SignRefine: Adapting Foundational Video Models for Sign Language Generation

Abstract

Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign language research, we present NVSign, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.

Problem & Motivation

Sign languages are the primary means of communication for Deaf communities worldwide and video is their natural medium. Automatic sign language video generation is therefore a prerequisite for accessible content creation. Yet the field faces a stark duality: general-purpose video models generalize beautifully but cannot sign, while sign-specific models do not generalize.

Meaning lives in a few pixels Hands and face carry the core linguistic content, but occupy only ~15% of each frame. In the patchified latent space where video diffusion models operate, these regions are aggressively compressed and their structural detail is overwhelmed by the global reconstruction objective.

Signing is out-of-distribution Foundational video models are trained on spoken-language video. The complex articulation and rapid, high-frequency handshape changes required for signing fall outside that distribution, so full-body conditioning alone cannot recover the lost definition — producing corrupted, under-articulated hands.

Specialist models overfit Sign-specific generators are trained from scratch on small, constrained corpora. They overfit to particular signer appearances, scales and studio environments, output low resolution, and often depend on dense modalities such as edge maps or posed 3D hand meshes that limit practical use.

Existing data is visually sterile Available datasets are dominated by interpreted broadcast or lab footage in static environments, carrying structural biases from the source spoken language and lacking the spatio-temporal complexity and fluidity of natural, native signing.

Contributions

1 · Generalizable pose-to-video signing The first generalizable pose-to-video generation approach designed to improve the comprehensibility of generated sign language content, rather than pixel fidelity alone.

2 · Local adapters with spatial grounding A novel conditioning scheme for DiT-based human video models that enables targeted refinement of hand and face regions without degrading the quality or generalizability of the frozen backbone.

3 · NVSign dataset A large-scale, multi-modal, natively signed dataset with extensive multi-party conversational content, diverse appearances and dynamic camera work.

4 · Evaluation with sign language users Rigorous quantitative evaluation plus perceptual and comprehension studies with signers of varying proficiency, demonstrating that the generated video is measurably more understandable — not just more attractive.

Architecture

SignRefine is built on Wan2.1-1.3B-Fun-Control, a pretrained video DiT conditioned on skeleton sequences. The full-body skeleton drives global body movement effectively, but the backbone lacks precision exactly where sign language needs it. We keep the backbone frozen, preserving its quality and generalization, and add trainable local adapters for hand and face regions. They extract regional conditions at higher resolution and inject them into selected DiT blocks through masked cross-attention.

Fig. 2 — A reference image and a full-body skeleton sequence guide the DiT backbone. Regional conditions for the face, left hand and right hand are processed by separate encoders and injected into the frozen backbone through trainable local adapters.

Regional Conditions with Spatial Grounding

Per-channel condition encoders Signing operates over two articulatory channels with different morphology: manual (hands) and non-manual (face). Each region is therefore processed by an independent encoder, each free to specialize its representation.

Magnified regional crops Each encoder sees its region rendered at 256×256 — far larger than the area the hands occupy in the original frame — recovering the structural detail lost to latent compression. A ResNet-style backbone downsamples to a 16×16 grid and causal 1D convolutions compress T frames to T/4, aligning the condition with the backbone's latent temporal resolution.

CoordConv spatial grounding A crop loses the memory of where it came from. We inject each pixel's absolute normalized coordinate in the original frame alongside the RGB channels, embedding the crop-to-frame geometric mapping directly into the regional features.

Localized Feature Injection

Interleaved cross-attention Face and hand adapters inject at offset intervals across the 30 transformer blocks — face at {0, 6, 12, 18, 24}, hands at {2, 8, 14, 20, 26} — so no single block absorbs multiple adapter residuals and the conditioning load is distributed through the network. Left and right hand conditions are concatenated before injection to resolve inter-hand dependencies and occlusions.

Masked attention with a sink token A binary spatial mask separates articulator patches from background on the latent grid. Foreground queries attend to the high-resolution regional features; background queries are routed to a learned sink token that safely absorbs them, preventing the adapter from bleeding hand or face features into unrelated areas.

Region-weighted training objective Output projections are zero-initialized for stability on the frozen backbone. The MSE loss is weighted per region, raising the articulators' share of the gradient to match their linguistic importance while retaining roughly a third of it on the background to preserve global coherence and identity.

NVSign Dataset

44,759

Signing segments

78.1 h

High-res video

2,184

Unique signers

~40%

Conversational

NVSign is a large-scale dataset of web video content natively produced in sign language: 44,759 signing segments sourced from 317 longer videos, in varied real-world environments with dynamic camera work. Roughly 40% of the data involves two or more participants, unlocking research directions that are impossible with single-signer corpora: multi-signer dynamics, turn-taking, signer diarisation, and non-manual features used as backchannels and discourse markers.

Each video ships with a comprehensive multi-modal annotation set — time-aligned English transcripts, extracted 2D body keypoints, per-signer activity scores, depth maps, SMPL body, MANO hand and FLAME face parameters. Three data partitions navigate different trade-offs between signer overlap and content diversity, enabling nuanced appearance-independent evaluation.

Quantitative Results

Because the goal is articulator quality, evaluation concentrates on regional metrics: MPJPE for keypoint accuracy and structural precision, LPIPS and SSIM for perceptual quality, computed on face and hand crops. NVSign appearance-independent test set, Phoenix14T and BSL Corpus are used for testing.

Regional quality across three datasets

Method	Face			Right Hand			Left Hand
Method	MPJPE ↓	LPIPS ↓	SSIM ↑	MPJPE ↓	LPIPS ↓	SSIM ↑	MPJPE ↓	LPIPS ↓	SSIM ↑
NVSign
SignGAN	10.424	0.564	0.302	24.774	0.627	0.388	21.852	0.620	0.384
SignViP	10.576	0.534	0.356	23.957	0.581	0.428	24.383	0.588	0.424
Wan-VACE	4.067	0.422	0.367	15.904	0.525	0.380	15.431	0.503	0.407
Wan-FC	1.835	0.197	0.713	9.565	0.364	0.583	9.341	0.349	0.597
SignRefine	1.770	0.194	0.714	6.602	0.330	0.629	6.638	0.321	0.640
Phoenix14T
SignGAN	5.457	0.585	0.343	28.507	0.707	0.469	25.373	0.657	0.470
SignViP	7.194	0.456	0.276	10.345	0.534	0.425	14.405	0.566	0.376
Wan-VACE	2.162	0.383	0.413	11.717	0.551	0.367	13.214	0.546	0.299
Wan-FC	1.151	0.223	0.688	5.754	0.402	0.629	5.719	0.374	0.642
SignRefine	0.916	0.214	0.701	4.582	0.386	0.660	5.101	0.366	0.668
BSL Corpus
SignGAN	9.415	0.634	0.330	22.726	0.720	0.406	16.858	0.689	0.390
SignViP	28.222	0.626	0.289	23.085	0.611	0.381	22.111	0.593	0.372
Wan-VACE	3.172	0.436	0.380	17.524	0.567	0.351	15.385	0.557	0.338
Wan-FC	1.790	0.221	0.685	8.232	0.389	0.559	7.002	0.373	0.563
SignRefine	1.524	0.222	0.690	6.314	0.365	0.608	5.406	0.353	0.607

Table 1 — SignRefine is consistently outperforming the baselines on all the metrics. Gains are most pronounced on the hands — roughly a 30% reduction in MPJPE over the strongest baseline on NVSign, with face precision improving by up to 20% on Phoenix14T and 15% on BSL Corpus. Note the generalization pattern: SignViP, trained exclusively on Phoenix14T, collapses on NVSign and BSL Corpus, while SignRefine holds stable performance across all three without any dataset-specific adaptation.

User study

Two studies complement the automated metrics. In the perceptual study, fifteen participants of varying BSL proficiency answered 17 questions each, ranking models by naturalness and hand/face quality, plus semantic fidelity against an English reference for intermediate-or-higher signers. SignRefine obtains the lowest mean rank overall and in every category (Table 2); SignRefine wins more than 83% of head-to-head comparisons against each baseline, giving the ordering Ours > Wan-FC > Wan-VACE > SignGAN ≈ SignViP. In the comprehension study, six fluent BSL signers wrote blind free-text translations of ten clips rendered by SignRefine and Wan-FC. Viewers of SignRefine recover more key content, land closer to the reference translation and abandon fewer clips as unintelligible (Table 3).

Category	Ours	Wan-FC	Wan-VACE	SignViP	SignGAN
Overall ↓	1.22	1.89	2.46	3.56	3.41
NVSign ↓	1.16	1.95	2.41	3.81	3.23
Phoenix14T ↓	1.41	1.89	2.60	2.65	3.95
Text Ref. ↓	1.16	1.63	2.44	3.76	3.22

Table 2 — Mean rank assigned by participants (1 = best, 5 = worst).

Metric	Ours	Wan-FC
Keyword Recall ↑	0.54	0.30
BLEURT ↑	0.31	0.19
LLM adequacy ↑	2.07	1.50
Failure Rate ↓	0.17	0.27

Table 3 — Comprehension study, SignRefine vs. Wan-FC.

Acknowledgements

This work was supported by EPSRC grant APP24554 (SignGPT — EP/Z535370/1), EPSRC grant APP78083 (UMCS — UKRI3927), and through funding from Google.org via the AI for Global Goals scheme. The authors acknowledge the use of Isambard-AI National AI Research Resource (AIRR), funded by UK DSIT via UKRI and STFC [ST/AIRR/I-A-I/1023]. This work reflects only the authors' views; the funders are not responsible for any use that may be made of the information it contains.

BibTeX

arXiv preprint — the proceedings entry will be added once published.

@misc{pelykh2026signrefine,
  title         = {SignRefine: Adapting Foundational Video Models
                   for Sign Language Generation},
  author        = {Anton Pelykh and Edward Fish and
                   Ozge Mercanoglu Sincan and Richard Bowden},
  year          = {2026},
  eprint        = {2609.08496},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.08496}
}