CSLR-XL: Scaling Continuous Sign Language Recognition with Synthetic Pretraining

Abstract

Continuous sign language recognition (CSLR) is a crucial task in sign language understanding, providing the temporally ordered, sign-level supervision that downstream tasks such as translation can build upon. Unlike sign language translation, which maps signing to spoken-language text, CSLR focuses on aligning the visual stream to its underlying sequence of signs. Yet its applications remain constrained by the scarcity of annotated data. In this work, we present CSLR-XL, a framework that addresses these gaps through synthetic pretraining and visually grounded pseudo-glossing. We introduce SignSynth-X, a large-scale synthetic corpus of densely annotated continuous pose sequences produced by stitching sign spottings sourced from continuous video. Unlike prior isolated-dictionary stitching, our method approximates natural co-articulation between signs, enabling the learning of sign priors from synthetic data. To demonstrate the value of the resulting synthetic sequences, we train a multi-stream skeleton-based recogniser. Relative to the MSKA recogniser, our EHM representation and synthetic pretraining together reduce test WER by 3.6 points on CSL-Daily and 5.2 on MeineDGS, of which synthetic pretraining alone contributes 2.6 and 2.5, respectively. The model remains competitive with skeleton-based methods on Phoenix2014T. We further introduce a force-aligned CTC pseudo-glosser, trained only on synthetic data, that scores complete candidate gloss sequences against the visual stream. On Phoenix2014T, it reduces pseudo-gloss WER from 60.9 to 41.8 compared with the strongest prior baseline, an absolute improvement of 19.1 points and a relative reduction of 31%. SignSynth-X and our code will be made publicly available.

SignSynth-X Examples

Stitched synthetic sequences from SignSynth-X, visualising the LBS-skinned EHM mesh overlaid with frame-level BIO spans, sign segmentation, and gloss annotation tiers — one example per benchmark subset.

Phoenix-Synth — German Sign Language (DGS)

CSL-Synth — Chinese Sign Language (CSL)

MeineDGS-Synth — German Sign Language (DGS)

BSLCP-Synth — British Sign Language (BSL)

BIO tier: Begin (green), In (yellow), Out (grey) frames · Seg tier: sign spans S₀…S_N · SS tier: gloss boundaries.

Quantitative Results

41.8WER

        Pseudo-gloss WER on Phoenix2014T — our force-aligned CTC pseudo-glosser
        reduces WER from 60.9 (strongest prior baseline) to 41.8, a
        19.1-point absolute improvement and a 31% relative reduction,
        demonstrating the value of grounding pseudo-glosses in visual evidence.
      

Continuous Sign Language Recognition — Skeleton-Based Methods (WER %)

Method	Phoenix2014		Phoenix2014T		CSL-Daily		MeineDGS		BSLCP
Method	Dev	Test	Dev	Test	Dev	Test	Dev	Test	Dev	Test
TwoStream-SLR	28.6	28.0	27.1	27.2	34.6	34.1	–	–	–	–
SignBERT+	34.0	34.1	32.9	33.6	–	–	–	–	–	–
CoSign-1s	20.9	21.2	20.4	20.6	29.5	29.1	–	–	–	–
MSKA	21.7	22.1	20.1	20.5	28.2	27.8	54.1^†	54.7^†	71.7^†	71.3^†
CSLR-XL (Ours)	20.4	21.3	19.1	20.5	25.5	24.2	49.1	49.5	65.6	64.9

CSLR results across five benchmarks; lower WER is better. ^† We train MSKA on MeineDGS and BSLCP, as the original work reports no results on these benchmarks. CSLR-XL attains the lowest CSL-Daily test WER (24.2) among reported methods and the strongest skeleton-only results on MeineDGS and BSLCP, while remaining competitive on Phoenix2014/2014T.

Effect of Synthetic Pretraining Scale — CSL-Daily (WER %)

Synthetic Scale	Dev WER ↓	Test WER ↓
None	27.5	26.8
2.5×	26.8	25.5
5×	25.7	24.9
7.5×	25.5	24.4
10×	25.5	24.2

Effect of synthetic pretraining data scale on CSL-Daily, relative to the real training set. Improvements become substantial at around 2.5× the real training set and continue to improve with further scale.

Acknowledgements

This work was supported by EPSRC grant APP24554 (SignGPT — EP/Z535370/1), EPSRC grant APP78083 (UMCS — UKRI3927), and through funding from Google.org via the AI for Global Goals scheme. The authors acknowledge the use of Isambard-AI National AI Research Resource (AIRR), funded by UK DSIT via UKRI and STFC [ST/AIRR/I-A-I/1023]. This work reflects only the authors' views; the funders are not responsible for any use that may be made of the information it contains.

BibTeX

@inproceedings{sahin2027cslrxl,
  title     = {CSLR-XL: Scaling Continuous Sign Language Recognition
               with Synthetic Pretraining},
  author    = {Sahin, Karahan and Sincan, Ozge Mercanoglu and
               Fish, Edward and Bowden, Richard},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on
               Applications of Computer Vision (WACV)},
  year      = {2027},
}