SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation

Abstract

Current sign language translation (SLT) systems attempt to learn all aspects of signing—manual gestures, high-speed fingerspelling, and asynchronous non-manual facial cues—within a single end-to-end network. Learning multiple tasks without detailed supervision leads to poor recognition of fingerspelled proper nouns and technical terms, and leaves rich disambiguating information from lip movements largely unexploited.

We introduce SignBind-LLM, a modular framework that addresses these limitations through three dedicated expert streams: one for continuous signing, one for fingerspelling, and one for lipreading. Each expert is pre-trained independently using CTC on approximately two million automatically generated pseudo-gloss sequences, removing the need for manual gloss annotation. A lightweight transformer with learned temporal alignment fuses the expert outputs, and a pre-trained language model translates the resulting pseudo-gloss sequences into fluent spoken English. At matched decoder scale (250M parameters), our architecture already surpasses all prior methods, confirming that the gains are architectural rather than a consequence of scaling the language model. Scaling to a larger decoder sets a new state-of-the-art across How2Sign: 23.1, BOBSL: 7.0, and ChicagoFSWild+: 73.6%, while requiring significantly lower training cost than prior approaches.

Method

Multi-Stream Expert Architecture Three dedicated processing streams—continuous signing, fingerspelling, and lipreading—each pre-trained independently via CTC. A dynamic router with straight-through Gumbel-Softmax assigns video segments to the appropriate manual expert during both training and inference.

Automatic Pseudo-Gloss Generation Approximately 2 million synthetic pseudo-gloss sequences are generated from subtitle text using GPT-4o, providing weak CTC supervision without any manual gloss annotation. Pseudo-glosses capture sign-language word order by removing function words and reordering toward subject–object–verb structure.

Temporal-Aware Fusion A cross-attention mechanism with learned projection matrices dynamically fuses the manual and non-manual streams, handling variable temporal asynchrony without fixed alignment assumptions. Manual-stream positions query the lipreading stream to retrieve the most relevant non-manual context, reflecting the linguistic structure of signing.

LLM Decoding A fine-tuned encoder–decoder language model (Flan-T5-XL, 3.0B parameters) translates fused pseudo-gloss sequences into fluent English. The LLM is trained independently on text pairs and fully decoupled from the visual pipeline, enabling flexible swapping of decoder sizes without retraining the visual encoders.

Quantitative Results

How2Sign — Sign Language Translation

Method	LM Decoder	#P	B-1 ↑	B-4 ↑	R-L ↑
GloFE-VN	—	—	14.9	2.2	12.6
MSLU	mT5-B	582M	20.1	2.4	17.2
SLT-IV	Trans.	≈30M	34.0	8.0	—
C²RL	mBART	680M	29.1	9.4	27.0
FLa-LLM	mBART	680M	29.8	9.7	—
YouTube-ASL	T5-B	250M	37.8	12.4	—
SignMusk	mT5-B	250M	41.5	14.3	36.0
Uni-Sign	mT5-B	582M	40.2	14.9	36.0
Geo-Sign	mT5-B	582M	40.8	15.1	35.4
SSVP-SLT	T5-B	250M	43.2	15.5	38.4
Ours_base	FT5-B	250M	45.0	16.5	31.5
Ours_large	FT5-L	780M	46.7	18.6	36.9
Ours_xl (SignBind-LLM)	FT5-XL	3.0B	50.2	23.1	42.7

At matched decoder scale (250M), SignBind-LLM_base already surpasses all prior methods including those with 582M decoders, demonstrating that gains are architectural. Scaling to Flan-T5-XL (3.0B) sets a new SOTA with +7.6 B-4 over SSVP-SLT at only 259 GPU-hours total training cost.

BOBSL — British Sign Language Translation

Method	B-1 ↑	B-4 ↑	R-L ↑
GFSLT	—	0.6	7.4
Sign2GPT	—	0.9	11.4
Albanie et al.	12.8	1.0	10.2
Sincan et al.	18.8	1.3	8.9
Lost in Translation	—	2.6	15.6
Ours (SignBind-LLM)	28.5	7.0	25.8

SignBind-LLM achieves 7.0 B-4 and 25.8 R-L on BOBSL, improving B-4 by +4.4 over the previous best, confirming that multi-stream fusion generalises from ASL to BSL without architectural modification.

ChicagoFSWild+ — Fingerspelling Recognition

Method	Letter Accuracy ↑
FS Recognition	41.2
Iterative Attention	46.7
TDC-SL	50.0
CtoML	54.9
FS Attention	57.8
FSS-Net	64.4
FS PoseNet	71.1
Ours (SignBind-LLM)	73.6

SignBind-LLM achieves 73.6% letter accuracy on ChicagoFSWild+, a 2.5 pp improvement over the previous best (FS PoseNet: 71.1%), despite not being specifically designed as a fingerspelling system. Precision 76.2%, Recall 70.4%.

BibTeX

@inproceedings{thomas2026signbind,
  title     = {SignBind-LLM: Multi-Stage Modality Fusion
               for Sign Language Translation},
  author    = {Thomas, Marshall and Fish, Edward and Bowden, Richard},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}