Research

The first strand of work addresses the fundamental perceptual challenge: extracting linguistically meaningful representations from raw BSL video. A signer communicates simultaneously through hand shape (handshape), hand position relative to the body and signing space (location), direction and speed of movement (movement), palm/finger orientation, and a range of non-manual articulators — facial expression, eye gaze, mouth patterns, and head position.

Existing visual backbones are poorly suited to this multi-channel, fine-grained signal. SignGPT will develop purpose-built video encoders that factorize the sign stream into its phonological parameters, enabling the downstream transformer to reason at the appropriate level of linguistic abstraction.

A key research challenge is handling the continuous, co-articulated nature of natural signing: signs blur into one another without clear boundaries, and many sign classes differ only in subtle spatial or kinematic details. We will investigate learned temporal segmentation, phoneme-level tokenisation, and self-supervised objectives derived from BSL linguistic structure.

The centrepiece of SignGPT is a novel generative predictive transformer trained to model the joint distribution over visual sign tokens and their linguistic meaning. Inspired by the success of large language models for spoken language — which learn by predicting the next token — our architecture extends this paradigm to the visual-linguistic domain of sign language.

Bidirectional training is central to our approach: a model that can produce plausible BSL from English has, by necessity, learned a generative model of BSL structure, which in turn improves translation accuracy.

The flagship demonstration of this work is a real-time, two-way conversational interface — a sign language equivalent of ChatGPT — enabling Deaf users to interact with an LLM entirely in BSL. This conversational SignGPT system, integrating recognition, generation, and a large language model, is the programme's headline demonstration of end-to-end bidirectional sign language AI.

BSL is a full natural language with its own distinct grammar — it is not a manual encoding of English. Its grammatical devices include use of three-dimensional signing space to establish referents, and classifier/depicting constructions that indicate shape, size, and movement of objects [Schembri 2003; Cormier et al. 2012]. Non-manual articulators — including facial expression, mouthing, eye gaze, and head position — interact with manual signs to convey essential grammatical information, including distinctions between question types, negation, and discourse-level structure.

This strand of work ensures that the transformer architecture is informed by — and evaluated against — this rich structural knowledge. Specific contributions include: (i) theoretically-grounded linguistic descriptions of BSL grammar to inform model design and evaluation; (ii) annotated training data capturing classifier/depicting constructions and other grammatical phenomena; (iii) linguistically motivated evaluation metrics that go beyond surface BLEU scores.

Technical excellence alone does not guarantee that an AI system serves the community it is intended to benefit. This strand of work embeds Deaf community participation throughout the programme — not as a downstream validation step, but as a co-design activity that shapes research questions, dataset construction, and evaluation criteria from the outset.

We work with the British Deaf Association (BDA), the Royal Association for Deaf people (RAD), LumoTV, and Signapse Ltd to co-create recording protocols, annotation guidelines, and test sets that reflect the diversity of BSL usage. An Advisory Board — including Dr Andrew McParland (BBC R&D), Dr Oscar Koller (Microsoft Munich), Mark Wheatley (RAD), Graham Turner (BDA), and Dr Robert Adam (Heriot-Watt University) — provides programme-level oversight and strategic input.

Evaluation extends beyond automatic metrics to structured Deaf-user studies measuring intelligibility, naturalness, and communicative adequacy of system outputs. Community feedback is integrated into each major development cycle, creating a continuous co-design loop between researchers and end users.