SignON | Aditya Parikh

SignON (May 2021 – December 2023) was a European H2020 user-centred project that aimed to facilitate the exchange of information between deaf, hard-of-hearing, and hearing people across Europe. I was a Junior Researcher working on the Inclusive AI / ASR side of the project.

What I worked on

Designed and deployed ASR components for a multilingual, multimodal mobile application supporting Deaf and Hard of Hearing (DHH) users.
Developed and optimised ASR models for multiple languages, including under-resourced languages, adapting systems to handle diverse and non-standard speech patterns.
Delivered low-latency ASR services via REST APIs on on-premises infrastructure, meeting real-time performance requirements for mobile applications.
Evolved ASR pipelines from traditional hybrid systems to modern self-supervised and end-to-end models, improving robustness and maintainability.
Applied data augmentation and ensemble techniques to enhance model performance and reliability in low-resource and noisy conditions.

Languages covered

Spoken/written: English, Irish, Dutch, Spanish Sign languages: ISL, BSL, NGT (Dutch), VGT (Flemish), LSE (Spanish)

(Vandeghinste et al., 2023; Shterionov et al., 2022; Parikh et al., 2022; Parikh et al., 2023)

2023

EAMT

SignON: Sign Language Translation. Progress and challenges

Vincent Vandeghinste, Dimitar Shterionov, Mirella De Sisto, and 20 more authors

In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, Jun 2023

Abs Bib

SignON is a Horizon 2020 project, running from 2021 until the end of 2023, which addresses the lack of technology and services for the automatic translation between sign languages (SLs) and spoken languages, through an inclusive, human-centric solution, hence contributing to the repertoire of communication media for deaf, hard of hearing (DHH) and hearing individuals. In this paper, we present an update of the status of the project, describing the approaches developed to address the challenges and peculiarities of SL machine translation (SLMT).

@inproceedings{vandeghinste-etal-2023-signon,
  title = {{S}ign{ON}: Sign Language Translation. Progress and challenges},
  author = {Vandeghinste, Vincent and Shterionov, Dimitar and Sisto, Mirella De and Brady, Aoife and Coster, Mathieu De and Leeson, Lorraine and Blat, Josep and Picron, Frankie and Scipioni, Marcello Paolo and Parikh, Aditya and ten Bosch, Louis and O'Flaherty, John and Dambre, Joni and Rijckaert, Jorn and Vanroy, Bram and Nogales, Victor Ubieto and Gomez, Santiago Egea and Schuurman, Ineke and Labaka, Gorka and Núnez-Marcos, Adrián and Murtagh, Irene and McGill, Euan and Saggion, Horacio},
  booktitle = {Proceedings of the 24th Annual Conference of the European Association for Machine Translation},
  month = jun,
  year = {2023},
  address = {Tampere, Finland},
  publisher = {European Association for Machine Translation},
  url = {https://aclanthology.org/2023.eamt-1.53/},
  pages = {501--502}
}

ICNLSP
Comparing Modular and End-To-End Approaches in ASR for Well-Resourced and Low-Resourced Languages

Aditya Parikh, Louis Bosch, Henk Heuvel, and 1 more author

In Proceedings of the 6th International Conference on Natural Language and Speech Processing (ICNLSP 2023), Dec 2023

Abs Bib

We present a comparative study of a state-of-the-art traditional modular Automatic Speech Recognition (Kaldi ASR) and an end-to-end ASR (wav2vec 2.0) for a well-resourced language (Spanish) and a low-resourced language (Irish). We created ASRs for both languages and evaluated their performance under different update regimes. Our results show that the end-to-end wav2vec 2.0 outperforms the modular ASR for both languages in terms of Word Error Rate (WER) but performs worst in terms of real-time decoding. We also addressed the issue of non-lexical words in wav2vec 2.0’s output. We found that in wav2vec 2.0 by LM integration with shallow fusion and increasing LM weight to 0.7 and 0.8 respectively for the Spanish and Irish provided the optimum ASR performance by reducing non-lexical words. However, this does not eliminate all non-lexical words. Finally, our study found that Kaldi ASR would perform best for real-time decoding for longer audio inputs compared to wav2vec 2.0 model trained on the same dataset on the minimal infrastructure, although wav2vec 2.0’s performance can be improved with a GPU acceleration in backend. These results may have significant implications for creating real-time ASR services, especially for low-resourced languages.
@inproceedings{parikh-etal-2023-comparing, title = {Comparing Modular and End-To-End Approaches in {ASR} for Well-Resourced and Low-Resourced Languages}, author = {Parikh, Aditya and ten Bosch, Louis and van den Heuvel, Henk and Tejedor-Garcia, Cristian}, booktitle = {Proceedings of the 6th International Conference on Natural Language and Speech Processing (ICNLSP 2023)}, month = dec, year = {2023}, address = {Online}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.icnlsp-1.28/}, pages = {266--273} }

2022

EAMT

Sign Language Translation: Ongoing Development, Challenges and Innovations in the SignON Project

Dimitar Shterionov, Mirella De Sisto, Vincent Vandeghinste, and 11 more authors

In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, Jun 2022

Abs Bib

The SignON project focuses on the research and development of a Sign Language (SL) translation mobile application and an open communications framework. SignON rectifies the lack of technology and services for the automatic translation between signed and spoken languages, through an inclusive, human-centric solution which facilitates communication between deaf, hard of hearing (DHH) and hearing individuals.

@inproceedings{shterionov-etal-2022-sign,
  title = {Sign Language Translation: Ongoing Development, Challenges and Innovations in the {S}ign{ON} Project},
  author = {Shterionov, Dimitar and De Sisto, Mirella and Vandeghinste, Vincent and Brady, Aoife and De Coster, Mathieu and Leeson, Lorraine and Blat, Josep and Picron, Frankie and Scipioni, Marcello Paolo and Parikh, Aditya and ten Bosh, Louis and O'Flaherty, John and Dambre, Joni and Rijckaert, Jorn},
  booktitle = {Proceedings of the 23rd Annual Conference of the European Association for Machine Translation},
  month = jun,
  year = {2022},
  address = {Ghent, Belgium},
  publisher = {European Association for Machine Translation},
  url = {https://aclanthology.org/2022.eamt-1.52/},
  pages = {325--326}
}

CLIN
Design Principles of an Automatic Speech Recognition Functionality in a User-centric Signed and Spoken Language Translation System

Aditya K. Parikh, Louis F. M. Bosch, Henk Heuvel, and 1 more author

Computational Linguistics in the Netherlands Journal, Dec 2022

Abs Bib

The European project SignON aims at designing a user-oriented and community-driven platform for communication among deaf, hard of hearing, and hearing individuals in both sign language and spoken languages. Inclusion, easy access to translation services and the use of state-of-the-art Artificial Intelligence (AI) are the key aspects of the platform design. This paper addresses the current state-of-the-art ASR component in SignON and the conceptual choices underlying the design, operation, and integration of the ASR component in the SignON application.
@article{parikh2022design, title = {Design Principles of an Automatic Speech Recognition Functionality in a User-centric Signed and Spoken Language Translation System}, author = {Parikh, Aditya K. and ten Bosch, Louis F. M. and van den Heuvel, Henk and Tejedor García, Cristian}, journal = {Computational Linguistics in the Netherlands Journal}, volume = {12}, year = {2022}, month = dec, pages = {19--32}, url = {https://www.clinjournal.org/clinj/article/view/145} }

Funder: European Commission (H2020)
Duration: May 2021 – December 2023
Website: signon-project.eu

What I worked on

Languages covered

Related publications

2023

2022