🔍 Read the full analysis: Open ASR Leaderboard Expands With First Language From The Global South on ThorstenMeyerAI.com
TL;DR
The Open ASR Leaderboard on Hugging Face has expanded to include Hindi and Indian English, the first languages from the Global South on the platform. This development introduces new datasets with diverse speaker attributes, aiming to improve the evaluation of speech recognition models across populations.
The Hugging Face Open ASR Leaderboard has incorporated two new evaluation datasets, Monsoon en-IN for Indian English and Monsoon hi-IN for Hindi, making these the first Indic and Global South languages included on the platform. This development is discussed in the original analysis. This expansion aims to address the lack of diverse language representation in automatic speech recognition (ASR) benchmarks, providing a more comprehensive assessment of model performance across different populations.
The new datasets are designed with a focus on diversity, capturing speech from 4,888 speakers across India, with recordings collected from unscripted, spontaneous conversations. The datasets include public and private splits for self-scoring and benchmarking, with hours of audio from speakers across various regions, ages, genders, and socio-economic backgrounds. Learn more about the significance of diverse speech datasets in the original analysis. Each clip records 12 speaker attributes such as occupation, education, income, and device used, enabling detailed analysis of model bias and performance disparities.
Specifically, the Indian English set comprises approximately 11 hours of audio from over 2,800 speakers, while the Hindi set includes over 6 hours from nearly 1,000 speakers. The recordings feature diverse acoustic environments, speech styles, and vocabulary, reflecting real-world usage rather than scripted speech. The Hindi dataset introduces a lattice-based normalisation to handle spelling variations, addressing challenges unique to Indic languages.
The datasets were compiled with contributions from speakers across hundreds of districts, using their own devices and environments—indoors and outdoors—rather than controlled recording setups. This approach aims to better simulate practical deployment scenarios and uncover biases in current ASR models. For more context, see the detailed report on the original analysis.
Implications for Global South Language Representation
The inclusion of Hindi and Indian English on the Open ASR Leaderboard marks a significant step toward more equitable and representative benchmarks in speech recognition technology. Historically, most benchmarks have focused on European languages, limiting the development of models suitable for diverse linguistic and socio-economic contexts. By adding languages spoken by over half a billion people, this move signals a shift toward more inclusive AI development, encouraging models that perform well across different populations and dialects.
Moreover, the detailed speaker metadata enables researchers to analyze model biases related to age, gender, region, and device type, addressing concerns about disparities highlighted in prior research. This can lead to more trustworthy and fair ASR systems, especially vital for applications in education, healthcare, and governance in the Global South.
automatic speech recognition device
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background on ASR Benchmarks and Diversity Gaps
Until now, the Open ASR Leaderboard on Hugging Face primarily evaluated models on European languages, with limited representation from other regions. Prior studies, such as ‘Racial Disparities in Automated Speech Recognition,’ have shown that commercial systems perform worse for Black speakers and other underrepresented groups. The focus on standardised datasets has often overlooked linguistic and socio-economic diversity, resulting in models that may not generalise well across different populations.
The addition of Hindi and Indian English addresses this gap by providing datasets that reflect real-world variability in speech patterns, accents, and environmental conditions typical of speakers in India. The datasets were curated to include speakers from various districts, socio-economic backgrounds, and device types, aiming to challenge models to perform reliably across diverse scenarios.
This development aligns with ongoing efforts to improve fairness and reduce bias in AI, emphasizing the importance of inclusive benchmarks for advancing global AI capabilities.
“Adding Hindi and Indian English datasets to the Open ASR Leaderboard is a crucial step toward more inclusive and representative speech recognition benchmarks.”
— Thorsten Meyer, Lead Developer at Hugging Face
Outstanding Questions About Dataset Impact and Model Performance
It remains unclear how existing leading ASR models perform on these new datasets, as baseline results have not yet been published. The stability of model rankings given the relatively small size of the Hindi set (1.33 hours) also requires further validation. Additionally, the effectiveness of the lattice normalisation approach for Hindi compared to traditional methods has not been demonstrated through published comparisons. Whether the detailed speaker attributes will be used by participants to disaggregate results is also yet to be seen. Further research is needed to assess how these datasets influence model improvements and bias mitigation over time.
Next Steps for Benchmarking and Language Inclusion
The public splits are now available for self-scoring, and model developers are expected to evaluate their systems on these datasets. In the coming months, benchmark results from top models will reveal how well current systems handle Indian English and Hindi, providing insights into existing gaps. Researchers will likely explore the impact of speaker attributes on model performance and bias. Additionally, efforts may expand to include more languages from the Global South, further diversifying the benchmark landscape and encouraging the development of more equitable ASR technologies.
Key Questions
Why is the inclusion of Hindi and Indian English significant for speech recognition?
The inclusion of Hindi and Indian English addresses the lack of representation of languages spoken by over half a billion people, helping develop models that are more inclusive and effective across diverse populations.
What makes the new datasets different from previous benchmarks?
The datasets feature spontaneous, unscripted speech from a highly diverse set of speakers across India, with detailed metadata and multiple spelling variants for Hindi, reflecting real-world variability.
Will existing ASR models perform well on these new datasets?
Baseline results are not yet available, so it is unclear how current models will perform. The datasets are designed to challenge models with diverse speech and environmental conditions.
How might this development influence future AI research?
It encourages the development of more inclusive and fair speech recognition systems, potentially inspiring more datasets and benchmarks for underrepresented languages and regions.
Are there plans to include more languages from the Global South?
While not confirmed, this move sets a precedent that may motivate further inclusion of additional languages from the Global South in future benchmark expansions.
Primary source: Hugging Face · via ThorstenMeyerAI.com
