Teaching ASR to keep the words that matter

Word error rate quietly rewards a transcriber for discarding exactly what a language tutor needs. If a learner says “I went to Yogyakarta with Rina,” a system that returns “I went to Jakarta with Rena” barely moves its WER — two short words out of seven — while getting wrong the only two things the sentence was about. The same is true in the other direction: every production ASR system strips um, uh, and false starts by design, and those are precisely the signal a fluency assessment is computed from.

This was the problem behind my first-author paper at INTERSPEECH 2026, Beyond WER: Entity and Disfluency Recall in Accented Conversational ASR, with Ankit Pandey and Yash Singh. The argument is that WER is the wrong optimisation target for a language-learning product, that entity recall and filler recall should be measured directly, and that doing so changes which model you would pick.

The pipeline

Curating entity-rich data with SQL rather than annotators. Only 24–28% of real learner utterances contain a proper noun, so random sampling produces training data that is mostly entity-free — you pay for annotation and buy very little of what you need. Instead I wrote lexical filters that run in the data warehouse before anything is labelled, keeping utterances of four or more words that contain consecutive capitalised words, acronyms, honorifics, or mid-sentence capitalisation. The filtered pool runs at 70–78% entity density, roughly 2.8× enrichment, which cut annotation effort by about 65%.

Regional adapters rather than one model. India, Indonesia, and Latin America fail differently, so each got its own LoRA adapter on Qwen2.5-Omni-3B — rank 32, attention projections only, base weights frozen, two epochs on 9k utterances. About six hours per adapter on a single A100.

Two transcripts from one forward pass. Each inference returns structured JSON containing both a verbatim transcript, which preserves fillers, accent-influenced pronunciations, and false starts, and a corrected one. A boolean comprehensible flag gates the output, so genuinely degraded audio returns empty strings instead of a confident hallucination. Every metric below is computed on the harder verbatim transcript.

A judge to diagnose the errors. A six-category error taxonomy — phonetic substitution, named-entity error, gibberish, hallucination, omission, non-actionable audio — scored by an LLM. I benchmarked eighteen candidate judge models against 210 human labels before selecting one at 83.8% exact agreement.

Results

Entity recall moved from 53–55% to 80–85%, filler recall from under 5% to 76–86%, and WER fell by 53–60% relative to the prior baseline. The fine-tuned 3B model matches or beats a zero-shot 30B model on entity recall with ten times fewer parameters, and serves at around 800ms P95 against roughly 16.5 seconds for a commercial API.

The result I find most interesting is the ablation. Training an otherwise identical adapter on randomly sampled data isolates what the curation stage was worth on its own: paired bootstrap tests over 10,000 iterations attribute +4.19pp, +2.84pp, and +2.76pp of entity recall to curation alone, all at p<0.0001. A day of writing SQL was worth more than most of the modelling.

What I would flag about it

The test-set references are silver-standard — generated by a large multimodal model, with only 250 utterances per region human-verified. A gap remains against the 30B model on filler recall specifically. And the evaluation is English-only, which leaves code-switching, constant in all three of these markets, entirely to future work.