Start with the production decision
Specify what the transcript will support: searchable meetings, captions, customer-service records or published quotations. The consequence of an error changes which examples and metrics belong in the evaluation.
List expected languages, accents, microphones, rooms, background sounds, speaker counts and vocabulary. Sample those conditions deliberately; a convenient archive of clean read speech is unlikely to represent a noisy, overlapping workplace call.
Keep the holdout genuinely held out
Separate speakers from training and development sets. Where possible, collect evaluation audio after the model's training or tuning period and from sources not used to build prompts or dictionaries. Track provenance so accidental reuse can be detected.
Do not repeatedly tune against every failed holdout example. Maintain a development set for iteration, a frozen comparison set and a rotating set of fresh cases. This preserves learning while reducing the chance that the evaluation becomes another training target.
Audit references before scoring
A word error rate calculation assumes the reference transcript is correct. Review a sample at minimum, and route disagreements among systems to independent listeners. Record whether punctuation, casing, disfluencies and numbers are normalized.
ArmBench-ASR reports strict and normalized views because each answers a different question. Hume's recent work goes further by probing cases where models reproduce an erroneous public reference. Both approaches show why a single score can conceal the source of apparent accuracy.
Weight errors by impact
Calculate WER and character error rate, then separately score names, amounts, dates, negation and domain terms. Add diarization, timestamp, latency and long-form stability when the product uses them.
Create an error taxonomy and review the worst categories. One wrong medication or customer identifier can matter more than many missing filler words, so product acceptance should use risk-weighted thresholds alongside aggregate accuracy.
Publish a reproducible evaluation record
Save model identifiers, API dates, decoding settings, language hints, preprocessing and the exact metric implementation. Hosted systems can change, making a date and configuration necessary for later comparison.
Report distributions by condition rather than only an overall rank, disclose private portions of the dataset and state what was not measured. The goal is not a universal winner; it is evidence that a system works for a defined population and workflow.
Sources & further reading
Social-media activity is treated as a signal of attention, not proof. Product claims are attributed to the linked publisher or announcement.