Word error rate: how to test transcription accuracy

We took one line from a sample meeting, changed four words by hand, and scored the pair four ways: 37.5%, 35.3%, 25.0% and 23.5%. Only the scorer's clean-up rules changed.

So a WER figure without its audio, reference and normaliser tells you little. It still matters: notes are written from the transcript, and AI meeting notes that cite the recording let you hear the line each one came from. Every source linked below was read on 16 October 2026.

The word error rate formula

WER = (S + D + I) / N. S is substitutions (a word replaced), D deletions (a word missing), I insertions (an extra word), C correct words, and N the words in the reference, a transcript a person wrote and checked; the software's output is the hypothesis. Hugging Face's evaluate writes it as "WER = (S + D + I) / N = (S + D + I) / (S + D + C)", "the average number of errors per reference word" (WER README).

Errors are counted after aligning the two with the fewest edits (a minimum edit distance). Comparing word 1 with word 1 fails: one word dropped early shifts every later word into an "error". NIST's sclite, "a tool for scoring and evaluating the output of speech recognition systems", weights correct words, insertions, deletions and substitutions at 0, 3, 3 and 4 when aligning (sclite docs); our code uses equal costs, so the two can split errors differently.

Why WER can go over 100%

WER passes 100% because insertions have no ceiling. A made-up example: reference "yes", transcript "yes yes yes" is 2 insertions over 1 word, so WER = 200%.

The Python scorer jiwer shows jiwer.wer('', 'peaceful silence') returning 2, its convention for an empty reference (N = 0 would divide by zero), not part of the formula. Character error rate is the same; Hugging Face's CER README says it is "not always a number between 0 and 1".

WER is not accuracy

"Accuracy = 100% minus WER" is common shorthand. In our view it misleads: it goes negative past 100% and hides what kind of error happened. Morris, Maier and Green (Interspeech 2004) wrote that WER "measures the cost of restoring the output word sequence" and argued that "a more meaningful performance measure would be given by the proportion of information communicated" (ISCA archive).

How to calculate word error rate, worked by hand

This pair is hand-made, not engine output. The line is Sara's decision at 1:11 in the sync we used to compare a transcript with minutes and notes.

Reference:  Okay, so that's a decision: notifications follow
Hypothesis: Okay so that's the decision, notification follows

Reference:  the account language. Lina, can that land
Hypothesis: the account language. Lena, can that land

Reference:  before Thursday?
Hypothesis: before Thursday?

1. Normalise both with our rules (next section). The reference goes from 16 words to 17: "that's" splits into "that" and "s".
2. Align them: 13 words match.
3. Count: 4 substitutions, 0 deletions, 0 insertions.
4. WER = 4 / 17 = 23.5%.

Add a filler, "before, um, Thursday", and that is 1 insertion: 5 / 17 = 29.4%.

Each substitution costs the same 5.9 points. Three change nothing anyone would act on. "Lina" to "Lena" changes who delivers by Thursday, and WER can't tell it from "a" to "the".

Normalisation: one transcript, four scores

Clean-up rules alone move this pair's score by 14 points:

Same pair as above, scored on 16 October 2026 with jiwer 4.0.0 and our own code. Rules per jiwer's transformations page.

With punctuation kept, "Okay," and "Okay" differ, as do "decision:" and "decision,"; "that's" is one word or two, so N is 16 or 17.

The Whisper paper says WER "penalizes all differences between the model's output and the reference transcript including innocuous differences in transcript style". After applying their own normaliser: "For several datasets, we observe WER drops of up to 50 percent" (arXiv 2212.04356). Often the cause was style, such as split contractions.

Our rules:

• lowercase;
• strip Latin accents and Arabic tashkeel and tatweel;
• fold أ, إ, آ and ٱ to ا, ى to ي and ة to ه;
• turn Arabic-Indic digits into ASCII;
• split on anything that isn't a letter or digit.

"Twenty" against "20" is still an error. Folding letter forms handles only part of a known problem: Ali, Khalifa and Habash (Interspeech 2019) note that Arabic dialect spelling isn't standardised, which makes "standard word error rate (WER) inadequate as an evaluation metric" (ISCA archive).

What is a good word error rate?

The only primary-sourced bands we found are on Microsoft Learn's page "Test accuracy of a custom speech model": "A WER of 5-10% is considered good quality and is ready to use. A WER of 20% is acceptable, but you might want to consider more training. A WER of 30% or more signals poor quality and requires customization and training." (Microsoft Learn).

That is Microsoft's guidance for testing Azure speech to text models, base or custom, not an industry standard, and the page adds: "How the errors are distributed is important." In our view they only hold on the same audio, reference and normaliser: our worked example sits just above the 20% "acceptable" mark at 23.5% and in the "30% or more" band at 37.5%, depending only on the scorer.

What we measured on four engines

We ran four speech-to-text engines on four synthetic sample meetings on 8 and 9 October 2026:

• Audio: macOS text-to-speech voices read our scripts, so each reference is exact.
• Runs: one per engine and sample, on 8 and 9 October 2026.
• Scoring: our normaliser, equal-cost alignment over the whole meeting. No CER.
• Columns: "speakers" is the share of speech time given to the right person; "line language" the share of lines in the right language.

• Small gaps are a few words. On the 655-word sync, 2.7% against 3.1% is roughly 18 errors against 20. The order also changes by sample, so we don't rank them.
• WER missed the biggest failures. Whisper scored 2.7% on the English sync with every line under one speaker. Nova-3 lost the Spanish lines on the mixed call. Spanish is about a fifth of that sample's words, so the loss shows up as a 22% WER, but the number alone doesn't say a whole language went missing.
• Settings matter. With "en" as a language hint, Melia 1 wrote an Arabic call as romanised English nonsense; without it, Arabic was detected correctly.

Limits: clean text-to-speech audio with no noise or crosstalk; small samples; no Gulf-dialect speakers; one run per engine. Real, noisy meetings will likely score worse; we haven't measured them.

What word error rate can't see

WER scores words, so it misses the errors that change the notes:

• Who said it. Speaker labels have their own metric, the diarisation error rate: (false alarm + missed detection + speaker confusion) / total speech time, which pyannote.metrics calls "the de facto standard metric for evaluating and comparing speaker diarization systems". On a real 9-minute call of ours (no reference, so no WER), the engine found three speakers for five people and an action item went to the wrong owner.
• Which word was wrong. Lina against Lena.
• Meaning. In a verbatim reference, a dropped "not" costs one word, like a dropped "um", yet a summary written from it says the opposite, which an AI summary accuracy check catches.

That is why cited notes link each line to the moment it was said: when a name or owner looks wrong, you play the second it came from instead of trusting a score. It is also the reason we cite every note.

How to test your own tool's transcription accuracy

1. Pick a 3 to 5 minute clip of a real meeting, with the participants' consent, containing names, numbers, jargon and any second language. You can record a browser tab in Chrome or Edge, with no bot.
2. Write the reference by hand. Listen twice, and write down your spelling rules for numbers, names and fillers first.
3. Run the same file through each tool, not a fresh recording per tool.
4. Normalise both sides the same way and note the rules.
5. Score it. This prints the 25.0% row above (checked with jiwer 4.0.0):

import jiwer

reference = "Okay, so that's a decision: notifications follow the account language. Lina, can that land before Thursday?"
hypothesis = "Okay so that's the decision, notification follows the account language. Lena, can that land before Thursday?"

clean = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToListOfListOfWords(),
])
out = jiwer.process_words(reference, hypothesis,
                          reference_transform=clean, hypothesis_transform=clean)
print(out.wer, out.substitutions, out.deletions, out.insertions)
print(jiwer.visualize_alignment(out))

6. Read the errors, not just the number. Mark the ones that change meaning: names, owners, numbers, dates, negations. Then check the summary's decisions, owners and dates against the call, because notes can be wrong even when the transcript is right.
7. Check speakers separately. For each action item, is the right person speaking?
8. Compare like with like. On a 300-word clip, one error moves WER by a third of a point.

Questions people ask

What is the difference between WER and CER?

CER applies the same formula to characters. The Hugging Face CER README: "CER is similar to Word Error Rate (WER), but operates on character instead of word."

Can word error rate be over 100%?

Yes. Word error rate has no ceiling because insertions count, so accuracy isn't 100% minus WER.

How do I calculate WER in Python?

Use jiwer or Hugging Face evaluate, and state which clean-up rules you used.

What is a good word error rate for meetings?

We found no sourced standard for meetings. Microsoft's bands are for its own models; test on your own audio.