Speaker diarization: why transcripts mix up speakers

A transcript that says Speaker 2 doesn't know who Speaker 2 is. Diarization only decides that two stretches of audio sound alike. When it's wrong, the transcript still reads cleanly and a task lands on the wrong person.

It matters most for notes that cite the transcript line they came from: a citation can play the right moment and still show the wrong name. Sources below were read on 29 October 2026.

What speaker diarization does: Speaker 1, Speaker 2

The common approach, described in a review by Park et al., turns each short stretch of speech into a voice fingerprint: numbers that describe how it sounds. Similar fingerprints are grouped, and the engine also decides how many groups there are. Too few groups and two people merge; too many and one person splits.

Each group gets a number: these are the speaker labels in a transcription. In MeetNotely, speakers are numbered in the order they first speak, and an unnamed speaker shows as "Speaker 1", "Speaker 2" in the reader's language (المتحدث 1 in Arabic).

Speaker diarization vs speaker identification

Speaker diarization separates voices and numbers them; speaker identification matches a voice to a known person, which needs something to match against, such as a voice profile. Microsoft's help page on Teams meeting rooms shows what happens without that step: "Without speaker recognition, audio will be attributed to the room in AI notes." (Microsoft Support) Everyone in that room shares one label.

Diarization vs separate channels

When each person is recorded on their own channel, the labels can come from the channel instead of the voice. Deepgram's docs draw the line: diarization "focuses on giving information about different speakers," while multichannel "focuses on identifying different audio channels." (Deepgram docs) A single mixed recording has only the voices to go on.

Why transcripts mix up speakers

Mix-ups happen when the voices give the engine too little to separate. Five conditions make that more likely:

1. People talk over each other. AssemblyAI's documentation: "Overlapping speech between speakers can reduce diarization accuracy." (AssemblyAI docs) Park et al. explain why for clustering: "A clustering-based speaker diarization method can handle a large number of speakers but unable to handle overlapped speech." (arXiv 2101.09624)
2. Turns are short. AssemblyAI's page also advises that "each speaker should have at least 30 seconds of continuous speech." That is one vendor's guidance, but someone who only says "yes" twice gives any engine little to group.
3. One microphone serves a room. The Teams room above, or several people around one laptop.
4. The engine is told the wrong number of speakers. AssemblyAI again: "Only set speakers_expected when you are certain of the exact speaker count."
5. Everyone arrives on one mixed track. When you record from the browser, the other voices arrive already mixed by the meeting app, and our recorder adds your microphone to that same track. This one is our reasoning; we haven't measured its effect.

Of these, only crosstalk was ruled out in the mix-up we traced: the audio was one mixed track, and no turn reached AssemblyAI's 30 seconds.

A mix-up we traced: five voices, no crosstalk

This is a 9-minute sample meeting we scripted and voiced with text-to-speech: five people, 59 scripted turns, one mono track, with a short silence between every line (0.25 to 0.7 seconds), so nobody ever talks over anyone. Three voices are female (Karen, Helen, Jordan) and two male (Greg, Rishi).

On 9 October 2026 we ran it through two speech-to-text engines, one run per engine and setting:

• One engine returned three speakers for five people. Our notes record Karen and Helen as merged; three labels for five voices means at least one more pair was too.
• With its speaker sensitivity set to 0.7, the same engine returned four, still merging Jordan into Karen.
• Another engine returned six labels for five voices: it kept Jordan (first heard at 0:26) and Helen (0:39) apart, and one voice or stretch got a label of its own.

Notes written from the three-speaker transcript gave one of the four action items to the wrong person. We didn't record which item, which second pair or what the sixth label held, and we counted labels rather than scoring them.

Our judgement: every merge we recorded is among the female voices, so similar-sounding voices are a plausible reason that we haven't measured. AssemblyAI's documentation says the same of its own engine: "If speakers sound similar, the model may have difficulty distinguishing between them."

How we scored speakers on four samples (and why it isn't DER)

We scored four more meetings we scripted and voiced with text-to-speech, one run per engine, on 8 and 9 October 2026. Each figure is the share of speech time given to the right speaker; "N found" is how many speakers the engine returned when that didn't match the script.

OpenAI Whisper large-v3-turbo returned no speaker labels on any sample.

We add up the time each found speaker overlaps each scripted one, pair them one to one (largest overlap first), and divide the paired time by all of it: roughly 1 minus speaker confusion. A merge costs the time of whoever lost the label. The order changes by sample, so we don't rank engines. The Arabic row was built to be unfair and says nothing about real Arabic speakers.

The diarization error rate (DER) adds false alarm (non-speech counted as speech), missed detection (speech counted as non-speech) and speaker confusion, divided by total speech time; pyannote.metrics calls it "the de facto standard metric for evaluating and comparing speaker diarization systems." Our score counts only speaker confusion, and our reference times are exact, which hand-labelled references for real meetings aren't. The diarization error rate, set out next to word error rate, shows what each metric leaves out.

Merged, split or misnamed: what each does to your notes

Each error does different damage:

• Merged: two people under one label, so both people's lines and tasks show one name.
• Split: one person spread over two labels, so their lines and tasks carry two names.
• Misnamed: a label someone named after the wrong person. The grouping is right, and renaming fixes it.

In MeetNotely, transcripts with speaker names you can rename work per meeting: "Rename speakers" gives one field per speaker. Action items are linked to the speaker, so a rename changes the owner shown on every action item and the name on every transcript line of that speaker.

You can't split a merged speaker, move a line or set an owner directly, so renaming a merged label gives both people one name. Notes text written before the rename keeps the old label. For example, "Speaker 2 will send the deck" stays.

Timestamps survive a merge or a rename: a note's timestamp is matched by words, not by speaker.

What to do when the speaker labels look wrong

Find out which error you have first. The Speakers card shows each label's share of talk time (hover it for the number of turns). If one label has far more talk time than that person had, suspect a merge: click that speaker in the card to show only their lines in the transcript, and listen to a few.

1. If you hear two voices under one label, renaming won't help.
2. For a merge or a split, correct the owner where you share the notes, and say so.
3. Check each action item's owner against the call; owners are where a wrong label costs most.
4. For formal minutes, minutes that start from a transcript with speaker names need those names checked first.

For the next call, fewer people per microphone and less talking over each other help.

What our tests can't tell you

They show a merge can happen on clean audio, not how often.

• Every sample is scripted and voiced with text-to-speech, one run per engine. Real meetings will likely do worse; we haven't measured them.
• The five-voice run has label counts only.
• Our score isn't comparable with published DER figures.
• We haven't measured mix-ups from the browser's mixed tab-and-microphone track.

Questions people ask

Why does my transcript put two people under one name?

Overlapping speech, short turns, one shared microphone and, plausibly, similar-sounding voices (AssemblyAI's documentation warns about them too) can merge two people, even on clean audio.

Can renaming a speaker fix a merged speaker?

No. Renaming changes the name on a speaker label, so both people merged under it get the new name. MeetNotely can't split a speaker, so note the correction when you share the notes.

Is speaker diarization the same as speaker identification?

No. Speaker diarization separates voices and numbers them; speaker identification matches a voice to a known person.

Is the diarization error rate the same as word error rate?

No. Word error rate counts wrong words; the diarization error rate measures time given to the wrong speaker, missed, or wrongly counted as speech.