Tools
Transcription
When you transcribe a recording of a meeting or an interview as it is, what comes back is a single run of text with no indication of who said what. Splitting that transcript up by speaker is called "speaker diarization", and if the plan is to read it back later as minutes, skipping this step means listening to the audio again anyway.
Whisper itself, which is what most transcription runs on, has no such feature. Cloud services do offer speaker diarization even on their free tiers, but the monthly allowance is capped and uploading your audio data is a precondition.
This article explains how to run a transcription with speaker diarization entirely in the browser. The second half covers the accuracy I measured while implementing the feature, and the situations where that accuracy does not hold up.
- What you need — a PC, Chrome or Edge, and a recording with a conversation in it
- What you do not need — an account, an app install, or a file upload
How speaker diarization relates to Whisper
You sometimes see it written that Whisper has a speaker diarization feature, which is wrong. Whisper is a model that turns audio into text, and it has no mechanism at all for telling voices apart.
Separating out who spoke requires a different model, one built for that job. The one used most often in this area is "pyannote.audio", which slices the audio finely, extracts the characteristics of each voice as a vector, and groups similar vectors together to reconstruct one person's utterances across the whole recording.
A transcript with speaker diarization is therefore built from a transcription model and a diarization model working together. Cloud services run both of them on their own servers, which is why using one means uploading your audio data.
Comparing the free options
There are broadly three free routes. The preparation each one takes, the file sizes they handle, and whether your audio data is sent to a server all differ, so it is worth lining them up first.
Method | Preparation | Length limit | Sent to a server |
|---|---|---|---|
Cloud services | An account | Free tiers run around 120–300 min/month | Yes |
Building it in Python | Python and a GPU environment | None | No |
Processing in the browser | None | None | No |
Building it in Python gives you the most freedom, but getting CUDA or ROCm working is a common place to get stuck. I started down that route myself, then decided it was unreasonable to expect every reader to set that environment up, and rebuilt the same processing so that it runs entirely inside the browser.
What follows is the browser route. If building it yourself in Python is the more interesting option, there is an article at the end that starts from the environment setup.
Steps: separate speakers in the browser
The tool here is "Transcriber", which I built. Both diarization and transcription run inside the browser, so the audio you recorded never leaves the device.
Transcribe audio and video files entirely in your browser. No account and no upload — your audio never leaves your device. Free with no time or usage limits. Speaker diarization and SRT subtitle export powered by Whisper AI, supporting MP3, WAV, M4A, MP4 and more.
- Open "Transcriber" in the browser.
- Turn "Speaker diarization" on.
- If you know how many people took part, set it in "Number of speakers (0 = auto)".
- Choose the "Model" and the "Language".
- Load the recording into the drop area. Processing starts as soon as the file is loaded.
Setting the number of speakers improves accuracy
Leaving "Number of speakers (0 = auto)" at 0 means the count is estimated automatically. That estimate is harder than it looks, and in recordings where short utterances alternate repeatedly, a conversation between two people can end up split across three.
In my own evaluation a 30 second English conversation came out as three speakers under automatic detection. Setting the count to 2 removes that error entirely, so if you know who was in the meeting, setting it is the safer choice.
Fixing the parts that come out wrong
Speaker diarization is not perfect. It is weakest on short utterances, and interjections or calls of around a second get assigned to the same speaker as the longer utterance next to them. This weakness stays no matter which model you use, so working on the assumption that you will fix it afterwards ends up being quicker.
When I looked into a recording of a Diet committee session, there were six short calls along the lines of "Chair, Mr Yamanoi.", and in four of them the next speaker had already absorbed the call at the point where the audio is segmented. The problem is not in how the segments are grouped but one step earlier, where the distinction is already lost, so no amount of moving the threshold fixes it.
Since the model cannot fix this, what matters more is that a person can fix it afterwards. The results screen therefore lets you change the speaker on a single line, rename a speaker, merge two speakers that were split by mistake, and split a long line in two.
- Clicking a speaker badge lets you rename that speaker or merge them into another
- To change only one line, pick a different speaker from that line's badge
- "Undo speaker edits" reverts the speaker assignments alone, leaving your text edits intact
How it works, and the accuracy I measured (for engineers)
What follows is about the internal logic, so feel free to skip it if you only want to use the tool. Knowing how it works does make it clear why short utterances fail, and gives you a sense of where to look when fixing the output.
The processing has three stages. The audio is first sliced into ten second chunks, and within each chunk the model estimates who is speaking and when. Each chunk then yields a vector describing the characteristics of the voice speaking in it, and finally similar vectors are grouped together to reconstruct one person's utterances across the whole recording.
The first stage uses pyannote's segmentation model and the second uses WeSpeaker; the segmentation model is only 6 MB, so loading it in the browser involves no waiting. Most of the cost sits in the second stage embeddings and the grouping that follows.
How one person speaking alone became eight speakers
The first implementation subtracted the overall mean from each voice characteristic vector. The intent was to cancel out differences in microphones and rooms between recordings, but measuring it afterwards showed this was what was lowering the accuracy.
In a recording with only one speaker, subtracting the overall mean leaves nothing behind that distinguishes speakers, because the mean matches that person's voice characteristics. With two speakers the distance between utterances from the same person widens instead, and in my simulation that distance opened up from 0.39 to 1.00.
The result was that a recording of one person speaking alone and a conversation between two people both split into the maximum of eight speakers under automatic detection. The grouping threshold had been chosen without measurement as well, so rather than adjusting the threshold, I started by building something that could produce numbers.
Measurements after the rebuild
I rebuilt it with the same structure as pyannote 3.1 and measured again using the same audio. The metric is DER, which expresses what proportion of the total time was attributed to the wrong speaker; a lower value means the speakers were separated more accurately.
Audio | Old (auto) | New (auto) | New (speaker count set) |
|---|---|---|---|
English conversation, 30 s, 2 speakers | 8 speakers, DER 57.1% | 2 speakers, 5.1% | 5.1% |
Japanese monologue, 41 s, 1 speaker | 8 speakers, 80.3% | 1 speaker, 0.1% | — |
The conversation above repeated 25 times, 12.5 min | — | 2 speakers, 5.9% | — |
DER stays at 5.9% even stretched to 12.5 minutes, so accuracy does not suddenly collapse on longer recordings. Note, though, that these are values measured on two pieces of audio with hand-made reference labels, and they do not mean every recording reaches this accuracy. Meetings with many participants, or recordings that include speech far from the microphone, come out worse.
Porting pyannote as-is did not reach this accuracy
Carrying the pyannote procedure across unchanged split the 30 second sample into three speakers (DER 12.3%). Chunks where short utterances overlap produced large numbers of mutually similar vectors, and those were being grouped together as a third speaker who did not exist.
Excluding vectors whose speech within a chunk falls short of two seconds from the cluster seeds, and doing so only under automatic detection, resolved it. Those vectors are kept when the speaker count is set explicitly.
The other awkward part was that the browser-side library had no post-processing for the segmentation model. That model reports its results as combinations of who is speaking simultaneously, so turning those back into per-speaker time ranges was something I had to write myself.
Frequently asked questions
How many speakers can it separate?
The limit is eight, both under automatic detection and when the count is set explicitly. Meetings with more participants than that cannot be fully separated, and people with similar voices end up merged into one.
Is it really free? Are there limits on how many times I can use it?
It is free to use. There is no limit on the number of runs and no limit on length. Both diarization and transcription run on the visitor's own PC, so no server costs land on my side.
Is there any chance my audio data is sent to a server?
There is not. The diarization model and the transcription model both run inside the browser, so the audio never leaves the device. The only network traffic is the initial download of the models.
How much slower does turning on speaker diarization make it?
It costs the time it takes to read through the same audio a second time, separately from the transcription. With WebGPU available it takes roughly as long as the transcription itself; without it, longer.
Can I use it on a phone?
It runs, but I would not recommend it. Diarization uses memory on top of what the transcription needs, which makes it that much more likely to stop partway. Record on the phone and process on a PC.
Can I change speaker names to real names?
You can. Names are changed from the speaker badge, and the names you set there carry through to the TXT and SRT files you export.
Summary
Running Whisper on its own tells you nothing about who spoke. Speaker diarization is handled by an entirely separate model, and combining the two gets you to something readable as minutes without leaving the browser.
- Turn "Speaker diarization" on before loading the audio file
- Set the number of speakers if you know it
- Short interjections and calls are its weak point, so fix them on the results screen
Related articles and tools
A step-by-step guide to transcribing m4a recordings from the iPhone Voice Memos app without converting the format. No upload and no account are needed, and separating speakers and exporting subtitles are done entirely in the browser.
This article explains in detail how to transcribe audio files such as videos and meeting recordings into text completely free of charge using the Python language.
A step-by-step guide to turning a recorded X (formerly Twitter) Space into text for free. Covers downloading the recording, separating speakers and exporting subtitle files, all in the browser without an account or an app install.
Programing
Python
Hosting