Scribe v3 vs Scribe v2: We Tested 102 Languages Before Switching Loopdesk's Captions
We moved Loopdesk's captions from ElevenLabs Scribe v2 to Scribe v3 after running both on the same 2,550 clips in 102 languages. Scribe v3 made 15 to 19% fewer errors overall, and it had lower error in all 13 Indian languages we send to it when the caption language is set, at the same speed and price. It was worse in two languages, and it has one habit we are watching.
Results measured on November 11, 2026. Disclosure: Loopdesk is our product, and ElevenLabs is one of our transcription providers.
ElevenLabs released Scribe v3 on November 10, 2026 and reported 27% fewer word errors than Scribe v2 across 102 languages. Switching is a one-line change of model name. We tested first anyway. Captions are read word by word, and a vendor's benchmark runs on the vendor's setup, not on ours.
What does Loopdesk use Scribe for?
Loopdesk sends each caption job to the speech-to-text engine that suits the language, as described in Captions Without Borders. Scribe has three jobs in that system:
- Transcribing 13 Indian languages: Hindi, Telugu, Tamil, Kannada, Malayalam, Bengali, Marathi, Nepali, Gujarati, Punjabi, Odia, Urdu and Assamese. Scribe returns a timestamp for every word, which word-by-word caption highlighting needs.
- Detecting the language when you leave the caption language on auto.
- Backing up the other engines if one of them fails.
That gave the test three questions. Is v3 more accurate in those 13 languages? Does it detect languages at least as well? Does anything get worse?
How did we test Scribe v3 against v2?
We used FLEURS, the public benchmark ElevenLabs quotes. It has short read sentences in 102 languages, each with a reference transcript, so errors can be counted.
| Part of the test | What we did |
|---|---|
| Sample | 25 test clips per language, each a different sentence: 2,550 clips and 9.3 hours of audio |
| Models | Scribe v2 and Scribe v3 on the same clips, with the request settings our captions use |
| Pass 1 | No language given, as when captions are left on auto |
| Pass 2 | Language given, as when you pick the caption language |
| Calls | 10,200, and none failed |
| Scoring | Word error rate against the reference, after lower-casing and removing punctuation. Seven languages written without spaces between words are scored by character |
| Headline figure | The average of the 102 per-language rates, so each language counts equally |
Two limits apply to everything below. Twenty-five clips is a small sample for one language, so a gap of a few points in a single language is usually noise. We call a language clearly better or clearly worse only when a paired bootstrap test puts the 95% interval of the difference entirely on one side of zero. And FLEURS is clean read speech with one speaker, which is easier than most real recordings.
How much more accurate is Scribe v3?
Average error fell from 16.4% to 13.7% with no language given, and from 16.0% to 13.6% with the language given.
| Measure | Scribe v2 | Scribe v3 |
|---|---|---|
| Average error, no language given | 16.4% | 13.7% |
| Average error, language given | 16.0% | 13.6% |
| Median language, no language given | 10.4% | 8.5% |
| Clips with more than half the words wrong, no language given | 177 | 136 |
| Empty transcripts, no language given | 4 | 0 |
| Median response time per clip | 2.5 seconds | 2.5 seconds |
That is 16% fewer errors on the first pass (95% interval 13 to 20%) and 15% on the second (12 to 18%). With Serbian compared in one script, which we explain below, the gains are 19% and 18%.
ElevenLabs reports 27%. Our figure is lower, and the two are not directly comparable: we tested a sample of FLEURS, not all of it, and we normalise text our own way. What we can say is that the improvement holds on our setup and is smaller than the headline.
With no language given, Scribe v3 had lower error in 83 of the 102 languages, higher in 17 and equal in 2. By the stricter test it was clearly better in 42, clearly worse in 2, and not clearly different in 58.
The largest gains were in Cantonese (28.6% to 3.7%, by character), Telugu (21.3% to 11.9%), Mongolian (19.5% to 8.7%) and Amharic (30.1% to 17.2%). Among widely used languages, English went from 4.4% to 3.0%, Spanish from 1.6% to 0.6%, German from 3.1% to 1.7% and Mandarin from 9.9% to 5.9%.
What changed for the 13 Indian languages?
These are the languages Loopdesk sends to Scribe, so this table decided the migration. The figures are word error rates, and lower is better.
| Language | v2, auto | v3, auto | v2, language set | v3, language set |
|---|---|---|---|---|
| Hindi | 10.7% | 10.6% | 11.4% | 10.0% |
| Telugu | 21.3% | 11.9% | 21.8% | 11.0% |
| Tamil | 25.0% | 20.5% | 25.2% | 21.6% |
| Kannada | 19.6% | 14.1% | 19.6% | 14.8% |
| Malayalam | 22.0% | 16.0% | 22.0% | 16.0% |
| Bengali | 13.1% | 10.7% | 12.9% | 9.9% |
| Marathi | 18.5% | 14.5% | 18.3% | 14.5% |
| Nepali | 24.6% | 18.8% | 26.5% | 20.0% |
| Gujarati | 14.5% | 12.2% | 14.3% | 12.8% |
| Punjabi | 13.6% | 11.4% | 13.9% | 11.1% |
| Odia | 19.7% | 18.5% | 21.1% | 18.3% |
| Urdu | 29.7% | 32.0% | 20.1% | 15.5% |
| Assamese | 23.0% | 17.4% | 25.2% | 17.6% |
| Average | 19.6% | 16.0% | 19.4% | 14.9% |
Scribe v3 was lower in all 13 languages with the language set and in 12 of 13 on auto. It was clearly better in 8 and 9 of them, and clearly worse in none. Hindi barely moved on auto (10.7% to 10.6%).
Urdu is the exception on auto, and the cause is language detection. Spoken Urdu and Hindi are close, and both models sometimes labelled an Urdu clip as Hindi and wrote it in Devanagari script: v2 on 4 of 25 clips, v3 on 6. With Urdu selected, that stopped, and v3's error was 15.5% against v2's 20.1%. If you know the language of your video, pick it.
Does Scribe v3 detect languages as well as v2?
For our purposes, yes. Loopdesk only needs to know which engine a clip should go to. Both models got that right on 99.8% of clips, and they agreed with each other on 99.7%.
Naming the exact language is harder. Both models reported the right language on 74% of clips. Ten of the 13 Indian languages were identified on every clip by both. Assamese and Nepali improved with v3 (88% to 100% and 84% to 92%), and Urdu is covered above. Low-resource languages were often reported as a better-known neighbour, and that did not change between versions.
Where is Scribe v3 worse?
Three languages scored clearly worse, and one of the three is not really a loss.
| Language | Scribe v2 | Scribe v3 | What happened |
|---|---|---|---|
| Asturian, language set | 11.3% | 28.2% | v3 often writes standard Spanish instead of Asturian |
| Somali, language set | 35.5% | 41.0% | More word errors |
| Serbian, auto, as scored | 43.0% | 77.4% | A change of script, not of accuracy |
Serbian can be written in Latin or Cyrillic script. Scribe v2 wrote Latin on 18 of our 25 clips, and Scribe v3 wrote Cyrillic on 22. Most of the FLEURS references in our sample are in Latin, so v3 was marked wrong for words it had heard correctly. With both sides converted to Latin, the error rates are 7.1% for v2 and 3.7% for v3. If you publish Serbian captions in Latin script, though, the change is visible.
Six more languages were slightly higher in both passes: Afrikaans, Bulgarian, Welsh, Romanian, Swedish and Turkish. Each gap is under 3 points and within the noise of 25 clips.
None of these are languages Loopdesk sends to Scribe for transcription, unless another engine is down.
Does Scribe v3 change what was said?
Once, in a way worth knowing about. One FLEURS sentence says the Allies invaded southern France on 15 August 1940, in Operation Dragoon. The real invasion was in 1944. That sentence is in 15 of our clips, in 15 languages, and every reference transcript says 1940.
| Model, no language given | Wrote 1940 | Wrote 1944 | Wrote no year |
|---|---|---|---|
| Scribe v2 | 12 | 2 | 1 |
| Scribe v3 | 4 | 11 | 0 |
Scribe v3 wrote the historically correct year instead of the spoken one in 11 clips, across 11 languages. We did not listen to the recordings. The evidence that the speakers said 1940 is the script they read from and what Scribe v2 wrote.
This is a single sentence, so it shows the behaviour exists, not how often it happens. On numbers in general v3 did slightly better: when each model wrote digits, they matched the reference 81% of the time for v3 and 77% for v2. But captions should show what was said, including a speaker's mistakes. Check names, dates and figures before you publish. Every caption in Loopdesk is editable.
What happened on a real recording?
FLEURS is read speech, so we also ran a real 15-minute interview with two speakers through both models. There is no reference transcript for it, so this shows how the models differ, not which one is right.
| Measure | Scribe v2 | Scribe v3 |
|---|---|---|
| Words transcribed | 2,889 | 2,831 |
| False starts, such as "s-" or "w-" | 40 | 13 |
| Filler words (uh, um, mm) | 87 | 75 |
| Words with zero duration | 41 | 16 |
- Wording: 96% of v2's words appear in v3's transcript in the same order. Most of what v3 left out was false starts, fillers and one-word interjections. About 30 passages differ in wording and would need a listen to judge.
- Timing: on the words both models agree on, the median difference in timestamps was 0 milliseconds. Eleven of 2,782 words started more than a tenth of a second apart.
- Speakers: every matching word carried the same speaker label.
- Names: v3 spelled two company names correctly that v2 got wrong, and in one place wrote "okay" where v2 heard a person's name.
Why did we migrate to Scribe v3?
Five reasons, in order of weight.
- It is more accurate where we use it. Across the 13 Indian languages, average error fell from 19.4% to 14.9% with the language set, and no language was clearly worse.
- Nothing our captions depend on changed. The response format, word timing, speaker labels, engine routing and speed were the same in our tests.
- It fails less often. No empty transcripts against four, and 136 clips with more than half the words wrong against 177.
- The price is the same. ElevenLabs lists Scribe v3 at the same price per hour of audio as Scribe v2.
- It is reversible. ElevenLabs still offers Scribe v2, and the model is one name in our captions service.
What we did not do is turn on Scribe v3's new options. It can format numbers as digits or words, and it can take a hint about a second language in the audio. We will test those separately before any caption depends on them.
What have we not tested yet?
- Mixed-language speech, such as Hindi with English. FLEURS has none, and it is common in the videos our creators make.
- Three or more speakers. Every FLEURS clip has one speaker, and our interview had two.
- Noisy audio and music under speech.
- Filler-word cleanup in Indian languages. Scribe v3 writes down fewer fillers and false starts, and filler-word removal works from the transcript.
Those tests are next.
Frequently Asked Questions
Is Scribe v3 more accurate than Scribe v2?
In our test of 2,550 FLEURS clips in 102 languages, yes. Average error fell from 16.4% to 13.7% with no language given and from 16.0% to 13.6% with it. Scribe v3 was lower in 83 languages and clearly worse in Asturian and Somali.
Why is our result lower than the 27% ElevenLabs reports?
ElevenLabs measured 27% fewer word errors on its own version of FLEURS. We measured 15 to 19% on a sample of 25 clips per language with our own text normalisation. The methods differ, so the two figures are not directly comparable.
Which Loopdesk captions use Scribe v3?
Captions in 13 Indian languages, including Hindi, Tamil, Telugu and Bengali, are transcribed by Scribe v3. It also detects the language when captions are left on auto, and it backs up our other speech-to-text engines.
Does Scribe v3 change caption timing?
Not in our tests. On a 15-minute interview, the median difference between v2 and v3 word timestamps was 0 milliseconds, and 11 of 2,782 matching words started more than a tenth of a second apart.
Should I set the caption language or leave it on auto?
Set it if you know it. Urdu shows why: on auto, both models sometimes wrote Urdu speech as Hindi in Devanagari script. With Urdu selected, Scribe v3's error rate fell from 32.0% to 15.5%.
Does Scribe v3 cost more than Scribe v2?
No. ElevenLabs lists Scribe v3 at the same price per hour of audio as Scribe v2, and Loopdesk's plan prices did not change.
Captioning in Hindi, Tamil, Telugu or another Indian language? Open Loopdesk, add captions, and check the names and numbers before you publish.