Three reported baselines; one keyterm projection. Local and hosted runtimes differ.
Method & limits
Historical reported summary from open-weight-stt-findings.md, not a fresh run. WER is a fraction; latency is the reported median in milliseconds. Whisper and Parakeet ran locally on an M4 GPU; Deepgram used a hosted API, so timing includes different compute and network conditions, not live-turn latency. The keyterm row is the same Deepgram system: full-corpus WER is projected from seven name clips, and 204 ms is inherited from the baseline, not remeasured. The separate 240-row probe pools six audio conditions and is not this 8 kHz summary.
Match the calls
I built a custom benchmark around the agents' actual domain and use cases: conversational requests, short replies, names, and digits. The corpus uses synthesized audio with telephone-bandwidth and noise conditions. It is not a dataset of customer call recordings.
Revisit the scorer
Seeing issues in Converse made me revisit the scorer. It was penalizing digits written as numerals instead of spoken words. Number-aware scoring changed Parakeet's digit result on the same transcripts; this was a correction to the evaluation, not an improvement to the model.
Test a counterexample
I tried post-hoc snapping, replacing recognized text with nearby directory names, as a counterexample. The naive version made name errors worse in the report. I kept that separate from supplying directory vocabulary during recognition, where the keyterm experiment improved the name-clip result.
A decision for our agents
In my evaluation, replacing the stack with Parakeet would need extra turn detection, and I had concerns about Punjabi accuracy. I considered a hybrid but thought it could be slower and less accurate for our calls. Those are my project-specific judgments, not universal claims about the providers. The benchmark's public repository is pending.
What the evidence says.
measured
Parakeet digit WER after scorer correction
benchmark
0.303 WER / to0.052 WER
Arithmetic means of the stored pk_wer and pk_wer2 scores for the 60 digit rows: 0.302882 and 0.052357. The same transcripts span six audio conditions. This measures a scoring correction, not new recognition performance; the number-aware scorer is not included in the retained probe.
Source
parakeet_bench_results.json: digit-row scores
Sample
10 synthesized digit clips across 6 conditions; 60 paired rows
Recorded
2026-08-05
measured
Paired probe WER across all conditions
benchmark
0.0486 Deepgram / 0.0537 Parakeet
Arithmetic means of stored dg_wer2 and pk_wer2 scores across all 240 paired rows. The pool includes 16 kHz clean audio, 8 kHz telephony, and four noise levels; it is not the separate 8 kHz summary or a production-call evaluation.
Source
parakeet_bench_results.json: all-condition scores
Sample
40 synthesized clips across 6 conditions; 240 paired rows, each with both model outputs
Recorded
2026-08-05
reported
Name-clip WER with directory keyterms
benchmark
0.121 WER / to0.014 WER
Historical report of the same seven synthesized name clips processed through 8 kHz mu-law audio, with and without directory keyterms. This is a name-subset comparison, not full-corpus WER or measured live-call routing success.
Source
open-weight-stt-findings.md: name-clip comparison
Sample
7 synthesized name clips
Recorded
2026-08-05
projected
Projected full-corpus WER with keyterms
benchmark
0.050 WER / to0.033 WER
The historical report projects the seven-clip name improvement across the full corpus. The accompanying 204 ms latency is inherited from the Deepgram baseline, not a new timing measurement.
Source
open-weight-stt-findings.md: summary table and projection footnote
Sample
40-clip corpus projection from a 7-clip name subset
Recorded
2026-08-05
Credits
Benchmark design, voice research, and analysis by DongYeop Lee at Hemut.
