Parakeet vs Whisper — default-model bake-off (2026-07-16)

Pre-registered comparison (criteria fixed before any numbers existed — see the plan on branch test_parakeet) between WhispAssist's shipping Whisper base.en paths and NVIDIA Parakeet TDT 0.6B v2, to decide whether the default transcription model should change.

Provenance

Machine: Intel Core Ultra 5 135U (Meteor Lake: 12C/14T CPU, Intel Arc iGPU, Intel AI Boost NPU, driver 32.0.100.4724), Windows 11 Pro 26200. App version under test: 0.7.3.
Harness: src-tauri/examples/asr_bench.rs — drives the production transcribers (WhisperTranscriber, OnnxTranscriber) and a sherpa-onnx TransducerRecognizer; one contender per process (clean CPU accounting). WER = word-level Levenshtein after lowercase/punctuation-strip normalization (self-tested).
Corpus: first 30 LibriSpeech test-clean utterances (241.7 s, 644 reference words), 16 kHz mono, staged by scripts/bench/fetch-corpus.ps1. Live latency: sequential 10 s windows over a 3-minute concatenation (16 windows).
Models: ggml base.en-q5_1, medium.en-q5_0; WA ONNX base.en (merged decoder, encoder on OpenVINO/NPU + decoder on OpenVINO/GPU — EP confirmed in-run); sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8 (encoder 652 MB + decoder 7 MB + joiner 2 MB).
Raw data: bench-results/raw.csv, bench-results/raw-w1-novulkan.csv (committed).

Results — corpus (accuracy + cost)

Contender	Engine / device	WER	RTF (wall)	CPU-sec per audio-sec	Load
W1 base.en (as shipped, 14 thr)	whisper.cpp, CPU	2.48 %	⚠ 20.1	⚠ 232.6	112 ms
W1 base.en (thread-cap fix)	whisper.cpp, CPU	2.48 %	0.218	0.827	163 ms
W2 base.en	whisper.cpp Vulkan, iGPU	2.64 %	0.071	0.033	248 ms
W3 medium.en	whisper.cpp Vulkan, iGPU	3.26 %	0.304	0.088	724 ms
W4 base.en (default)	ort ONNX, NPU + iGPU	3.11 %	0.058	0.049	~4.7 s
P1 parakeet int8, t8	sherpa-onnx, CPU	0.47 %	0.047	0.396	~4.6 s
P1 parakeet int8, t4	sherpa-onnx, CPU	0.47 %	0.049	0.224	~4.6 s

Parakeet: 3 errors in 644 words, vs 20 for the current default — an 85 % relative WER improvement — while also posting the best RTF in the field, on the CPU alone.
Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads); the NPU+iGPU Whisper path costs ≈0.05.
✅ W1 anomaly RESOLVED (2026-07-16 follow-up): root cause was n_threads = available_parallelism() (14) putting ggml spin-barrier workers on the two slow LP E-cores — every graph-node barrier waits on the slowest worker while the rest burn CPU. 12 threads is healthy (RTF 0.200), 14 collapses (RTF ~22). The original suspect (opt-level = "z" / unoptimized ggml) was disproven: a verified /O2 build was equally slow. Fix: default capped at 4 threads, whisper.cpp's own upstream default (WA_WHISPER_THREADS env overrides); re-run row above — RTF 20.1 → 0.218 (92×), CPU 232.6 → 0.827 (281×), WER identical. (The build was also silently unoptimized — the cmake crate clobbers MSVC /O2 when no generator is set; fixed via CMAKE_GENERATOR=Ninja in .cargo/config.toml as build hygiene, though it wasn't the bottleneck.) Raw data: bench-results/raw-w1-fixed.csv.
W3 (medium) scoring worse than base on this corpus (21 vs 17 errors) is small-sample noise on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general.

Results — live windows (10 s, n=16)

Contender	p50	p95	max
W2 Vulkan base.en	1055 ms	3722 ms	4148 ms
W4 NPU+iGPU (default)	519 ms	630 ms	928 ms
P1 parakeet t8 (CPU)	450 ms	497 ms	514 ms

All hold real-time; Parakeet is fastest and near-flat (64 ms p50→max spread), which would make the live transcript feel noticeably steadier.

Evidence sample (file 1089-134686-0000, 10.4 s)

Reference: HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND CARROTS AND BRUISED POTATOES AND FAT MUTTON PIECES TO BE LADLED OUT IN THICK PEPPERED FLOUR FATTENED SAUCE

W4 (default): "He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour-fatted sauce." (2 err)
P1 (parakeet): " He hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce" (0 err)

Note the trade visible even here: Whisper punctuates richly; Parakeet's int8 output carries casing but sparse punctuation — a real consideration for notes.md readability (see integration costs).

Qualitative pass — real recording (2026-07-16 follow-up)

bench-corpus/test_wavs/bank_deposits.wav (62 s real recording, 48 kHz stereo — financial disclosure read aloud; downmixed/resampled by the production reader). No reference transcript, so transcripts were compared by eye; raw hypotheses in bench-results/raw-qual-bank-deposits.csv (its WER column is against a placeholder — ignore it).

P1 (parakeet t4): cleanest of the three — zero audible-word errors, and the best punctuation, including commas around a parenthetical clause ("…the bank deposit details section, which appears later in this statement, for information…"). The "sparse punctuation" concern from the LibriSpeech sample did not reproduce on this longer real clip.
W2 (Vulkan base.en): near-perfect; one function-word slip ("effective for cash balances and your FDIC insured…" — W4 and P1 both hear "in your").
W4 (NPU ONNX, shipping default): worst of the three — "details" → "D-Tales" twice, "are SIPC protected" → "or SIPC protected", and a segmentation break at the end ("statement period" → "state. period.").

One clip, one speaker, clean audio — directionally consistent with the corpus WERs (P1 ≥ W2 > W4), and it removes punctuation as an argument against Parakeet. Multi-speaker crosstalk remains untested.

Pre-registered criteria, evaluated

Criterion	Result
(a) WER ≥30 % relatively better than W4	PASS — 85 % better (0.47 % vs 3.11 %)
(b) CPU rung: beat W1 on WER and CPU-seconds	PASS — vs the fixed W1 (follow-up): 5.3× better WER (0.47 % vs 2.48 %), 3.7× less CPU (0.224 vs 0.827 cpu-sec/audio-sec)
(b) NPU rung: accelerator execution or CPU ≤ W4	FAIL (strictly) — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below)
(c) Live windows hold real-time	PASS — best in field, p95 497 ms

Phase-4 finding (accelerated Parakeet, timeboxed): the sherpa route is packaging-blocked — sherpa-rs's directml feature is mutually exclusive with download-binaries (compile_error! in sherpa-rs-sys 0.6.8), and escaping via SHERPA_LIB_PATH + a version-matched prebuilt DirectML archive would require replacing the base dependency configuration for the whole branch, with transducer-on-DirectML performance unproven (and a prior DirectML-on-Intel-GPU crash on record). The genuine route is a TDT decode loop on our own ort/OpenVINO stack (istupakov-style ONNX exports) — estimated days, not hours, and only worth it if the NPU rung must flip. The bench-directml cargo feature + p2-parakeet-dml contender remain in the harness as the documented attempt.

Recommendation (per hardware rung)

CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8. Even against the fixed W1 (thread cap, follow-up above) it is ~5× more accurate and 3.7× cheaper in CPU-seconds at comparable wall RTF. This is the unambiguous win — just no longer by three orders of magnitude, since the W1 pathology itself is fixed.
NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX — Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the steadiest live latency. That trade is the user's call, not the benchmark's; if low-CPU stays the hard constraint, the NPU rung flips only after the ort/OpenVINO TDT work lands.
GPU (Vulkan) machines without NPU: W2 is the cheapest-CPU option (0.033), but Parakeet's accuracy gap applies here too — same judgment call as the NPU rung, cheaper to revisit after the integration items below.

Integration costs before any default flip (not solved by this bench): timestamped segments (sherpa-rs's transducer API returns text only; the C API exposes timestamps — small upstream PR or direct -sys call), punctuation restoration (sparse on short LibriSpeech clips, though the real-recording qualitative pass showed good punctuation on longer audio — may be a non-issue), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for non-English), a ~660 MB model download, and the model catalog/dispatch work (same shape as the 0.7.3 merged-decoder swap).

Caveats

LibriSpeech is clean read speech; WERs are comparative, not absolute product claims. The qualitative pass over a real recording was run as a follow-up (section above) — still single-speaker; multi-speaker meeting audio with crosstalk remains untested.
int8 quantization slightly penalizes Parakeet vs published fp32 figures — it still won by 85 %; the fp32 bundle exists if a tighter number is ever needed.
Single machine, single run per cell (30 files each). Differences here are far larger than run-to-run noise, but treat third-decimal RTF differences as noise.

9.6 KiB Raw Blame History Unescape Escape