149 lines
9.6 KiB
Markdown
149 lines
9.6 KiB
Markdown
# Parakeet vs Whisper — default-model bake-off (2026-07-16)
|
||
|
||
Pre-registered comparison (criteria fixed before any numbers existed — see the plan on branch
|
||
`test_parakeet`) between WhispAssist's shipping Whisper `base.en` paths and NVIDIA Parakeet
|
||
TDT 0.6B v2, to decide whether the default transcription model should change.
|
||
|
||
## Provenance
|
||
|
||
- **Machine:** Intel Core Ultra 5 135U (Meteor Lake: 12C/14T CPU, Intel Arc iGPU, Intel AI Boost
|
||
NPU, driver 32.0.100.4724), Windows 11 Pro 26200. App version under test: 0.7.3.
|
||
- **Harness:** `src-tauri/examples/asr_bench.rs` — drives the *production* transcribers
|
||
(`WhisperTranscriber`, `OnnxTranscriber`) and a sherpa-onnx `TransducerRecognizer`; one
|
||
contender per process (clean CPU accounting). WER = word-level Levenshtein after
|
||
lowercase/punctuation-strip normalization (self-tested).
|
||
- **Corpus:** first 30 LibriSpeech test-clean utterances (241.7 s, 644 reference words), 16 kHz
|
||
mono, staged by `scripts/bench/fetch-corpus.ps1`. Live latency: sequential 10 s windows over a
|
||
3-minute concatenation (16 windows).
|
||
- **Models:** ggml `base.en-q5_1`, `medium.en-q5_0`; WA ONNX `base.en` (merged decoder, encoder
|
||
on OpenVINO/NPU + decoder on OpenVINO/GPU — EP confirmed in-run);
|
||
`sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8` (encoder 652 MB + decoder 7 MB + joiner 2 MB).
|
||
- **Raw data:** `bench-results/raw.csv`, `bench-results/raw-w1-novulkan.csv` (committed).
|
||
|
||
## Results — corpus (accuracy + cost)
|
||
|
||
| Contender | Engine / device | WER | RTF (wall) | CPU-sec per audio-sec | Load |
|
||
|---|---|---|---|---|---|
|
||
| W1 base.en (as shipped, 14 thr) | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
|
||
| W1 base.en (thread-cap fix) | whisper.cpp, CPU | 2.48 % | 0.218 | 0.827 | 163 ms |
|
||
| W2 base.en | whisper.cpp Vulkan, iGPU | 2.64 % | 0.071 | **0.033** | 248 ms |
|
||
| W3 medium.en | whisper.cpp Vulkan, iGPU | 3.26 % | 0.304 | 0.088 | 724 ms |
|
||
| W4 base.en (default) | ort ONNX, NPU + iGPU | 3.11 % | 0.058 | 0.049 | ~4.7 s |
|
||
| **P1 parakeet int8, t8** | sherpa-onnx, CPU | **0.47 %** | **0.047** | 0.396 | ~4.6 s |
|
||
| **P1 parakeet int8, t4** | sherpa-onnx, CPU | **0.47 %** | 0.049 | 0.224 | ~4.6 s |
|
||
|
||
- **Parakeet: 3 errors in 644 words**, vs 20 for the current default — an **85 % relative WER
|
||
improvement** — while also posting the best RTF in the field, on the CPU alone.
|
||
- Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads);
|
||
the NPU+iGPU Whisper path costs ≈0.05.
|
||
- ✅ **W1 anomaly RESOLVED (2026-07-16 follow-up):** root cause was `n_threads =
|
||
available_parallelism()` (14) putting ggml spin-barrier workers on the two slow LP E-cores —
|
||
every graph-node barrier waits on the slowest worker while the rest burn CPU. 12 threads is
|
||
healthy (RTF 0.200), 14 collapses (RTF ~22). The original suspect (`opt-level = "z"` /
|
||
unoptimized ggml) was **disproven**: a verified `/O2` build was equally slow. Fix: default
|
||
capped at 4 threads, whisper.cpp's own upstream default (`WA_WHISPER_THREADS` env overrides);
|
||
re-run row above — RTF 20.1 → 0.218 (92×), CPU 232.6 → 0.827 (281×), WER identical.
|
||
(The build *was* also silently unoptimized — the cmake crate clobbers MSVC `/O2` when no
|
||
generator is set; fixed via `CMAKE_GENERATOR=Ninja` in `.cargo/config.toml` as build hygiene,
|
||
though it wasn't the bottleneck.) Raw data: `bench-results/raw-w1-fixed.csv`.
|
||
- W3 (medium) scoring *worse* than base on this corpus (21 vs 17 errors) is small-sample noise
|
||
on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general.
|
||
|
||
## Results — live windows (10 s, n=16)
|
||
|
||
| Contender | p50 | p95 | max |
|
||
|---|---|---|---|
|
||
| W2 Vulkan base.en | 1055 ms | 3722 ms | 4148 ms |
|
||
| W4 NPU+iGPU (default) | 519 ms | 630 ms | 928 ms |
|
||
| **P1 parakeet t8 (CPU)** | **450 ms** | **497 ms** | **514 ms** |
|
||
|
||
All hold real-time; Parakeet is fastest **and** near-flat (64 ms p50→max spread), which would
|
||
make the live transcript feel noticeably steadier.
|
||
|
||
## Evidence sample (file 1089-134686-0000, 10.4 s)
|
||
|
||
> **Reference:** HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND CARROTS AND BRUISED
|
||
> POTATOES AND FAT MUTTON PIECES TO BE LADLED OUT IN THICK PEPPERED FLOUR FATTENED SAUCE
|
||
|
||
- **W4 (default):** "He hoped there would be stew for dinner, turnips and carrots and bruised
|
||
potatoes and fat mutton pieces to be ladled out in thick peppered **flour-fatted** sauce." (2 err)
|
||
- **P1 (parakeet):** " He hoped there would be stew for dinner turnips and carrots and bruised
|
||
potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce" (0 err)
|
||
|
||
Note the trade visible even here: Whisper punctuates richly; Parakeet's int8 output carries
|
||
casing but **sparse punctuation** — a real consideration for notes.md readability (see
|
||
integration costs).
|
||
|
||
## Qualitative pass — real recording (2026-07-16 follow-up)
|
||
|
||
`bench-corpus/test_wavs/bank_deposits.wav` (62 s real recording, 48 kHz stereo — financial
|
||
disclosure read aloud; downmixed/resampled by the production reader). No reference transcript,
|
||
so transcripts were compared by eye; raw hypotheses in `bench-results/raw-qual-bank-deposits.csv`
|
||
(its WER column is against a placeholder — ignore it).
|
||
|
||
- **P1 (parakeet t4):** cleanest of the three — zero audible-word errors, and the **best
|
||
punctuation**, including commas around a parenthetical clause ("…the bank deposit details
|
||
section, which appears later in this statement, for information…"). The "sparse punctuation"
|
||
concern from the LibriSpeech sample did **not** reproduce on this longer real clip.
|
||
- **W2 (Vulkan base.en):** near-perfect; one function-word slip ("effective for cash balances
|
||
*and* your FDIC insured…" — W4 and P1 both hear "*in* your").
|
||
- **W4 (NPU ONNX, shipping default):** worst of the three — "details" → "**D-Tales**" twice,
|
||
"are SIPC protected" → "**or** SIPC protected", and a segmentation break at the end
|
||
("statement period" → "**state. period.**").
|
||
|
||
One clip, one speaker, clean audio — directionally consistent with the corpus WERs
|
||
(P1 ≥ W2 > W4), and it removes punctuation as an argument against Parakeet. Multi-speaker
|
||
crosstalk remains untested.
|
||
|
||
## Pre-registered criteria, evaluated
|
||
|
||
| Criterion | Result |
|
||
|---|---|
|
||
| (a) WER ≥30 % relatively better than W4 | **PASS** — 85 % better (0.47 % vs 3.11 %) |
|
||
| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — vs the *fixed* W1 (follow-up): 5.3× better WER (0.47 % vs 2.48 %), 3.7× less CPU (0.224 vs 0.827 cpu-sec/audio-sec) |
|
||
| (b) NPU rung: accelerator execution or CPU ≤ W4 | **FAIL (strictly)** — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below) |
|
||
| (c) Live windows hold real-time | **PASS** — best in field, p95 497 ms |
|
||
|
||
**Phase-4 finding (accelerated Parakeet, timeboxed):** the sherpa route is packaging-blocked —
|
||
`sherpa-rs`'s `directml` feature is mutually exclusive with `download-binaries`
|
||
(`compile_error!` in `sherpa-rs-sys` 0.6.8), and escaping via `SHERPA_LIB_PATH` + a
|
||
version-matched prebuilt DirectML archive would require replacing the base dependency
|
||
configuration for the whole branch, with transducer-on-DirectML performance unproven (and a
|
||
prior DirectML-on-Intel-GPU crash on record). The genuine route is a TDT decode loop on our own
|
||
ort/OpenVINO stack (istupakov-style ONNX exports) — estimated **days, not hours**, and only
|
||
worth it if the NPU rung must flip. The `bench-directml` cargo feature + `p2-parakeet-dml`
|
||
contender remain in the harness as the documented attempt.
|
||
|
||
## Recommendation (per hardware rung)
|
||
|
||
1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** Even against the
|
||
*fixed* W1 (thread cap, follow-up above) it is ~5× more accurate and 3.7× cheaper in
|
||
CPU-seconds at comparable wall RTF. This is the unambiguous win — just no longer by three
|
||
orders of magnitude, since the W1 pathology itself is fixed.
|
||
2. **NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX** —
|
||
Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the
|
||
numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the
|
||
steadiest live latency. That trade is the user's call, not the benchmark's; if low-CPU stays
|
||
the hard constraint, the NPU rung flips only after the ort/OpenVINO TDT work lands.
|
||
3. **GPU (Vulkan) machines without NPU:** W2 is the cheapest-CPU option (0.033), but Parakeet's
|
||
accuracy gap applies here too — same judgment call as the NPU rung, cheaper to revisit after
|
||
the integration items below.
|
||
|
||
**Integration costs before any default flip** (not solved by this bench): timestamped segments
|
||
(sherpa-rs's `transducer` API returns text only; the C API exposes timestamps — small upstream
|
||
PR or direct `-sys` call), punctuation restoration (sparse on short LibriSpeech clips, though
|
||
the real-recording qualitative pass showed *good* punctuation on longer audio — may be a
|
||
non-issue), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for
|
||
non-English), a ~660 MB model download, and the model catalog/dispatch work (same shape as the
|
||
0.7.3 merged-decoder swap).
|
||
|
||
## Caveats
|
||
|
||
- LibriSpeech is clean read speech; WERs are comparative, not absolute product claims. The
|
||
qualitative pass over a real recording was run as a follow-up (section above) — still
|
||
single-speaker; multi-speaker meeting audio with crosstalk remains untested.
|
||
- int8 quantization slightly penalizes Parakeet vs published fp32 figures — it *still* won by
|
||
85 %; the fp32 bundle exists if a tighter number is ever needed.
|
||
- Single machine, single run per cell (30 files each). Differences here are far larger than
|
||
run-to-run noise, but treat third-decimal RTF differences as noise.
|