diff --git a/docs/benchmarks/2026-07-parakeet-vs-whisper.md b/docs/benchmarks/2026-07-parakeet-vs-whisper.md new file mode 100644 index 0000000..25481a0 --- /dev/null +++ b/docs/benchmarks/2026-07-parakeet-vs-whisper.md @@ -0,0 +1,121 @@ +# Parakeet vs Whisper — default-model bake-off (2026-07-16) + +Pre-registered comparison (criteria fixed before any numbers existed — see the plan on branch +`test_parakeet`) between WhispAssist's shipping Whisper `base.en` paths and NVIDIA Parakeet +TDT 0.6B v2, to decide whether the default transcription model should change. + +## Provenance + +- **Machine:** Intel Core Ultra 5 135U (Meteor Lake: 12C/14T CPU, Intel Arc iGPU, Intel AI Boost + NPU, driver 32.0.100.4724), Windows 11 Pro 26200. App version under test: 0.7.3. +- **Harness:** `src-tauri/examples/asr_bench.rs` — drives the *production* transcribers + (`WhisperTranscriber`, `OnnxTranscriber`) and a sherpa-onnx `TransducerRecognizer`; one + contender per process (clean CPU accounting). WER = word-level Levenshtein after + lowercase/punctuation-strip normalization (self-tested). +- **Corpus:** first 30 LibriSpeech test-clean utterances (241.7 s, 644 reference words), 16 kHz + mono, staged by `scripts/bench/fetch-corpus.ps1`. Live latency: sequential 10 s windows over a + 3-minute concatenation (16 windows). +- **Models:** ggml `base.en-q5_1`, `medium.en-q5_0`; WA ONNX `base.en` (merged decoder, encoder + on OpenVINO/NPU + decoder on OpenVINO/GPU — EP confirmed in-run); + `sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8` (encoder 652 MB + decoder 7 MB + joiner 2 MB). +- **Raw data:** `bench-results/raw.csv`, `bench-results/raw-w1-novulkan.csv` (committed). + +## Results — corpus (accuracy + cost) + +| Contender | Engine / device | WER | RTF (wall) | CPU-sec per audio-sec | Load | +|---|---|---|---|---|---| +| W1 base.en | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms | +| W2 base.en | whisper.cpp Vulkan, iGPU | 2.64 % | 0.071 | **0.033** | 248 ms | +| W3 medium.en | whisper.cpp Vulkan, iGPU | 3.26 % | 0.304 | 0.088 | 724 ms | +| W4 base.en (default) | ort ONNX, NPU + iGPU | 3.11 % | 0.058 | 0.049 | ~4.7 s | +| **P1 parakeet int8, t8** | sherpa-onnx, CPU | **0.47 %** | **0.047** | 0.396 | ~4.6 s | +| **P1 parakeet int8, t4** | sherpa-onnx, CPU | **0.47 %** | 0.049 | 0.224 | ~4.6 s | + +- **Parakeet: 3 errors in 644 words**, vs 20 for the current default — an **85 % relative WER + improvement** — while also posting the best RTF in the field, on the CPU alone. +- Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads); + the NPU+iGPU Whisper path costs ≈0.05. +- ⚠ **W1 anomaly:** whisper.cpp forced to CPU is pathologically slow in *both* the Vulkan and + non-Vulkan builds of this crate (reproduced twice; ~12 cores busy at 20× real-time). Its WER + is valid; its speed numbers are not representative of whisper.cpp generally (published CPU + RTFs for base.en are ~0.3–0.5) and point at a real product bug for CPU-only machines — prime + suspect: the release profile's `opt-level = "z"` applied to ggml's kernels. **Follow-up filed; + even a healthy W1 would lose the CPU rung to P1 by an order of magnitude.** +- W3 (medium) scoring *worse* than base on this corpus (21 vs 17 errors) is small-sample noise + on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general. + +## Results — live windows (10 s, n=16) + +| Contender | p50 | p95 | max | +|---|---|---|---| +| W2 Vulkan base.en | 1055 ms | 3722 ms | 4148 ms | +| W4 NPU+iGPU (default) | 519 ms | 630 ms | 928 ms | +| **P1 parakeet t8 (CPU)** | **450 ms** | **497 ms** | **514 ms** | + +All hold real-time; Parakeet is fastest **and** near-flat (64 ms p50→max spread), which would +make the live transcript feel noticeably steadier. + +## Evidence sample (file 1089-134686-0000, 10.4 s) + +> **Reference:** HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND CARROTS AND BRUISED +> POTATOES AND FAT MUTTON PIECES TO BE LADLED OUT IN THICK PEPPERED FLOUR FATTENED SAUCE + +- **W4 (default):** "He hoped there would be stew for dinner, turnips and carrots and bruised + potatoes and fat mutton pieces to be ladled out in thick peppered **flour-fatted** sauce." (2 err) +- **P1 (parakeet):** " He hoped there would be stew for dinner turnips and carrots and bruised + potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce" (0 err) + +Note the trade visible even here: Whisper punctuates richly; Parakeet's int8 output carries +casing but **sparse punctuation** — a real consideration for notes.md readability (see +integration costs). + +## Pre-registered criteria, evaluated + +| Criterion | Result | +|---|---| +| (a) WER ≥30 % relatively better than W4 | **PASS** — 85 % better (0.47 % vs 3.11 %) | +| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — 5× better WER, ≥1000× less CPU than measured W1 (≥10× vs even a healthy W1) | +| (b) NPU rung: accelerator execution or CPU ≤ W4 | **FAIL (strictly)** — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below) | +| (c) Live windows hold real-time | **PASS** — best in field, p95 497 ms | + +**Phase-4 finding (accelerated Parakeet, timeboxed):** the sherpa route is packaging-blocked — +`sherpa-rs`'s `directml` feature is mutually exclusive with `download-binaries` +(`compile_error!` in `sherpa-rs-sys` 0.6.8), and escaping via `SHERPA_LIB_PATH` + a +version-matched prebuilt DirectML archive would require replacing the base dependency +configuration for the whole branch, with transducer-on-DirectML performance unproven (and a +prior DirectML-on-Intel-GPU crash on record). The genuine route is a TDT decode loop on our own +ort/OpenVINO stack (istupakov-style ONNX exports) — estimated **days, not hours**, and only +worth it if the NPU rung must flip. The `bench-directml` cargo feature + `p2-parakeet-dml` +contender remain in the harness as the documented attempt. + +## Recommendation (per hardware rung) + +1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** It is ~5× more + accurate than base.en, decodes at 0.049 RTF using a fraction of one core, and sidesteps the + W1 whisper.cpp-CPU performance bug outright. This is the unambiguous win. +2. **NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX** — + Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the + numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the + steadiest live latency. That trade is the user's call, not the benchmark's; if low-CPU stays + the hard constraint, the NPU rung flips only after the ort/OpenVINO TDT work lands. +3. **GPU (Vulkan) machines without NPU:** W2 is the cheapest-CPU option (0.033), but Parakeet's + accuracy gap applies here too — same judgment call as the NPU rung, cheaper to revisit after + the integration items below. + +**Integration costs before any default flip** (not solved by this bench): timestamped segments +(sherpa-rs's `transducer` API returns text only; the C API exposes timestamps — small upstream +PR or direct `-sys` call), punctuation restoration (sparse in Parakeet output; matters for +notes.md), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for +non-English), a ~660 MB model download, and the model catalog/dispatch work (same shape as the +0.7.3 merged-decoder swap). + +## Caveats + +- LibriSpeech is clean read speech; WERs are comparative, not absolute product claims. The + planned qualitative pass over real meeting audio was skipped because all retained recordings + are vault-sealed (by design); to run it, export a meeting bundle from an unlocked session and + drop the WAV into `bench-corpus/test_wavs/`. +- int8 quantization slightly penalizes Parakeet vs published fp32 figures — it *still* won by + 85 %; the fp32 bundle exists if a tighter number is ever needed. +- Single machine, single run per cell (30 files each). Differences here are far larger than + run-to-run noise, but treat third-decimal RTF differences as noise.