docs/benchmarks/2026-07-parakeet-vs-whisper.md

@@ -0,0 +1,121 @@
+# Parakeet vs Whisper — default-model bake-off (2026-07-16)
+
+Pre-registered comparison (criteria fixed before any numbers existed — see the plan on branch
+`test_parakeet`) between WhispAssist's shipping Whisper `base.en` paths and NVIDIA Parakeet
+TDT 0.6B v2, to decide whether the default transcription model should change.
+
+## Provenance
+
+- **Machine:** Intel Core Ultra 5 135U (Meteor Lake: 12C/14T CPU, Intel Arc iGPU, Intel AI Boost
+  NPU, driver 32.0.100.4724), Windows 11 Pro 26200. App version under test: 0.7.3.
+- **Harness:** `src-tauri/examples/asr_bench.rs` — drives the *production* transcribers
+  (`WhisperTranscriber`, `OnnxTranscriber`) and a sherpa-onnx `TransducerRecognizer`; one
+  contender per process (clean CPU accounting). WER = word-level Levenshtein after
+  lowercase/punctuation-strip normalization (self-tested).
+- **Corpus:** first 30 LibriSpeech test-clean utterances (241.7 s, 644 reference words), 16 kHz
+  mono, staged by `scripts/bench/fetch-corpus.ps1`. Live latency: sequential 10 s windows over a
+  3-minute concatenation (16 windows).
+- **Models:** ggml `base.en-q5_1`, `medium.en-q5_0`; WA ONNX `base.en` (merged decoder, encoder
+  on OpenVINO/NPU + decoder on OpenVINO/GPU — EP confirmed in-run);
+  `sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8` (encoder 652 MB + decoder 7 MB + joiner 2 MB).
+- **Raw data:** `bench-results/raw.csv`, `bench-results/raw-w1-novulkan.csv` (committed).
+
+## Results — corpus (accuracy + cost)
+
+| Contender | Engine / device | WER | RTF (wall) | CPU-sec per audio-sec | Load |
+|---|---|---|---|---|---|
+| W1 base.en | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
+| W2 base.en | whisper.cpp Vulkan, iGPU | 2.64 % | 0.071 | **0.033** | 248 ms |
+| W3 medium.en | whisper.cpp Vulkan, iGPU | 3.26 % | 0.304 | 0.088 | 724 ms |
+| W4 base.en (default) | ort ONNX, NPU + iGPU | 3.11 % | 0.058 | 0.049 | ~4.7 s |
+| **P1 parakeet int8, t8** | sherpa-onnx, CPU | **0.47 %** | **0.047** | 0.396 | ~4.6 s |
+| **P1 parakeet int8, t4** | sherpa-onnx, CPU | **0.47 %** | 0.049 | 0.224 | ~4.6 s |
+
+- **Parakeet: 3 errors in 644 words**, vs 20 for the current default — an **85 % relative WER
+  improvement** — while also posting the best RTF in the field, on the CPU alone.
+- Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads);
+  the NPU+iGPU Whisper path costs ≈0.05.
+- ⚠ **W1 anomaly:** whisper.cpp forced to CPU is pathologically slow in *both* the Vulkan and
+  non-Vulkan builds of this crate (reproduced twice; ~12 cores busy at 20× real-time). Its WER
+  is valid; its speed numbers are not representative of whisper.cpp generally (published CPU
+  RTFs for base.en are ~0.3–0.5) and point at a real product bug for CPU-only machines — prime
+  suspect: the release profile's `opt-level = "z"` applied to ggml's kernels. **Follow-up filed;
+  even a healthy W1 would lose the CPU rung to P1 by an order of magnitude.**
+- W3 (medium) scoring *worse* than base on this corpus (21 vs 17 errors) is small-sample noise
+  on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general.
+
+## Results — live windows (10 s, n=16)
+
+| Contender | p50 | p95 | max |
+|---|---|---|---|
+| W2 Vulkan base.en | 1055 ms | 3722 ms | 4148 ms |
+| W4 NPU+iGPU (default) | 519 ms | 630 ms | 928 ms |
+| **P1 parakeet t8 (CPU)** | **450 ms** | **497 ms** | **514 ms** |
+
+All hold real-time; Parakeet is fastest **and** near-flat (64 ms p50→max spread), which would
+make the live transcript feel noticeably steadier.
+
+## Evidence sample (file 1089-134686-0000, 10.4 s)
+
+> **Reference:** HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND CARROTS AND BRUISED
+> POTATOES AND FAT MUTTON PIECES TO BE LADLED OUT IN THICK PEPPERED FLOUR FATTENED SAUCE
+
+- **W4 (default):** "He hoped there would be stew for dinner, turnips and carrots and bruised
+  potatoes and fat mutton pieces to be ladled out in thick peppered **flour-fatted** sauce." (2 err)
+- **P1 (parakeet):** " He hoped there would be stew for dinner turnips and carrots and bruised
+  potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce" (0 err)
+
+Note the trade visible even here: Whisper punctuates richly; Parakeet's int8 output carries
+casing but **sparse punctuation** — a real consideration for notes.md readability (see
+integration costs).
+
+## Pre-registered criteria, evaluated
+
+| Criterion | Result |
+|---|---|
+| (a) WER ≥30 % relatively better than W4 | **PASS** — 85 % better (0.47 % vs 3.11 %) |
+| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — 5× better WER, ≥1000× less CPU than measured W1 (≥10× vs even a healthy W1) |
+| (b) NPU rung: accelerator execution or CPU ≤ W4 | **FAIL (strictly)** — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below) |
+| (c) Live windows hold real-time | **PASS** — best in field, p95 497 ms |
+
+**Phase-4 finding (accelerated Parakeet, timeboxed):** the sherpa route is packaging-blocked —
+`sherpa-rs`'s `directml` feature is mutually exclusive with `download-binaries`
+(`compile_error!` in `sherpa-rs-sys` 0.6.8), and escaping via `SHERPA_LIB_PATH` + a
+version-matched prebuilt DirectML archive would require replacing the base dependency
+configuration for the whole branch, with transducer-on-DirectML performance unproven (and a
+prior DirectML-on-Intel-GPU crash on record). The genuine route is a TDT decode loop on our own
+ort/OpenVINO stack (istupakov-style ONNX exports) — estimated **days, not hours**, and only
+worth it if the NPU rung must flip. The `bench-directml` cargo feature + `p2-parakeet-dml`
+contender remain in the harness as the documented attempt.
+
+## Recommendation (per hardware rung)
+
+1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** It is ~5× more
+   accurate than base.en, decodes at 0.049 RTF using a fraction of one core, and sidesteps the
+   W1 whisper.cpp-CPU performance bug outright. This is the unambiguous win.
+2. **NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX** —
+   Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the
+   numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the
+   steadiest live latency. That trade is the user's call, not the benchmark's; if low-CPU stays
+   the hard constraint, the NPU rung flips only after the ort/OpenVINO TDT work lands.
+3. **GPU (Vulkan) machines without NPU:** W2 is the cheapest-CPU option (0.033), but Parakeet's
+   accuracy gap applies here too — same judgment call as the NPU rung, cheaper to revisit after
+   the integration items below.
+
+**Integration costs before any default flip** (not solved by this bench): timestamped segments
+(sherpa-rs's `transducer` API returns text only; the C API exposes timestamps — small upstream
+PR or direct `-sys` call), punctuation restoration (sparse in Parakeet output; matters for
+notes.md), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for
+non-English), a ~660 MB model download, and the model catalog/dispatch work (same shape as the
+0.7.3 merged-decoder swap).
+
+## Caveats
+
+- LibriSpeech is clean read speech; WERs are comparative, not absolute product claims. The
+  planned qualitative pass over real meeting audio was skipped because all retained recordings
+  are vault-sealed (by design); to run it, export a meeting bundle from an unlocked session and
+  drop the WAV into `bench-corpus/test_wavs/`.
+- int8 quantization slightly penalizes Parakeet vs published fp32 figures — it *still* won by
+  85 %; the fp32 bundle exists if a tighter number is ever needed.
+- Single machine, single run per cell (30 files each). Differences here are far larger than
+  run-to-run noise, but treat third-decimal RTF differences as noise.