docs/benchmarks/2026-07-parakeet-vs-whisper.md

@@ -24,7 +24,8 @@ TDT 0.6B v2, to decide whether the default transcription model should change.
 
 | Contender | Engine / device | WER | RTF (wall) | CPU-sec per audio-sec | Load |
 |---|---|---|---|---|---|
-| W1 base.en | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
+| W1 base.en (as shipped, 14 thr) | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
+| W1 base.en (thread-cap fix) | whisper.cpp, CPU | 2.48 % | 0.218 | 0.827 | 163 ms |
 | W2 base.en | whisper.cpp Vulkan, iGPU | 2.64 % | 0.071 | **0.033** | 248 ms |
 | W3 medium.en | whisper.cpp Vulkan, iGPU | 3.26 % | 0.304 | 0.088 | 724 ms |
 | W4 base.en (default) | ort ONNX, NPU + iGPU | 3.11 % | 0.058 | 0.049 | ~4.7 s |
@@ -35,12 +36,16 @@ TDT 0.6B v2, to decide whether the default transcription model should change.
   improvement** — while also posting the best RTF in the field, on the CPU alone.
 - Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads);
   the NPU+iGPU Whisper path costs ≈0.05.
-- ⚠ **W1 anomaly:** whisper.cpp forced to CPU is pathologically slow in *both* the Vulkan and
-  non-Vulkan builds of this crate (reproduced twice; ~12 cores busy at 20× real-time). Its WER
-  is valid; its speed numbers are not representative of whisper.cpp generally (published CPU
-  RTFs for base.en are ~0.3–0.5) and point at a real product bug for CPU-only machines — prime
-  suspect: the release profile's `opt-level = "z"` applied to ggml's kernels. **Follow-up filed;
-  even a healthy W1 would lose the CPU rung to P1 by an order of magnitude.**
+- ✅ **W1 anomaly RESOLVED (2026-07-16 follow-up):** root cause was `n_threads =
+  available_parallelism()` (14) putting ggml spin-barrier workers on the two slow LP E-cores —
+  every graph-node barrier waits on the slowest worker while the rest burn CPU. 12 threads is
+  healthy (RTF 0.200), 14 collapses (RTF ~22). The original suspect (`opt-level = "z"` /
+  unoptimized ggml) was **disproven**: a verified `/O2` build was equally slow. Fix: default
+  capped at 4 threads, whisper.cpp's own upstream default (`WA_WHISPER_THREADS` env overrides);
+  re-run row above — RTF 20.1 → 0.218 (92×), CPU 232.6 → 0.827 (281×), WER identical.
+  (The build *was* also silently unoptimized — the cmake crate clobbers MSVC `/O2` when no
+  generator is set; fixed via `CMAKE_GENERATOR=Ninja` in `.cargo/config.toml` as build hygiene,
+  though it wasn't the bottleneck.) Raw data: `bench-results/raw-w1-fixed.csv`.
 - W3 (medium) scoring *worse* than base on this corpus (21 vs 17 errors) is small-sample noise
   on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general.
 
@@ -95,7 +100,7 @@ crosstalk remains untested.
 | Criterion | Result |
 |---|---|
 | (a) WER ≥30 % relatively better than W4 | **PASS** — 85 % better (0.47 % vs 3.11 %) |
-| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — 5× better WER, ≥1000× less CPU than measured W1 (≥10× vs even a healthy W1) |
+| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — vs the *fixed* W1 (follow-up): 5.3× better WER (0.47 % vs 2.48 %), 3.7× less CPU (0.224 vs 0.827 cpu-sec/audio-sec) |
 | (b) NPU rung: accelerator execution or CPU ≤ W4 | **FAIL (strictly)** — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below) |
 | (c) Live windows hold real-time | **PASS** — best in field, p95 497 ms |
 
@@ -111,9 +116,10 @@ contender remain in the harness as the documented attempt.
 
 ## Recommendation (per hardware rung)
 
-1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** It is ~5× more
-   accurate than base.en, decodes at 0.049 RTF using a fraction of one core, and sidesteps the
-   W1 whisper.cpp-CPU performance bug outright. This is the unambiguous win.
+1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** Even against the
+   *fixed* W1 (thread cap, follow-up above) it is ~5× more accurate and 3.7× cheaper in
+   CPU-seconds at comparable wall RTF. This is the unambiguous win — just no longer by three
+   orders of magnitude, since the W1 pathology itself is fixed.
 2. **NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX** —
    Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the
    numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the