docs(bench): W1 anomaly resolved — LP-E-core spin-barrier root cause, fixed numbers, criteria updated

This commit is contained in:
iamdoubz committed 2026-07-16 08:44:17 -05:00
1 parent 6cd736bec1
commit be49c53940
1 file changed
+17 -11
+17 -11
View File
@@ -24,7 +24,8 @@ TDT 0.6B v2, to decide whether the default transcription model should change.
| Contender | Engine / device | WER | RTF (wall) | CPU-sec per audio-sec | Load |
|---|---|---|---|---|---|
| W1 base.en | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
| W1 base.en (as shipped, 14 thr) | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
| W1 base.en (thread-cap fix) | whisper.cpp, CPU | 2.48 % | 0.218 | 0.827 | 163 ms |
| W2 base.en | whisper.cpp Vulkan, iGPU | 2.64 % | 0.071 | **0.033** | 248 ms |
| W3 medium.en | whisper.cpp Vulkan, iGPU | 3.26 % | 0.304 | 0.088 | 724 ms |
| W4 base.en (default) | ort ONNX, NPU + iGPU | 3.11 % | 0.058 | 0.049 | ~4.7 s |
@@ -35,12 +36,16 @@ TDT 0.6B v2, to decide whether the default transcription model should change.
improvement** — while also posting the best RTF in the field, on the CPU alone.
- Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads);
the NPU+iGPU Whisper path costs ≈0.05.
- ⚠ **W1 anomaly:** whisper.cpp forced to CPU is pathologically slow in *both* the Vulkan and
non-Vulkan builds of this crate (reproduced twice; ~12 cores busy at 20× real-time). Its WER
is valid; its speed numbers are not representative of whisper.cpp generally (published CPU
RTFs for base.en are ~0.3–0.5) and point at a real product bug for CPU-only machines — prime
suspect: the release profile's `opt-level = "z"` applied to ggml's kernels. **Follow-up filed;
even a healthy W1 would lose the CPU rung to P1 by an order of magnitude.**
- ✅ **W1 anomaly RESOLVED (2026-07-16 follow-up):** root cause was `n_threads =
available_parallelism()` (14) putting ggml spin-barrier workers on the two slow LP E-cores —
every graph-node barrier waits on the slowest worker while the rest burn CPU. 12 threads is
healthy (RTF 0.200), 14 collapses (RTF ~22). The original suspect (`opt-level = "z"` /
unoptimized ggml) was **disproven**: a verified `/O2` build was equally slow. Fix: default
capped at 4 threads, whisper.cpp's own upstream default (`WA_WHISPER_THREADS` env overrides);
re-run row above — RTF 20.1 → 0.218 (92×), CPU 232.6 → 0.827 (281×), WER identical.
(The build *was* also silently unoptimized — the cmake crate clobbers MSVC `/O2` when no
generator is set; fixed via `CMAKE_GENERATOR=Ninja` in `.cargo/config.toml` as build hygiene,
though it wasn't the bottleneck.) Raw data: `bench-results/raw-w1-fixed.csv`.
- W3 (medium) scoring *worse* than base on this corpus (21 vs 17 errors) is small-sample noise
on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general.
@@ -95,7 +100,7 @@ crosstalk remains untested.
| Criterion | Result |
|---|---|
| (a) WER ≥30 % relatively better than W4 | **PASS** — 85 % better (0.47 % vs 3.11 %) |
| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — 5× better WER, ≥1000× less CPU than measured W1 (≥10× vs even a healthy W1) |
| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — vs the *fixed* W1 (follow-up): 5.3× better WER (0.47 % vs 2.48 %), 3.7× less CPU (0.224 vs 0.827 cpu-sec/audio-sec) |
| (b) NPU rung: accelerator execution or CPU ≤ W4 | **FAIL (strictly)** — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below) |
| (c) Live windows hold real-time | **PASS** — best in field, p95 497 ms |
@@ -111,9 +116,10 @@ contender remain in the harness as the documented attempt.
## Recommendation (per hardware rung)
1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** It is ~5× more
accurate than base.en, decodes at 0.049 RTF using a fraction of one core, and sidesteps the
W1 whisper.cpp-CPU performance bug outright. This is the unambiguous win.
1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** Even against the
*fixed* W1 (thread cap, follow-up above) it is ~5× more accurate and 3.7× cheaper in
CPU-seconds at comparable wall RTF. This is the unambiguous win — just no longer by three
orders of magnitude, since the W1 pathology itself is fixed.
2. **NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX** —
Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the
numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the