# Parakeet vs Whisper — default-model bake-off (2026-07-16)

Pre-registered comparison (criteria fixed before any numbers existed — see the plan on branch
`test_parakeet`) between WhispAssist's shipping Whisper `base.en` paths and NVIDIA Parakeet
TDT 0.6B v2, to decide whether the default transcription model should change.

## Provenance

- **Machine:** Intel Core Ultra 5 135U (Meteor Lake: 12C/14T CPU, Intel Arc iGPU, Intel AI Boost
  NPU, driver 32.0.100.4724), Windows 11 Pro 26200. App version under test: 0.7.3.
- **Harness:** `src-tauri/examples/asr_bench.rs` — drives the *production* transcribers
  (`WhisperTranscriber`, `OnnxTranscriber`) and a sherpa-onnx `TransducerRecognizer`; one
  contender per process (clean CPU accounting). WER = word-level Levenshtein after
  lowercase/punctuation-strip normalization (self-tested).
- **Corpus:** first 30 LibriSpeech test-clean utterances (241.7 s, 644 reference words), 16 kHz
  mono, staged by `scripts/bench/fetch-corpus.ps1`. Live latency: sequential 10 s windows over a
  3-minute concatenation (16 windows).
- **Models:** ggml `base.en-q5_1`, `medium.en-q5_0`; WA ONNX `base.en` (merged decoder, encoder
  on OpenVINO/NPU + decoder on OpenVINO/GPU — EP confirmed in-run);
  `sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8` (encoder 652 MB + decoder 7 MB + joiner 2 MB).
- **Raw data:** `bench-results/raw.csv`, `bench-results/raw-w1-novulkan.csv` (committed).

## Results — corpus (accuracy + cost)

| Contender | Engine / device | WER | RTF (wall) | CPU-sec per audio-sec | Load |
|---|---|---|---|---|---|
| W1 base.en (as shipped, 14 thr) | whisper.cpp, CPU | 2.48 % | ⚠ 20.1 | ⚠ 232.6 | 112 ms |
| W1 base.en (thread-cap fix) | whisper.cpp, CPU | 2.48 % | 0.218 | 0.827 | 163 ms |
| W2 base.en | whisper.cpp Vulkan, iGPU | 2.64 % | 0.071 | **0.033** | 248 ms |
| W3 medium.en | whisper.cpp Vulkan, iGPU | 3.26 % | 0.304 | 0.088 | 724 ms |
| W4 base.en (default) | ort ONNX, NPU + iGPU | 3.11 % | 0.058 | 0.049 | ~4.7 s |
| **P1 parakeet int8, t8** | sherpa-onnx, CPU | **0.47 %** | **0.047** | 0.396 | ~4.6 s |
| **P1 parakeet int8, t4** | sherpa-onnx, CPU | **0.47 %** | 0.049 | 0.224 | ~4.6 s |

- **Parakeet: 3 errors in 644 words**, vs 20 for the current default — an **85 % relative WER
  improvement** — while also posting the best RTF in the field, on the CPU alone.
- Parakeet at 4 threads costs ≈0.22 CPU-cores sustained (~1.6 % of this machine's 14 threads);
  the NPU+iGPU Whisper path costs ≈0.05.
- ✅ **W1 anomaly RESOLVED (2026-07-16 follow-up):** root cause was `n_threads =
  available_parallelism()` (14) putting ggml spin-barrier workers on the two slow LP E-cores —
  every graph-node barrier waits on the slowest worker while the rest burn CPU. 12 threads is
  healthy (RTF 0.200), 14 collapses (RTF ~22). The original suspect (`opt-level = "z"` /
  unoptimized ggml) was **disproven**: a verified `/O2` build was equally slow. Fix: default
  capped at 4 threads, whisper.cpp's own upstream default (`WA_WHISPER_THREADS` env overrides);
  re-run row above — RTF 20.1 → 0.218 (92×), CPU 232.6 → 0.827 (281×), WER identical.
  (The build *was* also silently unoptimized — the cmake crate clobbers MSVC `/O2` when no
  generator is set; fixed via `CMAKE_GENERATOR=Ninja` in `.cargo/config.toml` as build hygiene,
  though it wasn't the bottleneck.) Raw data: `bench-results/raw-w1-fixed.csv`.
- W3 (medium) scoring *worse* than base on this corpus (21 vs 17 errors) is small-sample noise
  on clean read speech; treat W3's WER as "≈base" here, not as medium being worse in general.

## Results — live windows (10 s, n=16)

| Contender | p50 | p95 | max |
|---|---|---|---|
| W2 Vulkan base.en | 1055 ms | 3722 ms | 4148 ms |
| W4 NPU+iGPU (default) | 519 ms | 630 ms | 928 ms |
| **P1 parakeet t8 (CPU)** | **450 ms** | **497 ms** | **514 ms** |

All hold real-time; Parakeet is fastest **and** near-flat (64 ms p50→max spread), which would
make the live transcript feel noticeably steadier.

## Evidence sample (file 1089-134686-0000, 10.4 s)

> **Reference:** HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND CARROTS AND BRUISED
> POTATOES AND FAT MUTTON PIECES TO BE LADLED OUT IN THICK PEPPERED FLOUR FATTENED SAUCE

- **W4 (default):** "He hoped there would be stew for dinner, turnips and carrots and bruised
  potatoes and fat mutton pieces to be ladled out in thick peppered **flour-fatted** sauce." (2 err)
- **P1 (parakeet):** " He hoped there would be stew for dinner turnips and carrots and bruised
  potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce" (0 err)

Note the trade visible even here: Whisper punctuates richly; Parakeet's int8 output carries
casing but **sparse punctuation** — a real consideration for notes.md readability (see
integration costs).

## Qualitative pass — real recording (2026-07-16 follow-up)

`bench-corpus/test_wavs/bank_deposits.wav` (62 s real recording, 48 kHz stereo — financial
disclosure read aloud; downmixed/resampled by the production reader). No reference transcript,
so transcripts were compared by eye; raw hypotheses in `bench-results/raw-qual-bank-deposits.csv`
(its WER column is against a placeholder — ignore it).

- **P1 (parakeet t4):** cleanest of the three — zero audible-word errors, and the **best
  punctuation**, including commas around a parenthetical clause ("…the bank deposit details
  section, which appears later in this statement, for information…"). The "sparse punctuation"
  concern from the LibriSpeech sample did **not** reproduce on this longer real clip.
- **W2 (Vulkan base.en):** near-perfect; one function-word slip ("effective for cash balances
  *and* your FDIC insured…" — W4 and P1 both hear "*in* your").
- **W4 (NPU ONNX, shipping default):** worst of the three — "details" → "**D-Tales**" twice,
  "are SIPC protected" → "**or** SIPC protected", and a segmentation break at the end
  ("statement period" → "**state. period.**").

One clip, one speaker, clean audio — directionally consistent with the corpus WERs
(P1 ≥ W2 > W4), and it removes punctuation as an argument against Parakeet. Multi-speaker
crosstalk remains untested.

## Pre-registered criteria, evaluated

| Criterion | Result |
|---|---|
| (a) WER ≥30 % relatively better than W4 | **PASS** — 85 % better (0.47 % vs 3.11 %) |
| (b) CPU rung: beat W1 on WER **and** CPU-seconds | **PASS** — vs the *fixed* W1 (follow-up): 5.3× better WER (0.47 % vs 2.48 %), 3.7× less CPU (0.224 vs 0.827 cpu-sec/audio-sec) |
| (b) NPU rung: accelerator execution or CPU ≤ W4 | **FAIL (strictly)** — 0.224 vs 0.049 CPU-sec/audio-sec; accelerated Parakeet not achieved (below) |
| (c) Live windows hold real-time | **PASS** — best in field, p95 497 ms |

**Phase-4 finding (accelerated Parakeet, timeboxed):** the sherpa route is packaging-blocked —
`sherpa-rs`'s `directml` feature is mutually exclusive with `download-binaries`
(`compile_error!` in `sherpa-rs-sys` 0.6.8), and escaping via `SHERPA_LIB_PATH` + a
version-matched prebuilt DirectML archive would require replacing the base dependency
configuration for the whole branch, with transducer-on-DirectML performance unproven (and a
prior DirectML-on-Intel-GPU crash on record). The genuine route is a TDT decode loop on our own
ort/OpenVINO stack (istupakov-style ONNX exports) — estimated **days, not hours**, and only
worth it if the NPU rung must flip. The `bench-directml` cargo feature + `p2-parakeet-dml`
contender remain in the harness as the documented attempt.

## Recommendation (per hardware rung)

1. **CPU-only machines: change the default to Parakeet TDT 0.6B v2 int8.** Even against the
   *fixed* W1 (thread cap, follow-up above) it is ~5× more accurate and 3.7× cheaper in
   CPU-seconds at comparable wall RTF. This is the unambiguous win — just no longer by three
   orders of magnitude, since the W1 pathology itself is fixed.
2. **NPU+iGPU machines (this machine): the pre-registered rule says keep Whisper-ONNX** —
   Parakeet-on-CPU costs 4.6× the CPU of the current near-idle path. The override argument the
   numbers support: 0.22 cores is still objectively tiny, and it buys 85 % fewer errors plus the
   steadiest live latency. That trade is the user's call, not the benchmark's; if low-CPU stays
   the hard constraint, the NPU rung flips only after the ort/OpenVINO TDT work lands.
3. **GPU (Vulkan) machines without NPU:** W2 is the cheapest-CPU option (0.033), but Parakeet's
   accuracy gap applies here too — same judgment call as the NPU rung, cheaper to revisit after
   the integration items below.

**Integration costs before any default flip** (not solved by this bench): timestamped segments
(sherpa-rs's `transducer` API returns text only; the C API exposes timestamps — small upstream
PR or direct `-sys` call), punctuation restoration (sparse on short LibriSpeech clips, though
the real-recording qualitative pass showed *good* punctuation on longer audio — may be a
non-issue), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for
non-English), a ~660 MB model download, and the model catalog/dispatch work (same shape as the
0.7.3 merged-decoder swap).

## Caveats

- LibriSpeech is clean read speech; WERs are comparative, not absolute product claims. The
  qualitative pass over a real recording was run as a follow-up (section above) — still
  single-speaker; multi-speaker meeting audio with crosstalk remains untested.
- int8 quantization slightly penalizes Parakeet vs published fp32 figures — it *still* won by
  85 %; the fp32 bundle exists if a tighter number is ever needed.
- Single machine, single run per cell (30 files each). Differences here are far larger than
  run-to-run noise, but treat third-decimal RTF differences as noise.