diff --git a/docs/benchmarks/2026-07-parakeet-vs-whisper.md b/docs/benchmarks/2026-07-parakeet-vs-whisper.md index 25481a0..449f3b7 100644 --- a/docs/benchmarks/2026-07-parakeet-vs-whisper.md +++ b/docs/benchmarks/2026-07-parakeet-vs-whisper.md @@ -69,6 +69,27 @@ Note the trade visible even here: Whisper punctuates richly; Parakeet's int8 out casing but **sparse punctuation** — a real consideration for notes.md readability (see integration costs). +## Qualitative pass — real recording (2026-07-16 follow-up) + +`bench-corpus/test_wavs/bank_deposits.wav` (62 s real recording, 48 kHz stereo — financial +disclosure read aloud; downmixed/resampled by the production reader). No reference transcript, +so transcripts were compared by eye; raw hypotheses in `bench-results/raw-qual-bank-deposits.csv` +(its WER column is against a placeholder — ignore it). + +- **P1 (parakeet t4):** cleanest of the three — zero audible-word errors, and the **best + punctuation**, including commas around a parenthetical clause ("…the bank deposit details + section, which appears later in this statement, for information…"). The "sparse punctuation" + concern from the LibriSpeech sample did **not** reproduce on this longer real clip. +- **W2 (Vulkan base.en):** near-perfect; one function-word slip ("effective for cash balances + *and* your FDIC insured…" — W4 and P1 both hear "*in* your"). +- **W4 (NPU ONNX, shipping default):** worst of the three — "details" → "**D-Tales**" twice, + "are SIPC protected" → "**or** SIPC protected", and a segmentation break at the end + ("statement period" → "**state. period.**"). + +One clip, one speaker, clean audio — directionally consistent with the corpus WERs +(P1 ≥ W2 > W4), and it removes punctuation as an argument against Parakeet. Multi-speaker +crosstalk remains untested. + ## Pre-registered criteria, evaluated | Criterion | Result | @@ -104,17 +125,17 @@ contender remain in the harness as the documented attempt. **Integration costs before any default flip** (not solved by this bench): timestamped segments (sherpa-rs's `transducer` API returns text only; the C API exposes timestamps — small upstream -PR or direct `-sys` call), punctuation restoration (sparse in Parakeet output; matters for -notes.md), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for +PR or direct `-sys` call), punctuation restoration (sparse on short LibriSpeech clips, though +the real-recording qualitative pass showed *good* punctuation on longer audio — may be a +non-issue), English-only v2 (v3 is multilingual — the whisper multilingual path must remain for non-English), a ~660 MB model download, and the model catalog/dispatch work (same shape as the 0.7.3 merged-decoder swap). ## Caveats - LibriSpeech is clean read speech; WERs are comparative, not absolute product claims. The - planned qualitative pass over real meeting audio was skipped because all retained recordings - are vault-sealed (by design); to run it, export a meeting bundle from an unlocked session and - drop the WAV into `bench-corpus/test_wavs/`. + qualitative pass over a real recording was run as a follow-up (section above) — still + single-speaker; multi-speaker meeting audio with crosstalk remains untested. - int8 quantization slightly penalizes Parakeet vs published fp32 figures — it *still* won by 85 %; the fp32 bundle exists if a tighter number is ever needed. - Single machine, single run per cell (30 files each). Differences here are far larger than