Parakeet v Whisper h2h #26

2026-07-16T10:23:09-05:00

iamdoubz commented

2026-07-16 10:23:09 -05:00

This comparison found a huge issue when compiling for CPU and this pull request increases throughput for CPU setups by 92x... so yeah, it was bad to start with.

This comparison found a huge issue when compiling for CPU and this pull request increases throughput for CPU setups by **92x**... so yeah, it was bad to start with.

iamdoubz added 21 commits 2026-07-16 10:23:09 -05:00

test(bench): corpus staging script + gitignore for bench-corpus (parakeet bake-off) 96072a2f75

test(bench): parakeet-tdt-0.6b-v2 int8 staging script 965f1adc86

test(bench): typeperf CPU/GPU/NPU counter capture script 4dddcdfe1a

test(bench): asr_bench harness — W1-W4 + Parakeet contenders, WER/RTF/CPU-seconds, live-window mode ad4495f220

test(bench): sequential matrix runner (6 corpus + 3 live runs) 297c412488

fix(bench): named Set-Content params — positional+NoNewline silently wrote nothing on PS 5.1 68632db6ef

test(bench): bench-directml feature for DirectML sherpa binaries (P2, never default) a682ffccdd

test(bench): p2-parakeet-dml contender (Parakeet on the iGPU via DirectML EP) ca0563039d

docs(bench): parakeet-vs-whisper results — pre-registered criteria evaluated, per-rung recommendation 21691d3756

test(bench): raw benchmark CSVs (30-utterance LibriSpeech matrix + W1 non-vulkan rerun) 32e0d3c8d9

fix(build): restore /O2 /DNDEBUG for native deps clobbered by cmake-rs on MSVC d89b99db6c

cmake-rs hardcodes opt_level(0) and replaces CMake's Release C flags, so
ggml/whisper.cpp compiled at /Od — root cause of the W1 CPU-pathology
(RTF ~20) in docs/benchmarks/2026-07-parakeet-vs-whisper.md. cc-rs appends
env CFLAGS/CXXFLAGS last, restoring optimization for all native deps.

test(bench): qualitative meeting-audio pass — W2/W4/P1 on bank_deposits.wav 181af150e5

62 s real recording (48 kHz stereo, downmixed/resampled by the production
reader). Reference .txt is a placeholder; WER column is meaningless here,
only the hypothesis text counts.

docs(bench): qualitative real-recording pass — P1 cleanest + best punctuation, W4 worst 4e6c5a357e

fix(build): use Ninja generator so cmake-rs stops clobbering MSVC /O2 c7899e72a2

The cmake crate strips -O*/O* from CFLAGS and, with no explicit generator
on MSVC, overwrites CMAKE_C_FLAGS_RELEASE — so the previous CFLAGS=/O2
attempt could never work (verified: /DNDEBUG survived, /O2 stripped).
With CMAKE_GENERATOR=Ninja the override branch is skipped and CMake's
default Release flags (/O2 /Ob2 /DNDEBUG) reach ggml.

fix(transcription): cap whisper n_threads at 4 — hybrid-CPU spin-barrier pathology 0d2eee7416

available_parallelism() put ggml workers on Meteor Lake LP E-cores; every
per-node spin barrier then waits on the slowest worker while the rest burn
CPU: RTF ~22 at 14 threads vs 0.28 at 4 threads (~80x), same build. This
was the real W1 CPU-pathology root cause; /Od-vs-/O2 made no difference.
4 matches whisper.cpp upstream default and is the CPU-frugal choice
(NFR-RES-1). WA_WHISPER_THREADS env overrides for support/bench.

test(bench): W1 re-run after thread-cap fix — RTF 0.218, 0.827 cpu-sec/audio-sec, WER unchanged 6cd736bec1

docs(bench): W1 anomaly resolved — LP-E-core spin-barrier root cause, fixed numbers, criteria updated be49c53940

chore(bench): rustfmt asr_bench + allow large_enum_variant (one Engine per process) 616d2f4f97

chore: bump version 0.7.4 fcf72da9db

docs(release): v0.7.4 release notes with SHA-256 checksums a71299803f

Bench test npu 2d1c9fa478

iamdoubz merged commit 8cdd0f61df into main

2026-07-16 10:23:17 -05:00

iamdoubz deleted branch test_parakeet

2026-07-16 10:23:17 -05:00

Sign in to join this conversation.