Feature chore bug 010 #25

Merged
iamdoubz merged 10 commits from feature_chore_bug_010 into main 2026-07-15 20:42:25 -05:00
Owner
No description provided.
iamdoubz added 10 commits 2026-07-15 20:42:08 -05:00
Merged-decoder (use_cache_branch/past_key_values) loop discovered from the
graph at load; step 1 runs the no-cache branch over the prompt, later steps
feed one token + cached KVs (decoder slots refreshed, encoder cross-attn KVs
kept from step 1). Decoder session lands on the Intel iGPU via the OpenVINO
GPU plugin (same staged runtime) with CPU-EP fallback and a
WA_ONNX_DECODER_DEVICE=cpu|gpu override; DirectML backends keep the CPU
decoder. Legacy decoder_model.onnx is removed once the merged graph loads.

Measured (Core Ultra 5 135U, base.en, 19s TTS clip, identical transcripts):
decoder on OpenVINO/GPU infer=1288ms vs CPU-EP infer=6098ms
iamdoubz merged commit 87fb665e9e into main 2026-07-15 20:42:25 -05:00
iamdoubz deleted branch feature_chore_bug_010 2026-07-15 20:42:25 -05:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: iamdoubz/WhispAssist#25