ADR-0004 — Hardware acceleration strategy & detection ladder

Status: Accepted
Date: 2026-06-30
Context source: Design doc §"Hardware Detection and Transcription Strategy"

Context

The design mandates a strict preference order NPU → NVIDIA → AMD → Intel → CPU, detected at startup and on demand, with graceful fallback and the chosen backend surfaced to the user.

Decision

Introduce a hardware module that performs capability detection and returns a ranked list of available Backends; the transcription layer selects the highest-ranked one and constructs the matching Transcriber.

Detection mapping:

Tier	Detection	Execution path
NPU	ONNX Runtime / Windows ML enumeration; DirectML `device_filter="npu"`	ONNX `Transcriber` (Whisper-ONNX) via `ort` + DirectML
NVIDIA GPU	adapter enumeration (DXGI) + CUDA availability	whisper.cpp CUDA
AMD GPU	DXGI adapter; Vulkan device present	whisper.cpp Vulkan (DirectML alt.)
Intel GPU	DXGI adapter; Vulkan device present	whisper.cpp Vulkan
CPU	always	whisper.cpp CPU (AVX2/AVX-512 if present)

ONNX Runtime execution providers also fall back internally (NPU→GPU→CPU) per-operator, giving a second safety net on the NPU path.
The selected backend, model size, and an estimated real-time factor are reported via the hardware_status command and shown in Settings + the recording bar (FR-HW-3).
A user override lets advanced users pin a specific backend (FR-HW-2); "Low overhead" preset forces CPU + small model + no real-time summarization (NFR-RES-3).

Consequences

Positive: one abstraction satisfies the full ladder; per-operator EP fallback makes the NPU path robust; clean place to add future vendors/runtimes (Windows ML is abstracted behind Backend, so a DirectML→Windows ML migration is local to this module).
Negative: NPU support is hardware/driver dependent and the newest, least-tested tier — must degrade silently to GPU/CPU and never hard-fail; requires pinning ort↔ONNX Runtime binary versions on Windows builds.

Revisit if

Windows ML supersedes DirectML for our models, or a vendor SDK gives materially better NPU throughput than the ONNX/DirectML path.

Update (2026-07-16): NPU path uses OpenVINO; decoder split off the CPU

As built (T3.4 + follow-up), the NPU tier runs on ONNX Runtime + the OpenVINO EP (device_type=NPU), not DirectML — Intel's prebuilt OpenVINO ORT runtime is downloaded on demand (DirectML remains the GPU fallback EP for AMD/Intel in non-Vulkan builds). The engine splits Whisper across devices:

encoder (fixed shape): OpenVINO NPU;
decoder (autoregressive, KV-cache greedy via Optimum's merged export): OpenVINO GPU (Intel iGPU, same runtime bundle) when available, else the CPU EP — WA_ONNX_DECODER_DEVICE=cpu|gpu overrides.

Rationale: the dynamic-shaped decode loop is a poor NPU fit, and pre-KV-cache it re-ran the full token prefix every step on the CPU — the dominant CPU cost while transcribing. With the merged decoder + iGPU placement, sustained CPU during transcription is near-idle on NPU+iGPU systems. The engine logs both EPs at load (transcription engine: ONNX …).

3.2 KiB Raw Blame History