64 lines
3.2 KiB
Markdown
64 lines
3.2 KiB
Markdown
# ADR-0004 — Hardware acceleration strategy & detection ladder
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-06-30
|
|
- **Context source:** Design doc §"Hardware Detection and Transcription Strategy"
|
|
|
|
## Context
|
|
|
|
The design mandates a strict preference order **NPU → NVIDIA → AMD → Intel → CPU**, detected at
|
|
startup and on demand, with graceful fallback and the chosen backend surfaced to the user.
|
|
|
|
## Decision
|
|
|
|
Introduce a `hardware` module that performs **capability detection** and returns a ranked list of
|
|
available `Backend`s; the transcription layer selects the highest-ranked one and constructs the
|
|
matching `Transcriber`.
|
|
|
|
Detection mapping:
|
|
|
|
| Tier | Detection | Execution path |
|
|
|---|---|---|
|
|
| NPU | ONNX Runtime / Windows ML enumeration; DirectML `device_filter="npu"` | ONNX `Transcriber` (Whisper-ONNX) via `ort` + DirectML |
|
|
| NVIDIA GPU | adapter enumeration (DXGI) + CUDA availability | whisper.cpp **CUDA** |
|
|
| AMD GPU | DXGI adapter; Vulkan device present | whisper.cpp **Vulkan** (DirectML alt.) |
|
|
| Intel GPU | DXGI adapter; Vulkan device present | whisper.cpp **Vulkan** |
|
|
| CPU | always | whisper.cpp **CPU** (AVX2/AVX-512 if present) |
|
|
|
|
- ONNX Runtime execution providers also fall back internally (NPU→GPU→CPU) per-operator, giving a
|
|
second safety net on the NPU path.
|
|
- The selected backend, model size, and an estimated real-time factor are reported via the
|
|
`hardware_status` command and shown in Settings + the recording bar (FR-HW-3).
|
|
- A user override lets advanced users pin a specific backend (FR-HW-2); "Low overhead" preset
|
|
forces CPU + small model + no real-time summarization (NFR-RES-3).
|
|
|
|
## Consequences
|
|
|
|
- **Positive:** one abstraction satisfies the full ladder; per-operator EP fallback makes the
|
|
NPU path robust; clean place to add future vendors/runtimes (Windows ML is abstracted behind
|
|
`Backend`, so a DirectML→Windows ML migration is local to this module).
|
|
- **Negative:** NPU support is hardware/driver dependent and the newest, least-tested tier —
|
|
must degrade silently to GPU/CPU and never hard-fail; requires pinning `ort`↔ONNX Runtime
|
|
binary versions on Windows builds.
|
|
|
|
## Revisit if
|
|
Windows ML supersedes DirectML for our models, or a vendor SDK gives materially better NPU
|
|
throughput than the ONNX/DirectML path.
|
|
|
|
## Update (2026-07-16): NPU path uses OpenVINO; decoder split off the CPU
|
|
|
|
As built (T3.4 + follow-up), the NPU tier runs on **ONNX Runtime + the OpenVINO EP**
|
|
(`device_type=NPU`), not DirectML — Intel's prebuilt OpenVINO ORT runtime is downloaded on
|
|
demand (DirectML remains the *GPU* fallback EP for AMD/Intel in non-Vulkan builds). The engine
|
|
splits Whisper across devices:
|
|
|
|
- **encoder** (fixed shape): OpenVINO **NPU**;
|
|
- **decoder** (autoregressive, KV-cache greedy via Optimum's merged export): OpenVINO **GPU**
|
|
(Intel iGPU, same runtime bundle) when available, else the CPU EP —
|
|
`WA_ONNX_DECODER_DEVICE=cpu|gpu` overrides.
|
|
|
|
Rationale: the dynamic-shaped decode loop is a poor NPU fit, and pre-KV-cache it re-ran the full
|
|
token prefix every step on the CPU — the dominant CPU cost while transcribing. With the merged
|
|
decoder + iGPU placement, sustained CPU during transcription is near-idle on NPU+iGPU systems.
|
|
The engine logs both EPs at load (`transcription engine: ONNX …`).
|