86 lines
5.0 KiB
Markdown
86 lines
5.0 KiB
Markdown
# ADR-0005 — Speaker diarization: sherpa-onnx
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-06-30
|
|
- **Context source:** Design doc §"Speaker Diarization and Naming"
|
|
|
|
## Context
|
|
|
|
WA must label transcript segments by speaker (Speaker 1/2…), allow naming during and after a
|
|
meeting, support merging over-split speakers, and feed calendar participant names into the
|
|
naming UI — all offline.
|
|
|
|
## Decision
|
|
|
|
Use **`sherpa-onnx`** offline speaker diarization: a pyannote **segmentation** model + a
|
|
speaker-**embedding** extractor (e.g. 3D-Speaker ERes2Net) + **clustering**, all ONNX and fully
|
|
local. Wrap it behind a `diarization::Diarizer` trait. Diarization runs as **post-processing**
|
|
over the recorded audio (audio is the source of truth, ADR-0006), producing speaker-ID-tagged
|
|
spans that are aligned to whisper segments by timestamp overlap.
|
|
|
|
During live recording, show provisional speaker turns from segmentation (cheap), then refine
|
|
labels in the post-meeting pass; this keeps latency low while improving final accuracy.
|
|
|
|
## Consequences
|
|
|
|
- **Positive:** purpose-built, offline, ONNX (shares the runtime with the NPU transcription path);
|
|
supports the segmentation+embedding+clustering pipeline the design assumes; models are
|
|
downloadable and swappable.
|
|
- **Negative:** separate models to download/manage (model-management UI, FR-MODEL-1); C/C++ FFI
|
|
to integrate; clustering may over/under-split → the **merge** workflow (FR-SPK-3) is required,
|
|
not optional.
|
|
- Speaker IDs (`S1`, `S2`, …) are internal and stable per meeting; name mappings live in the DB
|
|
and are applied at render/export time, never destructively rewritten onto segments.
|
|
|
|
### Phase 3 refinement — per-stream "You" attribution (FR-SPK, 2026-07-14)
|
|
|
|
The single-pass-over-the-whole-recording model above blind-clusters a **summed mono** signal
|
|
(mic + loopback), then guesses which cluster is the user via a voiceprint match. In practice that
|
|
clustering is unreliable in both directions on the summed signal — it over-split a 2-speaker call
|
|
into 83 clusters at one threshold and merged two clearly distinct voices into one at another — and
|
|
the voiceprint can only *label* a cluster, never *create* the separation.
|
|
|
|
So when the **microphone is enabled**, WA no longer relies on clustering to find the user:
|
|
|
|
- A live **mic-activity timeline** (`audio::MicActivity`) records, per 100 ms of `audio.wav` frame
|
|
time, when the mic was speech-level — captured in the loopback writer, the one place the mic and
|
|
loopback exist separately in the recording's own timebase. Those ranges become **"You"** spans
|
|
directly (no clustering, no embedding).
|
|
- sherpa clustering is then run over the recording with the "You" ranges **masked out** (zeroed),
|
|
so it only ever splits the **far side** into `Speaker N`. The mic can never pollute or merge into
|
|
a far-side cluster.
|
|
- The timeline is persisted as `mic_activity.json` (retained meetings) so `reprocess_transcript`
|
|
re-attributes identically without the live capture.
|
|
|
|
The original blind-clustering + voiceprint pass **remains the fallback** for mic-off recordings and
|
|
imports (no timeline). Naming stays uniform (`build_name_map`: "You", then `Speaker 2…`). Segment
|
|
IDs and the names-in-DB rule are unchanged; only the *source* of the spans changes.
|
|
|
|
### Phase 3.5 refinement — dual-channel capture (supersedes Phase 3's timeline, 2026-07-14)
|
|
|
|
The Phase 3 timeline above reconstructed the mic/far-side split *after the fact* from a summed-mono
|
|
`audio.wav` plus a `mic_activity.json` sidecar, masking the mic ranges before clustering. That was
|
|
reliable at stop but **fragile on reprocess** (re-aligning a sidecar against a mono mix — it
|
|
collapsed to a single speaker) and carried a parallel persistence path.
|
|
|
|
**Decision:** when the mic is enabled, record `audio.wav` as **stereo with the streams separated —
|
|
left = microphone ("You"), right = system/loopback ("Speaker")** instead of summing them. The
|
|
separation is then intrinsic to the recording:
|
|
|
|
- Diarization runs on the **right channel only** → `Speaker N`; "You" comes from **left-channel**
|
|
voice activity (`vad_spans`). The mic is never clustered, by construction.
|
|
- Reprocess recomputes both from the file — no sidecar, no masking; stop and reprocess agree.
|
|
- Transcription still downmixes (L+R) to the same summed mono; playback folds to mono and bundle
|
|
export folds to dual-mono so shared/played audio is normal.
|
|
- A `meetings.audio_layout` flag (`summed` | `split`, carried in the bundle manifest) distinguishes
|
|
recordings; existing `summed` recordings keep the blind-clustering + voiceprint path. On a stereo
|
|
render endpoint this is **size-neutral** (the file was already 2ch with the mic summed into both).
|
|
|
|
This **retires** `MicActivity` / `mic_activity.json` / masked diarization from Phase 3 (see the
|
|
retirement note at `attribute_split` in `commands.rs`); the capture-silence handling,
|
|
`diarize_samples`, and `build_name_map` naming are retained. Segment IDs and names-in-DB unchanged.
|
|
|
|
## Revisit if
|
|
A single model gives joint ASR + diarization with better accuracy, or whisper.cpp gains
|
|
production diarization.
|