244 lines
14 KiB
Markdown
244 lines
14 KiB
Markdown
# Phase 3 design sketch: per-stream speaker attribution (FR-SPK)
|
||
|
||
Status: **signed off & in progress**, 2026-07-13. Deepens Phase 3 of
|
||
[`2026-07-13-diarization-speaker-accuracy.md`](./2026-07-13-diarization-speaker-accuracy.md).
|
||
The §§1–8 below are the original sketch (ADR-0005 review + alternatives); the **finalized
|
||
design** the user signed off on is at the top and supersedes the RMS-dominance heuristic.
|
||
|
||
---
|
||
|
||
## Finalized design (signed off 2026-07-13)
|
||
|
||
Simpler than the sketch's dominance heuristic, because the user records on **headphones** so the
|
||
mic never hears the far side (no bleed):
|
||
|
||
> **The microphone stream is "You" by definition. Diarization runs on the loopback (speaker)
|
||
> audio only, never the mic. Any loopback speaker is "Speaker N".**
|
||
|
||
Mechanism, all in **`audio.wav` frame time** (so You-spans and sherpa-spans share one clock):
|
||
|
||
1. **Mic-activity timeline** — captured in the loopback writer, the one place mic and loopback
|
||
exist separately in `audio.wav`'s timebase (`capture_loop`, where `bridge.pull()` is folded
|
||
in). Per 100 ms, flag whether the pulled mic chunk was speech-level (reuse `VOICE_ENERGY_FLOOR`).
|
||
Held in a shared `MicActivity` (sibling to `VoiceSample`); persisted for reprocess.
|
||
2. **At stop / reprocess:**
|
||
- Collapse active windows → **"You" spans** (dropping sub-`MIN_SPAN_MS` blips).
|
||
- **Mask** the You ranges out of `audio.wav` (zero them) and run sherpa on the remainder → far
|
||
side splits into **"Speaker N"** spans. Clustering never sees the mic, so the mic can't
|
||
pollute or merge with a far-side cluster.
|
||
- Merge span lists; `assign_by_overlap` unchanged.
|
||
3. **Voiceprint (Phase 1/2) becomes the fallback** for meetings with no timeline (imports, older
|
||
recordings). Live-tick voiceprint "You" is retired once the timeline drives labels.
|
||
|
||
Timebase note: with the capture-silence fix (FR-CAP-7) `audio.wav` is now continuous, so the
|
||
frame-indexed timeline aligns cleanly with what sherpa reads back via `read_wav_mono_16k`.
|
||
|
||
Implementation status (all landed 2026-07-13/14): **(a)** `MicActivity` struct + capture wiring —
|
||
done; **(b)** stop-path attribution + masked diarize (`phase3_attribute`) — done; **(c)** persist
|
||
`mic_activity.json` + reprocess uses it (imports fall back, no timeline) — done; **(d)** ADR-0005 +
|
||
`docs/02`/`03` amended — done. **Not yet done:** empirical user test; and the *live* 15s tick still
|
||
uses the Phase 2 voiceprint, not the timeline (a later polish so live labels match the stop result).
|
||
|
||
---
|
||
|
||
_Original sketch (ADR-0005 review, risks, alternatives) follows._
|
||
|
||
Phases 0–2 shipped: cluster explosion tamed (0.7 + 700 ms floor), reprocess re-diarizes with a
|
||
persistent `voiceprint.wav`, and live "You" reaches the UI. Those all still treat diarization as
|
||
**blind clustering of a summed mono signal**, then guess which cluster is the mic. Phase 3
|
||
removes the guess for the mic side by using information WA throws away today.
|
||
|
||
---
|
||
|
||
## 1. ADR-0005 review — what it commits us to
|
||
|
||
ADR-0005 (Accepted, 2026-06-30) decides: sherpa-onnx offline diarization (pyannote segmentation
|
||
+ ERes2Net embedding + clustering), behind the `diarization::Diarizer` trait, run as a
|
||
**post-processing pass over recorded audio**, aligned to whisper segments by **timestamp
|
||
overlap**. Speaker IDs (`S1`…) are internal/stable; names map in the DB, applied at render time,
|
||
**never rewritten onto segments**.
|
||
|
||
What Phase 3 must respect vs. what it may change:
|
||
|
||
- **Keeps (contract-level):** the `Diarizer` trait, post-pass-over-audio model, `SpeakerSpan`
|
||
→ segment alignment by overlap (`assign_by_overlap`), stable internal labels, names-in-DB.
|
||
Phase 3 produces *more* `SpeakerSpan`s from a better source; it does not rewrite segments or
|
||
move naming.
|
||
- **Bends (needs the ADR noted/updated):** ADR-0005 assumes one clustering pass over "the
|
||
recorded audio." Phase 3 introduces a **second, non-clustered source of spans** (the mic
|
||
activity timeline) and restricts clustering to the **far-side** audio. That is new enough to
|
||
warrant an ADR amendment or a short ADR-0005a, because a future reader will otherwise expect
|
||
all spans to come from sherpa.
|
||
- **ADR's "Revisit if"** (joint ASR+diarization model, or whisper.cpp diarization) is unrelated
|
||
— Phase 3 is orthogonal and does not trigger it.
|
||
|
||
Conclusion: Phase 3 fits inside the trait and the overlap-alignment contract. The only doc debt
|
||
is recording that "mic-dominant spans bypass clustering," which is a genuine deviation from
|
||
ADR-0005's single-source assumption. **Recommend: amend ADR-0005** (Consequences section) rather
|
||
than a new ADR — same decision, refined.
|
||
|
||
---
|
||
|
||
## 2. The asset we currently discard
|
||
|
||
When the mic is enabled (FR-CAP-7), `start_recording` (`commands.rs:342`) wires two independent
|
||
16 kHz-mono streams into `spawn_mixer` (`audio/mod.rs:1020`). The `Mixer` (`audio/mod.rs:979`)
|
||
holds `loopback` and `mic` **time-aligned in the same buffer** and sums them sample-for-sample
|
||
into the transcript stream — then the per-stream identity is **gone**. Diarization later reads
|
||
`audio.wav` (the summed signal) and has to reverse-engineer which cluster was the mic. That
|
||
reversal is the entire reason `voiceprint.rs` exists.
|
||
|
||
Two facts make this cheap to exploit:
|
||
|
||
1. **Per-stream RMS already exists.** `audio_level(mono, mic)` (`audio/mod.rs:522`) runs on every
|
||
WASAPI chunk for *both* directions (`is_loopback` at :564, emitted at :678). We already know,
|
||
per ~10 ms chunk, how loud each side is. We just don't persist it against time.
|
||
2. **The two streams are already aligned at the mixer**, in the **same timebase the transcript
|
||
segments derive from** (the streaming worker consumes the mixer's summed output; segment
|
||
timestamps are cumulative-samples-fed / 16). So a timeline indexed by cumulative mixer-output
|
||
samples shares the segments' clock — tighter than sherpa spans, which come from `audio.wav`.
|
||
|
||
---
|
||
|
||
## 3. Core idea
|
||
|
||
> The mic stream is a **known** speaker. Don't cluster to find it — record when it's dominant,
|
||
> call those spans "You", and run sherpa only on the **far-side** audio to split the *other*
|
||
> participants.
|
||
|
||
Pipeline at stop (mic enabled + retained):
|
||
|
||
```
|
||
mixer ─┬─► loopback samples ──► [far-side WAV] ──► sherpa cluster ──► Speaker 2..N spans
|
||
│ │
|
||
└─► mic vs loopback RMS per 100 ms ──► mic-dominant ranges ──► "You" spans
|
||
│
|
||
merge span lists ──► assign_by_overlap ──► segments
|
||
```
|
||
|
||
The mic-dominant spans need no embedding, no clustering, no voiceprint — they are attribution by
|
||
construction. Voiceprint (Phases 1–2) stays as the **fallback** for meetings without a timeline
|
||
(imports, pre-Phase-3 recordings, mic-disabled meetings).
|
||
|
||
---
|
||
|
||
## 4. The two real risks (why this needs sign-off, not just a ticket)
|
||
|
||
### 4a. Acoustic bleed — the mic is not purely "You"
|
||
|
||
Without echo cancellation (WA has none — fully local, minimal), a user on **speakers** (not
|
||
headphones) has the far side playing into the room and back into the mic. Naive "mic has signal →
|
||
You" would attribute the far side to the user whenever the far side is loud.
|
||
|
||
Mitigation (this is why the parent plan says *mic-**dominant**,* not *mic-active*): a range is
|
||
"You" only when **mic RMS meaningfully exceeds loopback RMS** in that window (e.g.
|
||
`mic_rms > k · loopback_rms` with `k ≈ 2`, plus an absolute mic-VAD floor so silence isn't
|
||
"You"). When both are comparably loud → treat as far-side/ambiguous, let sherpa/overlap decide.
|
||
This degrades gracefully: headphone users get near-perfect mic isolation; speaker users get
|
||
"You" only on clear self-speech and fall back to the old behavior during overlap. **`k` is a
|
||
calibration knob, not a constant to bury** — real rooms differ.
|
||
|
||
### 4b. Timebase — three clocks, currently reconciled by tolerance
|
||
|
||
- **Segments** are in *mixer-output* time (streaming worker over the summed stream).
|
||
- **sherpa spans** are in *`audio.wav`* time (loopback thread's byte-accurate WAV, mic bridged in
|
||
at native rate then resampled to 16 kHz on read).
|
||
- The **new mic-timeline** would be in *mixer-output* time.
|
||
|
||
Today `assign_by_overlap` already bridges segments↔sherpa-spans across the first two clocks and
|
||
tolerates the small skew. Phase 3 merges a *third* source. The mic-timeline is actually the
|
||
**closest** to segment time (shared origin), so aligning You-spans to segments is *more* reliable
|
||
than sherpa's. The residual risk is only that You-spans (mixer time) and Speaker-2..N spans
|
||
(`audio.wav` time) coexist in one merged list with a small relative skew at the boundaries. In
|
||
practice that costs at most a fraction of a `MIN_SPAN_MS` at each turn boundary — below segment
|
||
granularity. **Verification item, not a blocker.** If it proves visible, the clean fix is to run
|
||
the far-side sherpa pass over the mixer's *loopback lane* buffered to a WAV rather than over
|
||
`audio.wav`, putting everything on one clock; heavier, so deferred until measured.
|
||
|
||
---
|
||
|
||
## 5. Concrete design
|
||
|
||
**Capture (live, near-zero cost).** Add a mic-activity accumulator the mixer writes to. In
|
||
`Mixer::drain_ready` (or the `mixer_loop`), for each emitted window compute `mic_rms` and
|
||
`loopback_rms` over the aligned overlap (samples are already in hand there) and append a coarse
|
||
flag per 100 ms:
|
||
|
||
```rust
|
||
// audio/mod.rs — new, alongside Mixer
|
||
struct MicActivity { flags: Vec<bool>, /* one per 100 ms of mixer output */ }
|
||
// ponytail: Vec<bool> at 10 Hz = 10 bytes/s ≈ 36 KB/hour. No ring buffer, no config.
|
||
```
|
||
|
||
Expose it via the same `Arc` sharing pattern `VoiceSample` uses so `stop_recording` can read it.
|
||
Reuse `audio_level`'s RMS math — do **not** add a second RMS implementation.
|
||
|
||
**Persist (retained meetings).** At stop, alongside `voiceprint.wav`, write the timeline as
|
||
`mic_activity.json` (or pack into an existing artifact) so **reprocess** can reuse it — same
|
||
ADR-0009 retention/consent gate as `voiceprint.wav`, same lives-and-dies-with-`audio.wav` rule
|
||
(delete it everywhere `audio.wav` is deleted). Tiny, plaintext-or-sealed to match.
|
||
|
||
**Attribute (stop + reprocess).**
|
||
1. Collapse the flag timeline into `You` `SpeakerSpan`s (merge adjacent flags, drop runs
|
||
`< MIN_SPAN_MS`, reuse Phase 0's floor).
|
||
2. Build a **far-side-only** 16 kHz signal: zero out (or excise) mic-dominant ranges from the
|
||
loopback lane, run the existing `SherpaDiarizer::diarize` over it → `Speaker 2..N` spans
|
||
(label numbering starts at 2, mirroring `voiceprint::build_name_map`).
|
||
3. Merge the two span lists; `assign_by_overlap` unchanged.
|
||
4. Fallback chain: no timeline → today's Phase-1/2 voiceprint path; no models → raw `S1`.
|
||
|
||
**Contract/doc changes.**
|
||
- Amend **ADR-0005** Consequences: "mic-dominant spans are attributed directly from a per-stream
|
||
activity timeline and bypass clustering; sherpa runs on the far-side residual only."
|
||
- `docs/02-architecture.md` data-flow: note the mixer emits a mic-activity side-channel.
|
||
- `docs/03-data-model.md`: add `mic_activity.json` to the meeting-dir list (like `voiceprint.wav`).
|
||
- No new egress; no new default-on setting; `k` threshold lives in code with a calibration comment.
|
||
|
||
**Tests.**
|
||
- Unit: flag-run → span collapse (boundaries, sub-`MIN_SPAN_MS` drop) — pure, table-driven.
|
||
- Unit: dominance classifier (`mic`, `loop` RMS pairs → You / far / ambiguous), incl. the bleed
|
||
case (both loud → not You).
|
||
- Manual: 2-person call on **headphones** → exactly "You" + 1 speaker, near-zero errors; repeat
|
||
on **speakers** → verify bleed doesn't mislabel far-side as You.
|
||
|
||
---
|
||
|
||
## 6. Alternatives considered (and why not)
|
||
|
||
- **Do nothing / stop at Phase 2.** Legitimate. Phases 0–2 already fix the reported bugs
|
||
(83-cluster explosion, reprocess collapse, live "You"). Phase 3 is an **accuracy ceiling
|
||
raise**, not a bug fix. *If 2-person calls are the dominant case and Phase 2's voiceprint "You"
|
||
tests well in the field, Phase 3 may not clear the cost/benefit bar — measure Phase 2 first.*
|
||
This is the ponytail-honest recommendation: **verify Phase 2 empirically before building 3.**
|
||
- **"Mic active → You" (no dominance test).** Simpler, but wrong under speaker bleed (4a). The
|
||
dominance test is the minimum that survives real rooms.
|
||
- **Full source separation / AEC.** Much larger, new deps, defeats "minimal + local." The
|
||
dominance heuristic gets ~90 % of the benefit for ~5 % of the effort.
|
||
- **Better voiceprint only** (bigger sample, re-match every tick — already done in Phase 2).
|
||
Still bottlenecked on sherpa's ability to *form* a clean mic cluster in the summed signal;
|
||
Phase 3's point is to stop relying on that.
|
||
|
||
---
|
||
|
||
## 7. Open questions for sign-off
|
||
|
||
1. **Build it now, or verify Phase 2 first?** Recommendation: run the empirical 2-person
|
||
re-diarization (outstanding for Phases 0–2) and a Phase-2 live "You" check **before**
|
||
committing to Phase 3. If Phase 2 accuracy is acceptable, Phase 3 becomes optional.
|
||
2. **Dominance threshold `k` and mic-VAD floor** — accept a code-constant default with a
|
||
calibration comment (proposed), or expose a hidden setting? Recommendation: constant first.
|
||
3. **Far-side signal construction** — zero-out mic-dominant ranges in the loopback lane (simple,
|
||
preserves timeline) vs. excise-and-concat (shorter audio, shifts timestamps → needs remap).
|
||
Recommendation: **zero-out**, keeps one timebase.
|
||
4. **ADR: amend 0005 vs. new ADR-0005a?** Recommendation: amend.
|
||
5. **Scope of `mic_activity.json`** — persist for reprocess (proposed) or compute live-only and
|
||
accept that reprocess falls back to voiceprint? Persisting is cheap and keeps reprocess at
|
||
parity.
|
||
|
||
## 8. Rough effort
|
||
|
||
Backend-only until the far-side pass; no frontend change (labels already flow via Phase 2). Est.
|
||
~1–1.5 days: mixer side-channel + accumulator (½ day), span collapse + far-side pass + merge
|
||
(½ day), persistence + reprocess wiring + docs/tests (½ day). Contained, reversible, behind the
|
||
existing mic-enabled + retention gates.
|