docs(plan): Phase 3 per-stream attribution design sketch + ADR-0005 review (FR-SPK)
This commit is contained in:
iamdoubz
@@ -0,0 +1,205 @@
|
||||
# Phase 3 design sketch: per-stream speaker attribution (FR-SPK)
|
||||
|
||||
Status: **draft for sign-off**, 2026-07-13. Deepens Phase 3 of
|
||||
[`2026-07-13-diarization-speaker-accuracy.md`](./2026-07-13-diarization-speaker-accuracy.md).
|
||||
Requires ADR-0005 review (done below) and a decision before any code lands — this changes
|
||||
the diarization data flow, so it is gated per the parent plan.
|
||||
|
||||
Phases 0–2 shipped: cluster explosion tamed (0.7 + 700 ms floor), reprocess re-diarizes with a
|
||||
persistent `voiceprint.wav`, and live "You" reaches the UI. Those all still treat diarization as
|
||||
**blind clustering of a summed mono signal**, then guess which cluster is the mic. Phase 3
|
||||
removes the guess for the mic side by using information WA throws away today.
|
||||
|
||||
---
|
||||
|
||||
## 1. ADR-0005 review — what it commits us to
|
||||
|
||||
ADR-0005 (Accepted, 2026-06-30) decides: sherpa-onnx offline diarization (pyannote segmentation
|
||||
+ ERes2Net embedding + clustering), behind the `diarization::Diarizer` trait, run as a
|
||||
**post-processing pass over recorded audio**, aligned to whisper segments by **timestamp
|
||||
overlap**. Speaker IDs (`S1`…) are internal/stable; names map in the DB, applied at render time,
|
||||
**never rewritten onto segments**.
|
||||
|
||||
What Phase 3 must respect vs. what it may change:
|
||||
|
||||
- **Keeps (contract-level):** the `Diarizer` trait, post-pass-over-audio model, `SpeakerSpan`
|
||||
→ segment alignment by overlap (`assign_by_overlap`), stable internal labels, names-in-DB.
|
||||
Phase 3 produces *more* `SpeakerSpan`s from a better source; it does not rewrite segments or
|
||||
move naming.
|
||||
- **Bends (needs the ADR noted/updated):** ADR-0005 assumes one clustering pass over "the
|
||||
recorded audio." Phase 3 introduces a **second, non-clustered source of spans** (the mic
|
||||
activity timeline) and restricts clustering to the **far-side** audio. That is new enough to
|
||||
warrant an ADR amendment or a short ADR-0005a, because a future reader will otherwise expect
|
||||
all spans to come from sherpa.
|
||||
- **ADR's "Revisit if"** (joint ASR+diarization model, or whisper.cpp diarization) is unrelated
|
||||
— Phase 3 is orthogonal and does not trigger it.
|
||||
|
||||
Conclusion: Phase 3 fits inside the trait and the overlap-alignment contract. The only doc debt
|
||||
is recording that "mic-dominant spans bypass clustering," which is a genuine deviation from
|
||||
ADR-0005's single-source assumption. **Recommend: amend ADR-0005** (Consequences section) rather
|
||||
than a new ADR — same decision, refined.
|
||||
|
||||
---
|
||||
|
||||
## 2. The asset we currently discard
|
||||
|
||||
When the mic is enabled (FR-CAP-7), `start_recording` (`commands.rs:342`) wires two independent
|
||||
16 kHz-mono streams into `spawn_mixer` (`audio/mod.rs:1020`). The `Mixer` (`audio/mod.rs:979`)
|
||||
holds `loopback` and `mic` **time-aligned in the same buffer** and sums them sample-for-sample
|
||||
into the transcript stream — then the per-stream identity is **gone**. Diarization later reads
|
||||
`audio.wav` (the summed signal) and has to reverse-engineer which cluster was the mic. That
|
||||
reversal is the entire reason `voiceprint.rs` exists.
|
||||
|
||||
Two facts make this cheap to exploit:
|
||||
|
||||
1. **Per-stream RMS already exists.** `audio_level(mono, mic)` (`audio/mod.rs:522`) runs on every
|
||||
WASAPI chunk for *both* directions (`is_loopback` at :564, emitted at :678). We already know,
|
||||
per ~10 ms chunk, how loud each side is. We just don't persist it against time.
|
||||
2. **The two streams are already aligned at the mixer**, in the **same timebase the transcript
|
||||
segments derive from** (the streaming worker consumes the mixer's summed output; segment
|
||||
timestamps are cumulative-samples-fed / 16). So a timeline indexed by cumulative mixer-output
|
||||
samples shares the segments' clock — tighter than sherpa spans, which come from `audio.wav`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Core idea
|
||||
|
||||
> The mic stream is a **known** speaker. Don't cluster to find it — record when it's dominant,
|
||||
> call those spans "You", and run sherpa only on the **far-side** audio to split the *other*
|
||||
> participants.
|
||||
|
||||
Pipeline at stop (mic enabled + retained):
|
||||
|
||||
```
|
||||
mixer ─┬─► loopback samples ──► [far-side WAV] ──► sherpa cluster ──► Speaker 2..N spans
|
||||
│ │
|
||||
└─► mic vs loopback RMS per 100 ms ──► mic-dominant ranges ──► "You" spans
|
||||
│
|
||||
merge span lists ──► assign_by_overlap ──► segments
|
||||
```
|
||||
|
||||
The mic-dominant spans need no embedding, no clustering, no voiceprint — they are attribution by
|
||||
construction. Voiceprint (Phases 1–2) stays as the **fallback** for meetings without a timeline
|
||||
(imports, pre-Phase-3 recordings, mic-disabled meetings).
|
||||
|
||||
---
|
||||
|
||||
## 4. The two real risks (why this needs sign-off, not just a ticket)
|
||||
|
||||
### 4a. Acoustic bleed — the mic is not purely "You"
|
||||
|
||||
Without echo cancellation (WA has none — fully local, minimal), a user on **speakers** (not
|
||||
headphones) has the far side playing into the room and back into the mic. Naive "mic has signal →
|
||||
You" would attribute the far side to the user whenever the far side is loud.
|
||||
|
||||
Mitigation (this is why the parent plan says *mic-**dominant**,* not *mic-active*): a range is
|
||||
"You" only when **mic RMS meaningfully exceeds loopback RMS** in that window (e.g.
|
||||
`mic_rms > k · loopback_rms` with `k ≈ 2`, plus an absolute mic-VAD floor so silence isn't
|
||||
"You"). When both are comparably loud → treat as far-side/ambiguous, let sherpa/overlap decide.
|
||||
This degrades gracefully: headphone users get near-perfect mic isolation; speaker users get
|
||||
"You" only on clear self-speech and fall back to the old behavior during overlap. **`k` is a
|
||||
calibration knob, not a constant to bury** — real rooms differ.
|
||||
|
||||
### 4b. Timebase — three clocks, currently reconciled by tolerance
|
||||
|
||||
- **Segments** are in *mixer-output* time (streaming worker over the summed stream).
|
||||
- **sherpa spans** are in *`audio.wav`* time (loopback thread's byte-accurate WAV, mic bridged in
|
||||
at native rate then resampled to 16 kHz on read).
|
||||
- The **new mic-timeline** would be in *mixer-output* time.
|
||||
|
||||
Today `assign_by_overlap` already bridges segments↔sherpa-spans across the first two clocks and
|
||||
tolerates the small skew. Phase 3 merges a *third* source. The mic-timeline is actually the
|
||||
**closest** to segment time (shared origin), so aligning You-spans to segments is *more* reliable
|
||||
than sherpa's. The residual risk is only that You-spans (mixer time) and Speaker-2..N spans
|
||||
(`audio.wav` time) coexist in one merged list with a small relative skew at the boundaries. In
|
||||
practice that costs at most a fraction of a `MIN_SPAN_MS` at each turn boundary — below segment
|
||||
granularity. **Verification item, not a blocker.** If it proves visible, the clean fix is to run
|
||||
the far-side sherpa pass over the mixer's *loopback lane* buffered to a WAV rather than over
|
||||
`audio.wav`, putting everything on one clock; heavier, so deferred until measured.
|
||||
|
||||
---
|
||||
|
||||
## 5. Concrete design
|
||||
|
||||
**Capture (live, near-zero cost).** Add a mic-activity accumulator the mixer writes to. In
|
||||
`Mixer::drain_ready` (or the `mixer_loop`), for each emitted window compute `mic_rms` and
|
||||
`loopback_rms` over the aligned overlap (samples are already in hand there) and append a coarse
|
||||
flag per 100 ms:
|
||||
|
||||
```rust
|
||||
// audio/mod.rs — new, alongside Mixer
|
||||
struct MicActivity { flags: Vec<bool>, /* one per 100 ms of mixer output */ }
|
||||
// ponytail: Vec<bool> at 10 Hz = 10 bytes/s ≈ 36 KB/hour. No ring buffer, no config.
|
||||
```
|
||||
|
||||
Expose it via the same `Arc` sharing pattern `VoiceSample` uses so `stop_recording` can read it.
|
||||
Reuse `audio_level`'s RMS math — do **not** add a second RMS implementation.
|
||||
|
||||
**Persist (retained meetings).** At stop, alongside `voiceprint.wav`, write the timeline as
|
||||
`mic_activity.json` (or pack into an existing artifact) so **reprocess** can reuse it — same
|
||||
ADR-0009 retention/consent gate as `voiceprint.wav`, same lives-and-dies-with-`audio.wav` rule
|
||||
(delete it everywhere `audio.wav` is deleted). Tiny, plaintext-or-sealed to match.
|
||||
|
||||
**Attribute (stop + reprocess).**
|
||||
1. Collapse the flag timeline into `You` `SpeakerSpan`s (merge adjacent flags, drop runs
|
||||
`< MIN_SPAN_MS`, reuse Phase 0's floor).
|
||||
2. Build a **far-side-only** 16 kHz signal: zero out (or excise) mic-dominant ranges from the
|
||||
loopback lane, run the existing `SherpaDiarizer::diarize` over it → `Speaker 2..N` spans
|
||||
(label numbering starts at 2, mirroring `voiceprint::build_name_map`).
|
||||
3. Merge the two span lists; `assign_by_overlap` unchanged.
|
||||
4. Fallback chain: no timeline → today's Phase-1/2 voiceprint path; no models → raw `S1`.
|
||||
|
||||
**Contract/doc changes.**
|
||||
- Amend **ADR-0005** Consequences: "mic-dominant spans are attributed directly from a per-stream
|
||||
activity timeline and bypass clustering; sherpa runs on the far-side residual only."
|
||||
- `docs/02-architecture.md` data-flow: note the mixer emits a mic-activity side-channel.
|
||||
- `docs/03-data-model.md`: add `mic_activity.json` to the meeting-dir list (like `voiceprint.wav`).
|
||||
- No new egress; no new default-on setting; `k` threshold lives in code with a calibration comment.
|
||||
|
||||
**Tests.**
|
||||
- Unit: flag-run → span collapse (boundaries, sub-`MIN_SPAN_MS` drop) — pure, table-driven.
|
||||
- Unit: dominance classifier (`mic`, `loop` RMS pairs → You / far / ambiguous), incl. the bleed
|
||||
case (both loud → not You).
|
||||
- Manual: 2-person call on **headphones** → exactly "You" + 1 speaker, near-zero errors; repeat
|
||||
on **speakers** → verify bleed doesn't mislabel far-side as You.
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternatives considered (and why not)
|
||||
|
||||
- **Do nothing / stop at Phase 2.** Legitimate. Phases 0–2 already fix the reported bugs
|
||||
(83-cluster explosion, reprocess collapse, live "You"). Phase 3 is an **accuracy ceiling
|
||||
raise**, not a bug fix. *If 2-person calls are the dominant case and Phase 2's voiceprint "You"
|
||||
tests well in the field, Phase 3 may not clear the cost/benefit bar — measure Phase 2 first.*
|
||||
This is the ponytail-honest recommendation: **verify Phase 2 empirically before building 3.**
|
||||
- **"Mic active → You" (no dominance test).** Simpler, but wrong under speaker bleed (4a). The
|
||||
dominance test is the minimum that survives real rooms.
|
||||
- **Full source separation / AEC.** Much larger, new deps, defeats "minimal + local." The
|
||||
dominance heuristic gets ~90 % of the benefit for ~5 % of the effort.
|
||||
- **Better voiceprint only** (bigger sample, re-match every tick — already done in Phase 2).
|
||||
Still bottlenecked on sherpa's ability to *form* a clean mic cluster in the summed signal;
|
||||
Phase 3's point is to stop relying on that.
|
||||
|
||||
---
|
||||
|
||||
## 7. Open questions for sign-off
|
||||
|
||||
1. **Build it now, or verify Phase 2 first?** Recommendation: run the empirical 2-person
|
||||
re-diarization (outstanding for Phases 0–2) and a Phase-2 live "You" check **before**
|
||||
committing to Phase 3. If Phase 2 accuracy is acceptable, Phase 3 becomes optional.
|
||||
2. **Dominance threshold `k` and mic-VAD floor** — accept a code-constant default with a
|
||||
calibration comment (proposed), or expose a hidden setting? Recommendation: constant first.
|
||||
3. **Far-side signal construction** — zero-out mic-dominant ranges in the loopback lane (simple,
|
||||
preserves timeline) vs. excise-and-concat (shorter audio, shifts timestamps → needs remap).
|
||||
Recommendation: **zero-out**, keeps one timebase.
|
||||
4. **ADR: amend 0005 vs. new ADR-0005a?** Recommendation: amend.
|
||||
5. **Scope of `mic_activity.json`** — persist for reprocess (proposed) or compute live-only and
|
||||
accept that reprocess falls back to voiceprint? Persisting is cheap and keeps reprocess at
|
||||
parity.
|
||||
|
||||
## 8. Rough effort
|
||||
|
||||
Backend-only until the far-side pass; no frontend change (labels already flow via Phase 2). Est.
|
||||
~1–1.5 days: mixer side-channel + accumulator (½ day), span collapse + far-side pass + merge
|
||||
(½ day), persistence + reprocess wiring + docs/tests (½ day). Contained, reversible, behind the
|
||||
existing mic-enabled + retention gates.
|
||||
Reference in New Issue
Block a user