iamdoubz/WhispAssist
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 13 Wiki Activity
Files
21f3d76ed056aa7f50792e160413600629e0513d
WhispAssist/docs/plans/2026-07-13-diarization-phase3-per-stream-attribution.md
T

14 KiB
Raw Blame History

Phase 3 design sketch: per-stream speaker attribution (FR-SPK)

Status: signed off & in progress, 2026-07-13. Deepens Phase 3 of 2026-07-13-diarization-speaker-accuracy.md. The §§1–8 below are the original sketch (ADR-0005 review + alternatives); the finalized design the user signed off on is at the top and supersedes the RMS-dominance heuristic.

Finalized design (signed off 2026-07-13)

Simpler than the sketch's dominance heuristic, because the user records on headphones so the mic never hears the far side (no bleed):

The microphone stream is "You" by definition. Diarization runs on the loopback (speaker) audio only, never the mic. Any loopback speaker is "Speaker N".

Mechanism, all in audio.wav frame time (so You-spans and sherpa-spans share one clock):

  1. Mic-activity timeline — captured in the loopback writer, the one place mic and loopback exist separately in audio.wav's timebase (capture_loop, where bridge.pull() is folded in). Per 100 ms, flag whether the pulled mic chunk was speech-level (reuse VOICE_ENERGY_FLOOR). Held in a shared MicActivity (sibling to VoiceSample); persisted for reprocess.
  2. At stop / reprocess:
    • Collapse active windows → "You" spans (dropping sub-MIN_SPAN_MS blips).
    • Mask the You ranges out of audio.wav (zero them) and run sherpa on the remainder → far side splits into "Speaker N" spans. Clustering never sees the mic, so the mic can't pollute or merge with a far-side cluster.
    • Merge span lists; assign_by_overlap unchanged.
  3. Voiceprint (Phase 1/2) becomes the fallback for meetings with no timeline (imports, older recordings). Live-tick voiceprint "You" is retired once the timeline drives labels.

Timebase note: with the capture-silence fix (FR-CAP-7) audio.wav is now continuous, so the frame-indexed timeline aligns cleanly with what sherpa reads back via read_wav_mono_16k.

Implementation status (all landed 2026-07-13/14): (a) MicActivity struct + capture wiring — done; (b) stop-path attribution + masked diarize (phase3_attribute) — done; (c) persist mic_activity.json + reprocess uses it (imports fall back, no timeline) — done; (d) ADR-0005 + docs/02/03 amended — done. Not yet done: empirical user test; and the live 15s tick still uses the Phase 2 voiceprint, not the timeline (a later polish so live labels match the stop result).

Original sketch (ADR-0005 review, risks, alternatives) follows.

Phases 0–2 shipped: cluster explosion tamed (0.7 + 700 ms floor), reprocess re-diarizes with a persistent voiceprint.wav, and live "You" reaches the UI. Those all still treat diarization as blind clustering of a summed mono signal, then guess which cluster is the mic. Phase 3 removes the guess for the mic side by using information WA throws away today.

1. ADR-0005 review — what it commits us to

ADR-0005 (Accepted, 2026-06-30) decides: sherpa-onnx offline diarization (pyannote segmentation

  • ERes2Net embedding + clustering), behind the diarization::Diarizer trait, run as a post-processing pass over recorded audio, aligned to whisper segments by timestamp overlap. Speaker IDs (S1…) are internal/stable; names map in the DB, applied at render time, never rewritten onto segments.

What Phase 3 must respect vs. what it may change:

  • Keeps (contract-level): the Diarizer trait, post-pass-over-audio model, SpeakerSpan → segment alignment by overlap (assign_by_overlap), stable internal labels, names-in-DB. Phase 3 produces more SpeakerSpans from a better source; it does not rewrite segments or move naming.
  • Bends (needs the ADR noted/updated): ADR-0005 assumes one clustering pass over "the recorded audio." Phase 3 introduces a second, non-clustered source of spans (the mic activity timeline) and restricts clustering to the far-side audio. That is new enough to warrant an ADR amendment or a short ADR-0005a, because a future reader will otherwise expect all spans to come from sherpa.
  • ADR's "Revisit if" (joint ASR+diarization model, or whisper.cpp diarization) is unrelated — Phase 3 is orthogonal and does not trigger it.

Conclusion: Phase 3 fits inside the trait and the overlap-alignment contract. The only doc debt is recording that "mic-dominant spans bypass clustering," which is a genuine deviation from ADR-0005's single-source assumption. Recommend: amend ADR-0005 (Consequences section) rather than a new ADR — same decision, refined.

2. The asset we currently discard

When the mic is enabled (FR-CAP-7), start_recording (commands.rs:342) wires two independent 16 kHz-mono streams into spawn_mixer (audio/mod.rs:1020). The Mixer (audio/mod.rs:979) holds loopback and mic time-aligned in the same buffer and sums them sample-for-sample into the transcript stream — then the per-stream identity is gone. Diarization later reads audio.wav (the summed signal) and has to reverse-engineer which cluster was the mic. That reversal is the entire reason voiceprint.rs exists.

Two facts make this cheap to exploit:

  1. Per-stream RMS already exists. audio_level(mono, mic) (audio/mod.rs:522) runs on every WASAPI chunk for both directions (is_loopback at :564, emitted at :678). We already know, per ~10 ms chunk, how loud each side is. We just don't persist it against time.
  2. The two streams are already aligned at the mixer, in the same timebase the transcript segments derive from (the streaming worker consumes the mixer's summed output; segment timestamps are cumulative-samples-fed / 16). So a timeline indexed by cumulative mixer-output samples shares the segments' clock — tighter than sherpa spans, which come from audio.wav.

3. Core idea

The mic stream is a known speaker. Don't cluster to find it — record when it's dominant, call those spans "You", and run sherpa only on the far-side audio to split the other participants.

Pipeline at stop (mic enabled + retained):

mixer  ─┬─► loopback samples ──► [far-side WAV] ──► sherpa cluster ──► Speaker 2..N spans
        │                                                                      │
        └─► mic vs loopback RMS per 100 ms ──► mic-dominant ranges ──► "You" spans
                                                                               │
                              merge span lists ──► assign_by_overlap ──► segments

The mic-dominant spans need no embedding, no clustering, no voiceprint — they are attribution by construction. Voiceprint (Phases 1–2) stays as the fallback for meetings without a timeline (imports, pre-Phase-3 recordings, mic-disabled meetings).

4. The two real risks (why this needs sign-off, not just a ticket)

4a. Acoustic bleed — the mic is not purely "You"

Without echo cancellation (WA has none — fully local, minimal), a user on speakers (not headphones) has the far side playing into the room and back into the mic. Naive "mic has signal → You" would attribute the far side to the user whenever the far side is loud.

Mitigation (this is why the parent plan says mic-dominant, not mic-active): a range is "You" only when mic RMS meaningfully exceeds loopback RMS in that window (e.g. mic_rms > k · loopback_rms with k ≈ 2, plus an absolute mic-VAD floor so silence isn't "You"). When both are comparably loud → treat as far-side/ambiguous, let sherpa/overlap decide. This degrades gracefully: headphone users get near-perfect mic isolation; speaker users get "You" only on clear self-speech and fall back to the old behavior during overlap. k is a calibration knob, not a constant to bury — real rooms differ.

4b. Timebase — three clocks, currently reconciled by tolerance

  • Segments are in mixer-output time (streaming worker over the summed stream).
  • sherpa spans are in audio.wav time (loopback thread's byte-accurate WAV, mic bridged in at native rate then resampled to 16 kHz on read).
  • The new mic-timeline would be in mixer-output time.

Today assign_by_overlap already bridges segments↔sherpa-spans across the first two clocks and tolerates the small skew. Phase 3 merges a third source. The mic-timeline is actually the closest to segment time (shared origin), so aligning You-spans to segments is more reliable than sherpa's. The residual risk is only that You-spans (mixer time) and Speaker-2..N spans (audio.wav time) coexist in one merged list with a small relative skew at the boundaries. In practice that costs at most a fraction of a MIN_SPAN_MS at each turn boundary — below segment granularity. Verification item, not a blocker. If it proves visible, the clean fix is to run the far-side sherpa pass over the mixer's loopback lane buffered to a WAV rather than over audio.wav, putting everything on one clock; heavier, so deferred until measured.

5. Concrete design

Capture (live, near-zero cost). Add a mic-activity accumulator the mixer writes to. In Mixer::drain_ready (or the mixer_loop), for each emitted window compute mic_rms and loopback_rms over the aligned overlap (samples are already in hand there) and append a coarse flag per 100 ms:

// audio/mod.rs — new, alongside Mixer
struct MicActivity { flags: Vec<bool>, /* one per 100 ms of mixer output */ }
// ponytail: Vec<bool> at 10 Hz = 10 bytes/s ≈ 36 KB/hour. No ring buffer, no config.

Expose it via the same Arc sharing pattern VoiceSample uses so stop_recording can read it. Reuse audio_level's RMS math — do not add a second RMS implementation.

Persist (retained meetings). At stop, alongside voiceprint.wav, write the timeline as mic_activity.json (or pack into an existing artifact) so reprocess can reuse it — same ADR-0009 retention/consent gate as voiceprint.wav, same lives-and-dies-with-audio.wav rule (delete it everywhere audio.wav is deleted). Tiny, plaintext-or-sealed to match.

Attribute (stop + reprocess).

  1. Collapse the flag timeline into You SpeakerSpans (merge adjacent flags, drop runs < MIN_SPAN_MS, reuse Phase 0's floor).
  2. Build a far-side-only 16 kHz signal: zero out (or excise) mic-dominant ranges from the loopback lane, run the existing SherpaDiarizer::diarize over it → Speaker 2..N spans (label numbering starts at 2, mirroring voiceprint::build_name_map).
  3. Merge the two span lists; assign_by_overlap unchanged.
  4. Fallback chain: no timeline → today's Phase-1/2 voiceprint path; no models → raw S1.

Contract/doc changes.

  • Amend ADR-0005 Consequences: "mic-dominant spans are attributed directly from a per-stream activity timeline and bypass clustering; sherpa runs on the far-side residual only."
  • docs/02-architecture.md data-flow: note the mixer emits a mic-activity side-channel.
  • docs/03-data-model.md: add mic_activity.json to the meeting-dir list (like voiceprint.wav).
  • No new egress; no new default-on setting; k threshold lives in code with a calibration comment.

Tests.

  • Unit: flag-run → span collapse (boundaries, sub-MIN_SPAN_MS drop) — pure, table-driven.
  • Unit: dominance classifier (mic, loop RMS pairs → You / far / ambiguous), incl. the bleed case (both loud → not You).
  • Manual: 2-person call on headphones → exactly "You" + 1 speaker, near-zero errors; repeat on speakers → verify bleed doesn't mislabel far-side as You.

6. Alternatives considered (and why not)

  • Do nothing / stop at Phase 2. Legitimate. Phases 0–2 already fix the reported bugs (83-cluster explosion, reprocess collapse, live "You"). Phase 3 is an accuracy ceiling raise, not a bug fix. If 2-person calls are the dominant case and Phase 2's voiceprint "You" tests well in the field, Phase 3 may not clear the cost/benefit bar — measure Phase 2 first. This is the ponytail-honest recommendation: verify Phase 2 empirically before building 3.
  • "Mic active → You" (no dominance test). Simpler, but wrong under speaker bleed (4a). The dominance test is the minimum that survives real rooms.
  • Full source separation / AEC. Much larger, new deps, defeats "minimal + local." The dominance heuristic gets ~90 % of the benefit for ~5 % of the effort.
  • Better voiceprint only (bigger sample, re-match every tick — already done in Phase 2). Still bottlenecked on sherpa's ability to form a clean mic cluster in the summed signal; Phase 3's point is to stop relying on that.

7. Open questions for sign-off

  1. Build it now, or verify Phase 2 first? Recommendation: run the empirical 2-person re-diarization (outstanding for Phases 0–2) and a Phase-2 live "You" check before committing to Phase 3. If Phase 2 accuracy is acceptable, Phase 3 becomes optional.
  2. Dominance threshold k and mic-VAD floor — accept a code-constant default with a calibration comment (proposed), or expose a hidden setting? Recommendation: constant first.
  3. Far-side signal construction — zero-out mic-dominant ranges in the loopback lane (simple, preserves timeline) vs. excise-and-concat (shorter audio, shifts timestamps → needs remap). Recommendation: zero-out, keeps one timebase.
  4. ADR: amend 0005 vs. new ADR-0005a? Recommendation: amend.
  5. Scope of mic_activity.json — persist for reprocess (proposed) or compute live-only and accept that reprocess falls back to voiceprint? Persisting is cheap and keeps reprocess at parity.

8. Rough effort

Backend-only until the far-side pass; no frontend change (labels already flow via Phase 2). Est. ~1–1.5 days: mixer side-channel + accumulator (½ day), span collapse + far-side pass + merge (½ day), persistence + reprocess wiring + docs/tests (½ day). Contained, reversible, behind the existing mic-enabled + retention gates.

Reference in New Issue View Git Blame Copy Permalink