From e449c194efe111bf40f02de5b0a0e76e6f40e775 Mon Sep 17 00:00:00 2001 From: iamdoubz <> Date: Tue, 14 Jul 2026 11:28:16 -0500 Subject: [PATCH] docs(plan): draft dual-channel (mic-L/loopback-R) capture plan + ADR-0005 amendment for review (FR-SPK) --- .../2026-07-14-stereo-dual-channel-capture.md | 148 ++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/plans/2026-07-14-stereo-dual-channel-capture.md diff --git a/docs/plans/2026-07-14-stereo-dual-channel-capture.md b/docs/plans/2026-07-14-stereo-dual-channel-capture.md new file mode 100644 index 0000000..f6355db --- /dev/null +++ b/docs/plans/2026-07-14-stereo-dual-channel-capture.md @@ -0,0 +1,148 @@ +# Plan: dual-channel capture (mic = left, loopback = right) — FR-SPK / FR-CAP + +Status: **draft for review**, 2026-07-14. Supersedes the summed-mono + masked-diarize approach +in [`2026-07-13-diarization-phase3-per-stream-attribution.md`](./2026-07-13-diarization-phase3-per-stream-attribution.md). +The proposed **ADR-0005 amendment** is at the bottom — apply it on approval. + +## Motivation + +Every diarization problem this month traces to one root: **`audio.wav` is a summed mono mix** of +mic + loopback, so the two sources are irrecoverable after the fact. Phase 3 works around that by +(a) recording a separate `mic_activity.json` "You" timeline and (b) diarizing the mix with the You +ranges masked out. It works at stop but is fragile on **reprocess** — the persisted timeline has to +re-align against freshly-transcribed segments over a mono mix, and it currently collapses to a +single "Speaker 2". The timelines are being written correctly (verified: real `you_spans`), so the +failure is the masked-mono re-alignment itself, not the capture. + +**Fix the root, not the instance:** keep both sources *physically separate in the recording* by +recording **stereo — left channel = microphone ("You"), right channel = system/loopback +("Speaker")**. The separation then lives in the file forever; reprocess and diarization always have +both streams; no sidecar timeline, no masking, no summed-mono clustering. + +## Key facts that make this cheap + +- `audio.wav` is **already 2-channel** on a normal (stereo) render endpoint — today we *sum the mic + into both channels*. Writing mic→L and loopback(downmixed)→R is the **same channel count, same + bytes** — no size increase in the common case. +- Transcription already downmixes to mono: `read_wav_mono_16k` averaging L+R of a split file yields + `(mic + loopback)/2` = the same summed signal it gets today, so **transcription is unchanged**. +- The mic↔loopback alignment machinery (`MicBridge`, the drain fix, the capture-silence fix) carries + over unchanged — we just route the aligned samples to two channels instead of summing them. + +## Design + +### 1. Capture write path (`audio`) +- Force the recording spec to **2ch, 16-bit, native rate** when the mic is enabled ("split" + layout). Loopback-only (mic off) keeps today's behavior. +- New writer (replacing `write_wav_bytes`'s summing for split recordings): per frame, + **L = mic sample** (0 when the mic underran), **R = downmix(loopback channels)**. +- Idle path (loopback silent): **L = drained mic, R = 0** — same silence-synthesis + device-paced + drain we just fixed, now writing to L instead of summing. + +### 2. Layout versioning (distinguish old "summed" from new "split") +Content alone can't tell a summed-stereo file from a split-stereo one, so mark it explicitly: +- Add `audio_layout TEXT` to the `meetings` row: `'summed'` (default / NULL for all existing rows + and imports) or `'split'` (new mic-enabled recordings). +- Include it in the **`MeetingBundle` manifest** so exported recordings stay interpretable on + another machine. +- Playback and diarization branch on it; old recordings keep every current code path untouched. + +### 3. Read helpers (`audio`) +- `read_wav_mono_16k` — **unchanged** (downmix all channels → mono 16k). Used for transcription of + both layouts; on a split file it returns the summed signal, exactly as wanted. +- New `read_wav_channel_16k(path, channel)` — extract one channel → mono 16k. Diarization reads the + **right** channel (far side); "You" detection reads the **left** (mic). + +### 4. Diarization / attribution (`commands`) — *simpler* than Phase 3 +For a **split** recording, one path replaces the timeline + masking: +``` +far_spans = diarizer.diarize_samples( read_wav_channel_16k(wav, RIGHT) ) // clean far side +you_spans = vad_spans( read_wav_channel_16k(wav, LEFT) ) // energy-windowed mic +merge + assign_by_overlap + build_name_map("You") // You + Speaker N +``` +- The mic is **never** in the clustering input — structurally, not procedurally. +- `vad_spans` is the `MicActivity` windowing logic (energy floor + 100 ms windows) applied to the L + channel from the file — so **stop and reprocess share one path** and produce identical results. +- **This deletes `MicActivity`, `mic_activity.json` (persist + read + delete), and `mask_ranges`.** + Net: less code than the current Phase 3. +- Mic-off (`'summed'`) recordings keep the whole-signal pass + voiceprint fallback. + +### 5. Playback (`commands::serve_recording`) +- A split file plays mic-in-left / computer-in-right — wrong for a listener. On playback, **downmix + L+R → mono** (dual-mono) so both sources come out both ears. +- `serve_recording` already reads+decrypts the whole file into memory; add a downmix-to-mono step + for `'split'` layout and serve that mono WAV (range/seek computed over the transcoded bytes). + `'summed'` files stream as today. + +### 6. Migration & portability +- **Existing recordings** (`'summed'`): every current path unchanged. Their reprocess falls back to + the voiceprint match (acceptable — they predate split). We may retire the `MicActivity`/mask code + once split is default; the handful of recent test recordings lose Phase 3 on reprocess, which is + fine. +- **Bundle export**: export **downmixes the canonical split `audio.wav` to mono** so a shared file + plays normally in any player; the working file stays split for WA's own re-processing. + +## What this removes / simplifies + +| Removed | Replaced by | +|---|---| +| `MicActivity` struct + capture wiring | L-channel VAD read from the file | +| `mic_activity.json` (write/read/delete) | nothing — recompute from L channel | +| `mask_ranges` + masked diarize | diarize the R channel directly | +| stop vs reprocess divergence | one shared split-attribution path | + +The capture-silence fix, the device-paced `drain`, `diarize_samples`, and the `build_name_map` +naming all **stay**. + +## Costs / risks (none fatal) +- **New code:** split writer, `audio_layout` column + migration + manifest field, channel read + helper, playback downmix, `vad_spans`, and the split-vs-summed branch. Bigger than a patch, but + it *retires* comparable Phase 3 code. +- **Loopback stereo image lost** (R = loopback downmixed to mono) — irrelevant for speech. +- **Mono render endpoint** (rare): 1ch→2ch doubles that recording's size. +- **Raw file plays split** in a third-party player until exported (export downmixes) — minor. +- **Clock drift** between mic/loopback now shows as tiny L/R desync instead of dropped summed + samples — harmless (diarization tolerates it; playback sums it away). + +## Open questions for sign-off +1. Layout marker: **DB column** (proposed) vs. a self-describing WAV chunk? DB column is simpler and + travels via the bundle manifest; recommend it. +2. Retire `MicActivity`/`mask_ranges` outright, or keep them for existing `'summed'` recordings' + reprocess? Recommend **retire** (summed → voiceprint fallback) to avoid two live systems. +3. Export downmix to mono — agree that's the right default for shared/synced copies? + +## Implementation order +1. `audio_layout` column + `MeetingBundle` field + read on `get_meeting` (no behavior yet). +2. Split writer + force 2ch when mic on; set `audio_layout='split'`. (`audio` + `start_recording`.) +3. `read_wav_channel_16k` + `vad_spans`; split-attribution path in stop + reprocess. +4. Playback downmix for split; bundle-export downmix. +5. Delete `MicActivity` / `mic_activity.json` / `mask_ranges`; update tests. +6. Apply the ADR-0005 amendment + update `docs/02`/`03`. + +--- + +## Proposed ADR-0005 amendment (apply on approval) + +> ### Phase 3.5 refinement — dual-channel capture (FR-SPK/FR-CAP, 2026-07-14) +> +> The Phase 3 per-stream attribution (above) reconstructs the mic/far-side split *after the fact* +> from a summed-mono `audio.wav` plus a `mic_activity.json` timeline, then masks the mic ranges +> before clustering. This is reliable at stop but fragile on reprocess (re-aligning a sidecar +> timeline against a mono mix), and it carries a parallel persistence path. +> +> **Decision:** when the microphone is enabled, record `audio.wav` as **stereo with the streams +> separated — left = microphone ("You"), right = system/loopback ("Speaker")** — instead of summing +> them. The separation is then intrinsic to the recording: +> - Diarization runs on the **right channel only** → `Speaker N`; "You" comes from **left-channel** +> voice activity. The mic is never clustered, by construction. +> - Reprocess recomputes both from the file — no sidecar timeline, no masking. +> - Transcription still downmixes (L+R) to the same summed mono it uses today; playback and bundle +> export downmix to mono so shared/played audio is normal. +> - A `meetings.audio_layout` flag (`summed` | `split`, in the bundle manifest for portability) +> distinguishes recordings; existing `summed` recordings keep the original blind-clustering + +> voiceprint path. On a stereo render endpoint this is **size-neutral** (the file was already 2ch +> with the mic summed into both). +> +> This **supersedes** `MicActivity` / `mic_activity.json` / masked diarization from Phase 3; the +> capture-silence handling, `diarize_samples`, and the `build_name_map` naming are retained. Segment +> IDs and the names-in-DB rule are unchanged.