WhispAssist/docs/plans/2026-07-14-stereo-dual-channel-capture.md

Plan: dual-channel capture (mic = left, loopback = right) — FR-SPK / FR-CAP

Status: implemented 2026-07-14 (all 3 sign-off answers: DB column; retire MicActivity with a code note; export → dual-mono). Migration 0009 + write_wav_split + attribute_split + fold_wav + ADR-0005 Phase 3.5. Supersedes the summed-mono + masked-diarize approach in 2026-07-13-diarization-phase3-per-stream-attribution.md. The proposed ADR-0005 amendment is at the bottom — apply it on approval.

Motivation

Every diarization problem this month traces to one root: audio.wav is a summed mono mix of mic + loopback, so the two sources are irrecoverable after the fact. Phase 3 works around that by (a) recording a separate mic_activity.json "You" timeline and (b) diarizing the mix with the You ranges masked out. It works at stop but is fragile on reprocess — the persisted timeline has to re-align against freshly-transcribed segments over a mono mix, and it currently collapses to a single "Speaker 2". The timelines are being written correctly (verified: real you_spans), so the failure is the masked-mono re-alignment itself, not the capture.

Fix the root, not the instance: keep both sources physically separate in the recording by recording stereo — left channel = microphone ("You"), right channel = system/loopback ("Speaker"). The separation then lives in the file forever; reprocess and diarization always have both streams; no sidecar timeline, no masking, no summed-mono clustering.

Key facts that make this cheap

• audio.wav is already 2-channel on a normal (stereo) render endpoint — today we sum the mic into both channels. Writing mic→L and loopback(downmixed)→R is the same channel count, same bytes — no size increase in the common case.
• Transcription already downmixes to mono: read_wav_mono_16k averaging L+R of a split file yields (mic + loopback)/2 = the same summed signal it gets today, so transcription is unchanged.
• The mic↔loopback alignment machinery (MicBridge, the drain fix, the capture-silence fix) carries over unchanged — we just route the aligned samples to two channels instead of summing them.

Design

1. Capture write path (audio)

• Force the recording spec to 2ch, 16-bit, native rate when the mic is enabled ("split" layout). Loopback-only (mic off) keeps today's behavior.
• New writer (replacing write_wav_bytes's summing for split recordings): per frame, L = mic sample (0 when the mic underran), R = downmix(loopback channels).
• Idle path (loopback silent): L = drained mic, R = 0 — same silence-synthesis + device-paced drain we just fixed, now writing to L instead of summing.

2. Layout versioning (distinguish old "summed" from new "split")

Content alone can't tell a summed-stereo file from a split-stereo one, so mark it explicitly:

• Add audio_layout TEXT to the meetings row: 'summed' (default / NULL for all existing rows and imports) or 'split' (new mic-enabled recordings).
• Include it in the MeetingBundle manifest so exported recordings stay interpretable on another machine.
• Playback and diarization branch on it; old recordings keep every current code path untouched.

3. Read helpers (audio)

• read_wav_mono_16k — unchanged (downmix all channels → mono 16k). Used for transcription of both layouts; on a split file it returns the summed signal, exactly as wanted.
• New read_wav_channel_16k(path, channel) — extract one channel → mono 16k. Diarization reads the right channel (far side); "You" detection reads the left (mic).

4. Diarization / attribution (commands) — simpler than Phase 3

For a split recording, one path replaces the timeline + masking:

far_spans = diarizer.diarize_samples( read_wav_channel_16k(wav, RIGHT) )   // clean far side
you_spans = vad_spans( read_wav_channel_16k(wav, LEFT) )                    // energy-windowed mic
merge + assign_by_overlap + build_name_map("You")                           // You + Speaker N

• The mic is never in the clustering input — structurally, not procedurally.
• vad_spans is the MicActivity windowing logic (energy floor + 100 ms windows) applied to the L channel from the file — so stop and reprocess share one path and produce identical results.
• This deletes MicActivity, mic_activity.json (persist + read + delete), and mask_ranges. Net: less code than the current Phase 3.
• Mic-off ('summed') recordings keep the whole-signal pass + voiceprint fallback.

5. Playback (commands::serve_recording)

• A split file plays mic-in-left / computer-in-right — wrong for a listener. On playback, downmix L+R → mono (dual-mono) so both sources come out both ears.
• serve_recording already reads+decrypts the whole file into memory; add a downmix-to-mono step for 'split' layout and serve that mono WAV (range/seek computed over the transcoded bytes). 'summed' files stream as today.

6. Migration & portability

• Existing recordings ('summed'): every current path unchanged. Their reprocess falls back to the voiceprint match (acceptable — they predate split). We may retire the MicActivity/mask code once split is default; the handful of recent test recordings lose Phase 3 on reprocess, which is fine.
• Bundle export: export downmixes the canonical split audio.wav to mono so a shared file plays normally in any player; the working file stays split for WA's own re-processing.

What this removes / simplifies

Removed	Replaced by
MicActivity struct + capture wiring	L-channel VAD read from the file
mic_activity.json (write/read/delete)	nothing — recompute from L channel
mask_ranges + masked diarize	diarize the R channel directly
stop vs reprocess divergence	one shared split-attribution path

The capture-silence fix, the device-paced drain, diarize_samples, and the build_name_map naming all stay.

Costs / risks (none fatal)

• New code: split writer, audio_layout column + migration + manifest field, channel read helper, playback downmix, vad_spans, and the split-vs-summed branch. Bigger than a patch, but it retires comparable Phase 3 code.
• Loopback stereo image lost (R = loopback downmixed to mono) — irrelevant for speech.
• Mono render endpoint (rare): 1ch→2ch doubles that recording's size.
• Raw file plays split in a third-party player until exported (export downmixes) — minor.
• Clock drift between mic/loopback now shows as tiny L/R desync instead of dropped summed samples — harmless (diarization tolerates it; playback sums it away).

Open questions for sign-off

1. Layout marker: DB column (proposed) vs. a self-describing WAV chunk? DB column is simpler and travels via the bundle manifest; recommend it.
2. Retire MicActivity/mask_ranges outright, or keep them for existing 'summed' recordings' reprocess? Recommend retire (summed → voiceprint fallback) to avoid two live systems.
3. Export downmix to mono — agree that's the right default for shared/synced copies?

Implementation order

1. audio_layout column + MeetingBundle field + read on get_meeting (no behavior yet).
2. Split writer + force 2ch when mic on; set audio_layout='split'. (audio + start_recording.)
3. read_wav_channel_16k + vad_spans; split-attribution path in stop + reprocess.
4. Playback downmix for split; bundle-export downmix.
5. Delete MicActivity / mic_activity.json / mask_ranges; update tests.
6. Apply the ADR-0005 amendment + update docs/02/03.

---

Proposed ADR-0005 amendment (apply on approval)

Phase 3.5 refinement — dual-channel capture (FR-SPK/FR-CAP, 2026-07-14)

The Phase 3 per-stream attribution (above) reconstructs the mic/far-side split after the fact from a summed-mono audio.wav plus a mic_activity.json timeline, then masks the mic ranges before clustering. This is reliable at stop but fragile on reprocess (re-aligning a sidecar timeline against a mono mix), and it carries a parallel persistence path.

Decision: when the microphone is enabled, record audio.wav as stereo with the streams separated — left = microphone ("You"), right = system/loopback ("Speaker") — instead of summing them. The separation is then intrinsic to the recording:

• Diarization runs on the right channel only → Speaker N; "You" comes from left-channel voice activity. The mic is never clustered, by construction.
• Reprocess recomputes both from the file — no sidecar timeline, no masking.
• Transcription still downmixes (L+R) to the same summed mono it uses today; playback and bundle export downmix to mono so shared/played audio is normal.
• A meetings.audio_layout flag (summed | split, in the bundle manifest for portability) distinguishes recordings; existing summed recordings keep the original blind-clustering + voiceprint path. On a stereo render endpoint this is size-neutral (the file was already 2ch with the mic summed into both).

This supersedes MicActivity / mic_activity.json / masked diarization from Phase 3; the capture-silence handling, diarize_samples, and the build_name_map naming are retained. Segment IDs and the names-in-DB rule are unchanged.