151 lines
9.3 KiB
Markdown
151 lines
9.3 KiB
Markdown
# Plan: dual-channel capture (mic = left, loopback = right) — FR-SPK / FR-CAP
|
|
|
|
Status: **implemented** 2026-07-14 (all 3 sign-off answers: DB column; retire MicActivity with a
|
|
code note; export → dual-mono). Migration 0009 + `write_wav_split` + `attribute_split` +
|
|
`fold_wav` + ADR-0005 Phase 3.5. Supersedes the summed-mono + masked-diarize approach
|
|
in [`2026-07-13-diarization-phase3-per-stream-attribution.md`](./2026-07-13-diarization-phase3-per-stream-attribution.md).
|
|
The proposed **ADR-0005 amendment** is at the bottom — apply it on approval.
|
|
|
|
## Motivation
|
|
|
|
Every diarization problem this month traces to one root: **`audio.wav` is a summed mono mix** of
|
|
mic + loopback, so the two sources are irrecoverable after the fact. Phase 3 works around that by
|
|
(a) recording a separate `mic_activity.json` "You" timeline and (b) diarizing the mix with the You
|
|
ranges masked out. It works at stop but is fragile on **reprocess** — the persisted timeline has to
|
|
re-align against freshly-transcribed segments over a mono mix, and it currently collapses to a
|
|
single "Speaker 2". The timelines are being written correctly (verified: real `you_spans`), so the
|
|
failure is the masked-mono re-alignment itself, not the capture.
|
|
|
|
**Fix the root, not the instance:** keep both sources *physically separate in the recording* by
|
|
recording **stereo — left channel = microphone ("You"), right channel = system/loopback
|
|
("Speaker")**. The separation then lives in the file forever; reprocess and diarization always have
|
|
both streams; no sidecar timeline, no masking, no summed-mono clustering.
|
|
|
|
## Key facts that make this cheap
|
|
|
|
- `audio.wav` is **already 2-channel** on a normal (stereo) render endpoint — today we *sum the mic
|
|
into both channels*. Writing mic→L and loopback(downmixed)→R is the **same channel count, same
|
|
bytes** — no size increase in the common case.
|
|
- Transcription already downmixes to mono: `read_wav_mono_16k` averaging L+R of a split file yields
|
|
`(mic + loopback)/2` = the same summed signal it gets today, so **transcription is unchanged**.
|
|
- The mic↔loopback alignment machinery (`MicBridge`, the drain fix, the capture-silence fix) carries
|
|
over unchanged — we just route the aligned samples to two channels instead of summing them.
|
|
|
|
## Design
|
|
|
|
### 1. Capture write path (`audio`)
|
|
- Force the recording spec to **2ch, 16-bit, native rate** when the mic is enabled ("split"
|
|
layout). Loopback-only (mic off) keeps today's behavior.
|
|
- New writer (replacing `write_wav_bytes`'s summing for split recordings): per frame,
|
|
**L = mic sample** (0 when the mic underran), **R = downmix(loopback channels)**.
|
|
- Idle path (loopback silent): **L = drained mic, R = 0** — same silence-synthesis + device-paced
|
|
drain we just fixed, now writing to L instead of summing.
|
|
|
|
### 2. Layout versioning (distinguish old "summed" from new "split")
|
|
Content alone can't tell a summed-stereo file from a split-stereo one, so mark it explicitly:
|
|
- Add `audio_layout TEXT` to the `meetings` row: `'summed'` (default / NULL for all existing rows
|
|
and imports) or `'split'` (new mic-enabled recordings).
|
|
- Include it in the **`MeetingBundle` manifest** so exported recordings stay interpretable on
|
|
another machine.
|
|
- Playback and diarization branch on it; old recordings keep every current code path untouched.
|
|
|
|
### 3. Read helpers (`audio`)
|
|
- `read_wav_mono_16k` — **unchanged** (downmix all channels → mono 16k). Used for transcription of
|
|
both layouts; on a split file it returns the summed signal, exactly as wanted.
|
|
- New `read_wav_channel_16k(path, channel)` — extract one channel → mono 16k. Diarization reads the
|
|
**right** channel (far side); "You" detection reads the **left** (mic).
|
|
|
|
### 4. Diarization / attribution (`commands`) — *simpler* than Phase 3
|
|
For a **split** recording, one path replaces the timeline + masking:
|
|
```
|
|
far_spans = diarizer.diarize_samples( read_wav_channel_16k(wav, RIGHT) ) // clean far side
|
|
you_spans = vad_spans( read_wav_channel_16k(wav, LEFT) ) // energy-windowed mic
|
|
merge + assign_by_overlap + build_name_map("You") // You + Speaker N
|
|
```
|
|
- The mic is **never** in the clustering input — structurally, not procedurally.
|
|
- `vad_spans` is the `MicActivity` windowing logic (energy floor + 100 ms windows) applied to the L
|
|
channel from the file — so **stop and reprocess share one path** and produce identical results.
|
|
- **This deletes `MicActivity`, `mic_activity.json` (persist + read + delete), and `mask_ranges`.**
|
|
Net: less code than the current Phase 3.
|
|
- Mic-off (`'summed'`) recordings keep the whole-signal pass + voiceprint fallback.
|
|
|
|
### 5. Playback (`commands::serve_recording`)
|
|
- A split file plays mic-in-left / computer-in-right — wrong for a listener. On playback, **downmix
|
|
L+R → mono** (dual-mono) so both sources come out both ears.
|
|
- `serve_recording` already reads+decrypts the whole file into memory; add a downmix-to-mono step
|
|
for `'split'` layout and serve that mono WAV (range/seek computed over the transcoded bytes).
|
|
`'summed'` files stream as today.
|
|
|
|
### 6. Migration & portability
|
|
- **Existing recordings** (`'summed'`): every current path unchanged. Their reprocess falls back to
|
|
the voiceprint match (acceptable — they predate split). We may retire the `MicActivity`/mask code
|
|
once split is default; the handful of recent test recordings lose Phase 3 on reprocess, which is
|
|
fine.
|
|
- **Bundle export**: export **downmixes the canonical split `audio.wav` to mono** so a shared file
|
|
plays normally in any player; the working file stays split for WA's own re-processing.
|
|
|
|
## What this removes / simplifies
|
|
|
|
| Removed | Replaced by |
|
|
|---|---|
|
|
| `MicActivity` struct + capture wiring | L-channel VAD read from the file |
|
|
| `mic_activity.json` (write/read/delete) | nothing — recompute from L channel |
|
|
| `mask_ranges` + masked diarize | diarize the R channel directly |
|
|
| stop vs reprocess divergence | one shared split-attribution path |
|
|
|
|
The capture-silence fix, the device-paced `drain`, `diarize_samples`, and the `build_name_map`
|
|
naming all **stay**.
|
|
|
|
## Costs / risks (none fatal)
|
|
- **New code:** split writer, `audio_layout` column + migration + manifest field, channel read
|
|
helper, playback downmix, `vad_spans`, and the split-vs-summed branch. Bigger than a patch, but
|
|
it *retires* comparable Phase 3 code.
|
|
- **Loopback stereo image lost** (R = loopback downmixed to mono) — irrelevant for speech.
|
|
- **Mono render endpoint** (rare): 1ch→2ch doubles that recording's size.
|
|
- **Raw file plays split** in a third-party player until exported (export downmixes) — minor.
|
|
- **Clock drift** between mic/loopback now shows as tiny L/R desync instead of dropped summed
|
|
samples — harmless (diarization tolerates it; playback sums it away).
|
|
|
|
## Open questions for sign-off
|
|
1. Layout marker: **DB column** (proposed) vs. a self-describing WAV chunk? DB column is simpler and
|
|
travels via the bundle manifest; recommend it.
|
|
2. Retire `MicActivity`/`mask_ranges` outright, or keep them for existing `'summed'` recordings'
|
|
reprocess? Recommend **retire** (summed → voiceprint fallback) to avoid two live systems.
|
|
3. Export downmix to mono — agree that's the right default for shared/synced copies?
|
|
|
|
## Implementation order
|
|
1. `audio_layout` column + `MeetingBundle` field + read on `get_meeting` (no behavior yet).
|
|
2. Split writer + force 2ch when mic on; set `audio_layout='split'`. (`audio` + `start_recording`.)
|
|
3. `read_wav_channel_16k` + `vad_spans`; split-attribution path in stop + reprocess.
|
|
4. Playback downmix for split; bundle-export downmix.
|
|
5. Delete `MicActivity` / `mic_activity.json` / `mask_ranges`; update tests.
|
|
6. Apply the ADR-0005 amendment + update `docs/02`/`03`.
|
|
|
|
---
|
|
|
|
## Proposed ADR-0005 amendment (apply on approval)
|
|
|
|
> ### Phase 3.5 refinement — dual-channel capture (FR-SPK/FR-CAP, 2026-07-14)
|
|
>
|
|
> The Phase 3 per-stream attribution (above) reconstructs the mic/far-side split *after the fact*
|
|
> from a summed-mono `audio.wav` plus a `mic_activity.json` timeline, then masks the mic ranges
|
|
> before clustering. This is reliable at stop but fragile on reprocess (re-aligning a sidecar
|
|
> timeline against a mono mix), and it carries a parallel persistence path.
|
|
>
|
|
> **Decision:** when the microphone is enabled, record `audio.wav` as **stereo with the streams
|
|
> separated — left = microphone ("You"), right = system/loopback ("Speaker")** — instead of summing
|
|
> them. The separation is then intrinsic to the recording:
|
|
> - Diarization runs on the **right channel only** → `Speaker N`; "You" comes from **left-channel**
|
|
> voice activity. The mic is never clustered, by construction.
|
|
> - Reprocess recomputes both from the file — no sidecar timeline, no masking.
|
|
> - Transcription still downmixes (L+R) to the same summed mono it uses today; playback and bundle
|
|
> export downmix to mono so shared/played audio is normal.
|
|
> - A `meetings.audio_layout` flag (`summed` | `split`, in the bundle manifest for portability)
|
|
> distinguishes recordings; existing `summed` recordings keep the original blind-clustering +
|
|
> voiceprint path. On a stereo render endpoint this is **size-neutral** (the file was already 2ch
|
|
> with the mic summed into both).
|
|
>
|
|
> This **supersedes** `MicActivity` / `mic_activity.json` / masked diarization from Phase 3; the
|
|
> capture-silence handling, `diarize_samples`, and the `build_name_map` naming are retained. Segment
|
|
> IDs and the names-in-DB rule are unchanged.
|