docs(plan): draft dual-channel (mic-L/loopback-R) capture plan + ADR-0005 amendment for review (FR-SPK)
This commit is contained in:
iamdoubz committed
1 parent
6cea767dec
commit
e449c194ef
1 file changed
+148
@@ -0,0 +1,148 @@
|
||||
# Plan: dual-channel capture (mic = left, loopback = right) — FR-SPK / FR-CAP
|
||||
|
||||
Status: **draft for review**, 2026-07-14. Supersedes the summed-mono + masked-diarize approach
|
||||
in [`2026-07-13-diarization-phase3-per-stream-attribution.md`](./2026-07-13-diarization-phase3-per-stream-attribution.md).
|
||||
The proposed **ADR-0005 amendment** is at the bottom — apply it on approval.
|
||||
|
||||
## Motivation
|
||||
|
||||
Every diarization problem this month traces to one root: **`audio.wav` is a summed mono mix** of
|
||||
mic + loopback, so the two sources are irrecoverable after the fact. Phase 3 works around that by
|
||||
(a) recording a separate `mic_activity.json` "You" timeline and (b) diarizing the mix with the You
|
||||
ranges masked out. It works at stop but is fragile on **reprocess** — the persisted timeline has to
|
||||
re-align against freshly-transcribed segments over a mono mix, and it currently collapses to a
|
||||
single "Speaker 2". The timelines are being written correctly (verified: real `you_spans`), so the
|
||||
failure is the masked-mono re-alignment itself, not the capture.
|
||||
|
||||
**Fix the root, not the instance:** keep both sources *physically separate in the recording* by
|
||||
recording **stereo — left channel = microphone ("You"), right channel = system/loopback
|
||||
("Speaker")**. The separation then lives in the file forever; reprocess and diarization always have
|
||||
both streams; no sidecar timeline, no masking, no summed-mono clustering.
|
||||
|
||||
## Key facts that make this cheap
|
||||
|
||||
- `audio.wav` is **already 2-channel** on a normal (stereo) render endpoint — today we *sum the mic
|
||||
into both channels*. Writing mic→L and loopback(downmixed)→R is the **same channel count, same
|
||||
bytes** — no size increase in the common case.
|
||||
- Transcription already downmixes to mono: `read_wav_mono_16k` averaging L+R of a split file yields
|
||||
`(mic + loopback)/2` = the same summed signal it gets today, so **transcription is unchanged**.
|
||||
- The mic↔loopback alignment machinery (`MicBridge`, the drain fix, the capture-silence fix) carries
|
||||
over unchanged — we just route the aligned samples to two channels instead of summing them.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Capture write path (`audio`)
|
||||
- Force the recording spec to **2ch, 16-bit, native rate** when the mic is enabled ("split"
|
||||
layout). Loopback-only (mic off) keeps today's behavior.
|
||||
- New writer (replacing `write_wav_bytes`'s summing for split recordings): per frame,
|
||||
**L = mic sample** (0 when the mic underran), **R = downmix(loopback channels)**.
|
||||
- Idle path (loopback silent): **L = drained mic, R = 0** — same silence-synthesis + device-paced
|
||||
drain we just fixed, now writing to L instead of summing.
|
||||
|
||||
### 2. Layout versioning (distinguish old "summed" from new "split")
|
||||
Content alone can't tell a summed-stereo file from a split-stereo one, so mark it explicitly:
|
||||
- Add `audio_layout TEXT` to the `meetings` row: `'summed'` (default / NULL for all existing rows
|
||||
and imports) or `'split'` (new mic-enabled recordings).
|
||||
- Include it in the **`MeetingBundle` manifest** so exported recordings stay interpretable on
|
||||
another machine.
|
||||
- Playback and diarization branch on it; old recordings keep every current code path untouched.
|
||||
|
||||
### 3. Read helpers (`audio`)
|
||||
- `read_wav_mono_16k` — **unchanged** (downmix all channels → mono 16k). Used for transcription of
|
||||
both layouts; on a split file it returns the summed signal, exactly as wanted.
|
||||
- New `read_wav_channel_16k(path, channel)` — extract one channel → mono 16k. Diarization reads the
|
||||
**right** channel (far side); "You" detection reads the **left** (mic).
|
||||
|
||||
### 4. Diarization / attribution (`commands`) — *simpler* than Phase 3
|
||||
For a **split** recording, one path replaces the timeline + masking:
|
||||
```
|
||||
far_spans = diarizer.diarize_samples( read_wav_channel_16k(wav, RIGHT) ) // clean far side
|
||||
you_spans = vad_spans( read_wav_channel_16k(wav, LEFT) ) // energy-windowed mic
|
||||
merge + assign_by_overlap + build_name_map("You") // You + Speaker N
|
||||
```
|
||||
- The mic is **never** in the clustering input — structurally, not procedurally.
|
||||
- `vad_spans` is the `MicActivity` windowing logic (energy floor + 100 ms windows) applied to the L
|
||||
channel from the file — so **stop and reprocess share one path** and produce identical results.
|
||||
- **This deletes `MicActivity`, `mic_activity.json` (persist + read + delete), and `mask_ranges`.**
|
||||
Net: less code than the current Phase 3.
|
||||
- Mic-off (`'summed'`) recordings keep the whole-signal pass + voiceprint fallback.
|
||||
|
||||
### 5. Playback (`commands::serve_recording`)
|
||||
- A split file plays mic-in-left / computer-in-right — wrong for a listener. On playback, **downmix
|
||||
L+R → mono** (dual-mono) so both sources come out both ears.
|
||||
- `serve_recording` already reads+decrypts the whole file into memory; add a downmix-to-mono step
|
||||
for `'split'` layout and serve that mono WAV (range/seek computed over the transcoded bytes).
|
||||
`'summed'` files stream as today.
|
||||
|
||||
### 6. Migration & portability
|
||||
- **Existing recordings** (`'summed'`): every current path unchanged. Their reprocess falls back to
|
||||
the voiceprint match (acceptable — they predate split). We may retire the `MicActivity`/mask code
|
||||
once split is default; the handful of recent test recordings lose Phase 3 on reprocess, which is
|
||||
fine.
|
||||
- **Bundle export**: export **downmixes the canonical split `audio.wav` to mono** so a shared file
|
||||
plays normally in any player; the working file stays split for WA's own re-processing.
|
||||
|
||||
## What this removes / simplifies
|
||||
|
||||
| Removed | Replaced by |
|
||||
|---|---|
|
||||
| `MicActivity` struct + capture wiring | L-channel VAD read from the file |
|
||||
| `mic_activity.json` (write/read/delete) | nothing — recompute from L channel |
|
||||
| `mask_ranges` + masked diarize | diarize the R channel directly |
|
||||
| stop vs reprocess divergence | one shared split-attribution path |
|
||||
|
||||
The capture-silence fix, the device-paced `drain`, `diarize_samples`, and the `build_name_map`
|
||||
naming all **stay**.
|
||||
|
||||
## Costs / risks (none fatal)
|
||||
- **New code:** split writer, `audio_layout` column + migration + manifest field, channel read
|
||||
helper, playback downmix, `vad_spans`, and the split-vs-summed branch. Bigger than a patch, but
|
||||
it *retires* comparable Phase 3 code.
|
||||
- **Loopback stereo image lost** (R = loopback downmixed to mono) — irrelevant for speech.
|
||||
- **Mono render endpoint** (rare): 1ch→2ch doubles that recording's size.
|
||||
- **Raw file plays split** in a third-party player until exported (export downmixes) — minor.
|
||||
- **Clock drift** between mic/loopback now shows as tiny L/R desync instead of dropped summed
|
||||
samples — harmless (diarization tolerates it; playback sums it away).
|
||||
|
||||
## Open questions for sign-off
|
||||
1. Layout marker: **DB column** (proposed) vs. a self-describing WAV chunk? DB column is simpler and
|
||||
travels via the bundle manifest; recommend it.
|
||||
2. Retire `MicActivity`/`mask_ranges` outright, or keep them for existing `'summed'` recordings'
|
||||
reprocess? Recommend **retire** (summed → voiceprint fallback) to avoid two live systems.
|
||||
3. Export downmix to mono — agree that's the right default for shared/synced copies?
|
||||
|
||||
## Implementation order
|
||||
1. `audio_layout` column + `MeetingBundle` field + read on `get_meeting` (no behavior yet).
|
||||
2. Split writer + force 2ch when mic on; set `audio_layout='split'`. (`audio` + `start_recording`.)
|
||||
3. `read_wav_channel_16k` + `vad_spans`; split-attribution path in stop + reprocess.
|
||||
4. Playback downmix for split; bundle-export downmix.
|
||||
5. Delete `MicActivity` / `mic_activity.json` / `mask_ranges`; update tests.
|
||||
6. Apply the ADR-0005 amendment + update `docs/02`/`03`.
|
||||
|
||||
---
|
||||
|
||||
## Proposed ADR-0005 amendment (apply on approval)
|
||||
|
||||
> ### Phase 3.5 refinement — dual-channel capture (FR-SPK/FR-CAP, 2026-07-14)
|
||||
>
|
||||
> The Phase 3 per-stream attribution (above) reconstructs the mic/far-side split *after the fact*
|
||||
> from a summed-mono `audio.wav` plus a `mic_activity.json` timeline, then masks the mic ranges
|
||||
> before clustering. This is reliable at stop but fragile on reprocess (re-aligning a sidecar
|
||||
> timeline against a mono mix), and it carries a parallel persistence path.
|
||||
>
|
||||
> **Decision:** when the microphone is enabled, record `audio.wav` as **stereo with the streams
|
||||
> separated — left = microphone ("You"), right = system/loopback ("Speaker")** — instead of summing
|
||||
> them. The separation is then intrinsic to the recording:
|
||||
> - Diarization runs on the **right channel only** → `Speaker N`; "You" comes from **left-channel**
|
||||
> voice activity. The mic is never clustered, by construction.
|
||||
> - Reprocess recomputes both from the file — no sidecar timeline, no masking.
|
||||
> - Transcription still downmixes (L+R) to the same summed mono it uses today; playback and bundle
|
||||
> export downmix to mono so shared/played audio is normal.
|
||||
> - A `meetings.audio_layout` flag (`summed` | `split`, in the bundle manifest for portability)
|
||||
> distinguishes recordings; existing `summed` recordings keep the original blind-clustering +
|
||||
> voiceprint path. On a stereo render endpoint this is **size-neutral** (the file was already 2ch
|
||||
> with the mic summed into both).
|
||||
>
|
||||
> This **supersedes** `MicActivity` / `mic_activity.json` / masked diarization from Phase 3; the
|
||||
> capture-silence handling, `diarize_samples`, and the `build_name_map` naming are retained. Segment
|
||||
> IDs and the names-in-DB rule are unchanged.
|
||||
Reference in new issue
Block a user