diff --git a/.gitignore b/.gitignore index 352527e..66bf6b8 100644 --- a/.gitignore +++ b/.gitignore @@ -41,3 +41,8 @@ Thumbs.db # NPU runtime bundle: a large binary artifact hosted as a Gitea package, not # committed. The folder's README is tracked; the zip is produced locally. packaging/npu-runtime/*.zip + +# Vulkan loader staged next to the exe / into src-tauri by build.rs for the +# --features vulkan build (bundled into the installer); a redistributable blob, +# not committed. +src-tauri/vulkan-1.dll diff --git a/README.md b/README.md index 2ad5e85..70be6be 100644 --- a/README.md +++ b/README.md @@ -7,11 +7,13 @@ on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures th Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and transcripts **never leave the machine** unless you explicitly configure a destination. -> **Status: working application (v0.1.5).** Capture, transcription (CPU / Intel NPU / Vulkan -> GPU), speaker diarization, storage + crash recovery, local-LLM summaries, opt-in recording, -> at-rest encryption, and self-hosted sync are implemented and ship as signed **MSI + NSIS** -> installers. Outlook `.pst`/calendar context and the coding-agent (MCP) handoff are in -> progress. Build order and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md). +> **Status: working application (v0.2.0).** Capture (system audio **+ your microphone**), +> transcription (CPU / Intel NPU / Vulkan GPU), speaker diarization, storage + crash recovery, +> local-LLM summaries, AI tags, opt-in recording with **in-app playback**, at-rest encryption, +> and self-hosted sync are implemented and ship as a single signed **MSI + NSIS** universal +> installer. Outlook `.pst`/calendar context (recurring-event import + filtering) is landing; +> the coding-agent (MCP) handoff is in progress. Build order and remaining tasks are in +> [`docs/05-roadmap.md`](docs/05-roadmap.md). ## Why WhispAssist — Granola vs Meetily vs WhispAssist @@ -44,10 +46,12 @@ accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a coding-agent handoff. -## What's built (v0.1.5) +## What's built (v0.2.0) -- **Bot-free capture** — WASAPI loopback records the system mix (all participants) with no - meeting bot and no per-app plumbing. +- **Bot-free capture — now both sides** — WASAPI loopback records the system mix (all + participants), and an optional **microphone** path captures your own voice, mixed into both the + live transcript and the saved recording. Pick a specific output/input or turn the mic off in + **Settings ▸ Hardware**. No meeting bot, no per-app plumbing. - **Local transcription with a hardware ladder** — whisper.cpp via `whisper-rs` on CPU; the **Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend @@ -61,17 +65,24 @@ coding-agent handoff. - **Storage & crash recovery** — SQLite + on-disk audio/transcripts under `%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate after a crash. -- **Opt-in recording** — off by default; `.wav` retained only when you turn it on, after a - one-time consent notice. +- **Opt-in recording + in-app playback** — off by default; `.wav` retained only when you turn it + on, after a one-time consent notice. Play a saved recording back in the app — encrypted + recordings are decrypted **in memory on the fly** (nothing plaintext is written to disk). + Recordings are 16-bit for roughly half the size, and an accidental recording can be **cancelled** + (audio + transcript deleted). +- **Notes, summaries & AI tags** — Markdown notes with an **Editor/Preview** toggle; local-LLM + summaries and one-click **tag generation** with a chip-based tag editor and tag filtering. - **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts, notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock. - **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve, Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with - backoff; **client-side encryption before upload** so the destination holds only ciphertext. - Credentials live only in the OS credential store. + backoff and **live per-item upload progress**; **client-side encryption before upload** so the + destination holds only ciphertext. Credentials live only in the OS credential store. - **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same `LlmProvider` interface, off by default (third-party egress, keys in the OS credential store). -- **Installers** — signed MSI and NSIS `-setup.exe`. +- **One universal installer** — a single signed MSI and NSIS `-setup.exe` that covers every + machine: Vulkan for all GPUs, the Intel NPU path, and CPU fallback. The Vulkan loader is bundled + so it launches even on machines without a GPU driver. **In progress:** Outlook `.pst` + calendar context, the local **MCP server** that hands meeting context to your coding agents (Claude, Codex, Copilot, OpenCode), and MS Graph calendar. @@ -142,18 +153,28 @@ npm install npm run tauri dev # CPU/NPU build ``` -**GPU (Vulkan) build.** whisper.cpp's GPU backends are compiled in (not downloaded at runtime), -so a GPU build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and -a short target dir (to dodge Windows' 260-char path limit in the shader build): +**Release build (single universal installer).** whisper.cpp's GPU backends are compiled in (not +downloaded at runtime), so the release build needs a one-time toolchain setup — the **Vulkan SDK**, +a **Ninja** generator, and a short target dir (to dodge Windows' 260-char path limit in the shader +build): ```bash # after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded set VULKAN_SDK=C:\VulkanSDK\1.4.350.0 set CMAKE_GENERATOR=Ninja set CARGO_TARGET_DIR=C:\wt -npm run tauri build -- --features vulkan +npm run tauri build -- --features vulkan --config src-tauri/tauri.vulkan.conf.json ``` +This one build covers **every** machine: Vulkan accelerates all GPUs (NVIDIA/AMD/Intel), the Intel +NPU path works via the runtime OpenVINO download, and CPU is the fallback. The `--features vulkan` +binary links `vulkan-1.dll`, so `build.rs` stages the redistributable Vulkan **loader** (from +`VULKAN_SDK\Bin`, or System32) next to the exe and `tauri.vulkan.conf.json` bundles it into the +installer — the app then launches even on a machine with no GPU driver (it reports zero Vulkan +devices and decodes on the CPU). DirectML is intentionally not offered here because Vulkan already +covers those GPUs; it's the GPU path only in the plain `npm run tauri build` (no Vulkan) variant, +kept as an internal fallback. + CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build recipe lives in the project notes. diff --git a/docs/01-requirements.md b/docs/01-requirements.md index 0965f71..a0082f8 100644 --- a/docs/01-requirements.md +++ b/docs/01-requirements.md @@ -16,7 +16,9 @@ Each requirement has a stable ID used across the roadmap, tests, and commits. Pr | FR-CAP-4 | M | 1 | Show an unambiguous "recording active" indicator (in-app banner + tray icon). | | FR-CAP-5 | S | 7 | Render a live input waveform / level meter while recording. | | FR-CAP-6 | S | 7 | Handle audio device changes mid-recording without losing the session. | -| FR-CAP-7 | S | 1 | Optionally capture the user's **microphone** alongside loopback and mix it into the live transcript (default ON, local-only/no egress, selectable device + "off"). | +| FR-CAP-7 | S | 1 | Optionally capture the user's **microphone** alongside loopback, mixing it into **both** the live transcript and the saved recording (default ON, local-only/no egress, selectable device + "off"). | +| FR-CAP-8 | S | 1 | Write the retained recording as **16-bit PCM** at the device's native rate/channels — roughly half the size of the 32-bit-float mix, with no material quality loss for speech. | +| FR-CAP-9 | S | 1 | **Cancel** an in-progress recording: stop capture, delete working files, and remove the meeting from the DB entirely (for one started by mistake — no finalize/transcript/sync). | ### Recording retention & consent (REC) — see ADR-0009 @@ -26,6 +28,7 @@ Each requirement has a stable ID used across the roadmap, tests, and commits. Pr | FR-REC-2 | M | 1 | Before retaining a recording for the first time (and shown near the toggle thereafter), display a consent notice: recording without participants' consent may be illegal in some regions; advise checking local laws. Require a one-time acknowledgment; store it. This is a caution, not legal advice. | | FR-REC-3 | M | 1 | When retention is on, the UI indicates the meeting is being **saved** (in addition to the "recording active" indicator, FR-CAP-4). | | FR-REC-4 | M | 2 | Deleting working audio on finalize happens only **after** the transcript is successfully finalized; never race with crash recovery (audio stays source of truth until then). | +| FR-REC-5 | S | 2 | Play a meeting's retained `.wav` back in the app (decrypting a vault-sealed recording on demand for playback). | ### Hardware acceleration (HW) @@ -153,6 +156,7 @@ the local LLM endpoint, and it is **off by default**. Primary targets are self-h | FR-SYNC-8 | M | 9 | Surface sync state in the UI and label targets: self-hosted/primary as "your server"; third-party clouds carry a clear "data leaves your device to a third party" banner. | | FR-SYNC-9 | S | 9 | **Secondary** targets via provider APIs + OAuth 2.0 (PKCE, loopback redirect): **OneDrive** (MS Graph), **Dropbox**, **Box**. | | FR-SYNC-10 | C | 9 | Optional client-side encryption of artifacts before upload (ties to FR-SEC-3): destination holds only ciphertext. | +| FR-SYNC-11 | S | 9 | Show **live per-item upload progress** while syncing (stream the upload body and report bytes sent per artifact). | ### UX, accessibility, recovery (UX) diff --git a/docs/04-api-contracts.md b/docs/04-api-contracts.md index 0f1f25f..68a43a3 100644 --- a/docs/04-api-contracts.md +++ b/docs/04-api-contracts.md @@ -125,7 +125,7 @@ privacy_self_check(): { ## 2. Tauri events (Rust → frontend) ```ts -"recording://state" { meetingId, state: "recording"|"paused"|"stopped", elapsedMs } +"recording://state" { meetingId, state: "recording"|"paused"|"stopped"|"cancelled", elapsedMs } "recording://level" { meetingId, rms: number, peak: number } // waveform (FR-CAP-5) "recording://device" { meetingId, recovered: boolean, message: string } // capture device change (FR-CAP-6) "transcript://segment" { meetingId, segment: TranscriptSegment } // live segments (FR-TRX-2) diff --git a/package.json b/package.json index 7012e7b..72c9b4b 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "whispassist", "private": true, - "version": "0.1.6", + "version": "0.2.0", "type": "module", "description": "Privacy-first, fully local Windows meeting assistant.", "license": "MIT OR Apache-2.0", diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 7820615..ee6d6e8 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -5975,7 +5975,7 @@ dependencies = [ [[package]] name = "whispassist" -version = "0.1.6" +version = "0.2.0" dependencies = [ "argon2", "async-trait", diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index d4e9803..e9adce4 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "whispassist" -version = "0.1.6" +version = "0.2.0" description = "Privacy-first, fully local Windows meeting assistant" authors = ["WhispAssist contributors"] license = "MIT OR Apache-2.0" diff --git a/src-tauri/build.rs b/src-tauri/build.rs index 4784e65..c1df508 100644 --- a/src-tauri/build.rs +++ b/src-tauri/build.rs @@ -14,5 +14,51 @@ fn main() { println!("cargo:rustc-env=WA_GIT_HASH={hash}"); println!("cargo:rerun-if-changed=../.git/logs/HEAD"); + // The `vulkan` build links `vulkan-1.dll` at load time, so the exe won't + // launch on a machine that lacks the Vulkan loader (no GPU driver / bare VM). + // Bundling the redistributable loader (Apache-2.0) next to the exe makes the + // single universal installer start everywhere — with no GPU it simply reports + // zero devices and we fall back to CPU. Copied both next to the built exe (so + // `tauri dev`/`cargo run` work) and into the crate dir where the bundler picks + // it up as a resource (see tauri.vulkan.conf.json). + if std::env::var_os("CARGO_FEATURE_VULKAN").is_some() { + stage_vulkan_loader(); + } + tauri_build::build(); } + +/// Locate `vulkan-1.dll` (Vulkan SDK first, then System32) and copy it beside +/// the compiled exe and into the crate dir for bundling. Warns rather than fails +/// so a dev build on a machine with the loader already on PATH still succeeds. +fn stage_vulkan_loader() { + use std::path::{Path, PathBuf}; + + let source = std::env::var_os("VULKAN_SDK") + .map(|sdk| Path::new(&sdk).join("Bin").join("vulkan-1.dll")) + .filter(|p| p.exists()) + .or_else(|| { + let sys = PathBuf::from(r"C:\Windows\System32\vulkan-1.dll"); + sys.exists().then_some(sys) + }); + let Some(source) = source else { + println!( + "cargo:warning=vulkan feature is on but vulkan-1.dll wasn't found \ + (set VULKAN_SDK); the installer won't bundle the Vulkan loader" + ); + return; + }; + + // Beside the exe: OUT_DIR is target//build/-/out, so three + // parents up is target/ (correct even under CARGO_TARGET_DIR=C:\wt). + if let Some(out_dir) = std::env::var_os("OUT_DIR") { + if let Some(exe_dir) = Path::new(&out_dir).ancestors().nth(3) { + let _ = std::fs::copy(&source, exe_dir.join("vulkan-1.dll")); + } + } + // Into the crate dir for the bundler resource. + let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR"); + if let Err(e) = std::fs::copy(&source, Path::new(&manifest_dir).join("vulkan-1.dll")) { + println!("cargo:warning=failed to stage vulkan-1.dll for bundling: {e}"); + } +} diff --git a/src-tauri/src/audio/mod.rs b/src-tauri/src/audio/mod.rs index d3e29fa..0908c25 100644 --- a/src-tauri/src/audio/mod.rs +++ b/src-tauri/src/audio/mod.rs @@ -9,9 +9,11 @@ //! crate's actual source and doesn't hold for this dependency; the capture loop //! below waits on the WASAPI event handle (with a short timeout so it can also //! notice a stop/pause request) rather than busy-polling. -//! - Writes PCM to disk continuously, in the device's native format, so `audio.wav` -//! is a byte-accurate capture (audio = source of truth). A downmixed/resampled -//! 16kHz-mono copy is pushed into the bounded `FrameSink` for live transcription; +//! - Writes PCM to disk continuously as 16-bit at the device's native rate/channels +//! (FR-CAP-8), so `audio.wav` is a compact but faithful recording (audio = source +//! of truth) — about half the size of the 32-bit-float mix. A downmixed/resampled +//! 16kHz-mono copy (from the full-resolution samples) is pushed into the bounded +//! `FrameSink` for live transcription; //! a full consumer drops preview frames (`try_send`) without blocking the disk //! write or the capture thread. //! - Does no inference itself. @@ -20,9 +22,9 @@ use hound::{SampleFormat, WavSpec, WavWriter}; use std::fs::File; use std::io::BufWriter; use std::path::Path; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; use std::sync::mpsc::SyncSender; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use std::thread::{self, JoinHandle}; use std::time::{Duration, Instant}; use wasapi::{ @@ -152,6 +154,63 @@ pub fn list_capture_devices() -> Result, AudioError> { list_devices(&Direction::Capture) } +/// Shared channel the microphone capture fills and the loopback capture drains, +/// so the user's voice is mixed into the recording at the loopback's native rate +/// (FR-CAP-7 in the saved `.wav`, not just the transcript). The loopback thread +/// publishes its `rate` once it opens so the mic knows what to resample to; `buf` +/// is capped to ~0.5s to bound the two independent clocks' drift. +#[cfg(feature = "audio")] +pub struct MicBridge { + rate: AtomicU32, + buf: Mutex>, +} + +#[cfg(feature = "audio")] +impl MicBridge { + pub fn shared() -> Arc { + Arc::new(Self { + rate: AtomicU32::new(0), + buf: Mutex::new(std::collections::VecDeque::new()), + }) + } + + /// Loopback: take `n` mic samples aligned to this callback's frames, padding + /// with silence if the mic hasn't produced enough yet (underrun). + fn pull(&self, n: usize) -> Vec { + let mut out = Vec::with_capacity(n); + if let Ok(mut buf) = self.buf.lock() { + for _ in 0..n { + out.push(buf.pop_front().unwrap_or(0.0)); + } + } else { + out.resize(n, 0.0); + } + out + } + + /// Mic: append resampled samples, dropping the oldest beyond `cap` so a + /// faster mic clock (overrun) can't grow the buffer or the latency unbounded. + fn push(&self, samples: &[f32], cap: usize) { + if let Ok(mut buf) = self.buf.lock() { + buf.extend(samples); + while buf.len() > cap { + buf.pop_front(); + } + } + } +} + +/// Number of audio frames in a raw WASAPI byte buffer of the given format. +#[cfg(feature = "audio")] +fn frame_count(bytes: &[u8], format: &WaveFormat) -> usize { + let bytes_per_frame = (format.get_bitspersample() as usize / 8) * format.get_nchannels() as usize; + if bytes_per_frame == 0 { + 0 + } else { + bytes.len() / bytes_per_frame + } +} + /// Default Windows WASAPI implementation (Phase 1, feature `audio`). #[cfg(feature = "audio")] pub struct WasapiCapture; @@ -162,6 +221,7 @@ impl WasapiCapture { /// `Some` only for the loopback stream (the byte-accurate recording); the mic /// stream produces frames for transcription but no WAV. `direction` selects /// loopback (Render) vs. plain microphone capture (Capture). + #[allow(clippy::too_many_arguments)] fn start_capture( &self, thread_name: &str, @@ -170,6 +230,7 @@ impl WasapiCapture { device_id: Option<&str>, frame_sink: FrameSink, event_sink: EventSink, + bridge: Option>, ) -> Result { let running = Arc::new(AtomicBool::new(true)); let paused = Arc::new(AtomicBool::new(false)); @@ -189,6 +250,7 @@ impl WasapiCapture { &event_sink, &running_th, &paused_th, + bridge.as_ref(), ) }) .map_err(|e| AudioError::Capture(format!("spawn failed: {e}")))?; @@ -199,6 +261,48 @@ impl WasapiCapture { thread, }) } + + /// Loopback capture that mixes the microphone (delivered via `bridge`) into + /// the recording, so the saved `.wav` contains both sides of the meeting at + /// the loopback's native rate/channels (FR-CAP-7). + pub fn start_loopback_recording( + &self, + wav_path: &Path, + device_id: Option<&str>, + frame_sink: FrameSink, + event_sink: EventSink, + bridge: Arc, + ) -> Result { + self.start_capture( + "wa-audio-capture", + Some(wav_path), + Direction::Render, + device_id, + frame_sink, + event_sink, + Some(bridge), + ) + } + + /// Microphone capture that both feeds the live-transcript mixer (`frame_sink`) + /// and pushes its audio into `bridge` for the loopback thread to record. + pub fn start_microphone_recording( + &self, + device_id: Option<&str>, + frame_sink: FrameSink, + event_sink: EventSink, + bridge: Arc, + ) -> Result { + self.start_capture( + "wa-mic-capture", + None, + Direction::Capture, + device_id, + frame_sink, + event_sink, + Some(bridge), + ) + } } #[cfg(feature = "audio")] @@ -217,6 +321,7 @@ impl AudioCapture for WasapiCapture { device_id, frame_sink, event_sink, + None, ) } @@ -233,6 +338,7 @@ impl AudioCapture for WasapiCapture { device_id, frame_sink, event_sink, + None, ) } @@ -376,6 +482,7 @@ const LEVEL_EMIT_INTERVAL: Duration = Duration::from_millis(50); /// WASAPI client and the WAV writer; exits (and finalizes the WAV) once `running` /// is cleared. #[cfg(feature = "audio")] +#[allow(clippy::too_many_arguments)] fn capture_loop( wav_path: Option<&Path>, direction: Direction, @@ -384,15 +491,25 @@ fn capture_loop( event_sink: &EventSink, running: &AtomicBool, paused: &AtomicBool, + bridge: Option<&Arc>, ) -> Result { wasapi::initialize_mta() .ok() .map_err(|e| AudioError::Device(format!("COM init failed: {e}")))?; + let is_loopback = matches!(direction, Direction::Render); // Only the loopback stream emits level updates (FR-CAP-5) and writes a WAV; // the mic stream just contributes frames to the transcript mixer (FR-CAP-7). let emit_level = wav_path.is_some(); let mut session = open_capture_session(&direction, device_id)?; + // Loopback publishes its rate so the mic knows what to resample to before + // pushing into the shared bridge (mic-into-recording, FR-CAP-7). + if is_loopback { + if let Some(b) = bridge { + b.rate + .store(session.format.get_samplespersec(), Ordering::Relaxed); + } + } let spec = wav_spec_for(&session.format)?; let mut writer = match wav_path { Some(path) => Some(WavWriter::create(path, spec).map_err(|e| { @@ -407,6 +524,9 @@ fn capture_loop( .map_err(|e| AudioError::Capture(format!("start_stream failed: {e}")))?; let mut resampler = Resampler::new(session.format.get_samplespersec()); + // Mic-only: resamples the mic to the loopback's rate for the recording bridge + // (created lazily once the loopback has published its rate). + let mut bridge_resampler: Option = None; let mut queue: std::collections::VecDeque = std::collections::VecDeque::new(); let mut frames_written: u64 = 0; let mut last_level_emit = Instant::now() - LEVEL_EMIT_INTERVAL; @@ -438,6 +558,13 @@ fn capture_loop( .start_stream() .map_err(|e| AudioError::Capture(format!("restart_stream failed: {e}")))?; resampler = Resampler::new(new_session.format.get_samplespersec()); + bridge_resampler = None; + if is_loopback { + if let Some(b) = bridge { + b.rate + .store(new_session.format.get_samplespersec(), Ordering::Relaxed); + } + } session = new_session; let _ = event_sink.try_send(CaptureEvent::DeviceChanged { recovered: true, @@ -456,11 +583,34 @@ fn capture_loop( continue; } + // Loopback: mix the mic in per-frame as we write (mic silence when it + // hasn't produced enough), so the recording holds both sides (FR-CAP-7). if let Some(w) = writer.as_mut() { - frames_written += write_wav_bytes(w, &bytes, &session.format)?; + let mic = match bridge { + Some(b) if is_loopback => b.pull(frame_count(&bytes, &session.format)), + _ => Vec::new(), + }; + frames_written += write_wav_bytes(w, &bytes, &session.format, &mic)?; } let mono = decode_mono_f32(&bytes, &session.format)?; + // Mic: feed the shared bridge (resampled to the loopback's rate) so the + // loopback thread can fold it into the recording. + if !is_loopback { + if let Some(b) = bridge { + let rate = b.rate.load(Ordering::Relaxed); + if rate != 0 { + let rs = bridge_resampler.get_or_insert_with(|| { + Resampler::new_to(session.format.get_samplespersec(), rate) + }); + let resampled = rs.process(&mono); + if !resampled.is_empty() { + b.push(&resampled, (rate / 2) as usize); + } + } + } + } + if emit_level && last_level_emit.elapsed() >= LEVEL_EMIT_INTERVAL { let _ = event_sink.try_send(CaptureEvent::Level(audio_level(&mono))); last_level_emit = Instant::now(); @@ -486,58 +636,79 @@ fn capture_loop( }) } +/// The on-disk recording is always 16-bit PCM at the device's native rate and +/// channel count — half the size of the 32-bit-float mix format WASAPI hands us, +/// with no perceptible quality loss for speech (FR-CAP-8). The full-resolution +/// native samples still feed transcription untouched (see `decode_mono_f32`). fn wav_spec_for(format: &WaveFormat) -> Result { - let sample_format = match format - .get_subformat() - .map_err(|e| AudioError::Device(format!("unrecognized mix format: {e}")))? - { - SampleType::Float => SampleFormat::Float, - SampleType::Int => SampleFormat::Int, - }; Ok(WavSpec { channels: format.get_nchannels(), sample_rate: format.get_samplespersec(), - bits_per_sample: format.get_bitspersample(), - sample_format, + bits_per_sample: 16, + sample_format: SampleFormat::Int, }) } -/// Write raw WASAPI capture bytes to the WAV writer in their native format. -/// Returns the number of frames written. Only the two mix formats WASAPI shared -/// mode actually produces in practice (32-bit float, 16-bit PCM) are supported; -/// anything else is a loud error rather than a silently corrupt recording. +/// Write raw WASAPI capture bytes to the WAV writer as 16-bit PCM (FR-CAP-8), +/// quantizing the 32-bit-float mix down so `audio.wav` is ~half the size, and +/// adding one `mic` sample per frame to every channel so the recording carries +/// the user's voice too (FR-CAP-7); `mic` may be shorter than the frame count +/// (missing entries mix as silence) or empty (loopback only). Returns the number +/// of frames written. Only the two mix formats WASAPI shared mode actually +/// produces (32-bit float, 16-bit PCM) are supported; anything else is a loud +/// error rather than a silently corrupt recording. fn write_wav_bytes( writer: &mut WavWriter>, bytes: &[u8], format: &WaveFormat, + mic: &[f32], ) -> Result { let sample_type = format .get_subformat() .map_err(|e| AudioError::Device(format!("unrecognized mix format: {e}")))?; - let channels = format.get_nchannels() as u64; + let channels = format.get_nchannels() as usize; + if channels == 0 { + return Ok(0); + } + let mut frames = 0u64; match (sample_type, format.get_bitspersample()) { (SampleType::Float, 32) => { - for chunk in bytes.chunks_exact(4) { - let v = f32::from_le_bytes(chunk.try_into().unwrap()); - writer - .write_sample(v) - .map_err(|e| AudioError::Capture(format!("wav write: {e}")))?; + for frame in bytes.chunks_exact(4 * channels) { + let add = mic.get(frames as usize).copied().unwrap_or(0.0); + for c in frame.chunks_exact(4) { + let v = f32::from_le_bytes(c.try_into().unwrap()) + add; + writer + .write_sample(f32_to_i16(v)) + .map_err(|e| AudioError::Capture(format!("wav write: {e}")))?; + } + frames += 1; } - Ok(bytes.len() as u64 / 4 / channels.max(1)) } (SampleType::Int, 16) => { - for chunk in bytes.chunks_exact(2) { - let v = i16::from_le_bytes(chunk.try_into().unwrap()); - writer - .write_sample(v) - .map_err(|e| AudioError::Capture(format!("wav write: {e}")))?; + for frame in bytes.chunks_exact(2 * channels) { + let add = mic.get(frames as usize).copied().unwrap_or(0.0); + for c in frame.chunks_exact(2) { + let v = i16::from_le_bytes(c.try_into().unwrap()) as f32 / i16::MAX as f32 + add; + writer + .write_sample(f32_to_i16(v)) + .map_err(|e| AudioError::Capture(format!("wav write: {e}")))?; + } + frames += 1; } - Ok(bytes.len() as u64 / 2 / channels.max(1)) } - (st, bits) => Err(AudioError::Device(format!( - "unsupported capture format: {st} {bits}-bit" - ))), + (st, bits) => { + return Err(AudioError::Device(format!( + "unsupported capture format: {st} {bits}-bit" + ))) + } } + Ok(frames) +} + +/// Quantize a `[-1.0, 1.0]` float sample to 16-bit PCM, clamping out-of-range +/// values so an over-unity sample wraps to a click instead of full-scale noise. +fn f32_to_i16(v: f32) -> i16 { + (v.clamp(-1.0, 1.0) * i16::MAX as f32) as i16 } /// Downmix raw WASAPI capture bytes to mono `f32` in `[-1.0, 1.0]`, at the @@ -588,14 +759,22 @@ const TARGET_SAMPLE_RATE: u32 = 16_000; /// resampler if transcription accuracy complaints trace back to aliasing.` struct Resampler { src_rate: u32, + dst_rate: u32, pos: f64, carry: Vec, } impl Resampler { fn new(src_rate: u32) -> Self { + Self::new_to(src_rate, TARGET_SAMPLE_RATE) + } + + /// Resample `src_rate` -> `dst_rate` (used for the mic->loopback-rate bridge + /// as well as the 16kHz transcription path). + fn new_to(src_rate: u32, dst_rate: u32) -> Self { Self { src_rate, + dst_rate, pos: 0.0, carry: Vec::new(), } @@ -605,7 +784,7 @@ impl Resampler { if mono_in.is_empty() { return Vec::new(); } - if self.src_rate == TARGET_SAMPLE_RATE { + if self.src_rate == self.dst_rate { return mono_in.to_vec(); } @@ -616,7 +795,7 @@ impl Resampler { return Vec::new(); } - let ratio = self.src_rate as f64 / TARGET_SAMPLE_RATE as f64; + let ratio = self.src_rate as f64 / self.dst_rate as f64; let mut out = Vec::new(); while (self.pos as usize) + 1 < buf.len() { let i = self.pos as usize; @@ -835,38 +1014,95 @@ mod tests { } #[test] - fn wav_writer_roundtrip_produces_valid_header_and_duration() { + fn wav_writer_writes_16bit_from_a_float_mix_and_keeps_duration() { + // A 32-bit-float mix is captured but written as 16-bit PCM at the same + // rate/channels (FR-CAP-8) — half the bytes, same frame count. let dir = std::env::temp_dir().join(format!("wa-test-{}", uuid::Uuid::new_v4())); std::fs::create_dir_all(&dir).unwrap(); let path = dir.join("audio.wav"); - let spec = WavSpec { - channels: 2, - sample_rate: 48_000, - bits_per_sample: 32, - sample_format: SampleFormat::Float, - }; - let mut writer = WavWriter::create(&path, spec).unwrap(); - let samples: Vec = (0..2000).map(|i| (i as f32 / 2000.0) - 0.5).collect(); - let bytes: Vec = samples.iter().flat_map(|s| s.to_le_bytes()).collect(); - // format.get_subformat()/get_bitspersample() need a real WaveFormat; build one // via WaveFormat::new instead of a live device for this pure unit test. let format = WaveFormat::new(32, 32, &SampleType::Float, 48_000, 2, None); - write_wav_bytes(&mut writer, &bytes, &format).unwrap(); + let spec = wav_spec_for(&format).unwrap(); + assert_eq!(spec.bits_per_sample, 16); + assert_eq!(spec.sample_format, SampleFormat::Int); + + let mut writer = WavWriter::create(&path, spec).unwrap(); + let samples: Vec = (0..2000).map(|i| (i as f32 / 2000.0) - 0.5).collect(); + let bytes: Vec = samples.iter().flat_map(|s| s.to_le_bytes()).collect(); + write_wav_bytes(&mut writer, &bytes, &format, &[]).unwrap(); writer.finalize().unwrap(); let reader = hound::WavReader::open(&path).unwrap(); let read_spec = reader.spec(); assert_eq!(read_spec.channels, 2); assert_eq!(read_spec.sample_rate, 48_000); - assert_eq!(read_spec.bits_per_sample, 32); + assert_eq!(read_spec.bits_per_sample, 16); let frame_count = reader.duration(); assert_eq!(frame_count as usize, samples.len() / 2); let _ = std::fs::remove_dir_all(&dir); } + #[test] + fn f32_to_i16_clamps_and_scales() { + assert_eq!(f32_to_i16(0.0), 0); + assert_eq!(f32_to_i16(1.0), i16::MAX); + assert_eq!(f32_to_i16(2.0), i16::MAX); // clamped, no wrap + assert_eq!(f32_to_i16(-2.0), -i16::MAX); // clamped to -32767 + } + + #[test] + fn write_wav_bytes_mixes_the_mic_into_every_channel() { + // Silent stereo loopback + one mic sample of 0.5 for the single frame + // should land 0.5 (16383) on BOTH channels — the user's voice recorded. + let dir = std::env::temp_dir().join(format!("wa-test-{}", uuid::Uuid::new_v4())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("audio.wav"); + let format = WaveFormat::new(32, 32, &SampleType::Float, 48_000, 2, None); + let mut writer = WavWriter::create(&path, wav_spec_for(&format).unwrap()).unwrap(); + // One stereo frame of silence (L=0.0, R=0.0). + let bytes: Vec = [0.0f32, 0.0] + .iter() + .flat_map(|s| s.to_le_bytes()) + .collect(); + let frames = write_wav_bytes(&mut writer, &bytes, &format, &[0.5]).unwrap(); + writer.finalize().unwrap(); + assert_eq!(frames, 1); + + let samples: Vec = hound::WavReader::open(&path) + .unwrap() + .into_samples::() + .map(|s| s.unwrap()) + .collect(); + assert_eq!(samples, vec![16383, 16383]); // mic folded into both channels + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn mic_bridge_pull_pads_with_silence_and_push_caps() { + let bridge = MicBridge::shared(); + // Pull from empty -> all silence. + assert_eq!(bridge.pull(3), vec![0.0, 0.0, 0.0]); + // Push 5 with a cap of 3 -> only the newest 3 survive. + bridge.push(&[1.0, 2.0, 3.0, 4.0, 5.0], 3); + assert_eq!(bridge.pull(3), vec![3.0, 4.0, 5.0]); + // Now empty again -> padded. + assert_eq!(bridge.pull(2), vec![0.0, 0.0]); + } + + #[test] + fn resampler_new_to_downsamples_between_arbitrary_rates() { + // 44.1kHz -> 16kHz still lands on a passthrough when rates match… + let mut same = Resampler::new_to(16_000, 16_000); + assert_eq!(same.process(&[0.1, 0.2, 0.3]), vec![0.1, 0.2, 0.3]); + // …and produces fewer samples when downsampling (48k -> 44.1k). + let mut down = Resampler::new_to(48_000, 44_100); + let out = down.process(&(0..480).map(|i| i as f32).collect::>()); + assert!(out.len() >= 430 && out.len() <= 450, "got {}", out.len()); + } + #[test] fn read_wav_mono_16k_reads_a_normally_finalized_file() { let dir = std::env::temp_dir().join(format!("wa-test-{}", uuid::Uuid::new_v4())); @@ -1023,6 +1259,116 @@ mod tests { assert_eq!(ids.len(), devices.len()); } + #[test] + #[ignore = "captures real loopback while playing a sound; run with --ignored"] + fn loopback_16bit_wav_captures_played_audio() { + // Definitive check that the 32-bit-float -> 16-bit write path (FR-CAP-8) + // records real loopback audio (not silence): start capture, play a known + // WAV through the default device, stop, and assert the 16-bit WAV has + // non-zero samples. + let dir = std::env::temp_dir().join(format!("wa-loop-{}", uuid::Uuid::new_v4())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("audio.wav"); + let (frame_tx, _frame_rx) = std::sync::mpsc::sync_channel::>(64); + let (event_tx, _event_rx) = std::sync::mpsc::sync_channel::(8); + + let handle = match WasapiCapture.start(&path, None, frame_tx, event_tx) { + Ok(h) => h, + Err(e) => { + eprintln!("[loopback] could not open default render loopback: {e} (skipping)"); + return; + } + }; + // Play an audible system sound through the default render device so + // loopback has something to capture. PlaySync blocks for its duration. + let _ = std::process::Command::new("powershell") + .args([ + "-NoProfile", + "-Command", + "(New-Object Media.SoundPlayer 'C:\\Windows\\Media\\Alarm01.wav').PlaySync()", + ]) + .status(); + std::thread::sleep(Duration::from_millis(500)); + let summary = WasapiCapture.stop(handle).unwrap(); + + let raw = std::fs::read(&path).unwrap(); + let data_at = raw.windows(4).position(|w| w == b"data").unwrap() + 8; + let pcm = &raw[data_at..]; + let max_abs = pcm + .chunks_exact(2) + .map(|c| i16::from_le_bytes([c[0], c[1]]).unsigned_abs()) + .max() + .unwrap_or(0); + eprintln!( + "[loopback] {}Hz {}ch, {} data bytes, peak i16={max_abs}", + summary.sample_rate, + summary.channels, + pcm.len() + ); + let _ = std::fs::remove_dir_all(&dir); + assert!( + max_abs > 0, + "16-bit loopback WAV is silent despite playing audio — real conversion bug" + ); + } + + #[test] + #[ignore = "captures loopback+mic together while playing a sound; run with --ignored"] + fn loopback_recording_with_mic_bridge_captures_played_audio() { + // The exact production path when the mic is on: loopback + mic share a + // MicBridge and the recording must still contain the loopback audio (mic + // silence here just mixes in as 0). Proves the bridge doesn't zero or + // deadlock the recording. + let dir = std::env::temp_dir().join(format!("wa-mixrec-{}", uuid::Uuid::new_v4())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("audio.wav"); + let bridge = MicBridge::shared(); + let (loop_tx, _lr) = std::sync::mpsc::sync_channel::>(64); + let (mic_tx, _mr) = std::sync::mpsc::sync_channel::>(64); + let (ev_tx, _ev) = std::sync::mpsc::sync_channel::(8); + let (ev_tx2, _ev2) = std::sync::mpsc::sync_channel::(8); + + let loop_h = + match WasapiCapture.start_loopback_recording(&path, None, loop_tx, ev_tx, bridge.clone()) + { + Ok(h) => h, + Err(e) => { + eprintln!("[mixrec] no loopback device: {e} (skipping)"); + return; + } + }; + let mic_h = WasapiCapture + .start_microphone_recording(None, mic_tx, ev_tx2, bridge) + .ok(); + + let _ = std::process::Command::new("powershell") + .args([ + "-NoProfile", + "-Command", + "(New-Object Media.SoundPlayer 'C:\\Windows\\Media\\Alarm01.wav').PlaySync()", + ]) + .status(); + std::thread::sleep(Duration::from_millis(500)); + let summary = WasapiCapture.stop(loop_h).unwrap(); + if let Some(m) = mic_h { + let _ = WasapiCapture.stop(m); + } + + let raw = std::fs::read(&path).unwrap(); + let data_at = raw.windows(4).position(|w| w == b"data").unwrap() + 8; + let peak = raw[data_at..] + .chunks_exact(2) + .map(|c| i16::from_le_bytes([c[0], c[1]]).unsigned_abs()) + .max() + .unwrap_or(0); + eprintln!( + "[mixrec] {}Hz {}ch peak i16={peak}", + summary.sample_rate, summary.channels + ); + let _ = std::fs::remove_dir_all(&dir); + assert!(peak > 0, "combined loopback+mic recording is silent"); + } + #[test] fn find_render_device_resolves_a_real_enumerated_id() { let devices = list_render_devices().unwrap(); diff --git a/src-tauri/src/commands.rs b/src-tauri/src/commands.rs index 8aaf10a..0b6cede 100644 --- a/src-tauri/src/commands.rs +++ b/src-tauri/src/commands.rs @@ -296,17 +296,27 @@ pub async fn start_recording( // back to loopback-only (the mixer forwards loopback alone once its sink // drops). let (capture, mic_capture) = if settings.microphone_enabled { + // The mixer sums both streams for the live transcript; the bridge carries + // the mic into the loopback thread so the recorded WAV holds both sides + // at native quality (FR-CAP-7). + let bridge = crate::audio::MicBridge::shared(); let (loop_sink, mic_sink) = crate::audio::spawn_mixer(frame_tx); let capture = WasapiCapture - .start( + .start_loopback_recording( &wav_path, settings.audio_output_device.as_deref(), loop_sink, event_tx.clone(), + bridge.clone(), ) .map_err(|e| WaError::new("audio", e.to_string()))?; let mic = WasapiCapture - .start_microphone(settings.audio_input_device.as_deref(), mic_sink, event_tx) + .start_microphone_recording( + settings.audio_input_device.as_deref(), + mic_sink, + event_tx, + bridge, + ) .map_err(|e| tracing::warn!("microphone capture unavailable: {e}")) .ok(); (capture, mic) @@ -640,6 +650,58 @@ pub async fn stop_recording( Ok(()) } +/// Abandon the in-progress recording: stop capture, drop the transcript, and +/// delete the meeting row + its working files entirely — for a recording that +/// was started by mistake. Unlike `stop_recording`, nothing is finalized, +/// transcribed further, diarized, retained, or synced. +#[tauri::command] +pub async fn cancel_recording( + app: AppHandle, + state: State<'_, AppState>, + meeting_id: MeetingId, +) -> WaResult<()> { + let mut guard = state.session.lock().await; + let session = match guard.take() { + Some(s) if s.meeting_id == meeting_id => s, + Some(s) => { + *guard = Some(s); + return Err(WaError::new( + "recording", + "meeting_id does not match the active recording", + )); + } + None => { + return Err(WaError::new( + "recording", + "no meeting is currently recording", + )) + } + }; + drop(guard); + + // Stop both captures so their frame sinks drop and the transcription worker + // exits; join it so nothing is still touching the files we're about to delete. + let _ = WasapiCapture.stop(session.capture); + if let Some(mic) = session.mic_capture { + let _ = WasapiCapture.stop(mic); + } + let _ = session.transcription_worker.join(); + + // Remove the DB row + the whole meeting folder (working audio.wav included). + state + .store + .delete_meeting(&meeting_id) + .await + .map_err(|e| WaError::new("storage", e.to_string()))?; + + crate::update_tray_tooltip(&app, "WhispAssist — idle"); + let _ = app.emit( + "recording://state", + serde_json::json!({ "meetingId": meeting_id, "state": "cancelled", "elapsedMs": 0 }), + ); + Ok(()) +} + #[tauri::command] pub async fn pause_recording( app: AppHandle, @@ -1586,6 +1648,141 @@ pub async fn delete_meeting(state: State<'_, AppState>, meeting_id: MeetingId) - .map_err(|e| WaError::new("storage", e.to_string())) } +/// Return the `waaudio://` URL the in-app player loads for a meeting's recording +/// (FR-REC-5). The bytes are decrypted in memory on demand by `serve_recording` +/// — nothing plaintext is ever written to disk. Prechecks that a recording +/// exists and (if sealed) the vault is unlocked, so the UI can show a clear +/// error before playback; also clears any stale plaintext `audio.play.wav` left +/// by the previous file-based player. +#[tauri::command] +pub async fn recording_playback_path(meeting_id: MeetingId) -> WaResult { + let id = meeting_id.clone(); + tauri::async_runtime::spawn_blocking(move || { + let dir = meeting_dir(&id); + let wav = dir.join("audio.wav"); + if !wav.exists() { + return Err(WaError::new( + "recording", + "no saved recording for this meeting", + )); + } + // Cheap sealed check — read only the magic prefix, not the whole file. + let mut head = [0u8; 8]; + let sealed = std::fs::File::open(&wav) + .and_then(|mut f| { + use std::io::Read; + let n = f.read(&mut head)?; + Ok(n) + }) + .map(|n| crate::vault::is_sealed(&head[..n])) + .unwrap_or(false); + if sealed && !crate::vault::is_unlocked() { + return Err(WaError::new( + "recording", + "unlock the vault to play this recording", + )); + } + // Drop any plaintext temp the old file-based player left behind. + let _ = std::fs::remove_file(dir.join("audio.play.wav")); + Ok(()) + }) + .await + .map_err(|e| WaError::new("recording", e.to_string()))??; + Ok(format!("http://waaudio.localhost/{meeting_id}")) +} + +/// Custom-scheme handler backing the `waaudio://` URL: reads the meeting's +/// `audio.wav`, decrypts it in memory if sealed (T8.8), and streams the PCM to +/// the `