Feature chore bug 003 #17

Merged
iamdoubz merged 30 commits from feature_chore_bug_003 into main 2026-07-06 20:06:53 -05:00
20 changed files with 937 additions and 109 deletions
+5
View File
@@ -41,3 +41,8 @@ Thumbs.db
# NPU runtime bundle: a large binary artifact hosted as a Gitea package, not
# committed. The folder's README is tracked; the zip is produced locally.
packaging/npu-runtime/*.zip
# Vulkan loader staged next to the exe / into src-tauri by build.rs for the
# --features vulkan build (bundled into the installer); a redistributable blob,
# not committed.
src-tauri/vulkan-1.dll
+38 -17
View File
@@ -7,11 +7,13 @@ on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures th
Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and
transcripts **never leave the machine** unless you explicitly configure a destination.
> **Status: working application (v0.1.5).** Capture, transcription (CPU / Intel NPU / Vulkan
> GPU), speaker diarization, storage + crash recovery, local-LLM summaries, opt-in recording,
> at-rest encryption, and self-hosted sync are implemented and ship as signed **MSI + NSIS**
> installers. Outlook `.pst`/calendar context and the coding-agent (MCP) handoff are in
> progress. Build order and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md).
> **Status: working application (v0.2.0).** Capture (system audio **+ your microphone**),
> transcription (CPU / Intel NPU / Vulkan GPU), speaker diarization, storage + crash recovery,
> local-LLM summaries, AI tags, opt-in recording with **in-app playback**, at-rest encryption,
> and self-hosted sync are implemented and ship as a single signed **MSI + NSIS** universal
> installer. Outlook `.pst`/calendar context (recurring-event import + filtering) is landing;
> the coding-agent (MCP) handoff is in progress. Build order and remaining tasks are in
> [`docs/05-roadmap.md`](docs/05-roadmap.md).
## Why WhispAssist — Granola vs Meetily vs WhispAssist
@@ -44,10 +46,12 @@ accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern
everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a
coding-agent handoff.
## What's built (v0.1.5)
## What's built (v0.2.0)
- **Bot-free capture** — WASAPI loopback records the system mix (all participants) with no
meeting bot and no per-app plumbing.
- **Bot-free capture — now both sides** — WASAPI loopback records the system mix (all
participants), and an optional **microphone** path captures your own voice, mixed into both the
live transcript and the saved recording. Pick a specific output/input or turn the mic off in
**Settings ▸ Hardware**. No meeting bot, no per-app plumbing.
- **Local transcription with a hardware ladder** — whisper.cpp via `whisper-rs` on CPU; the
**Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on
NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend
@@ -61,17 +65,24 @@ coding-agent handoff.
- **Storage & crash recovery** — SQLite + on-disk audio/transcripts under
`%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate
after a crash.
- **Opt-in recording** — off by default; `.wav` retained only when you turn it on, after a
one-time consent notice.
- **Opt-in recording + in-app playback** — off by default; `.wav` retained only when you turn it
on, after a one-time consent notice. Play a saved recording back in the app — encrypted
recordings are decrypted **in memory on the fly** (nothing plaintext is written to disk).
Recordings are 16-bit for roughly half the size, and an accidental recording can be **cancelled**
(audio + transcript deleted).
- **Notes, summaries & AI tags** — Markdown notes with an **Editor/Preview** toggle; local-LLM
summaries and one-click **tag generation** with a chip-based tag editor and tag filtering.
- **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts,
notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock.
- **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve,
Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with
backoff; **client-side encryption before upload** so the destination holds only ciphertext.
Credentials live only in the OS credential store.
backoff and **live per-item upload progress**; **client-side encryption before upload** so the
destination holds only ciphertext. Credentials live only in the OS credential store.
- **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same
`LlmProvider` interface, off by default (third-party egress, keys in the OS credential store).
- **Installers** — signed MSI and NSIS `-setup.exe`.
- **One universal installer** — a single signed MSI and NSIS `-setup.exe` that covers every
machine: Vulkan for all GPUs, the Intel NPU path, and CPU fallback. The Vulkan loader is bundled
so it launches even on machines without a GPU driver.
**In progress:** Outlook `.pst` + calendar context, the local **MCP server** that hands meeting
context to your coding agents (Claude, Codex, Copilot, OpenCode), and MS Graph calendar.
@@ -142,18 +153,28 @@ npm install
npm run tauri dev # CPU/NPU build
```
**GPU (Vulkan) build.** whisper.cpp's GPU backends are compiled in (not downloaded at runtime),
so a GPU build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and
a short target dir (to dodge Windows' 260-char path limit in the shader build):
**Release build (single universal installer).** whisper.cpp's GPU backends are compiled in (not
downloaded at runtime), so the release build needs a one-time toolchain setup — the **Vulkan SDK**,
a **Ninja** generator, and a short target dir (to dodge Windows' 260-char path limit in the shader
build):
```bash
# after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded
set VULKAN_SDK=C:\VulkanSDK\1.4.350.0
set CMAKE_GENERATOR=Ninja
set CARGO_TARGET_DIR=C:\wt
npm run tauri build -- --features vulkan
npm run tauri build -- --features vulkan --config src-tauri/tauri.vulkan.conf.json
```
This one build covers **every** machine: Vulkan accelerates all GPUs (NVIDIA/AMD/Intel), the Intel
NPU path works via the runtime OpenVINO download, and CPU is the fallback. The `--features vulkan`
binary links `vulkan-1.dll`, so `build.rs` stages the redistributable Vulkan **loader** (from
`VULKAN_SDK\Bin`, or System32) next to the exe and `tauri.vulkan.conf.json` bundles it into the
installer — the app then launches even on a machine with no GPU driver (it reports zero Vulkan
devices and decodes on the CPU). DirectML is intentionally not offered here because Vulkan already
covers those GPUs; it's the GPU path only in the plain `npm run tauri build` (no Vulkan) variant,
kept as an internal fallback.
CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build
recipe lives in the project notes.
+5 -1
View File
@@ -16,7 +16,9 @@ Each requirement has a stable ID used across the roadmap, tests, and commits. Pr
| FR-CAP-4 | M | 1 | Show an unambiguous "recording active" indicator (in-app banner + tray icon). |
| FR-CAP-5 | S | 7 | Render a live input waveform / level meter while recording. |
| FR-CAP-6 | S | 7 | Handle audio device changes mid-recording without losing the session. |
| FR-CAP-7 | S | 1 | Optionally capture the user's **microphone** alongside loopback and mix it into the live transcript (default ON, local-only/no egress, selectable device + "off"). |
| FR-CAP-7 | S | 1 | Optionally capture the user's **microphone** alongside loopback, mixing it into **both** the live transcript and the saved recording (default ON, local-only/no egress, selectable device + "off"). |
| FR-CAP-8 | S | 1 | Write the retained recording as **16-bit PCM** at the device's native rate/channels — roughly half the size of the 32-bit-float mix, with no material quality loss for speech. |
| FR-CAP-9 | S | 1 | **Cancel** an in-progress recording: stop capture, delete working files, and remove the meeting from the DB entirely (for one started by mistake — no finalize/transcript/sync). |
### Recording retention & consent (REC) — see ADR-0009
@@ -26,6 +28,7 @@ Each requirement has a stable ID used across the roadmap, tests, and commits. Pr
| FR-REC-2 | M | 1 | Before retaining a recording for the first time (and shown near the toggle thereafter), display a consent notice: recording without participants' consent may be illegal in some regions; advise checking local laws. Require a one-time acknowledgment; store it. This is a caution, not legal advice. |
| FR-REC-3 | M | 1 | When retention is on, the UI indicates the meeting is being **saved** (in addition to the "recording active" indicator, FR-CAP-4). |
| FR-REC-4 | M | 2 | Deleting working audio on finalize happens only **after** the transcript is successfully finalized; never race with crash recovery (audio stays source of truth until then). |
| FR-REC-5 | S | 2 | Play a meeting's retained `.wav` back in the app (decrypting a vault-sealed recording on demand for playback). |
### Hardware acceleration (HW)
@@ -153,6 +156,7 @@ the local LLM endpoint, and it is **off by default**. Primary targets are self-h
| FR-SYNC-8 | M | 9 | Surface sync state in the UI and label targets: self-hosted/primary as "your server"; third-party clouds carry a clear "data leaves your device to a third party" banner. |
| FR-SYNC-9 | S | 9 | **Secondary** targets via provider APIs + OAuth 2.0 (PKCE, loopback redirect): **OneDrive** (MS Graph), **Dropbox**, **Box**. |
| FR-SYNC-10 | C | 9 | Optional client-side encryption of artifacts before upload (ties to FR-SEC-3): destination holds only ciphertext. |
| FR-SYNC-11 | S | 9 | Show **live per-item upload progress** while syncing (stream the upload body and report bytes sent per artifact). |
### UX, accessibility, recovery (UX)
+1 -1
View File
@@ -125,7 +125,7 @@ privacy_self_check(): {
## 2. Tauri events (Rust → frontend)
```ts
"recording://state" { meetingId, state: "recording"|"paused"|"stopped", elapsedMs }
"recording://state" { meetingId, state: "recording"|"paused"|"stopped"|"cancelled", elapsedMs }
"recording://level" { meetingId, rms: number, peak: number } // waveform (FR-CAP-5)
"recording://device" { meetingId, recovered: boolean, message: string } // capture device change (FR-CAP-6)
"transcript://segment" { meetingId, segment: TranscriptSegment } // live segments (FR-TRX-2)
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "whispassist",
"private": true,
"version": "0.1.6",
"version": "0.2.0",
"type": "module",
"description": "Privacy-first, fully local Windows meeting assistant.",
"license": "MIT OR Apache-2.0",
+1 -1
View File
@@ -5975,7 +5975,7 @@ dependencies = [
[[package]]
name = "whispassist"
version = "0.1.6"
version = "0.2.0"
dependencies = [
"argon2",
"async-trait",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "whispassist"
version = "0.1.6"
version = "0.2.0"
description = "Privacy-first, fully local Windows meeting assistant"
authors = ["WhispAssist contributors"]
license = "MIT OR Apache-2.0"
+46
View File
@@ -14,5 +14,51 @@ fn main() {
println!("cargo:rustc-env=WA_GIT_HASH={hash}");
println!("cargo:rerun-if-changed=../.git/logs/HEAD");
// The `vulkan` build links `vulkan-1.dll` at load time, so the exe won't
// launch on a machine that lacks the Vulkan loader (no GPU driver / bare VM).
// Bundling the redistributable loader (Apache-2.0) next to the exe makes the
// single universal installer start everywhere — with no GPU it simply reports
// zero devices and we fall back to CPU. Copied both next to the built exe (so
// `tauri dev`/`cargo run` work) and into the crate dir where the bundler picks
// it up as a resource (see tauri.vulkan.conf.json).
if std::env::var_os("CARGO_FEATURE_VULKAN").is_some() {
stage_vulkan_loader();
}
tauri_build::build();
}
/// Locate `vulkan-1.dll` (Vulkan SDK first, then System32) and copy it beside
/// the compiled exe and into the crate dir for bundling. Warns rather than fails
/// so a dev build on a machine with the loader already on PATH still succeeds.
fn stage_vulkan_loader() {
use std::path::{Path, PathBuf};
let source = std::env::var_os("VULKAN_SDK")
.map(|sdk| Path::new(&sdk).join("Bin").join("vulkan-1.dll"))
.filter(|p| p.exists())
.or_else(|| {
let sys = PathBuf::from(r"C:\Windows\System32\vulkan-1.dll");
sys.exists().then_some(sys)
});
let Some(source) = source else {
println!(
"cargo:warning=vulkan feature is on but vulkan-1.dll wasn't found \
(set VULKAN_SDK); the installer won't bundle the Vulkan loader"
);
return;
};
// Beside the exe: OUT_DIR is target/<profile>/build/<pkg>-<hash>/out, so three
// parents up is target/<profile> (correct even under CARGO_TARGET_DIR=C:\wt).
if let Some(out_dir) = std::env::var_os("OUT_DIR") {
if let Some(exe_dir) = Path::new(&out_dir).ancestors().nth(3) {
let _ = std::fs::copy(&source, exe_dir.join("vulkan-1.dll"));
}
}
// Into the crate dir for the bundler resource.
let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR");
if let Err(e) = std::fs::copy(&source, Path::new(&manifest_dir).join("vulkan-1.dll")) {
println!("cargo:warning=failed to stage vulkan-1.dll for bundling: {e}");
}
}
+396 -50
View File
@@ -9,9 +9,11 @@
//! crate's actual source and doesn't hold for this dependency; the capture loop
//! below waits on the WASAPI event handle (with a short timeout so it can also
//! notice a stop/pause request) rather than busy-polling.
//! - Writes PCM to disk continuously, in the device's native format, so `audio.wav`
//! is a byte-accurate capture (audio = source of truth). A downmixed/resampled
//! 16kHz-mono copy is pushed into the bounded `FrameSink` for live transcription;
//! - Writes PCM to disk continuously as 16-bit at the device's native rate/channels
//! (FR-CAP-8), so `audio.wav` is a compact but faithful recording (audio = source
//! of truth) — about half the size of the 32-bit-float mix. A downmixed/resampled
//! 16kHz-mono copy (from the full-resolution samples) is pushed into the bounded
//! `FrameSink` for live transcription;
//! a full consumer drops preview frames (`try_send`) without blocking the disk
//! write or the capture thread.
//! - Does no inference itself.
@@ -20,9 +22,9 @@ use hound::{SampleFormat, WavSpec, WavWriter};
use std::fs::File;
use std::io::BufWriter;
use std::path::Path;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::mpsc::SyncSender;
use std::sync::Arc;
use std::sync::{Arc, Mutex};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
use wasapi::{
@@ -152,6 +154,63 @@ pub fn list_capture_devices() -> Result<Vec<AudioDeviceInfo>, AudioError> {
list_devices(&Direction::Capture)
}
/// Shared channel the microphone capture fills and the loopback capture drains,
/// so the user's voice is mixed into the recording at the loopback's native rate
/// (FR-CAP-7 in the saved `.wav`, not just the transcript). The loopback thread
/// publishes its `rate` once it opens so the mic knows what to resample to; `buf`
/// is capped to ~0.5s to bound the two independent clocks' drift.
#[cfg(feature = "audio")]
pub struct MicBridge {
rate: AtomicU32,
buf: Mutex<std::collections::VecDeque<f32>>,
}
#[cfg(feature = "audio")]
impl MicBridge {
pub fn shared() -> Arc<Self> {
Arc::new(Self {
rate: AtomicU32::new(0),
buf: Mutex::new(std::collections::VecDeque::new()),
})
}
/// Loopback: take `n` mic samples aligned to this callback's frames, padding
/// with silence if the mic hasn't produced enough yet (underrun).
fn pull(&self, n: usize) -> Vec<f32> {
let mut out = Vec::with_capacity(n);
if let Ok(mut buf) = self.buf.lock() {
for _ in 0..n {
out.push(buf.pop_front().unwrap_or(0.0));
}
} else {
out.resize(n, 0.0);
}
out
}
/// Mic: append resampled samples, dropping the oldest beyond `cap` so a
/// faster mic clock (overrun) can't grow the buffer or the latency unbounded.
fn push(&self, samples: &[f32], cap: usize) {
if let Ok(mut buf) = self.buf.lock() {
buf.extend(samples);
while buf.len() > cap {
buf.pop_front();
}
}
}
}
/// Number of audio frames in a raw WASAPI byte buffer of the given format.
#[cfg(feature = "audio")]
fn frame_count(bytes: &[u8], format: &WaveFormat) -> usize {
let bytes_per_frame = (format.get_bitspersample() as usize / 8) * format.get_nchannels() as usize;
if bytes_per_frame == 0 {
0
} else {
bytes.len() / bytes_per_frame
}
}
/// Default Windows WASAPI implementation (Phase 1, feature `audio`).
#[cfg(feature = "audio")]
pub struct WasapiCapture;
@@ -162,6 +221,7 @@ impl WasapiCapture {
/// `Some` only for the loopback stream (the byte-accurate recording); the mic
/// stream produces frames for transcription but no WAV. `direction` selects
/// loopback (Render) vs. plain microphone capture (Capture).
#[allow(clippy::too_many_arguments)]
fn start_capture(
&self,
thread_name: &str,
@@ -170,6 +230,7 @@ impl WasapiCapture {
device_id: Option<&str>,
frame_sink: FrameSink,
event_sink: EventSink,
bridge: Option<Arc<MicBridge>>,
) -> Result<CaptureHandle, AudioError> {
let running = Arc::new(AtomicBool::new(true));
let paused = Arc::new(AtomicBool::new(false));
@@ -189,6 +250,7 @@ impl WasapiCapture {
&event_sink,
&running_th,
&paused_th,
bridge.as_ref(),
)
})
.map_err(|e| AudioError::Capture(format!("spawn failed: {e}")))?;
@@ -199,6 +261,48 @@ impl WasapiCapture {
thread,
})
}
/// Loopback capture that mixes the microphone (delivered via `bridge`) into
/// the recording, so the saved `.wav` contains both sides of the meeting at
/// the loopback's native rate/channels (FR-CAP-7).
pub fn start_loopback_recording(
&self,
wav_path: &Path,
device_id: Option<&str>,
frame_sink: FrameSink,
event_sink: EventSink,
bridge: Arc<MicBridge>,
) -> Result<CaptureHandle, AudioError> {
self.start_capture(
"wa-audio-capture",
Some(wav_path),
Direction::Render,
device_id,
frame_sink,
event_sink,
Some(bridge),
)
}
/// Microphone capture that both feeds the live-transcript mixer (`frame_sink`)
/// and pushes its audio into `bridge` for the loopback thread to record.
pub fn start_microphone_recording(
&self,
device_id: Option<&str>,
frame_sink: FrameSink,
event_sink: EventSink,
bridge: Arc<MicBridge>,
) -> Result<CaptureHandle, AudioError> {
self.start_capture(
"wa-mic-capture",
None,
Direction::Capture,
device_id,
frame_sink,
event_sink,
Some(bridge),
)
}
}
#[cfg(feature = "audio")]
@@ -217,6 +321,7 @@ impl AudioCapture for WasapiCapture {
device_id,
frame_sink,
event_sink,
None,
)
}
@@ -233,6 +338,7 @@ impl AudioCapture for WasapiCapture {
device_id,
frame_sink,
event_sink,
None,
)
}
@@ -376,6 +482,7 @@ const LEVEL_EMIT_INTERVAL: Duration = Duration::from_millis(50);
/// WASAPI client and the WAV writer; exits (and finalizes the WAV) once `running`
/// is cleared.
#[cfg(feature = "audio")]
#[allow(clippy::too_many_arguments)]
fn capture_loop(
wav_path: Option<&Path>,
direction: Direction,
@@ -384,15 +491,25 @@ fn capture_loop(
event_sink: &EventSink,
running: &AtomicBool,
paused: &AtomicBool,
bridge: Option<&Arc<MicBridge>>,
) -> Result<CaptureSummary, AudioError> {
wasapi::initialize_mta()
.ok()
.map_err(|e| AudioError::Device(format!("COM init failed: {e}")))?;
let is_loopback = matches!(direction, Direction::Render);
// Only the loopback stream emits level updates (FR-CAP-5) and writes a WAV;
// the mic stream just contributes frames to the transcript mixer (FR-CAP-7).
let emit_level = wav_path.is_some();
let mut session = open_capture_session(&direction, device_id)?;
// Loopback publishes its rate so the mic knows what to resample to before
// pushing into the shared bridge (mic-into-recording, FR-CAP-7).
if is_loopback {
if let Some(b) = bridge {
b.rate
.store(session.format.get_samplespersec(), Ordering::Relaxed);
}
}
let spec = wav_spec_for(&session.format)?;
let mut writer = match wav_path {
Some(path) => Some(WavWriter::create(path, spec).map_err(|e| {
@@ -407,6 +524,9 @@ fn capture_loop(
.map_err(|e| AudioError::Capture(format!("start_stream failed: {e}")))?;
let mut resampler = Resampler::new(session.format.get_samplespersec());
// Mic-only: resamples the mic to the loopback's rate for the recording bridge
// (created lazily once the loopback has published its rate).
let mut bridge_resampler: Option<Resampler> = None;
let mut queue: std::collections::VecDeque<u8> = std::collections::VecDeque::new();
let mut frames_written: u64 = 0;
let mut last_level_emit = Instant::now() - LEVEL_EMIT_INTERVAL;
@@ -438,6 +558,13 @@ fn capture_loop(
.start_stream()
.map_err(|e| AudioError::Capture(format!("restart_stream failed: {e}")))?;
resampler = Resampler::new(new_session.format.get_samplespersec());
bridge_resampler = None;
if is_loopback {
if let Some(b) = bridge {
b.rate
.store(new_session.format.get_samplespersec(), Ordering::Relaxed);
}
}
session = new_session;
let _ = event_sink.try_send(CaptureEvent::DeviceChanged {
recovered: true,
@@ -456,11 +583,34 @@ fn capture_loop(
continue;
}
// Loopback: mix the mic in per-frame as we write (mic silence when it
// hasn't produced enough), so the recording holds both sides (FR-CAP-7).
if let Some(w) = writer.as_mut() {
frames_written += write_wav_bytes(w, &bytes, &session.format)?;
let mic = match bridge {
Some(b) if is_loopback => b.pull(frame_count(&bytes, &session.format)),
_ => Vec::new(),
};
frames_written += write_wav_bytes(w, &bytes, &session.format, &mic)?;
}
let mono = decode_mono_f32(&bytes, &session.format)?;
// Mic: feed the shared bridge (resampled to the loopback's rate) so the
// loopback thread can fold it into the recording.
if !is_loopback {
if let Some(b) = bridge {
let rate = b.rate.load(Ordering::Relaxed);
if rate != 0 {
let rs = bridge_resampler.get_or_insert_with(|| {
Resampler::new_to(session.format.get_samplespersec(), rate)
});
let resampled = rs.process(&mono);
if !resampled.is_empty() {
b.push(&resampled, (rate / 2) as usize);
}
}
}
}
if emit_level && last_level_emit.elapsed() >= LEVEL_EMIT_INTERVAL {
let _ = event_sink.try_send(CaptureEvent::Level(audio_level(&mono)));
last_level_emit = Instant::now();
@@ -486,58 +636,79 @@ fn capture_loop(
})
}
/// The on-disk recording is always 16-bit PCM at the device's native rate and
/// channel count — half the size of the 32-bit-float mix format WASAPI hands us,
/// with no perceptible quality loss for speech (FR-CAP-8). The full-resolution
/// native samples still feed transcription untouched (see `decode_mono_f32`).
fn wav_spec_for(format: &WaveFormat) -> Result<WavSpec, AudioError> {
let sample_format = match format
.get_subformat()
.map_err(|e| AudioError::Device(format!("unrecognized mix format: {e}")))?
{
SampleType::Float => SampleFormat::Float,
SampleType::Int => SampleFormat::Int,
};
Ok(WavSpec {
channels: format.get_nchannels(),
sample_rate: format.get_samplespersec(),
bits_per_sample: format.get_bitspersample(),
sample_format,
bits_per_sample: 16,
sample_format: SampleFormat::Int,
})
}
/// Write raw WASAPI capture bytes to the WAV writer in their native format.
/// Returns the number of frames written. Only the two mix formats WASAPI shared
/// mode actually produces in practice (32-bit float, 16-bit PCM) are supported;
/// anything else is a loud error rather than a silently corrupt recording.
/// Write raw WASAPI capture bytes to the WAV writer as 16-bit PCM (FR-CAP-8),
/// quantizing the 32-bit-float mix down so `audio.wav` is ~half the size, and
/// adding one `mic` sample per frame to every channel so the recording carries
/// the user's voice too (FR-CAP-7); `mic` may be shorter than the frame count
/// (missing entries mix as silence) or empty (loopback only). Returns the number
/// of frames written. Only the two mix formats WASAPI shared mode actually
/// produces (32-bit float, 16-bit PCM) are supported; anything else is a loud
/// error rather than a silently corrupt recording.
fn write_wav_bytes(
writer: &mut WavWriter<BufWriter<File>>,
bytes: &[u8],
format: &WaveFormat,
mic: &[f32],
) -> Result<u64, AudioError> {
let sample_type = format
.get_subformat()
.map_err(|e| AudioError::Device(format!("unrecognized mix format: {e}")))?;
let channels = format.get_nchannels() as u64;
let channels = format.get_nchannels() as usize;
if channels == 0 {
return Ok(0);
}
let mut frames = 0u64;
match (sample_type, format.get_bitspersample()) {
(SampleType::Float, 32) => {
for chunk in bytes.chunks_exact(4) {
let v = f32::from_le_bytes(chunk.try_into().unwrap());
writer
.write_sample(v)
.map_err(|e| AudioError::Capture(format!("wav write: {e}")))?;
for frame in bytes.chunks_exact(4 * channels) {
let add = mic.get(frames as usize).copied().unwrap_or(0.0);
for c in frame.chunks_exact(4) {
let v = f32::from_le_bytes(c.try_into().unwrap()) + add;
writer
.write_sample(f32_to_i16(v))
.map_err(|e| AudioError::Capture(format!("wav write: {e}")))?;
}
frames += 1;
}
Ok(bytes.len() as u64 / 4 / channels.max(1))
}
(SampleType::Int, 16) => {
for chunk in bytes.chunks_exact(2) {
let v = i16::from_le_bytes(chunk.try_into().unwrap());
writer
.write_sample(v)
.map_err(|e| AudioError::Capture(format!("wav write: {e}")))?;
for frame in bytes.chunks_exact(2 * channels) {
let add = mic.get(frames as usize).copied().unwrap_or(0.0);
for c in frame.chunks_exact(2) {
let v = i16::from_le_bytes(c.try_into().unwrap()) as f32 / i16::MAX as f32 + add;
writer
.write_sample(f32_to_i16(v))
.map_err(|e| AudioError::Capture(format!("wav write: {e}")))?;
}
frames += 1;
}
Ok(bytes.len() as u64 / 2 / channels.max(1))
}
(st, bits) => Err(AudioError::Device(format!(
"unsupported capture format: {st} {bits}-bit"
))),
(st, bits) => {
return Err(AudioError::Device(format!(
"unsupported capture format: {st} {bits}-bit"
)))
}
}
Ok(frames)
}
/// Quantize a `[-1.0, 1.0]` float sample to 16-bit PCM, clamping out-of-range
/// values so an over-unity sample wraps to a click instead of full-scale noise.
fn f32_to_i16(v: f32) -> i16 {
(v.clamp(-1.0, 1.0) * i16::MAX as f32) as i16
}
/// Downmix raw WASAPI capture bytes to mono `f32` in `[-1.0, 1.0]`, at the
@@ -588,14 +759,22 @@ const TARGET_SAMPLE_RATE: u32 = 16_000;
/// resampler if transcription accuracy complaints trace back to aliasing.`
struct Resampler {
src_rate: u32,
dst_rate: u32,
pos: f64,
carry: Vec<f32>,
}
impl Resampler {
fn new(src_rate: u32) -> Self {
Self::new_to(src_rate, TARGET_SAMPLE_RATE)
}
/// Resample `src_rate` -> `dst_rate` (used for the mic->loopback-rate bridge
/// as well as the 16kHz transcription path).
fn new_to(src_rate: u32, dst_rate: u32) -> Self {
Self {
src_rate,
dst_rate,
pos: 0.0,
carry: Vec::new(),
}
@@ -605,7 +784,7 @@ impl Resampler {
if mono_in.is_empty() {
return Vec::new();
}
if self.src_rate == TARGET_SAMPLE_RATE {
if self.src_rate == self.dst_rate {
return mono_in.to_vec();
}
@@ -616,7 +795,7 @@ impl Resampler {
return Vec::new();
}
let ratio = self.src_rate as f64 / TARGET_SAMPLE_RATE as f64;
let ratio = self.src_rate as f64 / self.dst_rate as f64;
let mut out = Vec::new();
while (self.pos as usize) + 1 < buf.len() {
let i = self.pos as usize;
@@ -835,38 +1014,95 @@ mod tests {
}
#[test]
fn wav_writer_roundtrip_produces_valid_header_and_duration() {
fn wav_writer_writes_16bit_from_a_float_mix_and_keeps_duration() {
// A 32-bit-float mix is captured but written as 16-bit PCM at the same
// rate/channels (FR-CAP-8) — half the bytes, same frame count.
let dir = std::env::temp_dir().join(format!("wa-test-{}", uuid::Uuid::new_v4()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("audio.wav");
let spec = WavSpec {
channels: 2,
sample_rate: 48_000,
bits_per_sample: 32,
sample_format: SampleFormat::Float,
};
let mut writer = WavWriter::create(&path, spec).unwrap();
let samples: Vec<f32> = (0..2000).map(|i| (i as f32 / 2000.0) - 0.5).collect();
let bytes: Vec<u8> = samples.iter().flat_map(|s| s.to_le_bytes()).collect();
// format.get_subformat()/get_bitspersample() need a real WaveFormat; build one
// via WaveFormat::new instead of a live device for this pure unit test.
let format = WaveFormat::new(32, 32, &SampleType::Float, 48_000, 2, None);
write_wav_bytes(&mut writer, &bytes, &format).unwrap();
let spec = wav_spec_for(&format).unwrap();
assert_eq!(spec.bits_per_sample, 16);
assert_eq!(spec.sample_format, SampleFormat::Int);
let mut writer = WavWriter::create(&path, spec).unwrap();
let samples: Vec<f32> = (0..2000).map(|i| (i as f32 / 2000.0) - 0.5).collect();
let bytes: Vec<u8> = samples.iter().flat_map(|s| s.to_le_bytes()).collect();
write_wav_bytes(&mut writer, &bytes, &format, &[]).unwrap();
writer.finalize().unwrap();
let reader = hound::WavReader::open(&path).unwrap();
let read_spec = reader.spec();
assert_eq!(read_spec.channels, 2);
assert_eq!(read_spec.sample_rate, 48_000);
assert_eq!(read_spec.bits_per_sample, 32);
assert_eq!(read_spec.bits_per_sample, 16);
let frame_count = reader.duration();
assert_eq!(frame_count as usize, samples.len() / 2);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn f32_to_i16_clamps_and_scales() {
assert_eq!(f32_to_i16(0.0), 0);
assert_eq!(f32_to_i16(1.0), i16::MAX);
assert_eq!(f32_to_i16(2.0), i16::MAX); // clamped, no wrap
assert_eq!(f32_to_i16(-2.0), -i16::MAX); // clamped to -32767
}
#[test]
fn write_wav_bytes_mixes_the_mic_into_every_channel() {
// Silent stereo loopback + one mic sample of 0.5 for the single frame
// should land 0.5 (16383) on BOTH channels — the user's voice recorded.
let dir = std::env::temp_dir().join(format!("wa-test-{}", uuid::Uuid::new_v4()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("audio.wav");
let format = WaveFormat::new(32, 32, &SampleType::Float, 48_000, 2, None);
let mut writer = WavWriter::create(&path, wav_spec_for(&format).unwrap()).unwrap();
// One stereo frame of silence (L=0.0, R=0.0).
let bytes: Vec<u8> = [0.0f32, 0.0]
.iter()
.flat_map(|s| s.to_le_bytes())
.collect();
let frames = write_wav_bytes(&mut writer, &bytes, &format, &[0.5]).unwrap();
writer.finalize().unwrap();
assert_eq!(frames, 1);
let samples: Vec<i16> = hound::WavReader::open(&path)
.unwrap()
.into_samples::<i16>()
.map(|s| s.unwrap())
.collect();
assert_eq!(samples, vec![16383, 16383]); // mic folded into both channels
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn mic_bridge_pull_pads_with_silence_and_push_caps() {
let bridge = MicBridge::shared();
// Pull from empty -> all silence.
assert_eq!(bridge.pull(3), vec![0.0, 0.0, 0.0]);
// Push 5 with a cap of 3 -> only the newest 3 survive.
bridge.push(&[1.0, 2.0, 3.0, 4.0, 5.0], 3);
assert_eq!(bridge.pull(3), vec![3.0, 4.0, 5.0]);
// Now empty again -> padded.
assert_eq!(bridge.pull(2), vec![0.0, 0.0]);
}
#[test]
fn resampler_new_to_downsamples_between_arbitrary_rates() {
// 44.1kHz -> 16kHz still lands on a passthrough when rates match…
let mut same = Resampler::new_to(16_000, 16_000);
assert_eq!(same.process(&[0.1, 0.2, 0.3]), vec![0.1, 0.2, 0.3]);
// …and produces fewer samples when downsampling (48k -> 44.1k).
let mut down = Resampler::new_to(48_000, 44_100);
let out = down.process(&(0..480).map(|i| i as f32).collect::<Vec<_>>());
assert!(out.len() >= 430 && out.len() <= 450, "got {}", out.len());
}
#[test]
fn read_wav_mono_16k_reads_a_normally_finalized_file() {
let dir = std::env::temp_dir().join(format!("wa-test-{}", uuid::Uuid::new_v4()));
@@ -1023,6 +1259,116 @@ mod tests {
assert_eq!(ids.len(), devices.len());
}
#[test]
#[ignore = "captures real loopback while playing a sound; run with --ignored"]
fn loopback_16bit_wav_captures_played_audio() {
// Definitive check that the 32-bit-float -> 16-bit write path (FR-CAP-8)
// records real loopback audio (not silence): start capture, play a known
// WAV through the default device, stop, and assert the 16-bit WAV has
// non-zero samples.
let dir = std::env::temp_dir().join(format!("wa-loop-{}", uuid::Uuid::new_v4()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("audio.wav");
let (frame_tx, _frame_rx) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
let (event_tx, _event_rx) = std::sync::mpsc::sync_channel::<CaptureEvent>(8);
let handle = match WasapiCapture.start(&path, None, frame_tx, event_tx) {
Ok(h) => h,
Err(e) => {
eprintln!("[loopback] could not open default render loopback: {e} (skipping)");
return;
}
};
// Play an audible system sound through the default render device so
// loopback has something to capture. PlaySync blocks for its duration.
let _ = std::process::Command::new("powershell")
.args([
"-NoProfile",
"-Command",
"(New-Object Media.SoundPlayer 'C:\\Windows\\Media\\Alarm01.wav').PlaySync()",
])
.status();
std::thread::sleep(Duration::from_millis(500));
let summary = WasapiCapture.stop(handle).unwrap();
let raw = std::fs::read(&path).unwrap();
let data_at = raw.windows(4).position(|w| w == b"data").unwrap() + 8;
let pcm = &raw[data_at..];
let max_abs = pcm
.chunks_exact(2)
.map(|c| i16::from_le_bytes([c[0], c[1]]).unsigned_abs())
.max()
.unwrap_or(0);
eprintln!(
"[loopback] {}Hz {}ch, {} data bytes, peak i16={max_abs}",
summary.sample_rate,
summary.channels,
pcm.len()
);
let _ = std::fs::remove_dir_all(&dir);
assert!(
max_abs > 0,
"16-bit loopback WAV is silent despite playing audio — real conversion bug"
);
}
#[test]
#[ignore = "captures loopback+mic together while playing a sound; run with --ignored"]
fn loopback_recording_with_mic_bridge_captures_played_audio() {
// The exact production path when the mic is on: loopback + mic share a
// MicBridge and the recording must still contain the loopback audio (mic
// silence here just mixes in as 0). Proves the bridge doesn't zero or
// deadlock the recording.
let dir = std::env::temp_dir().join(format!("wa-mixrec-{}", uuid::Uuid::new_v4()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("audio.wav");
let bridge = MicBridge::shared();
let (loop_tx, _lr) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
let (mic_tx, _mr) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
let (ev_tx, _ev) = std::sync::mpsc::sync_channel::<CaptureEvent>(8);
let (ev_tx2, _ev2) = std::sync::mpsc::sync_channel::<CaptureEvent>(8);
let loop_h =
match WasapiCapture.start_loopback_recording(&path, None, loop_tx, ev_tx, bridge.clone())
{
Ok(h) => h,
Err(e) => {
eprintln!("[mixrec] no loopback device: {e} (skipping)");
return;
}
};
let mic_h = WasapiCapture
.start_microphone_recording(None, mic_tx, ev_tx2, bridge)
.ok();
let _ = std::process::Command::new("powershell")
.args([
"-NoProfile",
"-Command",
"(New-Object Media.SoundPlayer 'C:\\Windows\\Media\\Alarm01.wav').PlaySync()",
])
.status();
std::thread::sleep(Duration::from_millis(500));
let summary = WasapiCapture.stop(loop_h).unwrap();
if let Some(m) = mic_h {
let _ = WasapiCapture.stop(m);
}
let raw = std::fs::read(&path).unwrap();
let data_at = raw.windows(4).position(|w| w == b"data").unwrap() + 8;
let peak = raw[data_at..]
.chunks_exact(2)
.map(|c| i16::from_le_bytes([c[0], c[1]]).unsigned_abs())
.max()
.unwrap_or(0);
eprintln!(
"[mixrec] {}Hz {}ch peak i16={peak}",
summary.sample_rate, summary.channels
);
let _ = std::fs::remove_dir_all(&dir);
assert!(peak > 0, "combined loopback+mic recording is silent");
}
#[test]
fn find_render_device_resolves_a_real_enumerated_id() {
let devices = list_render_devices().unwrap();
+229 -9
View File
@@ -296,17 +296,27 @@ pub async fn start_recording(
// back to loopback-only (the mixer forwards loopback alone once its sink
// drops).
let (capture, mic_capture) = if settings.microphone_enabled {
// The mixer sums both streams for the live transcript; the bridge carries
// the mic into the loopback thread so the recorded WAV holds both sides
// at native quality (FR-CAP-7).
let bridge = crate::audio::MicBridge::shared();
let (loop_sink, mic_sink) = crate::audio::spawn_mixer(frame_tx);
let capture = WasapiCapture
.start(
.start_loopback_recording(
&wav_path,
settings.audio_output_device.as_deref(),
loop_sink,
event_tx.clone(),
bridge.clone(),
)
.map_err(|e| WaError::new("audio", e.to_string()))?;
let mic = WasapiCapture
.start_microphone(settings.audio_input_device.as_deref(), mic_sink, event_tx)
.start_microphone_recording(
settings.audio_input_device.as_deref(),
mic_sink,
event_tx,
bridge,
)
.map_err(|e| tracing::warn!("microphone capture unavailable: {e}"))
.ok();
(capture, mic)
@@ -640,6 +650,58 @@ pub async fn stop_recording(
Ok(())
}
/// Abandon the in-progress recording: stop capture, drop the transcript, and
/// delete the meeting row + its working files entirely — for a recording that
/// was started by mistake. Unlike `stop_recording`, nothing is finalized,
/// transcribed further, diarized, retained, or synced.
#[tauri::command]
pub async fn cancel_recording(
app: AppHandle,
state: State<'_, AppState>,
meeting_id: MeetingId,
) -> WaResult<()> {
let mut guard = state.session.lock().await;
let session = match guard.take() {
Some(s) if s.meeting_id == meeting_id => s,
Some(s) => {
*guard = Some(s);
return Err(WaError::new(
"recording",
"meeting_id does not match the active recording",
));
}
None => {
return Err(WaError::new(
"recording",
"no meeting is currently recording",
))
}
};
drop(guard);
// Stop both captures so their frame sinks drop and the transcription worker
// exits; join it so nothing is still touching the files we're about to delete.
let _ = WasapiCapture.stop(session.capture);
if let Some(mic) = session.mic_capture {
let _ = WasapiCapture.stop(mic);
}
let _ = session.transcription_worker.join();
// Remove the DB row + the whole meeting folder (working audio.wav included).
state
.store
.delete_meeting(&meeting_id)
.await
.map_err(|e| WaError::new("storage", e.to_string()))?;
crate::update_tray_tooltip(&app, "WhispAssist — idle");
let _ = app.emit(
"recording://state",
serde_json::json!({ "meetingId": meeting_id, "state": "cancelled", "elapsedMs": 0 }),
);
Ok(())
}
#[tauri::command]
pub async fn pause_recording(
app: AppHandle,
@@ -1586,6 +1648,141 @@ pub async fn delete_meeting(state: State<'_, AppState>, meeting_id: MeetingId) -
.map_err(|e| WaError::new("storage", e.to_string()))
}
/// Return the `waaudio://` URL the in-app player loads for a meeting's recording
/// (FR-REC-5). The bytes are decrypted in memory on demand by `serve_recording`
/// — nothing plaintext is ever written to disk. Prechecks that a recording
/// exists and (if sealed) the vault is unlocked, so the UI can show a clear
/// error before playback; also clears any stale plaintext `audio.play.wav` left
/// by the previous file-based player.
#[tauri::command]
pub async fn recording_playback_path(meeting_id: MeetingId) -> WaResult<String> {
let id = meeting_id.clone();
tauri::async_runtime::spawn_blocking(move || {
let dir = meeting_dir(&id);
let wav = dir.join("audio.wav");
if !wav.exists() {
return Err(WaError::new(
"recording",
"no saved recording for this meeting",
));
}
// Cheap sealed check — read only the magic prefix, not the whole file.
let mut head = [0u8; 8];
let sealed = std::fs::File::open(&wav)
.and_then(|mut f| {
use std::io::Read;
let n = f.read(&mut head)?;
Ok(n)
})
.map(|n| crate::vault::is_sealed(&head[..n]))
.unwrap_or(false);
if sealed && !crate::vault::is_unlocked() {
return Err(WaError::new(
"recording",
"unlock the vault to play this recording",
));
}
// Drop any plaintext temp the old file-based player left behind.
let _ = std::fs::remove_file(dir.join("audio.play.wav"));
Ok(())
})
.await
.map_err(|e| WaError::new("recording", e.to_string()))??;
Ok(format!("http://waaudio.localhost/{meeting_id}"))
}
/// Custom-scheme handler backing the `waaudio://` URL: reads the meeting's
/// `audio.wav`, decrypts it in memory if sealed (T8.8), and streams the PCM to
/// the `<audio>` element with byte-range support for seeking. The plaintext
/// never touches the filesystem, so playback can't undermine encryption at rest.
pub(crate) fn serve_recording(
request: &tauri::http::Request<Vec<u8>>,
) -> tauri::http::Response<Vec<u8>> {
use tauri::http::{header, Response, StatusCode};
let fail = |code: StatusCode| {
Response::builder()
.status(code)
.body(Vec::new())
.expect("static error response")
};
// Guard the id against path traversal before joining it into a path.
let id = request.uri().path().trim_start_matches('/').to_string();
if id.is_empty() || !id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-') {
return fail(StatusCode::BAD_REQUEST);
}
let raw = match std::fs::read(meeting_dir(&id).join("audio.wav")) {
Ok(b) => b,
Err(_) => return fail(StatusCode::NOT_FOUND),
};
let plain = match crate::vault::open(&raw) {
Ok(p) => p,
Err(_) => return fail(StatusCode::FORBIDDEN), // sealed + vault locked
};
let total = plain.len();
let base = || {
Response::builder()
.header(header::CONTENT_TYPE, "audio/wav")
.header(header::ACCEPT_RANGES, "bytes")
};
// Honour a single `Range` request so the player can seek.
if let Some(range) = request
.headers()
.get(header::RANGE)
.and_then(|v| v.to_str().ok())
{
if let Some((start, end)) = parse_byte_range(range, total) {
return base()
.status(StatusCode::PARTIAL_CONTENT)
.header(header::CONTENT_RANGE, format!("bytes {start}-{end}/{total}"))
.header(header::CONTENT_LENGTH, (end - start + 1).to_string())
.body(plain[start..=end].to_vec())
.unwrap_or_else(|_| fail(StatusCode::INTERNAL_SERVER_ERROR));
}
}
base()
.status(StatusCode::OK)
.header(header::CONTENT_LENGTH, total.to_string())
.body(plain)
.unwrap_or_else(|_| fail(StatusCode::INTERNAL_SERVER_ERROR))
}
/// Parse a single `bytes=start-end` range against `total`, returning inclusive
/// clamped indices. Supports open-ended (`start-`) and suffix (`-n`) forms; only
/// the common single-range case is handled (enough for an `<audio>` element).
fn parse_byte_range(header: &str, total: usize) -> Option<(usize, usize)> {
if total == 0 {
return None;
}
let (s, e) = header.strip_prefix("bytes=")?.split_once('-')?;
if s.is_empty() {
let n: usize = e.parse().ok()?;
return Some((total.saturating_sub(n), total - 1));
}
let start: usize = s.parse().ok()?;
let end = if e.is_empty() {
total - 1
} else {
e.parse::<usize>().ok()?.min(total - 1)
};
(start <= end && start < total).then_some((start, end))
}
/// Remove any leftover plaintext `audio.play.wav` files from the previous
/// file-based player, so no decrypted audio lingers on disk (T8.8).
pub(crate) fn cleanup_playback_temp() {
let Ok(entries) = std::fs::read_dir(crate::paths::meetings_dir()) else {
return;
};
for entry in entries.flatten() {
let play = entry.path().join("audio.play.wav");
if play.exists() {
let _ = std::fs::remove_file(play);
}
}
}
#[tauri::command]
pub async fn update_notes(
state: State<'_, AppState>,
@@ -2762,6 +2959,7 @@ async fn upload_job(
target: &dyn crate::sync::SyncTarget,
row: &crate::storage::SyncJobRow,
encrypt: bool,
mut on_progress: impl FnMut(u64, u64) + Send + 'static,
) -> Result<u64, crate::sync::SyncError> {
use crate::sync::SyncError;
let remote_dir = row
@@ -2771,6 +2969,22 @@ async fn upload_job(
.unwrap_or("");
target.ensure_dir(remote_dir).await?;
let (tx, rx) = std::sync::mpsc::channel();
// Forward live upload progress off the async task (the sink is a blocking
// std channel): throttled so a large recording emits a steady stream of
// `sync://job` updates without flooding the UI. Ends when `put` drops `tx`.
let fallback_total = row.bytes_total.unwrap_or(0) as u64;
let drain = std::thread::spawn(move || {
let mut last = 0u64;
let mut last_emit = std::time::Instant::now() - std::time::Duration::from_millis(500);
while let Ok((sent, total)) = rx.recv() {
last = sent;
if last_emit.elapsed() >= std::time::Duration::from_millis(250) {
on_progress(sent, total);
last_emit = std::time::Instant::now();
}
}
last
});
if encrypt {
// Client-side encryption before upload (T9.12, FR-SYNC-10): seal to a
// temp file (idempotent if already sealed at rest) so the destination
@@ -2792,12 +3006,10 @@ async fn upload_job(
.put(Path::new(&row.local_path), &row.remote_path, tx)
.await?;
}
let sent = rx
.try_iter()
.last()
.map(|(s, _)| s)
.unwrap_or(row.bytes_total.unwrap_or(0) as u64);
Ok(sent)
// `tx` is dropped now that `put` returned, so the drain thread finishes and
// hands back the final byte count it observed.
let sent = drain.join().unwrap_or(fallback_total);
Ok(if sent > 0 { sent } else { fallback_total })
}
/// Drive all due jobs once: upload each, emit `sync://job` on every transition,
@@ -2822,8 +3034,16 @@ pub(crate) async fn pump_sync(app: &AppHandle, store: &dyn crate::storage::Store
let _ = store.update_sync_job(job.clone()).await;
emit_sync_job(app, &job);
// Emit a live `sync://job` (status still "uploading") as bytes go out, so
// the UI shows a moving per-item progress bar (FR-SYNC-11).
let app_prog = app.clone();
let mut prog_row = job.clone();
let on_progress = move |sent: u64, _total: u64| {
prog_row.bytes_sent = sent as i64;
emit_sync_job(&app_prog, &prog_row);
};
let outcome = match crate::sync::build_sync_target(&target) {
Ok(t) => upload_job(t.as_ref(), &job, target.encrypt_before_upload).await,
Ok(t) => upload_job(t.as_ref(), &job, target.encrypt_before_upload, on_progress).await,
Err(e) => Err(e),
};
match outcome {
+11
View File
@@ -88,6 +88,11 @@ pub fn run() {
tauri::Builder::default()
.plugin(tauri_plugin_dialog::init())
// In-memory streaming of recordings for the player (FR-REC-5): decrypts
// on the fly so no plaintext audio is ever written to disk.
.register_uri_scheme_protocol("waaudio", |_ctx, request| {
commands::serve_recording(&request)
})
.manage(AppState {
store,
session: Mutex::new(None),
@@ -127,6 +132,10 @@ pub fn run() {
// Startup recovery + retention + reminder-reconcile pass (FR-REL-1,
// FR-STORE-2, FR-CAL-5). Spawned so it never blocks the window from
// showing (NFR-PERF-4); nothing here repeats on a timer (NFR-RES-1).
// Sweep away any plaintext playback temp files left by the previous
// file-based player (T8.8) — playback is now in-memory only.
commands::cleanup_playback_temp();
let store = store_for_setup;
let startup_app = app.handle().clone();
tauri::async_runtime::spawn(async move {
@@ -187,6 +196,8 @@ pub fn run() {
.invoke_handler(tauri::generate_handler![
commands::start_recording,
commands::stop_recording,
commands::cancel_recording,
commands::recording_playback_path,
commands::pause_recording,
commands::resume_recording,
commands::set_recording_retention,
+29 -10
View File
@@ -241,21 +241,42 @@ impl SyncTarget for WebDavTarget {
let url = self.url_for(remote_path);
let resp = self
.request(reqwest::Method::PUT, &url)?
.body(bytes)
.header(reqwest::header::CONTENT_LENGTH, total)
.body(progress_body(bytes, prog))
.send()
.await
.map_err(|e| SyncError::Upload(e.to_string()))?;
match resp.status().as_u16() {
code if (200..300).contains(&code) => {
let _ = prog.send((total, total));
Ok(())
}
code if (200..300).contains(&code) => Ok(()),
401 | 403 => Err(SyncError::Auth),
other => Err(SyncError::Upload(format!("PUT {other}"))),
}
}
}
/// Wrap in-memory bytes as a reqwest streaming body that reports cumulative
/// bytes-sent to `prog` as the request drains it — the basis for live per-item
/// upload progress (FR-SYNC-11). Chunked so the UI sees steady movement on a
/// large recording instead of a single 0→100% jump at the end.
#[cfg(feature = "sync")]
fn progress_body(bytes: Vec<u8>, prog: ProgressSink) -> reqwest::Body {
const CHUNK: usize = 64 * 1024;
let total = bytes.len() as u64;
let stream = futures_util::stream::unfold((bytes, 0usize), move |(bytes, pos)| {
let prog = prog.clone();
async move {
if pos >= bytes.len() {
return None;
}
let end = (pos + CHUNK).min(bytes.len());
let chunk = bytes[pos..end].to_vec();
let _ = prog.send((end as u64, total));
Some((Ok::<Vec<u8>, std::io::Error>(chunk), (bytes, end)))
}
});
reqwest::Body::wrap_stream(stream)
}
/// Shared HTTP client (rustls TLS via reqwest's `rustls-tls` feature).
#[cfg(feature = "sync")]
fn http_client() -> Result<reqwest::Client, SyncError> {
@@ -667,15 +688,13 @@ impl SyncTarget for OneDriveTarget {
let resp = http_client()?
.put(Self::item_url(remote_path, ":/content"))
.bearer_auth(token)
.body(bytes)
.header(reqwest::header::CONTENT_LENGTH, total)
.body(progress_body(bytes, prog))
.send()
.await
.map_err(|e| SyncError::Upload(e.to_string()))?;
match resp.status().as_u16() {
code if (200..300).contains(&code) => {
let _ = prog.send((total, total));
Ok(())
}
code if (200..300).contains(&code) => Ok(()),
401 | 403 => Err(SyncError::Auth),
other => Err(SyncError::Upload(format!("PUT {other}"))),
}
+6
View File
@@ -216,6 +216,12 @@ pub fn seal(plaintext: &[u8]) -> Result<Vec<u8>, VaultError> {
}
}
/// True if `data` begins with the vault's sealed-file magic (i.e. it's
/// ciphertext at rest, not a plaintext/pre-vault file).
pub fn is_sealed(data: &[u8]) -> bool {
data.len() >= MAGIC.len() && &data[..MAGIC.len()] == MAGIC
}
/// Inverse of `seal`. Plaintext (no magic) passes through unchanged; sealed data
/// requires the vault to be unlocked.
pub fn open(data: &[u8]) -> Result<Vec<u8>, VaultError> {
+2 -2
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "WhispAssist",
"version": "0.1.6",
"version": "0.2.0",
"identifier": "bet.dou.whispassist",
"build": {
"frontendDist": "../dist",
@@ -21,7 +21,7 @@
}
],
"security": {
"csp": "default-src 'self'; connect-src 'self' http://localhost:* http://127.0.0.1:*; img-src 'self' data:; style-src 'self' 'unsafe-inline'"
"csp": "default-src 'self'; connect-src 'self' http://localhost:* http://127.0.0.1:*; img-src 'self' data:; media-src 'self' http://waaudio.localhost; style-src 'self' 'unsafe-inline'"
},
"trayIcon": {
"iconPath": "icons/tray.png",
+6
View File
@@ -0,0 +1,6 @@
{
"$schema": "gen/schemas/desktop-schema.json",
"bundle": {
"resources": ["vulkan-1.dll"]
}
}
+34 -1
View File
@@ -14,7 +14,7 @@
import { api, type NoteTemplate } from "./lib/api";
import { onMount } from "svelte";
import ThemeToggle from "./lib/components/ThemeToggle.svelte";
import { Circle, Square, Settings as SettingsIcon, AlertTriangle } from "@lucide/svelte";
import { Circle, Square, Trash2, Settings as SettingsIcon, AlertTriangle } from "@lucide/svelte";
let showSettings = $state(false);
let showConsent = $state(false);
@@ -99,6 +99,12 @@
else recording.stop();
}
async function cancelRecording() {
if (!confirm("Discard this recording? Its audio and transcript will be deleted.")) return;
await recording.cancel();
meetings.deselect();
}
// Global shortcuts (T7.4, FR-UX-3): record start/stop, view toggles.
function isEditableTarget(target: EventTarget | null): boolean {
if (!(target instanceof HTMLElement)) return false;
@@ -199,6 +205,14 @@
<Square size={11} fill="currentColor" aria-hidden="true" />
Stop
</button>
<button
class="cancel-btn"
onclick={cancelRecording}
title="Discard this recording and delete it"
>
<Trash2 size={12} aria-hidden="true" />
Cancel
</button>
<span class="rec">
<span class="rec-dot" aria-hidden="true"></span>
Recording…
@@ -461,6 +475,25 @@
.stop-btn:hover {
background: var(--border);
}
.cancel-btn {
display: inline-flex;
align-items: center;
gap: 0.35rem;
padding: 0.35rem 0.7rem;
border-radius: var(--radius-full);
background: transparent;
color: var(--muted);
border: 1px solid var(--border);
font-size: 0.85rem;
cursor: pointer;
transition:
color 150ms ease-out,
border-color 150ms ease-out;
}
.cancel-btn:hover {
color: var(--danger);
border-color: var(--danger);
}
.rec {
display: inline-flex;
align-items: center;
+5
View File
@@ -339,6 +339,11 @@ export const api = {
}),
listNoteTemplates: () => invoke<NoteTemplate[]>("list_note_templates"),
stopRecording: (meetingId: MeetingId) => invoke<void>("stop_recording", { meetingId }),
// Abandon an accidental recording: stop + delete files + drop the DB row.
cancelRecording: (meetingId: MeetingId) => invoke<void>("cancel_recording", { meetingId }),
// Absolute path to a playable audio.wav (decrypted if sealed), for convertFileSrc.
recordingPlaybackPath: (meetingId: MeetingId) =>
invoke<string>("recording_playback_path", { meetingId }),
pauseRecording: (meetingId: MeetingId) => invoke<void>("pause_recording", { meetingId }),
resumeRecording: (meetingId: MeetingId) => invoke<void>("resume_recording", { meetingId }),
setRecordingRetention: (meetingId: MeetingId, record: boolean) =>
+18 -3
View File
@@ -18,10 +18,14 @@ class RecordingStore {
async init() {
await events.onRecordingState((p) => {
const e = p as { state: "recording" | "paused" | "stopped"; elapsedMs: number };
this.state = e.state === "stopped" ? "idle" : e.state;
const e = p as {
state: "recording" | "paused" | "stopped" | "cancelled";
elapsedMs: number;
};
const ended = e.state === "stopped" || e.state === "cancelled";
this.state = ended ? "idle" : (e.state as "recording" | "paused");
this.elapsedMs = e.elapsedMs ?? this.elapsedMs;
if (e.state === "stopped") {
if (ended) {
this.levelRms = 0;
this.levelPeak = 0;
}
@@ -60,6 +64,17 @@ class RecordingStore {
this.deviceNotice = null;
}
/** Abandon an accidental recording: stop capture and delete it entirely. */
async cancel() {
if (this.meetingId) await api.cancelRecording(this.meetingId);
this.meetingId = null;
this.segments = [];
this.state = "idle";
this.levelRms = 0;
this.levelPeak = 0;
this.deviceNotice = null;
}
/** Toggle audio retention mid-meeting (FR-REC-1); caller must have gated consent already. */
async setRetention(record: boolean) {
if (!this.meetingId) return;
+62
View File
@@ -26,6 +26,26 @@
onMount(() => calendar.load());
// ---- Recording playback (FR-REC-5) ----
let audioSrc = $state<string | null>(null);
let audioError = $state<string | null>(null);
$effect(() => {
const m = meetings.selected;
audioSrc = null;
audioError = null;
if (m?.recorded) {
const id = m.id;
api
.recordingPlaybackPath(id)
.then((url) => {
if (meetings.selected?.id === id) audioSrc = url;
})
.catch((e) => {
if (meetings.selected?.id === id) audioError = errorMessage(e);
});
}
});
// ---- Summary + action items (T5.4/T5.5/T5.6, FR-LLM-2/3/4) ----
let llmStatus = $state<LlmStatus | null>(null);
onMount(async () => {
@@ -233,6 +253,26 @@
</script>
<div class="wrap">
{#if meetings.selected?.recorded}
<h3><Mic2 size={14} aria-hidden="true" /> Recording</h3>
{#if audioSrc}
<!-- svelte-ignore a11y_media_has_caption -- a meeting recording has no caption track -->
<!-- controlsList/contextmenu: no download affordance — the decrypted audio
must not be savable to disk (would undermine encryption at rest). -->
<audio
class="player"
controls
controlsList="nodownload noplaybackrate"
oncontextmenu={(e) => e.preventDefault()}
src={audioSrc}
></audio>
{:else if audioError}
<p class="muted">{audioError}</p>
{:else}
<p class="muted">Loading recording…</p>
{/if}
{/if}
{#if meetings.selected && settings.settings.sync_enabled}
<h3><UploadCloud size={14} aria-hidden="true" /> Sync</h3>
<button
@@ -252,6 +292,7 @@
<span class="artifact">{job.artifact}</span>
<span class="job-status {job.status}">{job.status}</span>
{#if job.status === "uploading" && job.bytesTotal}
<progress class="job-bar" max={job.bytesTotal} value={job.bytesSent}></progress>
<span class="muted">{Math.round((100 * job.bytesSent) / job.bytesTotal)}%</span>
{/if}
{#if job.status === "failed"}
@@ -683,6 +724,27 @@
.artifact {
min-width: 5rem;
}
.job-bar {
flex: 1;
height: 0.4rem;
min-width: 3rem;
border: none;
border-radius: var(--radius-full);
overflow: hidden;
accent-color: var(--accent, #2563eb);
}
.job-bar::-webkit-progress-bar {
background: var(--bg-hover);
border-radius: var(--radius-full);
}
.job-bar::-webkit-progress-value {
background: var(--accent, #2563eb);
border-radius: var(--radius-full);
}
.player {
width: 100%;
margin: 0.2rem 0 0.6rem;
}
.job-status {
text-transform: capitalize;
color: var(--muted);
+41 -12
View File
@@ -21,6 +21,8 @@
RefreshCw,
MessageSquareText,
NotebookPen,
Eye,
Pencil,
} from "@lucide/svelte";
function speakerName(label: string, speakers: SpeakerInfo[] = []): string {
@@ -28,6 +30,9 @@
}
let notesText = $state("");
// Notes is a single pane: raw markdown ("Editor") or the rendered result
// ("Preview"), toggled by one button whose label flips to the other mode.
let notesPreview = $state(false);
let editorEl: HTMLTextAreaElement | undefined = $state();
let saveTimer: ReturnType<typeof setTimeout> | undefined;
let loadedForId: string | null = null;
@@ -201,6 +206,20 @@
>
<ListChecks size={14} aria-hidden="true" />
</button>
<button
class="toggle"
onclick={() => (notesPreview = !notesPreview)}
title={notesPreview ? "Edit the raw markdown" : "Render the markdown"}
aria-pressed={notesPreview}
>
{#if notesPreview}
<Pencil size={14} aria-hidden="true" />
Editor
{:else}
<Eye size={14} aria-hidden="true" />
Preview
{/if}
</button>
<span class="spacer"></span>
<button onclick={exportMd} title="Export notes as .md">
<FileText size={13} aria-hidden="true" />
@@ -220,14 +239,17 @@
</button>
</div>
<div class="editor-preview">
<textarea
bind:this={editorEl}
bind:value={notesText}
oninput={scheduleSave}
placeholder="Notes…"
></textarea>
<!-- eslint-disable-next-line svelte/no-at-html-tags -- sanitized via renderMarkdown() -->
<div class="preview">{@html renderMarkdown(notesText)}</div>
{#if notesPreview}
<!-- eslint-disable-next-line svelte/no-at-html-tags -- sanitized via renderMarkdown() -->
<div class="preview">{@html renderMarkdown(notesText)}</div>
{:else}
<textarea
bind:this={editorEl}
bind:value={notesText}
oninput={scheduleSave}
placeholder="Notes…"
></textarea>
{/if}
</div>
</div>
</div>
@@ -366,11 +388,14 @@
}
.editor-preview {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 0.75rem;
height: calc(100% - 2.5rem);
}
.toolbar .toggle {
font-weight: 600;
}
.toolbar .toggle[aria-pressed="true"] {
background: var(--bg-hover);
}
textarea {
resize: none;
width: 100%;
@@ -388,8 +413,12 @@
border-color: var(--accent);
}
.preview {
height: 100%;
box-sizing: border-box;
overflow: auto;
padding: 0.25rem 0.5rem;
padding: 0.6rem;
border: 1px solid var(--border);
border-radius: var(--radius-md);
font-size: 0.9rem;
line-height: 1.5;
}