WhispAssist/docs/02-architecture.md

02 — Architecture

Component map

┌─────────────────────────────────────────────────────────────────────┐
│  WebView2 window                                                      │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │  Svelte frontend (TypeScript)                                  │  │
│  │   views: MeetingsList · TranscriptNotes · SummaryPanel ·       │  │
│  │          Settings · SpeakerReview · CalendarView               │  │
│  │   stores: recording, meeting, transcript, hardware, settings   │  │
│  └───────────────▲───────────────────────────┬───────────────────┘  │
│        Tauri events (push)        Tauri commands (request/response)   │
└──────────────────┼───────────────────────────┼──────────────────────┘
                   │                            │
┌──────────────────┴───────────────────────────▼──────────────────────┐
│  Rust core (Tauri)                                                    │
│                                                                       │
│   commands.rs  ──orchestrates──►  AppState { service handles }        │
│                                                                       │
│   ┌─────────┐  ┌──────────┐  ┌────────────┐  ┌──────────────┐         │
│   │ audio   │  │ hardware │  │transcription│  │ diarization │         │
│   │ (WASAPI)│  │(detect)  │  │(whisper/onnx)│ │(sherpa-onnx)│         │
│   └────┬────┘  └────┬─────┘  └─────┬──────┘  └──────┬───────┘         │
│        │            │              │                │                 │
│   ┌────▼────────────▼──────────────▼────────────────▼───────┐         │
│   │ storage (SQLite + files)   notes (markdown)             │         │
│   └────┬───────────────────────────────────┬───────────────┘         │
│        │                                    │                         │
│   ┌────▼─────┐   ┌──────────────┐  ┌──────────┐  ┌─────▼──────┐       │
│   │   llm    │   │     sync     │  │   mcp    │  │  calendar  │       │
│   │(Ollama / │   │(WebDAV/OAuth)│  │ (server) │  │ (.pst/Graph)│      │
│   │ cloud)   │   └──────┬───────┘  └────▲─────┘  └────────────┘       │
│   └────┬─────┘          │               │ loopback (INBOUND)          │
└────────┼────────────────┼───────────────┼──────────────────────────────┘
         │ HTTP/HTTPS     │ HTTPS         │ MCP over stdio / http://127.0.0.1/mcp
         │ (local or      │ (targets,     │ (off by default; token; NOT egress)
         │  cloud AI)     │  off default) │
   ┌─────▼─────┐    ┌──────▼───────────┐  │   ┌──────────────────────────────┐
   │ Ollama /  │    │ Nextcloud·Seafile│  └───│ Claude · Codex · Copilot ·   │
   │ Anthropic │    │ ·Cloudreve·…     │      │ OpenCode (user's own agents) │
   │ /OpenAI   │    │ |OneDrive·Dropbox│      │ — they may forward to their  │
   └───────────┘    └──────────────────┘      │ own cloud (outside WA)       │
                                              └──────────────────────────────┘

The mcp arrow points inward: agents connect to WA, not the reverse. WA opens no socket for it.

Each Rust service lives in its own module under src-tauri/src/<service>/ and exposes a trait plus a default implementation. Callers depend on the trait (NFR-MNT-1/2). The frontend never touches files, DB, or network directly — only Tauri commands/events (04-api-contracts.md).

Services (responsibilities & key dependencies)

Service	Responsibility	Primary crate(s)
audio	WASAPI loopback capture (+ optional microphone, mixed into the transcript stream, FR-CAP-7); PCM ring buffer; write WAV to disk; pause/resume	wasapi, hound
hardware	Enumerate NPU/GPU/CPU; rank backends; report capabilities	ort, DXGI via windows
transcription	Load model on a backend; stream segments (whisper.cpp) or NPU (ONNX)	whisper-rs, ort
diarization	Post-process audio → speaker spans; align to segments; merge	sherpa-onnx (FFI)
storage	SQLite index + file layout; CRUD; FTS search; retention; recovery scan	sqlx, serde_json
notes	Assemble Markdown from segments+speakers+summary; render/export	pulldown-cmark
llm	Talk to Ollama/custom endpoint; build prompts; stream; parse action items	reqwest, serde
calendar	Parse .pst events/attendees; (future) Graph/ICS sources	outlook-pst
sync	Upload artifacts to user-configured targets; durable queue + retry; WebDAV + OAuth providers; OS-keychain creds	reqwest, reqwest_dav, keyring
mcp	Host a local (loopback) MCP server exposing meeting context as tools; build feature briefs; scope + token + audit	rmcp, axum/hyper

The audio service also owns recording retention (ADR-0009): the working WAV is kept as the permanent audio.wav only when "Record this meeting" is on; otherwise it is deleted on finalize.

The llm service is extended (ADR-0011, Layer 1) with optional hosted providers (Anthropic Messages, OpenAI-compatible) behind the same LlmProvider trait, in addition to local Ollama. An optional agent module (Layer 3, later) spawns local coding-agent CLIs and/or creates task-tracker issues from feature briefs.

Data flow — a typical meeting

1. User selects/creates a meeting (optionally from a calendar event) → start_recording.
2. audio begins WASAPI loopback capture, writing PCM to audio.wav continuously (FR-CAP-2), and pushes frames into an in-memory ring buffer.
3. transcription consumes windowed audio from the ring buffer on the selected hardware backend and emits interim TranscriptSegments → transcript_segment events to the UI (FR-TRX-2).
4. Provisional speaker turns (from cheap segmentation) tag segments live; the user may name a speaker, applied immediately to past/future segments (FR-SPK-2).
5. The notes view renders Markdown that updates as segments arrive (FR-NOTE-1, FR-NOTE-6).
6. On stop_recording: finalize the WAV; run diarization, align speaker IDs to segments, apply name mappings; persist transcript JSON + metadata via storage (FR-STORE-1). When the mic is on, audio.wav is a dual-channel split (left = mic, right = loopback; ADR-0005 Phase 3.5, FR-SPK), so attribution is per-stream: sherpa clusters the right channel only → Speaker N, and "You" comes from left-channel voice activity. Blind whole-signal clustering
   • voiceprint is the mic-off (summed) fallback. If "Record this meeting" is off, delete the working WAV after the transcript is finalized; if on, keep it as audio.wav (ADR-0009, FR-REC-1/4).
7. If an LLM provider is configured: llm builds a prompt (transcript + metadata + template), streams a summary/decisions/action-items into the summary panel (FR-LLM-2/4).
8. Action items are parsed and presented for confirmation; confirmed ones persist and may raise local OS reminders (FR-LLM-3, FR-CAL-5).
9. If sync is configured and enabled, storage finalization enqueues the selected artifacts (transcript/notes/summary, and the recording if retained) onto the sync queue; sync uploads them to each target with retry/backoff and reports progress via events (FR-SYNC-3/5). Nothing is uploaded if no target is enabled (FR-SYNC-1).

Data flow — meeting → code (agent handoff, ADR-0011)

The "customer asks for a feature, get started right away" path, pull model:

1. WA distills the relevant part of the transcript into a feature brief (via the configured llm): problem, desired outcome, acceptance criteria, target repo/context, source meeting (FR-MCP-4). It is stored with the meeting and offered as a tool result.
2. The user has the mcp server enabled (off by default) with a chosen exposure scope (FR-MCP-1/3).
3. In their own agent (Claude Code, Codex, Copilot, OpenCode), the developer asks it to act; the agent connects to WA's loopback MCP server and calls get_feature_brief / get_action_items to pull structured context (FR-MCP-2), then scaffolds/branches/PRs in their repo.
4. WA logs what was read and reminds the user that the agent may forward served data to its own cloud (FR-MCP-5). WA itself sends nothing (FR-MCP-7).
5. (Layer 3, later) Alternatively, WA pushes: an AgentRunner spawns a CLI agent headless against a repo, or WA opens a GitHub issue (optionally assigned to Copilot's cloud agent) from the brief (FR-AGENT-1/2).

Threading & concurrency model

• UI/IPC thread: Tauri commands return quickly. Nothing that runs for the length of a meeting blocks a command — long work is spawned and reports via events.
• Audio thread: a dedicated thread runs the WASAPI capture loop, waiting on the WASAPI event handle (short timeout so it can also notice a stop/pause request) rather than busy-polling — see 07-research-findings.md for why the earlier "polling only" note was corrected. It only does capture + disk write + ring-buffer push; no inference.
• Inference worker(s): transcription runs on its own worker, pulling audio windows from the ring buffer; diarization runs as a discrete post-stop job. Backends are constructed once per session from the hardware ranking.
• Async runtime: tokio for storage (sqlx), llm (reqwest streaming), and orchestration.
• State: a tauri::State<AppState> holds Arc-wrapped service handles and a RecordingSession guarded so start/stop/pause transitions are atomic.
• Backpressure: the ring buffer is bounded; if transcription falls behind (CPU-only, large model), it drops to interim/skip strategy for the live view while the on-disk audio remains complete for an accurate post-meeting pass.

Idle behavior (NFR-RES-1)

When not recording, no audio thread, inference worker, or polling timer runs. Background indexing (FTS, retention) is event-driven or scheduled with idle-priority, and is reduced further under the "Low overhead" preset (NFR-RES-3). Models are unloaded from memory when idle and lazy-loaded on the next recording (NFR-PERF-4).

Error handling & resilience

• Each module defines a thiserror error enum; commands surface typed errors to the frontend.
• Hardware/backend failures degrade one tier at a time down to CPU (FR-HW-4, NFR-REL-2) and surface a non-blocking notice; they never abort a recording.
• audio is the source of truth: a crash leaves a recoverable audio.wav; on next launch a reconcile scan finds meetings with audio but no finalized transcript and offers recovery (FR-REL-1, NFR-REL-3).
• PST parse failures and missing LLM providers are non-fatal: core capture/transcription/notes keep working.

Security boundaries (FR-SEC-1, NFR-SEC-*)

• WA opens sockets only to destinations the user configured: the LLM endpoint (local Ollama or, if chosen, a hosted AI provider — ADR-0011), enabled sync target hosts, a configured task tracker (Layer 3), and model-download hosts during an explicit download. These form an allowlist derived from settings; a privacy self-check (and a CI network test) asserts no connection to any off-allowlist host (NFR-SEC-3). With nothing configured, WA makes no content egress at all.
• MCP server (ADR-0011) is inbound, not egress. WA listens on loopback for the user's agents; it opens no outbound socket and adds nothing to the allowlist (FR-MCP-7/NFR-SEC-5). It is off by default, token-gated, scope-limited (never serves recordings unless allowed), and audited. The data-leaves-device disclosure for this path is "the agent you connected may forward served data to its own provider" — outside WA's control, like the remote-LLM caveat (ADR-0007).
• Sync is off by default and is an explicit user export (ADR-0010): credentials live in the OS credential store (never in settings/DB, FR-SYNC-6/NFR-SEC-4); TLS is required (plaintext only via explicit per-target LAN opt-in, FR-SYNC-7); third-party targets are clearly labeled (FR-SYNC-8).
• The WebView CSP also restricts connect-src to localhost, so the frontend itself cannot reach the network — all egress goes through the Rust sync/llm services where the allowlist is enforced.
• Data is stored under the user profile honoring ACLs; no admin rights required; optional at-rest encryption in Phase 8, optionally applied before upload so a sync target holds only ciphertext (FR-SYNC-10).

Portability seam (NFR-MNT-3)

Windows-specific code (WASAPI, DXGI enumeration, toast notifications, PST) sits behind the service traits. A future macOS/Linux port implements the same traits with platform equivalents (CoreAudio/PulseAudio capture, Metal/Vulkan backends, etc.) without changing callers or the frontend.