WhispAssist/docs/02-architecture.md

02 — Architecture

Component map

┌─────────────────────────────────────────────────────────────────────┐
│  WebView2 window                                                      │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │  Svelte frontend (TypeScript)                                  │  │
│  │   views: MeetingsList · TranscriptNotes · SummaryPanel ·       │  │
│  │          Settings · SpeakerReview · CalendarView               │  │
│  │   stores: recording, meeting, transcript, hardware, settings   │  │
│  └───────────────▲───────────────────────────┬───────────────────┘  │
│        Tauri events (push)        Tauri commands (request/response)   │
└──────────────────┼───────────────────────────┼──────────────────────┘
                   │                            │
┌──────────────────┴───────────────────────────▼──────────────────────┐
│  Rust core (Tauri)                                                    │
│                                                                       │
│   commands.rs  ──orchestrates──►  AppState { service handles }        │
│                                                                       │
│   ┌─────────┐  ┌──────────┐  ┌────────────┐  ┌──────────────┐         │
│   │ audio   │  │ hardware │  │transcription│  │ diarization │         │
│   │ (WASAPI)│  │(detect)  │  │(whisper/onnx)│ │(sherpa-onnx)│         │
│   └────┬────┘  └────┬─────┘  └─────┬──────┘  └──────┬───────┘         │
│        │            │              │                │                 │
│   ┌────▼────────────▼──────────────▼────────────────▼───────┐         │
│   │ storage (SQLite + files)   notes (markdown)             │         │
│   └────┬───────────────────────────────────┬───────────────┘         │
│        │                                    │                         │
│   ┌────▼─────┐   ┌──────────────┐  ┌──────────┐  ┌─────▼──────┐       │
│   │   llm    │   │     sync     │  │   mcp    │  │  calendar  │       │
│   │(Ollama / │   │(WebDAV/OAuth)│  │ (server) │  │ (.pst/Graph)│      │
│   │ cloud)   │   └──────┬───────┘  └────▲─────┘  └────────────┘       │
│   └────┬─────┘          │               │ loopback (INBOUND)          │
└────────┼────────────────┼───────────────┼──────────────────────────────┘
         │ HTTP/HTTPS     │ HTTPS         │ MCP over stdio / http://127.0.0.1/mcp
         │ (local or      │ (targets,     │ (off by default; token; NOT egress)
         │  cloud AI)     │  off default) │
   ┌─────▼─────┐    ┌──────▼───────────┐  │   ┌──────────────────────────────┐
   │ Ollama /  │    │ Nextcloud·Seafile│  └───│ Claude · Codex · Copilot ·   │
   │ Anthropic │    │ ·Cloudreve·…     │      │ OpenCode (user's own agents) │
   │ /OpenAI   │    │ |OneDrive·Dropbox│      │ — they may forward to their  │
   └───────────┘    └──────────────────┘      │ own cloud (outside WA)       │
                                              └──────────────────────────────┘

The mcp arrow points inward: agents connect to WA, not the reverse. WA opens no socket for it.

Each Rust service lives in its own module under src-tauri/src/<service>/ and exposes a trait plus a default implementation. Callers depend on the trait (NFR-MNT-1/2). The frontend never touches files, DB, or network directly — only Tauri commands/events (04-api-contracts.md).

Services (responsibilities & key dependencies)

Service	Responsibility	Primary crate(s)
audio	WASAPI loopback capture (+ optional microphone, mixed into the transcript stream, FR-CAP-7); PCM ring buffer; write WAV to disk; pause/resume	wasapi, hound
hardware	Enumerate NPU/GPU/CPU; rank backends; report capabilities	ort, DXGI via windows
transcription	Load model on a backend; stream segments (whisper.cpp) or NPU (ONNX)	whisper-rs, ort
diarization	Post-process audio → speaker spans; align to segments; merge	sherpa-onnx (FFI)
storage	SQLite index + file layout; CRUD; FTS search; retention; recovery scan	sqlx, serde_json
notes	Assemble Markdown from segments+speakers+summary; render/export	pulldown-cmark
llm	Talk to Ollama/custom endpoint; build prompts; stream; parse action items	reqwest, serde
calendar	Parse .pst events/attendees; (future) Graph/ICS sources	outlook-pst
sync	Upload artifacts to user-configured targets; durable queue + retry; WebDAV + OAuth providers; OS-keychain creds	reqwest, reqwest_dav, keyring
mcp	Host a local (loopback) MCP server exposing meeting context as tools; build feature briefs; scope + token + audit	rmcp, axum/hyper

The audio service also owns recording retention (ADR-0009): the working WAV is kept as the permanent audio.wav only when "Record this meeting" is on; otherwise it is deleted on finalize.

The llm service is extended (ADR-0011, Layer 1) with optional hosted providers (Anthropic Messages, OpenAI-compatible) behind the same LlmProvider trait, in addition to local Ollama. An optional agent module (Layer 3, later) spawns local coding-agent CLIs and/or creates task-tracker issues from feature briefs.

Data flow — a typical meeting

1. User selects/creates a meeting (optionally from a calendar event) → start_recording.
2. audio begins WASAPI loopback capture, writing PCM to audio.wav continuously (FR-CAP-2), and pushes frames into an in-memory ring buffer.
3. transcription consumes windowed audio from the ring buffer on the selected hardware backend and emits interim TranscriptSegments → transcript_segment events to the UI (FR-TRX-2).
4. Provisional speaker turns (from cheap segmentation) tag segments live; the user may name a speaker, applied immediately to past/future segments (FR-SPK-2).
5. The notes view renders Markdown that updates as segments arrive (FR-NOTE-1, FR-NOTE-6).
6. On stop_recording: finalize the WAV; run diarization, align speaker IDs to segments, apply name mappings; persist transcript JSON + metadata via storage (FR-STORE-1). When the mic is on, attribution is per-stream (ADR-0005 Phase 3, FR-SPK): the mic-activity timeline that audio recorded (in audio.wav frame time, where the mic is folded into the loopback writer) is the user's "You" spans directly, and sherpa clusters only the far side — the recording diarized with those ranges masked out. Blind whole-signal clustering + voiceprint is the mic-off fallback. If "Record this meeting" is off, delete the working WAV after the transcript is finalized; if on, keep it as audio.wav (ADR-0009, FR-REC-1/4).
7. If an LLM provider is configured: llm builds a prompt (transcript + metadata + template), streams a summary/decisions/action-items into the summary panel (FR-LLM-2/4).
8. Action items are parsed and presented for confirmation; confirmed ones persist and may raise local OS reminders (FR-LLM-3, FR-CAL-5).
9. If sync is configured and enabled, storage finalization enqueues the selected artifacts (transcript/notes/summary, and the recording if retained) onto the sync queue; sync uploads them to each target with retry/backoff and reports progress via events (FR-SYNC-3/5). Nothing is uploaded if no target is enabled (FR-SYNC-1).

Data flow — meeting → code (agent handoff, ADR-0011)

The "customer asks for a feature, get started right away" path, pull model:

1. WA distills the relevant part of the transcript into a feature brief (via the configured llm): problem, desired outcome, acceptance criteria, target repo/context, source meeting (FR-MCP-4). It is stored with the meeting and offered as a tool result.
2. The user has the mcp server enabled (off by default) with a chosen exposure scope (FR-MCP-1/3).
3. In their own agent (Claude Code, Codex, Copilot, OpenCode), the developer asks it to act; the agent connects to WA's loopback MCP server and calls get_feature_brief / get_action_items to pull structured context (FR-MCP-2), then scaffolds/branches/PRs in their repo.
4. WA logs what was read and reminds the user that the agent may forward served data to its own cloud (FR-MCP-5). WA itself sends nothing (FR-MCP-7).
5. (Layer 3, later) Alternatively, WA pushes: an AgentRunner spawns a CLI agent headless against a repo, or WA opens a GitHub issue (optionally assigned to Copilot's cloud agent) from the brief (FR-AGENT-1/2).

Threading & concurrency model

• UI/IPC thread: Tauri commands return quickly. Nothing that runs for the length of a meeting blocks a command — long work is spawned and reports via events.
• Audio thread: a dedicated thread runs the WASAPI capture loop, waiting on the WASAPI event handle (short timeout so it can also notice a stop/pause request) rather than busy-polling — see 07-research-findings.md for why the earlier "polling only" note was corrected. It only does capture + disk write + ring-buffer push; no inference.
• Inference worker(s): transcription runs on its own worker, pulling audio windows from the ring buffer; diarization runs as a discrete post-stop job. Backends are constructed once per session from the hardware ranking.
• Async runtime: tokio for storage (sqlx), llm (reqwest streaming), and orchestration.
• State: a tauri::State<AppState> holds Arc-wrapped service handles and a RecordingSession guarded so start/stop/pause transitions are atomic.
• Backpressure: the ring buffer is bounded; if transcription falls behind (CPU-only, large model), it drops to interim/skip strategy for the live view while the on-disk audio remains complete for an accurate post-meeting pass.

Idle behavior (NFR-RES-1)

When not recording, no audio thread, inference worker, or polling timer runs. Background indexing (FTS, retention) is event-driven or scheduled with idle-priority, and is reduced further under the "Low overhead" preset (NFR-RES-3). Models are unloaded from memory when idle and lazy-loaded on the next recording (NFR-PERF-4).

Error handling & resilience

• Each module defines a thiserror error enum; commands surface typed errors to the frontend.
• Hardware/backend failures degrade one tier at a time down to CPU (FR-HW-4, NFR-REL-2) and surface a non-blocking notice; they never abort a recording.
• audio is the source of truth: a crash leaves a recoverable audio.wav; on next launch a reconcile scan finds meetings with audio but no finalized transcript and offers recovery (FR-REL-1, NFR-REL-3).
• PST parse failures and missing LLM providers are non-fatal: core capture/transcription/notes keep working.

Security boundaries (FR-SEC-1, NFR-SEC-*)

• WA opens sockets only to destinations the user configured: the LLM endpoint (local Ollama or, if chosen, a hosted AI provider — ADR-0011), enabled sync target hosts, a configured task tracker (Layer 3), and model-download hosts during an explicit download. These form an allowlist derived from settings; a privacy self-check (and a CI network test) asserts no connection to any off-allowlist host (NFR-SEC-3). With nothing configured, WA makes no content egress at all.
• MCP server (ADR-0011) is inbound, not egress. WA listens on loopback for the user's agents; it opens no outbound socket and adds nothing to the allowlist (FR-MCP-7/NFR-SEC-5). It is off by default, token-gated, scope-limited (never serves recordings unless allowed), and audited. The data-leaves-device disclosure for this path is "the agent you connected may forward served data to its own provider" — outside WA's control, like the remote-LLM caveat (ADR-0007).
• Sync is off by default and is an explicit user export (ADR-0010): credentials live in the OS credential store (never in settings/DB, FR-SYNC-6/NFR-SEC-4); TLS is required (plaintext only via explicit per-target LAN opt-in, FR-SYNC-7); third-party targets are clearly labeled (FR-SYNC-8).
• The WebView CSP also restricts connect-src to localhost, so the frontend itself cannot reach the network — all egress goes through the Rust sync/llm services where the allowlist is enforced.
• Data is stored under the user profile honoring ACLs; no admin rights required; optional at-rest encryption in Phase 8, optionally applied before upload so a sync target holds only ciphertext (FR-SYNC-10).

Portability seam (NFR-MNT-3)

Windows-specific code (WASAPI, DXGI enumeration, toast notifications, PST) sits behind the service traits. A future macOS/Linux port implements the same traits with platform equivalents (CoreAudio/PulseAudio capture, Metal/Vulkan backends, etc.) without changing callers or the frontend.