184 lines
14 KiB
Markdown
184 lines
14 KiB
Markdown
# 02 — Architecture
|
|
|
|
## Component map
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ WebView2 window │
|
|
│ ┌───────────────────────────────────────────────────────────────┐ │
|
|
│ │ Svelte frontend (TypeScript) │ │
|
|
│ │ views: MeetingsList · TranscriptNotes · SummaryPanel · │ │
|
|
│ │ Settings · SpeakerReview · CalendarView │ │
|
|
│ │ stores: recording, meeting, transcript, hardware, settings │ │
|
|
│ └───────────────▲───────────────────────────┬───────────────────┘ │
|
|
│ Tauri events (push) Tauri commands (request/response) │
|
|
└──────────────────┼───────────────────────────┼──────────────────────┘
|
|
│ │
|
|
┌──────────────────┴───────────────────────────▼──────────────────────┐
|
|
│ Rust core (Tauri) │
|
|
│ │
|
|
│ commands.rs ──orchestrates──► AppState { service handles } │
|
|
│ │
|
|
│ ┌─────────┐ ┌──────────┐ ┌────────────┐ ┌──────────────┐ │
|
|
│ │ audio │ │ hardware │ │transcription│ │ diarization │ │
|
|
│ │ (WASAPI)│ │(detect) │ │(whisper/onnx)│ │(sherpa-onnx)│ │
|
|
│ └────┬────┘ └────┬─────┘ └─────┬──────┘ └──────┬───────┘ │
|
|
│ │ │ │ │ │
|
|
│ ┌────▼────────────▼──────────────▼────────────────▼───────┐ │
|
|
│ │ storage (SQLite + files) notes (markdown) │ │
|
|
│ └────┬───────────────────────────────────┬───────────────┘ │
|
|
│ │ │ │
|
|
│ ┌────▼─────┐ ┌──────────────┐ ┌──────────┐ ┌─────▼──────┐ │
|
|
│ │ llm │ │ sync │ │ mcp │ │ calendar │ │
|
|
│ │(Ollama / │ │(WebDAV/OAuth)│ │ (server) │ │ (.pst/Graph)│ │
|
|
│ │ cloud) │ └──────┬───────┘ └────▲─────┘ └────────────┘ │
|
|
│ └────┬─────┘ │ │ loopback (INBOUND) │
|
|
└────────┼────────────────┼───────────────┼──────────────────────────────┘
|
|
│ HTTP/HTTPS │ HTTPS │ MCP over stdio / http://127.0.0.1/mcp
|
|
│ (local or │ (targets, │ (off by default; token; NOT egress)
|
|
│ cloud AI) │ off default) │
|
|
┌─────▼─────┐ ┌──────▼───────────┐ │ ┌──────────────────────────────┐
|
|
│ Ollama / │ │ Nextcloud·Seafile│ └───│ Claude · Codex · Copilot · │
|
|
│ Anthropic │ │ ·Cloudreve·… │ │ OpenCode (user's own agents) │
|
|
│ /OpenAI │ │ |OneDrive·Dropbox│ │ — they may forward to their │
|
|
└───────────┘ └──────────────────┘ │ own cloud (outside WA) │
|
|
└──────────────────────────────┘
|
|
```
|
|
|
|
The `mcp` arrow points **inward**: agents connect to WA, not the reverse. WA opens no socket for it.
|
|
|
|
Each Rust service lives in its own module under `src-tauri/src/<service>/` and exposes a **trait**
|
|
plus a default implementation. Callers depend on the trait (NFR-MNT-1/2). The frontend never
|
|
touches files, DB, or network directly — only Tauri commands/events (`04-api-contracts.md`).
|
|
|
|
## Services (responsibilities & key dependencies)
|
|
|
|
| Service | Responsibility | Primary crate(s) |
|
|
|---|---|---|
|
|
| `audio` | WASAPI loopback capture (+ optional microphone, mixed into the transcript stream, FR-CAP-7); PCM ring buffer; write WAV to disk; pause/resume | `wasapi`, `hound` |
|
|
| `hardware` | Enumerate NPU/GPU/CPU; rank backends; report capabilities | `ort`, DXGI via `windows` |
|
|
| `transcription` | Load model on a backend; stream segments (whisper.cpp) or NPU (ONNX) | `whisper-rs`, `ort` |
|
|
| `diarization` | Post-process audio → speaker spans; align to segments; merge | `sherpa-onnx` (FFI) |
|
|
| `storage` | SQLite index + file layout; CRUD; FTS search; retention; recovery scan | `sqlx`, `serde_json` |
|
|
| `notes` | Assemble Markdown from segments+speakers+summary; render/export | `pulldown-cmark` |
|
|
| `llm` | Talk to Ollama/custom endpoint; build prompts; stream; parse action items | `reqwest`, `serde` |
|
|
| `calendar` | Parse `.pst` events/attendees; (future) Graph/ICS sources | `outlook-pst` |
|
|
| `sync` | Upload artifacts to user-configured targets; durable queue + retry; WebDAV + OAuth providers; OS-keychain creds | `reqwest`, `reqwest_dav`, `keyring` |
|
|
| `mcp` | Host a local (loopback) MCP server exposing meeting context as tools; build feature briefs; scope + token + audit | `rmcp`, `axum`/`hyper` |
|
|
|
|
The `audio` service also owns **recording retention** (ADR-0009): the working WAV is kept as the
|
|
permanent `audio.wav` only when "Record this meeting" is on; otherwise it is deleted on finalize.
|
|
|
|
The `llm` service is extended (ADR-0011, Layer 1) with optional **hosted** providers (Anthropic
|
|
Messages, OpenAI-compatible) behind the same `LlmProvider` trait, in addition to local Ollama. An
|
|
optional `agent` module (Layer 3, later) spawns local coding-agent CLIs and/or creates task-tracker
|
|
issues from feature briefs.
|
|
|
|
## Data flow — a typical meeting
|
|
|
|
1. User selects/creates a meeting (optionally from a calendar event) → `start_recording`.
|
|
2. `audio` begins WASAPI loopback capture, writing PCM to `audio.wav` **continuously** (FR-CAP-2),
|
|
and pushes frames into an in-memory ring buffer.
|
|
3. `transcription` consumes windowed audio from the ring buffer on the selected `hardware` backend
|
|
and emits interim `TranscriptSegment`s → `transcript_segment` events to the UI (FR-TRX-2).
|
|
4. Provisional speaker turns (from cheap segmentation) tag segments live; the user may name a
|
|
speaker, applied immediately to past/future segments (FR-SPK-2).
|
|
5. The notes view renders Markdown that updates as segments arrive (FR-NOTE-1, FR-NOTE-6).
|
|
6. On `stop_recording`: finalize the WAV; run `diarization`, align speaker IDs to segments,
|
|
apply name mappings; persist transcript JSON + metadata via `storage` (FR-STORE-1). When the mic
|
|
is on, `audio.wav` is a **dual-channel split** (left = mic, right = loopback; ADR-0005 Phase 3.5,
|
|
FR-SPK), so attribution is **per-stream**: sherpa clusters the **right channel only** →
|
|
`Speaker N`, and "You" comes from **left-channel** voice activity. Blind whole-signal clustering
|
|
+ voiceprint is the mic-off (`summed`) fallback. If "Record this meeting" is **off**, delete the
|
|
working WAV **after** the transcript is finalized; if **on**, keep it as `audio.wav` (ADR-0009,
|
|
FR-REC-1/4).
|
|
7. If an LLM provider is configured: `llm` builds a prompt (transcript + metadata + template),
|
|
streams a summary/decisions/action-items into the summary panel (FR-LLM-2/4).
|
|
8. Action items are parsed and presented for confirmation; confirmed ones persist and may raise
|
|
local OS reminders (FR-LLM-3, FR-CAL-5).
|
|
9. If sync is configured and enabled, `storage` finalization enqueues the selected artifacts
|
|
(transcript/notes/summary, and the recording if retained) onto the `sync` queue; `sync` uploads
|
|
them to each target with retry/backoff and reports progress via events (FR-SYNC-3/5). Nothing is
|
|
uploaded if no target is enabled (FR-SYNC-1).
|
|
|
|
## Data flow — meeting → code (agent handoff, ADR-0011)
|
|
|
|
The "customer asks for a feature, get started right away" path, **pull** model:
|
|
|
|
1. WA distills the relevant part of the transcript into a **feature brief** (via the configured
|
|
`llm`): problem, desired outcome, acceptance criteria, target repo/context, source meeting
|
|
(FR-MCP-4). It is stored with the meeting and offered as a tool result.
|
|
2. The user has the `mcp` server enabled (off by default) with a chosen exposure scope (FR-MCP-1/3).
|
|
3. In **their own** agent (Claude Code, Codex, Copilot, OpenCode), the developer asks it to act;
|
|
the agent connects to WA's loopback MCP server and calls `get_feature_brief` / `get_action_items`
|
|
to pull structured context (FR-MCP-2), then scaffolds/branches/PRs in their repo.
|
|
4. WA logs what was read and reminds the user that the agent may forward served data to its own cloud
|
|
(FR-MCP-5). WA itself sends nothing (FR-MCP-7).
|
|
5. *(Layer 3, later)* Alternatively, WA **pushes**: an `AgentRunner` spawns a CLI agent headless
|
|
against a repo, or WA opens a GitHub issue (optionally assigned to Copilot's cloud agent) from the
|
|
brief (FR-AGENT-1/2).
|
|
|
|
## Threading & concurrency model
|
|
|
|
- **UI/IPC thread:** Tauri commands return quickly. Nothing that runs for the length of a meeting
|
|
blocks a command — long work is spawned and reports via **events**.
|
|
- **Audio thread:** a dedicated thread runs the WASAPI capture loop, waiting on the WASAPI event
|
|
handle (short timeout so it can also notice a stop/pause request) rather than busy-polling —
|
|
see `07-research-findings.md` for why the earlier "polling only" note was corrected. It only
|
|
does capture + disk write + ring-buffer push; no inference.
|
|
- **Inference worker(s):** transcription runs on its own worker, pulling audio windows from the
|
|
ring buffer; diarization runs as a discrete post-stop job. Backends are constructed once per
|
|
session from the `hardware` ranking.
|
|
- **Async runtime:** `tokio` for `storage` (sqlx), `llm` (reqwest streaming), and orchestration.
|
|
- **State:** a `tauri::State<AppState>` holds `Arc`-wrapped service handles and a `RecordingSession`
|
|
guarded so start/stop/pause transitions are atomic.
|
|
- **Backpressure:** the ring buffer is bounded; if transcription falls behind (CPU-only, large
|
|
model), it drops to interim/skip strategy for the live view while the on-disk audio remains
|
|
complete for an accurate post-meeting pass.
|
|
|
|
## Idle behavior (NFR-RES-1)
|
|
|
|
When not recording, no audio thread, inference worker, or polling timer runs. Background indexing
|
|
(FTS, retention) is event-driven or scheduled with idle-priority, and is reduced further under the
|
|
"Low overhead" preset (NFR-RES-3). Models are unloaded from memory when idle and lazy-loaded on the
|
|
next recording (NFR-PERF-4).
|
|
|
|
## Error handling & resilience
|
|
|
|
- Each module defines a `thiserror` error enum; commands surface typed errors to the frontend.
|
|
- Hardware/backend failures degrade one tier at a time down to CPU (FR-HW-4, NFR-REL-2) and surface
|
|
a non-blocking notice; they never abort a recording.
|
|
- `audio` is the source of truth: a crash leaves a recoverable `audio.wav`; on next launch a
|
|
reconcile scan finds meetings with audio but no finalized transcript and offers recovery
|
|
(FR-REL-1, NFR-REL-3).
|
|
- PST parse failures and missing LLM providers are non-fatal: core capture/transcription/notes
|
|
keep working.
|
|
|
|
## Security boundaries (FR-SEC-1, NFR-SEC-*)
|
|
|
|
- WA opens sockets only to destinations the user configured: the LLM endpoint (**local** Ollama or,
|
|
if chosen, a **hosted** AI provider — ADR-0011), **enabled sync target hosts**, a configured task
|
|
tracker (Layer 3), and model-download hosts during an explicit download. These form an **allowlist
|
|
derived from settings**; a privacy self-check (and a CI network test) asserts no connection to any
|
|
off-allowlist host (NFR-SEC-3). With nothing configured, WA makes no content egress at all.
|
|
- **MCP server (ADR-0011) is inbound, not egress.** WA *listens* on loopback for the user's agents;
|
|
it opens no outbound socket and adds nothing to the allowlist (FR-MCP-7/NFR-SEC-5). It is off by
|
|
default, token-gated, scope-limited (never serves recordings unless allowed), and audited. The
|
|
data-leaves-device disclosure for this path is "the agent you connected may forward served data to
|
|
its own provider" — outside WA's control, like the remote-LLM caveat (ADR-0007).
|
|
- **Sync** is off by default and is an explicit user export (ADR-0010): credentials live in the OS
|
|
credential store (never in settings/DB, FR-SYNC-6/NFR-SEC-4); TLS is required (plaintext only via
|
|
explicit per-target LAN opt-in, FR-SYNC-7); third-party targets are clearly labeled (FR-SYNC-8).
|
|
- The WebView CSP also restricts `connect-src` to localhost, so the frontend itself cannot reach the
|
|
network — all egress goes through the Rust `sync`/`llm` services where the allowlist is enforced.
|
|
- Data is stored under the user profile honoring ACLs; no admin rights required; optional at-rest
|
|
encryption in Phase 8, optionally applied **before** upload so a sync target holds only ciphertext
|
|
(FR-SYNC-10).
|
|
|
|
## Portability seam (NFR-MNT-3)
|
|
|
|
Windows-specific code (WASAPI, DXGI enumeration, toast notifications, PST) sits behind the service
|
|
traits. A future macOS/Linux port implements the same traits with platform equivalents
|
|
(CoreAudio/PulseAudio capture, Metal/Vulkan backends, etc.) without changing callers or the
|
|
frontend.
|