Add all scaffolding, design, and decision documentation AKA first commit
This commit is contained in:
iamdoubz committed
1 parent
b19e01c171
commit
aae22a0e23
63 files changed
+4690
No files matched your search
@@ -0,0 +1,68 @@
|
||||
# 00 — Overview
|
||||
|
||||
## Vision
|
||||
|
||||
WhispAssist (WA) is a privacy-first, Windows-native meeting assistant that runs **entirely on the
|
||||
local machine**. It captures system audio, transcribes it with Whisper-class models using
|
||||
on-device acceleration (NPU → GPU → CPU), structures the result into Markdown notes, and
|
||||
optionally augments those notes with a locally hosted LLM. It combines **Meetily's** local-only
|
||||
architecture with **Granola's** calendar-aware, role-aware note workflows, and leans harder into
|
||||
Windows hardware acceleration and Outlook `.pst` integration than either.
|
||||
|
||||
## Product goals
|
||||
|
||||
1. High-quality, real-time or near-real-time transcription, fully on-device.
|
||||
2. Strict local-only data handling — recordings and transcripts never leave the machine except by
|
||||
explicit user export.
|
||||
3. Intelligent note-taking and summarization via a locally configured LLM endpoint.
|
||||
4. Calendar and local Outlook `.pst` integration for meeting context and speaker mapping.
|
||||
5. A modern, responsive desktop experience: light/dark themes, fast startup, low-latency
|
||||
interaction, and near-zero idle footprint.
|
||||
|
||||
## Non-goals (for v1)
|
||||
|
||||
- Cloud sync, multi-device, or team sharing of recordings/transcripts.
|
||||
- Joining meetings as a bot.
|
||||
- Mobile apps (Windows-on-ARM is a "later", not a v1 target).
|
||||
- Cloud AI for transcription or summarization (a user *may* point the LLM at a remote
|
||||
OpenAI-compatible endpoint, but that is off by default and clearly labeled).
|
||||
|
||||
## How to read this plan
|
||||
|
||||
| Doc | Answers |
|
||||
|---|---|
|
||||
| `00-overview.md` (this) | What and why, at a glance; glossary |
|
||||
| `01-requirements.md` | Exactly what it must do (FR) and how well (NFR), with IDs |
|
||||
| `02-architecture.md` | How the pieces fit; data flow; threading |
|
||||
| `03-data-model.md` | Database schema, file layout, transcript JSON |
|
||||
| `04-api-contracts.md` | Tauri commands/events + internal Rust traits |
|
||||
| `05-roadmap.md` | Build order: 8 phases, tasks, acceptance criteria |
|
||||
| `06-test-strategy.md` | How each phase is proven correct |
|
||||
| `07-research-findings.md` | Evidence the stack works (with sources) |
|
||||
| `adr/*` | The big decisions and their trade-offs |
|
||||
|
||||
Requirement IDs (e.g. `FR-CAP-1`) are referenced throughout the roadmap, tests, and code commits
|
||||
so any line of work traces back to a requirement.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **System / loopback audio** — the audio the OS plays back (what you hear), captured without a
|
||||
microphone or meeting bot, via WASAPI loopback.
|
||||
- **Backend** — the hardware execution path for inference (NPU, NVIDIA, AMD, Intel, CPU).
|
||||
- **Segment** — a timestamped chunk of transcript text with an associated speaker ID.
|
||||
- **Speaker ID** — an internal label (`S1`, `S2`, …) assigned by diarization, later mapped to a
|
||||
human name or calendar participant.
|
||||
- **Diarization** — partitioning audio by "who spoke when".
|
||||
- **Real-time factor (RTF)** — processing time ÷ audio duration; < 1.0 means faster than real time.
|
||||
- **Provider** — a configured local LLM endpoint (Ollama by default).
|
||||
- **Meeting** — the top-level record: audio + transcript + notes + metadata + speakers.
|
||||
|
||||
## One-paragraph architecture
|
||||
|
||||
A **Svelte** frontend in a **WebView2** window talks over Tauri IPC to a **Rust core** split into
|
||||
independent services — `audio` (WASAPI capture), `hardware` (backend detection), `transcription`
|
||||
(whisper.cpp / ONNX), `diarization` (sherpa-onnx), `storage` (SQLite + files), `llm` (Ollama
|
||||
HTTP), `calendar` (`.pst`/future Graph), and `notes`. Capture writes audio to disk first (source
|
||||
of truth), streams it to transcription, which emits segments to the UI live; on stop, diarization
|
||||
and (optionally) the LLM refine and summarize, and everything is persisted locally. See
|
||||
`02-architecture.md`.
|
||||
@@ -0,0 +1,228 @@
|
||||
# 01 — Requirements
|
||||
|
||||
Each requirement has a stable ID used across the roadmap, tests, and commits. Priority:
|
||||
**M** = must (v1), **S** = should, **C** = could (later). Each FR notes the phase that delivers it
|
||||
(see `05-roadmap.md`).
|
||||
|
||||
## Functional requirements
|
||||
|
||||
### Audio capture (CAP)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-CAP-1 | M | 1 | Capture system (loopback) audio on Windows via WASAPI without a meeting bot or microphone. |
|
||||
| FR-CAP-2 | M | 1 | Write captured audio to disk continuously during recording (audio = source of truth). |
|
||||
| FR-CAP-3 | M | 1 | Start, stop, pause, and resume recording from the UI. |
|
||||
| FR-CAP-4 | M | 1 | Show an unambiguous "recording active" indicator (in-app banner + tray icon). |
|
||||
| FR-CAP-5 | S | 7 | Render a live input waveform / level meter while recording. |
|
||||
| FR-CAP-6 | S | 7 | Handle audio device changes mid-recording without losing the session. |
|
||||
|
||||
### Recording retention & consent (REC) — see ADR-0009
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-REC-1 | M | 1 | "Record this meeting" is **off by default** (global default + per-meeting toggle). When on, the meeting's audio is retained and saved as a `.wav` file; when off, working audio is deleted on finalize and only the transcript/notes persist. |
|
||||
| FR-REC-2 | M | 1 | Before retaining a recording for the first time (and shown near the toggle thereafter), display a consent notice: recording without participants' consent may be illegal in some regions; advise checking local laws. Require a one-time acknowledgment; store it. This is a caution, not legal advice. |
|
||||
| FR-REC-3 | M | 1 | When retention is on, the UI indicates the meeting is being **saved** (in addition to the "recording active" indicator, FR-CAP-4). |
|
||||
| FR-REC-4 | M | 2 | Deleting working audio on finalize happens only **after** the transcript is successfully finalized; never race with crash recovery (audio stays source of truth until then). |
|
||||
|
||||
### Hardware acceleration (HW)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-HW-1 | M | 3 | Detect available compute backends at startup and on demand: NPU, NVIDIA, AMD, Intel GPU, CPU. |
|
||||
| FR-HW-2 | S | 3 | Select the best backend by the order NPU→NVIDIA→AMD→Intel→CPU; allow a manual override. |
|
||||
| FR-HW-3 | M | 3 | Display the active backend, model size, and estimated RTF in Settings and the recording bar. |
|
||||
| FR-HW-4 | M | 3 | Fall back gracefully to a lower tier (down to CPU) when a backend is unavailable or fails. |
|
||||
|
||||
### Transcription (TRX)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-TRX-1 | M | 1 | Transcribe audio to timestamped text segments using a Whisper-class model (CPU baseline). |
|
||||
| FR-TRX-2 | M | 1 | Stream interim transcript segments to the UI in near real time during recording. |
|
||||
| FR-TRX-3 | S | 3 | Offer a post-meeting batch re-transcription with a larger model for higher accuracy. |
|
||||
| FR-TRX-4 | S | 8 | Support multiple languages via multilingual Whisper models; expose language selection/auto. |
|
||||
| FR-TRX-5 | M | 3 | Run accelerated inference (NPU/GPU) when available; identical segment output across backends. |
|
||||
|
||||
### Speaker diarization & naming (SPK)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-SPK-1 | M | 4 | Assign internal speaker IDs (S1, S2, …) to transcript segments via offline diarization. |
|
||||
| FR-SPK-2 | M | 4 | Let the user name a speaker during recording; apply to that speaker's past & future segments. |
|
||||
| FR-SPK-3 | M | 4 | Provide a post-meeting review to rename speakers and **merge** over-split speakers. |
|
||||
| FR-SPK-4 | S | 6 | Populate the naming UI with attendees from the linked calendar event, plus "add new name". |
|
||||
| FR-SPK-5 | M | 4 | Persist speaker→name mappings and apply them at render/export (never destructively rewrite). |
|
||||
|
||||
### Notes & Markdown (NOTE)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-NOTE-1 | M | 2 | Present meeting notes as rendered Markdown with speaker-tagged dialogue sections. |
|
||||
| FR-NOTE-2 | M | 2 | Edit notes (headings, bold/italic, lists, checkboxes) via Markdown syntax and/or a toolbar. |
|
||||
| FR-NOTE-3 | M | 2 | Export a meeting's notes as `.md`. |
|
||||
| FR-NOTE-4 | S | 8 | Export notes as PDF and/or Word via local conversion (no cloud). |
|
||||
| FR-NOTE-5 | C | 8 | User-defined note templates by meeting type (e.g. Sales Call, 1:1, Standup). |
|
||||
| FR-NOTE-6 | S | 2 | Show raw transcript and rendered notes side-by-side. |
|
||||
|
||||
### Local LLM (LLM)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-LLM-1 | M | 5 | Configure a local LLM provider (Ollama default, custom OpenAI-compatible endpoint, or off). |
|
||||
| FR-LLM-2 | M | 5 | Generate a summary, decisions, and action items from transcript + metadata (+ optional template). |
|
||||
| FR-LLM-3 | M | 5 | Parse LLM output into editable action items the user confirms before they become tasks. |
|
||||
| FR-LLM-4 | S | 5 | Stream LLM output token-by-token into the summary panel. |
|
||||
| FR-LLM-5 | S | 5 | Detect a missing provider and offer guided install + hardware-aware model suggestions. |
|
||||
| FR-LLM-6 | M | 5 | Validate the endpoint is local; if remote/proxying, show a clear "data leaves WA" banner. |
|
||||
|
||||
### External AI providers (AI) — see ADR-0011 (Layer 1), ADR-0007 update
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-AI-1 | S | 10 | Optionally configure a **hosted** summary provider behind the same `LlmProvider` model: Anthropic Messages API and an OpenAI-compatible client (OpenAI, OpenRouter, LM Studio, gateways). Off by default. |
|
||||
| FR-AI-2 | M | 10 | Hosted providers are explicit third-party egress: off by default, labeled "data leaves your device", host added to the settings-derived allowlist, API keys in the OS credential store (never settings/DB). |
|
||||
| FR-AI-3 | S | 10 | Per-use provider selection is allowed (e.g. local Ollama for one meeting, Claude for another); the active provider is shown wherever a summary is generated. |
|
||||
|
||||
### MCP server — WhispAssist as a tool source (MCP) — see ADR-0011 (Layer 2)
|
||||
|
||||
WA exposes its own meeting context to the user's coding agents (Claude, Codex, Copilot, OpenCode,
|
||||
…) via a local MCP server. This is the primary "get started right away" handoff (**pull** model).
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-MCP-1 | S | 10 | WA can expose a **local MCP server**, **off by default**, bound to loopback, requiring a token; explicit enable. |
|
||||
| FR-MCP-2 | M | 10 | Capabilities are exposed as MCP **tools** (not resources/prompts) for max client compatibility (Copilot cloud agent supports tools only): e.g. `list_recent_meetings`, `get_transcript`, `get_action_items`, `get_feature_brief`. |
|
||||
| FR-MCP-3 | M | 10 | **Scope control:** the user chooses which meetings/artifacts are exposed; recordings (`.wav`) are never exposed unless explicitly allowed. |
|
||||
| FR-MCP-4 | S | 10 | **Feature brief** primitive: distill a transcript (via the configured LLM) into a structured, agent-ready spec — problem, desired outcome, acceptance criteria, target repo/context, source meeting — served by `get_feature_brief`/`create_feature_brief`. |
|
||||
| FR-MCP-5 | M | 10 | **Disclosure + audit:** the UI states that a connected agent may send served data to its provider's cloud (outside WA's control); WA logs what each agent read. |
|
||||
| FR-MCP-6 | S | 10 | Support both MCP transports: **stdio** (thin adapter) and **Streamable HTTP** on loopback (`/mcp`). HTTP+SSE (deprecated) is not used. |
|
||||
| FR-MCP-7 | M | all | The MCP server is **inbound on loopback** and adds no WA egress; enabling it must not add any host to the egress allowlist. |
|
||||
|
||||
### Agent push & task-tracker handoff (AGENT) — see ADR-0011 (Layer 3, later)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-AGENT-1 | C | 10c | Optional `AgentRunner`: spawn a local coding-agent CLI headless (`claude -p`, `codex exec`, `opencode run`, `copilot`) against a user-chosen repo to produce a branch/PR from a feature brief. |
|
||||
| FR-AGENT-2 | C | 10c | Optional task-tracker handoff: create a GitHub issue from a confirmed action item / feature brief; optionally assign Copilot's cloud agent. Third-party egress: off by default, labeled, allowlisted. |
|
||||
|
||||
### Storage & persistence (STORE)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-STORE-1 | M | 2 | Persist audio, transcript, notes, and metadata locally (files + SQLite index). |
|
||||
| FR-STORE-2 | S | 2 | Configure base storage directory and a retention policy (max size and/or max age). |
|
||||
| FR-STORE-3 | M | 2 | Single-meeting export (audio + transcript + notes) to a chosen folder. |
|
||||
| FR-STORE-4 | S | 8 | Bulk export by date range or tag. |
|
||||
| FR-STORE-5 | M | 2 | List, open, and delete meetings. |
|
||||
|
||||
### Search & tagging (SEARCH)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-SEARCH-1 | S | 8 | Full-text search across transcripts and notes (SQLite FTS5). |
|
||||
| FR-SEARCH-2 | S | 8 | Tag meetings (project/client/topic) and filter the list by date, tag, or participant. |
|
||||
|
||||
### Calendar & Outlook .pst (CAL)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-CAL-1 | S | 6 | Read a user-selected `.pst` (with optional password) and extract events + attendees. |
|
||||
| FR-CAL-2 | S | 6 | Show upcoming/recent meetings; let the user attach a recording to an event. |
|
||||
| FR-CAL-3 | S | 6 | Display pre-meeting context: title, organizer, participants. |
|
||||
| FR-CAL-4 | C | 6 | Link a recording to a historical event from `.pst` history. |
|
||||
| FR-CAL-5 | C | 8 | Create local reminders / OS notifications for action items. |
|
||||
| FR-CAL-6 | C | later | Optional Microsoft Graph calendar source (explicit consent; calendar metadata only). |
|
||||
|
||||
### Remote sync / upload (SYNC) — see ADR-0010
|
||||
|
||||
Sync is an **explicit, user-configured export**; it is the only egress for meeting content besides
|
||||
the local LLM endpoint, and it is **off by default**. Primary targets are self-hostable.
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-SYNC-1 | M | 9 | Sync is disabled by default. No artifact is uploaded anywhere unless the user has configured a target and explicitly enabled it (with acknowledgment). |
|
||||
| FR-SYNC-2 | M | 9 | Configure one or more **WebDAV** targets covering the primary set — **Nextcloud, ownCloud, Cloudreve, Seafile** — and **Synology** (URL + username + app password; user-chosen remote base path). |
|
||||
| FR-SYNC-3 | M | 9 | Choose, per target, **what** to upload (transcript, notes, summary, and — only if retained per FR-REC-1 — the `.wav` recording) and **when** (on finalize, manual "Upload now", and automatic retry). |
|
||||
| FR-SYNC-4 | M | 9 | "Test connection" validates a target's reachability + credentials before saving. |
|
||||
| FR-SYNC-5 | M | 9 | Durable upload queue with per-job status, exponential-backoff retry, SHA-256 skip-if-unchanged (idempotent), and resumable/chunked upload for large recordings. |
|
||||
| FR-SYNC-6 | M | 9 | Store target credentials in the OS credential store (Windows Credential Manager / DPAPI), never in `settings.json` or the database. |
|
||||
| FR-SYNC-7 | M | 9 | Require TLS; refuse plaintext `http://` unless the user explicitly allows it for a LAN address, with a warning. |
|
||||
| FR-SYNC-8 | M | 9 | Surface sync state in the UI and label targets: self-hosted/primary as "your server"; third-party clouds carry a clear "data leaves your device to a third party" banner. |
|
||||
| FR-SYNC-9 | S | 9 | **Secondary** targets via provider APIs + OAuth 2.0 (PKCE, loopback redirect): **OneDrive** (MS Graph), **Dropbox**, **Box**. |
|
||||
| FR-SYNC-10 | C | 9 | Optional client-side encryption of artifacts before upload (ties to FR-SEC-3): destination holds only ciphertext. |
|
||||
|
||||
### UX, accessibility, recovery (UX)
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-UX-1 | M | 7 | Three-pane layout: meetings list · transcript/notes · summary/action-items/participants. |
|
||||
| FR-UX-2 | M | 7 | Light/dark themes following system setting, with user override. |
|
||||
| FR-UX-3 | S | 7 | Keyboard shortcuts for start/stop recording, view toggles, and template apply. |
|
||||
| FR-UX-4 | S | 7 | Screen-reader-friendly labels and sufficient contrast. |
|
||||
| FR-REL-1 | M | 2 | Crash recovery: regenerate transcript/notes from persisted audio after an unexpected exit. |
|
||||
| FR-REL-2 | M | 2 | Auto-save transcript and notes at intervals. |
|
||||
|
||||
### Privacy & security (SEC) — see also NFR-SEC
|
||||
|
||||
| ID | Pri | Phase | Requirement |
|
||||
|---|---|---|---|
|
||||
| FR-SEC-1 | M | all | WA originates **no** outbound connection for audio or transcript content except to destinations the user has explicitly configured: the LLM endpoint (local or, if chosen, a hosted AI provider — ADR-0007/0011), enabled sync targets (ADR-0010), and (Layer 3) a configured task tracker. With none configured, WA makes no content egress at all. Model downloads are the only other allowed egress and are explicit. The **local MCP server is inbound on loopback and is not egress** (FR-MCP-7). |
|
||||
| FR-SEC-2 | M | 7 | A privacy panel shows current hardware backend, LLM/AI provider endpoint, **every enabled sync target** (self-hosted vs third-party), and **MCP server state** (and that connected agents may forward served data to their own provider), stating exactly what leaves the device. |
|
||||
| FR-SEC-3 | C | 8 | Optional at-rest encryption of stored audio/transcripts; optional password-protected vault. |
|
||||
| FR-MODEL-1 | S | 3 | Model management UI: choose Whisper size, download/remove transcription & diarization models. |
|
||||
|
||||
## Non-functional requirements
|
||||
|
||||
### Performance (PERF)
|
||||
- **NFR-PERF-1 (M):** Real-time transcription latency under ~2–3 s per phrase on accelerated paths.
|
||||
- **NFR-PERF-2 (M):** Batch transcription of a 60-min meeting completes within ≤1–2× meeting length
|
||||
on CPU-only; faster on NPU/GPU.
|
||||
- **NFR-PERF-3 (M):** Out-of-the-box defaults run acceptably on a typical Windows laptop with no
|
||||
dedicated GPU/NPU (small/efficient model, conservative settings).
|
||||
- **NFR-PERF-4 (S):** Cold start to interactive UI under ~2 s; heavy components (models) lazy-load
|
||||
after the window is shown.
|
||||
|
||||
### Resource usage (RES)
|
||||
- **NFR-RES-1 (M):** Near-zero CPU/GPU/NPU and minimal memory/disk when idle (no recording or active
|
||||
transcription) — behave like a lightweight background app; no polling timers when idle.
|
||||
- **NFR-RES-2 (M):** During recording, prefer accelerated backends but stay responsive on CPU-only.
|
||||
- **NFR-RES-3 (S):** A "Low overhead" preset caps model size, disables real-time summarization, and
|
||||
reduces background indexing.
|
||||
- **NFR-RES-4 (M):** Do not add WA to OS startup or run invasive background tasks without explicit
|
||||
user consent.
|
||||
|
||||
### Reliability (REL)
|
||||
- **NFR-REL-1 (M):** Tolerate audio glitches and device changes without crashing or corrupting data.
|
||||
- **NFR-REL-2 (M):** Graceful fallback when hardware acceleration is unavailable.
|
||||
- **NFR-REL-3 (M):** No data loss on crash — audio persisted continuously; derived artifacts
|
||||
regenerable.
|
||||
|
||||
### Security (SEC)
|
||||
- **NFR-SEC-1 (M):** Store data under the user profile, respecting Windows ACLs.
|
||||
- **NFR-SEC-2 (M):** Operate without admin privileges.
|
||||
- **NFR-SEC-3 (S):** Network egress is restricted to an allowlist derived from user configuration:
|
||||
the LLM/AI-provider endpoint, enabled sync target hosts, a configured task tracker, and explicit
|
||||
model downloads; nothing else. Verifiable (network test in CI / a privacy self-check). Any
|
||||
connection to a host not on the derived allowlist is a defect. The MCP server is inbound and does
|
||||
not add to the allowlist.
|
||||
- **NFR-SEC-4 (M):** Sync and AI-provider credentials/keys live only in the OS credential store; TLS
|
||||
is required for sync transport (plaintext only via explicit per-target LAN opt-in). See FR-SYNC-6/7,
|
||||
FR-AI-2.
|
||||
- **NFR-SEC-5 (M):** The MCP server binds to **loopback only**, requires a token, is **off by
|
||||
default**, and honors the configured exposure scope; it never serves recordings unless explicitly
|
||||
allowed, and logs agent reads (FR-MCP-1/3/5).
|
||||
|
||||
### Maintainability & portability (MNT)
|
||||
- **NFR-MNT-1 (M):** Modular services (audio, transcription, diarization, storage, llm, calendar,
|
||||
hardware, notes, sync, mcp) with trait-defined boundaries enabling component replacement.
|
||||
- **NFR-MNT-2 (S):** Engines selectable/swappable without changes to callers.
|
||||
- **NFR-MNT-3 (C):** Architecture portable to macOS/Linux/Windows-on-ARM later without a rewrite.
|
||||
- **NFR-MNT-4 (M):** CPU-only build always compiles; accelerated backends behind Cargo features.
|
||||
|
||||
## Traceability
|
||||
|
||||
The roadmap (`05-roadmap.md`) tags each task with the FR/NFR it satisfies; the test strategy
|
||||
(`06-test-strategy.md`) maps tests to the same IDs. A requirement is "done" only when its
|
||||
acceptance criteria (in the roadmap) pass the tests named for it.
|
||||
@@ -0,0 +1,178 @@
|
||||
# 02 — Architecture
|
||||
|
||||
## Component map
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ WebView2 window │
|
||||
│ ┌───────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Svelte frontend (TypeScript) │ │
|
||||
│ │ views: MeetingsList · TranscriptNotes · SummaryPanel · │ │
|
||||
│ │ Settings · SpeakerReview · CalendarView │ │
|
||||
│ │ stores: recording, meeting, transcript, hardware, settings │ │
|
||||
│ └───────────────▲───────────────────────────┬───────────────────┘ │
|
||||
│ Tauri events (push) Tauri commands (request/response) │
|
||||
└──────────────────┼───────────────────────────┼──────────────────────┘
|
||||
│ │
|
||||
┌──────────────────┴───────────────────────────▼──────────────────────┐
|
||||
│ Rust core (Tauri) │
|
||||
│ │
|
||||
│ commands.rs ──orchestrates──► AppState { service handles } │
|
||||
│ │
|
||||
│ ┌─────────┐ ┌──────────┐ ┌────────────┐ ┌──────────────┐ │
|
||||
│ │ audio │ │ hardware │ │transcription│ │ diarization │ │
|
||||
│ │ (WASAPI)│ │(detect) │ │(whisper/onnx)│ │(sherpa-onnx)│ │
|
||||
│ └────┬────┘ └────┬─────┘ └─────┬──────┘ └──────┬───────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ ┌────▼────────────▼──────────────▼────────────────▼───────┐ │
|
||||
│ │ storage (SQLite + files) notes (markdown) │ │
|
||||
│ └────┬───────────────────────────────────┬───────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌────▼─────┐ ┌──────────────┐ ┌──────────┐ ┌─────▼──────┐ │
|
||||
│ │ llm │ │ sync │ │ mcp │ │ calendar │ │
|
||||
│ │(Ollama / │ │(WebDAV/OAuth)│ │ (server) │ │ (.pst/Graph)│ │
|
||||
│ │ cloud) │ └──────┬───────┘ └────▲─────┘ └────────────┘ │
|
||||
│ └────┬─────┘ │ │ loopback (INBOUND) │
|
||||
└────────┼────────────────┼───────────────┼──────────────────────────────┘
|
||||
│ HTTP/HTTPS │ HTTPS │ MCP over stdio / http://127.0.0.1/mcp
|
||||
│ (local or │ (targets, │ (off by default; token; NOT egress)
|
||||
│ cloud AI) │ off default) │
|
||||
┌─────▼─────┐ ┌──────▼───────────┐ │ ┌──────────────────────────────┐
|
||||
│ Ollama / │ │ Nextcloud·Seafile│ └───│ Claude · Codex · Copilot · │
|
||||
│ Anthropic │ │ ·Cloudreve·… │ │ OpenCode (user's own agents) │
|
||||
│ /OpenAI │ │ |OneDrive·Dropbox│ │ — they may forward to their │
|
||||
└───────────┘ └──────────────────┘ │ own cloud (outside WA) │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
The `mcp` arrow points **inward**: agents connect to WA, not the reverse. WA opens no socket for it.
|
||||
|
||||
Each Rust service lives in its own module under `src-tauri/src/<service>/` and exposes a **trait**
|
||||
plus a default implementation. Callers depend on the trait (NFR-MNT-1/2). The frontend never
|
||||
touches files, DB, or network directly — only Tauri commands/events (`04-api-contracts.md`).
|
||||
|
||||
## Services (responsibilities & key dependencies)
|
||||
|
||||
| Service | Responsibility | Primary crate(s) |
|
||||
|---|---|---|
|
||||
| `audio` | WASAPI loopback capture; PCM ring buffer; write WAV to disk; pause/resume | `wasapi`, `hound` |
|
||||
| `hardware` | Enumerate NPU/GPU/CPU; rank backends; report capabilities | `ort`, DXGI via `windows` |
|
||||
| `transcription` | Load model on a backend; stream segments (whisper.cpp) or NPU (ONNX) | `whisper-rs`, `ort` |
|
||||
| `diarization` | Post-process audio → speaker spans; align to segments; merge | `sherpa-onnx` (FFI) |
|
||||
| `storage` | SQLite index + file layout; CRUD; FTS search; retention; recovery scan | `sqlx`, `serde_json` |
|
||||
| `notes` | Assemble Markdown from segments+speakers+summary; render/export | `pulldown-cmark` |
|
||||
| `llm` | Talk to Ollama/custom endpoint; build prompts; stream; parse action items | `reqwest`, `serde` |
|
||||
| `calendar` | Parse `.pst` events/attendees; (future) Graph/ICS sources | `outlook-pst` |
|
||||
| `sync` | Upload artifacts to user-configured targets; durable queue + retry; WebDAV + OAuth providers; OS-keychain creds | `reqwest`, `reqwest_dav`, `keyring` |
|
||||
| `mcp` | Host a local (loopback) MCP server exposing meeting context as tools; build feature briefs; scope + token + audit | `rmcp`, `axum`/`hyper` |
|
||||
|
||||
The `audio` service also owns **recording retention** (ADR-0009): the working WAV is kept as the
|
||||
permanent `audio.wav` only when "Record this meeting" is on; otherwise it is deleted on finalize.
|
||||
|
||||
The `llm` service is extended (ADR-0011, Layer 1) with optional **hosted** providers (Anthropic
|
||||
Messages, OpenAI-compatible) behind the same `LlmProvider` trait, in addition to local Ollama. An
|
||||
optional `agent` module (Layer 3, later) spawns local coding-agent CLIs and/or creates task-tracker
|
||||
issues from feature briefs.
|
||||
|
||||
## Data flow — a typical meeting
|
||||
|
||||
1. User selects/creates a meeting (optionally from a calendar event) → `start_recording`.
|
||||
2. `audio` begins WASAPI loopback capture, writing PCM to `audio.wav` **continuously** (FR-CAP-2),
|
||||
and pushes frames into an in-memory ring buffer.
|
||||
3. `transcription` consumes windowed audio from the ring buffer on the selected `hardware` backend
|
||||
and emits interim `TranscriptSegment`s → `transcript_segment` events to the UI (FR-TRX-2).
|
||||
4. Provisional speaker turns (from cheap segmentation) tag segments live; the user may name a
|
||||
speaker, applied immediately to past/future segments (FR-SPK-2).
|
||||
5. The notes view renders Markdown that updates as segments arrive (FR-NOTE-1, FR-NOTE-6).
|
||||
6. On `stop_recording`: finalize the WAV; run full `diarization`, align speaker IDs to segments,
|
||||
apply name mappings; persist transcript JSON + metadata via `storage` (FR-STORE-1). If "Record
|
||||
this meeting" is **off**, delete the working WAV **after** the transcript is finalized; if **on**,
|
||||
keep it as `audio.wav` (ADR-0009, FR-REC-1/4).
|
||||
7. If an LLM provider is configured: `llm` builds a prompt (transcript + metadata + template),
|
||||
streams a summary/decisions/action-items into the summary panel (FR-LLM-2/4).
|
||||
8. Action items are parsed and presented for confirmation; confirmed ones persist and may raise
|
||||
local OS reminders (FR-LLM-3, FR-CAL-5).
|
||||
9. If sync is configured and enabled, `storage` finalization enqueues the selected artifacts
|
||||
(transcript/notes/summary, and the recording if retained) onto the `sync` queue; `sync` uploads
|
||||
them to each target with retry/backoff and reports progress via events (FR-SYNC-3/5). Nothing is
|
||||
uploaded if no target is enabled (FR-SYNC-1).
|
||||
|
||||
## Data flow — meeting → code (agent handoff, ADR-0011)
|
||||
|
||||
The "customer asks for a feature, get started right away" path, **pull** model:
|
||||
|
||||
1. WA distills the relevant part of the transcript into a **feature brief** (via the configured
|
||||
`llm`): problem, desired outcome, acceptance criteria, target repo/context, source meeting
|
||||
(FR-MCP-4). It is stored with the meeting and offered as a tool result.
|
||||
2. The user has the `mcp` server enabled (off by default) with a chosen exposure scope (FR-MCP-1/3).
|
||||
3. In **their own** agent (Claude Code, Codex, Copilot, OpenCode), the developer asks it to act;
|
||||
the agent connects to WA's loopback MCP server and calls `get_feature_brief` / `get_action_items`
|
||||
to pull structured context (FR-MCP-2), then scaffolds/branches/PRs in their repo.
|
||||
4. WA logs what was read and reminds the user that the agent may forward served data to its own cloud
|
||||
(FR-MCP-5). WA itself sends nothing (FR-MCP-7).
|
||||
5. *(Layer 3, later)* Alternatively, WA **pushes**: an `AgentRunner` spawns a CLI agent headless
|
||||
against a repo, or WA opens a GitHub issue (optionally assigned to Copilot's cloud agent) from the
|
||||
brief (FR-AGENT-1/2).
|
||||
|
||||
## Threading & concurrency model
|
||||
|
||||
- **UI/IPC thread:** Tauri commands return quickly. Nothing that runs for the length of a meeting
|
||||
blocks a command — long work is spawned and reports via **events**.
|
||||
- **Audio thread:** a dedicated thread runs the WASAPI **polling** capture loop. (WASAPI loopback
|
||||
cannot use event-callback mode, so we poll with a small interval — see `07-research-findings.md`.)
|
||||
It only does capture + disk write + ring-buffer push; no inference.
|
||||
- **Inference worker(s):** transcription runs on its own worker, pulling audio windows from the
|
||||
ring buffer; diarization runs as a discrete post-stop job. Backends are constructed once per
|
||||
session from the `hardware` ranking.
|
||||
- **Async runtime:** `tokio` for `storage` (sqlx), `llm` (reqwest streaming), and orchestration.
|
||||
- **State:** a `tauri::State<AppState>` holds `Arc`-wrapped service handles and a `RecordingSession`
|
||||
guarded so start/stop/pause transitions are atomic.
|
||||
- **Backpressure:** the ring buffer is bounded; if transcription falls behind (CPU-only, large
|
||||
model), it drops to interim/skip strategy for the live view while the on-disk audio remains
|
||||
complete for an accurate post-meeting pass.
|
||||
|
||||
## Idle behavior (NFR-RES-1)
|
||||
|
||||
When not recording, no audio thread, inference worker, or polling timer runs. Background indexing
|
||||
(FTS, retention) is event-driven or scheduled with idle-priority, and is reduced further under the
|
||||
"Low overhead" preset (NFR-RES-3). Models are unloaded from memory when idle and lazy-loaded on the
|
||||
next recording (NFR-PERF-4).
|
||||
|
||||
## Error handling & resilience
|
||||
|
||||
- Each module defines a `thiserror` error enum; commands surface typed errors to the frontend.
|
||||
- Hardware/backend failures degrade one tier at a time down to CPU (FR-HW-4, NFR-REL-2) and surface
|
||||
a non-blocking notice; they never abort a recording.
|
||||
- `audio` is the source of truth: a crash leaves a recoverable `audio.wav`; on next launch a
|
||||
reconcile scan finds meetings with audio but no finalized transcript and offers recovery
|
||||
(FR-REL-1, NFR-REL-3).
|
||||
- PST parse failures and missing LLM providers are non-fatal: core capture/transcription/notes
|
||||
keep working.
|
||||
|
||||
## Security boundaries (FR-SEC-1, NFR-SEC-*)
|
||||
|
||||
- WA opens sockets only to destinations the user configured: the LLM endpoint (**local** Ollama or,
|
||||
if chosen, a **hosted** AI provider — ADR-0011), **enabled sync target hosts**, a configured task
|
||||
tracker (Layer 3), and model-download hosts during an explicit download. These form an **allowlist
|
||||
derived from settings**; a privacy self-check (and a CI network test) asserts no connection to any
|
||||
off-allowlist host (NFR-SEC-3). With nothing configured, WA makes no content egress at all.
|
||||
- **MCP server (ADR-0011) is inbound, not egress.** WA *listens* on loopback for the user's agents;
|
||||
it opens no outbound socket and adds nothing to the allowlist (FR-MCP-7/NFR-SEC-5). It is off by
|
||||
default, token-gated, scope-limited (never serves recordings unless allowed), and audited. The
|
||||
data-leaves-device disclosure for this path is "the agent you connected may forward served data to
|
||||
its own provider" — outside WA's control, like the remote-LLM caveat (ADR-0007).
|
||||
- **Sync** is off by default and is an explicit user export (ADR-0010): credentials live in the OS
|
||||
credential store (never in settings/DB, FR-SYNC-6/NFR-SEC-4); TLS is required (plaintext only via
|
||||
explicit per-target LAN opt-in, FR-SYNC-7); third-party targets are clearly labeled (FR-SYNC-8).
|
||||
- The WebView CSP also restricts `connect-src` to localhost, so the frontend itself cannot reach the
|
||||
network — all egress goes through the Rust `sync`/`llm` services where the allowlist is enforced.
|
||||
- Data is stored under the user profile honoring ACLs; no admin rights required; optional at-rest
|
||||
encryption in Phase 8, optionally applied **before** upload so a sync target holds only ciphertext
|
||||
(FR-SYNC-10).
|
||||
|
||||
## Portability seam (NFR-MNT-3)
|
||||
|
||||
Windows-specific code (WASAPI, DXGI enumeration, toast notifications, PST) sits behind the service
|
||||
traits. A future macOS/Linux port implements the same traits with platform equivalents
|
||||
(CoreAudio/PulseAudio capture, Metal/Vulkan backends, etc.) without changing callers or the
|
||||
frontend.
|
||||
@@ -0,0 +1,311 @@
|
||||
# 03 — Data Model
|
||||
|
||||
## On-disk layout
|
||||
|
||||
Default root: `%LOCALAPPDATA%\WhispAssist\` (user-configurable, FR-STORE-2).
|
||||
|
||||
```
|
||||
%LOCALAPPDATA%\WhispAssist\
|
||||
├── wa.db # SQLite index (relations, metadata, FTS)
|
||||
├── settings.json # app settings (theme, provider, storage, presets)
|
||||
├── models\ # downloaded models (whisper, diarization)
|
||||
│ ├── whisper-base.q5.bin
|
||||
│ ├── seg-pyannote-3.0.onnx
|
||||
│ └── spk-eres2net.onnx
|
||||
└── meetings\
|
||||
└── <meeting_id>\ # one folder per meeting (uuid)
|
||||
├── audio.wav # canonical recording — present ONLY if "Record" was on (ADR-0009)
|
||||
├── transcript.json # canonical transcript (segments+speakers+timings)
|
||||
├── notes.md # user-editable Markdown notes
|
||||
├── summary.json # LLM summary, decisions, action items (if generated)
|
||||
└── briefs/ # feature briefs distilled from this meeting (ADR-0011), if any
|
||||
└── <brief_id>.json # agent-ready spec served via the MCP `get_feature_brief` tool
|
||||
```
|
||||
|
||||
Rule: while a meeting is in progress a working WAV is the source of truth for crash recovery. On
|
||||
finalize, it is **kept** as `audio.wav` if "Record this meeting" was on, or **deleted** if not
|
||||
(FR-REC-1/4) — deletion happens only after `transcript.json` is finalized. `transcript.json`,
|
||||
`notes.md`, and `summary.json` are **derived** and regenerable (regenerable only while the audio
|
||||
still exists — i.e. for recorded meetings).
|
||||
|
||||
## SQLite schema (`wa.db`)
|
||||
|
||||
```sql
|
||||
-- A meeting is the top-level record.
|
||||
CREATE TABLE meetings (
|
||||
id TEXT PRIMARY KEY, -- uuid v4
|
||||
title TEXT NOT NULL DEFAULT 'Untitled meeting',
|
||||
started_at INTEGER NOT NULL, -- unix epoch seconds
|
||||
ended_at INTEGER, -- null while recording
|
||||
duration_secs INTEGER, -- finalized on stop
|
||||
folder_path TEXT NOT NULL, -- absolute path to meeting folder
|
||||
audio_path TEXT, -- audio.wav; NULL if not retained (ADR-0009)
|
||||
recorded INTEGER NOT NULL DEFAULT 0, -- 1 = audio retained as .wav, 0 = transcript-only
|
||||
status TEXT NOT NULL, -- recording|transcribing|ready|recovering|error
|
||||
language TEXT, -- detected/selected language code
|
||||
backend_used TEXT, -- npu|nvidia|amd|intel|cpu
|
||||
model_used TEXT, -- e.g. whisper-base
|
||||
calendar_event_id TEXT, -- FK -> calendar_events.id (nullable)
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- Internal speakers detected per meeting (S1, S2, …) and their assigned names.
|
||||
CREATE TABLE speakers (
|
||||
id TEXT PRIMARY KEY, -- uuid
|
||||
meeting_id TEXT NOT NULL REFERENCES meetings(id) ON DELETE CASCADE,
|
||||
label TEXT NOT NULL, -- 'S1','S2',… (internal, stable per meeting)
|
||||
display_name TEXT, -- user/participant name (nullable)
|
||||
participant_id TEXT REFERENCES participants(id), -- if mapped to a calendar attendee
|
||||
color TEXT, -- UI color hint
|
||||
UNIQUE(meeting_id, label)
|
||||
);
|
||||
|
||||
-- People known from calendar/.pst; reused across meetings for continuity.
|
||||
CREATE TABLE participants (
|
||||
id TEXT PRIMARY KEY, -- uuid
|
||||
name TEXT NOT NULL,
|
||||
email TEXT, -- nullable
|
||||
UNIQUE(name, email)
|
||||
);
|
||||
|
||||
CREATE TABLE meeting_participants ( -- attendee list per meeting
|
||||
meeting_id TEXT NOT NULL REFERENCES meetings(id) ON DELETE CASCADE,
|
||||
participant_id TEXT NOT NULL REFERENCES participants(id),
|
||||
role TEXT, -- organizer|required|optional
|
||||
PRIMARY KEY (meeting_id, participant_id)
|
||||
);
|
||||
|
||||
-- Calendar events imported from .pst (or future sources).
|
||||
CREATE TABLE calendar_events (
|
||||
id TEXT PRIMARY KEY, -- uuid (stable from source uid if available)
|
||||
source TEXT NOT NULL, -- pst|graph|ics
|
||||
subject TEXT,
|
||||
organizer TEXT,
|
||||
starts_at INTEGER,
|
||||
ends_at INTEGER,
|
||||
description TEXT,
|
||||
raw_uid TEXT -- source's own id for dedup
|
||||
);
|
||||
|
||||
CREATE TABLE tags (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE
|
||||
);
|
||||
CREATE TABLE meeting_tags (
|
||||
meeting_id TEXT NOT NULL REFERENCES meetings(id) ON DELETE CASCADE,
|
||||
tag_id TEXT NOT NULL REFERENCES tags(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (meeting_id, tag_id)
|
||||
);
|
||||
|
||||
-- Action items parsed from LLM output, then user-confirmed.
|
||||
CREATE TABLE action_items (
|
||||
id TEXT PRIMARY KEY,
|
||||
meeting_id TEXT NOT NULL REFERENCES meetings(id) ON DELETE CASCADE,
|
||||
text TEXT NOT NULL,
|
||||
owner TEXT, -- assignee name (nullable)
|
||||
due_at INTEGER, -- nullable
|
||||
confirmed INTEGER NOT NULL DEFAULT 0, -- 0=suggested, 1=confirmed by user
|
||||
reminder_set INTEGER NOT NULL DEFAULT 0,
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- Feature briefs distilled from a meeting for coding-agent handoff (ADR-0011). The JSON body
|
||||
-- lives in the meeting's briefs/ folder; this table indexes it for the MCP tools.
|
||||
CREATE TABLE feature_briefs (
|
||||
id TEXT PRIMARY KEY, -- uuid
|
||||
meeting_id TEXT NOT NULL REFERENCES meetings(id) ON DELETE CASCADE,
|
||||
title TEXT NOT NULL,
|
||||
target_repo TEXT, -- optional repo hint for the agent
|
||||
path TEXT NOT NULL, -- briefs/<id>.json
|
||||
exposed INTEGER NOT NULL DEFAULT 0, -- visible to the MCP server? (scope control, FR-MCP-3)
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- Audit of what an MCP client (agent) read (FR-MCP-5).
|
||||
CREATE TABLE mcp_access_log (
|
||||
id TEXT PRIMARY KEY,
|
||||
at INTEGER NOT NULL,
|
||||
tool TEXT NOT NULL, -- e.g. get_feature_brief
|
||||
meeting_id TEXT, -- subject, if any
|
||||
client TEXT -- client-reported name, if provided
|
||||
);
|
||||
|
||||
-- Configured sync/upload destinations (ADR-0010). Credentials are NOT stored here —
|
||||
-- only a reference into the OS credential store (FR-SYNC-6).
|
||||
CREATE TABLE sync_targets (
|
||||
id TEXT PRIMARY KEY, -- uuid
|
||||
name TEXT NOT NULL, -- user label, e.g. "Home Nextcloud"
|
||||
kind TEXT NOT NULL, -- webdav|onedrive|dropbox|box
|
||||
provider_hint TEXT, -- nextcloud|owncloud|cloudreve|seafile|synology|generic
|
||||
base_url TEXT, -- WebDAV URL (kind=webdav)
|
||||
remote_base_path TEXT NOT NULL DEFAULT '/WhispAssist',
|
||||
username TEXT, -- WebDAV username (secret is in credential store)
|
||||
credential_ref TEXT NOT NULL, -- key into OS credential store
|
||||
enabled INTEGER NOT NULL DEFAULT 0, -- off by default (FR-SYNC-1)
|
||||
upload_transcript INTEGER NOT NULL DEFAULT 1,
|
||||
upload_notes INTEGER NOT NULL DEFAULT 1,
|
||||
upload_summary INTEGER NOT NULL DEFAULT 1,
|
||||
upload_recording INTEGER NOT NULL DEFAULT 0, -- only meaningful if a meeting is recorded
|
||||
trigger_on_finalize INTEGER NOT NULL DEFAULT 1,
|
||||
allow_plaintext_lan INTEGER NOT NULL DEFAULT 0, -- FR-SYNC-7
|
||||
encrypt_before_upload INTEGER NOT NULL DEFAULT 0, -- FR-SYNC-10
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- One upload job per (artifact, target). The durable queue (FR-SYNC-5).
|
||||
CREATE TABLE sync_jobs (
|
||||
id TEXT PRIMARY KEY, -- uuid
|
||||
target_id TEXT NOT NULL REFERENCES sync_targets(id) ON DELETE CASCADE,
|
||||
meeting_id TEXT NOT NULL REFERENCES meetings(id) ON DELETE CASCADE,
|
||||
artifact TEXT NOT NULL, -- transcript|notes|summary|recording
|
||||
local_path TEXT NOT NULL,
|
||||
remote_path TEXT NOT NULL,
|
||||
sha256 TEXT, -- skip-if-unchanged idempotency
|
||||
status TEXT NOT NULL, -- pending|uploading|done|failed|skipped
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT,
|
||||
next_attempt_at INTEGER, -- backoff schedule
|
||||
bytes_total INTEGER,
|
||||
bytes_sent INTEGER NOT NULL DEFAULT 0,
|
||||
updated_at INTEGER NOT NULL,
|
||||
UNIQUE(target_id, meeting_id, artifact)
|
||||
);
|
||||
|
||||
-- Full-text search over transcript + notes (FR-SEARCH-1).
|
||||
CREATE VIRTUAL TABLE meeting_fts USING fts5(
|
||||
meeting_id UNINDEXED,
|
||||
title,
|
||||
transcript_text,
|
||||
notes_text,
|
||||
tokenize = 'porter unicode61'
|
||||
);
|
||||
|
||||
CREATE INDEX idx_meetings_started ON meetings(started_at DESC);
|
||||
CREATE INDEX idx_speakers_meeting ON speakers(meeting_id);
|
||||
CREATE INDEX idx_action_meeting ON action_items(meeting_id);
|
||||
CREATE INDEX idx_syncjobs_status ON sync_jobs(status, next_attempt_at);
|
||||
CREATE INDEX idx_briefs_meeting ON feature_briefs(meeting_id);
|
||||
```
|
||||
|
||||
Schema migrations are versioned (e.g. `sqlx::migrate!`), each migration numbered and forward-only.
|
||||
|
||||
## `transcript.json` (canonical transcript)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"schema": 1,
|
||||
"meeting_id": "f1c2…",
|
||||
"language": "en",
|
||||
"model": "whisper-base",
|
||||
"backend": "nvidia",
|
||||
"segments": [
|
||||
{
|
||||
"id": 0,
|
||||
"start_ms": 1240, // offset from recording start
|
||||
"end_ms": 4880,
|
||||
"speaker": "S1", // internal speaker id; name resolved at render time
|
||||
"text": "Let's start with the roadmap.",
|
||||
"confidence": 0.91, // optional
|
||||
"interim": false // true while streaming, false once finalized
|
||||
}
|
||||
],
|
||||
"speakers": [
|
||||
{ "label": "S1", "display_name": "Alex" },
|
||||
{ "label": "S2", "display_name": null }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Names are stored as a mapping and applied at render/export — segments keep the internal `speaker`
|
||||
label so re-diarization or renaming never requires rewriting every segment (FR-SPK-5).
|
||||
|
||||
## `summary.json`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"schema": 1,
|
||||
"generated_at": 1751299200,
|
||||
"provider": "ollama",
|
||||
"model": "llama3",
|
||||
"summary_md": "## Summary\n…",
|
||||
"decisions": ["Adopt Tauri for the shell"],
|
||||
"action_items": [
|
||||
{ "text": "Send the API contract draft", "owner": "Jordan", "due": "2026-07-03" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## `briefs/<brief_id>.json` (feature brief — ADR-0011)
|
||||
|
||||
Agent-ready spec the MCP `get_feature_brief` tool returns. Designed to drop straight into a coding
|
||||
agent's context.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"schema": 1,
|
||||
"id": "b7a1…",
|
||||
"meeting_id": "f1c2…",
|
||||
"title": "Bulk CSV export for the reporting view",
|
||||
"problem": "Customer can't get their data out for offline analysis.",
|
||||
"desired_outcome": "One-click CSV export of the current filtered report.",
|
||||
"acceptance_criteria": [
|
||||
"Export button on the report toolbar",
|
||||
"Respects active filters and column order",
|
||||
"Streams large exports without blocking the UI"
|
||||
],
|
||||
"target_repo": "acme/reporting-web", // optional hint for the agent
|
||||
"context_excerpts": [ // minimal transcript quotes that ground the request
|
||||
{ "speaker": "Customer", "text": "We really need to pull this into our own spreadsheets." }
|
||||
],
|
||||
"source": { "meeting_title": "Acme quarterly sync", "at": 1751299200 }
|
||||
}
|
||||
```
|
||||
|
||||
## `settings.json`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"theme": "system", // system|light|dark
|
||||
"storage_root": "%LOCALAPPDATA%\\WhispAssist",
|
||||
"retention": { "max_age_days": 90, "max_size_gb": 10 },
|
||||
"transcription": { "model": "base", "language": "auto", "batch_reprocess": false },
|
||||
"hardware": { "preferred_backend": "auto", "low_overhead": false },
|
||||
"recording": {
|
||||
"default_record": false, // "Record this meeting" default — OFF (FR-REC-1)
|
||||
"consent_acknowledged": false // set true after the one-time consent notice (FR-REC-2)
|
||||
},
|
||||
"llm": {
|
||||
"provider": "ollama", // ollama|custom|anthropic|openai|off (ADR-0007/0011)
|
||||
"endpoint": "http://localhost:11434",
|
||||
"model": "llama3",
|
||||
"stream": true
|
||||
// API keys for hosted providers (anthropic|openai) live in the OS credential store, not here.
|
||||
},
|
||||
// Sync target rows live in wa.db (sync_targets); secrets live in the OS credential store.
|
||||
// settings.json only holds the global default. No credentials here (FR-SYNC-6).
|
||||
"sync": { "enabled": false }, // master off switch — OFF by default (FR-SYNC-1)
|
||||
// Local MCP server (ADR-0011). OFF by default; inbound loopback only; token in credential store.
|
||||
"mcp": {
|
||||
"enabled": false, // FR-MCP-1
|
||||
"transport": "http", // http (127.0.0.1/mcp) | stdio
|
||||
"port": 4849,
|
||||
"expose": "selected", // none | selected | all (scope control, FR-MCP-3)
|
||||
"expose_recordings": false // never serve .wav unless explicitly true (FR-MCP-3)
|
||||
},
|
||||
"privacy": { "encrypt_at_rest": false }
|
||||
}
|
||||
```
|
||||
|
||||
## Retention & recovery semantics
|
||||
|
||||
- **Retention** (FR-STORE-2): a background job deletes whole meeting folders + rows once a meeting
|
||||
exceeds the age/size policy, oldest first; never runs while a meeting is `recording`/`transcribing`.
|
||||
- **Recovery** (FR-REL-1): on startup, any meeting whose **working WAV still exists** but whose
|
||||
`transcript.json` is missing/partial is marked `recovering`; the user can re-run transcription from
|
||||
the audio. (A cleanly finalized non-recorded meeting has no audio and needs no recovery; an
|
||||
interrupted one still has its working WAV, so it is always recoverable.)
|
||||
- **Sync** (FR-SYNC-5): on startup and after each finalize, the `sync` service scans `sync_jobs` for
|
||||
`pending`/`failed` jobs whose `next_attempt_at` has passed and resumes them with backoff. Deleting
|
||||
a meeting cascades to its `sync_jobs` (local rows only; already-uploaded remote copies are left to
|
||||
the user/server).
|
||||
@@ -0,0 +1,243 @@
|
||||
# 04 — API Contracts
|
||||
|
||||
Two contract surfaces:
|
||||
1. **Frontend ⇄ Rust** — Tauri **commands** (request/response) and **events** (Rust→UI push).
|
||||
2. **Rust internal** — service **traits** that decouple callers from concrete engines (NFR-MNT-1/2).
|
||||
|
||||
All payloads are `serde`-serializable; timestamps are unix epoch ms unless noted. Errors are typed:
|
||||
each command returns `Result<T, WaError>` where `WaError` carries a `kind` (machine-readable) and a
|
||||
`message` (human-readable).
|
||||
|
||||
## 1. Tauri commands (frontend → Rust)
|
||||
|
||||
```ts
|
||||
// ---- Recording lifecycle ----
|
||||
// `record` (default false) controls audio RETENTION (ADR-0009). When false, working audio is
|
||||
// deleted on finalize and only the transcript/notes persist. It can be toggled mid-meeting.
|
||||
start_recording(input: { meetingTitle?: string; calendarEventId?: string; record?: boolean }): MeetingId
|
||||
stop_recording(input: { meetingId: MeetingId }): MeetingSummaryRef
|
||||
pause_recording(input: { meetingId: MeetingId }): void
|
||||
resume_recording(input: { meetingId: MeetingId }): void
|
||||
set_recording_retention(input: { meetingId: MeetingId; record: boolean }): void // toggle mid-meeting (FR-REC-1)
|
||||
acknowledge_recording_consent(): void // one-time (FR-REC-2)
|
||||
|
||||
// ---- Hardware ----
|
||||
hardware_status(): { backends: BackendInfo[]; active: BackendId; modelSize: string; estRtf: number }
|
||||
set_preferred_backend(input: { backend: BackendId | "auto" }): void
|
||||
|
||||
// ---- Transcription / models ----
|
||||
reprocess_transcript(input: { meetingId: MeetingId; model: string }): void // batch mode (FR-TRX-3)
|
||||
list_models(): ModelInfo[]
|
||||
download_model(input: { kind: "whisper" | "diar-seg" | "diar-emb"; id: string }): void // emits progress events
|
||||
remove_model(input: { id: string }): void
|
||||
|
||||
// ---- Speakers ----
|
||||
rename_speaker(input: { meetingId: MeetingId; label: string; name: string }): void
|
||||
merge_speakers(input: { meetingId: MeetingId; from: string[]; into: string }): void
|
||||
map_speaker_to_participant(input: { meetingId: MeetingId; label: string; participantId: string }): void
|
||||
|
||||
// ---- Meetings / storage ----
|
||||
list_meetings(input: { query?: string; tag?: string; participantId?: string; limit?: number; offset?: number }): MeetingListItem[]
|
||||
get_meeting(input: { meetingId: MeetingId }): Meeting // includes transcript + speakers
|
||||
delete_meeting(input: { meetingId: MeetingId }): void
|
||||
export_meeting(input: { meetingId: MeetingId; dest: string; format: "md" | "pdf" | "docx" | "bundle" }): string
|
||||
update_notes(input: { meetingId: MeetingId; markdown: string }): void
|
||||
search(input: { query: string }): SearchHit[] // FTS (FR-SEARCH-1)
|
||||
set_tags(input: { meetingId: MeetingId; tags: string[] }): void
|
||||
|
||||
// ---- LLM / AI provider (ADR-0007/0011) ----
|
||||
// provider ∈ ollama | custom | anthropic | openai | off. Hosted-provider API keys are passed to
|
||||
// set_llm_provider but stored only in the OS credential store; never returned by llm_status.
|
||||
llm_status(): { provider: string; reachable: boolean; isLocal: boolean; models: string[] }
|
||||
set_llm_provider(input: { provider: string; endpoint?: string; model?: string; apiKey?: string }): void // FR-AI-1/2
|
||||
generate_summary(input: { meetingId: MeetingId; templateId?: string }): void // streams via events (FR-LLM-2/4)
|
||||
confirm_action_items(input: { meetingId: MeetingId; items: ActionItem[] }): void
|
||||
|
||||
// ---- Calendar / .pst ----
|
||||
import_pst(input: { path: string; password?: string }): { eventsImported: number } // FR-CAL-1
|
||||
list_calendar_events(input: { from?: number; to?: number }): CalendarEvent[]
|
||||
attach_meeting_to_event(input: { meetingId: MeetingId; eventId: string }): void
|
||||
|
||||
// ---- Sync / upload (ADR-0010) ---- secrets are passed to add/update but stored only in the OS
|
||||
// credential store; they are NEVER returned by list_sync_targets.
|
||||
list_sync_targets(): SyncTargetInfo[]
|
||||
add_sync_target(input: SyncTargetConfig & { secret: string }): SyncTargetInfo // FR-SYNC-2/9
|
||||
update_sync_target(input: { id: string } & Partial<SyncTargetConfig> & { secret?: string }): SyncTargetInfo
|
||||
remove_sync_target(input: { id: string }): void
|
||||
test_sync_target(input: { id: string } | SyncTargetConfig & { secret: string }): { ok: boolean; message: string } // FR-SYNC-4
|
||||
set_sync_enabled(input: { enabled: boolean }): void // master switch (FR-SYNC-1)
|
||||
sync_meeting(input: { meetingId: MeetingId; targetId?: string }): void // manual "Upload now" (FR-SYNC-3)
|
||||
sync_status(input?: { meetingId?: MeetingId }): SyncJobInfo[] // queue state (FR-SYNC-5)
|
||||
retry_sync_job(input: { jobId: string }): void
|
||||
// OAuth (secondary targets, FR-SYNC-9): begins loopback-redirect PKCE flow, returns when linked.
|
||||
begin_oauth_link(input: { kind: "onedrive" | "dropbox" | "box" }): { ok: boolean; account?: string }
|
||||
|
||||
// ---- Feature briefs + MCP server (ADR-0011) ----
|
||||
create_feature_brief(input: { meetingId: MeetingId; targetRepo?: string }): FeatureBrief // FR-MCP-4 (LLM-distilled)
|
||||
list_feature_briefs(input: { meetingId?: MeetingId }): FeatureBriefInfo[]
|
||||
get_feature_brief(input: { id: string }): FeatureBrief
|
||||
set_brief_exposed(input: { id: string; exposed: boolean }): void // scope control (FR-MCP-3)
|
||||
mcp_status(): { enabled: boolean; transport: "http" | "stdio"; endpoint: string; tokenSet: boolean; exposeScope: string }
|
||||
set_mcp_enabled(input: { enabled: boolean; transport?: "http" | "stdio"; port?: number }): { endpoint: string; token: string } // FR-MCP-1/6
|
||||
set_mcp_scope(input: { expose: "none" | "selected" | "all"; exposeRecordings?: boolean }): void // FR-MCP-3
|
||||
mcp_access_log(input?: { limit?: number }): McpAccessEntry[] // audit (FR-MCP-5)
|
||||
// (Layer 3, later) push handoff — spawn a local agent CLI / open a tracker issue from a brief.
|
||||
run_agent(input: { briefId: string; tool: "claude" | "codex" | "opencode" | "copilot"; repoPath: string }): { ok: boolean } // FR-AGENT-1
|
||||
create_issue_from_brief(input: { briefId: string; tracker: "github"; assignCopilot?: boolean }): { url: string } // FR-AGENT-2
|
||||
|
||||
// ---- Settings ----
|
||||
get_settings(): Settings
|
||||
update_settings(input: Partial<Settings>): Settings
|
||||
// Reports the full egress allowlist so the UI can prove exactly what may leave the device (FR-SEC-2).
|
||||
privacy_self_check(): {
|
||||
llmEndpoint: string; llmIsLocal: boolean;
|
||||
syncEnabled: boolean;
|
||||
syncTargets: { name: string; host: string; thirdParty: boolean; tls: boolean }[];
|
||||
allowlistedHosts: string[];
|
||||
}
|
||||
```
|
||||
|
||||
### Conventions
|
||||
- Commands return promptly; anything long-running (recording, transcription, summary, model
|
||||
download, PST import) reports progress/results through **events** below.
|
||||
- `MeetingId` is a uuid string. `BackendId` ∈ `"npu" | "nvidia" | "amd" | "intel" | "cpu"`.
|
||||
|
||||
## 2. Tauri events (Rust → frontend)
|
||||
|
||||
```ts
|
||||
"recording://state" { meetingId, state: "recording"|"paused"|"stopped", elapsedMs }
|
||||
"recording://level" { meetingId, rms: number, peak: number } // waveform (FR-CAP-5)
|
||||
"transcript://segment" { meetingId, segment: TranscriptSegment } // live segments (FR-TRX-2)
|
||||
"transcript://finalized" { meetingId, segmentCount }
|
||||
"diarization://updated" { meetingId, speakers: SpeakerInfo[] } // after post-pass (FR-SPK)
|
||||
"llm://token" { meetingId, text } // streamed summary (FR-LLM-4)
|
||||
"llm://done" { meetingId, summary: SummaryRef }
|
||||
"model://progress" { id, receivedBytes, totalBytes }
|
||||
"pst://progress" { processed, total }
|
||||
"hardware://changed" { active: BackendId, reason: string } // fallback occurred (FR-HW-4)
|
||||
"recording://retention" { meetingId, record: boolean } // retention toggled (FR-REC-1/3)
|
||||
"sync://job" { jobId, meetingId, targetId, artifact, status, bytesSent, bytesTotal } // FR-SYNC-5
|
||||
"sync://done" { meetingId, targetId, uploaded: number, failed: number }
|
||||
"mcp://access" { at, tool, meetingId?, client? } // agent read something (FR-MCP-5)
|
||||
"agent://progress" { briefId, tool, line } // push run output (FR-AGENT-1)
|
||||
"error" { kind, message, context? }
|
||||
```
|
||||
|
||||
## 3. Internal Rust service traits
|
||||
|
||||
These define the seams that let engines be swapped without touching callers. Signatures are
|
||||
indicative (async where I/O-bound).
|
||||
|
||||
```rust
|
||||
// audio/mod.rs
|
||||
pub trait AudioCapture: Send + Sync {
|
||||
/// Begin WASAPI loopback capture, writing PCM to `wav_path`; frames also pushed to `sink`.
|
||||
fn start(&self, wav_path: &Path, sink: FrameSink) -> Result<CaptureHandle, AudioError>;
|
||||
fn pause(&self, h: &CaptureHandle) -> Result<(), AudioError>;
|
||||
fn resume(&self, h: &CaptureHandle) -> Result<(), AudioError>;
|
||||
fn stop(&self, h: CaptureHandle) -> Result<CaptureSummary, AudioError>;
|
||||
}
|
||||
|
||||
// hardware/mod.rs
|
||||
pub trait HardwareDetector: Send + Sync {
|
||||
fn detect(&self) -> Vec<BackendInfo>; // ranked NPU→NVIDIA→AMD→Intel→CPU
|
||||
fn best(&self, preferred: Option<BackendId>) -> BackendInfo;
|
||||
}
|
||||
|
||||
// transcription/mod.rs
|
||||
pub trait Transcriber: Send + Sync {
|
||||
fn load(model: &Path, backend: BackendId) -> Result<Self, TrxError> where Self: Sized;
|
||||
/// Stream interim + final segments for an audio window.
|
||||
fn transcribe_stream(&self, audio: AudioWindow, out: SegmentSink) -> Result<(), TrxError>;
|
||||
/// One-shot batch transcription (higher accuracy).
|
||||
fn transcribe_file(&self, wav: &Path) -> Result<Vec<TranscriptSegment>, TrxError>;
|
||||
}
|
||||
|
||||
// diarization/mod.rs
|
||||
pub trait Diarizer: Send + Sync {
|
||||
fn diarize(&self, wav: &Path) -> Result<Vec<SpeakerSpan>, DiarError>;
|
||||
fn assign(&self, segments: &mut [TranscriptSegment], spans: &[SpeakerSpan]);
|
||||
}
|
||||
|
||||
// storage/mod.rs (async, sqlx)
|
||||
#[async_trait] pub trait Store: Send + Sync {
|
||||
async fn create_meeting(&self, m: NewMeeting) -> Result<MeetingId, StoreError>;
|
||||
async fn finalize_meeting(&self, id: &MeetingId, s: FinalizeMeeting) -> Result<(), StoreError>;
|
||||
async fn list_meetings(&self, f: MeetingFilter) -> Result<Vec<MeetingListItem>, StoreError>;
|
||||
async fn get_meeting(&self, id: &MeetingId) -> Result<Meeting, StoreError>;
|
||||
async fn delete_meeting(&self, id: &MeetingId) -> Result<(), StoreError>;
|
||||
async fn search(&self, q: &str) -> Result<Vec<SearchHit>, StoreError>;
|
||||
async fn recover_scan(&self) -> Result<Vec<MeetingId>, StoreError>; // FR-REL-1
|
||||
async fn enforce_retention(&self, policy: Retention) -> Result<u32, StoreError>;
|
||||
}
|
||||
|
||||
// llm/mod.rs
|
||||
#[async_trait] pub trait LlmProvider: Send + Sync {
|
||||
async fn status(&self) -> LlmStatus; // reachable? local? models
|
||||
async fn summarize(&self, prompt: Prompt, out: TokenSink) -> Result<Summary, LlmError>;
|
||||
fn is_local(&self) -> bool; // FR-LLM-6 guard
|
||||
}
|
||||
|
||||
// calendar/mod.rs
|
||||
pub trait CalendarSource: Send + Sync {
|
||||
fn import(&self, input: CalImport) -> Result<Vec<CalendarEvent>, CalError>; // pst|graph|ics
|
||||
fn attendees(&self, event_id: &str) -> Result<Vec<Participant>, CalError>;
|
||||
}
|
||||
|
||||
// notes/mod.rs
|
||||
pub trait NotesRenderer: Send + Sync {
|
||||
fn to_markdown(&self, t: &Transcript, speakers: &[SpeakerInfo], s: Option<&Summary>) -> String;
|
||||
fn export(&self, md: &str, dest: &Path, fmt: ExportFormat) -> Result<PathBuf, NotesError>;
|
||||
}
|
||||
|
||||
// sync/mod.rs
|
||||
// One impl per provider; `WebDavTarget` covers Nextcloud/ownCloud/Cloudreve/Seafile/Synology.
|
||||
// Secondary OAuth impls: OneDriveTarget, DropboxTarget, BoxTarget (ADR-0010).
|
||||
#[async_trait] pub trait SyncTarget: Send + Sync {
|
||||
fn kind(&self) -> SyncKind;
|
||||
fn is_third_party(&self) -> bool; // false for self-hosted WebDAV
|
||||
async fn test(&self) -> Result<(), SyncError>; // reachability + auth (FR-SYNC-4)
|
||||
async fn ensure_dir(&self, remote_dir: &str) -> Result<(), SyncError>;
|
||||
async fn exists(&self, remote_path: &str, sha256: &str) -> Result<bool, SyncError>; // skip-if-unchanged
|
||||
/// Upload a file; resumable/chunked for large artifacts. Reports progress via `prog`.
|
||||
async fn put(&self, local: &Path, remote_path: &str, prog: ProgressSink) -> Result<(), SyncError>;
|
||||
}
|
||||
|
||||
// Owns the durable queue, retry/backoff, credential resolution, and TLS enforcement.
|
||||
#[async_trait] pub trait SyncManager: Send + Sync {
|
||||
async fn enqueue_meeting(&self, meeting_id: &MeetingId, target_id: Option<&str>) -> Result<(), SyncError>;
|
||||
async fn pump(&self) -> Result<(), SyncError>; // drive pending jobs (called on finalize + startup + timer)
|
||||
async fn status(&self, meeting_id: Option<&MeetingId>) -> Result<Vec<SyncJobInfo>, SyncError>;
|
||||
async fn retry(&self, job_id: &str) -> Result<(), SyncError>;
|
||||
}
|
||||
|
||||
// llm/mod.rs — `LlmProvider` (above) gains hosted impls behind the same trait (ADR-0011):
|
||||
// OllamaProvider (local) · OpenAiCompatProvider (/v1/chat/completions) · AnthropicProvider (/v1/messages)
|
||||
// `is_local()` stays the egress guard; hosted impls return false and require an API key from the keychain.
|
||||
|
||||
// mcp/mod.rs — WA as an MCP server (loopback, off by default). Tools-first (FR-MCP-2).
|
||||
#[async_trait] pub trait McpServer: Send + Sync {
|
||||
async fn start(&self, cfg: McpConfig) -> Result<McpHandle, McpError>; // returns endpoint + token
|
||||
async fn stop(&self, h: McpHandle) -> Result<(), McpError>;
|
||||
fn tools(&self) -> Vec<McpToolDescriptor>; // list_recent_meetings, get_transcript, get_action_items, get_feature_brief
|
||||
}
|
||||
// Builds the agent-ready spec from a transcript via the configured LlmProvider.
|
||||
#[async_trait] pub trait FeatureBriefBuilder: Send + Sync {
|
||||
async fn build(&self, meeting_id: &MeetingId, target_repo: Option<&str>) -> Result<FeatureBrief, BriefError>;
|
||||
}
|
||||
|
||||
// agent/mod.rs — Layer 3 (later). Push handoff; one impl per CLI / tracker.
|
||||
#[async_trait] pub trait AgentRunner: Send + Sync {
|
||||
async fn run(&self, brief: &FeatureBrief, repo: &Path, out: LineSink) -> Result<RunOutcome, AgentError>; // claude -p / codex exec / …
|
||||
}
|
||||
#[async_trait] pub trait IssueTracker: Send + Sync {
|
||||
async fn create_issue(&self, brief: &FeatureBrief, assign_copilot: bool) -> Result<String /*url*/, AgentError>;
|
||||
}
|
||||
```
|
||||
|
||||
## Versioning
|
||||
|
||||
- Command/event names and payload shapes are versioned implicitly by `schema` fields in persisted
|
||||
JSON (`03-data-model.md`) and explicitly in a `CONTRACTS_VERSION` constant. Breaking a command
|
||||
shape requires bumping it and updating the frontend client in the same change (see `CLAUDE.md`
|
||||
"source of truth" rule).
|
||||
@@ -0,0 +1,237 @@
|
||||
# 05 — Roadmap
|
||||
|
||||
Eight phases, each a shippable increment. Every task carries an ID and the requirement(s) it
|
||||
satisfies; every phase has **acceptance criteria** that gate "done" (verified by the tests in
|
||||
`06-test-strategy.md`). Phases are ordered so each builds on a working previous one. Suggested
|
||||
effort labels: **S** ≤1 day, **M** a few days, **L** ~1–2 weeks (team-dependent, indicative only).
|
||||
|
||||
Legend: `[T1.2]` = task; → FR/NFR satisfied.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Foundation: capture + CPU transcription
|
||||
**Goal:** a window where you click Record, system audio is captured to disk, and plain transcript
|
||||
text streams in on CPU.
|
||||
|
||||
- `[T1.1]` Scaffold Tauri 2 + Svelte project; window, build, dev loop. **M** → ADR-0001/0002
|
||||
- `[T1.2]` `audio`: WASAPI loopback capture thread (polling), PCM ring buffer, WAV writer. **L** → FR-CAP-1/2
|
||||
- `[T1.3]` Recording lifecycle commands + `recording://state` events; start/stop/pause/resume. **M** → FR-CAP-3
|
||||
- `[T1.4]` Recording indicator: in-app banner + tray icon. **S** → FR-CAP-4
|
||||
- `[T1.5]` `transcription`: integrate `whisper-rs` CPU; load small model; window+overlap feeder. **L** → FR-TRX-1, NFR-MNT-4
|
||||
- `[T1.6]` Stream interim/final segments → `transcript://segment`; minimal transcript view. **M** → FR-TRX-2
|
||||
- `[T1.7]` Idle discipline: no threads/timers when not recording; model unloaded when idle. **S** → NFR-RES-1
|
||||
- `[T1.8]` "Record this meeting" toggle (default OFF) + mid-meeting `set_recording_retention`; "saving" indicator. **S** → FR-REC-1/3
|
||||
- `[T1.9]` Consent notice + one-time acknowledgment before first retained recording. **S** → FR-REC-2
|
||||
|
||||
**Acceptance:** Record a 5-min meeting; transcript appears live within a few seconds of speech on a
|
||||
CPU-only laptop; stopping leaves the app idle with near-zero CPU. With "Record" ON, a playable
|
||||
`audio.wav` remains; with it OFF (default), no audio file remains after finalize. The consent notice
|
||||
appears before the first retained recording and must be acknowledged once. (Tests: P1 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Storage, notes, persistence, recovery
|
||||
**Goal:** meetings are saved, listable, editable as Markdown, exportable, and survive a crash.
|
||||
|
||||
- `[T2.1]` `storage`: SQLite schema + migrations; meeting folder layout; transactional writes. **L** → FR-STORE-1
|
||||
- `[T2.2]` Persist `transcript.json` + metadata on stop; status lifecycle. **M** → FR-STORE-1
|
||||
- `[T2.3]` Meetings list (create/open/delete) UI + commands. **M** → FR-STORE-5
|
||||
- `[T2.4]` `notes`: assemble Markdown (speaker-tagged) from segments; render + raw side-by-side. **M** → FR-NOTE-1/6
|
||||
- `[T2.5]` Markdown editor with toolbar (headings/bold/italic/lists/checkboxes); `update_notes`. **M** → FR-NOTE-2
|
||||
- `[T2.6]` Export single meeting to `.md` and bundle (audio+transcript+notes). **M** → FR-NOTE-3, FR-STORE-3
|
||||
- `[T2.10]` Finalize path honors retention: delete working WAV only **after** transcript finalized; set `recorded`/`audio_path` (FR-REC-4). **S** → FR-REC-1/4
|
||||
- `[T2.7]` Auto-save notes/transcript at intervals. **S** → FR-REL-2
|
||||
- `[T2.8]` Startup recover-scan: audio-without-transcript → `recovering`; re-run flow. **M** → FR-REL-1, NFR-REL-3
|
||||
- `[T2.9]` Storage settings: base directory + retention policy (size/age) + background enforcement. **M** → FR-STORE-2
|
||||
|
||||
**Acceptance:** Kill the app mid-meeting; on relaunch the meeting is recoverable from audio and
|
||||
transcription can be regenerated. Notes edit + `.md` export round-trip. Retention deletes the
|
||||
oldest meeting when the cap is exceeded and never touches a recording in progress. (Tests: P2 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Hardware acceleration + model management
|
||||
**Goal:** detect and use NPU/GPU; switch models; show the active backend; batch re-transcribe.
|
||||
|
||||
- `[T3.1]` `hardware`: enumerate CPU/GPU (DXGI) + NPU (ONNX/Windows ML); rank backends. **L** → FR-HW-1
|
||||
- `[T3.2]` Backend selection (auto order + manual override); construct matching Transcriber. **M** → FR-HW-2
|
||||
- `[T3.3]` whisper.cpp accelerated builds behind Cargo features (CUDA/Vulkan); CPU always builds. **L** → FR-TRX-5, NFR-MNT-4
|
||||
- `[T3.4]` NPU path: `ort` + DirectML Whisper-ONNX `Transcriber` (same trait/output). **L** → FR-TRX-5, ADR-0004
|
||||
- `[T3.5]` Graceful fallback NPU→GPU→CPU on failure; `hardware://changed` notice. **M** → FR-HW-4, NFR-REL-2
|
||||
- `[T3.6]` Show active backend, model size, est. RTF in Settings + recording bar. **S** → FR-HW-3
|
||||
- `[T3.7]` Model management UI: choose Whisper size; download/remove with progress. **M** → FR-MODEL-1
|
||||
- `[T3.8]` Batch re-transcription with a larger model post-meeting. **M** → FR-TRX-3
|
||||
- `[T3.9]` "Low overhead" preset (CPU + small model + no realtime summary + reduced indexing). **S** → NFR-RES-3, NFR-PERF-3
|
||||
|
||||
**Acceptance:** On a machine with a supported GPU/NPU, the active backend is detected and shown,
|
||||
and transcription is measurably faster than the CPU baseline; forcing a backend failure falls back
|
||||
cleanly to CPU without aborting. Output segments are equivalent across backends. (Tests: P3 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Speaker diarization & naming
|
||||
**Goal:** segments get speaker labels; users name/merge speakers during and after meetings.
|
||||
|
||||
- `[T4.1]` `diarization`: integrate `sherpa-onnx` (segmentation+embedding+clustering) via FFI. **L** → FR-SPK-1, ADR-0005
|
||||
- `[T4.2]` Align speaker spans to transcript segments by timestamp overlap. **M** → FR-SPK-1
|
||||
- `[T4.3]` Live provisional speaker turns from cheap segmentation during recording. **M** → FR-SPK-2
|
||||
- `[T4.4]` In-session naming: name a speaker → applies to past & future segments. **M** → FR-SPK-2
|
||||
- `[T4.5]` Post-meeting review screen: rename + **merge** over-split speakers. **M** → FR-SPK-3
|
||||
- `[T4.6]` Persist speaker→name mapping; apply at render/export (non-destructive). **S** → FR-SPK-5
|
||||
- `[T4.7]` Diarization models in model-management; download/select. **S** → FR-MODEL-1
|
||||
|
||||
**Acceptance:** A two-person recording yields ≥2 speakers; naming a speaker relabels all their
|
||||
segments live; merging two IDs into one updates notes and export; renaming never rewrites segment
|
||||
speaker IDs in storage. (Tests: P4 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Local LLM integration
|
||||
**Goal:** configure a local LLM and get summaries, decisions, and editable action items.
|
||||
|
||||
- `[T5.1]` `llm`: Ollama client (`/api/tags`, `/api/chat` streaming) behind `LlmProvider`. **M** → FR-LLM-1, ADR-0007
|
||||
- `[T5.2]` Provider settings: Ollama | custom endpoint | off; `llm_status` + reachability. **M** → FR-LLM-1
|
||||
- `[T5.3]` Local-endpoint guard + "data leaves WA" banner for remote/proxy endpoints. **S** → FR-LLM-6, FR-SEC-1
|
||||
- `[T5.4]` Prompt assembly: transcript + metadata + optional template → summary/decisions/actions. **M** → FR-LLM-2
|
||||
- `[T5.5]` Stream tokens to summary panel (`llm://token`/`done`); persist `summary.json`. **M** → FR-LLM-4
|
||||
- `[T5.6]` Parse action items into editable list; confirm → `action_items` rows. **M** → FR-LLM-3
|
||||
- `[T5.7]` Guided provider install + hardware-aware model suggestions when none detected. **M** → FR-LLM-5
|
||||
|
||||
**Acceptance:** With Ollama running, a finished meeting produces a streamed summary plus a list of
|
||||
action items the user can edit and confirm; with the LLM off or unreachable, capture/transcription/
|
||||
notes still work and the UI degrades gracefully. (Tests: P5 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Calendar & Outlook .pst integration
|
||||
**Goal:** import `.pst` events/attendees, attach recordings to events, suggest speaker names.
|
||||
|
||||
- `[T6.1]` `calendar`: `.pst` reader (`outlook-pst`; libpff fallback) → events + attendees. **L** → FR-CAL-1, ADR-0008
|
||||
- `[T6.2]` `import_pst` command with optional password; progress events; persist events/participants. **M** → FR-CAL-1
|
||||
- `[T6.3]` Calendar/meetings view: upcoming/recent; attach a recording to an event. **M** → FR-CAL-2
|
||||
- `[T6.4]` Pre-meeting context panel: title, organizer, participants. **S** → FR-CAL-3
|
||||
- `[T6.5]` Participant-aware naming: attendee dropdown in speaker UI + "add new name". **M** → FR-SPK-4
|
||||
- `[T6.6]` Link recording ↔ historical `.pst` event; continuity of named speakers. **M** → FR-CAL-4
|
||||
|
||||
**Acceptance:** Importing a sample `.pst` yields events with attendees; selecting an event prefills
|
||||
the speaker-naming dropdown; PST parse failure is non-fatal and surfaced clearly. (Tests: P6 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — UX polish, themes, accessibility, privacy panel
|
||||
**Goal:** the modern, fast, themeable, accessible experience.
|
||||
|
||||
- `[T7.1]` Final three-pane layout (list · transcript/notes · summary/actions/participants). **M** → FR-UX-1
|
||||
- `[T7.2]` Light/dark themes following system + override; tokens/contrast pass. **M** → FR-UX-2
|
||||
- `[T7.3]` Live waveform/level meter; device-change resilience. **M** → FR-CAP-5/6
|
||||
- `[T7.4]` Keyboard shortcuts (record start/stop, view toggles, template apply). **S** → FR-UX-3
|
||||
- `[T7.5]` Screen-reader labels, focus order, ARIA; contrast audit. **M** → FR-UX-4
|
||||
- `[T7.6]` Privacy panel: active backend + LLM endpoint + local-only confirmation; `privacy_self_check`. **M** → FR-SEC-2
|
||||
- `[T7.7]` Startup/perf pass: cold start < ~2 s, lazy-load models, idle audit. **M** → NFR-PERF-4, NFR-RES-1
|
||||
|
||||
**Acceptance:** Themes switch with the OS and override; keyboard-only operation works for core
|
||||
flows; a contrast/screen-reader audit passes; the privacy panel accurately reflects egress; cold
|
||||
start meets target. (Tests: P7 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Advanced features
|
||||
**Goal:** templates, search/tagging, reminders, richer export, encryption.
|
||||
|
||||
- `[T8.1]` Note templates by meeting type (sections per type); apply on creation. **M** → FR-NOTE-5
|
||||
- `[T8.2]` Full-text search (FTS5) across transcripts + notes. **M** → FR-SEARCH-1
|
||||
- `[T8.3]` Tagging + list filters (date/tag/participant). **M** → FR-SEARCH-2
|
||||
- `[T8.4]` PDF/Word export via local conversion. **M** → FR-NOTE-4
|
||||
- `[T8.5]` Bulk export by date range/tag. **S** → FR-STORE-4
|
||||
- `[T8.6]` Local reminders / OS notifications for action items. **M** → FR-CAL-5
|
||||
- `[T8.7]` Multi-language transcription + UI localization scaffold. **M** → FR-TRX-4
|
||||
- `[T8.8]` Optional at-rest encryption + password-protected vault. **L** → FR-SEC-3
|
||||
- `[T8.9]` (Later) Microsoft Graph calendar source behind `CalendarSource` (opt-in, consented). **L** → FR-CAL-6
|
||||
|
||||
**Acceptance:** Search returns relevant meetings; templates structure new notes; PDF/Word exports
|
||||
render correctly offline; reminders fire locally; encryption (when enabled) protects the storage
|
||||
root and unlocks with the vault password. (Tests: P8 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 — Remote sync / upload
|
||||
**Goal:** optionally upload meeting artifacts to user-configured destinations — off by default,
|
||||
self-hostable first. (ADR-0010.) Can begin in parallel after Phase 2 since it only needs the
|
||||
storage/file layer; OAuth sub-phase is independent of the WebDAV one.
|
||||
|
||||
### 9a — Core + WebDAV primary set (Nextcloud, ownCloud, Cloudreve, Seafile, Synology)
|
||||
- `[T9.1]` `sync` module: `SyncTarget`/`SyncManager` traits; `sync_targets`/`sync_jobs` schema (migration 0002). **M** → FR-SYNC-1
|
||||
- `[T9.2]` `WebDavTarget`: PUT/MKCOL/PROPFIND; per-provider base paths; chunked upload for large files. **L** → FR-SYNC-2
|
||||
- `[T9.3]` Target config UI + `add/update/remove/list_sync_targets`; per-target artifact + trigger selection. **M** → FR-SYNC-2/3
|
||||
- `[T9.4]` `test_sync_target` (reachability + auth) with provider-specific setup hints (e.g. enable SeafDAV). **M** → FR-SYNC-4
|
||||
- `[T9.5]` Durable queue: enqueue on finalize / "Upload now"; backoff retry; SHA-256 skip-if-unchanged; `sync://job` events. **L** → FR-SYNC-5
|
||||
- `[T9.6]` Credentials in OS credential store (`keyring`); never in settings/DB. **M** → FR-SYNC-6, NFR-SEC-4
|
||||
- `[T9.7]` TLS enforcement; refuse plaintext http except explicit per-target LAN opt-in. **S** → FR-SYNC-7
|
||||
- `[T9.8]` Sync state + target labeling in UI (self-hosted vs third-party); wire into privacy panel + `privacy_self_check`. **M** → FR-SYNC-8, FR-SEC-2
|
||||
|
||||
### 9b — Secondary OAuth targets (OneDrive, Dropbox, Box)
|
||||
- `[T9.9]` OAuth 2.0 PKCE with loopback redirect; token storage in credential store; `begin_oauth_link`. **L** → FR-SYNC-9
|
||||
- `[T9.10]` `OneDriveTarget` (MS Graph), `DropboxTarget`, `BoxTarget`: upload + resumable sessions behind `SyncTarget`. **L** → FR-SYNC-9
|
||||
- `[T9.11]` Third-party "data leaves your device" banner + explicit enable acknowledgment. **S** → FR-SYNC-8
|
||||
|
||||
### 9c — Optional
|
||||
- `[T9.12]` Client-side encryption before upload (uses the Phase 8 vault): destination holds only ciphertext. **M** → FR-SYNC-10
|
||||
|
||||
**Acceptance:** With sync disabled (default), the egress allowlist contains no sync hosts and nothing
|
||||
uploads. After adding and enabling a WebDAV target (tested against a real Nextcloud/Seafile or an
|
||||
`rclone serve webdav` instance), finishing a meeting uploads the selected artifacts to the correct
|
||||
remote path; interrupting the network leaves a `failed` job that retries and completes; an unchanged
|
||||
re-upload is skipped. Credentials never appear in `settings.json`/`wa.db`. Third-party targets show
|
||||
the leaves-your-device banner. (Tests: P9 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Phase 10 — External AI & coding-agent integration
|
||||
**Goal:** optionally use hosted AI for summaries, and hand a meeting off to the user's coding agent
|
||||
to "get started right away." (ADR-0011.) All off by default. Needs Phase 5 (LLM) for distillation;
|
||||
otherwise independent.
|
||||
|
||||
### 10a — Cloud summary providers (Layer 1)
|
||||
- `[T10.1]` Extend `LlmProvider` with `OpenAiCompatProvider` (`/v1/chat/completions`) and `AnthropicProvider` (`/v1/messages`). **M** → FR-AI-1
|
||||
- `[T10.2]` `set_llm_provider`; API keys in OS credential store; host added to egress allowlist; third-party banner. **M** → FR-AI-2, FR-SEC-1
|
||||
- `[T10.3]` Per-use provider selection + active-provider display. **S** → FR-AI-3
|
||||
|
||||
### 10b — WhispAssist as an MCP server + feature briefs (Layer 2, the primary handoff)
|
||||
- `[T10.4]` `mcp` module on `rmcp`: loopback MCP server, **off by default**, token-gated; Streamable HTTP (`/mcp`) + stdio adapter. **L** → FR-MCP-1/6, NFR-SEC-5
|
||||
- `[T10.5]` Tools-first surface: `list_recent_meetings`, `get_transcript`, `get_action_items`, `get_feature_brief`. **M** → FR-MCP-2
|
||||
- `[T10.6]` `FeatureBriefBuilder`: distill transcript → structured brief; `create/list/get_feature_brief`; persist to `briefs/`. **L** → FR-MCP-4
|
||||
- `[T10.7]` Scope control (`none|selected|all`, recordings excluded by default) + `set_brief_exposed`. **M** → FR-MCP-3
|
||||
- `[T10.8]` Disclosure UI ("connected agents may forward data") + `mcp_access_log` audit. **M** → FR-MCP-5
|
||||
- `[T10.9]` Privacy panel + `privacy_self_check` show MCP state and confirm it adds no egress. **S** → FR-MCP-7, FR-SEC-2
|
||||
|
||||
### 10c — Push & task-tracker handoff (Layer 3, later)
|
||||
- `[T10.10]` `AgentRunner`: spawn `claude -p` / `codex exec` / `opencode run` / `copilot` headless against a repo; stream output. **L** → FR-AGENT-1
|
||||
- `[T10.11]` `IssueTracker`: create a GitHub issue from a brief; optional assign-to-Copilot-cloud. **M** → FR-AGENT-2
|
||||
|
||||
**Acceptance:** With everything off (default), no AI host is on the allowlist and no MCP port is
|
||||
open. Enabling a hosted provider routes a summary to it and shows the third-party banner. Enabling
|
||||
the MCP server lets a local agent (e.g. Claude Code) call `get_feature_brief` and receive a usable
|
||||
spec; the access is logged; WA opens no outbound socket for it (egress test unchanged). Recordings
|
||||
are never served unless explicitly allowed. (Tests: P10 suite.)
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting (every phase)
|
||||
- Privacy invariant FR-SEC-1 verified continuously: the CI network test asserts egress stays within
|
||||
the settings-derived allowlist (LLM + enabled sync hosts + model downloads). See `06-test-strategy.md`.
|
||||
- `cargo fmt`/`clippy -D warnings` + Prettier/ESLint clean.
|
||||
- Update `docs/` whenever a contract or behavior changes (CLAUDE.md rule).
|
||||
|
||||
## Dependency graph (high level)
|
||||
```
|
||||
P1 ─► P2 ─► P3 ─► P4 ─► P5 ─► P10 (AI/MCP; 10a/10b need P5's LLM)
|
||||
│ └► P6 ─► P7 ─► P8
|
||||
└► P9 (sync; needs only P2's storage layer — can run in parallel)
|
||||
P5 and P6 can proceed in parallel after P4; P7 needs P3–P6 features to polish; P8 last.
|
||||
P9 needs only P2; 9b (OAuth) is independent of 9a (WebDAV); 9c needs P8's vault.
|
||||
P10 needs P5; 10b (MCP) is the priority; 10c (push/issue) is later and optional.
|
||||
```
|
||||
|
||||
## Suggested first milestone (thin vertical slice)
|
||||
T1.1 → T1.2 → T1.5 → T1.6 → T2.1 → T2.2 → T2.4 gives a usable "record → live transcript → saved
|
||||
Markdown notes" loop — the smallest thing worth dogfooding.
|
||||
@@ -0,0 +1,147 @@
|
||||
# 06 — Test Strategy
|
||||
|
||||
Testing mirrors the architecture (per-service) and the roadmap (per-phase). Every requirement ID
|
||||
has at least one test that asserts its acceptance criteria. A phase is "done" only when its suite
|
||||
is green and the cross-cutting gates pass.
|
||||
|
||||
## Test levels
|
||||
|
||||
| Level | Scope | Tooling |
|
||||
|---|---|---|
|
||||
| Unit | One module/function; pure logic, parsers, mappers | Rust `#[cfg(test)]`; Vitest (frontend) |
|
||||
| Service/integration | One service against real deps (SQLite, files, Ollama, models) | Rust integration tests in `/tests` |
|
||||
| Contract | Tauri command/event payload shapes match `04-api-contracts.md` | shared TS↔Rust fixtures, schema assertions |
|
||||
| End-to-end | Full flows through the UI (record→notes→summary) | Tauri WebDriver / Playwright-style harness |
|
||||
| Non-functional | Performance, resource, privacy egress | Criterion benches; resource probes; network monitor |
|
||||
|
||||
## Key fixtures (`/tests/fixtures`)
|
||||
- Short stereo WAVs with known content (1-speaker, 2-speaker, overlap, silence, device-glitch).
|
||||
- A golden `transcript.json` for deterministic alignment/merge tests.
|
||||
- A small synthetic `.pst` with known events/attendees (or a generator script) — never real mail.
|
||||
- A canned Ollama response set (recorded) for offline LLM tests + a live-Ollama opt-in lane.
|
||||
- A disposable WebDAV server for sync tests (`rclone serve webdav` or a throwaway Nextcloud/Seafile
|
||||
container) + mocked OAuth responses for the secondary providers.
|
||||
|
||||
## Per-phase suites
|
||||
|
||||
### P1 — capture + CPU transcription
|
||||
- Unit: WAV writer produces valid headers/duration; ring buffer bounded + lossless under load.
|
||||
- Service: capture a fixed audio source → byte-accurate `audio.wav` (FR-CAP-1/2).
|
||||
- Service: feed a known WAV through the CPU Transcriber → expected segments within tolerance (FR-TRX-1).
|
||||
- E2E: start→speak→segments appear via `transcript://segment`; stop→idle (FR-CAP-3, FR-TRX-2).
|
||||
- NFR: idle audit — 0 audio threads / 0 polling timers after stop (NFR-RES-1).
|
||||
- Service: retention OFF (default) → no `audio.wav` after finalize; retention ON → playable
|
||||
`audio.wav` retained; toggling mid-meeting takes effect (FR-REC-1/3).
|
||||
- Unit/E2E: consent notice shown before first retained recording; acknowledgment persists and is not
|
||||
re-shown (FR-REC-2).
|
||||
|
||||
### P2 — storage, notes, recovery
|
||||
- Unit: schema migrations apply forward cleanly; Markdown assembly is speaker-tagged + deterministic.
|
||||
- Service: create→finalize→get round-trips meeting + transcript (FR-STORE-1/5).
|
||||
- Service: **crash-recovery** — write working audio, kill process, relaunch → meeting `recovering`,
|
||||
re-transcribe succeeds (FR-REL-1, NFR-REL-3).
|
||||
- Service: retention deletes oldest past cap; refuses to touch an in-progress meeting (FR-STORE-2).
|
||||
- Service: working-WAV deletion on finalize happens only after transcript is finalized; a crash
|
||||
before finalize leaves a recoverable WAV regardless of retention setting (FR-REC-4).
|
||||
- Contract: `.md` export round-trips notes (FR-NOTE-3); bundle export contains audio+transcript+notes.
|
||||
|
||||
### P3 — hardware + models
|
||||
- Unit: backend ranking honors NPU→NVIDIA→AMD→Intel→CPU; override respected (FR-HW-2).
|
||||
- Service: same WAV across available backends → equivalent segments (FR-TRX-5).
|
||||
- Service: simulated backend failure → `hardware://changed` + CPU fallback, no abort (FR-HW-4).
|
||||
- NFR bench: accelerated RTF < CPU RTF on capable hardware; latency target met (NFR-PERF-1/2).
|
||||
- Service: model download/remove with progress; corrupt/aborted download handled (FR-MODEL-1).
|
||||
|
||||
### P4 — diarization & naming
|
||||
- Service: 2-speaker fixture → ≥2 speaker IDs; alignment overlap accuracy ≥ threshold (FR-SPK-1).
|
||||
- Unit: rename applies to all of a speaker's segments at render; storage IDs unchanged (FR-SPK-5).
|
||||
- Unit: merge(from=[S2,S3], into=S1) updates notes/export; idempotent (FR-SPK-3).
|
||||
- E2E: in-session naming updates live view (FR-SPK-2).
|
||||
|
||||
### P5 — LLM
|
||||
- Service (recorded): prompt assembly includes transcript+metadata+template; output parsed into
|
||||
action items (FR-LLM-2/3).
|
||||
- Service: streaming tokens arrive via `llm://token`; `summary.json` persisted (FR-LLM-4).
|
||||
- Unit: local-endpoint guard flags non-loopback hosts → banner (FR-LLM-6, FR-SEC-1).
|
||||
- Resilience: provider off/unreachable → core app unaffected, clear status (FR-LLM-1/5).
|
||||
- Live lane (opt-in): against a real local Ollama, end-to-end summary sanity check.
|
||||
|
||||
### P6 — calendar / .pst
|
||||
- Service: parse synthetic `.pst` → expected events + attendees; password path; corrupt file →
|
||||
non-fatal error surfaced (FR-CAL-1).
|
||||
- Unit: attendee list populates speaker-naming dropdown; "add new name" path (FR-SPK-4).
|
||||
- Service: attach meeting↔event persists link; participant continuity across meetings (FR-CAL-2/4).
|
||||
|
||||
### P7 — UX / accessibility / privacy
|
||||
- E2E: theme follows OS + override; keyboard-only core flows (FR-UX-2/3).
|
||||
- A11y: automated axe-style audit (labels/roles/contrast) passes (FR-UX-4).
|
||||
- Service: `privacy_self_check` reports only allowed egress incl. configured sync targets with
|
||||
correct third-party/TLS labeling; backend/endpoint accurate (FR-SEC-2).
|
||||
- NFR: cold-start time under target; model lazy-load verified (NFR-PERF-4).
|
||||
|
||||
### P8 — advanced
|
||||
- FTS search relevance on fixture corpus (FR-SEARCH-1); tag/filter correctness (FR-SEARCH-2).
|
||||
- Template application structures notes by type (FR-NOTE-5).
|
||||
- PDF/Word export renders offline + opens (FR-NOTE-4); bulk export by range/tag (FR-STORE-4).
|
||||
- Reminder fires locally (FR-CAL-5). Encryption: locked store unreadable without vault password;
|
||||
unlock round-trips (FR-SEC-3).
|
||||
|
||||
### P9 — remote sync / upload
|
||||
- Service: **sync disabled (default)** → no `sync_jobs` enqueued, no sync hosts in the egress
|
||||
allowlist, nothing uploads (FR-SYNC-1). This is the most important sync test.
|
||||
- Service (WebDAV against `rclone serve webdav` or a disposable Nextcloud/Seafile container):
|
||||
finalize → selected artifacts land at the correct remote path (FR-SYNC-2/3).
|
||||
- Service: `test_sync_target` succeeds with good creds, fails clearly with bad creds/URL (FR-SYNC-4).
|
||||
- Resilience: kill the network mid-upload → job `failed`, retried with backoff, completes; unchanged
|
||||
re-upload is `skipped` via SHA-256; large-file resumable/chunked path exercised (FR-SYNC-5).
|
||||
- Security: credentials never written to `settings.json`/`wa.db` (assert by scanning both);
|
||||
`list_sync_targets` never returns secrets (FR-SYNC-6). Plaintext `http://` refused unless explicit
|
||||
LAN opt-in (FR-SYNC-7).
|
||||
- Unit: host-allowlist derivation includes only enabled targets; disabling a target removes its host.
|
||||
- Secondary (mocked OAuth): PKCE loopback flow links an account; token stored in credential store;
|
||||
third-party banner shown (FR-SYNC-8/9). Optional: encrypt-before-upload yields ciphertext remotely
|
||||
(FR-SYNC-10).
|
||||
|
||||
### P10 — external AI & coding-agent integration
|
||||
- Service: **all off (default)** → no AI host on the allowlist, MCP port closed, no briefs exposed.
|
||||
This is the most important P10 test.
|
||||
- Service (10a): summary routes to a mocked OpenAI-compatible and a mocked Anthropic endpoint
|
||||
(correct `/v1/chat/completions` vs `/v1/messages` shapes); API key read from the credential store,
|
||||
never from settings/DB; host appears on the allowlist only when configured (FR-AI-1/2).
|
||||
- Service (10b): with the MCP server enabled, an in-process MCP **client** calls `get_feature_brief`
|
||||
and receives a schema-valid brief; tools-only surface (no required resources/prompts) (FR-MCP-2/4).
|
||||
- Security: MCP server binds to **loopback only** (assert it does not bind a non-loopback address),
|
||||
requires a token, and **opens no outbound socket** — the egress test is unchanged with MCP on (FR-MCP-7,
|
||||
NFR-SEC-5). Recordings are not served unless `expose_recordings` is true (FR-MCP-3).
|
||||
- Audit: every tool call appends an `mcp_access_log` row / `mcp://access` event (FR-MCP-5).
|
||||
- Unit: `FeatureBriefBuilder` produces problem/outcome/acceptance-criteria from a golden transcript.
|
||||
- (10c, when built) push: `AgentRunner` invokes a stub CLI with the brief; `IssueTracker` creates a
|
||||
mocked GitHub issue and (optional) Copilot assignment (FR-AGENT-1/2).
|
||||
|
||||
## Cross-cutting gates (run in CI every change)
|
||||
|
||||
1. **Privacy egress test (FR-SEC-1):** run a representative flow under a network monitor; assert
|
||||
**no** outbound connection except hosts on the settings-derived allowlist — the configured LLM/AI
|
||||
provider endpoint, **enabled sync target hosts**, a configured task tracker, and explicit
|
||||
model-download hosts. With nothing configured, assert zero egress. **Enabling the MCP server must
|
||||
not change the egress set** (it is inbound/loopback). Any off-allowlist connection blocks merge.
|
||||
This is the product's most important automated guarantee.
|
||||
2. **Build matrix:** CPU-only build must always pass; accelerated features (CUDA/Vulkan/DirectML)
|
||||
built on capable runners, skipped (not failed) where hardware/SDK is absent (NFR-MNT-4).
|
||||
3. **Lint/format:** `cargo fmt --check`, `cargo clippy -- -D warnings`, Prettier, ESLint, `tsc`.
|
||||
4. **Contract check:** TS client types and Rust command/event payloads validated against shared
|
||||
fixtures so the IPC boundary can't silently drift.
|
||||
|
||||
## Performance & resource benchmarks (tracked over time)
|
||||
- Real-time latency per phrase by backend (NFR-PERF-1).
|
||||
- 60-min batch wall-clock vs meeting length by backend (NFR-PERF-2).
|
||||
- Idle CPU/memory snapshot (NFR-RES-1); recording CPU on CPU-only laptop (NFR-RES-2).
|
||||
- Cold-start to interactive (NFR-PERF-4). Regressions beyond a threshold fail the bench gate.
|
||||
|
||||
## Manual / exploratory checklist (pre-release)
|
||||
Real meeting dogfood; device hot-swap mid-recording; very long (2 h) meeting; low-disk during
|
||||
capture; abrupt power loss (recover from audio); high-DPI + both themes; screen-reader pass.
|
||||
|
||||
## Definition of done (recap from CLAUDE.md)
|
||||
Compiles, no clippy warnings, requirement acceptance criteria met, named phase tests pass, docs
|
||||
updated if contracts/behavior changed.
|
||||
@@ -0,0 +1,111 @@
|
||||
# 07 — Research Findings (Stack Validation)
|
||||
|
||||
This document records the technical feasibility checks performed before committing to a stack.
|
||||
Each finding includes the conclusion that fed into the ADRs. Dates reflect research done
|
||||
**June 2026**; re-verify versions at implementation time.
|
||||
|
||||
## Summary
|
||||
|
||||
Every load-bearing assumption in the design document is supported by a mature, offline-capable,
|
||||
permissively licensed component. No blocker was found. The one area needing care is the **NPU
|
||||
path** (newer, vendor-dependent) and **`.pst` parsing** (use a battle-tested library, treat as
|
||||
read-only). Both have viable primary + fallback options.
|
||||
|
||||
## Findings
|
||||
|
||||
### App shell — Tauri 2 ✅
|
||||
Tauri 2.0 reached stable in **October 2024** and is the current major line. On Windows it
|
||||
renders through **WebView2** (the OS Chromium-based webview) rather than bundling a browser,
|
||||
which is the key reason its binaries and memory footprint are far smaller than Electron's —
|
||||
directly serving WA's "fast, low memory" goal. Cross-platform (Windows/macOS/Linux) leaves the
|
||||
door open for the design doc's "future" non-Windows support without a rewrite.
|
||||
→ **ADR-0001.**
|
||||
|
||||
### Frontend — Svelte ✅
|
||||
Svelte compiles components to small imperative JS with no runtime framework shipped, giving the
|
||||
smallest bundle and lowest per-view memory among mainstream options — consistent with the
|
||||
low-overhead NFRs. Pairs cleanly with Tauri via Vite. → **ADR-0002.**
|
||||
|
||||
### Audio capture — WASAPI loopback in Rust ✅
|
||||
The `wasapi` crate provides safe Rust access to WASAPI including a **loopback** example
|
||||
(capture system output without a meeting bot). `cpal` is the cross-platform fallback. Known
|
||||
constraint: `AUDCLNT_STREAMFLAGS_LOOPBACK` and `AUDCLNT_STREAMFLAGS_EVENTCALLBACK` cannot be
|
||||
combined, so the capture loop must **poll** rather than rely on event callbacks. This shapes
|
||||
the audio thread design in `docs/02-architecture.md`. → **ADR-0001 / architecture.**
|
||||
|
||||
### Transcription — whisper-rs (+ ONNX for NPU) ✅
|
||||
`whisper-rs` wraps **whisper.cpp** with feature-gated acceleration: **CUDA** (NVIDIA),
|
||||
**Vulkan** (cross-vendor GPU incl. AMD/Intel), **ROCm/hipBLAS**, **Metal**, **OpenBLAS**, and
|
||||
plain CPU. whisper.cpp has **no native NPU backend**, so the NPU tier is served separately via
|
||||
**ONNX Runtime** (see below) running a Whisper-ONNX model. This split — whisper.cpp for
|
||||
CPU/GPU, ONNX for NPU — is the basis of the acceleration ladder. → **ADR-0003.**
|
||||
|
||||
### Hardware acceleration — ONNX Runtime `ort` + DirectML/Windows ML ✅
|
||||
The Rust `ort` crate exposes ONNX Runtime execution providers including **DirectML**, which can
|
||||
target an **NPU** (`device_filter = "npu"`) on Windows. Execution providers fall back in order
|
||||
(NPU → GPU → CPU) automatically when an operator is unsupported. **Windows ML** is now GA as the
|
||||
production on-device runtime on Windows 11. DirectX 12 is required for DirectML (ubiquitous on
|
||||
modern hardware). Note: Microsoft is steering new Windows ONNX work toward Windows ML over raw
|
||||
DirectML, so the hardware layer is abstracted (`hardware::Backend`) to allow swapping the
|
||||
provider without touching callers. → **ADR-0004.**
|
||||
|
||||
### Diarization — sherpa-onnx ✅
|
||||
`sherpa-onnx` (k2-fsa) provides **offline speaker diarization**: pyannote segmentation model +
|
||||
a speaker-embedding extractor (e.g. 3D-Speaker ERes2Net) + clustering, all as ONNX, fully
|
||||
offline. It ships C/C++ APIs callable from Rust via FFI. This satisfies "Speaker 1/2 …" labeling
|
||||
and post-meeting merge/rename without any cloud dependency. → **ADR-0005.**
|
||||
|
||||
### Storage — SQLite ✅
|
||||
SQLite (via `sqlx` for async or `rusqlite` for sync) is the standard embedded store, matches
|
||||
Meetily's proven model, requires no server, and supports full-text search (FTS5) for the
|
||||
transcript search requirement. → **ADR-0006.**
|
||||
|
||||
### Local LLM — Ollama HTTP ✅
|
||||
Ollama serves a REST API on **`http://localhost:11434`** with `/api/chat` (OpenAI-style
|
||||
messages), `/api/generate` (one-shot), `/api/tags` (list models), `/api/pull` (download), and
|
||||
token **streaming** via newline-delimited JSON. WA calls only this local endpoint; a "custom
|
||||
endpoint" option lets advanced users point at any OpenAI-compatible local server. → **ADR-0007.**
|
||||
|
||||
### Outlook `.pst` — outlook-pst / libpff ✅
|
||||
The Rust `outlook-pst` crate offers **read-only** access to PST files modeled on the published
|
||||
MS-PST specification (emails, folders, and — relevant here — calendar appointments/attendees).
|
||||
The mature C library **libpff** (with `pffexport`) is the fallback for tricky/encrypted files
|
||||
via FFI or a sidecar. Treat PST strictly as read-only input. → **ADR-0008.**
|
||||
|
||||
### External AI / MCP — all targets are MCP clients; Rust SDK exists ✅
|
||||
- **Claude** (Code & Desktop): MCP client over stdio + HTTP; Claude Code also runs **headless**
|
||||
(`claude -p`, `--bare`) and ships the **Claude Agent SDK** (Python/TS) — usable for the optional
|
||||
push path. Claude Code can itself act as an MCP server.
|
||||
- **OpenAI Codex** (CLI): MCP via **stdio** (local child process) **and remote Streamable HTTP**
|
||||
(OAuth/bearer); managed with `codex mcp`; `codex exec` for headless. Has an Agents SDK.
|
||||
- **GitHub Copilot:** MCP across **IDE, CLI, and the cloud agent**. Caveat: the cloud agent supports
|
||||
MCP **tools only** (no resources/prompts) and **no OAuth-remote** MCP, and runs in GitHub's cloud
|
||||
(can't reach a localhost server) — so WA exposes capabilities as **tools**, and Copilot *cloud* is
|
||||
better served by the GitHub-issue handoff.
|
||||
- **OpenCode:** full MCP client (local + remote), 75+ LLM providers, has an SDK.
|
||||
- **Transports:** MCP defines exactly **stdio** and **Streamable HTTP** (HTTP+SSE deprecated). For a
|
||||
long-running desktop app holding shared state, a **loopback Streamable HTTP** `/mcp` endpoint fits;
|
||||
a thin **stdio** adapter covers agents that spawn their server.
|
||||
- **Rust SDK:** the official **`rmcp`** crate (`modelcontextprotocol/rust-sdk`, v0.16+, `server`
|
||||
feature) implements tools/resources/prompts with pluggable transports — so WA's Rust core can host
|
||||
the server natively and lightweightly.
|
||||
- **Summary APIs:** **Anthropic** is native `/v1/messages` (needs its own adapter); **OpenAI** is
|
||||
`/v1/chat/completions`; most others are OpenAI-compatible. → **ADR-0011** (and ADR-0007 update).
|
||||
|
||||
## Open items to confirm at build time
|
||||
- Exact `whisper-rs` version and which acceleration features build cleanly on the Windows CI image.
|
||||
- `ort` ↔ ONNX Runtime binary version pinning for the DirectML/NPU path on target Windows builds.
|
||||
- `outlook-pst` coverage of password-protected and very large PST files; fall back to libpff if gaps.
|
||||
- Whether to consume `sherpa-onnx` as a prebuilt C library or build from source in CI.
|
||||
|
||||
## Sources
|
||||
- [Tauri 2.0 Stable Release](https://v2.tauri.app/blog/tauri-20/) · [Tauri vs Electron 2026](https://tech-insider.org/tauri-vs-electron-2026/)
|
||||
- [whisper-rs (crates.io)](https://crates.io/crates/whisper-rs) · [whisper-rs repo](https://github.com/tazz4843/whisper-rs)
|
||||
- [ort execution providers](https://ort.pyke.io/perf/execution-providers) · [ONNX Runtime DirectML EP](https://onnxruntime.ai/docs/execution-providers/DirectML-ExecutionProvider.html) · [Windows ML execution providers](https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/supported-execution-providers)
|
||||
- [sherpa-onnx speaker diarization](https://k2-fsa.github.io/sherpa/onnx/speaker-diarization/index.html)
|
||||
- [wasapi (crates.io)](https://crates.io/crates/wasapi) · [cpal](https://github.com/RustAudio/cpal)
|
||||
- [Ollama API docs](https://github.com/ollama/ollama/blob/main/docs/api.md)
|
||||
- [outlook-pst (docs.rs)](https://docs.rs/outlook-pst/latest/outlook_pst/) · [libpff](https://github.com/libyal/libpff)
|
||||
- MCP clients: [Copilot MCP + coding agent](https://docs.github.com/en/copilot/concepts/agents/coding-agent/mcp-and-coding-agent) · [Codex MCP](https://developers.openai.com/codex/mcp) · [OpenCode MCP/config](https://opencode.ai/docs/config/) · [Claude Code headless](https://code.claude.com/docs/en/headless)
|
||||
- MCP protocol: [Transports spec](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports) · [Rust SDK `rmcp`](https://github.com/modelcontextprotocol/rust-sdk) ([docs.rs](https://docs.rs/rmcp))
|
||||
- Summary APIs: [Anthropic Messages vs OpenAI Chat Completions](https://portkey.ai/blog/open-ai-responses-api-vs-chat-completions-vs-anthropic-anthropic-messages-api/)
|
||||
@@ -0,0 +1,49 @@
|
||||
# ADR-0001 — Application shell: Tauri 2
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Deciders:** WA core team
|
||||
- **Context source:** Design doc §"Desktop UI Layer", NFR resource-usage requirements
|
||||
|
||||
## Context
|
||||
|
||||
The design doc leaves the UI framework open (WPF/.NET, WinUI 3, Electron, or native) and the
|
||||
user's only hard constraint is: **fast and low memory footprint**. WA is Windows-first but the
|
||||
doc mentions possible future Windows-on-ARM and (implicitly) cross-platform support. The app
|
||||
must idle like a lightweight background process (NFR-RES-1) yet present a modern, themeable UI
|
||||
with waveforms, live transcript streaming, and rich note editing.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **Electron + native modules** — fastest UI development, largest ecosystem, but bundles
|
||||
Chromium → high baseline memory and large binaries. Conflicts with the low-overhead NFRs.
|
||||
2. **.NET + WinUI 3** — excellent native Windows + hardware story (Windows ML, DirectML,
|
||||
WASAPI), good performance. But heavier runtime than Rust/native, Windows-locked, and a
|
||||
larger install. Strong second choice.
|
||||
3. **C++ / Qt or fully native Win32** — lowest overhead, maximal control. Slowest UI iteration,
|
||||
most code, hardest to staff for an open-source project.
|
||||
4. **Tauri 2 (Rust core + WebView2 + web UI)** — uses the OS WebView2 instead of bundling a
|
||||
browser → small binaries, low idle memory; Rust core gives native-class performance for
|
||||
audio/ML; web frontend keeps modern UX fast to build; cross-platform for the future.
|
||||
|
||||
## Decision
|
||||
|
||||
**Use Tauri 2.** It is the best fit for "fast + low memory + modern UI + open source":
|
||||
WebView2 keeps the footprint near-native while the Rust core handles WASAPI capture, whisper.cpp
|
||||
inference, ONNX Runtime, SQLite, and HTTP to Ollama with no GC pauses and direct access to the
|
||||
exact crates validated in `07-research-findings.md`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** small footprint; one language (Rust) for all system-level services; mature
|
||||
access to `wasapi`, `whisper-rs`, `ort`, `sherpa-onnx`, `outlook-pst`; future macOS/Linux/ARM
|
||||
reachable without a rewrite.
|
||||
- **Negative / risks:** depends on WebView2 runtime (preinstalled on Win11, bootstrappable on
|
||||
Win10); WASAPI loopback must poll (no event callback) — handled in the audio thread design;
|
||||
web↔Rust boundary requires disciplined command/event contracts (`docs/04-api-contracts.md`).
|
||||
- **Rejected because:** Electron's memory cost contradicts the primary constraint; WinUI 3 and
|
||||
native were viable but more Windows-locked / slower to iterate for an OSS project.
|
||||
|
||||
## Revisit if
|
||||
WebView2 proves unreliable for real-time waveform/transcript rendering, or NPU tooling becomes
|
||||
materially better from .NET than from Rust.
|
||||
@@ -0,0 +1,39 @@
|
||||
# ADR-0002 — Frontend framework: Svelte + TypeScript
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"UX and Performance Principles", NFR-RES-1
|
||||
|
||||
## Context
|
||||
|
||||
Tauri (ADR-0001) renders a web frontend. We need light/dark theming, a three-pane layout
|
||||
(meetings list · transcript/notes · summary/participants), live-updating transcript and
|
||||
waveform, and a Markdown editor — all while keeping per-view memory and render cost low.
|
||||
|
||||
## Options considered
|
||||
|
||||
- **React** — largest ecosystem and component availability, but ships a runtime + virtual DOM;
|
||||
heavier than necessary for our footprint goals.
|
||||
- **Vue** — middle ground; runtime overhead between React and Svelte.
|
||||
- **Svelte** — compiles components to small imperative JS with no shipped framework runtime;
|
||||
smallest bundles and lowest memory; fine-grained reactivity ideal for high-frequency
|
||||
transcript/waveform updates.
|
||||
- **SolidJS** — comparably lean, but smaller ecosystem/talent pool than Svelte.
|
||||
|
||||
## Decision
|
||||
|
||||
**Use Svelte with TypeScript, bundled by Vite.** Best memory/performance profile and a good fit
|
||||
for streaming UI updates, with enough ecosystem maturity for an OSS project.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** smallest footprint; simple, fast reactive updates for live transcript; first-class
|
||||
Vite integration with Tauri.
|
||||
- **Negative:** smaller component ecosystem than React (mitigated: WA's UI is bespoke); team must
|
||||
know Svelte 5 reactivity model.
|
||||
- Theming via CSS custom properties + `prefers-color-scheme`, with a user override stored in
|
||||
settings (FR-UX-2).
|
||||
|
||||
## Revisit if
|
||||
We need a large third-party component (e.g. complex data grid) only available for React/Vue, or
|
||||
hiring constraints favor React.
|
||||
@@ -0,0 +1,41 @@
|
||||
# ADR-0003 — Transcription engine: whisper-rs (whisper.cpp) with pluggable backends
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"Transcription Engine", §"Transcription Modes"
|
||||
|
||||
## Context
|
||||
|
||||
WA needs accurate, offline, multilingual speech-to-text with both a low-latency streaming mode
|
||||
and a higher-accuracy batch mode, runnable across CPU and several GPU vendors, and (separately)
|
||||
on NPUs.
|
||||
|
||||
## Decision
|
||||
|
||||
Use **`whisper-rs`** (Rust bindings to **whisper.cpp**) as the primary engine, behind a
|
||||
`transcription::Transcriber` trait so the concrete engine is swappable. whisper.cpp gives:
|
||||
- CPU (AVX2/AVX-512) baseline that runs anywhere,
|
||||
- **Vulkan** for cross-vendor GPU (AMD/Intel/NVIDIA),
|
||||
- **CUDA** for NVIDIA fast path,
|
||||
- selectable model sizes (tiny→large) for the speed/accuracy and "low-overhead" presets,
|
||||
- multilingual models for FR-LANG-1.
|
||||
|
||||
The **NPU** path is *not* served by whisper.cpp (it has no NPU backend); it is a second
|
||||
`Transcriber` implementation using ONNX Runtime — see ADR-0004. Both implementations satisfy the
|
||||
same trait and emit the same `TranscriptSegment` stream.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** one mature dependency covers CPU + all desktop GPUs; model-size switching maps
|
||||
directly to the performance presets; trait boundary lets the NPU/ONNX engine slot in without
|
||||
touching callers (audio, storage, UI).
|
||||
- **Negative:** two engines (whisper.cpp + ONNX) to maintain for full hardware coverage;
|
||||
whisper.cpp diarization is weak, so diarization is a separate component (ADR-0005); building
|
||||
acceleration features on Windows CI needs the right toolchain (CUDA/Vulkan SDKs) — gated by
|
||||
Cargo features so CPU-only always builds.
|
||||
- **Streaming mode:** feed fixed audio windows with overlap; emit interim segments, then finalize.
|
||||
- **Batch mode:** optional re-run with a larger model after stop for maximum accuracy.
|
||||
|
||||
## Revisit if
|
||||
A single engine gains solid NPU + GPU + CPU coverage (would let us drop the ONNX engine), or a
|
||||
materially better local ASR model (e.g. Parakeet) outperforms Whisper for our languages.
|
||||
@@ -0,0 +1,46 @@
|
||||
# ADR-0004 — Hardware acceleration strategy & detection ladder
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"Hardware Detection and Transcription Strategy"
|
||||
|
||||
## Context
|
||||
|
||||
The design mandates a strict preference order **NPU → NVIDIA → AMD → Intel → CPU**, detected at
|
||||
startup and on demand, with graceful fallback and the chosen backend surfaced to the user.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce a `hardware` module that performs **capability detection** and returns a ranked list of
|
||||
available `Backend`s; the transcription layer selects the highest-ranked one and constructs the
|
||||
matching `Transcriber`.
|
||||
|
||||
Detection mapping:
|
||||
|
||||
| Tier | Detection | Execution path |
|
||||
|---|---|---|
|
||||
| NPU | ONNX Runtime / Windows ML enumeration; DirectML `device_filter="npu"` | ONNX `Transcriber` (Whisper-ONNX) via `ort` + DirectML |
|
||||
| NVIDIA GPU | adapter enumeration (DXGI) + CUDA availability | whisper.cpp **CUDA** |
|
||||
| AMD GPU | DXGI adapter; Vulkan device present | whisper.cpp **Vulkan** (DirectML alt.) |
|
||||
| Intel GPU | DXGI adapter; Vulkan device present | whisper.cpp **Vulkan** |
|
||||
| CPU | always | whisper.cpp **CPU** (AVX2/AVX-512 if present) |
|
||||
|
||||
- ONNX Runtime execution providers also fall back internally (NPU→GPU→CPU) per-operator, giving a
|
||||
second safety net on the NPU path.
|
||||
- The selected backend, model size, and an estimated real-time factor are reported via the
|
||||
`hardware_status` command and shown in Settings + the recording bar (FR-HW-3).
|
||||
- A user override lets advanced users pin a specific backend (FR-HW-2); "Low overhead" preset
|
||||
forces CPU + small model + no real-time summarization (NFR-RES-3).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** one abstraction satisfies the full ladder; per-operator EP fallback makes the
|
||||
NPU path robust; clean place to add future vendors/runtimes (Windows ML is abstracted behind
|
||||
`Backend`, so a DirectML→Windows ML migration is local to this module).
|
||||
- **Negative:** NPU support is hardware/driver dependent and the newest, least-tested tier —
|
||||
must degrade silently to GPU/CPU and never hard-fail; requires pinning `ort`↔ONNX Runtime
|
||||
binary versions on Windows builds.
|
||||
|
||||
## Revisit if
|
||||
Windows ML supersedes DirectML for our models, or a vendor SDK gives materially better NPU
|
||||
throughput than the ONNX/DirectML path.
|
||||
@@ -0,0 +1,37 @@
|
||||
# ADR-0005 — Speaker diarization: sherpa-onnx
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"Speaker Diarization and Naming"
|
||||
|
||||
## Context
|
||||
|
||||
WA must label transcript segments by speaker (Speaker 1/2…), allow naming during and after a
|
||||
meeting, support merging over-split speakers, and feed calendar participant names into the
|
||||
naming UI — all offline.
|
||||
|
||||
## Decision
|
||||
|
||||
Use **`sherpa-onnx`** offline speaker diarization: a pyannote **segmentation** model + a
|
||||
speaker-**embedding** extractor (e.g. 3D-Speaker ERes2Net) + **clustering**, all ONNX and fully
|
||||
local. Wrap it behind a `diarization::Diarizer` trait. Diarization runs as **post-processing**
|
||||
over the recorded audio (audio is the source of truth, ADR-0006), producing speaker-ID-tagged
|
||||
spans that are aligned to whisper segments by timestamp overlap.
|
||||
|
||||
During live recording, show provisional speaker turns from segmentation (cheap), then refine
|
||||
labels in the post-meeting pass; this keeps latency low while improving final accuracy.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** purpose-built, offline, ONNX (shares the runtime with the NPU transcription path);
|
||||
supports the segmentation+embedding+clustering pipeline the design assumes; models are
|
||||
downloadable and swappable.
|
||||
- **Negative:** separate models to download/manage (model-management UI, FR-MODEL-1); C/C++ FFI
|
||||
to integrate; clustering may over/under-split → the **merge** workflow (FR-SPK-3) is required,
|
||||
not optional.
|
||||
- Speaker IDs (`S1`, `S2`, …) are internal and stable per meeting; name mappings live in the DB
|
||||
and are applied at render/export time, never destructively rewritten onto segments.
|
||||
|
||||
## Revisit if
|
||||
A single model gives joint ASR + diarization with better accuracy, or whisper.cpp gains
|
||||
production diarization.
|
||||
@@ -0,0 +1,39 @@
|
||||
# ADR-0006 — Storage: SQLite index + on-disk audio/transcript files
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"Recording Persistence and Storage", §"Crash Recovery"
|
||||
|
||||
## Context
|
||||
|
||||
WA stores recordings, transcripts, notes, speakers, participants, tags, and meeting metadata
|
||||
locally; needs fast list/filter/search; must survive crashes; and must let users choose the base
|
||||
directory, set retention, and export.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Large/opaque artifacts on the filesystem:** audio (`.wav`/compressed) and the canonical
|
||||
transcript JSON live as files under the storage root, one folder per meeting.
|
||||
- **Index & relations in SQLite** (via `sqlx`): meetings, speakers, participants, tags, action
|
||||
items, and file paths; **FTS5** virtual table over transcript/notes text for full-text search
|
||||
(FR-SEARCH-1).
|
||||
- **Audio is the source of truth.** Audio is flushed to disk during capture; transcript/notes are
|
||||
derived and regenerable. Auto-save transcript/notes at intervals; on startup, detect meetings
|
||||
with audio but no finalized transcript and offer recovery (FR-REL-1).
|
||||
|
||||
Default root: `%LOCALAPPDATA%\WhispAssist` with per-meeting subfolders; user-configurable base
|
||||
directory and retention policy (size cap / age cap) in Settings (FR-STORE-2).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** embedded, serverless, zero-config; FTS5 covers search; files keep the DB small and
|
||||
make export/backup a folder copy; crash recovery is straightforward because audio persists first.
|
||||
- **Negative:** must keep DB rows and files consistent (transactional writes + a reconcile pass on
|
||||
startup); retention enforcement is a background job that must respect "audio is source of truth"
|
||||
(never orphan a meeting's audio while its transcript is mid-generation).
|
||||
- Optional at-rest encryption (NFR-SEC-3) can wrap the storage root and/or use SQLCipher; deferred
|
||||
to Phase 8.
|
||||
|
||||
## Revisit if
|
||||
We need multi-device sync (would change the data model and conflict story) — out of scope for the
|
||||
local-only product.
|
||||
@@ -0,0 +1,49 @@
|
||||
# ADR-0007 — Local LLM integration: Ollama HTTP (localhost-only)
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"Local LLM Integration (Ollama / Unsloth)"
|
||||
|
||||
## Context
|
||||
|
||||
WA augments notes with summaries, decisions, and action items via a **local** LLM, configurable
|
||||
by the user, and must never originate cloud calls for meeting content itself.
|
||||
|
||||
## Decision
|
||||
|
||||
Integrate via the **Ollama REST API** on `http://localhost:11434` behind an `llm::LlmProvider`
|
||||
trait. Use `/api/tags` to list installed models, `/api/chat` (OpenAI-style messages) for
|
||||
summarization with **streaming** token output, and `/api/pull` to assist guided model download.
|
||||
Provide three provider options in Settings:
|
||||
1. **Ollama** (default, autodetected),
|
||||
2. **Custom OpenAI-compatible endpoint** (any local server the user runs — covers Unsloth-served
|
||||
models, llama.cpp server, LM Studio, etc.),
|
||||
3. **Disabled** (notes-only, no LLM).
|
||||
|
||||
WA validates that the configured host resolves to **loopback/local**; if a user deliberately
|
||||
points at a remote/proxying endpoint, WA shows a clear banner that upstream behavior is outside
|
||||
WA's control (the design doc's Unsloth-proxy caveat).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** Ollama is the de-facto local LLM runtime with a simple, stable API and streaming;
|
||||
the trait + "custom endpoint" keeps us provider-agnostic and honors the design's pluggability;
|
||||
guided install/pull improves first-run UX with hardware-aware model suggestions.
|
||||
- **Negative:** depends on the user having Ollama (or another local server) installed → need
|
||||
detection + guided setup and a graceful "LLM unavailable, notes still work" path; summary quality
|
||||
varies by model.
|
||||
- Prompts are assembled from transcript + meeting metadata + optional template (FR-LLM-2); output
|
||||
is parsed into editable action items (FR-LLM-3) that the user confirms before they become tasks.
|
||||
|
||||
## Update (2026-06-30) — cloud providers behind the same trait
|
||||
|
||||
`LlmProvider` is extended (ADR-0011, Layer 1) with optional **hosted** providers for summaries:
|
||||
**Anthropic Messages API** (`/v1/messages` — native, not OpenAI-shaped) and an **OpenAI-compatible**
|
||||
client (OpenAI, OpenRouter, LM Studio, gateways). These are **off by default**, explicitly opt-in,
|
||||
and labeled as third-party egress; their hosts join the settings-derived egress allowlist
|
||||
(FR-SEC-1). The `is_local()` guard still distinguishes local from remote so the UI can warn. Ollama
|
||||
remains the default and the only zero-egress option.
|
||||
|
||||
## Revisit if
|
||||
A clearly better local-LLM runtime/protocol emerges, or we embed an inference engine directly
|
||||
(would remove the external dependency but enlarge the app).
|
||||
@@ -0,0 +1,38 @@
|
||||
# ADR-0008 — Calendar & Outlook .pst integration
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** Design doc §"Calendar Integration", §"Outlook .pst Integration"
|
||||
|
||||
## Context
|
||||
|
||||
WA enriches meetings with calendar context (title, organizer, attendees) and reads local Outlook
|
||||
`.pst` backups to build meeting history and suggest speaker names — all local, no cloud sync
|
||||
required.
|
||||
|
||||
## Decision
|
||||
|
||||
- **`.pst` reading:** use the Rust **`outlook-pst`** crate (read-only, MS-PST-based) to extract
|
||||
calendar appointments and attendees. Fall back to **libpff** (`pffexport`/FFI or a small
|
||||
sidecar) for password-protected or edge-case files. PST is **read-only input**; WA never writes
|
||||
to it.
|
||||
- **Live calendar:** abstract behind a `calendar::CalendarSource` trait. Phase 6 ships the PST
|
||||
source. Microsoft Graph (with explicit user consent) and local ICS files are future sources
|
||||
behind the same trait; Graph is the *one* allowed remote call and only for calendar metadata,
|
||||
never meeting content, and only with consent.
|
||||
- **Reminders:** action items become **local OS notifications** (Windows toast); optional
|
||||
calendar entry creation is deferred and, where added, stays local/consented.
|
||||
- **Participant-aware naming:** attendee names from the selected event populate the speaker-naming
|
||||
dropdown (FR-SPK-2), with an "add new name" escape hatch; mappings persist per meeting.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** delivers Granola-style context fully offline from data the user already has;
|
||||
trait keeps Graph/ICS additions non-invasive; read-only PST avoids corrupting user mail stores.
|
||||
- **Negative:** PST parsing is fiddly (encryption, large files, format variance) → keep libpff as
|
||||
a fallback and treat parse failures as non-fatal; Graph path (future) introduces OAuth + the
|
||||
product's only remote dependency, so it must be strictly opt-in and clearly labeled.
|
||||
|
||||
## Revisit if
|
||||
`outlook-pst` proves insufficient in practice (promote libpff to primary), or users need live
|
||||
Exchange more than PST (prioritize the Graph source).
|
||||
@@ -0,0 +1,45 @@
|
||||
# ADR-0009 — Optional audio recording & consent
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** User request (2026-06-30); design doc §"Audio Capture and Privacy"
|
||||
|
||||
## Context
|
||||
|
||||
WA captures system audio to transcribe it. Transcription needs audio only transiently — the design
|
||||
already treats `audio.wav` as the source of truth for crash recovery. However, **persisting** a
|
||||
full meeting recording is a distinct, higher-stakes choice: it is more sensitive than a transcript,
|
||||
and recording conversations without consent is **illegal in some jurisdictions** (e.g. all-party /
|
||||
two-party consent regions). The user wants recording to be a deliberate opt-in, saved as `.wav`,
|
||||
with a clear legal caution.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Recording is OFF by default.** A per-meeting and a global default toggle ("Record this
|
||||
meeting") control whether the captured audio is **retained** after processing.
|
||||
2. When recording is **off**, audio is still written to a working file during the session (so
|
||||
transcription and crash recovery work), but that working audio is **deleted** when the meeting
|
||||
is finalized — only the transcript/notes persist.
|
||||
3. When recording is **on**, the working audio is kept as the meeting's permanent `audio.wav`.
|
||||
4. **Consent notice.** The first time a user enables recording (and surfaced near the toggle
|
||||
thereafter), WA shows a non-blocking notice: *"Recording conversations without the consent of
|
||||
participants may be illegal in your region. Check your local recording laws."* The user must
|
||||
acknowledge once; the acknowledgment is stored. This is a caution, **not legal advice**.
|
||||
5. The recording indicator (FR-CAP-4) already makes active capture obvious; when retention is on,
|
||||
the UI additionally indicates the meeting is being **saved**.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** privacy-respecting default (transcript-only); users opt into the heavier artifact
|
||||
knowingly; the legal caution reduces the chance of inadvertent unlawful recording; reuses the
|
||||
existing working-audio path, so little new machinery.
|
||||
- **Negative / care:** the "delete working audio on finalize" path must be robust and must not race
|
||||
with crash recovery — deletion happens only after the transcript is successfully finalized
|
||||
(audio remains the source of truth until then, ADR-0006); retention/sync must treat a non-recorded
|
||||
meeting as "no audio artifact exists".
|
||||
- **Interaction with sync (ADR-0010):** only **retained** recordings are eligible to upload; a
|
||||
non-recorded meeting can still sync its transcript/notes.
|
||||
|
||||
## Revisit if
|
||||
We add region detection to tailor the consent copy, or a "record microphone too" feature (separate
|
||||
consent considerations).
|
||||
@@ -0,0 +1,72 @@
|
||||
# ADR-0010 — Remote sync / upload targets
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** User request (2026-06-30)
|
||||
|
||||
## Context
|
||||
|
||||
WA stores everything locally. The user wants the **option** to upload meeting artifacts
|
||||
(transcript, notes, summary, and optionally the `.wav` recording) to a destination they configure.
|
||||
Required (primary) targets: **Nextcloud, ownCloud, Cloudreve, Seafile**. Desired (secondary):
|
||||
**OneDrive, Dropbox, Box, Synology NAS**.
|
||||
|
||||
This must be reconciled with WA's core promise (local-only). The reconciliation: sync is an
|
||||
**explicit, user-configured export** — exactly the carve-out the privacy invariant already allows.
|
||||
It is **off by default**, never automatic without configuration, and clearly surfaced. The primary
|
||||
targets are notably all **self-hostable**, so the default story is "your data, on your server."
|
||||
|
||||
## Decision
|
||||
|
||||
### One protocol covers the entire primary set: WebDAV
|
||||
Nextcloud, ownCloud, Cloudreve, and Seafile (via SeafDAV) all expose **WebDAV (RFC 4918)**, as does
|
||||
**Synology** (WebDAV Server package). So a single `WebDavTarget` implementation serves all four
|
||||
primary targets **and** Synology:
|
||||
- Upload: `PUT`; create folders: `MKCOL`; existence/listing: `PROPFIND`.
|
||||
- Large files (`.wav`): chunked/resumable upload where the server supports it (e.g. Nextcloud
|
||||
chunked upload); otherwise single `PUT` with a size guard.
|
||||
- Per-server base paths, e.g. Nextcloud/ownCloud `…/remote.php/dav/files/<user>/`,
|
||||
Seafile `…/seafdav/`, Cloudreve `…/dav`. Seafile note: SeafDAV is **disabled by default**
|
||||
server-side and may need `LOCK` disabled — surfaced in setup help.
|
||||
- Auth: URL + username + **app password** (recommended over account password where supported).
|
||||
|
||||
### Secondary targets use provider APIs + OAuth (one impl each)
|
||||
`OneDriveTarget` (Microsoft Graph), `DropboxTarget` (Dropbox API v2), `BoxTarget` (Box API). Each
|
||||
uses **OAuth 2.0 Authorization Code + PKCE** with a **loopback redirect** (`http://127.0.0.1:<port>`)
|
||||
— desktop-appropriate, no client secret embedded. Resumable/upload-session APIs for large files.
|
||||
Synology can alternatively use its FileStation API, but WebDAV is the default path for it.
|
||||
|
||||
### Common architecture
|
||||
- A `SyncTarget` trait (`test`, `ensure_dir`, `put`, `exists`) abstracts all providers; a
|
||||
`SyncManager` owns a **durable queue** of upload jobs (per artifact × per target).
|
||||
- **What/when to upload is configurable** (FR-SYNC-3): choose artifacts (transcript / notes /
|
||||
summary / recording) and trigger (on finalize, manual "Upload now", or retry). Recording is
|
||||
uploadable only if it was retained (ADR-0009).
|
||||
- **Reliability:** queue with exponential-backoff retry; SHA-256 to skip unchanged files
|
||||
(idempotent); resumable upload for large recordings; status per job surfaced via events.
|
||||
- **Remote layout:** mirrors local meeting folders under a user-chosen base path.
|
||||
- **Credentials:** stored in the **OS credential store** (Windows Credential Manager / DPAPI),
|
||||
referenced by target id — **never** in `settings.json` or the DB.
|
||||
- **Transport security:** TLS required; plaintext `http://` is refused unless the user explicitly
|
||||
allows it for a LAN address, with a warning.
|
||||
- **Privacy labeling:** primary/self-hosted targets are labeled "your server"; third-party clouds
|
||||
(OneDrive/Dropbox/Box) carry a clear "data leaves your device to a third party" banner, mirroring
|
||||
the remote-LLM caveat (ADR-0007). Enabling any sync requires explicit acknowledgment.
|
||||
- **Optional client-side encryption** before upload (ties to FR-SEC-3): when the WA vault is
|
||||
enabled, artifacts can be encrypted locally so the destination only ever holds ciphertext —
|
||||
provider-agnostic, independent of any server-side E2EE.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** four primary targets + Synology delivered by **one** WebDAV implementation — small,
|
||||
testable surface; secondary set is additive and isolated behind the same trait; default-off +
|
||||
explicit config keeps the privacy promise intact; self-hostable primaries fit the ethos.
|
||||
- **Negative / care:** secondary targets each need an OAuth app registration (client IDs) and
|
||||
per-provider quirks (chunk sizes, path APIs) — hence "secondary"; sync introduces real network
|
||||
egress, so the privacy egress test (FR-SEC-1) must be widened to *allow only* configured target
|
||||
hosts and *fail* on any other; credential handling and TLS enforcement are security-critical.
|
||||
- **Phasing:** WebDAV primary set first, then OAuth secondary set (see Phase 9 in `05-roadmap.md`).
|
||||
|
||||
## Revisit if
|
||||
A primary target drops WebDAV, or we need provider-native features (e.g. Nextcloud Talk, Seafile
|
||||
library sharing) beyond plain file upload — would add a provider-specific path behind the trait.
|
||||
@@ -0,0 +1,88 @@
|
||||
# ADR-0011 — External AI providers & coding-agent integration
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-30
|
||||
- **Context source:** User request (2026-06-30); research in `07-research-findings.md` (MCP section)
|
||||
|
||||
## Context
|
||||
|
||||
WhispAssist already summarizes meetings with a **local** LLM (Ollama, ADR-0007). The user wants to
|
||||
also tie into external AI tools — **Claude, OpenAI Codex, GitHub Copilot, OpenCode, and eventually
|
||||
others** — for two purposes:
|
||||
|
||||
1. **Summaries / follow-ups** (what the local LLM does today), optionally using a hosted model.
|
||||
2. **Turn a meeting into actionable dev work** — e.g. a customer asks for a feature on a call and a
|
||||
developer can "get started right away" (scaffold, branch, PR) from that request.
|
||||
|
||||
These are **two different jobs** needing **two different mechanisms**. Codex, Copilot, and OpenCode
|
||||
are coding *agents*, not summarization endpoints — calling them for a summary is the wrong shape.
|
||||
|
||||
Research finding that drives the decision: **all four targets are MCP clients today** (Claude
|
||||
Code/Desktop, Codex CLI, Copilot across IDE/CLI/cloud, OpenCode), and an official **Rust** MCP SDK
|
||||
(`rmcp`) exists. So a single MCP **server** in WA's Rust core is consumable by all of them — and by
|
||||
future tools — without per-tool adapters.
|
||||
|
||||
## Decision — a layered model, MCP-server-first for handoff
|
||||
|
||||
### Layer 1 — Cloud summary providers (extends ADR-0007)
|
||||
Add hosted providers behind the existing `llm::LlmProvider` trait: **Anthropic Messages API** and an
|
||||
**OpenAI-compatible** client (covers OpenAI, OpenRouter, LM Studio, gateways). This is "use Claude/
|
||||
GPT to write summaries." Off by default; third-party "data leaves WA" banner; same provider model as
|
||||
Ollama. (Note: Anthropic's native endpoint is `/v1/messages`, not OpenAI's `/v1/chat/completions`,
|
||||
so it needs its own adapter.)
|
||||
|
||||
### Layer 2 — WhispAssist *is* an MCP server (the primary handoff; PULL model)
|
||||
WA hosts a **local MCP server**, off by default, exposing meeting context and a small set of
|
||||
**tools** (tools-first, because Copilot's cloud agent supports MCP tools but not resources/prompts):
|
||||
- `list_recent_meetings`, `get_transcript`, `get_action_items`,
|
||||
- `get_feature_brief(meeting_id)` / `create_feature_brief(...)` — the bridge primitive below.
|
||||
|
||||
The developer drives from **their own** agent ("Claude, grab the feature request from this morning's
|
||||
call and scaffold it"); the agent connects to WA as an MCP client and pulls structured context. WA
|
||||
stays **agent-agnostic** — implement once, works with Claude/Codex/Copilot/OpenCode and "eventually
|
||||
others."
|
||||
|
||||
- **Transports:** local **stdio** (agent spawns a thin WA MCP adapter) and **Streamable HTTP** on
|
||||
loopback (WA, already long-running and holding the DB, hosts `http://127.0.0.1:<port>/mcp`).
|
||||
HTTP+SSE is deprecated; do not use it.
|
||||
- **Auth/scope:** loopback-bound; token required; explicit per-server enable; **scope control** over
|
||||
which meetings/artifacts are exposed; recordings excluded unless explicitly allowed.
|
||||
- **Audit:** log what an agent read.
|
||||
|
||||
### The bridge primitive — "feature brief"
|
||||
WA distills a feature request from a transcript (via the configured LLM) into a structured,
|
||||
agent-ready spec: problem, desired outcome, acceptance criteria, target repo/context, source meeting.
|
||||
It is the object that turns "we talked about X" into "here's a branch." Exposed as the
|
||||
`get_feature_brief` MCP tool and reused by the push/issue paths (Layer 3).
|
||||
|
||||
### Layer 3 — Push & task-tracker handoff (documented now, built later)
|
||||
- **`AgentRunner`** trait: optionally spawn a local coding-agent CLI headless (`claude -p`,
|
||||
`codex exec`, `opencode run`, `copilot`) against a chosen repo to produce a branch/PR — a one-click
|
||||
"scaffold this" button in WA.
|
||||
- **Task-tracker handoff:** create a **GitHub issue** from a confirmed action item; optionally assign
|
||||
**Copilot's cloud agent**, which opens a PR. This is the right path for Copilot *cloud* (it runs in
|
||||
GitHub's cloud and cannot reach a localhost MCP server).
|
||||
|
||||
## Privacy reconciliation (critical)
|
||||
|
||||
- **The MCP server adds no WA egress.** It *listens* on loopback; WA opens no new outbound socket, so
|
||||
the egress allowlist and the CI network test are unaffected. The data only leaves the device when
|
||||
the *connected agent* sends it to its provider's cloud — outside WA's control. WA must **disclose**
|
||||
this ("WhispAssist serves this locally; the agent you connected may send it to its provider").
|
||||
- **Cloud summary providers (Layer 1) and task-tracker handoff (Layer 3) are real third-party
|
||||
egress** — off by default, explicit opt-in, clearly labeled, and added to the settings-derived
|
||||
allowlist exactly like remote sync (ADR-0010) and remote LLM (ADR-0007).
|
||||
- Everything here is **off by default**. With nothing configured, WA remains fully local.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** one MCP server covers all four agents + future ones (the "eventually others" is free);
|
||||
Rust `rmcp` keeps it lightweight; the local-only identity holds because the server is loopback and
|
||||
egress stays opt-in; the feature-brief primitive is a genuinely WA-specific value-add.
|
||||
- **Negative / care:** tools-first design required for Copilot compatibility; MCP server is an inbound
|
||||
surface, so loopback-binding + token + scope + audit are mandatory; Anthropic needs a bespoke
|
||||
adapter; the Layer-3 push/issue paths each need per-agent/per-tracker work, hence deferred.
|
||||
|
||||
## Revisit if
|
||||
A meeting-to-code standard emerges beyond MCP, or Copilot cloud gains localhost/OAuth-remote MCP
|
||||
(would let the pull model serve it directly without the issue handoff).
|
||||
Reference in new issue
Block a user