# 06 — Test Strategy

Testing mirrors the architecture (per-service) and the roadmap (per-phase). Every requirement ID
has at least one test that asserts its acceptance criteria. A phase is "done" only when its suite
is green and the cross-cutting gates pass.

## Test levels

| Level | Scope | Tooling |
|---|---|---|
| Unit | One module/function; pure logic, parsers, mappers | Rust `#[cfg(test)]`; Vitest (frontend) |
| Service/integration | One service against real deps (SQLite, files, Ollama, models) | Rust integration tests in `/tests` |
| Contract | Tauri command/event payload shapes match `04-api-contracts.md` | shared TS↔Rust fixtures, schema assertions |
| End-to-end | Full flows through the UI (record→notes→summary) | Tauri WebDriver / Playwright-style harness |
| Non-functional | Performance, resource, privacy egress | Criterion benches; resource probes; network monitor |

## Key fixtures (`/tests/fixtures`)
- Short stereo WAVs with known content (1-speaker, 2-speaker, overlap, silence, device-glitch).
- A golden `transcript.json` for deterministic alignment/merge tests.
- A small synthetic `.pst` with known events/attendees (or a generator script) — never real mail.
- A canned Ollama response set (recorded) for offline LLM tests + a live-Ollama opt-in lane.
- A disposable WebDAV server for sync tests (`rclone serve webdav` or a throwaway Nextcloud/Seafile
  container) + mocked OAuth responses for the secondary providers.

## Per-phase suites

### P1 — capture + CPU transcription
- Unit: WAV writer produces valid headers/duration; ring buffer bounded + lossless under load.
- Service: capture a fixed audio source → byte-accurate `audio.wav` (FR-CAP-1/2).
- Service: feed a known WAV through the CPU Transcriber → expected segments within tolerance (FR-TRX-1).
- E2E: start→speak→segments appear via `transcript://segment`; stop→idle (FR-CAP-3, FR-TRX-2).
- NFR: idle audit — 0 audio threads / 0 polling timers after stop (NFR-RES-1).
- Service: retention OFF (default) → no `audio.wav` after finalize; retention ON → playable
  `audio.wav` retained; toggling mid-meeting takes effect (FR-REC-1/3).
- Unit/E2E: consent notice shown before first retained recording; acknowledgment persists and is not
  re-shown (FR-REC-2).

### P2 — storage, notes, recovery
- Unit: schema migrations apply forward cleanly; Markdown assembly is speaker-tagged + deterministic.
- Service: create→finalize→get round-trips meeting + transcript (FR-STORE-1/5).
- Service: **crash-recovery** — write working audio, kill process, relaunch → meeting `recovering`,
  re-transcribe succeeds (FR-REL-1, NFR-REL-3).
- Service: retention deletes oldest past cap; refuses to touch an in-progress meeting (FR-STORE-2).
- Service: working-WAV deletion on finalize happens only after transcript is finalized; a crash
  before finalize leaves a recoverable WAV regardless of retention setting (FR-REC-4).
- Contract: `.md` export round-trips notes (FR-NOTE-3); bundle export contains audio+transcript+notes.

### P3 — hardware + models
- Unit: backend ranking honors NPU→NVIDIA→AMD→Intel→CPU; override respected (FR-HW-2).
- Service: same WAV across available backends → equivalent segments (FR-TRX-5).
- Service: simulated backend failure → `hardware://changed` + CPU fallback, no abort (FR-HW-4).
- NFR bench: accelerated RTF < CPU RTF on capable hardware; latency target met (NFR-PERF-1/2).
- Service: model download/remove with progress; corrupt/aborted download handled (FR-MODEL-1).

### P4 — diarization & naming
- Service: 2-speaker fixture → ≥2 speaker IDs; alignment overlap accuracy ≥ threshold (FR-SPK-1).
- Unit: rename applies to all of a speaker's segments at render; storage IDs unchanged (FR-SPK-5).
- Unit: merge(from=[S2,S3], into=S1) updates notes/export; idempotent (FR-SPK-3).
- E2E: in-session naming updates live view (FR-SPK-2).

### P5 — LLM
- Service (recorded): prompt assembly includes transcript+metadata+template; output parsed into
  action items (FR-LLM-2/3).
- Service: streaming tokens arrive via `llm://token`; `summary.json` persisted (FR-LLM-4).
- Unit: local-endpoint guard flags non-loopback hosts → banner (FR-LLM-6, FR-SEC-1).
- Resilience: provider off/unreachable → core app unaffected, clear status (FR-LLM-1/5).
- Live lane (opt-in): against a real local Ollama, end-to-end summary sanity check.

### P6 — calendar / .pst
- Service: parse synthetic `.pst` → expected events + attendees; password path; corrupt file →
  non-fatal error surfaced (FR-CAL-1).
- Unit: attendee list populates speaker-naming dropdown; "add new name" path (FR-SPK-4).
- Service: attach meeting↔event persists link; participant continuity across meetings (FR-CAL-2/4).

### P7 — UX / accessibility / privacy
- E2E: theme follows OS + override; keyboard-only core flows (FR-UX-2/3).
- A11y: automated axe-style audit (labels/roles/contrast) passes (FR-UX-4).
- Service: `privacy_self_check` reports only allowed egress incl. configured sync targets with
  correct third-party/TLS labeling; backend/endpoint accurate (FR-SEC-2).
- NFR: cold-start time under target; model lazy-load verified (NFR-PERF-4).

### P8 — advanced
- FTS search relevance on fixture corpus (FR-SEARCH-1); tag/filter correctness (FR-SEARCH-2).
- Template application structures notes by type (FR-NOTE-5).
- PDF/Word export renders offline + opens (FR-NOTE-4); bulk export by range/tag (FR-STORE-4).
- Reminder fires locally (FR-CAL-5). Encryption: locked store unreadable without vault password;
  unlock round-trips (FR-SEC-3).

### P9 — remote sync / upload
- Service: **sync disabled (default)** → no `sync_jobs` enqueued, no sync hosts in the egress
  allowlist, nothing uploads (FR-SYNC-1). This is the most important sync test.
- Service (WebDAV against `rclone serve webdav` or a disposable Nextcloud/Seafile container):
  finalize → selected artifacts land at the correct remote path (FR-SYNC-2/3).
- Service: `test_sync_target` succeeds with good creds, fails clearly with bad creds/URL (FR-SYNC-4).
- Resilience: kill the network mid-upload → job `failed`, retried with backoff, completes; unchanged
  re-upload is `skipped` via SHA-256; large-file resumable/chunked path exercised (FR-SYNC-5).
- Security: credentials never written to `settings.json`/`wa.db` (assert by scanning both);
  `list_sync_targets` never returns secrets (FR-SYNC-6). Plaintext `http://` refused unless explicit
  LAN opt-in (FR-SYNC-7).
- Unit: host-allowlist derivation includes only enabled targets; disabling a target removes its host.
- Secondary (mocked OAuth): PKCE loopback flow links an account; token stored in credential store;
  third-party banner shown (FR-SYNC-8/9). Optional: encrypt-before-upload yields ciphertext remotely
  (FR-SYNC-10).

### P10 — external AI & coding-agent integration
- Service: **all off (default)** → no AI host on the allowlist, MCP port closed, no briefs exposed.
  This is the most important P10 test.
- Service (10a): summary routes to a mocked OpenAI-compatible and a mocked Anthropic endpoint
  (correct `/v1/chat/completions` vs `/v1/messages` shapes); API key read from the credential store,
  never from settings/DB; host appears on the allowlist only when configured (FR-AI-1/2).
- Service (10b): with the MCP server enabled, an in-process MCP **client** calls `get_feature_brief`
  and receives a schema-valid brief; tools-only surface (no required resources/prompts) (FR-MCP-2/4).
- Security: MCP server binds to **loopback only** (assert it does not bind a non-loopback address),
  requires a token, and **opens no outbound socket** — the egress test is unchanged with MCP on (FR-MCP-7,
  NFR-SEC-5). Recordings are not served unless `expose_recordings` is true (FR-MCP-3).
- Audit: every tool call appends an `mcp_access_log` row / `mcp://access` event (FR-MCP-5).
- Unit (M1 — feature briefs, no network): (1) `parse_brief` splits a golden
  `## Title/## Problem/## Desired Outcome/## Acceptance Criteria` reply into the right fields, and an
  empty/malformed reply yields empty fields without panicking. (2) `FeatureBriefBuilder` over a golden
  transcript, driven by a `MockLlmProvider` that returns a fixed sectioned reply, produces a
  schema-valid `FeatureBrief` with non-empty `acceptance_criteria` and satisfies the **grounding
  invariant**: every `context_excerpts[].text` is a verbatim substring of some transcript segment
  (never model paraphrase). (3) Command-level: `create_feature_brief` with the LLM off/unreachable
  returns `Err` and writes no `briefs/*.json` and no `feature_briefs` row (no partial artifacts).
- (10c, when built) push: `AgentRunner` invokes a stub CLI with the brief; `IssueTracker` creates a
  mocked GitHub issue and (optional) Copilot assignment (FR-AGENT-1/2).

## Cross-cutting gates (run in CI every change)

1. **Privacy egress test (FR-SEC-1):** run a representative flow under a network monitor; assert
   **no** outbound connection except hosts on the settings-derived allowlist — the configured LLM/AI
   provider endpoint, **enabled sync target hosts**, a configured task tracker, and explicit
   model-download hosts. With nothing configured, assert zero egress. **Enabling the MCP server must
   not change the egress set** (it is inbound/loopback). Any off-allowlist connection blocks merge.
   This is the product's most important automated guarantee.
2. **Build matrix:** CPU-only build must always pass; accelerated features (CUDA/Vulkan/DirectML)
   built on capable runners, skipped (not failed) where hardware/SDK is absent (NFR-MNT-4).
3. **Lint/format:** `cargo fmt --check`, `cargo clippy -- -D warnings`, Prettier, ESLint, `tsc`.
4. **Contract check:** TS client types and Rust command/event payloads validated against shared
   fixtures so the IPC boundary can't silently drift.

## Performance & resource benchmarks (tracked over time)
- Real-time latency per phrase by backend (NFR-PERF-1).
- 60-min batch wall-clock vs meeting length by backend (NFR-PERF-2).
- Idle CPU/memory snapshot (NFR-RES-1); recording CPU on CPU-only laptop (NFR-RES-2).
- Cold-start to interactive (NFR-PERF-4). Regressions beyond a threshold fail the bench gate.

## Manual / exploratory checklist (pre-release)
Real meeting dogfood; device hot-swap mid-recording; very long (2 h) meeting; low-disk during
capture; abrupt power loss (recover from audio); high-DPI + both themes; screen-reader pass.

## Definition of done (recap from CLAUDE.md)
Compiles, no clippy warnings, requirement acceptance criteria met, named phase tests pass, docs
updated if contracts/behavior changed.