155 lines
10 KiB
Markdown
155 lines
10 KiB
Markdown
# 06 — Test Strategy
|
|
|
|
Testing mirrors the architecture (per-service) and the roadmap (per-phase). Every requirement ID
|
|
has at least one test that asserts its acceptance criteria. A phase is "done" only when its suite
|
|
is green and the cross-cutting gates pass.
|
|
|
|
## Test levels
|
|
|
|
| Level | Scope | Tooling |
|
|
|---|---|---|
|
|
| Unit | One module/function; pure logic, parsers, mappers | Rust `#[cfg(test)]`; Vitest (frontend) |
|
|
| Service/integration | One service against real deps (SQLite, files, Ollama, models) | Rust integration tests in `/tests` |
|
|
| Contract | Tauri command/event payload shapes match `04-api-contracts.md` | shared TS↔Rust fixtures, schema assertions |
|
|
| End-to-end | Full flows through the UI (record→notes→summary) | Tauri WebDriver / Playwright-style harness |
|
|
| Non-functional | Performance, resource, privacy egress | Criterion benches; resource probes; network monitor |
|
|
|
|
## Key fixtures (`/tests/fixtures`)
|
|
- Short stereo WAVs with known content (1-speaker, 2-speaker, overlap, silence, device-glitch).
|
|
- A golden `transcript.json` for deterministic alignment/merge tests.
|
|
- A small synthetic `.pst` with known events/attendees (or a generator script) — never real mail.
|
|
- A canned Ollama response set (recorded) for offline LLM tests + a live-Ollama opt-in lane.
|
|
- A disposable WebDAV server for sync tests (`rclone serve webdav` or a throwaway Nextcloud/Seafile
|
|
container) + mocked OAuth responses for the secondary providers.
|
|
|
|
## Per-phase suites
|
|
|
|
### P1 — capture + CPU transcription
|
|
- Unit: WAV writer produces valid headers/duration; ring buffer bounded + lossless under load.
|
|
- Service: capture a fixed audio source → byte-accurate `audio.wav` (FR-CAP-1/2).
|
|
- Service: feed a known WAV through the CPU Transcriber → expected segments within tolerance (FR-TRX-1).
|
|
- E2E: start→speak→segments appear via `transcript://segment`; stop→idle (FR-CAP-3, FR-TRX-2).
|
|
- NFR: idle audit — 0 audio threads / 0 polling timers after stop (NFR-RES-1).
|
|
- Service: retention OFF (default) → no `audio.wav` after finalize; retention ON → playable
|
|
`audio.wav` retained; toggling mid-meeting takes effect (FR-REC-1/3).
|
|
- Unit/E2E: consent notice shown before first retained recording; acknowledgment persists and is not
|
|
re-shown (FR-REC-2).
|
|
|
|
### P2 — storage, notes, recovery
|
|
- Unit: schema migrations apply forward cleanly; Markdown assembly is speaker-tagged + deterministic.
|
|
- Service: create→finalize→get round-trips meeting + transcript (FR-STORE-1/5).
|
|
- Service: **crash-recovery** — write working audio, kill process, relaunch → meeting `recovering`,
|
|
re-transcribe succeeds (FR-REL-1, NFR-REL-3).
|
|
- Service: retention deletes oldest past cap; refuses to touch an in-progress meeting (FR-STORE-2).
|
|
- Service: working-WAV deletion on finalize happens only after transcript is finalized; a crash
|
|
before finalize leaves a recoverable WAV regardless of retention setting (FR-REC-4).
|
|
- Contract: `.md` export round-trips notes (FR-NOTE-3); bundle export contains audio+transcript+notes.
|
|
|
|
### P3 — hardware + models
|
|
- Unit: backend ranking honors NPU→NVIDIA→AMD→Intel→CPU; override respected (FR-HW-2).
|
|
- Service: same WAV across available backends → equivalent segments (FR-TRX-5).
|
|
- Service: simulated backend failure → `hardware://changed` + CPU fallback, no abort (FR-HW-4).
|
|
- NFR bench: accelerated RTF < CPU RTF on capable hardware; latency target met (NFR-PERF-1/2).
|
|
- Service: model download/remove with progress; corrupt/aborted download handled (FR-MODEL-1).
|
|
|
|
### P4 — diarization & naming
|
|
- Service: 2-speaker fixture → ≥2 speaker IDs; alignment overlap accuracy ≥ threshold (FR-SPK-1).
|
|
- Unit: rename applies to all of a speaker's segments at render; storage IDs unchanged (FR-SPK-5).
|
|
- Unit: merge(from=[S2,S3], into=S1) updates notes/export; idempotent (FR-SPK-3).
|
|
- E2E: in-session naming updates live view (FR-SPK-2).
|
|
|
|
### P5 — LLM
|
|
- Service (recorded): prompt assembly includes transcript+metadata+template; output parsed into
|
|
action items (FR-LLM-2/3).
|
|
- Service: streaming tokens arrive via `llm://token`; `summary.json` persisted (FR-LLM-4).
|
|
- Unit: local-endpoint guard flags non-loopback hosts → banner (FR-LLM-6, FR-SEC-1).
|
|
- Resilience: provider off/unreachable → core app unaffected, clear status (FR-LLM-1/5).
|
|
- Live lane (opt-in): against a real local Ollama, end-to-end summary sanity check.
|
|
|
|
### P6 — calendar / .pst
|
|
- Service: parse synthetic `.pst` → expected events + attendees; password path; corrupt file →
|
|
non-fatal error surfaced (FR-CAL-1).
|
|
- Unit: attendee list populates speaker-naming dropdown; "add new name" path (FR-SPK-4).
|
|
- Service: attach meeting↔event persists link; participant continuity across meetings (FR-CAL-2/4).
|
|
|
|
### P7 — UX / accessibility / privacy
|
|
- E2E: theme follows OS + override; keyboard-only core flows (FR-UX-2/3).
|
|
- A11y: automated axe-style audit (labels/roles/contrast) passes (FR-UX-4).
|
|
- Service: `privacy_self_check` reports only allowed egress incl. configured sync targets with
|
|
correct third-party/TLS labeling; backend/endpoint accurate (FR-SEC-2).
|
|
- NFR: cold-start time under target; model lazy-load verified (NFR-PERF-4).
|
|
|
|
### P8 — advanced
|
|
- FTS search relevance on fixture corpus (FR-SEARCH-1); tag/filter correctness (FR-SEARCH-2).
|
|
- Template application structures notes by type (FR-NOTE-5).
|
|
- PDF/Word export renders offline + opens (FR-NOTE-4); bulk export by range/tag (FR-STORE-4).
|
|
- Reminder fires locally (FR-CAL-5). Encryption: locked store unreadable without vault password;
|
|
unlock round-trips (FR-SEC-3).
|
|
|
|
### P9 — remote sync / upload
|
|
- Service: **sync disabled (default)** → no `sync_jobs` enqueued, no sync hosts in the egress
|
|
allowlist, nothing uploads (FR-SYNC-1). This is the most important sync test.
|
|
- Service (WebDAV against `rclone serve webdav` or a disposable Nextcloud/Seafile container):
|
|
finalize → selected artifacts land at the correct remote path (FR-SYNC-2/3).
|
|
- Service: `test_sync_target` succeeds with good creds, fails clearly with bad creds/URL (FR-SYNC-4).
|
|
- Resilience: kill the network mid-upload → job `failed`, retried with backoff, completes; unchanged
|
|
re-upload is `skipped` via SHA-256; large-file resumable/chunked path exercised (FR-SYNC-5).
|
|
- Security: credentials never written to `settings.json`/`wa.db` (assert by scanning both);
|
|
`list_sync_targets` never returns secrets (FR-SYNC-6). Plaintext `http://` refused unless explicit
|
|
LAN opt-in (FR-SYNC-7).
|
|
- Unit: host-allowlist derivation includes only enabled targets; disabling a target removes its host.
|
|
- Secondary (mocked OAuth): PKCE loopback flow links an account; token stored in credential store;
|
|
third-party banner shown (FR-SYNC-8/9). Optional: encrypt-before-upload yields ciphertext remotely
|
|
(FR-SYNC-10).
|
|
|
|
### P10 — external AI & coding-agent integration
|
|
- Service: **all off (default)** → no AI host on the allowlist, MCP port closed, no briefs exposed.
|
|
This is the most important P10 test.
|
|
- Service (10a): summary routes to a mocked OpenAI-compatible and a mocked Anthropic endpoint
|
|
(correct `/v1/chat/completions` vs `/v1/messages` shapes); API key read from the credential store,
|
|
never from settings/DB; host appears on the allowlist only when configured (FR-AI-1/2).
|
|
- Service (10b): with the MCP server enabled, an in-process MCP **client** calls `get_feature_brief`
|
|
and receives a schema-valid brief; tools-only surface (no required resources/prompts) (FR-MCP-2/4).
|
|
- Security: MCP server binds to **loopback only** (assert it does not bind a non-loopback address),
|
|
requires a token, and **opens no outbound socket** — the egress test is unchanged with MCP on (FR-MCP-7,
|
|
NFR-SEC-5). Recordings are not served unless `expose_recordings` is true (FR-MCP-3).
|
|
- Audit: every tool call appends an `mcp_access_log` row / `mcp://access` event (FR-MCP-5).
|
|
- Unit (M1 — feature briefs, no network): (1) `parse_brief` splits a golden
|
|
`## Title/## Problem/## Desired Outcome/## Acceptance Criteria` reply into the right fields, and an
|
|
empty/malformed reply yields empty fields without panicking. (2) `FeatureBriefBuilder` over a golden
|
|
transcript, driven by a `MockLlmProvider` that returns a fixed sectioned reply, produces a
|
|
schema-valid `FeatureBrief` with non-empty `acceptance_criteria` and satisfies the **grounding
|
|
invariant**: every `context_excerpts[].text` is a verbatim substring of some transcript segment
|
|
(never model paraphrase). (3) Command-level: `create_feature_brief` with the LLM off/unreachable
|
|
returns `Err` and writes no `briefs/*.json` and no `feature_briefs` row (no partial artifacts).
|
|
- (10c, when built) push: `AgentRunner` invokes a stub CLI with the brief; `IssueTracker` creates a
|
|
mocked GitHub issue and (optional) Copilot assignment (FR-AGENT-1/2).
|
|
|
|
## Cross-cutting gates (run in CI every change)
|
|
|
|
1. **Privacy egress test (FR-SEC-1):** run a representative flow under a network monitor; assert
|
|
**no** outbound connection except hosts on the settings-derived allowlist — the configured LLM/AI
|
|
provider endpoint, **enabled sync target hosts**, a configured task tracker, and explicit
|
|
model-download hosts. With nothing configured, assert zero egress. **Enabling the MCP server must
|
|
not change the egress set** (it is inbound/loopback). Any off-allowlist connection blocks merge.
|
|
This is the product's most important automated guarantee.
|
|
2. **Build matrix:** CPU-only build must always pass; accelerated features (CUDA/Vulkan/DirectML)
|
|
built on capable runners, skipped (not failed) where hardware/SDK is absent (NFR-MNT-4).
|
|
3. **Lint/format:** `cargo fmt --check`, `cargo clippy -- -D warnings`, Prettier, ESLint, `tsc`.
|
|
4. **Contract check:** TS client types and Rust command/event payloads validated against shared
|
|
fixtures so the IPC boundary can't silently drift.
|
|
|
|
## Performance & resource benchmarks (tracked over time)
|
|
- Real-time latency per phrase by backend (NFR-PERF-1).
|
|
- 60-min batch wall-clock vs meeting length by backend (NFR-PERF-2).
|
|
- Idle CPU/memory snapshot (NFR-RES-1); recording CPU on CPU-only laptop (NFR-RES-2).
|
|
- Cold-start to interactive (NFR-PERF-4). Regressions beyond a threshold fail the bench gate.
|
|
|
|
## Manual / exploratory checklist (pre-release)
|
|
Real meeting dogfood; device hot-swap mid-recording; very long (2 h) meeting; low-disk during
|
|
capture; abrupt power loss (recover from audio); high-DPI + both themes; screen-reader pass.
|
|
|
|
## Definition of done (recap from CLAUDE.md)
|
|
Compiles, no clippy warnings, requirement acceptance criteria met, named phase tests pass, docs
|
|
updated if contracts/behavior changed.
|