Files

T

iamdoubz b40a7e2fbc docs(test): expand M1 golden-transcript brief tests (mock provider, grounding invariant)

2026-07-06 23:27:04 -05:00

10 KiB

Raw Blame History

06 — Test Strategy

Testing mirrors the architecture (per-service) and the roadmap (per-phase). Every requirement ID has at least one test that asserts its acceptance criteria. A phase is "done" only when its suite is green and the cross-cutting gates pass.

Test levels

Level	Scope	Tooling
Unit	One module/function; pure logic, parsers, mappers	Rust `#[cfg(test)]`; Vitest (frontend)
Service/integration	One service against real deps (SQLite, files, Ollama, models)	Rust integration tests in `/tests`
Contract	Tauri command/event payload shapes match `04-api-contracts.md`	shared TS↔Rust fixtures, schema assertions
End-to-end	Full flows through the UI (record→notes→summary)	Tauri WebDriver / Playwright-style harness
Non-functional	Performance, resource, privacy egress	Criterion benches; resource probes; network monitor

Key fixtures (`/tests/fixtures`)

Short stereo WAVs with known content (1-speaker, 2-speaker, overlap, silence, device-glitch).
A golden transcript.json for deterministic alignment/merge tests.
A small synthetic .pst with known events/attendees (or a generator script) — never real mail.
A canned Ollama response set (recorded) for offline LLM tests + a live-Ollama opt-in lane.
A disposable WebDAV server for sync tests (rclone serve webdav or a throwaway Nextcloud/Seafile container) + mocked OAuth responses for the secondary providers.

Per-phase suites

P1 — capture + CPU transcription

Unit: WAV writer produces valid headers/duration; ring buffer bounded + lossless under load.
Service: capture a fixed audio source → byte-accurate audio.wav (FR-CAP-1/2).
Service: feed a known WAV through the CPU Transcriber → expected segments within tolerance (FR-TRX-1).
E2E: start→speak→segments appear via transcript://segment; stop→idle (FR-CAP-3, FR-TRX-2).
NFR: idle audit — 0 audio threads / 0 polling timers after stop (NFR-RES-1).
Service: retention OFF (default) → no audio.wav after finalize; retention ON → playable audio.wav retained; toggling mid-meeting takes effect (FR-REC-1/3).
Unit/E2E: consent notice shown before first retained recording; acknowledgment persists and is not re-shown (FR-REC-2).

P2 — storage, notes, recovery

Unit: schema migrations apply forward cleanly; Markdown assembly is speaker-tagged + deterministic.
Service: create→finalize→get round-trips meeting + transcript (FR-STORE-1/5).
Service: crash-recovery — write working audio, kill process, relaunch → meeting recovering, re-transcribe succeeds (FR-REL-1, NFR-REL-3).
Service: retention deletes oldest past cap; refuses to touch an in-progress meeting (FR-STORE-2).
Service: working-WAV deletion on finalize happens only after transcript is finalized; a crash before finalize leaves a recoverable WAV regardless of retention setting (FR-REC-4).
Contract: .md export round-trips notes (FR-NOTE-3); bundle export contains audio+transcript+notes.

P3 — hardware + models

Unit: backend ranking honors NPU→NVIDIA→AMD→Intel→CPU; override respected (FR-HW-2).
Service: same WAV across available backends → equivalent segments (FR-TRX-5).
Service: simulated backend failure → hardware://changed + CPU fallback, no abort (FR-HW-4).
NFR bench: accelerated RTF < CPU RTF on capable hardware; latency target met (NFR-PERF-1/2).
Service: model download/remove with progress; corrupt/aborted download handled (FR-MODEL-1).

P4 — diarization & naming

Service: 2-speaker fixture → ≥2 speaker IDs; alignment overlap accuracy ≥ threshold (FR-SPK-1).
Unit: rename applies to all of a speaker's segments at render; storage IDs unchanged (FR-SPK-5).
Unit: merge(from=[S2,S3], into=S1) updates notes/export; idempotent (FR-SPK-3).
E2E: in-session naming updates live view (FR-SPK-2).

P5 — LLM

Service (recorded): prompt assembly includes transcript+metadata+template; output parsed into action items (FR-LLM-2/3).
Service: streaming tokens arrive via llm://token; summary.json persisted (FR-LLM-4).
Unit: local-endpoint guard flags non-loopback hosts → banner (FR-LLM-6, FR-SEC-1).
Resilience: provider off/unreachable → core app unaffected, clear status (FR-LLM-1/5).
Live lane (opt-in): against a real local Ollama, end-to-end summary sanity check.

P6 — calendar / .pst

Service: parse synthetic .pst → expected events + attendees; password path; corrupt file → non-fatal error surfaced (FR-CAL-1).
Unit: attendee list populates speaker-naming dropdown; "add new name" path (FR-SPK-4).
Service: attach meeting↔event persists link; participant continuity across meetings (FR-CAL-2/4).

P7 — UX / accessibility / privacy

E2E: theme follows OS + override; keyboard-only core flows (FR-UX-2/3).
A11y: automated axe-style audit (labels/roles/contrast) passes (FR-UX-4).
Service: privacy_self_check reports only allowed egress incl. configured sync targets with correct third-party/TLS labeling; backend/endpoint accurate (FR-SEC-2).
NFR: cold-start time under target; model lazy-load verified (NFR-PERF-4).

P8 — advanced

FTS search relevance on fixture corpus (FR-SEARCH-1); tag/filter correctness (FR-SEARCH-2).
Template application structures notes by type (FR-NOTE-5).
PDF/Word export renders offline + opens (FR-NOTE-4); bulk export by range/tag (FR-STORE-4).
Reminder fires locally (FR-CAL-5). Encryption: locked store unreadable without vault password; unlock round-trips (FR-SEC-3).

P9 — remote sync / upload

Service: sync disabled (default) → no sync_jobs enqueued, no sync hosts in the egress allowlist, nothing uploads (FR-SYNC-1). This is the most important sync test.
Service (WebDAV against rclone serve webdav or a disposable Nextcloud/Seafile container): finalize → selected artifacts land at the correct remote path (FR-SYNC-2/3).
Service: test_sync_target succeeds with good creds, fails clearly with bad creds/URL (FR-SYNC-4).
Resilience: kill the network mid-upload → job failed, retried with backoff, completes; unchanged re-upload is skipped via SHA-256; large-file resumable/chunked path exercised (FR-SYNC-5).
Security: credentials never written to settings.json/wa.db (assert by scanning both); list_sync_targets never returns secrets (FR-SYNC-6). Plaintext http:// refused unless explicit LAN opt-in (FR-SYNC-7).
Unit: host-allowlist derivation includes only enabled targets; disabling a target removes its host.
Secondary (mocked OAuth): PKCE loopback flow links an account; token stored in credential store; third-party banner shown (FR-SYNC-8/9). Optional: encrypt-before-upload yields ciphertext remotely (FR-SYNC-10).

P10 — external AI & coding-agent integration

Service: all off (default) → no AI host on the allowlist, MCP port closed, no briefs exposed. This is the most important P10 test.
Service (10a): summary routes to a mocked OpenAI-compatible and a mocked Anthropic endpoint (correct /v1/chat/completions vs /v1/messages shapes); API key read from the credential store, never from settings/DB; host appears on the allowlist only when configured (FR-AI-1/2).
Service (10b): with the MCP server enabled, an in-process MCP client calls get_feature_brief and receives a schema-valid brief; tools-only surface (no required resources/prompts) (FR-MCP-2/4).
Security: MCP server binds to loopback only (assert it does not bind a non-loopback address), requires a token, and opens no outbound socket — the egress test is unchanged with MCP on (FR-MCP-7, NFR-SEC-5). Recordings are not served unless expose_recordings is true (FR-MCP-3).
Audit: every tool call appends an mcp_access_log row / mcp://access event (FR-MCP-5).
Unit (M1 — feature briefs, no network): (1) parse_brief splits a golden ## Title/## Problem/## Desired Outcome/## Acceptance Criteria reply into the right fields, and an empty/malformed reply yields empty fields without panicking. (2) FeatureBriefBuilder over a golden transcript, driven by a MockLlmProvider that returns a fixed sectioned reply, produces a schema-valid FeatureBrief with non-empty acceptance_criteria and satisfies the grounding invariant: every context_excerpts[].text is a verbatim substring of some transcript segment (never model paraphrase). (3) Command-level: create_feature_brief with the LLM off/unreachable returns Err and writes no briefs/*.json and no feature_briefs row (no partial artifacts).
(10c, when built) push: AgentRunner invokes a stub CLI with the brief; IssueTracker creates a mocked GitHub issue and (optional) Copilot assignment (FR-AGENT-1/2).

Cross-cutting gates (run in CI every change)

Privacy egress test (FR-SEC-1): run a representative flow under a network monitor; assert no outbound connection except hosts on the settings-derived allowlist — the configured LLM/AI provider endpoint, enabled sync target hosts, a configured task tracker, and explicit model-download hosts. With nothing configured, assert zero egress. Enabling the MCP server must not change the egress set (it is inbound/loopback). Any off-allowlist connection blocks merge. This is the product's most important automated guarantee.
Build matrix: CPU-only build must always pass; accelerated features (CUDA/Vulkan/DirectML) built on capable runners, skipped (not failed) where hardware/SDK is absent (NFR-MNT-4).
Lint/format: cargo fmt --check, cargo clippy -- -D warnings, Prettier, ESLint, tsc.
Contract check: TS client types and Rust command/event payloads validated against shared fixtures so the IPC boundary can't silently drift.

Performance & resource benchmarks (tracked over time)

Real-time latency per phrase by backend (NFR-PERF-1).
60-min batch wall-clock vs meeting length by backend (NFR-PERF-2).
Idle CPU/memory snapshot (NFR-RES-1); recording CPU on CPU-only laptop (NFR-RES-2).
Cold-start to interactive (NFR-PERF-4). Regressions beyond a threshold fail the bench gate.

Manual / exploratory checklist (pre-release)

Real meeting dogfood; device hot-swap mid-recording; very long (2 h) meeting; low-disk during capture; abrupt power loss (recover from audio); high-DPI + both themes; screen-reader pass.

Definition of done (recap from CLAUDE.md)

Compiles, no clippy warnings, requirement acceptance criteria met, named phase tests pass, docs updated if contracts/behavior changed.

10 KiB Raw Blame History