iamdoubz/WhispAssist
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 13 Wiki Activity
Files
223dc5f34b90856782ae803c147131ea8cd36025
WhispAssist/docs/07-research-findings.md
T
iamdoubz 02834e9cd2 Phase 1
2026-06-30 22:17:30 -05:00

8.5 KiB
Raw Blame History

07 — Research Findings (Stack Validation)

This document records the technical feasibility checks performed before committing to a stack. Each finding includes the conclusion that fed into the ADRs. Dates reflect research done June 2026; re-verify versions at implementation time.

Summary

Every load-bearing assumption in the design document is supported by a mature, offline-capable, permissively licensed component. No blocker was found. The one area needing care is the NPU path (newer, vendor-dependent) and .pst parsing (use a battle-tested library, treat as read-only). Both have viable primary + fallback options.

Findings

App shell — Tauri 2 ✅

Tauri 2.0 reached stable in October 2024 and is the current major line. On Windows it renders through WebView2 (the OS Chromium-based webview) rather than bundling a browser, which is the key reason its binaries and memory footprint are far smaller than Electron's — directly serving WA's "fast, low memory" goal. Cross-platform (Windows/macOS/Linux) leaves the door open for the design doc's "future" non-Windows support without a rewrite. → ADR-0001.

Frontend — Svelte ✅

Svelte compiles components to small imperative JS with no runtime framework shipped, giving the smallest bundle and lowest per-view memory among mainstream options — consistent with the low-overhead NFRs. Pairs cleanly with Tauri via Vite. → ADR-0002.

Audio capture — WASAPI loopback in Rust ✅

The wasapi crate provides safe Rust access to WASAPI including a loopback example (capture system output without a meeting bot). cpal is the cross-platform fallback. Correction (Phase 1 implementation, verified against wasapi v0.15 source and current Windows): AUDCLNT_STREAMFLAGS_LOOPBACK and AUDCLNT_STREAMFLAGS_EVENTCALLBACK can be combined — the crate wires both flags together for a render-device-opened-as-capture client, and the capture loop waits on the WASAPI event handle (with a short timeout so it can also notice a stop/pause request) rather than busy-polling. The pre-implementation note below describing a polling-only design was inaccurate; docs/02-architecture.md's "Audio thread" section has been corrected to match.

Transcription — whisper-rs (+ ONNX for NPU) ✅

whisper-rs wraps whisper.cpp with feature-gated acceleration: CUDA (NVIDIA), Vulkan (cross-vendor GPU incl. AMD/Intel), ROCm/hipBLAS, Metal, OpenBLAS, and plain CPU. whisper.cpp has no native NPU backend, so the NPU tier is served separately via ONNX Runtime (see below) running a Whisper-ONNX model. This split — whisper.cpp for CPU/GPU, ONNX for NPU — is the basis of the acceleration ladder. → ADR-0003.

Hardware acceleration — ONNX Runtime ort + DirectML/Windows ML ✅

The Rust ort crate exposes ONNX Runtime execution providers including DirectML, which can target an NPU (device_filter = "npu") on Windows. Execution providers fall back in order (NPU → GPU → CPU) automatically when an operator is unsupported. Windows ML is now GA as the production on-device runtime on Windows 11. DirectX 12 is required for DirectML (ubiquitous on modern hardware). Note: Microsoft is steering new Windows ONNX work toward Windows ML over raw DirectML, so the hardware layer is abstracted (hardware::Backend) to allow swapping the provider without touching callers. → ADR-0004.

Diarization — sherpa-onnx ✅

sherpa-onnx (k2-fsa) provides offline speaker diarization: pyannote segmentation model + a speaker-embedding extractor (e.g. 3D-Speaker ERes2Net) + clustering, all as ONNX, fully offline. It ships C/C++ APIs callable from Rust via FFI. This satisfies "Speaker 1/2 …" labeling and post-meeting merge/rename without any cloud dependency. → ADR-0005.

Storage — SQLite ✅

SQLite (via sqlx for async or rusqlite for sync) is the standard embedded store, matches Meetily's proven model, requires no server, and supports full-text search (FTS5) for the transcript search requirement. → ADR-0006.

Local LLM — Ollama HTTP ✅

Ollama serves a REST API on http://localhost:11434 with /api/chat (OpenAI-style messages), /api/generate (one-shot), /api/tags (list models), /api/pull (download), and token streaming via newline-delimited JSON. WA calls only this local endpoint; a "custom endpoint" option lets advanced users point at any OpenAI-compatible local server. → ADR-0007.

Outlook .pst — outlook-pst / libpff ✅

The Rust outlook-pst crate offers read-only access to PST files modeled on the published MS-PST specification (emails, folders, and — relevant here — calendar appointments/attendees). The mature C library libpff (with pffexport) is the fallback for tricky/encrypted files via FFI or a sidecar. Treat PST strictly as read-only input. → ADR-0008.

External AI / MCP — all targets are MCP clients; Rust SDK exists ✅

  • Claude (Code & Desktop): MCP client over stdio + HTTP; Claude Code also runs headless (claude -p, --bare) and ships the Claude Agent SDK (Python/TS) — usable for the optional push path. Claude Code can itself act as an MCP server.
  • OpenAI Codex (CLI): MCP via stdio (local child process) and remote Streamable HTTP (OAuth/bearer); managed with codex mcp; codex exec for headless. Has an Agents SDK.
  • GitHub Copilot: MCP across IDE, CLI, and the cloud agent. Caveat: the cloud agent supports MCP tools only (no resources/prompts) and no OAuth-remote MCP, and runs in GitHub's cloud (can't reach a localhost server) — so WA exposes capabilities as tools, and Copilot cloud is better served by the GitHub-issue handoff.
  • OpenCode: full MCP client (local + remote), 75+ LLM providers, has an SDK.
  • Transports: MCP defines exactly stdio and Streamable HTTP (HTTP+SSE deprecated). For a long-running desktop app holding shared state, a loopback Streamable HTTP /mcp endpoint fits; a thin stdio adapter covers agents that spawn their server.
  • Rust SDK: the official rmcp crate (modelcontextprotocol/rust-sdk, v0.16+, server feature) implements tools/resources/prompts with pluggable transports — so WA's Rust core can host the server natively and lightweightly.
  • Summary APIs: Anthropic is native /v1/messages (needs its own adapter); OpenAI is /v1/chat/completions; most others are OpenAI-compatible. → ADR-0011 (and ADR-0007 update).

Open items to confirm at build time

  • Exact whisper-rs version and which acceleration features build cleanly on the Windows CI image.
  • ort ↔ ONNX Runtime binary version pinning for the DirectML/NPU path on target Windows builds.
  • outlook-pst coverage of password-protected and very large PST files; fall back to libpff if gaps.
  • Whether to consume sherpa-onnx as a prebuilt C library or build from source in CI.

Sources

Reference in New Issue View Git Blame Copy Permalink