From c24feeb542bd4135412fe69a688d87ce60ce0bce Mon Sep 17 00:00:00 2001 From: iamdoubz <> Date: Sun, 5 Jul 2026 17:58:26 -0500 Subject: [PATCH] docs(readme): reflect working v0.1.4 (CPU/NPU/Vulkan, vault, sync) + refresh Granola/Meetily comparison --- README.md | 170 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 117 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index b287f3e..a4c3735 100644 --- a/README.md +++ b/README.md @@ -3,88 +3,152 @@ **A privacy-first, Windows-native meeting assistant that runs entirely on-device.** WhispAssist captures system audio, transcribes it locally with Whisper-class models using -on-device acceleration (NPU → GPU → CPU), structures the result into Markdown notes, and -optionally augments them with a locally hosted LLM (Ollama). Audio and transcripts **never -leave the machine** unless the user explicitly exports them. +on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures the result into +Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and +transcripts **never leave the machine** unless you explicitly configure a destination. -> Status: **planning + scaffold**. This repository currently contains the full engineering -> plan (`docs/`) and a compiling-intent skeleton (`src-tauri/`, `src/`). No feature code is -> implemented yet. See [`docs/05-roadmap.md`](docs/05-roadmap.md) for the build order. +> **Status: working application (v0.1.4).** Capture, transcription (CPU / Intel NPU / Vulkan +> GPU), speaker diarization, storage + crash recovery, local-LLM summaries, opt-in recording, +> at-rest encryption, and self-hosted sync are implemented and ship as signed **MSI + NSIS** +> installers. Outlook `.pst`/calendar context and the coding-agent (MCP) handoff are in +> progress. Build order and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md). -## Why WhispAssist +## Why WhispAssist — Granola vs Meetily vs WhispAssist -| | Granola | Meetily | **WhispAssist** | +| | **Granola** | **Meetily** | **WhispAssist** | |---|---|---|---| -| Local transcription | ❌ cloud | ✅ | ✅ | -| Notes/summaries stay local | ❌ cloud AI | ⚠️ optional | ✅ local-only by design | -| NPU/GPU auto-acceleration | n/a | partial | ✅ NPU→GPU→CPU ladder | -| Calendar + Outlook `.pst` context | ✅ (cloud) | ❌ | ✅ local | -| Bot-free system-audio capture | ✅ | ✅ | ✅ | -| License | proprietary | MIT | open source | +| **Positioning** | Cloud AI notepad | Open-source self-hosted assistant | Local, Windows-native assistant | +| **Platform** | macOS, Windows | macOS, Windows, Linux | Windows 10/11 | +| **Bot-free system-audio capture** | ✅ | ✅ | ✅ WASAPI loopback | +| **Transcription** | ☁️ cloud | ✅ local (Whisper) | ✅ local (whisper.cpp) | +| **NPU / GPU auto-acceleration** | n/a (cloud) | ⚠️ CPU/GPU, manual | ✅ **NPU→NVIDIA→AMD→Intel→CPU** ladder, one binary | +| **Speaker diarization** | ✅ cloud | ⚠️ limited | ✅ offline (sherpa-onnx) | +| **Summaries / notes AI** | ☁️ cloud LLM | ✅ local (Ollama) / BYO | ✅ local (Ollama, localhost **or LAN**) + optional hosted | +| **Default data egress** | ☁️ audio + notes to cloud | 🔒 local (cloud optional) | 🔒 **none** — everything off by default, allowlist-enforced | +| **Calendar / Outlook `.pst` context** | ✅ cloud calendar | ❌ | ⚙️ local `.pst` — in progress | +| **Opt-in recording + consent notice** | ⚠️ partial | ❌ | ✅ off by default, one-time consent | +| **At-rest encryption** | ☁️ server-side | ❌ | ✅ vault: Argon2id + XChaCha20-Poly1305 | +| **Self-hosted sync w/ client-side encryption** | ❌ | ⚠️ | ✅ WebDAV + OAuth, **encrypt-before-upload** | +| **Coding-agent (MCP) handoff** | ❌ | ❌ | ⚙️ local MCP server — in progress | +| **License** | Proprietary | Open source (MIT) | Open source (MIT / Apache-2.0) | +| **Cost** | Subscription | Free | Free | + +Comparison reflects each project's public positioning as of mid-2026. Granola and Meetily +are independent products and their capabilities evolve — verify current details before relying +on any row. + +**The short version:** Granola is the polished cloud option (your audio and notes are processed +on their servers). Meetily is the closest peer — open-source and self-hosted — but is +cross-platform-generic and leans on manual setup. WhispAssist is the **Windows-native, hardware- +accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern laptops, keeps +everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a +coding-agent handoff. + +## What's built (v0.1.4) + +- **Bot-free capture** — WASAPI loopback records the system mix (all participants) with no + meeting bot and no per-app plumbing. +- **Local transcription with a hardware ladder** — whisper.cpp via `whisper-rs` on CPU; the + **Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on + NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend + (**NPU → NVIDIA → AMD → Intel → CPU**), streams partial transcripts live, and shows the active + backend in the UI. +- **Speaker diarization** — `sherpa-onnx` (pyannote segmentation + speaker-embedding + clustering), fully offline. +- **Notes & summaries** — Markdown notes; local-LLM summaries via **Ollama** on `localhost` + **or a private LAN endpoint** (RFC-1918), with a full advanced-parameter panel (system prompt, + `think`, `keep_alive`, `num_ctx`, sampling/repetition/mirostat, etc.). +- **Storage & crash recovery** — SQLite + on-disk audio/transcripts under + `%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate + after a crash. +- **Opt-in recording** — off by default; `.wav` retained only when you turn it on, after a + one-time consent notice. +- **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts, + notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock. +- **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve, + Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with + backoff; **client-side encryption before upload** so the destination holds only ciphertext. + Credentials live only in the OS credential store. +- **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same + `LlmProvider` interface, off by default (third-party egress, keys in the OS credential store). +- **Installers** — signed MSI and NSIS `-setup.exe`. + +**In progress:** Outlook `.pst` + calendar context, the local **MCP server** that hands meeting +context to your coding agents (Claude, Codex, Copilot, OpenCode), and MS Graph calendar. ## Technology -WhispAssist is built as a **Tauri 2** application: a small Rust core with a compiled -**Svelte** web frontend rendered through the OS WebView2 (no bundled browser → low idle -memory). The choice and its alternatives are recorded in [`docs/adr/`](docs/adr/). +WhispAssist is a **Tauri 2** application: a small Rust core with a compiled **Svelte + TypeScript** +frontend rendered through the OS WebView2 (no bundled browser → low idle memory). Every decision +and its alternatives are recorded in [`docs/adr/`](docs/adr/) (ADR-0001–0011). - **Shell / IPC:** Tauri 2 (Rust ⇄ WebView2) -- **Audio capture:** WASAPI loopback (`wasapi` crate) -- **Transcription:** `whisper-rs` (whisper.cpp: CPU/Vulkan/CUDA) + ONNX Runtime (`ort`) DirectML for the NPU path -- **Diarization:** `sherpa-onnx` (pyannote segmentation + speaker-embedding clustering), fully offline -- **Storage:** SQLite (`sqlx`/`rusqlite`) + on-disk audio/transcript files -- **Local LLM:** Ollama HTTP API on `localhost:11434` -- **Calendar / Outlook:** `outlook-pst` crate for `.pst`, OS notifications for reminders -- **Optional recording:** opt-in (default off), saved as `.wav`, with a consent reminder (ADR-0009) -- **Optional sync:** upload artifacts to your own server — WebDAV covers **Nextcloud, ownCloud, Cloudreve, Seafile, Synology**; **OneDrive/Dropbox/Box** via OAuth. Off by default (ADR-0010) -- **Optional AI/agent integration:** hosted summary providers (Anthropic, OpenAI-compatible) behind the same provider model, **and** a local **MCP server** so your own coding agents (**Claude, Codex, Copilot, OpenCode, …**) can pull meeting context and "feature briefs" to start coding. Off by default; the MCP server is inbound/loopback only (ADR-0011) +- **Audio capture:** WASAPI loopback +- **Transcription:** `whisper-rs` (whisper.cpp: CPU / **Vulkan** / CUDA) + ONNX Runtime (`ort`) + with the **OpenVINO** execution provider for the Intel NPU path +- **Diarization:** `sherpa-onnx`, fully offline +- **Storage:** SQLite + on-disk audio/transcript files +- **Local LLM:** Ollama HTTP API (localhost or a private LAN endpoint) +- **Encryption:** Argon2id + XChaCha20-Poly1305 envelope vault; secrets in the OS credential store +- **Calendar / Outlook:** `outlook-pst` for `.pst`, OS notifications for reminders *(in progress)* +- **Sync:** WebDAV primary set + OneDrive/Dropbox/Box via OAuth — off by default (ADR-0010) +- **External AI / agents:** hosted providers behind `LlmProvider`; a loopback-only **MCP server** + for coding-agent handoff — off by default (ADR-0011) ## Repository layout ``` WhispAssist/ ├── docs/ # The engineering plan (read this first) -│ ├── 00-overview.md Vision, goals, glossary -│ ├── 01-requirements.md Functional + non-functional requirements (traceable IDs) -│ ├── 02-architecture.md Components, data flow, threading model -│ ├── 03-data-model.md SQLite schema, file layout, transcript JSON -│ ├── 04-api-contracts.md Tauri commands/events + internal Rust service traits -│ ├── 05-roadmap.md 8 phases, task breakdown, acceptance criteria -│ ├── 06-test-strategy.md Test plan per phase + quality gates -│ ├── 07-research-findings.md Validated stack with sources -│ └── adr/ Architecture Decision Records (0001–0010) -├── src-tauri/ # Rust core (service module skeletons) -├── src/ # Svelte frontend skeleton -├── scripts/ # Dev/model-download helper scripts -└── tests/ # Test fixtures + integration test scaffolding +│ ├── 00-overview.md … 07-research-findings.md +│ └── adr/ Architecture Decision Records (0001–0011) +├── src-tauri/ # Rust core — implemented service modules: +│ └── src/{audio,transcription,diarization,storage,llm,calendar, +│ hardware,notes,sync,mcp,vault} +├── src/ # Svelte + TypeScript frontend (views, stores, API bindings) +├── packaging/ # NPU/OpenVINO runtime bundle + release assets +├── scripts/ # Dev / model-download helpers +└── tests/ # Fixtures + cross-service integration tests ``` ## Getting started (for builders) -Prerequisites once implementation begins: Rust (stable), Node.js 20+, the Tauri CLI, and -WebView2 runtime (preinstalled on Windows 11). Then: +Prerequisites: **Rust** (stable), **Node.js 20+**, the **Tauri CLI**, and the WebView2 runtime +(preinstalled on Windows 11). Native builds also need the **VS 2022 Build Tools** (load +`vcvars64.bat` first). ```bash npm install -npm run tauri dev # once src-tauri/Cargo.toml dependencies are filled in +npm run tauri dev # CPU/NPU build ``` -The current skeleton intentionally does **not** compile end-to-end — modules contain typed -stubs and `todo!()` markers that map 1:1 to roadmap tasks. Start at Phase 1 in -[`docs/05-roadmap.md`](docs/05-roadmap.md). +**GPU (Vulkan) build.** whisper.cpp's GPU backends are compiled in (not downloaded at runtime), +so a GPU build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and +a short target dir (to dodge Windows' 260-char path limit in the shader build): + +```bash +# after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded +set VULKAN_SDK=C:\VulkanSDK\1.4.350.0 +set CMAKE_GENERATOR=Ninja +set CARGO_TARGET_DIR=C:\wt +npm run tauri build -- --features vulkan +``` + +CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build +recipe lives in the project notes. ## Privacy guarantee WhispAssist originates **no outbound connection for audio or transcript content** except to -destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted AI -provider if you choose one) and any sync targets you enable — plus explicit model downloads. +destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted +AI provider if you choose one) and any sync targets you enable — plus explicit model downloads. Everything optional is **off by default**; with nothing configured, WA makes no content egress at -all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on loopback -and adds no egress** — data only leaves via the agent's own provider, which WA discloses. The set of -reachable hosts is an allowlist derived from your settings and enforced in the core (and verified by -a CI network test). Recording is opt-in; sync/AI credentials live in the OS credential store, never -in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`, `FR-SYNC-*`, `FR-MCP-*`) in -[`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 0009–0011. +all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on +loopback and adds no egress** — data only leaves via the agent's own provider, which WA discloses. +The set of reachable hosts is an allowlist derived from your settings and enforced in the core +(and verified by a CI network test). Recording is opt-in; sync/AI credentials live in the OS +credential store, never in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`, +`FR-SYNC-*`, `FR-MCP-*`) in [`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 0009–0011. ## License