176 lines
10 KiB
Markdown
176 lines
10 KiB
Markdown
# WhispAssist (WA)
|
||
|
||
**A privacy-first, Windows-native meeting assistant that runs entirely on-device.**
|
||
|
||
WhispAssist captures system audio, transcribes it locally with Whisper-class models using
|
||
on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures the result into
|
||
Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and
|
||
transcripts **never leave the machine** unless you explicitly configure a destination.
|
||
|
||
> **Status: working application (v0.1.5).** Capture, transcription (CPU / Intel NPU / Vulkan
|
||
> GPU), speaker diarization, storage + crash recovery, local-LLM summaries, opt-in recording,
|
||
> at-rest encryption, and self-hosted sync are implemented and ship as signed **MSI + NSIS**
|
||
> installers. Outlook `.pst`/calendar context and the coding-agent (MCP) handoff are in
|
||
> progress. Build order and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md).
|
||
|
||
## Why WhispAssist — Granola vs Meetily vs WhispAssist
|
||
|
||
| | **Granola** | **Meetily** | **WhispAssist** |
|
||
|---|---|---|---|
|
||
| **Positioning** | Cloud AI notepad | Open-source self-hosted assistant | Local, Windows-native assistant |
|
||
| **Platform** | macOS, Windows | macOS, Windows, Linux | Windows 10/11 |
|
||
| **Bot-free system-audio capture** | ✅ | ✅ | ✅ WASAPI loopback |
|
||
| **Transcription** | ☁️ cloud | ✅ local (Whisper) | ✅ local (whisper.cpp) |
|
||
| **NPU / GPU auto-acceleration** | n/a (cloud) | ⚠️ CPU/GPU, manual | ✅ **NPU→NVIDIA→AMD→Intel→CPU** ladder, one binary |
|
||
| **Speaker diarization** | ✅ cloud | ⚠️ limited | ✅ offline (sherpa-onnx) |
|
||
| **Summaries / notes AI** | ☁️ cloud LLM | ✅ local (Ollama) / BYO | ✅ local (Ollama, localhost **or LAN**) + optional hosted |
|
||
| **Default data egress** | ☁️ audio + notes to cloud | 🔒 local (cloud optional) | 🔒 **none** — everything off by default, allowlist-enforced |
|
||
| **Calendar / Outlook `.pst` context** | ✅ cloud calendar | ❌ | ⚙️ local `.pst` — in progress |
|
||
| **Opt-in recording + consent notice** | ⚠️ partial | ❌ | ✅ off by default, one-time consent |
|
||
| **At-rest encryption** | ☁️ server-side | ❌ | ✅ vault: Argon2id + XChaCha20-Poly1305 |
|
||
| **Self-hosted sync w/ client-side encryption** | ❌ | ⚠️ | ✅ WebDAV + OAuth, **encrypt-before-upload** |
|
||
| **Coding-agent (MCP) handoff** | ❌ | ❌ | ⚙️ local MCP server — in progress |
|
||
| **License** | Proprietary | Open source (MIT) | Open source (MIT / Apache-2.0) |
|
||
| **Cost** | Subscription | Free | Free |
|
||
|
||
<sub>Comparison reflects each project's public positioning as of mid-2026. Granola and Meetily
|
||
are independent products and their capabilities evolve — verify current details before relying
|
||
on any row.</sub>
|
||
|
||
**The short version:** Granola is the polished cloud option (your audio and notes are processed
|
||
on their servers). Meetily is the closest peer — open-source and self-hosted — but is
|
||
cross-platform-generic and leans on manual setup. WhispAssist is the **Windows-native, hardware-
|
||
accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern laptops, keeps
|
||
everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a
|
||
coding-agent handoff.
|
||
|
||
## What's built (v0.1.5)
|
||
|
||
- **Bot-free capture** — WASAPI loopback records the system mix (all participants) with no
|
||
meeting bot and no per-app plumbing.
|
||
- **Local transcription with a hardware ladder** — whisper.cpp via `whisper-rs` on CPU; the
|
||
**Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on
|
||
NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend
|
||
(**NPU → NVIDIA → AMD → Intel → CPU**), streams partial transcripts live, and shows the active
|
||
backend in the UI.
|
||
- **Speaker diarization** — `sherpa-onnx` (pyannote segmentation + speaker-embedding
|
||
clustering), fully offline.
|
||
- **Notes & summaries** — Markdown notes; local-LLM summaries via **Ollama** on `localhost`
|
||
**or a private LAN endpoint** (RFC-1918), with a full advanced-parameter panel (system prompt,
|
||
`think`, `keep_alive`, `num_ctx`, sampling/repetition/mirostat, etc.).
|
||
- **Storage & crash recovery** — SQLite + on-disk audio/transcripts under
|
||
`%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate
|
||
after a crash.
|
||
- **Opt-in recording** — off by default; `.wav` retained only when you turn it on, after a
|
||
one-time consent notice.
|
||
- **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts,
|
||
notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock.
|
||
- **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve,
|
||
Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with
|
||
backoff; **client-side encryption before upload** so the destination holds only ciphertext.
|
||
Credentials live only in the OS credential store.
|
||
- **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same
|
||
`LlmProvider` interface, off by default (third-party egress, keys in the OS credential store).
|
||
- **Installers** — signed MSI and NSIS `-setup.exe`.
|
||
|
||
**In progress:** Outlook `.pst` + calendar context, the local **MCP server** that hands meeting
|
||
context to your coding agents (Claude, Codex, Copilot, OpenCode), and MS Graph calendar.
|
||
|
||
## Quick start (install)
|
||
|
||
**Requirements:** Windows 10 or 11 (x64). WhispAssist needs the **WebView2** runtime — preinstalled
|
||
on Windows 11; on Windows 10 the installer fetches it automatically (so keep an internet connection
|
||
during setup).
|
||
|
||
1. Download the latest **`WhispAssist_<version>_x64_en-US.msi`** from the
|
||
[releases page](https://git.dou.bet/iamdoubz/WhispAssist/releases).
|
||
2. Double-click the `.msi` and follow the prompts, accepting the UAC prompt to install. The
|
||
installer is signed; if Windows SmartScreen appears, choose **More info → Run anyway**.
|
||
3. Launch **WhispAssist** from the Start menu.
|
||
|
||
On first run WA detects your hardware and picks the best transcription backend
|
||
(**NPU → NVIDIA → AMD → Intel → CPU**). It works on your **CPU or GPU** (GPU via Vulkan) out of the
|
||
box; to use an Intel **NPU**, open **Settings ▸ Hardware** and download the one-time NPU
|
||
acceleration package. The app runs without admin rights, never adds itself to startup, and keeps all
|
||
data under `%LOCALAPPDATA%\WhispAssist`.
|
||
|
||
Prefer the NSIS installer? Grab **`WhispAssist_<version>_x64-setup.exe`** from the same page.
|
||
|
||
## Technology
|
||
|
||
WhispAssist is a **Tauri 2** application: a small Rust core with a compiled **Svelte + TypeScript**
|
||
frontend rendered through the OS WebView2 (no bundled browser → low idle memory). Every decision
|
||
and its alternatives are recorded in [`docs/adr/`](docs/adr/) (ADR-0001–0011).
|
||
|
||
- **Shell / IPC:** Tauri 2 (Rust ⇄ WebView2)
|
||
- **Audio capture:** WASAPI loopback
|
||
- **Transcription:** `whisper-rs` (whisper.cpp: CPU / **Vulkan** / CUDA) + ONNX Runtime (`ort`)
|
||
with the **OpenVINO** execution provider for the Intel NPU path
|
||
- **Diarization:** `sherpa-onnx`, fully offline
|
||
- **Storage:** SQLite + on-disk audio/transcript files
|
||
- **Local LLM:** Ollama HTTP API (localhost or a private LAN endpoint)
|
||
- **Encryption:** Argon2id + XChaCha20-Poly1305 envelope vault; secrets in the OS credential store
|
||
- **Calendar / Outlook:** `outlook-pst` for `.pst`, OS notifications for reminders *(in progress)*
|
||
- **Sync:** WebDAV primary set + OneDrive/Dropbox/Box via OAuth — off by default (ADR-0010)
|
||
- **External AI / agents:** hosted providers behind `LlmProvider`; a loopback-only **MCP server**
|
||
for coding-agent handoff — off by default (ADR-0011)
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
WhispAssist/
|
||
├── docs/ # The engineering plan (read this first)
|
||
│ ├── 00-overview.md … 07-research-findings.md
|
||
│ └── adr/ Architecture Decision Records (0001–0011)
|
||
├── src-tauri/ # Rust core — implemented service modules:
|
||
│ └── src/{audio,transcription,diarization,storage,llm,calendar,
|
||
│ hardware,notes,sync,mcp,vault}
|
||
├── src/ # Svelte + TypeScript frontend (views, stores, API bindings)
|
||
├── packaging/ # NPU/OpenVINO runtime bundle + release assets
|
||
├── scripts/ # Dev / model-download helpers
|
||
└── tests/ # Fixtures + cross-service integration tests
|
||
```
|
||
|
||
## Getting started (for builders)
|
||
|
||
Prerequisites: **Rust** (stable), **Node.js 20+**, the **Tauri CLI**, and the WebView2 runtime
|
||
(preinstalled on Windows 11). Native builds also need the **VS 2022 Build Tools** (load
|
||
`vcvars64.bat` first).
|
||
|
||
```bash
|
||
npm install
|
||
npm run tauri dev # CPU/NPU build
|
||
```
|
||
|
||
**GPU (Vulkan) build.** whisper.cpp's GPU backends are compiled in (not downloaded at runtime),
|
||
so a GPU build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and
|
||
a short target dir (to dodge Windows' 260-char path limit in the shader build):
|
||
|
||
```bash
|
||
# after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded
|
||
set VULKAN_SDK=C:\VulkanSDK\1.4.350.0
|
||
set CMAKE_GENERATOR=Ninja
|
||
set CARGO_TARGET_DIR=C:\wt
|
||
npm run tauri build -- --features vulkan
|
||
```
|
||
|
||
CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build
|
||
recipe lives in the project notes.
|
||
|
||
## Privacy guarantee
|
||
|
||
WhispAssist originates **no outbound connection for audio or transcript content** except to
|
||
destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted
|
||
AI provider if you choose one) and any sync targets you enable — plus explicit model downloads.
|
||
Everything optional is **off by default**; with nothing configured, WA makes no content egress at
|
||
all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on
|
||
loopback and adds no egress** — data only leaves via the agent's own provider, which WA discloses.
|
||
The set of reachable hosts is an allowlist derived from your settings and enforced in the core
|
||
(and verified by a CI network test). Recording is opt-in; sync/AI credentials live in the OS
|
||
credential store, never in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`,
|
||
`FR-SYNC-*`, `FR-MCP-*`) in [`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 0009–0011.
|
||
|
||
## License
|
||
|
||
Dual-licensed under MIT or Apache-2.0, at your option. See [`LICENSE`](LICENSE).
|