# WhispAssist (WA)

**A privacy-first, Windows-native meeting assistant that runs entirely on-device.**

WhispAssist captures system audio, transcribes it locally with Whisper-class models using
on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures the result into
Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and
transcripts **never leave the machine** unless you explicitly configure a destination.

> **Status: working application (v0.1.5).** Capture, transcription (CPU / Intel NPU / Vulkan
> GPU), speaker diarization, storage + crash recovery, local-LLM summaries, opt-in recording,
> at-rest encryption, and self-hosted sync are implemented and ship as signed **MSI + NSIS**
> installers. Outlook `.pst`/calendar context and the coding-agent (MCP) handoff are in
> progress. Build order and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md).

## Why WhispAssist — Granola vs Meetily vs WhispAssist

| | **Granola** | **Meetily** | **WhispAssist** |
|---|---|---|---|
| **Positioning** | Cloud AI notepad | Open-source self-hosted assistant | Local, Windows-native assistant |
| **Platform** | macOS, Windows | macOS, Windows, Linux | Windows 10/11 |
| **Bot-free system-audio capture** | ✅ | ✅ | ✅ WASAPI loopback |
| **Transcription** | ☁️ cloud | ✅ local (Whisper) | ✅ local (whisper.cpp) |
| **NPU / GPU auto-acceleration** | n/a (cloud) | ⚠️ CPU/GPU, manual | ✅ **NPU→NVIDIA→AMD→Intel→CPU** ladder, one binary |
| **Speaker diarization** | ✅ cloud | ⚠️ limited | ✅ offline (sherpa-onnx) |
| **Summaries / notes AI** | ☁️ cloud LLM | ✅ local (Ollama) / BYO | ✅ local (Ollama, localhost **or LAN**) + optional hosted |
| **Default data egress** | ☁️ audio + notes to cloud | 🔒 local (cloud optional) | 🔒 **none** — everything off by default, allowlist-enforced |
| **Calendar / Outlook `.pst` context** | ✅ cloud calendar | ❌ | ⚙️ local `.pst` — in progress |
| **Opt-in recording + consent notice** | ⚠️ partial | ❌ | ✅ off by default, one-time consent |
| **At-rest encryption** | ☁️ server-side | ❌ | ✅ vault: Argon2id + XChaCha20-Poly1305 |
| **Self-hosted sync w/ client-side encryption** | ❌ | ⚠️ | ✅ WebDAV + OAuth, **encrypt-before-upload** |
| **Coding-agent (MCP) handoff** | ❌ | ❌ | ⚙️ local MCP server — in progress |
| **License** | Proprietary | Open source (MIT) | Open source (MIT / Apache-2.0) |
| **Cost** | Subscription | Free | Free |

<sub>Comparison reflects each project's public positioning as of mid-2026. Granola and Meetily
are independent products and their capabilities evolve — verify current details before relying
on any row.</sub>

**The short version:** Granola is the polished cloud option (your audio and notes are processed
on their servers). Meetily is the closest peer — open-source and self-hosted — but is
cross-platform-generic and leans on manual setup. WhispAssist is the **Windows-native, hardware-
accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern laptops, keeps
everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a
coding-agent handoff.

## What's built (v0.1.5)

- **Bot-free capture** — WASAPI loopback records the system mix (all participants) with no
  meeting bot and no per-app plumbing.
- **Local transcription with a hardware ladder** — whisper.cpp via `whisper-rs` on CPU; the
  **Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on
  NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend
  (**NPU → NVIDIA → AMD → Intel → CPU**), streams partial transcripts live, and shows the active
  backend in the UI.
- **Speaker diarization** — `sherpa-onnx` (pyannote segmentation + speaker-embedding
  clustering), fully offline.
- **Notes & summaries** — Markdown notes; local-LLM summaries via **Ollama** on `localhost`
  **or a private LAN endpoint** (RFC-1918), with a full advanced-parameter panel (system prompt,
  `think`, `keep_alive`, `num_ctx`, sampling/repetition/mirostat, etc.).
- **Storage & crash recovery** — SQLite + on-disk audio/transcripts under
  `%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate
  after a crash.
- **Opt-in recording** — off by default; `.wav` retained only when you turn it on, after a
  one-time consent notice.
- **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts,
  notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock.
- **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve,
  Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with
  backoff; **client-side encryption before upload** so the destination holds only ciphertext.
  Credentials live only in the OS credential store.
- **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same
  `LlmProvider` interface, off by default (third-party egress, keys in the OS credential store).
- **Installers** — signed MSI and NSIS `-setup.exe`.

**In progress:** Outlook `.pst` + calendar context, the local **MCP server** that hands meeting
context to your coding agents (Claude, Codex, Copilot, OpenCode), and MS Graph calendar.

## Quick start (install)

**Requirements:** Windows 10 or 11 (x64). WhispAssist needs the **WebView2** runtime — preinstalled
on Windows 11; on Windows 10 the installer fetches it automatically (so keep an internet connection
during setup).

1. Download the latest **`WhispAssist_<version>_x64_en-US.msi`** from the
   [releases page](https://git.dou.bet/iamdoubz/WhispAssist/releases).
2. Double-click the `.msi` and follow the prompts, accepting the UAC prompt to install. The
   installer is signed; if Windows SmartScreen appears, choose **More info → Run anyway**.
3. Launch **WhispAssist** from the Start menu.

On first run WA detects your hardware and picks the best transcription backend
(**NPU → NVIDIA → AMD → Intel → CPU**). It works on your **CPU or GPU** (GPU via Vulkan) out of the
box; to use an Intel **NPU**, open **Settings ▸ Hardware** and download the one-time NPU
acceleration package. The app runs without admin rights, never adds itself to startup, and keeps all
data under `%LOCALAPPDATA%\WhispAssist`.

Prefer the NSIS installer? Grab **`WhispAssist_<version>_x64-setup.exe`** from the same page.

## Technology

WhispAssist is a **Tauri 2** application: a small Rust core with a compiled **Svelte + TypeScript**
frontend rendered through the OS WebView2 (no bundled browser → low idle memory). Every decision
and its alternatives are recorded in [`docs/adr/`](docs/adr/) (ADR-0001–0011).

- **Shell / IPC:** Tauri 2 (Rust ⇄ WebView2)
- **Audio capture:** WASAPI loopback
- **Transcription:** `whisper-rs` (whisper.cpp: CPU / **Vulkan** / CUDA) + ONNX Runtime (`ort`)
  with the **OpenVINO** execution provider for the Intel NPU path
- **Diarization:** `sherpa-onnx`, fully offline
- **Storage:** SQLite + on-disk audio/transcript files
- **Local LLM:** Ollama HTTP API (localhost or a private LAN endpoint)
- **Encryption:** Argon2id + XChaCha20-Poly1305 envelope vault; secrets in the OS credential store
- **Calendar / Outlook:** `outlook-pst` for `.pst`, OS notifications for reminders *(in progress)*
- **Sync:** WebDAV primary set + OneDrive/Dropbox/Box via OAuth — off by default (ADR-0010)
- **External AI / agents:** hosted providers behind `LlmProvider`; a loopback-only **MCP server**
  for coding-agent handoff — off by default (ADR-0011)

## Repository layout

```
WhispAssist/
├── docs/                 # The engineering plan (read this first)
│   ├── 00-overview.md … 07-research-findings.md
│   └── adr/                  Architecture Decision Records (0001–0011)
├── src-tauri/            # Rust core — implemented service modules:
│   └── src/{audio,transcription,diarization,storage,llm,calendar,
│            hardware,notes,sync,mcp,vault}
├── src/                  # Svelte + TypeScript frontend (views, stores, API bindings)
├── packaging/            # NPU/OpenVINO runtime bundle + release assets
├── scripts/              # Dev / model-download helpers
└── tests/                # Fixtures + cross-service integration tests
```

## Getting started (for builders)

Prerequisites: **Rust** (stable), **Node.js 20+**, the **Tauri CLI**, and the WebView2 runtime
(preinstalled on Windows 11). Native builds also need the **VS 2022 Build Tools** (load
`vcvars64.bat` first).

```bash
npm install
npm run tauri dev      # CPU/NPU build
```

**GPU (Vulkan) build.** whisper.cpp's GPU backends are compiled in (not downloaded at runtime),
so a GPU build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and
a short target dir (to dodge Windows' 260-char path limit in the shader build):

```bash
# after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded
set VULKAN_SDK=C:\VulkanSDK\1.4.350.0
set CMAKE_GENERATOR=Ninja
set CARGO_TARGET_DIR=C:\wt
npm run tauri build -- --features vulkan
```

CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build
recipe lives in the project notes.

## Privacy guarantee

WhispAssist originates **no outbound connection for audio or transcript content** except to
destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted
AI provider if you choose one) and any sync targets you enable — plus explicit model downloads.
Everything optional is **off by default**; with nothing configured, WA makes no content egress at
all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on
loopback and adds no egress** — data only leaves via the agent's own provider, which WA discloses.
The set of reachable hosts is an allowlist derived from your settings and enforced in the core
(and verified by a CI network test). Recording is opt-in; sync/AI credentials live in the OS
credential store, never in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`,
`FR-SYNC-*`, `FR-MCP-*`) in [`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 0009–0011.

## License

Dual-licensed under MIT or Apache-2.0, at your option. See [`LICENSE`](LICENSE).