# WhispAssist (WA) **A privacy-first, Windows-native meeting assistant that runs entirely on-device.** WhispAssist captures system audio, transcribes it locally with Whisper-class models using on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures the result into Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and transcripts **never leave the machine** unless you explicitly configure a destination. > **Status: working application (v0.7.3).** Capture (system audio **+ your microphone**, with a > live **dual level meter** and an **M-key mic mute**), transcription (CPU / Intel NPU / Vulkan > GPU) with a **fluid live transcript**, speaker diarization that labels **You** live from your > own mic channel, storage + crash recovery, local-LLM summaries, AI tags, **AI-enhanced notes**, > opt-in recording with **in-app playback**, at-rest encryption, self-hosted sync, **Outlook > `.pst`/calendar import with optional auto-record**, **background import of an existing > recording from a file or URL** (with a step-by-step progress tracker), **close-to-tray**, and a > loopback **MCP server** for coding-agent handoff are all implemented and ship as a single > **MSI + NSIS** universal installer. On Intel Core Ultra machines, transcription runs with the > **CPU near-idle**: Whisper's encoder on the **NPU**, its decoder on the **iGPU**. Build order > and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md). ## Why WhispAssist — Granola vs Meetily vs WhispAssist | | **Granola** | **Meetily** | **WhispAssist** | | ---------------------------------------------- | ----------------------------------------------- | --------------------------------- | ----------------------------------------------------------- | | **Positioning** | Cloud AI notepad | Open-source self-hosted assistant | Local, Windows-native assistant | | **Platform** | macOS, Windows | macOS, Windows, Linux | Windows 10/11 | | **Bot-free system-audio capture** | ✅ | ✅ | ✅ WASAPI loopback | | **Transcription** | ☁️ cloud | ✅ local (Whisper) | ✅ local (whisper.cpp) | | **NPU / GPU auto-acceleration** | n/a (cloud) | ⚠️ CPU/GPU, manual | ✅ **NPU→NVIDIA→AMD→Intel→CPU** ladder, one binary | | **Speaker diarization** | ✅ cloud | ⚠️ limited | ✅ offline (sherpa-onnx) | | **Summaries / notes AI** | ☁️ cloud LLM | ✅ local (Ollama) / BYO | ✅ local (Ollama, localhost **or LAN**) + optional hosted | | **Interface localization (UI language)** | ⚠️ English-only UI (multilingual transcription) | ⚠️ English-only UI (more planned) | ✅ **20 languages** + RTL (Arabic/Urdu) | | **Default data egress** | ☁️ audio + notes to cloud | 🔒 local (cloud optional) | 🔒 **none** — everything off by default, allowlist-enforced | | **Calendar / Outlook `.pst` context** | ✅ cloud calendar | ❌ | ✅ local `.pst` import + **auto-record on events** | | **Import an existing recording (file/URL)** | ❌ live capture only | ❌ | ✅ any file or URL (ffmpeg + yt-dlp) | | **Opt-in recording + consent notice** | ⚠️ partial | ❌ | ✅ off by default, one-time consent | | **At-rest encryption** | ☁️ server-side | ❌ | ✅ vault: Argon2id + XChaCha20-Poly1305 | | **Self-hosted sync w/ client-side encryption** | ❌ | ⚠️ | ✅ WebDAV + OAuth, **encrypt-before-upload** | | **Coding-agent (MCP) server** | ✅ **cloud** MCP (notes via their servers) | ❌ | ✅ **local, loopback-only**, token-gated, adds no egress | | **License** | Proprietary | Open source (MIT) | Open source (MIT / Apache-2.0) | | **Cost** | Subscription | Free | Free | _{Comparison reflects each project's public positioning as of mid-2026. Granola now ships an MCP server too, but it is **cloud-hosted** — an agent reaching it pulls your notes through Granola's servers; WhispAssist's MCP server is **loopback-only and adds no egress of its own** (data leaves only via the connected agent's own provider, which WA discloses). Granola and Meetily are independent products and their capabilities evolve — verify current details before relying on any row.} **The short version:** Granola is the polished cloud option (your audio and notes are processed on their servers). Meetily is the closest peer — open-source and self-hosted — but is cross-platform-generic and leans on manual setup. WhispAssist is the **Windows-native, hardware- accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern laptops, keeps everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a coding-agent handoff. ## What's built (v0.7.3) - **Bot-free capture — now both sides, with a live dual meter and mic mute** — WASAPI loopback records the system mix (all participants), and an optional **microphone** path captures your own voice, mixed into both the live transcript and the saved recording. While recording, a **level meter overlays the system and microphone signals in two colours** so you can see both sides are being picked up, and the **M key (or the mic button) mutes/unmutes your mic mid-meeting**. Pick a specific output/input or turn the mic off in **Settings ▸ Hardware**. No meeting bot, no per-app plumbing. - **Local transcription with a hardware ladder — and a fluid live transcript** — whisper.cpp via `whisper-rs` on CPU; the **Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend (**NPU → NVIDIA → AMD → Intel → CPU**), and shows it in the UI. On NPU machines the engine **splits Whisper across accelerators — encoder on the NPU, KV-cache decoder on the Intel iGPU — so transcription leaves the CPU near-idle**. The **live transcript streams a growing line that refreshes ~once a second and commits at natural pauses** — words appear as they're spoken instead of in fixed multi-second blocks, so sentences aren't chopped across lines. - **Speaker diarization that knows who _you_ are** — `sherpa-onnx` (pyannote segmentation + speaker-embedding clustering), fully offline. Install the two diarization models in **Settings ▸ Hardware** and recordings are split by speaker (You, Speaker 2, …). With the mic enabled, your voice is recorded on its **own channel**, so your lines are labelled **You** live — from the first pass, not just after the meeting — and attributed by per-channel voice activity rather than clustering guesswork. Renaming a speaker updates the transcript **and the generated notes**. - **Import an existing recording — in the background, with a progress tracker** — add a meeting from a **local audio/video file or a URL** (YouTube, a streaming page, or a direct media link); it transcribes quietly in the background behind a four-step tracker (_Transcode → Transcribe → Identify speakers → Finalize_) while you keep working. Uses **`ffmpeg`** (and **`yt-dlp`** for URLs), which you install yourself — neither is bundled, and one-click download buttons are in the Import dialog. - **Notes & summaries** — Markdown notes; local-LLM summaries via **Ollama** on `localhost` **or a private LAN endpoint** (RFC-1918), with a full advanced-parameter panel (system prompt, `think`, `keep_alive`, `num_ctx`, sampling/repetition/mirostat, etc.). - **Storage & crash recovery** — SQLite + on-disk audio/transcripts under `%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate after a crash. - **Opt-in recording + in-app playback** — off by default; `.wav` retained only when you turn it on, after a one-time consent notice. Play a saved recording back in the app — encrypted recordings are decrypted **in memory on the fly** (nothing plaintext is written to disk). Recordings are 16-bit for roughly half the size, and an accidental recording can be **cancelled** (audio + transcript deleted). - **Notes, summaries & AI tags** — Markdown notes with an **Editor/Preview** toggle, a formatting toolbar, and **slash commands** (`/todo`, `/h1`, `/quote`, …); **AI-enhance** turns rough notes into clean structured Markdown via your local LLM (with one-step Undo); local-LLM summaries and one-click **tag generation** with a chip-based tag editor and tag filtering. - **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts, notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock. - **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve, Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with backoff and **live per-item upload progress**; **client-side encryption before upload** so the destination holds only ciphertext. Credentials live only in the OS credential store. - **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same `LlmProvider` interface, off by default (third-party egress, keys in the OS credential store). - **Outlook `.pst` / calendar context, with auto-record** — import events and attendees from a local Outlook `.pst` backup (read-only, range-limited, de-duplicated, with cleanup), attach meetings to events, and — opt-in — **auto-start recording when a calendar event begins** while the app is open (a one-shot timer, no background polling). The import shells out to `readpst` (libpst) — **Settings ▸ Calendar** has a one-click download button for it. - **Close to tray** — closing the window keeps WhispAssist running in the background; reopen it from the tray icon, quit fully from the tray menu. On by default, toggleable in **Settings ▸ Recording**. - **Local MCP server for coding-agent handoff** — hand meeting context to your own coding agents (Claude, Codex, Copilot, OpenCode) over a **loopback-only, token-gated** MCP server that is off by default, scope-limited, audited, and **adds no egress** — data leaves only via the agent's own provider, which WA discloses. - **20 interface languages, with RTL** — the whole UI (shell, meetings list, transcript/notes, summary, and Settings) is available in **20 languages**, chosen under **Settings ▸ Language**: Arabic (العربية), Bengali (বাংলা), Simplified Chinese (中文简体), English, Finnish (Suomi), French — France & Canada (Français), German (Deutsch), Hindi (हिन्दी), Korean (한국어), Norwegian Bokmål (Norsk bokmål), Portuguese — Brazil & Portugal (Português), Russian (Русский), Sinhala (සිංහල), Spanish — Spain & Mexico (Español), Swedish (Svenska), Tamil (தமிழ்), and Urdu (اردو). Selecting Arabic or Urdu flips the interface to **right-to-left**; any untranslated string falls back to English rather than showing a raw key. Adding a language is a one-file job. - **One universal installer** — a single MSI and NSIS `-setup.exe` that covers every machine: Vulkan for all GPUs, the Intel NPU path (+ DirectML fallback), and CPU. The Vulkan loader is bundled so it launches even on machines without a GPU driver. Every release ships **SHA-256 checksums** in its release notes. **In progress:** Microsoft Graph calendar (cloud calendar via OAuth) and an optional CUDA (NVIDIA-only) build variant. ## Quick start (install) **Requirements:** Windows 10 or 11 (x64). WhispAssist needs the **WebView2** runtime — preinstalled on Windows 11; on Windows 10 the installer fetches it automatically (so keep an internet connection during setup). 1. Download the latest **`WhispAssist__x64_en-US.msi`** from the [releases page](https://git.dou.bet/iamdoubz/WhispAssist/releases). 2. Double-click the `.msi` and follow the prompts, accepting the UAC prompt to install. The installers are currently unsigned, so if Windows SmartScreen appears, choose **More info → Run anyway** (verify the download against the SHA-256 checksums in the release notes first). 3. Launch **WhispAssist** from the Start menu. On first run WA detects your hardware and picks the best transcription backend (**NPU → NVIDIA → AMD → Intel → CPU**). It works on your **CPU or GPU** (GPU via Vulkan) out of the box; to use an Intel **NPU**, open **Settings ▸ Hardware** and download the one-time NPU acceleration package. The app runs without admin rights, does not add itself to startup unless you opt in (**Settings ▸ Recording ▸ Launch at login**), and keeps all data under `%LOCALAPPDATA%\WhispAssist`. Prefer the NSIS installer? Grab **`WhispAssist__x64-setup.exe`** from the same page — it lets you choose a **current-user** (no admin) or **all-users** install. **Deploying to many machines?** See [`docs/enterprise-deployment.md`](docs/enterprise-deployment.md) for silent install, custom install location, and presetting defaults with a `wa-defaults.ini` file. ## Optional dependencies If you do not have these installed, WhispAssist will still work, but some features will be unavailable. The app links to each one where it's needed: `ffmpeg`/`yt-dlp` in the Import dialog, `libpst` in **Settings ▸ Calendar**. - [yt-dlp](https://github.com/yt-dlp/yt-dlp/releases) — importing from URLs - [ffmpeg](https://www.ffmpeg.org/download.html#build-windows) — importing files/URLs - [libpst](https://sourceforge.net/projects/ezwinports/files/libpst-0.6.63-w32-bin.zip/) — Outlook `.pst` calendar import ## Technology WhispAssist is a **Tauri 2** application: a small Rust core with a compiled **Svelte + TypeScript** frontend rendered through the OS WebView2 (no bundled browser → low idle memory). Every decision and its alternatives are recorded in [`docs/adr/`](docs/adr/) (ADR-0001–0012). - **Shell / IPC:** Tauri 2 (Rust ⇄ WebView2) - **Audio capture:** WASAPI loopback - **Transcription:** `whisper-rs` (whisper.cpp: CPU / **Vulkan** / CUDA) + ONNX Runtime (`ort`) with the **OpenVINO** execution provider for the Intel NPU path - **Diarization:** `sherpa-onnx`, fully offline - **Storage:** SQLite + on-disk audio/transcript files - **Local LLM:** Ollama HTTP API (localhost or a private LAN endpoint) - **Encryption:** Argon2id + XChaCha20-Poly1305 envelope vault; secrets in the OS credential store - **Calendar / Outlook:** `readpst` for `.pst` import, OS scheduled toasts for action-item reminders - **Sync:** WebDAV primary set + OneDrive/Dropbox/Box via OAuth — off by default (ADR-0010) - **External AI / agents:** hosted providers behind `LlmProvider`; a loopback-only **MCP server** for coding-agent handoff — off by default (ADR-0011) ## Repository layout ``` WhispAssist/ ├── docs/ # The engineering plan (read this first) │ ├── 00-overview.md … 07-research-findings.md │ └── adr/ Architecture Decision Records (0001–0012) ├── src-tauri/ # Rust core — implemented service modules: │ └── src/{audio,transcription,diarization,storage,llm,calendar, │ hardware,notes,sync,mcp,vault} ├── src/ # Svelte + TypeScript frontend (views, stores, API bindings) ├── packaging/ # NPU/OpenVINO runtime bundle + release assets ├── scripts/ # Dev / model-download helpers └── tests/ # Fixtures + cross-service integration tests ``` ## Getting started (for builders) Prerequisites: **Rust** (stable), **Node.js 20+**, the **Tauri CLI**, and the WebView2 runtime (preinstalled on Windows 11). Native builds also need the **VS 2022 Build Tools** (load `vcvars64.bat` first). ```bash npm install npm run tauri dev # CPU/NPU build ``` **Release build (single universal installer).** whisper.cpp's GPU backends are compiled in (not downloaded at runtime), so the release build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and a short target dir (to dodge Windows' 260-char path limit in the shader build): ```bash # after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded set VULKAN_SDK=C:\VulkanSDK\1.4.350.0 set CMAKE_GENERATOR=Ninja set CARGO_TARGET_DIR=C:\wt npm run tauri build -- --features vulkan --config src-tauri/tauri.vulkan.conf.json ``` This one build covers **every** machine: Vulkan accelerates all GPUs (NVIDIA/AMD/Intel), the Intel NPU path works via the runtime OpenVINO download, and CPU is the fallback. The `--features vulkan` binary links `vulkan-1.dll`, so `build.rs` stages the redistributable Vulkan **loader** (from `VULKAN_SDK\Bin`, or System32) next to the exe and `tauri.vulkan.conf.json` bundles it into the installer — the app then launches even on a machine with no GPU driver (it reports zero Vulkan devices and decodes on the CPU). DirectML is intentionally not offered here because Vulkan already covers those GPUs; it's the GPU path only in the plain `npm run tauri build` (no Vulkan) variant, kept as an internal fallback. CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build recipe lives in the project notes. ## Privacy guarantee WhispAssist originates **no outbound connection for audio or transcript content** except to destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted AI provider if you choose one) and any sync targets you enable — plus explicit model downloads. Everything optional is **off by default**; with nothing configured, WA makes no content egress at all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on loopback and adds no egress** — data only leaves via the agent's own provider, which WA discloses. The set of reachable hosts is an allowlist derived from your settings and enforced in the core (and verified by a CI network test). Recording is opt-in; sync/AI credentials live in the OS credential store, never in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`, `FR-SYNC-*`, `FR-MCP-*`) in [`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 0009–0011. ## License Dual-licensed under MIT or Apache-2.0, at your option. See [`LICENSE`](LICENSE).