241 lines
16 KiB
Markdown
241 lines
16 KiB
Markdown
<p align="center">
|
||
<img src="bonus/clippy-says-hej.png" alt="WhispAssist logo with friendly Clippy saying hi there showing open source, private, note taker, and meeting advisor">
|
||
</p>
|
||
|
||
# WhispAssist (WA)
|
||
|
||
**A privacy-first, Windows-native meeting assistant that runs entirely on-device.**
|
||
|
||
WhispAssist captures system audio, transcribes it locally with Whisper-class models using
|
||
on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures the result into
|
||
Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and
|
||
transcripts **never leave the machine** unless you explicitly configure a destination.
|
||
|
||
> **Status: working application (v0.5.2).** Capture (system audio **+ your microphone**, with a
|
||
> live **dual level meter**), transcription (CPU / Intel NPU / Vulkan GPU) with a **fluid live
|
||
> transcript**, speaker diarization, storage + crash recovery, local-LLM summaries, AI tags,
|
||
> opt-in recording with **in-app playback**, at-rest encryption, self-hosted sync, **Outlook
|
||
> `.pst`/calendar import with optional auto-record**, **importing an existing recording from a
|
||
> file or URL**, and a loopback **MCP server** for coding-agent handoff are all implemented and
|
||
> ship as a single signed **MSI + NSIS** universal installer. Build order and remaining tasks
|
||
> are in [`docs/05-roadmap.md`](docs/05-roadmap.md).
|
||
|
||
## Why WhispAssist — Granola vs Meetily vs WhispAssist
|
||
|
||
| | **Granola** | **Meetily** | **WhispAssist** |
|
||
|---|---|---|---|
|
||
| **Positioning** | Cloud AI notepad | Open-source self-hosted assistant | Local, Windows-native assistant |
|
||
| **Platform** | macOS, Windows | macOS, Windows, Linux | Windows 10/11 |
|
||
| **Bot-free system-audio capture** | ✅ | ✅ | ✅ WASAPI loopback |
|
||
| **Transcription** | ☁️ cloud | ✅ local (Whisper) | ✅ local (whisper.cpp) |
|
||
| **NPU / GPU auto-acceleration** | n/a (cloud) | ⚠️ CPU/GPU, manual | ✅ **NPU→NVIDIA→AMD→Intel→CPU** ladder, one binary |
|
||
| **Speaker diarization** | ✅ cloud | ⚠️ limited | ✅ offline (sherpa-onnx) |
|
||
| **Summaries / notes AI** | ☁️ cloud LLM | ✅ local (Ollama) / BYO | ✅ local (Ollama, localhost **or LAN**) + optional hosted |
|
||
| **Interface localization (UI language)** | ⚠️ English-only UI (multilingual transcription) | ⚠️ English-only UI (more planned) | ✅ **20 languages** + RTL (Arabic/Urdu) |
|
||
| **Default data egress** | ☁️ audio + notes to cloud | 🔒 local (cloud optional) | 🔒 **none** — everything off by default, allowlist-enforced |
|
||
| **Calendar / Outlook `.pst` context** | ✅ cloud calendar | ❌ | ✅ local `.pst` import + **auto-record on events** |
|
||
| **Import an existing recording (file/URL)** | ❌ live capture only | ❌ | ✅ any file or URL (ffmpeg + yt-dlp) |
|
||
| **Opt-in recording + consent notice** | ⚠️ partial | ❌ | ✅ off by default, one-time consent |
|
||
| **At-rest encryption** | ☁️ server-side | ❌ | ✅ vault: Argon2id + XChaCha20-Poly1305 |
|
||
| **Self-hosted sync w/ client-side encryption** | ❌ | ⚠️ | ✅ WebDAV + OAuth, **encrypt-before-upload** |
|
||
| **Coding-agent (MCP) server** | ✅ **cloud** MCP (notes via their servers) | ❌ | ✅ **local, loopback-only**, token-gated, adds no egress |
|
||
| **License** | Proprietary | Open source (MIT) | Open source (MIT / Apache-2.0) |
|
||
| **Cost** | Subscription | Free | Free |
|
||
|
||
<sub>Comparison reflects each project's public positioning as of mid-2026. Granola now ships an
|
||
MCP server too, but it is **cloud-hosted** — an agent reaching it pulls your notes through
|
||
Granola's servers; WhispAssist's MCP server is **loopback-only and adds no egress of its own**
|
||
(data leaves only via the connected agent's own provider, which WA discloses). Granola and
|
||
Meetily are independent products and their capabilities evolve — verify current details before
|
||
relying on any row.</sub>
|
||
|
||
**The short version:** Granola is the polished cloud option (your audio and notes are processed
|
||
on their servers). Meetily is the closest peer — open-source and self-hosted — but is
|
||
cross-platform-generic and leans on manual setup. WhispAssist is the **Windows-native, hardware-
|
||
accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern laptops, keeps
|
||
everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a
|
||
coding-agent handoff.
|
||
|
||
## What's built (v0.5.2)
|
||
|
||
- **Bot-free capture — now both sides, with a live dual meter** — WASAPI loopback records the
|
||
system mix (all participants), and an optional **microphone** path captures your own voice,
|
||
mixed into both the live transcript and the saved recording. While recording, a **level meter
|
||
overlays the system and microphone signals in two colours** so you can see both sides are being
|
||
picked up. Pick a specific output/input or turn the mic off in **Settings ▸ Hardware**. No
|
||
meeting bot, no per-app plumbing.
|
||
- **Local transcription with a hardware ladder — and a fluid live transcript** — whisper.cpp via
|
||
`whisper-rs` on CPU; the **Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single
|
||
binary that runs on NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend
|
||
(**NPU → NVIDIA → AMD → Intel → CPU**), and shows it in the UI. The **live transcript streams a
|
||
growing line that refreshes ~once a second and commits at natural pauses** — words appear as
|
||
they're spoken instead of in fixed multi-second blocks, so sentences aren't chopped across
|
||
lines.
|
||
- **Speaker diarization** — `sherpa-onnx` (pyannote segmentation + speaker-embedding clustering),
|
||
fully offline. Install the two diarization models in **Settings ▸ Hardware** and finished
|
||
recordings are split by speaker (Speaker 1, Speaker 2, …); the microphone speaker is
|
||
auto-labelled from a short voiceprint.
|
||
- **Import an existing recording** — add a meeting from a **local audio/video file or a URL**
|
||
(YouTube, a streaming page, or a direct media link). WhispAssist transcribes and diarizes it
|
||
just like a live recording. Uses **`ffmpeg`** (and **`yt-dlp`** for URLs), which you install
|
||
yourself — neither is bundled.
|
||
- **Notes & summaries** — Markdown notes; local-LLM summaries via **Ollama** on `localhost`
|
||
**or a private LAN endpoint** (RFC-1918), with a full advanced-parameter panel (system prompt,
|
||
`think`, `keep_alive`, `num_ctx`, sampling/repetition/mirostat, etc.).
|
||
- **Storage & crash recovery** — SQLite + on-disk audio/transcripts under
|
||
`%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate
|
||
after a crash.
|
||
- **Opt-in recording + in-app playback** — off by default; `.wav` retained only when you turn it
|
||
on, after a one-time consent notice. Play a saved recording back in the app — encrypted
|
||
recordings are decrypted **in memory on the fly** (nothing plaintext is written to disk).
|
||
Recordings are 16-bit for roughly half the size, and an accidental recording can be **cancelled**
|
||
(audio + transcript deleted).
|
||
- **Notes, summaries & AI tags** — Markdown notes with an **Editor/Preview** toggle; local-LLM
|
||
summaries and one-click **tag generation** with a chip-based tag editor and tag filtering.
|
||
- **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts,
|
||
notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock.
|
||
- **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve,
|
||
Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with
|
||
backoff and **live per-item upload progress**; **client-side encryption before upload** so the
|
||
destination holds only ciphertext. Credentials live only in the OS credential store.
|
||
- **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same
|
||
`LlmProvider` interface, off by default (third-party egress, keys in the OS credential store).
|
||
- **Outlook `.pst` / calendar context, with auto-record** — import events and attendees from a
|
||
local Outlook `.pst` backup (read-only, range-limited, de-duplicated, with cleanup), attach
|
||
meetings to events, and — opt-in — **auto-start recording when a calendar event begins** while
|
||
the app is open (a one-shot timer, no background polling).
|
||
- **Local MCP server for coding-agent handoff** — hand meeting context to your own coding agents
|
||
(Claude, Codex, Copilot, OpenCode) over a **loopback-only, token-gated** MCP server that is off
|
||
by default, scope-limited, audited, and **adds no egress** — data leaves only via the agent's
|
||
own provider, which WA discloses.
|
||
- **20 interface languages, with RTL** — the whole UI (shell, meetings list, transcript/notes,
|
||
summary, and Settings) is available in **20 languages**, chosen under **Settings ▸ Language**:
|
||
Arabic (العربية), Bengali (বাংলা), Simplified Chinese (中文简体), English, Finnish (Suomi),
|
||
French — France & Canada (Français), German (Deutsch), Hindi (हिन्दी), Korean (한국어),
|
||
Norwegian Bokmål (Norsk bokmål), Portuguese — Brazil & Portugal (Português), Russian (Русский),
|
||
Sinhala (සිංහල), Spanish — Spain & Mexico (Español), Swedish (Svenska), Tamil (தமிழ்), and
|
||
Urdu (اردو). Selecting Arabic or Urdu flips the interface to **right-to-left**; any untranslated
|
||
string falls back to English rather than showing a raw key. Adding a language is a one-file job.
|
||
- **One universal installer** — a single signed MSI and NSIS `-setup.exe` that covers every
|
||
machine: Vulkan for all GPUs, the Intel NPU path (+ DirectML fallback), and CPU. The Vulkan
|
||
loader is bundled so it launches even on machines without a GPU driver.
|
||
|
||
**In progress:** Microsoft Graph calendar (cloud calendar via OAuth) and an optional CUDA
|
||
(NVIDIA-only) build variant.
|
||
|
||
## Quick start (install)
|
||
|
||
**Requirements:** Windows 10 or 11 (x64). WhispAssist needs the **WebView2** runtime — preinstalled
|
||
on Windows 11; on Windows 10 the installer fetches it automatically (so keep an internet connection
|
||
during setup).
|
||
|
||
1. Download the latest **`WhispAssist_<version>_x64_en-US.msi`** from the
|
||
[releases page](https://git.dou.bet/iamdoubz/WhispAssist/releases).
|
||
2. Double-click the `.msi` and follow the prompts, accepting the UAC prompt to install. The
|
||
installer is signed; if Windows SmartScreen appears, choose **More info → Run anyway**.
|
||
3. Launch **WhispAssist** from the Start menu.
|
||
|
||
On first run WA detects your hardware and picks the best transcription backend
|
||
(**NPU → NVIDIA → AMD → Intel → CPU**). It works on your **CPU or GPU** (GPU via Vulkan) out of the
|
||
box; to use an Intel **NPU**, open **Settings ▸ Hardware** and download the one-time NPU
|
||
acceleration package. The app runs without admin rights, never adds itself to startup, and keeps all
|
||
data under `%LOCALAPPDATA%\WhispAssist`.
|
||
|
||
Prefer the NSIS installer? Grab **`WhispAssist_<version>_x64-setup.exe`** from the same page.
|
||
|
||
## Optional dependencies
|
||
|
||
If you do not have these installed, WhispAssist will still work, but some features will be unavailable.
|
||
|
||
- [yt-dlp](https://github.com/yt-dlp/yt-dlp/releases)
|
||
- [ffmpeg](https://www.ffmpeg.org/download.html#build-windows)
|
||
- [libpst](https://sourceforge.net/projects/ezwinports/files/libpst-0.6.63-w32-bin.zip/)
|
||
|
||
## Technology
|
||
|
||
WhispAssist is a **Tauri 2** application: a small Rust core with a compiled **Svelte + TypeScript**
|
||
frontend rendered through the OS WebView2 (no bundled browser → low idle memory). Every decision
|
||
and its alternatives are recorded in [`docs/adr/`](docs/adr/) (ADR-0001–0011).
|
||
|
||
- **Shell / IPC:** Tauri 2 (Rust ⇄ WebView2)
|
||
- **Audio capture:** WASAPI loopback
|
||
- **Transcription:** `whisper-rs` (whisper.cpp: CPU / **Vulkan** / CUDA) + ONNX Runtime (`ort`)
|
||
with the **OpenVINO** execution provider for the Intel NPU path
|
||
- **Diarization:** `sherpa-onnx`, fully offline
|
||
- **Storage:** SQLite + on-disk audio/transcript files
|
||
- **Local LLM:** Ollama HTTP API (localhost or a private LAN endpoint)
|
||
- **Encryption:** Argon2id + XChaCha20-Poly1305 envelope vault; secrets in the OS credential store
|
||
- **Calendar / Outlook:** `readpst` for `.pst` import, OS scheduled toasts for action-item reminders
|
||
- **Sync:** WebDAV primary set + OneDrive/Dropbox/Box via OAuth — off by default (ADR-0010)
|
||
- **External AI / agents:** hosted providers behind `LlmProvider`; a loopback-only **MCP server**
|
||
for coding-agent handoff — off by default (ADR-0011)
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
WhispAssist/
|
||
├── docs/ # The engineering plan (read this first)
|
||
│ ├── 00-overview.md … 07-research-findings.md
|
||
│ └── adr/ Architecture Decision Records (0001–0011)
|
||
├── src-tauri/ # Rust core — implemented service modules:
|
||
│ └── src/{audio,transcription,diarization,storage,llm,calendar,
|
||
│ hardware,notes,sync,mcp,vault}
|
||
├── src/ # Svelte + TypeScript frontend (views, stores, API bindings)
|
||
├── packaging/ # NPU/OpenVINO runtime bundle + release assets
|
||
├── scripts/ # Dev / model-download helpers
|
||
└── tests/ # Fixtures + cross-service integration tests
|
||
```
|
||
|
||
## Getting started (for builders)
|
||
|
||
Prerequisites: **Rust** (stable), **Node.js 20+**, the **Tauri CLI**, and the WebView2 runtime
|
||
(preinstalled on Windows 11). Native builds also need the **VS 2022 Build Tools** (load
|
||
`vcvars64.bat` first).
|
||
|
||
```bash
|
||
npm install
|
||
npm run tauri dev # CPU/NPU build
|
||
```
|
||
|
||
**Release build (single universal installer).** whisper.cpp's GPU backends are compiled in (not
|
||
downloaded at runtime), so the release build needs a one-time toolchain setup — the **Vulkan SDK**,
|
||
a **Ninja** generator, and a short target dir (to dodge Windows' 260-char path limit in the shader
|
||
build):
|
||
|
||
```bash
|
||
# after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded
|
||
set VULKAN_SDK=C:\VulkanSDK\1.4.350.0
|
||
set CMAKE_GENERATOR=Ninja
|
||
set CARGO_TARGET_DIR=C:\wt
|
||
npm run tauri build -- --features vulkan --config src-tauri/tauri.vulkan.conf.json
|
||
```
|
||
|
||
This one build covers **every** machine: Vulkan accelerates all GPUs (NVIDIA/AMD/Intel), the Intel
|
||
NPU path works via the runtime OpenVINO download, and CPU is the fallback. The `--features vulkan`
|
||
binary links `vulkan-1.dll`, so `build.rs` stages the redistributable Vulkan **loader** (from
|
||
`VULKAN_SDK\Bin`, or System32) next to the exe and `tauri.vulkan.conf.json` bundles it into the
|
||
installer — the app then launches even on a machine with no GPU driver (it reports zero Vulkan
|
||
devices and decodes on the CPU). DirectML is intentionally not offered here because Vulkan already
|
||
covers those GPUs; it's the GPU path only in the plain `npm run tauri build` (no Vulkan) variant,
|
||
kept as an internal fallback.
|
||
|
||
CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build
|
||
recipe lives in the project notes.
|
||
|
||
## Privacy guarantee
|
||
|
||
WhispAssist originates **no outbound connection for audio or transcript content** except to
|
||
destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted
|
||
AI provider if you choose one) and any sync targets you enable — plus explicit model downloads.
|
||
Everything optional is **off by default**; with nothing configured, WA makes no content egress at
|
||
all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on
|
||
loopback and adds no egress** — data only leaves via the agent's own provider, which WA discloses.
|
||
The set of reachable hosts is an allowlist derived from your settings and enforced in the core
|
||
(and verified by a CI network test). Recording is opt-in; sync/AI credentials live in the OS
|
||
credential store, never in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`,
|
||
`FR-SYNC-*`, `FR-MCP-*`) in [`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 0009–0011.
|
||
|
||
## License
|
||
|
||
Dual-licensed under MIT or Apache-2.0, at your option. See [`LICENSE`](LICENSE).
|