Feature again here #13

Merged
iamdoubz merged 13 commits from feature_again_here into main 2026-07-05 18:20:16 -05:00
13 changed files with 1002 additions and 88 deletions
+117 -53
View File
@@ -3,88 +3,152 @@
**A privacy-first, Windows-native meeting assistant that runs entirely on-device.**
WhispAssist captures system audio, transcribes it locally with Whisper-class models using
on-device acceleration (NPU → GPU → CPU), structures the result into Markdown notes, and
optionally augments them with a locally hosted LLM (Ollama). Audio and transcripts **never
leave the machine** unless the user explicitly exports them.
on-device acceleration (**NPU → GPU → CPU**), labels speakers, structures the result into
Markdown notes, and optionally augments them with a locally hosted LLM (Ollama). Audio and
transcripts **never leave the machine** unless you explicitly configure a destination.
> Status: **planning + scaffold**. This repository currently contains the full engineering
> plan (`docs/`) and a compiling-intent skeleton (`src-tauri/`, `src/`). No feature code is
> implemented yet. See [`docs/05-roadmap.md`](docs/05-roadmap.md) for the build order.
> **Status: working application (v0.1.4).** Capture, transcription (CPU / Intel NPU / Vulkan
> GPU), speaker diarization, storage + crash recovery, local-LLM summaries, opt-in recording,
> at-rest encryption, and self-hosted sync are implemented and ship as signed **MSI + NSIS**
> installers. Outlook `.pst`/calendar context and the coding-agent (MCP) handoff are in
> progress. Build order and remaining tasks are in [`docs/05-roadmap.md`](docs/05-roadmap.md).
## Why WhispAssist
## Why WhispAssist — Granola vs Meetily vs WhispAssist
| | Granola | Meetily | **WhispAssist** |
| | **Granola** | **Meetily** | **WhispAssist** |
|---|---|---|---|
| Local transcription | ❌ cloud | ✅ | ✅ |
| Notes/summaries stay local | ❌ cloud AI | ⚠️ optional | ✅ local-only by design |
| NPU/GPU auto-acceleration | n/a | partial | ✅ NPU→GPU→CPU ladder |
| Calendar + Outlook `.pst` context | ✅ (cloud) | ❌ | ✅ local |
| Bot-free system-audio capture | ✅ | ✅ | ✅ |
| License | proprietary | MIT | open source |
| **Positioning** | Cloud AI notepad | Open-source self-hosted assistant | Local, Windows-native assistant |
| **Platform** | macOS, Windows | macOS, Windows, Linux | Windows 10/11 |
| **Bot-free system-audio capture** | ✅ | ✅ | ✅ WASAPI loopback |
| **Transcription** | ☁️ cloud | ✅ local (Whisper) | ✅ local (whisper.cpp) |
| **NPU / GPU auto-acceleration** | n/a (cloud) | ⚠️ CPU/GPU, manual | ✅ **NPU→NVIDIA→AMD→Intel→CPU** ladder, one binary |
| **Speaker diarization** | ✅ cloud | ⚠️ limited | ✅ offline (sherpa-onnx) |
| **Summaries / notes AI** | ☁️ cloud LLM | ✅ local (Ollama) / BYO | ✅ local (Ollama, localhost **or LAN**) + optional hosted |
| **Default data egress** | ☁️ audio + notes to cloud | 🔒 local (cloud optional) | 🔒 **none** — everything off by default, allowlist-enforced |
| **Calendar / Outlook `.pst` context** | ✅ cloud calendar | ❌ | ⚙️ local `.pst` — in progress |
| **Opt-in recording + consent notice** | ⚠️ partial | ❌ | ✅ off by default, one-time consent |
| **At-rest encryption** | ☁️ server-side | ❌ | ✅ vault: Argon2id + XChaCha20-Poly1305 |
| **Self-hosted sync w/ client-side encryption** | ❌ | ⚠️ | ✅ WebDAV + OAuth, **encrypt-before-upload** |
| **Coding-agent (MCP) handoff** | ❌ | ❌ | ⚙️ local MCP server — in progress |
| **License** | Proprietary | Open source (MIT) | Open source (MIT / Apache-2.0) |
| **Cost** | Subscription | Free | Free |
<sub>Comparison reflects each project's public positioning as of mid-2026. Granola and Meetily
are independent products and their capabilities evolve — verify current details before relying
on any row.</sub>
**The short version:** Granola is the polished cloud option (your audio and notes are processed
on their servers). Meetily is the closest peer — open-source and self-hosted — but is
cross-platform-generic and leans on manual setup. WhispAssist is the **Windows-native, hardware-
accelerated, zero-egress-by-default** option: it exploits the NPU/GPU in modern laptops, keeps
everything on the device unless you opt in, and adds Windows-specific context (Outlook) and a
coding-agent handoff.
## What's built (v0.1.4)
- **Bot-free capture** — WASAPI loopback records the system mix (all participants) with no
meeting bot and no per-app plumbing.
- **Local transcription with a hardware ladder** — whisper.cpp via `whisper-rs` on CPU; the
**Intel NPU** via ONNX Runtime + OpenVINO; **GPU via Vulkan** (a single binary that runs on
NVIDIA, AMD, and Intel). WA detects the hardware, picks the best backend
(**NPU → NVIDIA → AMD → Intel → CPU**), streams partial transcripts live, and shows the active
backend in the UI.
- **Speaker diarization** — `sherpa-onnx` (pyannote segmentation + speaker-embedding
clustering), fully offline.
- **Notes & summaries** — Markdown notes; local-LLM summaries via **Ollama** on `localhost`
**or a private LAN endpoint** (RFC-1918), with a full advanced-parameter panel (system prompt,
`think`, `keep_alive`, `num_ctx`, sampling/repetition/mirostat, etc.).
- **Storage & crash recovery** — SQLite + on-disk audio/transcripts under
`%LOCALAPPDATA%\WhispAssist`. Audio is the source of truth; notes and transcripts regenerate
after a crash.
- **Opt-in recording** — off by default; `.wav` retained only when you turn it on, after a
one-time consent notice.
- **At-rest encryption vault** — Argon2id key derivation + XChaCha20-Poly1305; transcripts,
notes, summaries, and recordings sealed on disk; startup unlock gate; keys zeroized on lock.
- **Self-hosted sync (optional, off by default)** — WebDAV (Nextcloud, ownCloud, Cloudreve,
Seafile, Synology) plus OneDrive/Dropbox/Box (OAuth 2.0 PKCE); durable retry queue with
backoff; **client-side encryption before upload** so the destination holds only ciphertext.
Credentials live only in the OS credential store.
- **Optional hosted AI** — Anthropic and OpenAI-compatible providers behind the same
`LlmProvider` interface, off by default (third-party egress, keys in the OS credential store).
- **Installers** — signed MSI and NSIS `-setup.exe`.
**In progress:** Outlook `.pst` + calendar context, the local **MCP server** that hands meeting
context to your coding agents (Claude, Codex, Copilot, OpenCode), and MS Graph calendar.
## Technology
WhispAssist is built as a **Tauri 2** application: a small Rust core with a compiled
**Svelte** web frontend rendered through the OS WebView2 (no bundled browser → low idle
memory). The choice and its alternatives are recorded in [`docs/adr/`](docs/adr/).
WhispAssist is a **Tauri 2** application: a small Rust core with a compiled **Svelte + TypeScript**
frontend rendered through the OS WebView2 (no bundled browser → low idle memory). Every decision
and its alternatives are recorded in [`docs/adr/`](docs/adr/) (ADR-00010011).
- **Shell / IPC:** Tauri 2 (Rust ⇄ WebView2)
- **Audio capture:** WASAPI loopback (`wasapi` crate)
- **Transcription:** `whisper-rs` (whisper.cpp: CPU/Vulkan/CUDA) + ONNX Runtime (`ort`) DirectML for the NPU path
- **Diarization:** `sherpa-onnx` (pyannote segmentation + speaker-embedding clustering), fully offline
- **Storage:** SQLite (`sqlx`/`rusqlite`) + on-disk audio/transcript files
- **Local LLM:** Ollama HTTP API on `localhost:11434`
- **Calendar / Outlook:** `outlook-pst` crate for `.pst`, OS notifications for reminders
- **Optional recording:** opt-in (default off), saved as `.wav`, with a consent reminder (ADR-0009)
- **Optional sync:** upload artifacts to your own server — WebDAV covers **Nextcloud, ownCloud, Cloudreve, Seafile, Synology**; **OneDrive/Dropbox/Box** via OAuth. Off by default (ADR-0010)
- **Optional AI/agent integration:** hosted summary providers (Anthropic, OpenAI-compatible) behind the same provider model, **and** a local **MCP server** so your own coding agents (**Claude, Codex, Copilot, OpenCode, …**) can pull meeting context and "feature briefs" to start coding. Off by default; the MCP server is inbound/loopback only (ADR-0011)
- **Audio capture:** WASAPI loopback
- **Transcription:** `whisper-rs` (whisper.cpp: CPU / **Vulkan** / CUDA) + ONNX Runtime (`ort`)
with the **OpenVINO** execution provider for the Intel NPU path
- **Diarization:** `sherpa-onnx`, fully offline
- **Storage:** SQLite + on-disk audio/transcript files
- **Local LLM:** Ollama HTTP API (localhost or a private LAN endpoint)
- **Encryption:** Argon2id + XChaCha20-Poly1305 envelope vault; secrets in the OS credential store
- **Calendar / Outlook:** `outlook-pst` for `.pst`, OS notifications for reminders *(in progress)*
- **Sync:** WebDAV primary set + OneDrive/Dropbox/Box via OAuth — off by default (ADR-0010)
- **External AI / agents:** hosted providers behind `LlmProvider`; a loopback-only **MCP server**
for coding-agent handoff — off by default (ADR-0011)
## Repository layout
```
WhispAssist/
├── docs/ # The engineering plan (read this first)
│ ├── 00-overview.md Vision, goals, glossary
── 01-requirements.md Functional + non-functional requirements (traceable IDs)
│ ├── 02-architecture.md Components, data flow, threading model
── 03-data-model.md SQLite schema, file layout, transcript JSON
├── 04-api-contracts.md Tauri commands/events + internal Rust service traits
│ ├── 05-roadmap.md 8 phases, task breakdown, acceptance criteria
│ ├── 06-test-strategy.md Test plan per phase + quality gates
│ ├── 07-research-findings.md Validated stack with sources
│ └── adr/ Architecture Decision Records (00010010)
├── src-tauri/ # Rust core (service module skeletons)
├── src/ # Svelte frontend skeleton
├── scripts/ # Dev/model-download helper scripts
└── tests/ # Test fixtures + integration test scaffolding
│ ├── 00-overview.md … 07-research-findings.md
── adr/ Architecture Decision Records (00010011)
├── src-tauri/ # Rust core — implemented service modules:
── src/{audio,transcription,diarization,storage,llm,calendar,
hardware,notes,sync,mcp,vault}
├── src/ # Svelte + TypeScript frontend (views, stores, API bindings)
├── packaging/ # NPU/OpenVINO runtime bundle + release assets
├── scripts/ # Dev / model-download helpers
└── tests/ # Fixtures + cross-service integration tests
```
## Getting started (for builders)
Prerequisites once implementation begins: Rust (stable), Node.js 20+, the Tauri CLI, and
WebView2 runtime (preinstalled on Windows 11). Then:
Prerequisites: **Rust** (stable), **Node.js 20+**, the **Tauri CLI**, and the WebView2 runtime
(preinstalled on Windows 11). Native builds also need the **VS 2022 Build Tools** (load
`vcvars64.bat` first).
```bash
npm install
npm run tauri dev # once src-tauri/Cargo.toml dependencies are filled in
npm run tauri dev # CPU/NPU build
```
The current skeleton intentionally does **not** compile end-to-end — modules contain typed
stubs and `todo!()` markers that map 1:1 to roadmap tasks. Start at Phase 1 in
[`docs/05-roadmap.md`](docs/05-roadmap.md).
**GPU (Vulkan) build.** whisper.cpp's GPU backends are compiled in (not downloaded at runtime),
so a GPU build needs a one-time toolchain setup — the **Vulkan SDK**, a **Ninja** generator, and
a short target dir (to dodge Windows' 260-char path limit in the shader build):
```bash
# after: Vulkan SDK installed, ninja.exe on PATH, vcvars64 loaded
set VULKAN_SDK=C:\VulkanSDK\1.4.350.0
set CMAKE_GENERATOR=Ninja
set CARGO_TARGET_DIR=C:\wt
npm run tauri build -- --features vulkan
```
CUDA (NVIDIA-only, faster) is planned as an optional variant. The full, gotcha-annotated build
recipe lives in the project notes.
## Privacy guarantee
WhispAssist originates **no outbound connection for audio or transcript content** except to
destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted AI
provider if you choose one) and any sync targets you enable — plus explicit model downloads.
destinations **you explicitly configure** — an LLM endpoint (local Ollama by default, or a hosted
AI provider if you choose one) and any sync targets you enable — plus explicit model downloads.
Everything optional is **off by default**; with nothing configured, WA makes no content egress at
all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on loopback
and adds no egress** — data only leaves via the agent's own provider, which WA discloses. The set of
reachable hosts is an allowlist derived from your settings and enforced in the core (and verified by
a CI network test). Recording is opt-in; sync/AI credentials live in the OS credential store, never
in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`, `FR-SYNC-*`, `FR-MCP-*`) in
[`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 00090011.
all. The local **MCP server** (for handing meetings to your coding agents) is **inbound on
loopback and adds no egress** — data only leaves via the agent's own provider, which WA discloses.
The set of reachable hosts is an allowlist derived from your settings and enforced in the core
(and verified by a CI network test). Recording is opt-in; sync/AI credentials live in the OS
credential store, never in config files. See the privacy requirements (`FR-SEC-*`, `NFR-SEC-*`,
`FR-SYNC-*`, `FR-MCP-*`) in [`docs/01-requirements.md`](docs/01-requirements.md) and ADRs 00090011.
## License
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "whispassist",
"private": true,
"version": "0.1.2",
"version": "0.1.4",
"type": "module",
"description": "Privacy-first, fully local Windows meeting assistant.",
"license": "MIT OR Apache-2.0",
+1 -1
View File
@@ -5945,7 +5945,7 @@ dependencies = [
[[package]]
name = "whispassist"
version = "0.1.1"
version = "0.1.4"
dependencies = [
"argon2",
"async-trait",
+8 -8
View File
@@ -1,6 +1,6 @@
[package]
name = "whispassist"
version = "0.1.2"
version = "0.1.4"
description = "Privacy-first, fully local Windows meeting assistant"
authors = ["WhispAssist contributors"]
license = "MIT OR Apache-2.0"
@@ -36,9 +36,9 @@ reqwest = { version = "0.12", default-features = false, features = ["json", "str
futures-util = "0.3"
sha2 = "0.10" # skip-if-unchanged hashing (sync)
# At-rest encryption vault (Phase 8, T8.8, FR-SEC-3). Pure-Rust RustCrypto no
# At-rest encryption vault (Phase 8, T8.8, FR-SEC-3). Pure-Rust RustCrypto — no
# C build. Argon2id derives the vault key from the password; XChaCha20-Poly1305
# (24-byte nonce safe random nonces) is the AEAD for artifacts + key wrapping.
# (24-byte nonce → safe random nonces) is the AEAD for artifacts + key wrapping.
argon2 = "0.5"
chacha20poly1305 = "0.10"
getrandom = "0.2"
@@ -52,7 +52,7 @@ hound = { version = "3", optional = true } # WAV I/O (Phase 1)
whisper-rs = { version = "0.16", optional = true } # whisper.cpp bindings (Phase 1)
# NPU transcription (Phase 3, T3.4): ONNX Runtime + OpenVINO EP. `load-dynamic`
# dlopens the on-demand-downloaded runtime, so cargo compiles no C++/OpenVINO
# dlopens the on-demand-downloaded runtime, so cargo compiles no C++/OpenVINO —
# keeps the CPU-only build untouched (NFR-MNT-4). `rustfft` powers the log-mel
# front-end (detok is hand-rolled off serde_json, no `tokenizers`/C dep).
ort = { version = "=2.0.0-rc.10", optional = true, default-features = false, features = ["load-dynamic", "openvino"] }
@@ -60,7 +60,7 @@ rustfft = { version = "6", optional = true }
sherpa-rs = { version = "0.6", optional = true, default-features = false, features = ["download-binaries"] } # sherpa-onnx bindings (Phase 4, ADR-0005)
tauri-plugin-dialog = "2" # native Save/choose-folder (Phase 2 export)
# notes export (Phase 8, FR-NOTE-4) pure-Rust, no external binary/cloud
# notes export (Phase 8, FR-NOTE-4) — pure-Rust, no external binary/cloud
# conversion service, consistent with the fully-local invariant.
pulldown-cmark = "0.12"
printpdf = "0.7"
@@ -74,7 +74,7 @@ windows = { version = "0.58", features = [
"Win32_Devices_DeviceAndDriverInstallation", # SetupAPI: NPU detection (Phase 3, T3.4)
"Win32_System_Com",
"Win32_UI_Shell", # SetCurrentProcessExplicitAppUserModelID (Phase 8, T8.6)
"UI_Notifications", # scheduled toast reminders (Phase 8, T8.6, FR-CAL-5) the
"UI_Notifications", # scheduled toast reminders (Phase 8, T8.6, FR-CAL-5) — the
"Data_Xml_Dom", # OS delivers these itself at the due time, no polling timer
"Foundation",
"Foundation_Collections", # IVectorView iteration (GetScheduledToastNotifications)
@@ -96,14 +96,14 @@ vulkan = ["whisper-rs?/vulkan"] # whisper.cpp Vulkan
npu = ["dep:ort", "dep:rustfft"] # ort + OpenVINO EP NPU path (T3.4)
# Phase 4 / 6 (added when integrated)
diarization = ["dep:sherpa-rs"] # sherpa-onnx
pst = [] # shells out to readpst (libpst) no crate dep, see ADR-0008 update
pst = [] # shells out to readpst (libpst) — no crate dep, see ADR-0008 update
# Phase 9
sync = ["dep:keyring"] # remote upload (WebDAV + OAuth providers)
# Phase 10
mcp = ["dep:rmcp", "dep:keyring"] # WhispAssist as an MCP server + hosted-AI creds
[profile.release]
opt-level = "z" # optimize for size keep the binary small (NFR-RES-1)
opt-level = "z" # optimize for size — keep the binary small (NFR-RES-1)
lto = true
codegen-units = 1
strip = true
+2
View File
@@ -51,6 +51,7 @@ fn default_settings() -> Settings {
llm_provider: "ollama".into(),
llm_endpoint: "http://localhost:11434".into(),
llm_model: "llama3".into(),
llm_advanced: serde_json::Value::Null,
preferred_backend: "auto".into(),
whisper_model: crate::paths::DEFAULT_WHISPER_MODEL.to_string(),
low_overhead: false,
@@ -1550,6 +1551,7 @@ fn llm_provider_from_settings(settings: &Settings) -> Option<Box<dyn crate::llm:
"ollama" => Some(Box::new(crate::llm::OllamaProvider {
endpoint: settings.llm_endpoint.clone(),
model: settings.llm_model.clone(),
advanced: settings.llm_advanced.clone(),
})),
"custom" => Some(Box::new(crate::llm::OpenAiCompatProvider {
endpoint: settings.llm_endpoint.clone(),
+43 -5
View File
@@ -57,7 +57,7 @@ const RESPONSE_FORMAT_INSTRUCTIONS: &str = "Respond in Markdown with exactly thr
are bullet lists (each line starting with \"- \"); write \"- None\" if a section has nothing \
to report. Do not add any other top-level sections.";
fn build_messages(prompt: &Prompt) -> Vec<serde_json::Value> {
fn build_messages(prompt: &Prompt, user_system: Option<&str>) -> Vec<serde_json::Value> {
let mut user = String::new();
if !prompt.metadata.is_empty() {
user.push_str(&prompt.metadata);
@@ -69,8 +69,19 @@ fn build_messages(prompt: &Prompt) -> Vec<serde_json::Value> {
}
user.push_str("Transcript:\n");
user.push_str(&prompt.transcript);
// A user's custom system prompt (Ollama advanced config) is *combined with*,
// not a replacement for, WA's output-format contract — otherwise summary /
// action-item parsing would break.
let mut system = String::new();
if let Some(us) = user_system.map(str::trim).filter(|s| !s.is_empty()) {
system.push_str(us);
system.push_str("\n\n");
}
system.push_str(RESPONSE_FORMAT_INSTRUCTIONS);
vec![
serde_json::json!({ "role": "system", "content": RESPONSE_FORMAT_INSTRUCTIONS }),
serde_json::json!({ "role": "system", "content": system }),
serde_json::json!({ "role": "user", "content": user }),
]
}
@@ -218,6 +229,9 @@ async fn stream_lines(
pub struct OllamaProvider {
pub endpoint: String, // e.g. http://localhost:11434
pub model: String,
/// Advanced tuning (`Settings.llm_advanced`): `{ system?, think?, keep_alive?,
/// options: {…} }`. Sparse — only non-default overrides. `Null` = all defaults.
pub advanced: serde_json::Value,
}
#[derive(Deserialize)]
@@ -278,11 +292,33 @@ impl LlmProvider for OllamaProvider {
}
async fn summarize(&self, prompt: Prompt, out: TokenSink) -> Result<Summary, LlmError> {
let body = serde_json::json!({
let user_system = self.advanced.get("system").and_then(|v| v.as_str());
let mut body = serde_json::json!({
"model": self.model,
"messages": build_messages(&prompt),
"messages": build_messages(&prompt, user_system),
"stream": true,
});
// Fold in the user's advanced tuning. `advanced` already holds only
// non-default values, so anything present is an intentional override.
if let Some(adv) = self.advanced.as_object() {
if let Some(think) = adv.get("think").filter(|v| !v.is_null()) {
body["think"] = think.clone();
}
if let Some(keep_alive) = adv
.get("keep_alive")
.and_then(|v| v.as_str())
.filter(|s| !s.trim().is_empty())
{
body["keep_alive"] = serde_json::json!(keep_alive);
}
if let Some(options) = adv
.get("options")
.and_then(|v| v.as_object())
.filter(|o| !o.is_empty())
{
body["options"] = serde_json::Value::Object(options.clone());
}
}
let resp = reqwest::Client::new()
.post(format!("{}/api/chat", self.base()))
.json(&body)
@@ -409,7 +445,9 @@ impl LlmProvider for OpenAiCompatProvider {
async fn summarize(&self, prompt: Prompt, out: TokenSink) -> Result<Summary, LlmError> {
let body = serde_json::json!({
"model": self.model,
"messages": build_messages(&prompt),
// Advanced tuning is Ollama-only; the OpenAI-compatible path uses no
// custom system prompt (its own server/model config governs that).
"messages": build_messages(&prompt, None),
"stream": true,
});
let req = self.auth(
+4
View File
@@ -192,6 +192,10 @@ pub struct Settings {
pub llm_provider: String, // ollama|custom|off
pub llm_endpoint: String,
pub llm_model: String,
/// Advanced Ollama tuning (T5.2): `{ system?, think?, keep_alive?, options: {…} }`.
/// Sparse — only user-overridden values are stored; `Null`/absent = all defaults.
#[serde(default)]
pub llm_advanced: serde_json::Value,
pub preferred_backend: String, // auto|npu|nvidia|amd|intel|cpu
pub whisper_model: String, // ModelInfo.id, e.g. "base.en-q5_1"
pub low_overhead: bool,
+33
View File
@@ -296,4 +296,37 @@ mod tests {
fn audio_ctx_is_capped_at_whispers_own_maximum() {
assert_eq!(audio_ctx_for_window(60 * 16_000), 1500); // 60s window
}
/// GPU spike (opt-in): times whisper.cpp on a chosen backend against a real
/// model + wav. With `--features vulkan` and `WA_BACKEND=intel` (or nvidia/amd)
/// whisper.cpp offloads to the GPU; `WA_BACKEND=cpu` is the baseline. Run:
/// WA_WHISPER_MODEL=…ggml-base.en-q5_1.bin WA_TEST_WAV=…tts_test.wav \
/// cargo test --features vulkan gpu_transcribes -- --ignored --nocapture
#[test]
#[ignore = "requires a whisper model + wav; GPU/CPU timing spike"]
fn gpu_transcribes_and_times() {
let model = std::env::var("WA_WHISPER_MODEL").expect("set WA_WHISPER_MODEL");
let wav = std::env::var("WA_TEST_WAV").expect("set WA_TEST_WAV");
let backend = match std::env::var("WA_BACKEND").as_deref() {
Ok("cpu") => BackendId::Cpu,
Ok("nvidia") => BackendId::Nvidia,
Ok("amd") => BackendId::Amd,
_ => BackendId::Intel, // use_gpu = true for any non-CPU backend
};
let t0 = std::time::Instant::now();
let transcriber = WhisperTranscriber::load(Path::new(&model), backend).expect("load");
let load_ms = t0.elapsed().as_millis();
let t1 = std::time::Instant::now();
let segments = transcriber
.transcribe_file(Path::new(&wav))
.expect("transcribe");
let infer_ms = t1.elapsed().as_millis();
let text = segments
.iter()
.map(|s| s.text.as_str())
.collect::<Vec<_>>()
.join(" ");
eprintln!("[spike] backend={backend:?} load={load_ms}ms infer={infer_ms}ms text={text:?}");
assert!(!text.trim().is_empty(), "transcript was empty");
}
}
+5 -1
View File
@@ -433,16 +433,20 @@ mod tests {
return;
};
let wav = std::env::var("WA_NPU_TEST_WAV").expect("set WA_NPU_TEST_WAV");
let t0 = std::time::Instant::now();
let t = OnnxNpuTranscriber::load(Path::new(&model_dir), BackendId::Npu)
.expect("load NPU transcriber");
let load_ms = t0.elapsed().as_millis();
let t1 = std::time::Instant::now();
let segs = t.transcribe_file(Path::new(&wav)).expect("transcribe");
let infer_ms = t1.elapsed().as_millis();
let text = segs
.iter()
.map(|s| s.text.as_str())
.collect::<Vec<_>>()
.join(" ")
.to_lowercase();
eprintln!("NPU transcript: {text:?}");
eprintln!("[spike] backend=Npu load={load_ms}ms infer={infer_ms}ms text={text:?}");
assert!(!text.trim().is_empty(), "transcript was empty");
if let Ok(expect) = std::env::var("WA_NPU_EXPECT") {
assert!(
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "WhispAssist",
"version": "0.1.2",
"version": "0.1.4",
"identifier": "bet.dou.whispassist",
"build": {
"frontendDist": "../dist",
+7
View File
@@ -253,6 +253,13 @@ export interface AppSettings {
llm_provider: string;
llm_endpoint: string;
llm_model: string;
/** Advanced Ollama tuning; sparse (only non-default overrides). */
llm_advanced?: {
system?: string;
think?: string;
keep_alive?: string;
options?: Record<string, number | boolean | string[]>;
} | null;
preferred_backend: string;
whisper_model: string;
low_overhead: boolean;
+317
View File
@@ -0,0 +1,317 @@
// Ollama advanced `options` parameters (T5.2) — the single source of truth that
// drives the Settings ▸ AI advanced form: rendering, defaults, per-field reset,
// and only-send-if-changed. System prompt / think / keep_alive are top-level
// request fields handled separately in the UI; everything here goes in `options`.
//
// Defaults + ranges per the Ollama Modelfile/API reference.
export type OllamaGroup = "sampling" | "repetition" | "mirostat" | "context" | "hardware";
export interface OllamaOption {
key: string;
group: OllamaGroup;
label: string;
kind: "number" | "bool" | "enum" | "tags";
/** Undefined = "auto" (no fixed default): the field is empty until set. */
default?: number | boolean;
min?: number;
max?: number;
step?: number;
enumOptions?: { value: number; label: string }[];
help: string;
/** Dimmed unless Mirostat is enabled (mirostat != 0). */
needsMirostat?: boolean;
}
export const OLLAMA_OPTIONS: OllamaOption[] = [
// ── Sampling ──────────────────────────────────────────────────────────────
{
key: "temperature",
group: "sampling",
label: "Temperature",
kind: "number",
default: 0.8,
min: 0,
max: 2,
step: 0.05,
help: "Randomness / creativity. Higher is more creative.",
},
{
key: "top_k",
group: "sampling",
label: "Top K",
kind: "number",
default: 40,
min: 0,
max: 200,
step: 1,
help: "Sample only from the K most likely tokens.",
},
{
key: "top_p",
group: "sampling",
label: "Top P",
kind: "number",
default: 0.9,
min: 0,
max: 1,
step: 0.05,
help: "Nucleus sampling cumulative probability.",
},
{
key: "min_p",
group: "sampling",
label: "Min P",
kind: "number",
default: 0.0,
min: 0,
max: 1,
step: 0.01,
help: "Minimum probability relative to the top token.",
},
{
key: "typical_p",
group: "sampling",
label: "Typical P",
kind: "number",
default: 1.0,
min: 0,
max: 1,
step: 0.05,
help: "Locally-typical sampling.",
},
{
key: "tfs_z",
group: "sampling",
label: "Tail-free (tfs_z)",
kind: "number",
default: 1.0,
min: 0,
max: 2,
step: 0.05,
help: "Tail-free sampling. 1.0 disables it (legacy).",
},
{
key: "seed",
group: "sampling",
label: "Seed",
kind: "number",
default: 0,
min: 0,
step: 1,
help: "Fixed seed → reproducible output. 0 = random.",
},
{
key: "num_predict",
group: "sampling",
label: "Max tokens (num_predict)",
kind: "number",
default: -1,
min: -2,
step: 1,
help: "Max tokens to generate. -1 = unlimited, -2 = fill context.",
},
// ── Repetition ────────────────────────────────────────────────────────────
{
key: "repeat_penalty",
group: "repetition",
label: "Repeat penalty",
kind: "number",
default: 1.1,
min: 0,
max: 2,
step: 0.05,
help: "How strongly to penalize repetition.",
},
{
key: "repeat_last_n",
group: "repetition",
label: "Repeat lookback",
kind: "number",
default: 64,
min: -1,
step: 1,
help: "Tokens to look back for repetition. 0 = off, -1 = num_ctx.",
},
{
key: "presence_penalty",
group: "repetition",
label: "Presence penalty",
kind: "number",
default: 0.0,
min: -2,
max: 2,
step: 0.1,
help: "Penalize tokens that already appeared.",
},
{
key: "frequency_penalty",
group: "repetition",
label: "Frequency penalty",
kind: "number",
default: 0.0,
min: -2,
max: 2,
step: 0.1,
help: "Penalize tokens by how often they appear.",
},
{
key: "penalize_newline",
group: "repetition",
label: "Penalize newlines",
kind: "bool",
default: true,
help: "Apply the repeat penalty to newline tokens.",
},
// ── Mirostat ──────────────────────────────────────────────────────────────
{
key: "mirostat",
group: "mirostat",
label: "Mirostat",
kind: "enum",
default: 0,
enumOptions: [
{ value: 0, label: "Off" },
{ value: 1, label: "Mirostat 1" },
{ value: 2, label: "Mirostat 2" },
],
help: "Adaptive perplexity control.",
},
{
key: "mirostat_tau",
group: "mirostat",
label: "Mirostat τ (tau)",
kind: "number",
default: 5.0,
min: 0,
max: 10,
step: 0.1,
help: "Balance of coherence vs diversity.",
needsMirostat: true,
},
{
key: "mirostat_eta",
group: "mirostat",
label: "Mirostat η (eta)",
kind: "number",
default: 0.1,
min: 0,
max: 1,
step: 0.01,
help: "How fast it adapts to feedback.",
needsMirostat: true,
},
// ── Context & session ─────────────────────────────────────────────────────
{
key: "num_ctx",
group: "context",
label: "Context size (num_ctx)",
kind: "number",
default: 4096,
min: 256,
step: 256,
help: "Context window in tokens.",
},
{
key: "num_keep",
group: "context",
label: "Keep tokens (num_keep)",
kind: "number",
default: 4,
min: 0,
step: 1,
help: "Tokens kept from the start of the prompt when truncating.",
},
{
key: "stop",
group: "context",
label: "Stop sequences",
kind: "tags",
help: "Comma-separated strings that end generation.",
},
// ── Runtime & hardware (rarely needed) ────────────────────────────────────
{
key: "num_batch",
group: "hardware",
label: "Batch size (num_batch)",
kind: "number",
default: 512,
min: 1,
step: 1,
help: "Prompt processing batch size.",
},
{
key: "num_gpu",
group: "hardware",
label: "GPU layers (num_gpu)",
kind: "number",
min: 0,
step: 1,
help: "Layers to offload to the GPU. Blank = auto.",
},
{
key: "main_gpu",
group: "hardware",
label: "Main GPU",
kind: "number",
default: 0,
min: 0,
step: 1,
help: "Which GPU to use for a single-GPU offload.",
},
{
key: "num_thread",
group: "hardware",
label: "CPU threads (num_thread)",
kind: "number",
min: 1,
step: 1,
help: "Threads for computation. Blank = auto.",
},
{
key: "low_vram",
group: "hardware",
label: "Low VRAM",
kind: "bool",
default: false,
help: "Reduce VRAM use at the cost of speed.",
},
{
key: "numa",
group: "hardware",
label: "NUMA",
kind: "bool",
default: false,
help: "Enable NUMA support.",
},
{
key: "use_mmap",
group: "hardware",
label: "Use mmap",
kind: "bool",
default: true,
help: "Memory-map the model file.",
},
{
key: "use_mlock",
group: "hardware",
label: "Use mlock",
kind: "bool",
default: false,
help: "Lock the model in RAM (no swap).",
},
];
export const THINK_OPTIONS = ["off", "low", "medium", "high", "max"] as const;
export const OLLAMA_GROUP_LABELS: Record<OllamaGroup, string> = {
sampling: "Sampling",
repetition: "Repetition",
mirostat: "Mirostat",
context: "Context & session",
hardware: "Runtime & hardware",
};
+463 -18
View File
@@ -21,7 +21,17 @@
UploadCloud,
ShieldCheck,
Sparkles,
RefreshCw,
ChevronRight,
RotateCcw,
} from "@lucide/svelte";
import {
OLLAMA_OPTIONS,
THINK_OPTIONS,
OLLAMA_GROUP_LABELS,
type OllamaOption,
type OllamaGroup,
} from "../llmParams";
let { onClose }: { onClose: () => void } = $props();
@@ -68,6 +78,93 @@
}
}
// ---- Advanced Ollama configuration (sparse; only overrides are stored) ----
const OPTION_GROUPS: OllamaGroup[] = ["sampling", "repetition", "mirostat", "context"];
type OptVal = number | boolean | string[];
let advSystem = $state("");
let advThink = $state("off");
let advKeepAlive = $state("");
let advOptions = $state<Record<string, OptVal>>({});
let advSaving = $state(false);
let advSaved = $state(false);
$effect(() => {
// (Re)load the form whenever persisted settings change.
const a = settings.settings.llm_advanced ?? {};
advSystem = a.system ?? "";
advThink = a.think ?? "off";
advKeepAlive = a.keep_alive ?? "";
advOptions = { ...(a.options ?? {}) };
});
let advChangedCount = $derived(
Object.keys(advOptions).length +
(advSystem.trim() ? 1 : 0) +
(advThink !== "off" ? 1 : 0) +
(advKeepAlive.trim() ? 1 : 0),
);
function mirostatOff(): boolean {
const m = advOptions["mirostat"];
return (m === undefined ? 0 : (m as number)) === 0;
}
function optDisplay(opt: OllamaOption): string {
const v = advOptions[opt.key];
if (opt.kind === "tags") return Array.isArray(v) ? v.join(", ") : "";
if (v !== undefined) return String(v);
return opt.default !== undefined ? String(opt.default) : "";
}
function isOverridden(opt: OllamaOption): boolean {
return opt.key in advOptions;
}
function commit(key: string, value: OptVal | undefined) {
const next = { ...advOptions };
if (value === undefined) delete next[key];
else next[key] = value;
advOptions = next;
advSaved = false;
}
function setNumber(opt: OllamaOption, raw: string) {
const t = raw.trim();
if (t === "") return commit(opt.key, undefined);
const n = Number(t);
if (Number.isNaN(n)) return;
commit(opt.key, opt.default !== undefined && n === opt.default ? undefined : n);
}
function setEnum(opt: OllamaOption, raw: string) {
const n = Number(raw);
commit(opt.key, opt.default !== undefined && n === opt.default ? undefined : n);
}
function setBool(opt: OllamaOption, checked: boolean) {
commit(opt.key, opt.default !== undefined && checked === opt.default ? undefined : checked);
}
function setTags(opt: OllamaOption, raw: string) {
const arr = raw
.split(",")
.map((s) => s.trim())
.filter(Boolean);
commit(opt.key, arr.length ? arr : undefined);
}
function resetAllAdvanced() {
advSystem = "";
advThink = "off";
advKeepAlive = "";
advOptions = {};
advSaved = false;
}
async function saveAdvanced() {
advSaving = true;
advSaved = false;
const adv: NonNullable<(typeof settings.settings)["llm_advanced"]> = {};
if (advSystem.trim()) adv.system = advSystem.trim();
if (advThink !== "off") adv.think = advThink;
if (advKeepAlive.trim()) adv.keep_alive = advKeepAlive.trim();
if (Object.keys(advOptions).length) adv.options = advOptions;
try {
await settings.patch({ llm_advanced: Object.keys(adv).length ? adv : null });
advSaved = true;
} finally {
advSaving = false;
}
}
// ---- Calendar / .pst import (T6.1/T6.2/T6.3, FR-CAL-1/2) ----
let pstPath = $state("");
let pstPassword = $state("");
@@ -698,25 +795,172 @@
<button class="primary" onclick={saveLlm} disabled={settings.llmSaving}>
{settings.llmSaving ? "Saving…" : "Save"}
</button>
<button onclick={() => settings.loadLlmStatus()}>Test / refresh</button>
{#if settings.llmStatus}
<span
class="test"
class:ok={settings.llmStatus.reachable}
class:fail={!settings.llmStatus.reachable}
>
{#if settings.llmStatus.reachable}<Check size={14} aria-hidden="true" />{:else}<X
size={14}
aria-hidden="true"
/>{/if}
{settings.llmStatus.reachable ? "reachable" : "unreachable"}
{#if settings.llmStatus.reachable && !settings.llmStatus.isLocal}· leaves your
network{/if}
</span>
{/if}
<button class="ghost" onclick={() => settings.loadLlmStatus()}>
<RefreshCw size={14} aria-hidden="true" /> Test connection
</button>
</div>
{#if settings.llmStatus?.reachable && settings.llmStatus.models.length}
<p class="muted">Models: {settings.llmStatus.models.slice(0, 6).join(", ")}</p>
{#if settings.llmStatus}
{@const s = settings.llmStatus}
<div class="status-card">
<span class="status-dot" class:ok={s.reachable} class:bad={!s.reachable}></span>
<span class="status-text">{s.reachable ? "Reachable" : "Unreachable"}</span>
{#if s.reachable}
<span class="muted"
>· {s.models.length} model{s.models.length === 1 ? "" : "s"}</span
>
<span class="muted" class:warn={!s.isLocal}
>· {s.isLocal ? "on your machine / LAN" : "leaves your network"}</span
>
{/if}
</div>
{#if s.reachable && s.models.length}
<p class="muted small">
{s.models.slice(0, 6).join(", ")}{s.models.length > 6 ? " …" : ""}
</p>
{/if}
{/if}
{#if llmProvider === "ollama"}
{#snippet paramRow(opt: OllamaOption)}
{@const disabled = opt.needsMirostat && mirostatOff()}
<div class="param" class:param-disabled={disabled}>
<div class="param-label">
<span class="param-name">{opt.label}</span>
<span class="param-help"
>{opt.help}{#if opt.default !== undefined}
Default {opt.default}.{/if}</span
>
</div>
<div class="param-control">
{#if opt.kind === "bool"}
<input
type="checkbox"
checked={(advOptions[opt.key] ?? opt.default) === true}
{disabled}
onchange={(e) => setBool(opt, e.currentTarget.checked)}
/>
{:else if opt.kind === "enum"}
<select
value={optDisplay(opt)}
onchange={(e) => setEnum(opt, e.currentTarget.value)}
>
{#each opt.enumOptions ?? [] as o (o.value)}
<option value={String(o.value)}>{o.label}</option>
{/each}
</select>
{:else if opt.kind === "tags"}
<input
value={optDisplay(opt)}
placeholder="e.g. User:, \n"
onchange={(e) => setTags(opt, e.currentTarget.value)}
/>
{:else}
<input
type="number"
value={optDisplay(opt)}
min={opt.min}
max={opt.max}
step={opt.step}
{disabled}
onchange={(e) => setNumber(opt, e.currentTarget.value)}
/>
{/if}
<button
class="reset-field"
class:hidden={!isOverridden(opt)}
title="Reset to default"
aria-label="Reset {opt.label}"
onclick={() => commit(opt.key, undefined)}
>
<RotateCcw size={13} aria-hidden="true" />
</button>
</div>
</div>
{/snippet}
<details class="advanced">
<summary>
<ChevronRight size={15} aria-hidden="true" class="chevron" />
<span>Advanced Ollama configuration</span>
{#if advChangedCount}<span class="changed-badge">{advChangedCount} changed</span
>{/if}
</summary>
<div class="adv-body">
{#if advChangedCount}
<button class="link reset-all" onclick={resetAllAdvanced}
>Reset all to defaults</button
>
{/if}
<label class="wide adv-system"
>System prompt
<textarea
rows="3"
bind:value={advSystem}
placeholder="e.g. You are a concise meeting summarizer."
></textarea>
</label>
<p class="muted small">
Added before WhispAssist's required output format, so your instructions can't
break summary/action-item parsing.
</p>
{#each OPTION_GROUPS as g (g)}
<fieldset class="adv-group">
<legend>{OLLAMA_GROUP_LABELS[g]}</legend>
{#if g === "context"}
<div class="param">
<div class="param-label">
<span class="param-name">Think</span>
<span class="param-help">Reasoning effort (reasoning models only).</span>
</div>
<div class="param-control">
<select bind:value={advThink}>
{#each THINK_OPTIONS as t (t)}<option value={t}>{t}</option>{/each}
</select>
</div>
</div>
<div class="param">
<div class="param-label">
<span class="param-name">Keep alive</span>
<span class="param-help"
>How long the model stays in RAM. e.g. 5m, 1h, 0 (unload), -1 (forever).</span
>
</div>
<div class="param-control">
<input bind:value={advKeepAlive} placeholder="5m" />
</div>
</div>
{/if}
{#each OLLAMA_OPTIONS.filter((o) => o.group === g) as opt (opt.key)}
{@render paramRow(opt)}
{/each}
</fieldset>
{/each}
<details class="adv-nested">
<summary>
<ChevronRight size={14} aria-hidden="true" class="chevron" />
<span>Runtime &amp; hardware</span>
<span class="muted small">— rarely needed</span>
</summary>
<fieldset class="adv-group">
{#each OLLAMA_OPTIONS.filter((o) => o.group === "hardware") as opt (opt.key)}
{@render paramRow(opt)}
{/each}
</fieldset>
</details>
<div class="actions">
<button class="primary" onclick={saveAdvanced} disabled={advSaving}>
{advSaving ? "Saving…" : "Save advanced"}
</button>
{#if advSaved}<span class="test ok"
><Check size={14} aria-hidden="true" /> Saved</span
>{/if}
</div>
</div>
</details>
{/if}
{:else}
<div class="actions">
@@ -1153,4 +1397,205 @@
ul.hosts li {
padding: 0.2rem 0;
}
/* ---- AI provider: status + advanced config ---- */
.ghost {
display: inline-flex;
align-items: center;
gap: 0.35rem;
background: transparent;
border: 1px solid var(--border);
color: var(--fg, inherit);
}
.ghost:hover {
background: var(--hover, rgba(127, 127, 127, 0.08));
}
.status-card {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 0.4rem;
margin-top: 0.6rem;
padding: 0.5rem 0.7rem;
border: 1px solid var(--border);
border-radius: 8px;
font-size: 0.85rem;
}
.status-dot {
width: 9px;
height: 9px;
border-radius: 50%;
background: var(--muted);
flex: none;
}
.status-dot.ok {
background: var(--success, #16a34a);
}
.status-dot.bad {
background: var(--danger, #dc2626);
}
.status-text {
font-weight: 600;
}
.muted.warn {
color: var(--danger, #dc2626);
}
.advanced,
.adv-nested {
margin-top: 0.9rem;
border: 1px solid var(--border);
border-radius: 8px;
}
.adv-nested {
margin-top: 0.6rem;
}
.advanced > summary,
.adv-nested > summary {
display: flex;
align-items: center;
gap: 0.45rem;
padding: 0.6rem 0.75rem;
cursor: pointer;
list-style: none;
font-weight: 600;
user-select: none;
}
.advanced > summary::-webkit-details-marker,
.adv-nested > summary::-webkit-details-marker {
display: none;
}
.advanced :global(.chevron),
.adv-nested :global(.chevron) {
transition: transform 0.18s ease;
flex: none;
color: var(--muted);
}
.advanced[open] > summary :global(.chevron),
.adv-nested[open] > summary :global(.chevron) {
transform: rotate(90deg);
}
@media (prefers-reduced-motion: reduce) {
.advanced :global(.chevron),
.adv-nested :global(.chevron) {
transition: none;
}
}
.changed-badge {
margin-left: auto;
font-size: 0.7rem;
font-weight: 500;
padding: 0.1rem 0.45rem;
border-radius: 10px;
background: var(--accent, #2563eb);
color: var(--accent-fg, #fff);
}
.adv-body {
padding: 0 0.75rem 0.85rem;
display: flex;
flex-direction: column;
gap: 0.5rem;
}
.reset-all {
align-self: flex-end;
font-size: 0.78rem;
}
.adv-system textarea {
width: 100%;
resize: vertical;
font-family: inherit;
font-size: 0.85rem;
padding: 0.4rem 0.5rem;
border: 1px solid var(--border-strong, var(--border));
border-radius: 6px;
}
.adv-group {
border: 1px solid var(--border);
border-radius: 8px;
padding: 0.4rem 0.75rem 0.6rem;
margin: 0;
display: flex;
flex-direction: column;
}
.adv-group legend {
font-size: 0.72rem;
font-weight: 700;
letter-spacing: 0.04em;
text-transform: uppercase;
color: var(--muted);
padding: 0 0.35rem;
}
.param {
display: flex;
align-items: center;
gap: 1rem;
padding: 0.4rem 0;
border-bottom: 1px solid var(--border);
}
.param:last-child {
border-bottom: none;
}
.param-label {
display: flex;
flex-direction: column;
gap: 0.1rem;
min-width: 0;
flex: 1;
}
.param-name {
font-size: 0.85rem;
font-weight: 500;
}
.param-help {
font-size: 0.74rem;
color: var(--muted);
line-height: 1.35;
}
.param-control {
display: flex;
align-items: center;
gap: 0.3rem;
flex: none;
}
.param-control input[type="number"],
.param-control select {
width: 6.5rem;
text-align: right;
font-variant-numeric: tabular-nums;
padding: 0.3rem 0.4rem;
border: 1px solid var(--border-strong, var(--border));
border-radius: 6px;
}
.param-control select {
text-align: left;
}
.param-control input:not([type="number"]):not([type="checkbox"]) {
width: 9rem;
padding: 0.3rem 0.4rem;
border: 1px solid var(--border-strong, var(--border));
border-radius: 6px;
}
.param-disabled {
opacity: 0.45;
}
.reset-field {
display: inline-flex;
align-items: center;
justify-content: center;
width: 1.6rem;
height: 1.6rem;
padding: 0;
border: none;
background: transparent;
color: var(--muted);
cursor: pointer;
border-radius: 5px;
}
.reset-field:hover {
background: var(--hover, rgba(127, 127, 127, 0.1));
color: var(--fg, inherit);
}
.reset-field.hidden {
visibility: hidden;
}
</style>