spng/selfphoneng
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 12 Wiki Activity
Files
7722f336392d3c94f58bf83faa086ac2c6222ff3
selfphoneng/docs/spec/00-overview.md
T
iamdoubz ba293f8703 spec: single sign-on (12), OpenID Connect and LDAP 
Owner decisions 2026-10-25: a user connects their OpenID account and signs in with either it or
their password; the admin picks passwords only, OpenID only (admins keep a password way in), or
mixed. No automatic accounts. OIDC sessions re-checked every 15 min with the refresh token;
signing out also signs out at the provider; password sign-ins keep two-step. LDAP is one more way
to check a password. Decision 0004: openid-client and ldapts. Tasks S1-S5.
2026-10-25 13:09:45 -05:00

396 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 00 — Overview
## 1. Goal
Build **SelfPhoneNG**: a fully open source, self-hosted
business phone system with the everyday features of 3CX:
- Extensions (users) with desk phones, softphones and a browser softphone
- A SIP trunk the customer brings (reference: voip.ms) carrying one or more DIDs (phone numbers)
- Inbound routing: DID → user / ring group / queue / IVR / voicemail, with business hours and holidays
- Outbound routing with permissions and toll-fraud protection
- Voicemail with email delivery and web playback
- IVR ("digital receptionist"), ring groups, call queues
- Call recording, call history (CDR), live call dashboard
- Transfer, hold, park, pickup, conference rooms, music on hold, paging/intercom
- Presence / BLF (busy lamp field)
- Desk phone auto-provisioning
- SMS through the trunk provider's API
- E911 handling that follows US multi-line telephone system law
- A REST API and webhooks
- Backups and one-command install with Docker Compose
Target user: one small business or home lab, 1–100 extensions, one server.
## 2. Non-goals for version 1
Do NOT build these in v1. Do not leave half-built hooks for them either.
- Multi-tenancy (several companies on one install)
- Native mobile apps (we document open source SIP apps instead, see `07-provisioning.md`)
- Video calls, screen sharing, team chat
- Fax
- CRM integrations (the REST API + webhooks are the integration point)
- Being a SIP carrier or selling numbers. The customer always brings their own trunk.
- High availability / clustering
- Our own SIP or RTP stack
## 3. Architecture
```
Internet
│
┌───────────────────┼─────────────────────────────────────────┐
│ Host (Ubuntu/Debian, Docker) │
│ │ │
│ ┌───────────────┴──────────┐ 443/tcp (HTTPS + WSS) │
│ │ Reverse proxy │◄──────── browsers, phones │
│ │ (bundled Caddy OR the │ (provisioning) │
│ │ user's own nginx) │ │
│ └───┬──────────────┬───────┘ │
│ │ /api /prov / │ /ws (SIP over WebSocket) │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────────────────────┐ 5060/udp+tcp SIP│
│ │ api │AMI│ asterisk │◄─── voip.ms │
│ │ (Node) ├──►│ (PJSIP, dialplan, │◄─── desk phones │
│ │ serves │ │ voicemail, queues, ...) │ 10000-20000/udp │
│ │ web UI │ └──────────┬───────────────┘ RTP audio │
│ └──┬───┬───┘ │ │
│ │ │ writes generated │ reads generated config, │
│ │ │ config files │ writes voicemail/recordings │
│ │ ▼ ▼ │
│ │ ┌────────────────────────────────┐ │
│ │ │ shared volumes (./data/...) │ │
│ │ └────────────────────────────────┘ │
│ ▼ │
│ ┌──────────┐ │
│ │ postgres │ │
│ └──────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### 3.1 Responsibilities
**Asterisk** (container `asterisk`): SIP registration of phones, SIP trunk
registration to voip.ms, call setup, audio, codecs, voicemail recording and
MWI, queues, conferences, music on hold, call recording to files, presence hints.
Asterisk holds **no business data of its own**. It reads config files that the
api generates. The only runtime state it keeps is in AstDB (§3.3).
**api** (container `api`, Node.js/TypeScript):
- Owns PostgreSQL. **PostgreSQL is the single source of truth.**
- Turns DB rows into Asterisk config files (the "generator", `03-config-generator.md`)
and tells Asterisk to reload.
- Connects to the Asterisk Manager Interface (AMI) to receive events (calls,
device states, registrations, CDRs) and send actions (originate, hangup, reload).
- Keeps an in-memory "live state" (active calls, extension states, trunk
status) and pushes it to browsers over a WebSocket.
- Serves the REST API, the provisioning endpoints, and the built web UI (static files).
- Runs background jobs: voicemail email, retention cleanup, backups, nightly regeneration.
**web** (package `apps/web`, React): the admin panel, the user portal and the
browser softphone. It is built to static files and served by the api. It is
not a separate container.
**postgres**: application data.
**Reverse proxy**: TLS termination for HTTPS and WSS. Either the bundled Caddy
(automatic Let's Encrypt) or the user's existing nginx (sample config provided).
### 3.2 The config loop (memorize this)
```
Admin changes something in the UI
→ API validates (zod) and writes to PostgreSQL
→ API calls generator.apply() (debounced 1 s, one at a time)
→ generator reads the whole DB, renders all Asterisk files as strings
→ writes them atomically to /etc/asterisk/selfphoneng/
→ sends AMI reload commands for the modules whose files changed
→ Asterisk picks up the new config
```
Generation is **pure and complete**: the same DB contents always produce the
same files, byte for byte. We never patch a generated file in place. We
regenerate everything every time. That keeps it simple and testable.
### 3.3 Runtime toggles (AstDB)
Some settings change many times a day and must work even when the user
toggles them from a desk phone with a feature code (for example DND, call
forwarding, queue login). Regenerating config for each toggle would be
wasteful, so these live in **AstDB**, Asterisk's built-in key/value store,
and dialplan reads them with `${DB(...)}`.
Rules:
- PostgreSQL still stores the value (table `user_runtime_state`, see `02-data-model.md`).
- When the API changes a toggle, it writes PostgreSQL first, then AstDB via AMI `DBPut`/`DBDel`.
- When a feature code changes a toggle, the dialplan writes AstDB and fires
`UserEvent(SpToggle,...)`. The API hears the event and writes PostgreSQL.
- On API startup (and after every AMI reconnect) the API pushes every toggle
from PostgreSQL into AstDB (PostgreSQL wins).
AstDB keys (family/key). This list is complete; do not add keys without updating it.
| Key | Value | Meaning |
|---|---|---|
| `SP/DND/<ext>` | `1` or absent | Do Not Disturb |
| `SP/CFU/<ext>` | dial string digits or absent | Call forward unconditional (always) |
| `SP/CFB/<ext>` | digits or absent | Call forward on busy |
| `SP/CFNA/<ext>` | digits or absent | Call forward on no answer (instead of voicemail) |
| `SP/TCOVR/<timeConditionId>` | `open`, `closed`, or absent (absent = automatic) | Business hours manual override |
| `SP/BLOCK/<callerNumber>` | `1` | Inbound caller blocked |
| `SP/QPAUSE/<queueNum>/<ext>` | `1` or absent | Agent logged out (paused) in queue |
## 4. Tech stack (fixed — do not change without a decision record)
| Area | Choice | Version |
|---|---|---|
| Telephony engine | Asterisk, built from source, PJSIP channel driver only (no chan_sip) | 24 LTS (latest 24.x; see 01 §2 while only release candidates exist) |
| Language | TypeScript (strict mode) | 7.x (see decisions/0001) |
| Runtime | Node.js | 24 LTS |
| Package manager | pnpm workspaces | 9.x or newer |
| API framework | Fastify | 5.x |
| Validation | zod | 4.x |
| ORM / migrations | drizzle-orm + drizzle-kit | latest |
| Database | PostgreSQL | 18 |
| Password hashing | `@node-rs/argon2` (argon2id) | latest |
| Email | nodemailer | latest |
| Job scheduling | croner | latest |
| File watching | chokidar | latest |
| WebSocket (api) | @fastify/websocket | latest |
| Multipart uploads | @fastify/multipart | latest |
| Rate limiting | @fastify/rate-limit | latest |
| Cookies | @fastify/cookie | latest (decisions/0002) |
| OpenID Connect client | openid-client | 6.x (decisions/0004) |
| LDAP client | ldapts | latest (decisions/0004) |
| PostgreSQL driver | pg | latest (decisions/0002) |
| QR codes | qrcode | latest |
| Logging | pino | latest |
| Tests | vitest | latest |
| Frontend | React 19 + Vite + TypeScript | latest |
| Routing (web) | react-router | 8.x |
| Data fetching (web) | @tanstack/react-query | 5.x |
| Styling | Tailwind CSS | 4.x |
| Forms (web) | react-hook-form + @hookform/resolvers | latest |
| Client state (web) | zustand | 5.x |
| Browser SIP | SIP.js | 0.21.x |
| Audio conversion | ffmpeg (in the api image) | distro version |
| Text-to-speech (optional, M12) | Piper | latest |
| Bundled reverse proxy | Caddy | 2.x |
| SIP test tools (dev only) | SIPp, baresip | distro versions |
The AMI client is written by us (`04-backend-api.md` §4). It's about 200 lines,
and existing npm AMI packages are unmaintained.
License for our code: **AGPL-3.0-or-later**. That fits a self-hosted network
service, and Asterisk is GPL-2.0. Put the license in `LICENSE` and a
`SPDX-License-Identifier: AGPL-3.0-or-later` header in every source file.
## 5. Repository layout
```
selfphoneng/ (repo root; the folder on disk may be named differently)
├── AGENTS.md
├── CLAUDE.md (owner's file — do not overwrite)
├── LICENSE
├── README.md
├── package.json (pnpm workspace root, scripts)
├── pnpm-workspace.yaml
├── docker-compose.yml
├── .env.example
├── docs/
│ ├── spec/ (this spec)
│ ├── decisions/ (short decision records, NNNN-title.md)
│ └── admin-guide/ (end-user docs, written in M11)
├── asterisk/
│ ├── Dockerfile
│ ├── entrypoint.sh
│ └── conf/ (STATIC Asterisk config, copied into the image)
│ ├── asterisk.conf
│ ├── modules.conf
│ ├── logger.conf
│ ├── manager.conf.template
│ ├── http.conf
│ ├── rtp.conf.template
│ ├── pjsip.conf (just #includes the generated file)
│ ├── extensions.conf (static helper contexts + #include generated)
│ ├── voicemail.conf (#include generated)
│ ├── queues.conf (#include generated)
│ ├── musiconhold.conf (#include generated)
│ ├── confbridge.conf
│ ├── features.conf
│ ├── res_parking.conf
│ └── cdr_manager.conf
├── apps/
│ ├── api/
│ │ ├── Dockerfile
│ │ ├── drizzle.config.ts
│ │ ├── src/
│ │ │ ├── index.ts (starts everything)
│ │ │ ├── config.ts (reads + validates env vars with zod)
│ │ │ ├── db/
│ │ │ │ ├── schema.ts (drizzle tables, matches 02-data-model.md)
│ │ │ │ ├── client.ts
│ │ │ │ └── migrations/ (generated by drizzle-kit)
│ │ │ ├── crypto/secrets.ts (AES-256-GCM encrypt/decrypt)
│ │ │ ├── auth/ (sessions, password hashing, API tokens, guards)
│ │ │ ├── ami/
│ │ │ │ ├── client.ts (TCP AMI client)
│ │ │ │ └── events.ts (typed event parsing)
│ │ │ ├── live/ (in-memory live state + WebSocket hub)
│ │ │ ├── generator/
│ │ │ │ ├── index.ts (render() and apply())
│ │ │ │ ├── sanitize.ts
│ │ │ │ ├── load.ts (loads a full snapshot from DB)
│ │ │ │ ├── destinations.ts
│ │ │ │ ├── pjsip.ts
│ │ │ │ ├── dialplan/ (one file per feature)
│ │ │ │ ├── voicemail.ts
│ │ │ │ ├── queues.ts
│ │ │ │ ├── moh.ts
│ │ │ │ └── __golden__/ (golden test fixtures)
│ │ │ ├── routes/ (Fastify route modules, one per resource)
│ │ │ ├── services/ (business logic used by routes + jobs)
│ │ │ ├── jobs/ (scheduled background jobs)
│ │ │ ├── provisioning/
│ │ │ │ └── templates/ (phone config templates)
│ │ │ ├── integrations/voipms/ (voip.ms REST API client, SMS)
│ │ │ └── media/ (ffmpeg audio conversion)
│ │ └── test/
│ └── web/
│ ├── index.html
│ ├── vite.config.ts
│ └── src/
│ ├── main.tsx
│ ├── api/ (typed fetch client)
│ ├── components/
│ ├── pages/admin/
│ ├── pages/user/
│ ├── softphone/ (SIP.js wrapper + UI)
│ └── live/ (WebSocket client)
├── packages/
│ └── shared/ (zod schemas + TS types shared by api and web)
├── deploy/
│ ├── caddy/Caddyfile
│ └── nginx/selfphoneng.conf.example
├── scripts/ (install.sh, backup.sh, restore.sh)
└── data/ (gitignored; runtime volumes)
├── postgres/
├── asterisk-generated/ (→ /etc/asterisk/selfphoneng)
├── voicemail/ (→ /var/spool/asterisk/voicemail)
├── recordings/ (→ /var/spool/asterisk/monitor)
├── sounds-custom/ (→ /var/lib/asterisk/sounds/custom)
├── moh/ (→ /var/lib/asterisk/moh)
├── astdb/ (→ /var/lib/asterisk/astdb.sqlite3 folder)
├── tls/
└── backups/
```
Shared zod schemas live in `packages/shared` so the web UI and API validate the same way.
## 6. Glossary (read if any telephony word is unfamiliar)
| Term | Meaning |
|---|---|
| **PBX** | Private Branch Exchange. The phone system for a business. That is what we are building. |
| **SIP** | Session Initiation Protocol. The text protocol phones and servers use to set up calls (INVITE, REGISTER, BYE...). |
| **RTP** | Real-time Transport Protocol. Carries the actual audio, on UDP ports 10000–20000. |
| **SIP trunk** | A SIP account at a provider (voip.ms) that connects the PBX to the public phone network. |
| **DID** | Direct Inward Dialing number: a real phone number like 312-555-1234 that rings into the PBX. |
| **Extension** | An internal number (e.g. 101) belonging to a user. |
| **Endpoint** | In Asterisk PJSIP: one SIP identity (a phone account or a trunk). |
| **AOR** | Address of Record. Where Asterisk finds an endpoint's registered devices ("contacts"). |
| **Contact** | One registered device. One user can have several (desk phone + softphone). |
| **Registration** | A device telling the server "I am 101, reach me at this IP:port". The PBX also registers *to* voip.ms. |
| **Dialplan** | Asterisk's routing script (`extensions.conf`). Contexts contain extensions, which contain numbered priorities (steps). |
| **Context** | A named section of the dialplan, e.g. `[sp-internal]`. A channel is always "in" a context. |
| **Channel** | One leg of a call inside Asterisk (e.g. `PJSIP/101-00000003`). A normal call has two channels bridged together. |
| **Bridge** | Asterisk object that connects channels so they hear each other. |
| **Local channel** | A fake channel (`Local/101@ctx`) that runs dialplan as if it were a device. Used for ring groups and queues. |
| **AMI** | Asterisk Manager Interface. A TCP text protocol (port 5038) to receive events and send commands. |
| **AstDB** | Asterisk's built-in key/value database. |
| **CDR** | Call Detail Record. One row per call leg: who called whom, when, how long, outcome. |
| **linkedid** | Asterisk ID shared by all channels belonging to one logical call. Used to group CDRs. |
| **MWI** | Message Waiting Indicator. The voicemail light on a desk phone. |
| **BLF** | Busy Lamp Field. Phone buttons that light up when a colleague is on a call. Uses SIP SUBSCRIBE to Asterisk "hints". |
| **Hint** | Dialplan line telling Asterisk which device state an extension number shows (`exten => 101,hint,PJSIP/101`). |
| **IVR** | Interactive Voice Response: "Press 1 for sales...". 3CX calls this a "digital receptionist". |
| **Ring group** | One number that rings several users (all at once or one after another). |
| **Queue** | Callers wait in line with music until an agent is free (call-center style). |
| **Park** | Put a call in a numbered slot so anyone can pick it up by dialing the slot. |
| **Pickup** | Answer a phone that is ringing on someone else's desk. |
| **REFER** | SIP message used for transfers. |
| **Blind / attended transfer** | Blind: send the call on without talking first. Attended: talk to the target first, then join. |
| **DTMF** | The tones of keypad digits. We use RFC 4733 (in-band RTP events). |
| **Codec** | Audio encoding. We use G.722 (HD), µ-law (PCMU, US default) and A-law (PCMA). |
| **WebRTC** | Browser real-time media. The browser softphone uses SIP over WebSocket + WebRTC media (DTLS-SRTP, ICE). |
| **ICE / STUN / TURN** | NAT traversal for WebRTC. STUN discovers the public IP. TURN relays audio when nothing else works. |
| **NAT** | Network Address Translation (home/office routers). The #1 cause of one-way or missing audio. |
| **E911** | Enhanced 911: an emergency call that carries a registered street address. |
| **MLTS** | Multi-Line Telephone System. A PBX, legally. US Kari's Law and RAY BAUM'S Act apply. |
| **Feature code** | Star codes like `*97` (check voicemail). |
| **Provisioning** | A desk phone downloads its config file from the PBX over HTTP(S). |
| **E.164** | International number format (`+13125551234`). Inside the PBX we store North American numbers as 11 digits `13125551234`. |
## 7. Numbering plan (defaults; configurable in Settings)
All internal numbers share **one namespace**. A number may belong to only one
object (user, ring group, queue, IVR, conference, parking). The API must
reject duplicates across all object types (see `04-backend-api.md` §7.2).
As built (owner, 2026-10-21): internal numbers have **1 to 6 digits**, and any free one can be
given to any object (a first user on `1`). The ranges below only choose the **Next free**
suggestion, and are edited on Settings → Numbering. Generated patterns that take an internal
number match one digit or more (`_X!`, `_**X!`, `_*45X!`, `_*72X!`, `_*80X!`; golden case
`short-numbers`, which the Asterisk load test dials). One more clash is refused, both ways: a
one-digit number that a digital receptionist with direct dialing also uses as a menu key
(pressing it always picks the option).
| Range | Object |
|---|---|
| 100–499 | Users (extensions) |
| 500–549 | Ring groups |
| 550–599 | Queues |
| 600–649 | IVRs (dialable internally so admins can test them) |
| 700 | Park (transfer a call here to park it) |
| 701–720 | Parking slots |
| 800–849 | Conference rooms |
| 850–869 | Paging / intercom groups |
| `*` codes | Feature codes (`03-config-generator.md` §9) |
**Reserved and never assignable:** `211, 311, 411, 511, 611, 711, 811, 911, 933`
and anything starting with `0` or `1` followed by more than 2 digits. N11 codes
are real public services, so a user with extension 911 or 411 would break
emergency or directory calls.
## 8. External number format
- Store and compare North American numbers as **11 digits** starting with `1` (`13125551234`).
- Store international numbers as `011` + country code + number.
- The function `normalizeNumber(input, defaultCountry='1')` in `packages/shared/src/phone.ts` is the only place that does this:
- strip spaces, dashes, dots, parentheses
- `+1XXXXXXXXXX` → `1XXXXXXXXXX`
- 10 digits starting with `[2-9]` → prefix `1`
- `+<other>` → `011<other>`
- otherwise return as-is if it contains only digits, `*` or `#`; else throw.
- Display: `1XXXXXXXXXX` → `(XXX) XXX-XXXX`.
## 9. Roles
| Role | Can do |
|---|---|
| `admin` | Everything |
| `user` | Their own extension: voicemail, call history, forwarding, DND, greetings, softphone, contacts, SMS on DIDs assigned to them |
(A third role, `manager`, with read-only reports and queue control, is a later addition. Don't build it in v1.)
## 10. Open questions for the owner
Project name: **SelfPhoneNG** (decided). In code and config it is written `selfphoneng`
(package scope `@selfphoneng/*`, paths, DB name). Short prefix `sp` (for SelfPhoneNG):
`sp-` for dialplan contexts, `SP_` for channel variables, `SP` for the AstDB family, `Sp…` for UserEvent names,
`sp_` for the session cookie, API token prefix and CDR variable mappings.
1. ~~Should v1 ship with the bundled Caddy enabled by default?~~ **Decided (2026-10-17):** Caddy stays an
optional compose profile, and the nginx example is documented.
Reference in new issue View Git Blame Copy Permalink