selfphoneng/docs/spec/00-overview.md

00 — Overview

1. Goal

Build SelfPhoneNG: a fully open source, self-hosted business phone system with the everyday features of 3CX:

• Extensions (users) with desk phones, softphones and a browser softphone
• A SIP trunk the customer brings (reference: voip.ms) carrying one or more DIDs (phone numbers)
• Inbound routing: DID → user / ring group / queue / IVR / voicemail, with business hours and holidays
• Outbound routing with permissions and toll-fraud protection
• Voicemail with email delivery and web playback
• IVR ("digital receptionist"), ring groups, call queues
• Call recording, call history (CDR), live call dashboard
• Transfer, hold, park, pickup, conference rooms, music on hold, paging/intercom
• Presence / BLF (busy lamp field)
• Desk phone auto-provisioning
• SMS through the trunk provider's API
• E911 handling that follows US multi-line telephone system law
• A REST API and webhooks
• Backups and one-command install with Docker Compose

Target user: one small business or home lab, 1–100 extensions, one server.

2. Non-goals for version 1

Do NOT build these in v1. Do not leave half-built hooks for them either.

• Multi-tenancy (several companies on one install)
• Native mobile apps (we document open source SIP apps instead, see 07-provisioning.md)
• Video calls, screen sharing, team chat
• Fax
• CRM integrations (the REST API + webhooks are the integration point)
• Being a SIP carrier or selling numbers. The customer always brings their own trunk.
• High availability / clustering
• Our own SIP or RTP stack

3. Architecture

                    Internet
                       │
   ┌───────────────────┼─────────────────────────────────────────┐
   │ Host (Ubuntu/Debian, Docker)                                │
   │                   │                                         │
   │   ┌───────────────┴──────────┐   443/tcp (HTTPS + WSS)       │
   │   │ Reverse proxy            │◄──────── browsers, phones     │
   │   │ (bundled Caddy OR the    │          (provisioning)       │
   │   │  user's own nginx)       │                               │
   │   └───┬──────────────┬───────┘                               │
   │       │ /api /prov / │ /ws (SIP over WebSocket)              │
   │       ▼              ▼                                       │
   │  ┌──────────┐   ┌──────────────────────────┐ 5060/udp+tcp SIP│
   │  │ api      │AMI│ asterisk                 │◄─── voip.ms     │
   │  │ (Node)   ├──►│ (PJSIP, dialplan,        │◄─── desk phones │
   │  │ serves   │   │  voicemail, queues, ...) │ 10000-20000/udp │
   │  │ web UI   │   └──────────┬───────────────┘ RTP audio       │
   │  └──┬───┬───┘              │                                 │
   │     │   │ writes generated │ reads generated config,         │
   │     │   │ config files     │ writes voicemail/recordings     │
   │     │   ▼                  ▼                                 │
   │     │  ┌────────────────────────────────┐                    │
   │     │  │ shared volumes (./data/...)    │                    │
   │     │  └────────────────────────────────┘                    │
   │     ▼                                                        │
   │  ┌──────────┐                                                │
   │  │ postgres │                                                │
   │  └──────────┘                                                │
   └──────────────────────────────────────────────────────────────┘

3.1 Responsibilities

Asterisk (container asterisk): SIP registration of phones, SIP trunk registration to voip.ms, call setup, audio, codecs, voicemail recording and MWI, queues, conferences, music on hold, call recording to files, presence hints. Asterisk holds no business data of its own. It reads config files that the api generates. The only runtime state it keeps is in AstDB (§3.3).

api (container api, Node.js/TypeScript):

• Owns PostgreSQL. PostgreSQL is the single source of truth.
• Turns DB rows into Asterisk config files (the "generator", 03-config-generator.md) and tells Asterisk to reload.
• Connects to the Asterisk Manager Interface (AMI) to receive events (calls, device states, registrations, CDRs) and send actions (originate, hangup, reload).
• Keeps an in-memory "live state" (active calls, extension states, trunk status) and pushes it to browsers over a WebSocket.
• Serves the REST API, the provisioning endpoints, and the built web UI (static files).
• Runs background jobs: voicemail email, retention cleanup, backups, nightly regeneration.

web (package apps/web, React): the admin panel, the user portal and the browser softphone. It is built to static files and served by the api. It is not a separate container.

postgres: application data.

Reverse proxy: TLS termination for HTTPS and WSS. Either the bundled Caddy (automatic Let's Encrypt) or the user's existing nginx (sample config provided).

3.2 The config loop (memorize this)

Admin changes something in the UI
  → API validates (zod) and writes to PostgreSQL
  → API calls generator.apply() (debounced 1 s, one at a time)
      → generator reads the whole DB, renders all Asterisk files as strings
      → writes them atomically to /etc/asterisk/selfphoneng/
      → sends AMI reload commands for the modules whose files changed
  → Asterisk picks up the new config

Generation is pure and complete: the same DB contents always produce the same files, byte for byte. We never patch a generated file in place. We regenerate everything every time. That keeps it simple and testable.

3.3 Runtime toggles (AstDB)

Some settings change many times a day and must work even when the user toggles them from a desk phone with a feature code (for example DND, call forwarding, queue login). Regenerating config for each toggle would be wasteful, so these live in AstDB, Asterisk's built-in key/value store, and dialplan reads them with ${DB(...)}.

Rules:

• PostgreSQL still stores the value (table user_runtime_state, see 02-data-model.md).
• When the API changes a toggle, it writes PostgreSQL first, then AstDB via AMI DBPut/DBDel.
• When a feature code changes a toggle, the dialplan writes AstDB and fires UserEvent(SpToggle,...). The API hears the event and writes PostgreSQL.
• On API startup (and after every AMI reconnect) the API pushes every toggle from PostgreSQL into AstDB (PostgreSQL wins).

AstDB keys (family/key). This list is complete; do not add keys without updating it.

Key	Value	Meaning
SP/DND/<ext>	1 or absent	Do Not Disturb
SP/CFU/<ext>	dial string digits or absent	Call forward unconditional (always)
SP/CFB/<ext>	digits or absent	Call forward on busy
SP/CFNA/<ext>	digits or absent	Call forward on no answer (instead of voicemail)
SP/TCOVR/<timeConditionId>	open, closed, or absent (absent = automatic)	Business hours manual override
SP/BLOCK/<callerNumber>	1	Inbound caller blocked
SP/QPAUSE/<queueNum>/<ext>	1 or absent	Agent logged out (paused) in queue

4. Tech stack (fixed — do not change without a decision record)

Area	Choice	Version
Telephony engine	Asterisk, built from source, PJSIP channel driver only (no chan_sip)	24 LTS (latest 24.x; see 01 §2 while only release candidates exist)
Language	TypeScript (strict mode)	7.x (see decisions/0001)
Runtime	Node.js	24 LTS
Package manager	pnpm workspaces	9.x or newer
API framework	Fastify	5.x
Validation	zod	4.x
ORM / migrations	drizzle-orm + drizzle-kit	latest
Database	PostgreSQL	18
Password hashing	@node-rs/argon2 (argon2id)	latest
Email	nodemailer	latest
Job scheduling	croner	latest
File watching	chokidar	latest
WebSocket (api)	@fastify/websocket	latest
Multipart uploads	@fastify/multipart	latest
Rate limiting	@fastify/rate-limit	latest
Cookies	@fastify/cookie	latest (decisions/0002)
PostgreSQL driver	pg	latest (decisions/0002)
QR codes	qrcode	latest
Logging	pino	latest
Tests	vitest	latest
Frontend	React 19 + Vite + TypeScript	latest
Routing (web)	react-router	8.x
Data fetching (web)	@tanstack/react-query	5.x
Styling	Tailwind CSS	4.x
Forms (web)	react-hook-form + @hookform/resolvers	latest
Client state (web)	zustand	5.x
Browser SIP	SIP.js	0.21.x
Audio conversion	ffmpeg (in the api image)	distro version
Text-to-speech (optional, M12)	Piper	latest
Bundled reverse proxy	Caddy	2.x
SIP test tools (dev only)	SIPp, baresip	distro versions

The AMI client is written by us (04-backend-api.md §4). It's about 200 lines, and existing npm AMI packages are unmaintained.

License for our code: AGPL-3.0-or-later. That fits a self-hosted network service, and Asterisk is GPL-2.0. Put the license in LICENSE and a SPDX-License-Identifier: AGPL-3.0-or-later header in every source file.

5. Repository layout

selfphoneng/                   (repo root; the folder on disk may be named differently)
├── AGENTS.md
├── CLAUDE.md                   (owner's file — do not overwrite)
├── LICENSE
├── README.md
├── package.json                (pnpm workspace root, scripts)
├── pnpm-workspace.yaml
├── docker-compose.yml
├── .env.example
├── docs/
│   ├── spec/                   (this spec)
│   ├── decisions/              (short decision records, NNNN-title.md)
│   └── admin-guide/            (end-user docs, written in M11)
├── asterisk/
│   ├── Dockerfile
│   ├── entrypoint.sh
│   └── conf/                   (STATIC Asterisk config, copied into the image)
│       ├── asterisk.conf
│       ├── modules.conf
│       ├── logger.conf
│       ├── manager.conf.template
│       ├── http.conf
│       ├── rtp.conf.template
│       ├── pjsip.conf          (just #includes the generated file)
│       ├── extensions.conf     (static helper contexts + #include generated)
│       ├── voicemail.conf      (#include generated)
│       ├── queues.conf         (#include generated)
│       ├── musiconhold.conf    (#include generated)
│       ├── confbridge.conf
│       ├── features.conf
│       ├── res_parking.conf
│       └── cdr_manager.conf
├── apps/
│   ├── api/
│   │   ├── Dockerfile
│   │   ├── drizzle.config.ts
│   │   ├── src/
│   │   │   ├── index.ts                (starts everything)
│   │   │   ├── config.ts               (reads + validates env vars with zod)
│   │   │   ├── db/
│   │   │   │   ├── schema.ts           (drizzle tables, matches 02-data-model.md)
│   │   │   │   ├── client.ts
│   │   │   │   └── migrations/         (generated by drizzle-kit)
│   │   │   ├── crypto/secrets.ts       (AES-256-GCM encrypt/decrypt)
│   │   │   ├── auth/                   (sessions, password hashing, API tokens, guards)
│   │   │   ├── ami/
│   │   │   │   ├── client.ts           (TCP AMI client)
│   │   │   │   └── events.ts           (typed event parsing)
│   │   │   ├── live/                   (in-memory live state + WebSocket hub)
│   │   │   ├── generator/
│   │   │   │   ├── index.ts            (render() and apply())
│   │   │   │   ├── sanitize.ts
│   │   │   │   ├── load.ts             (loads a full snapshot from DB)
│   │   │   │   ├── destinations.ts
│   │   │   │   ├── pjsip.ts
│   │   │   │   ├── dialplan/           (one file per feature)
│   │   │   │   ├── voicemail.ts
│   │   │   │   ├── queues.ts
│   │   │   │   ├── moh.ts
│   │   │   │   └── __golden__/         (golden test fixtures)
│   │   │   ├── routes/                 (Fastify route modules, one per resource)
│   │   │   ├── services/               (business logic used by routes + jobs)
│   │   │   ├── jobs/                   (scheduled background jobs)
│   │   │   ├── provisioning/
│   │   │   │   └── templates/          (phone config templates)
│   │   │   ├── integrations/voipms/    (voip.ms REST API client, SMS)
│   │   │   └── media/                  (ffmpeg audio conversion)
│   │   └── test/
│   └── web/
│       ├── index.html
│       ├── vite.config.ts
│       └── src/
│           ├── main.tsx
│           ├── api/                    (typed fetch client)
│           ├── components/
│           ├── pages/admin/
│           ├── pages/user/
│           ├── softphone/              (SIP.js wrapper + UI)
│           └── live/                   (WebSocket client)
├── packages/
│   └── shared/                         (zod schemas + TS types shared by api and web)
├── deploy/
│   ├── caddy/Caddyfile
│   └── nginx/selfphoneng.conf.example
├── scripts/                            (install.sh, backup.sh, restore.sh)
└── data/                               (gitignored; runtime volumes)
    ├── postgres/
    ├── asterisk-generated/             (→ /etc/asterisk/selfphoneng)
    ├── voicemail/                      (→ /var/spool/asterisk/voicemail)
    ├── recordings/                     (→ /var/spool/asterisk/monitor)
    ├── sounds-custom/                  (→ /var/lib/asterisk/sounds/custom)
    ├── moh/                            (→ /var/lib/asterisk/moh)
    ├── astdb/                          (→ /var/lib/asterisk/astdb.sqlite3 folder)
    ├── tls/
    └── backups/

Shared zod schemas live in packages/shared so the web UI and API validate the same way.

6. Glossary (read if any telephony word is unfamiliar)

Term	Meaning
PBX	Private Branch Exchange. The phone system for a business. That is what we are building.
SIP	Session Initiation Protocol. The text protocol phones and servers use to set up calls (INVITE, REGISTER, BYE...).
RTP	Real-time Transport Protocol. Carries the actual audio, on UDP ports 10000–20000.
SIP trunk	A SIP account at a provider (voip.ms) that connects the PBX to the public phone network.
DID	Direct Inward Dialing number: a real phone number like 312-555-1234 that rings into the PBX.
Extension	An internal number (e.g. 101) belonging to a user.
Endpoint	In Asterisk PJSIP: one SIP identity (a phone account or a trunk).
AOR	Address of Record. Where Asterisk finds an endpoint's registered devices ("contacts").
Contact	One registered device. One user can have several (desk phone + softphone).
Registration	A device telling the server "I am 101, reach me at this IP:port". The PBX also registers to voip.ms.
Dialplan	Asterisk's routing script (extensions.conf). Contexts contain extensions, which contain numbered priorities (steps).
Context	A named section of the dialplan, e.g. [sp-internal]. A channel is always "in" a context.
Channel	One leg of a call inside Asterisk (e.g. PJSIP/101-00000003). A normal call has two channels bridged together.
Bridge	Asterisk object that connects channels so they hear each other.
Local channel	A fake channel (Local/101@ctx) that runs dialplan as if it were a device. Used for ring groups and queues.
AMI	Asterisk Manager Interface. A TCP text protocol (port 5038) to receive events and send commands.
AstDB	Asterisk's built-in key/value database.
CDR	Call Detail Record. One row per call leg: who called whom, when, how long, outcome.
linkedid	Asterisk ID shared by all channels belonging to one logical call. Used to group CDRs.
MWI	Message Waiting Indicator. The voicemail light on a desk phone.
BLF	Busy Lamp Field. Phone buttons that light up when a colleague is on a call. Uses SIP SUBSCRIBE to Asterisk "hints".
Hint	Dialplan line telling Asterisk which device state an extension number shows (exten => 101,hint,PJSIP/101).
IVR	Interactive Voice Response: "Press 1 for sales...". 3CX calls this a "digital receptionist".
Ring group	One number that rings several users (all at once or one after another).
Queue	Callers wait in line with music until an agent is free (call-center style).
Park	Put a call in a numbered slot so anyone can pick it up by dialing the slot.
Pickup	Answer a phone that is ringing on someone else's desk.
REFER	SIP message used for transfers.
Blind / attended transfer	Blind: send the call on without talking first. Attended: talk to the target first, then join.
DTMF	The tones of keypad digits. We use RFC 4733 (in-band RTP events).
Codec	Audio encoding. We use G.722 (HD), µ-law (PCMU, US default) and A-law (PCMA).
WebRTC	Browser real-time media. The browser softphone uses SIP over WebSocket + WebRTC media (DTLS-SRTP, ICE).
ICE / STUN / TURN	NAT traversal for WebRTC. STUN discovers the public IP. TURN relays audio when nothing else works.
NAT	Network Address Translation (home/office routers). The #1 cause of one-way or missing audio.
E911	Enhanced 911: an emergency call that carries a registered street address.
MLTS	Multi-Line Telephone System. A PBX, legally. US Kari's Law and RAY BAUM'S Act apply.
Feature code	Star codes like *97 (check voicemail).
Provisioning	A desk phone downloads its config file from the PBX over HTTP(S).
E.164	International number format (+13125551234). Inside the PBX we store North American numbers as 11 digits 13125551234.

7. Numbering plan (defaults; configurable in Settings)

All internal numbers share one namespace. A number may belong to only one object (user, ring group, queue, IVR, conference, parking). The API must reject duplicates across all object types (see 04-backend-api.md §7.2).

Range	Object
100–499	Users (extensions)
500–549	Ring groups
550–599	Queues
600–649	IVRs (dialable internally so admins can test them)
700	Park (transfer a call here to park it)
701–720	Parking slots
800–849	Conference rooms
850–869	Paging / intercom groups
* codes	Feature codes (03-config-generator.md §9)

Reserved and never assignable: 211, 311, 411, 511, 611, 711, 811, 911, 933 and anything starting with 0 or 1 followed by more than 2 digits. N11 codes are real public services, so a user with extension 911 or 411 would break emergency or directory calls.

8. External number format

• Store and compare North American numbers as 11 digits starting with 1 (13125551234).
• Store international numbers as 011 + country code + number.
• The function normalizeNumber(input, defaultCountry='1') in packages/shared/src/phone.ts is the only place that does this:
  • strip spaces, dashes, dots, parentheses
  • +1XXXXXXXXXX → 1XXXXXXXXXX
  • 10 digits starting with [2-9] → prefix 1
  • +<other> → 011<other>
  • otherwise return as-is if it contains only digits, * or #; else throw.
• Display: 1XXXXXXXXXX → (XXX) XXX-XXXX.

9. Roles

Role	Can do
admin	Everything
user	Their own extension: voicemail, call history, forwarding, DND, greetings, softphone, contacts, SMS on DIDs assigned to them

(A third role, manager, with read-only reports and queue control, is a later addition. Don't build it in v1.)

10. Open questions for the owner

Project name: SelfPhoneNG (decided). In code and config it is written selfphoneng (package scope @selfphoneng/*, paths, DB name). Short prefix sp (for SelfPhoneNG): sp- for dialplan contexts, SP_ for channel variables, SP for the AstDB family, Sp… for UserEvent names, sp_ for the session cookie, API token prefix and CDR variable mappings.

1. Should v1 ship with the bundled Caddy enabled by default? Decided (2026-10-17): Caddy stays an optional compose profile, and the nginx example is documented.