# 14 — Spam check Owner, 2026-10-27: when an outside caller isn't in anyone's contacts, look their number up and show what's known about them. Why these providers and why it may leave the server: [decision 0006](../decisions/0006-spam-check.md). ## 1. What it does - An outside call rings a user, and the number is none of: an extension, a contact of that user, a company phonebook entry (the 11 §1 lookup finds no name). Then the api **checks the number** with a provider and the user sees the answer while it rings. - **Information only** (owner): a check never blocks, reroutes or delays a call. The call rings at once; the answer appears when it arrives. - **US numbers only** (owner: the callers are in the US): `1NXXNXXXXXX` after `normalizeNumber` (00 §8). Anything else, a withheld number, and internal calls are never checked. - Shown in the **browser softphone** and **call history**; on **desk phones** only when the answer is in before their caller-name lookup gives up (§5.2). - **Off by default** (owner). The admin turns it on in Settings → Spam check (§8). - **Never in the way:** a provider that fails or is slow only means no answer is shown. Each failure is one `warn` log line with the provider and the reason (HTTP status, timeout, the provider's message), never the key or the request URL (IPQualityScore puts its key in the path). ## 2. Providers The admin sets up either provider or both. The order is fixed (owner, 2026-10-27): **IPQualityScore first** when it's set up; the **FTC list** answers when IPQualityScore isn't set up or can't answer. v1 has these two: | | IPQualityScore | FTC Do Not Call complaints | |---|---|---| | What | Fraud score and flags for any number, live | Unwanted-call complaints people filed with the FTC | | Where it runs | An HTTPS request per check | A local copy, synced by a job; a check is a database query | | Sends out | The caller's number | Nothing (it only downloads public data) | | Key | IPQualityScore API key | api.data.gov key (free) | | Cost | Free plan 1,000 lookups a month; paid from about $99 a month | Free | "Can't answer" means: not set up, paused (§3), or this check failed. The FTC list is then asked for that check only; nothing is remembered about the failure. Neither set up → no checks. ### 2.1 IPQualityScore (`spam/ipqs.ts`) - `GET https://www.ipqualityscore.com/api/json/phone/{key}/{number}?country[]=US`, the number as 11 digits, 4 s timeout. - `success: false` whose message says the credits are used up ("You have insufficient credits to make this query…") → **exhausted**, §3. Any other `success: false`, an HTTP error or a timeout → a failure (logged, not cached, the FTC list asked). - Risk (owner, 2026-10-27): **high** if `spammer` or `fraud_score >= 85`; **medium** if `fraud_score >= 75`, `recent_abuse` or `risky`; else **low**. These follow IPQualityScore's general advice (75 suspicious, 85 high risk). - Kept (and nothing else from the answer): `fraud_score`, `spammer`, `recent_abuse`, `line_type`, `carrier`, `name`, `city`, `region`. - Credit Usage API: `GET https://www.ipqualityscore.com/api/json/account/{key}` → `credits` (left in this billing period), `usage`, `phone_usage`. Used by Test, the Settings tab and §3. `// TODO(spec): confirm with the owner's real key that this call costs no credit (SD2).` ### 2.2 FTC Do Not Call complaints (`spam/ftc.ts`) - Source: `GET https://api.ftc.gov/v0/dnc-complaints`, key in `X-Api-Key`, `created_date_from` / `created_date_to`, `sort_order=ASC`, `items_per_page=50` (the maximum), `offset`. Kept per complaint: `id`, `company-phone-number` (10 digits, stored with the leading 1), `created-date`, `subject`, `recorded-message-or-robocall`. - Not the daily CSV files on ftc.gov: the site answers scripted downloads with 403 (seen 2026-10-27). - **Sync job** `spamFtcSync`, every `syncHours` hours (default 24; the FTC posts weekday data by about noon Eastern, weekend data on Monday). It asks from the last synced `created-date` minus one day (complaint ids make the overlap harmless) up to now, then deletes complaints older than `windowDays` (default 30). The first sync fills the whole window. **Sync now** on the Settings tab starts one at once. - Volume: 5,000 to 15,000 complaints a day (probed 2026-10-26), so 100–300 requests a day, and about 6,000 for the first 30 days. api.data.gov allows 1,000 requests an hour per key, so the sync paces itself to 900 an hour, saves where it got to, and the next run carries on (the first fill takes about 7 hours). A 429 ends the run there; the next run carries on. - Risk from complaints in the window (owner, 2026-10-27): **high** at 3 or more, **medium** at 1–2, **low** at none. - Kept for the answer: the complaint count, how many were robocalls, the last complaint's date, the most common subject. - The reports aren't verified, and spammers fake real people's numbers: the softphone and the admin guide say "reported to the FTC", never "is spam". ## 3. When IPQualityScore's credits run out IPQualityScore counts credits per billing period, which needn't be a calendar month, and the free plan may also have a daily cap. So rather than wait for the month to end, the api asks IPQualityScore when credits are back: 1. A check answered "insufficient credits" → IPQualityScore is **paused** (`spam.state.ipqsPausedAt`): - one `system_events` row, `spam.quotaExhausted` (warning, source "Spam check"); - one `warn` log line; - the dashboard warning `spam_quota` (04 §7, `GET /api/system/warnings`). 2. While paused, no request goes to IPQualityScore. Checks go to the FTC list if it's set up, else no checks are made. Nothing is logged per call. 3. Job `spamQuotaCheck`, every 6 hours while paused, asks the Credit Usage API. `credits > 0` → resumed: `ipqsPausedAt` cleared, a `spam.quotaRestored` info event, the warning goes away. If the account call fails, the job tries again next time. The warning, shown once (owner): "IPQualityScore's lookups are used up. Spam checks use the FTC list until they come back." (or "…are off until they come back." without the FTC list). It has a **Dismiss** button (`POST /api/system/warnings/spam_quota/dismiss`, admin), which sets `spam.state.warningDismissedAt`. Dismissed, it stays hidden until the next time the credits run out. ## 4. Cache (IPQualityScore only) - `spam_lookups` (§7) keeps each answer, so a caller who rings again costs no credit: - **flagged** (medium or high) for `cache.flaggedDays`, default **14**; - **low** for `cache.cleanDays`, default **3**. - Both 1–365 days (owner: configurable). - Failures aren't cached. Expired rows go in `retentionCleanup` (04 §8). - One request per number at a time: a check already under way is shared (an in-memory map), so a ring group ringing five people costs one credit. - The FTC needs no cache: its check is already a local query. ## 5. Where checks happen Everything goes through `spam/check.ts` `checkNumber(number, { cachedOnly })` → `{ risk, provider, info } | null`. It runs the §1 rules, then the cache, then the providers (§2). It doesn't do the contacts test: `checkCaller` (§9.4) does that for the user being rung. ### 5.1 Browser softphone - `GET /api/me/caller-check?number=` → `{ check: { risk, provider, info } | null }`. It answers `null` when the number is in this user's books (11 §1), isn't checked (§1), or no provider answered within 5 s. - Only for a number that is ringing at one of the user's extensions, or rang there in the last 2 minutes (live state, 04 §5.1); otherwise `null` without checking. A user can't spend credits on numbers that never called. - The softphone asks once per incoming call, as it starts ringing, and doesn't retry. - Shown under the caller's name, in the ringing dialog and on the call card, for **high** and **medium** only; **low** shows nothing (owner, 2026-10-27): - a red "Likely spam" (high) or amber "Possible spam" (medium) badge; - a line with what's known: IPQualityScore "Fraud score 92 · spammer · VoIP · Verizon · Austin, TX" (and the name, if it has one); FTC "Reported to the FTC 4 times (3 robocalls), last on Oct 20: Warranties & protection plans". - Not in the browser notification: that appears as the call starts, before the answer. ### 5.2 Desk phones - In `GET /api/internal/caller-name` (11 §6.1): when there's no contact name, it waits for the check for up to 1.2 s (inside the dialplan's 2 s). A high or medium answer puts its label in front of the caller ID name: `deskPhoneLabels`, by default "SPAM LIKELY" and "Spam risk" (owner, 2026-10-27: configurable; no `?`, which `phoneSafeName` drops). The whole name goes through `phoneSafeName`, as now. A later answer is still cached for the softphone. - The dialplan doesn't change, so no generator change and no new golden file. ## 6. Call history - `calls.spam_risk` and `calls.spam_info` (§7) are written with the call row (CDR aggregation, incoming calls only) from `checkNumber(from_number, { cachedOnly: true })`: the cache and the local FTC copy, never a request out. - The Calls pages (`/me/calls`, `/admin/calls`) show the §5.1 badge and line, unless the viewer's books name the number (contacts come first, 11 §1). The call list API adds `spam: { risk, provider, info } | null` to each item. The mobile app's call history (13 MB5) gets it from the same API. ## 7. Data and settings Migration (SD1): ```sql create table spam_lookups ( -- IPQualityScore answers (14 §4) number text primary key, -- 1NXXNXXXXXX provider text not null, -- 'ipqs' risk text not null check (risk in ('low', 'medium', 'high')), info jsonb not null, -- the 14 §2.1 fields only expires_at timestamptz not null, -- updated_at is when it was checked created_at timestamptz not null default now(), updated_at timestamptz not null default now() ); create index on spam_lookups (expires_at); create table ftc_complaints ( -- the FTC copy (14 §2.2), kept windowDays; append-only id text primary key, -- the FTC's id number text not null, -- 1NXXNXXXXXX reported_at timestamptz not null, -- the complaint's created-date subject text, robocall boolean, created_at timestamptz not null default now() ); create index on ftc_complaints (number); create index on ftc_complaints (reported_at); alter table calls add column spam_risk text, add column spam_info jsonb; ``` Settings (02 §3): | Key | Type | Default | Notes | |---|---|---|---| | `spam` | object | below | the admin's choices | | `spam.state` | object | `{}` | read-only to the API, like `system.*`: `ipqsPausedAt`, `warningDismissedAt`, `ftcCursor` (`{day, offset}` while a download is under way), `ftcLastSyncAt`, `ftcLastError` | ```ts spam = { enabled: false, ipqs: { keyEnc: string } | null, // null = not used; first when set (§2) ftc: { keyEnc: string, syncHours: 24, windowDays: 30 } | null, // 1–168 hours, 7–365 days cache: { flaggedDays: 14, cleanDays: 3 }, // 1–365 each deskPhoneLabels: { high: 'SPAM LIKELY', medium: 'Spam risk' }, // 1–20 characters, §5.2 } ``` Keys are `_enc` (02, top: `crypto/secrets.ts`): the API returns `hasKey`, never the key; the audit log masks it. Validated with zod (`packages/shared`). ### 7.1 Settings → Spam check - **Spam check on** (the switch asks to confirm the first time, §8). - **Use IPQualityScore**: key, **Test** (the Credit Usage API: shows the credits left without spending one), "paused until credits come back" while paused. - **Use the FTC list**: api.data.gov key (a link to its sign-up), **Test** (one request), re-download every N hours, keep N days, last sync time and result, **Sync now**. With both on, the tab says IPQualityScore answers first and the FTC list when it can't. - Keep answers: flagged N days, others N days. - Desk phone labels: high, medium. Saving refuses a label `phoneSafeName` would change. - Test buttons use the stricter Test rate limit, as the others do (04 §2.2). ## 8. Privacy - Spam check is off until the admin turns it on. With IPQualityScore it is an exception to "nothing leaves the server": the caller's number (and nothing else: no name, no extension) goes to IPQualityScore. The FTC list sends nothing about calls. - Turning on IPQualityScore asks once: "The numbers of outside callers who aren't in your contacts will be sent to IPQualityScore. You may need to say so in your privacy notice, and to have a data processing agreement with them. In the EU and UK the GDPR applies." The audit log records who turned it on. - GDPR covers businesses in the EU or UK and calls from people there, so it isn't the owner's case (US callers only). The admin guide says so for everyone else, along with US state privacy laws such as California's. It's not legal advice. - Admin guide: a new page `docs/admin-guide/spam-check.md` (what each provider is, what leaves the server, the FTC caveat, the free allowance) and a line in `legal.md`. ## 9. Order of work | Task | What | Done when | |---|---|---| | SD1 | Decision 0006; this spec; the §7 migration and settings; `spam/check.ts` with both providers' lookups, the cache and the shared requests; Settings → Spam check with keys, Test, privacy confirmation | `pnpm lint && typecheck && test` pass; checks, cache times, the order (IPQualityScore, then the FTC list) and failures are tested against fake HTTP. Done, see §9.1 | | SD2 | §3: pause, events, the `spam_quota` warning and Dismiss, `spamQuotaCheck` | an exhausted fake pauses, falls back to the FTC list, warns once and resumes; the owner's real key shows the credit call is free. Done but the real-key check, see §9.2 | | SD3 | §2.2 sync: paging, pacing, the saved cursor, the window, Sync now | a fake FTC API with 3 days of data syncs, resumes after a 429 and drops old complaints; one real sync with the owner's key. Done but the real sync, see §9.3 | | SD4 | §5: `/api/me/caller-check`, the softphone badge and line, the desk phone labels | an unknown number rings the softphone and shows its answer; a contact or a low risk shows nothing; a desk phone shows "SPAM LIKELY" when the answer is in time. Done but a real call, see §9.4 | | SD5 | §6 call history; the admin guide page | a checked call keeps its badge in call history after its cache expires. Done, see §9.5 | ### 9.1 As built: SD1 - `spam/check.ts` `checkNumber(deps, number, { cachedOnly })`. There's no `wait`: a caller that can't wait (14 §5.1, §5.2) races it against its own timer, and the check carries on and fills the cache. The cache is only read while IPQualityScore is set up; turning it off leaves the rows to expire. - `spam/ipqs.ts`, `spam/ftc.ts` (the local lookup and Test; the sync is §9.3), `spam/http.ts` (`getJson`, 4 s timeout, and `SpamError`, the only error text that is logged or shown: never the URL or fetch's own error). A provider message is cut to 200 characters with the key taken out, in case it quotes it. - "Insufficient credits": see §9.2. - Settings: the `spam` key holds the admin's choices, `spam.state` is read-only to PATCH (like `system.*`). PATCH refuses a provider with no key typed or saved ("Enter the IPQualityScore key", "Enter the api.data.gov key") and spam check on with neither provider. Labels are 1–20 of `phoneSafeName`'s characters, runs of spaces made one. The audit log masks `keyEnc` (its `…Enc` rule). - `POST /api/settings/spam/ipqs/test` and `/ftc/test` `{ key? }` → `{ ok, detail | error }`: the typed key, else the saved one. IPQualityScore's is the Credit Usage API ("Works: 912 credits left this billing period."); the FTC's is one request for one complaint. - Settings → Spam check (`SpamTab`): the switch, a section per provider with its key and Test, the cache days and the desk phone labels. Saving with IPQualityScore newly in use opens the 14 §8 confirmation first. - `retentionCleanup` deletes expired `spam_lookups` rows. - Tests: `test/spam.test.ts` with a stubbed `fetch` (risk, kept fields, cache days and expiry, the shared request, IPQualityScore down / timing out → the FTC list, warnings without the key, `cachedOnly`, the settings and Test routes). ### 9.2 As built: SD2 - `spam/check.ts` `patchSpamState(db, patch, onlyIf?)` merges into `spam.state` in one statement (`value || patch`), so a check and a job never overwrite each other's fields; SD3's cursor uses it too. Pausing is `onlyIf` not paused yet, so of several checks running out at once, one logs the warning and the `spam.quotaExhausted` event. - Paused: the cache still answers (14 §4), IPQualityScore isn't asked, nothing is logged. - `spamQuotaCheck` runs at start and every 6 hours; it does nothing unless paused. A failing account call throws `SpamError` (no key in it), which the job's log line shows. - Also resumed (owner's intent, not in §3): saving a newly typed IPQualityScore key, so a new account isn't paused for up to 6 hours. - `Warning` has `dismissable`; the dashboard shows **Dismiss** next to **Fix** for it. Pausing clears `warningDismissedAt`, so the warning comes back the next time. - Not done: the owner's real key showing that the Credit Usage API costs no credit (the TODO in §2.1). If it does cost one, 4 calls a day while paused is the price. - Tests: `test/spam.test.ts` (two checks running out at once: one warning, one event; the FTC list answering; no requests while paused; the warning and Dismiss; the job leaving it paused at 0 credits and resuming above; the "are off" text without the FTC list; a new key; the job asking nothing unless paused and failing without the key). ### 9.3 As built: SD3 - `spam/ftc.ts` `ftcSync(db, { now, budget })`, job `spamFtcSync` **every minute**. It does nothing unless spam check and the FTC list are on and a download is due (`ftcLastSyncAt` older than `syncHours`, or never) or under way (`ftcCursor`). - One FTC date at a time (`created_date_from` / `_to` that day, `sort_order=ASC`, 50 a page, `offset`), so an offset never grows past one day's complaints. A short page moves on to the next day; today's short page ends the download: complaints older than `windowDays` are deleted and `ftcLastSyncAt` set. - Pacing: at most 15 pages a minute (900 an hour), and the place (`ftcCursor`) saved after each run, so a restart carries on. The first 30 days take about 7 hours. - Where a download starts: the newest complaint kept, less a day (for ones posted late, as on weekends), but no further back than `windowDays`. Ids make the overlap harmless. - The FTC's dates have no zone: taken as Eastern, so a day is midnight to midnight in Washington. Complaints without a 10-digit US number or a full date are skipped. - A failure (403 "api.data.gov refused the key", a 429 "…over its hourly limit", a timeout) keeps the place and is saved in `ftcLastError`; the next minute tries again from the same page. The job logs it (`warn`) only when it differs from the last one, so a long outage is one line. - Settings → Spam check: "Downloading: at the complaints of {day}", "Last download {time}", the last failure, and **Download now** (`POST /api/settings/spam/ftc/sync`, which clears `ftcLastSyncAt` so the next minute starts one; 400 without the FTC list). - Not done: one real download with the owner's api.data.gov key. - Tests: `test/ftc-sync.test.ts` against a fake api.ftc.gov (off; the first download a day at a time with three pages on one day, old complaints dropped, robocall and Eastern time; not again until `syncHours`, then from the newest day less one; 15 pages a minute with the place saved; a 429 stopping and the next minute carrying on without duplicates; the same failure logged once; a refused key). `test/spam.test.ts`: Download now. ### 9.4 As built: SD4 - `spam/check.ts` `checkCaller(deps, userId, number, waitMs)`: the contacts test (that user's books and the company's, `contactNames` without waiting on Contrack; books that can't be read count as "known", so nothing is shown), then `checkNumber` raced against `waitMs`. A late answer still fills the cache. `usNumber` normalizes (00 §8) or gives ''. - `GET /api/me/caller-check?number=` (5 s): only for an inbound call in live state (04 §5.1) from that number with one of my extensions among its channels (the browser's `PJSIP/101-web-…` and ring group / queue `Local/101@…` legs count). Live state already keeps the caller's number (the first channel's caller ID), so nothing was added there. - Not built: "or rang there in the last 2 minutes". The softphone asks as the call starts ringing, so it's in live state; a call gone before its question arrives has nothing to show. - `GET /api/internal/caller-name`: no contact → `checkCaller(…, 1200)`; high or medium → `deskPhoneLabels` and, from IPQualityScore, its name for the line. The label **replaces** the carrier's caller ID name rather than going in front of it: the dialplan doesn't send that name, and §5.2 keeps the dialplan as it is. Its failures log through the app's logger (the route's own is silent: the token is in the query). - Softphone: `CallerCheck` in `SoftphoneDock.tsx`, in the ringing dialog and on each call card (so the note stays after answering). Asked once per number (react-query, no retry), only for an incoming call from 10+ digits that the browser doesn't already know as a contact. - `components/SpamNote.tsx`: the badge and line (`spamLine`), for SD5's call history too. - Tests: `test/spam.test.ts` "while it rings" (only callers ringing me, +1 formatting, someone else's call, not ringing, a contact, low; desk labels, a contact's name, a number nobody has, a late answer then the cache, custom labels from the FTC list), `SpamNote.test.ts` (the line). - Not checked: a real call in the browser and on a desk phone. ### 9.5 As built: SD5 - `aggregateCall` (04 §5.4) runs `checkNumber(…, { cachedOnly: true })` for inbound calls and writes `spam_risk` and `spam_info` (`{ provider, info }`) with the row. A low answer is kept too (the FTC list answers low for a number nobody reported); null means not checked. - `CallDto.spam` on `/api/calls`, `/api/calls/:id` and `/api/me/calls`: null when the viewer's books (theirs and the company's) name the number, or when the books can't be read. - Calls pages: `SpamNote` under the caller, on `/me/calls` only for calls to me that the browser doesn't know as a contact. The CSV export has no spam columns. - Admin guide: `docs/admin-guide/spam-check.md`. - Tests: `test/spam.test.ts` (four calls aggregated: IPQualityScore's answer and the FTC's kept after the cache is emptied, a contact's call shows nothing, an unreported number low, and no request out). Later, not now: - **Report as spam** (owner, 2026-10-27: not now): a button in call history that reports the number to a provider that takes reports (OpenSpam has `POST /api/report`). - **OpenSpam** as a provider for Spanish numbers (the owner has a key; it has little US data). - **Low-risk details** (owner, 2026-10-27: nothing for now): show what IPQualityScore knows about a low-risk caller too (line type, carrier, city, name), without a badge.