selfphoneng/docs/spec/14-spam.md

14 — Spam check

Owner, 2026-10-27: when an outside caller isn't in anyone's contacts, look their number up and show what's known about them. Why these providers and why it may leave the server: decision 0006.

1. What it does

• An outside call rings a user, and the number is none of: an extension, a contact of that user, a company phonebook entry (the 11 §1 lookup finds no name). Then the api checks the number with a provider and the user sees the answer while it rings.
• Information only (owner): a check never blocks, reroutes or delays a call. The call rings at once; the answer appears when it arrives.
• US numbers only (owner: the callers are in the US): 1NXXNXXXXXX after normalizeNumber (00 §8). Anything else, a withheld number, and internal calls are never checked.
• Shown in the browser softphone and call history; on desk phones only when the answer is in before their caller-name lookup gives up (§5.2).
• Off by default (owner). The admin turns it on in Settings → Spam check (§8).
• Never in the way: a provider that fails or is slow only means no answer is shown. Each failure is one warn log line with the provider and the reason (HTTP status, timeout, the provider's message), never the key or the request URL (IPQualityScore puts its key in the path).

2. Providers

The admin sets up either provider or both. The order is fixed (owner, 2026-10-27): IPQualityScore first when it's set up; the FTC list answers when IPQualityScore isn't set up or can't answer. v1 has these two:

	IPQualityScore	FTC Do Not Call complaints
What	Fraud score and flags for any number, live	Unwanted-call complaints people filed with the FTC
Where it runs	An HTTPS request per check	A local copy, synced by a job; a check is a database query
Sends out	The caller's number	Nothing (it only downloads public data)
Key	IPQualityScore API key	api.data.gov key (free)
Cost	Free plan 1,000 lookups a month; paid from about $99 a month	Free

"Can't answer" means: not set up, paused (§3), or this check failed. The FTC list is then asked for that check only; nothing is remembered about the failure. Neither set up → no checks.

2.1 IPQualityScore (spam/ipqs.ts)

• GET https://www.ipqualityscore.com/api/json/phone/{key}/{number}?country[]=US, the number as 11 digits, 4 s timeout.
• success: false whose message says the credits are used up ("You have insufficient credits to make this query…") → exhausted, §3. Any other success: false, an HTTP error or a timeout → a failure (logged, not cached, the FTC list asked).
• Risk (owner, 2026-10-27): high if spammer or fraud_score >= 85; medium if fraud_score >= 75, recent_abuse or risky; else low. These follow IPQualityScore's general advice (75 suspicious, 85 high risk).
• Kept (and nothing else from the answer): fraud_score, spammer, recent_abuse, line_type, carrier, name, city, region.
• Credit Usage API: GET https://www.ipqualityscore.com/api/json/account/{key} → credits (left in this billing period), usage, phone_usage. Used by Test, the Settings tab and §3. // TODO(spec): confirm with the owner's real key that this call costs no credit (SD2).

2.2 FTC Do Not Call complaints (spam/ftc.ts)

• Source: GET https://api.ftc.gov/v0/dnc-complaints, key in X-Api-Key, created_date_from / created_date_to, sort_order=ASC, items_per_page=50 (the maximum), offset. Kept per complaint: id, company-phone-number (10 digits, stored with the leading 1), created-date, subject, recorded-message-or-robocall.
• Not the daily CSV files on ftc.gov: the site answers scripted downloads with 403 (seen 2026-10-27).
• Sync job spamFtcSync, every syncHours hours (default 24; the FTC posts weekday data by about noon Eastern, weekend data on Monday). It asks from the last synced created-date minus one day (complaint ids make the overlap harmless) up to now, then deletes complaints older than windowDays (default 30). The first sync fills the whole window. Sync now on the Settings tab starts one at once.
• Volume: 5,000 to 15,000 complaints a day (probed 2026-10-26), so 100–300 requests a day, and about 6,000 for the first 30 days. api.data.gov allows 1,000 requests an hour per key, so the sync paces itself to 900 an hour, saves where it got to, and the next run carries on (the first fill takes about 7 hours). A 429 ends the run there; the next run carries on.
• Risk from complaints in the window (owner, 2026-10-27): high at 3 or more, medium at 1–2, low at none.
• Kept for the answer: the complaint count, how many were robocalls, the last complaint's date, the most common subject.
• The reports aren't verified, and spammers fake real people's numbers: the softphone and the admin guide say "reported to the FTC", never "is spam".

3. When IPQualityScore's credits run out

IPQualityScore counts credits per billing period, which needn't be a calendar month, and the free plan may also have a daily cap. So rather than wait for the month to end, the api asks IPQualityScore when credits are back:

1. A check answered "insufficient credits" → IPQualityScore is paused (spam.state.ipqsPausedAt):
   • one system_events row, spam.quotaExhausted (warning, source "Spam check");
   • one warn log line;
   • the dashboard warning spam_quota (04 §7, GET /api/system/warnings).
2. While paused, no request goes to IPQualityScore. Checks go to the FTC list if it's set up, else no checks are made. Nothing is logged per call.
3. Job spamQuotaCheck, every 6 hours while paused, asks the Credit Usage API. credits > 0 → resumed: ipqsPausedAt cleared, a spam.quotaRestored info event, the warning goes away. If the account call fails, the job tries again next time.

The warning, shown once (owner): "IPQualityScore's lookups are used up. Spam checks use the FTC list until they come back." (or "…are off until they come back." without the FTC list). It has a Dismiss button (POST /api/system/warnings/spam_quota/dismiss, admin), which sets spam.state.warningDismissedAt. Dismissed, it stays hidden until the next time the credits run out.

4. Cache (IPQualityScore only)

• spam_lookups (§7) keeps each answer, so a caller who rings again costs no credit:
  • flagged (medium or high) for cache.flaggedDays, default 14;
  • low for cache.cleanDays, default 3.
  • Both 1–365 days (owner: configurable).
• Failures aren't cached. Expired rows go in retentionCleanup (04 §8).
• One request per number at a time: a check already under way is shared (an in-memory map), so a ring group ringing five people costs one credit.
• The FTC needs no cache: its check is already a local query.

5. Where checks happen

Everything goes through spam/check.ts checkNumber(number, { cachedOnly }) → { risk, provider, info } | null. It runs the §1 rules, then the cache, then the providers (§2). It doesn't do the contacts test: checkCaller (§9.4) does that for the user being rung.

5.1 Browser softphone

• GET /api/me/caller-check?number= → { check: { risk, provider, info } | null }. It answers null when the number is in this user's books (11 §1), isn't checked (§1), or no provider answered within 5 s.
• Only for a number that is ringing at one of the user's extensions, or rang there in the last 2 minutes (live state, 04 §5.1); otherwise null without checking. A user can't spend credits on numbers that never called.
• The softphone asks once per incoming call, as it starts ringing, and doesn't retry.
• Shown under the caller's name, in the ringing dialog and on the call card, for high and medium only; low shows nothing (owner, 2026-10-27):
  • a red "Likely spam" (high) or amber "Possible spam" (medium) badge;
  • a line with what's known: IPQualityScore "Fraud score 92 · spammer · VoIP · Verizon · Austin, TX" (and the name, if it has one); FTC "Reported to the FTC 4 times (3 robocalls), last on Oct 20: Warranties & protection plans".
  • Not in the browser notification: that appears as the call starts, before the answer.

5.2 Desk phones

• In GET /api/internal/caller-name (11 §6.1): when there's no contact name, it waits for the check for up to 1.2 s (inside the dialplan's 2 s). A high or medium answer puts its label in front of the caller ID name: deskPhoneLabels, by default "SPAM LIKELY" and "Spam risk" (owner, 2026-10-27: configurable; no ?, which phoneSafeName drops). The whole name goes through phoneSafeName, as now. A later answer is still cached for the softphone.
• The dialplan doesn't change, so no generator change and no new golden file.

6. Call history

• calls.spam_risk and calls.spam_info (§7) are written with the call row (CDR aggregation, incoming calls only) from checkNumber(from_number, { cachedOnly: true }): the cache and the local FTC copy, never a request out.
• The Calls pages (/me/calls, /admin/calls) show the §5.1 badge and line, unless the viewer's books name the number (contacts come first, 11 §1). The call list API adds spam: { risk, provider, info } | null to each item. The mobile app's call history (13 MB5) gets it from the same API.

7. Data and settings

Migration (SD1):

create table spam_lookups (            -- IPQualityScore answers (14 §4)
  number      text primary key,        -- 1NXXNXXXXXX
  provider    text not null,           -- 'ipqs'
  risk        text not null check (risk in ('low', 'medium', 'high')),
  info        jsonb not null,          -- the 14 §2.1 fields only
  expires_at  timestamptz not null,    -- updated_at is when it was checked
  created_at  timestamptz not null default now(),
  updated_at  timestamptz not null default now()
);
create index on spam_lookups (expires_at);

create table ftc_complaints (          -- the FTC copy (14 §2.2), kept windowDays; append-only
  id           text primary key,       -- the FTC's id
  number       text not null,          -- 1NXXNXXXXXX
  reported_at  timestamptz not null,   -- the complaint's created-date
  subject      text,
  robocall     boolean,
  created_at   timestamptz not null default now()
);
create index on ftc_complaints (number);
create index on ftc_complaints (reported_at);

alter table calls add column spam_risk text, add column spam_info jsonb;

Settings (02 §3):

Key	Type	Default	Notes
spam	object	below	the admin's choices
spam.state	object	{}	read-only to the API, like system.*: ipqsPausedAt, warningDismissedAt, ftcCursor ({day, offset} while a download is under way), ftcLastSyncAt, ftcLastError

spam = {
  enabled: false,
  ipqs: { keyEnc: string } | null,                       // null = not used; first when set (§2)
  ftc: { keyEnc: string, syncHours: 24, windowDays: 30 } | null,   // 1–168 hours, 7–365 days
  cache: { flaggedDays: 14, cleanDays: 3 },              // 1–365 each
  deskPhoneLabels: { high: 'SPAM LIKELY', medium: 'Spam risk' },  // 1–20 characters, §5.2
}

Keys are _enc (02, top: crypto/secrets.ts): the API returns hasKey, never the key; the audit log masks it. Validated with zod (packages/shared).

7.1 Settings → Spam check

• Spam check on (the switch asks to confirm the first time, §8).
• Use IPQualityScore: key, Test (the Credit Usage API: shows the credits left without spending one), "paused until credits come back" while paused.
• Use the FTC list: api.data.gov key (a link to its sign-up), Test (one request), re-download every N hours, keep N days, last sync time and result, Sync now. With both on, the tab says IPQualityScore answers first and the FTC list when it can't.
• Keep answers: flagged N days, others N days.
• Desk phone labels: high, medium. Saving refuses a label phoneSafeName would change.
• Test buttons use the stricter Test rate limit, as the others do (04 §2.2).

8. Privacy

• Spam check is off until the admin turns it on. With IPQualityScore it is an exception to "nothing leaves the server": the caller's number (and nothing else: no name, no extension) goes to IPQualityScore. The FTC list sends nothing about calls.
• Turning on IPQualityScore asks once: "The numbers of outside callers who aren't in your contacts will be sent to IPQualityScore. You may need to say so in your privacy notice, and to have a data processing agreement with them. In the EU and UK the GDPR applies." The audit log records who turned it on.
• GDPR covers businesses in the EU or UK and calls from people there, so it isn't the owner's case (US callers only). The admin guide says so for everyone else, along with US state privacy laws such as California's. It's not legal advice.
• Admin guide: a new page docs/admin-guide/spam-check.md (what each provider is, what leaves the server, the FTC caveat, the free allowance) and a line in legal.md.

9. Order of work

Task	What	Done when
SD1	Decision 0006; this spec; the §7 migration and settings; spam/check.ts with both providers' lookups, the cache and the shared requests; Settings → Spam check with keys, Test, privacy confirmation	pnpm lint && typecheck && test pass; checks, cache times, the order (IPQualityScore, then the FTC list) and failures are tested against fake HTTP. Done, see §9.1
SD2	§3: pause, events, the spam_quota warning and Dismiss, spamQuotaCheck	an exhausted fake pauses, falls back to the FTC list, warns once and resumes; the owner's real key shows the credit call is free. Done but the real-key check, see §9.2
SD3	§2.2 sync: paging, pacing, the saved cursor, the window, Sync now	a fake FTC API with 3 days of data syncs, resumes after a 429 and drops old complaints; one real sync with the owner's key. Done but the real sync, see §9.3
SD4	§5: /api/me/caller-check, the softphone badge and line, the desk phone labels	an unknown number rings the softphone and shows its answer; a contact or a low risk shows nothing; a desk phone shows "SPAM LIKELY" when the answer is in time. Done but a real call, see §9.4
SD5	§6 call history; the admin guide page	a checked call keeps its badge in call history after its cache expires. Done, see §9.5

9.1 As built: SD1

• spam/check.ts checkNumber(deps, number, { cachedOnly }). There's no wait: a caller that can't wait (14 §5.1, §5.2) races it against its own timer, and the check carries on and fills the cache. The cache is only read while IPQualityScore is set up; turning it off leaves the rows to expire.
• spam/ipqs.ts, spam/ftc.ts (the local lookup and Test; the sync is §9.3), spam/http.ts (getJson, 4 s timeout, and SpamError, the only error text that is logged or shown: never the URL or fetch's own error). A provider message is cut to 200 characters with the key taken out, in case it quotes it.
• "Insufficient credits": see §9.2.
• Settings: the spam key holds the admin's choices, spam.state is read-only to PATCH (like system.*). PATCH refuses a provider with no key typed or saved ("Enter the IPQualityScore key", "Enter the api.data.gov key") and spam check on with neither provider. Labels are 1–20 of phoneSafeName's characters, runs of spaces made one. The audit log masks keyEnc (its …Enc rule).
• POST /api/settings/spam/ipqs/test and /ftc/test { key? } → { ok, detail | error }: the typed key, else the saved one. IPQualityScore's is the Credit Usage API ("Works: 912 credits left this billing period."); the FTC's is one request for one complaint.
• Settings → Spam check (SpamTab): the switch, a section per provider with its key and Test, the cache days and the desk phone labels. Saving with IPQualityScore newly in use opens the 14 §8 confirmation first.
• retentionCleanup deletes expired spam_lookups rows.
• Tests: test/spam.test.ts with a stubbed fetch (risk, kept fields, cache days and expiry, the shared request, IPQualityScore down / timing out → the FTC list, warnings without the key, cachedOnly, the settings and Test routes).

9.2 As built: SD2

• spam/check.ts patchSpamState(db, patch, onlyIf?) merges into spam.state in one statement (value || patch), so a check and a job never overwrite each other's fields; SD3's cursor uses it too. Pausing is onlyIf not paused yet, so of several checks running out at once, one logs the warning and the spam.quotaExhausted event.
• Paused: the cache still answers (14 §4), IPQualityScore isn't asked, nothing is logged.
• spamQuotaCheck runs at start and every 6 hours; it does nothing unless paused. A failing account call throws SpamError (no key in it), which the job's log line shows.
• Also resumed (owner's intent, not in §3): saving a newly typed IPQualityScore key, so a new account isn't paused for up to 6 hours.
• Warning has dismissable; the dashboard shows Dismiss next to Fix for it. Pausing clears warningDismissedAt, so the warning comes back the next time.
• Not done: the owner's real key showing that the Credit Usage API costs no credit (the TODO in §2.1). If it does cost one, 4 calls a day while paused is the price.
• Tests: test/spam.test.ts (two checks running out at once: one warning, one event; the FTC list answering; no requests while paused; the warning and Dismiss; the job leaving it paused at 0 credits and resuming above; the "are off" text without the FTC list; a new key; the job asking nothing unless paused and failing without the key).

9.3 As built: SD3

• spam/ftc.ts ftcSync(db, { now, budget }), job spamFtcSync every minute. It does nothing unless spam check and the FTC list are on and a download is due (ftcLastSyncAt older than syncHours, or never) or under way (ftcCursor).
• One FTC date at a time (created_date_from / _to that day, sort_order=ASC, 50 a page, offset), so an offset never grows past one day's complaints. A short page moves on to the next day; today's short page ends the download: complaints older than windowDays are deleted and ftcLastSyncAt set.
• Pacing: at most 15 pages a minute (900 an hour), and the place (ftcCursor) saved after each run, so a restart carries on. The first 30 days take about 7 hours.
• Where a download starts: the newest complaint kept, less a day (for ones posted late, as on weekends), but no further back than windowDays. Ids make the overlap harmless.
• The FTC's dates have no zone: taken as Eastern, so a day is midnight to midnight in Washington. Complaints without a 10-digit US number or a full date are skipped.
• A failure (403 "api.data.gov refused the key", a 429 "…over its hourly limit", a timeout) keeps the place and is saved in ftcLastError; the next minute tries again from the same page. The job logs it (warn) only when it differs from the last one, so a long outage is one line.
• Settings → Spam check: "Downloading: at the complaints of {day}", "Last download {time}", the last failure, and Download now (POST /api/settings/spam/ftc/sync, which clears ftcLastSyncAt so the next minute starts one; 400 without the FTC list).
• Not done: one real download with the owner's api.data.gov key.
• Tests: test/ftc-sync.test.ts against a fake api.ftc.gov (off; the first download a day at a time with three pages on one day, old complaints dropped, robocall and Eastern time; not again until syncHours, then from the newest day less one; 15 pages a minute with the place saved; a 429 stopping and the next minute carrying on without duplicates; the same failure logged once; a refused key). test/spam.test.ts: Download now.

9.4 As built: SD4

• spam/check.ts checkCaller(deps, userId, number, waitMs): the contacts test (that user's books and the company's, contactNames without waiting on Contrack; books that can't be read count as "known", so nothing is shown), then checkNumber raced against waitMs. A late answer still fills the cache. usNumber normalizes (00 §8) or gives ''.
• GET /api/me/caller-check?number= (5 s): only for an inbound call in live state (04 §5.1) from that number with one of my extensions among its channels (the browser's PJSIP/101-web-… and ring group / queue Local/101@… legs count). Live state already keeps the caller's number (the first channel's caller ID), so nothing was added there.
• Not built: "or rang there in the last 2 minutes". The softphone asks as the call starts ringing, so it's in live state; a call gone before its question arrives has nothing to show.
• GET /api/internal/caller-name: no contact → checkCaller(…, 1200); high or medium → deskPhoneLabels and, from IPQualityScore, its name for the line. The label replaces the carrier's caller ID name rather than going in front of it: the dialplan doesn't send that name, and §5.2 keeps the dialplan as it is. Its failures log through the app's logger (the route's own is silent: the token is in the query).
• Softphone: CallerCheck in SoftphoneDock.tsx, in the ringing dialog and on each call card (so the note stays after answering). Asked once per number (react-query, no retry), only for an incoming call from 10+ digits that the browser doesn't already know as a contact.
• components/SpamNote.tsx: the badge and line (spamLine), for SD5's call history too.
• Tests: test/spam.test.ts "while it rings" (only callers ringing me, +1 formatting, someone else's call, not ringing, a contact, low; desk labels, a contact's name, a number nobody has, a late answer then the cache, custom labels from the FTC list), SpamNote.test.ts (the line).
• Not checked: a real call in the browser and on a desk phone.

9.5 As built: SD5

• aggregateCall (04 §5.4) runs checkNumber(…, { cachedOnly: true }) for inbound calls and writes spam_risk and spam_info ({ provider, info }) with the row. A low answer is kept too (the FTC list answers low for a number nobody reported); null means not checked.
• CallDto.spam on /api/calls, /api/calls/:id and /api/me/calls: null when the viewer's books (theirs and the company's) name the number, or when the books can't be read.
• Calls pages: SpamNote under the caller, on /me/calls only for calls to me that the browser doesn't know as a contact. The CSV export has no spam columns.
• Admin guide: docs/admin-guide/spam-check.md.
• Tests: test/spam.test.ts (four calls aggregated: IPQualityScore's answer and the FTC's kept after the cache is emptied, a contact's call shows nothing, an unreported number low, and no request out).

Later, not now:

• Report as spam (owner, 2026-10-27: not now): a button in call history that reports the number to a provider that takes reports (OpenSpam has POST /api/report).
• OpenSpam as a provider for Spanish numbers (the owner has a key; it has little US data).
• Low-risk details (owner, 2026-10-27: nothing for now): show what IPQualityScore knows about a low-risk caller too (line type, carrier, city, name), without a badge.