A full-stack price monitoring system for NAS hard drives. Tracks WD Red Pro, Seagate IronWolf Pro, and Toshiba N300 (12–18TB) across Amazon, Newegg, Best Buy, and manufacturer sites. Includes user registration, email verification, personalised price-drop email alerts, and a role-based admin panel for managing accounts.

Project Structure

nas_tracker/
│
├── config.py                   # .env loader — import this first in every entry-point
├── .env                        # Your local secrets (never commit — see .gitignore)
├── .env.example                # Template with every variable documented
├── .gitignore                  # Excludes .env, data/, logs/, __pycache__/
│
├── scraper/                    # Price-scraping engine
│   ├── __init__.py
│   ├── catalog.py              # All 12 target drives: model numbers, MSRPs, search terms
│   ├── database.py             # SQLite schema + every read/write helper for price data
│   ├── http_client.py          # Rate-limited HTTP with UA rotation, retries, proxy support
│   ├── retailers.py            # Per-retailer HTML parsers (Amazon, Newegg, Best Buy, Mfr)
│   ├── alerts.py               # Alert engine: price drop, new low, back-in-stock, new model
│   └── runner.py               # Orchestrator: --seed | --once | --schedule modes
│
├── auth/                       # User accounts, roles, and email verification
│   ├── __init__.py
│   ├── users.py                # User DB layer + admin CRUD + CLI management tool
│   ├── email.py                # SMTP email service + HTML templates for all email types
│   └── middleware.py           # Flask decorators: @require_auth, @require_admin, @optional_auth
│
├── api/                        # REST API server
│   ├── __init__.py
│   └── server.py               # Flask app: auth + admin + price-tracker endpoints (26 total)
│
├── frontend/                   # Browser UI (no build step)
│   ├── index.html              # Main dashboard: auth wall, price table, charts, alerts
│   ├── auth.js                 # Auth client: login, register, session storage, prefs
│   └── admin.html              # Admin panel: user management, role control, session viewer
│
├── data/                       # Runtime data (auto-created, excluded from git)
│   └── prices.db               # SQLite database (drives, prices, users, sessions, alerts)
│
└── logs/                       # Runtime logs (auto-created, excluded from git)
    └── scraper.log             # Scraper output log

Quick Start

1. Install dependencies

pip install requests beautifulsoup4 lxml flask python-dotenv

2. Configure environment

cp .env.example .env
# Open .env and fill in your SMTP credentials and other settings

See the Environment Variables section for all options.

3. Seed the database

Populates the drives table from the catalog. Run once on first setup.

python -m scraper.runner --seed

4. Run a scrape

python -m scraper.runner --once

5. Start the API server

python -m api.server
# → http://localhost:5000

6. Open the dashboard

Open frontend/index.html in your browser. It connects to http://localhost:5000 by default.

Register an account — the first account created is automatically promoted to admin. Verify your email, sign in, and the full price dashboard is visible. Admins see an "Admin Panel" link in their user menu.

Tip: To point the frontend at a different API host, set window.API_BASE = 'https://your-host' before the page scripts load, or serve both files from the same origin as the API.

User Roles

Every account has a role of either user or admin.

Role	Dashboard	Admin Panel	Admin API
`user`	✓	✗	✗ (403)
`admin`	✓	✓	✓

The first account registered is automatically given the admin role if no other admin exists yet. Subsequent registrations default to user. Role changes can be made via the admin panel or the CLI tool.

Configuration

All configuration is read from environment variables. The easiest way to set them is a .env file in the project root — it is loaded automatically on startup via config.py.

cp .env.example .env
# edit .env — real environment variables always take precedence over .env values

To use a custom .env path (e.g. for staging vs production):

NAS_TRACKER_ENV=/etc/nas-tracker/prod.env python -m api.server

Environment Variables

Email (SMTP)

Variable	Default	Description
`SMTP_HOST`	`localhost`	SMTP server hostname
`SMTP_PORT`	`587`	Port: 587 = STARTTLS (recommended), 465 = SSL, 25 = none
`SMTP_USER`	(none)	SMTP login username
`SMTP_PASS`	(none)	SMTP password or app-password
`SMTP_FROM_NAME`	`NAS Drive Tracker`	Display name shown in sent emails
`SMTP_FROM_EMAIL`	falls back to `SMTP_USER`	From address in sent emails
`APP_BASE_URL`	`http://localhost:5000`	Base URL for links in emails (no trailing slash)

Gmail setup: create an App Password at https://myaccount.google.com/apppasswords and use it as SMTP_PASS.

SendGrid setup: set SMTP_HOST=smtp.sendgrid.net, SMTP_USER=apikey, SMTP_PASS=<your-api-key>.

No SMTP configured? Registration still works. The API returns a verify_token in the response body, and the registration UI renders it as a clickable verification link so users are never locked out.

API Server

Variable	Default	Description
`PORT`	`5000`	Port the Flask API listens on
`FLASK_ENV`	(none)	Set to `development` for debug mode + dev shortcuts
`FLASK_SECRET`	random (regenerated on restart)	Flask session secret — set a fixed value in production

Production note: always set FLASK_SECRET to a fixed random string so sessions survive server restarts:
python3 -c "import secrets; print(secrets.token_hex(32))"

Scraper

Variable	Default	Description
`SCRAPE_INTERVAL_HOURS`	`6`	Hours between runs in `--schedule` mode
`SCRAPE_WORKERS`	`3`	Parallel threads per scrape run (keep low to avoid blocks)
`PROXY_URL`	(none)	HTTP/S proxy URL for all scraper requests

Alerts

Variable	Default	Description
`ALERT_DROP_PCT`	`20`	Fire alert when % off MSRP is at or above this value
`ALERT_DROP_ABS`	`30`	Fire alert when absolute $ drop from last price exceeds this
`ALERT_EMAIL_TO`	(none)	Comma-separated admin addresses for scraper-level alerts

Logging

Variable	Default	Description
`LOG_LEVEL`	`INFO`	Verbosity: `DEBUG` / `INFO` / `WARNING`

Scheduled Scraping

Run a scrape every N hours as a background process:

python -m scraper.runner --schedule &

Or schedule with cron for more reliable operation:

0 */6 * * * cd /path/to/nas_tracker && python -m scraper.runner --once >> logs/cron.log 2>&1

You can also trigger a one-off scrape from the dashboard sidebar, or directly via the API:

curl -X POST http://localhost:5000/api/scrape/trigger

API Reference

All endpoints return { "ok": true, "data": ... } on success, or { "ok": false, "error": "..." } on failure.

Auth column key: — = public, ✓ = requires valid session token (Authorization: Bearer <token>), ★ = requires admin role (returns 403 for non-admins).

Auth Endpoints

Method	Endpoint	Auth	Description
POST	`/api/auth/register`	—	Create account. Body: `{ name, email, dob, password }`
POST	`/api/auth/verify-email`	—	Consume email-verification token. Body: `{ token }`
POST	`/api/auth/resend-verify`	—	Resend verification email. Body: `{ email }`
POST	`/api/auth/login`	—	Get session token. Body: `{ email, password }`
POST	`/api/auth/logout`	✓	Invalidate the current session token
GET	`/api/auth/me`	✓	Current user profile (password fields never returned)
PUT	`/api/auth/me/prefs`	✓	Update alert preferences (see body below)
GET	`/api/auth/smtp-status`	✓	Test live SMTP connectivity (no credentials exposed)

Registration and verification flow

1. POST /api/auth/register
     → account created with email_verified = false
     → verification email sent (or verify_token returned in body if SMTP is unconfigured)

2. User clicks the link in the email (or uses the verify_token from the response)
     → browser opens /?token=<token>
     → frontend POSTs token to /api/auth/verify-email
     → email_verified = true; welcome email sent

3. POST /api/auth/login
     → returns { token, user }
     → store token; send as Authorization: Bearer <token> on all subsequent requests

Development shortcut

When FLASK_ENV=development (or when SMTP is unconfigured), the register response includes a dev_verify_token / verify_token field:

# Register
curl -s -X POST http://localhost:5000/api/auth/register \
  -H "Content-Type: application/json" \
  -d '{"name":"Ada","email":"ada@test.com","dob":"1990-01-01","password":"pass1234"}'

# Copy verify_token from the response, then:
curl -s -X POST http://localhost:5000/api/auth/verify-email \
  -H "Content-Type: application/json" \
  -d '{"token":"<verify_token>"}'

# Log in:
curl -s -X POST http://localhost:5000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"ada@test.com","password":"pass1234"}'

Alert preferences body (`PUT /api/auth/me/prefs`)

{
  "alert_enabled":   true,
  "alert_threshold": 20.0,
  "alert_brands":    "all"
}

alert_brands accepts "all" or a JSON array: ["WD", "Seagate", "Toshiba"]. alert_threshold is clamped to the range 1–80.

Admin Endpoints

All admin endpoints require an admin-role session. Non-admins receive HTTP 403. Unauthenticated requests receive HTTP 401.

Method	Endpoint	Auth	Description
GET	`/api/admin/stats`	★	Summary counts: users, admins, verified, pending, suspended, active sessions
GET	`/api/admin/users`	★	Paginated user list. Query: `search`, `role`, `page`, `per_page`
GET	`/api/admin/users/<id>`	★	Single user detail including recent sessions
POST	`/api/admin/users`	★	Create a user directly (pre-verified, no rate limiting)
PATCH	`/api/admin/users/<id>`	★	Update role, active status, verified flag, or notes
POST	`/api/admin/users/<id>/password`	★	Force-reset a user's password (invalidates all sessions)
DELETE	`/api/admin/users/<id>/sessions`	★	Invalidate all active sessions for a user (force logout)
DELETE	`/api/admin/users/<id>`	★	Permanently delete a user and all their data

Self-protection rules

Admins cannot perform the following operations on their own account via the API or UI:

Delete their own account
Demote their own role to user
Suspend their own account

The API returns HTTP 400 if any of these are attempted.

`GET /api/admin/users` query parameters

Parameter	Default	Description
`search`	(none)	Case-insensitive substring match on name/email
`role`	(none)	Filter to `admin` or `user`
`page`	`1`	Page number (1-based)
`per_page`	`50`	Results per page (max 200)

`POST /api/admin/users` body

{
  "name":     "Full Name",
  "email":    "user@example.com",
  "dob":      "1990-01-01",
  "password": "securepassword",
  "role":     "user",
  "verified": true
}

verified defaults to true when creating via the admin API — an admin creating an account implies the address is trusted.

`PATCH /api/admin/users/<id>` body

All fields are optional. Send only what you want to change.

{
  "role":           "admin",
  "is_active":      false,
  "email_verified": true,
  "notes":          "Internal admin note — only visible to admins"
}

Price-Tracker Endpoints

Method	Endpoint	Auth	Description
GET	`/api/drives`	—	All drives with latest prices and computed stats
GET	`/api/drives/<id>`	—	Single drive: specs, all retailer prices, history
GET	`/api/drives/<id>/history`	—	Daily price history (`?days=30&retailer=amazon`)
GET	`/api/prices`	—	Flat list of all latest (drive, retailer) rows
GET	`/api/alerts`	—	Recent scraper alerts (`?limit=50`)
GET	`/api/alerts/unread`	—	Unacknowledged scraper alerts only
POST	`/api/alerts/<id>/ack`	—	Mark a scraper alert as acknowledged
GET	`/api/stats`	—	Dashboard summary stats
GET	`/api/scrape/status`	—	Last scrape run metadata
POST	`/api/scrape/trigger`	—	Start a background scrape (returns immediately)
GET	`/health`	—	Health check: `{ "status": "ok", "time": "..." }`

CLI — Backend User Management

auth/users.py doubles as a command-line tool for managing users directly against the database, without the API server running. Useful for initial setup, emergency access, and automation.

python -m auth.users <command> [arguments]

Commands

`list` — show all users

python -m auth.users list

Prints a table of all users with ID, role, verified status, active status, email, and name.

`create` — create a new user

python -m auth.users create <email> <password> [options]

Option	Default	Description
`--name`	derived from email	Full display name
`--dob`	`1990-01-01`	Date of birth (YYYY-MM-DD)
`--role`	`user`	Role: `user` or `admin`
`--unverified`	(flag, off)	Leave email_verified = false

Examples:

# Create a regular user (email pre-verified)
python -m auth.users create alice@example.com securepass123 --name "Alice Smith"

# Create an admin account
python -m auth.users create admin@example.com adminpass123 --name "Site Admin" --role admin

# Create unverified (user must verify via email)
python -m auth.users create bob@example.com bobpass123 --unverified

`promote` — grant admin role

python -m auth.users promote <email>

`demote` — revoke admin role

python -m auth.users demote <email>

Refused if the target is the last remaining admin account.

`suspend` — disable a user account

python -m auth.users suspend <email>

Sets is_active = 0 and immediately invalidates all active sessions.

`activate` — re-enable a suspended account

python -m auth.users activate <email>

`reset-password` — force-set a new password

python -m auth.users reset-password <email> <new_password>

Invalidates all active sessions for the user after the reset.

`delete` — permanently remove a user

python -m auth.users delete <email>

Requires interactive confirmation. Refused if the target is the last admin account. Cascades to sessions and tokens.

`info` — show full account details

python -m auth.users info <email>

Prints all non-sensitive fields for the account.

Admin Panel (`frontend/admin.html`)

Open frontend/admin.html in your browser. It has its own auth wall — non-admins who sign in are rejected immediately with an appropriate message.

Features

Stats row — live counts for total users, admins, verified accounts, pending verification, suspended accounts, and active sessions.

User table — paginated (25 per page), with real-time search by name or email and a role filter dropdown. Each row shows: ID, name and email, role badge, active/suspended status, verification status, join date, and last login.

Per-row actions:

Action	Description
Detail	Full profile modal with all fields and recent session list
Edit	Change role, toggle email verified, add/edit admin notes
Pwd	Force-reset password (all sessions invalidated)
Suspend / Activate	Toggle account access (suspending invalidates sessions immediately)
Delete	Permanent deletion with confirmation dialog

Create User modal — create an account directly with name, email, DOB, password, role, and verified checkbox. Bypasses registration rate limiting.

Detail modal — shows every account field plus a list of recent sessions with IP address, user agent, creation date, and active/expired status. Includes an "Invalidate All Sessions" button for forcing a user to log out everywhere.

Admin Panel link — visible only to admins in the user dropdown on the main dashboard (index.html).

Database Schema

All data lives in a single SQLite file at data/prices.db. Tables are created automatically on first run. New columns are added to existing databases via safe ALTER TABLE migrations on startup.

Price-tracker tables

Table	Purpose
`drives`	Drive catalog: brand, series, model number, MSRP, specs
`price_snapshots`	Every scraped price point: drive, retailer, price, stock, timestamp, status
`alerts`	Fired alert records: type, old/new price, % off MSRP, message
`scrape_runs`	Per-run metadata: start/end times, prices found, error counts

Auth tables

Table	Purpose
`users`	Accounts: name, email, DOB, hashed password, role, alert prefs, notes
`sessions`	Active login sessions: token, user ID, expiry, IP, user agent
`email_verify_tokens`	Single-use email verification tokens (24-hour expiry)
`password_reset_tokens`	Single-use password reset tokens (reserved for future use)
`reg_attempts`	Per-IP registration attempt log for rate limiting

Auth & Security Details

Password hashing

Passwords are hashed with PBKDF2-SHA256 at 260,000 iterations (NIST SP 800-63B recommendation) with a unique 32-byte random salt per user. The hash and salt are stored in separate columns; the plaintext password is never persisted.

The login handler always runs the full PBKDF2 computation even when the email address does not exist, so response timing cannot reveal whether an address is registered.

Session tokens

Session tokens are secrets.token_urlsafe(32) (256 bits of entropy), expire after 30 days, and are invalidated immediately on logout. Suspending a user or resetting their password invalidates all of their active sessions.

Email verification tokens

Single-use secrets.token_urlsafe(32) values that expire after 24 hours. Consuming a token writes used_at immediately, preventing reuse. Requesting a new verification email invalidates any outstanding token for that user.

Registration rate limiting

Registration is limited to 5 attempts per IP per 60-minute window, tracked in reg_attempts. Entries older than 2 hours are pruned automatically.

Anti-enumeration

POST /api/auth/resend-verify always returns HTTP 200 with the same body regardless of whether the address is registered.

Admin self-protection

The API prevents admins from deleting, demoting, or suspending their own account, and will refuse to delete the last remaining admin account entirely (via CLI or API).

Sensitive field stripping

_safe_user() removes pw_hash and pw_salt before any user object is returned from the API or CLI.

Email Templates

Event	Subject line
Account registered	`Verify your email — NAS Drive Tracker`
Email address verified	`You're in — NAS Drive Tracker`
Price drops below user threshold	`Price drop: <drive> — N% off MSRP`
Drive hits all-time price low	`New all-time low: <drive>`

Test SMTP without sending a real email:

# Connectivity check only
python3 -c "from auth.email import test_smtp_config; print(test_smtp_config())"

# Send a real test email to yourself
python -m auth.email test your@email.com

SMTP status is also shown live in the dashboard: sign in → user avatar → Alert Preferences → indicator at the bottom of the modal.

Dashboard Features

Auth wall

The dashboard is gated behind login. On first load an auth wall covers the page with three tabbed screens: Sign In, Register, and Resend Verification. The session token is stored in localStorage and restored automatically on return visits. auth.js must be in the same directory as index.html.

Email verification flow

If the URL contains ?token=<token> (linked from verification emails, or shown in the UI when SMTP is unconfigured), the page intercepts it before rendering the auth wall, posts the token to the API, and shows a result screen with a link back to sign-in.

Once logged in, a user avatar appears in the top-right header. The dropdown contains: Alert Preferences, Admin Panel (admins only), and Sign Out.

Per-user alert settings: email alerts on/off, minimum % off MSRP threshold (5–50%), brand filter, and a live SMTP connectivity indicator.

Price dashboard

Metric cards — best price today, drives at alert threshold, best $/TB, total prices tracked
Alert banner — appears when any drive breaches the active threshold
Price matrix table — sortable by any column; per-retailer prices with in-stock dots; all-time low, % off MSRP, $/TB
30-day price chart — all 4 retailers overlaid; switchable to 7/14/30/90-day views
Alerts feed — timestamped alert history with type badges; click to acknowledge
Detail drawer — full spec sheet, retailer comparison table, mini chart, direct buy link
Sidebar filters — brand, capacity, alert-only, in-stock-only, adjustable threshold slider
Live polling — auto-refreshes every 60 seconds; manual refresh button
Scrape trigger — run a fresh scrape from the sidebar without leaving the page

Scraper Architecture

Anti-bot measures

User-Agent rotation — 7 real browser UA strings, selected randomly per request
Per-domain rate limiting — randomised delays: Amazon 4–9s, Newegg 5–11s, Best Buy 6–12s, manufacturer sites 2–5s
Exponential back-off retry — on 429 and 5xx responses, up to 3 attempts with 2s base delay
Bot-wall detection — _is_bot_wall() checks responses for Cloudflare/CAPTCHA signals and returns blocked rather than silently failing
Price sanity check — prices outside $50–$2000 are discarded to prevent bot-wall text matching the price regex
Session reuse — one persistent requests.Session per domain
Optional proxy — set PROXY_URL

Parser strategies

Each retailer has 2–4 fallback strategies tried in order.

Amazon: twister-plus JSON block → corePriceDisplay / apex_desktop price block → JSON-LD → search page fallback

Newegg: direct product page (JSON-LD → price-current li → data-price) → search page fallback

Best Buy: direct SKU product page (JSON-LD → data-testid="customer-price") → search page (45s timeout) → priceView-customer-price → aria-label

Manufacturer sites: JSON-LD → itemprop="price" → .product-price → .price

Scrape run modes

python -m scraper.runner --seed       # Populate drives table from catalog (run once)
python -m scraper.runner --once       # Run a full scrape across all 12 drives
python -m scraper.runner --schedule   # Loop forever at SCRAPE_INTERVAL_HOURS

Alert types

Type	Trigger
`price_drop`	Price is ≥ `ALERT_DROP_PCT`% below MSRP or absolute drop ≥ `ALERT_DROP_ABS`
`new_low`	Price beats the all-time recorded minimum
`back_in_stock`	Drive transitions from out-of-stock to in-stock
`new_model`	A model not in the catalog is detected on a retailer page

Alerts deduplicate within 24 hours. On fire, the engine writes to alerts, emails matching subscribers, and emails ALERT_EMAIL_TO addresses.

Scraper Reliability Notes

When scrapers return blocked or not_found consistently:

Check logs/scraper.log for status codes and error messages
Set PROXY_URL to a residential proxy (Oxylabs, Bright Data)
Reduce SCRAPE_WORKERS to 1
For Amazon: consider the Product Advertising API

Query scraper health directly:

SELECT retailer, scrape_status, COUNT(*) AS n
FROM price_snapshots
WHERE scraped_at > DATE('now', '-1 day')
GROUP BY retailer, scrape_status
ORDER BY retailer, n DESC;

README.md Unescape Escape

NAS Drive Price Tracker