Massprice/CLAUDE.md

# MassPrice Scheduler

A Flask app that drives a real headless-Chromium session against
`massprice.tcgplayer.com` (TCGplayer's bulk pricing-rule tool) to click
"Run Rule" for each of the user's pricing rules on a schedule set through
this app's own web UI. Runs as one Docker container on the user's Unraid
home server, deployed the same way as the other projects on that box
(`arr-summary`, `card-grader`, `lunch-notifier`, `n64-tracker`).

## Why this exists

MassPrice has its own native per-rule scheduler (day-of-week + one
time-of-day), but it's capped at 15 active schedules account-wide and
only allows **one run per day per rule** — no multiple runs/day. Native
runs also showed real clock drift during discovery (6–22 minutes late
against their scheduled time). This tool **replaces** native scheduling
entirely: each rule's native schedule should be set to Inactive once this
tool is confirmed working, so there's exactly one thing triggering runs.

## Architecture

- **`app.py`** — Flask app, single shared-password session auth
  (`LOGIN_PASSWORD` env var, `hmac.compare_digest`) — this protects the
  *admin UI itself*, unrelated to the MassPrice/TCGplayer login below.
  Routes: `/` (Dashboard — rule cards with schedule editor), `/activity`
  (run history), `/session` (login-session import), and a small JSON API
  (`/api/rules/refresh`, `/api/rules/<id>/schedule`,
  `/api/rules/<id>/run`, `/api/session/import`) that the frontend JS
  calls without full page reloads.
- **`scheduler.py`** — an in-process APScheduler `BackgroundScheduler`
  holding one `CronTrigger` job per `(rule, schedule entry)` pair,
  rebuilt from `schedules.json` on startup and whenever the webui saves
  a change. Triggered jobs (and manual "Run Now" clicks) are pushed onto
  one `queue.Queue`, drained by a single dedicated worker thread — this
  guarantees only one Playwright browser session is ever active at a
  time, matching how a human would actually operate the site and
  avoiding concurrent-session weirdness.
  - **Must run in exactly one process.** `gunicorn.conf.py` pins
    `workers = 1` for this reason — a second worker would instantiate a
    second scheduler and duplicate every run. Fine for a low-traffic
    single-user tool.
  - Also drives the **completion poll loop**: after a run is confirmed
    sent, a one-off `verify-<entry_id>-<attempt>` job (APScheduler
    `date` trigger, not recurring) fires `VERIFY_POLL_MINUTES` later and
    calls `automation.check_completion()`. Not complete yet? Another
    one-off job reschedules the next check the same interval later —
    `_run_verification_check()` in `scheduler.py` owns this loop — up to
    `VERIFY_MAX_ATTEMPTS` times, after which it's marked
    `"not_confirmed"` for good. `rebuild_jobs()` only ever removes
    `rule-*` jobs, deliberately, so editing a schedule mid-flight can't
    cancel a poll loop that's already in progress for an earlier run.
    These one-off jobs live only in the scheduler's in-memory jobstore
    (no persistent jobstore is configured) — a container restart mid-poll
    silently drops that one entry's remaining checks, leaving it stuck
    on whatever state it was last in (usually "pending"). Low-impact
    (an hour-long window, restarts are infrequent) but worth knowing if
    an entry is ever seen stuck on "Pending" long after the fact.
- **`automation.py`** — all Playwright logic. `list_rules()` scrapes
  `https://massprice.tcgplayer.com/Rule/List` for the live rule
  id/name pairs (cached to `rules_cache.json`, refreshed via the
  Dashboard's "Refresh Rules" button or on first boot). `run_rule(id)`
  finds that rule's `li.rule.widget-rule` row, clicks its
  `.dropdown-ellipsis` three-dot toggle, clicks the row's
  `a.run-rule[data-rule-id=...]` item — mirroring exactly what a human
  does in the UI — **then must click through a confirmation modal**
  (`#ConfirmationModal` → `#Confirmation_Run`; the same generic modal
  is reused for Delete Rule etc., so the confirm click is scoped to
  that specific modal instance). That confirmation step is not
  optional or cosmetic: confirmed by live testing that skipping it
  fires **zero** network requests and the rule silently never runs,
  even though the initial dropdown click "succeeds" with no error —
  an earlier version of this code did exactly that and every reported
  "success" was a false positive for a real stretch of production use.
  If a future site redesign changes this modal, watch for the same
  symptom: local history says success, MassPrice's own Activity Log
  never gets a matching entry. **No public API was found or used** —
  the "Run Rule" click handler is JS/AJAX-bound (no plain href, a
  delegated jQuery handler on `document.body`, not the element itself)
  and its request (`POST /Rule/Run`, confirmed by live network capture)
  is almost certainly anti-forgery-token protected, so automation
  drives the real page rather than replaying a raw HTTP POST.
- **Run verification** — confirming the modal only means MassPrice
  *accepted* the run; it still processes asynchronously on their side
  (queued → running → Complete), taking anywhere from ~1 minute to over
  20 in observed practice, and its queue can back up further under
  load (observed directly: a run still sitting in "Queued" 15+ minutes
  in). `run_rule()` captures a "watermark" — the `data-activitylogid` of
  that rule's current top entry on `/Activity/List` (a stable,
  monotonically increasing ID; deliberately *not* parsing the displayed
  timestamp string, which would need fragile date/timezone handling) —
  **before** submitting, logs the run as `status: "sent"` with
  `verification: "pending"`, and the caller in `scheduler.py` starts a
  poll loop (`automation.check_completion()`, every
  `VERIFY_POLL_MINUTES`, up to `VERIFY_MAX_ATTEMPTS` times — see
  `scheduler.py`'s own section above for the mechanics). Each check
  re-scans `/Activity/List`: a new entry (higher `data-activitylogid`
  than the watermark) whose status text contains "Complete" →
  `verification: "complete"`, loop stops; otherwise the loop keeps
  `verification: "pending"` and just updates `verify_message` with the
  latest note, until attempts run out and it's marked
  `verification: "not_confirmed"` for good — which can mean "still
  processing somewhere past our patience", not necessarily "failed".
  Also worth knowing: a still-"Queued" MassPrice entry has no
  `data-activitylogid` yet (no "View" link exists until it finishes),
  so `_find_latest_activity()` — shared by the watermark capture and
  every poll — correctly falls through to the *previous* completed
  entry rather than matching the in-flight one; that's why "no ID
  higher than the watermark" is the right "not done yet" signal, not a
  bug. The Activity page's "Complete" column surfaces all of this;
  "Status" shows whether the click itself was sent, independent of
  whether MassPrice finished it.
- **Session persistence** — a Playwright `storage_state` (cookies +
  localStorage) is saved to `/data/auth_state.json` and reused across
  runs: scheduled runs never touch the login form at all, they just load
  this saved session. When it's missing or has expired (detected by an
  unauthenticated visit to `/Rule/List` silently redirecting to the
  public marketing homepage `/` — there's no `/login` URL to check
  against), the run fails with a clear "session expired or missing"
  error rather than attempting to log in — see below for why.
- **`storage.py`** — tiny JSON read/write helpers (temp-file +
  `os.replace`, so a crash mid-write can't corrupt state) for
  everything under `/data`: `schedules.json`, `auth_state.json`,
  `run_history.json` (bounded to the last 200 entries), `rules_cache.json`,
  `notify_state.json` (the session-expiry alert's send-once dedupe flag).
- **`templates/` / `static/`** — a small, plain dark dashboard. This is
  a deliberately different, simpler visual identity than Arr Summary's
  teletext system — that design language is specific to that app, not a
  house style to reuse here.

## Login — why it's manual, and how it works

MassPrice's login (`store.tcgplayer.com/oauth/login`, TCGplayer's shared
OAuth identity provider) is protected by **hCaptcha**. Two automated
approaches were tried and confirmed broken by live testing during
development — there is no selector fix for either, **fully automated
login is not achievable here**, full stop:
1. Filling in the real username/password and clicking Submit — the
   invisible captcha check silently never resolves for a headless/
   automated browser, and the form just refuses to submit. An earlier
   version of this app tried `MASSPRICE_USERNAME`/`MASSPRICE_PASSWORD`
   this way; removed once confirmed broken.
2. Driving a real human's clicks through a headless Chromium exposed via
   Chrome remote debugging (`chrome://inspect`) — also removed after
   testing. Still fails hCaptcha, because the browser itself (a
   `HeadlessChrome` user-agent, `navigator.webdriver = true`) is what
   gets flagged, independent of who's actually clicking. Deliberately
   not pursued further than that one test: disguising that browser to
   get past hCaptcha's bot detection would be captcha evasion, not a
   session-handling technique, regardless of how legitimate running
   automation on your own account is.

What actually works: log in with an ordinary, non-automated browser —
hCaptcha passes the way it always does for a real person, because
nothing about that browser looks automated — then copy that session's
cookies over. The **Session** page in the webui drives this:
1. The user logs into massprice.tcgplayer.com in their own regular
   Chrome and exports its cookies with a browser extension (Cookie-Editor,
   "Export as JSON").
2. They paste that JSON into the Session page and click "Import Session".
3. `automation.import_cookies()` parses it, converts each cookie from
   Cookie-Editor's shape into Playwright's `storage_state` cookie shape
   (`_convert_cookie()` — mainly the `sameSite` enum values and the
   `expirationDate` → `expires` field rename), and — critically —
   **verifies it actually authenticates** (a real headless request to
   `/Rule/List`, checked with the same `_is_logged_out()` used
   everywhere else) before saving it to `/data/auth_state.json`. A bad
   paste never overwrites a working session.

This session is a real, live bearer credential for the user's MassPrice
account for as long as it's valid — same trust model as `LOGIN_PASSWORD`/
`SECRET_KEY` in `.env` (server-only, gitignored, never in git), not
something to treat as inert config. It isn't permanent: every successful
automated run re-saves it (`_save_state()`), which tends to slide a
"remember me"-style cookie's expiration forward, but it will eventually
expire (inactivity, a password change, TCGplayer forcing re-auth) and
need re-importing.

The Dashboard shows a red banner (`_session_warning()` in `app.py`) when
the most recent run/refresh failed specifically because of a missing/
expired session, so this doesn't just look like "no rules found."

## Session-expiry email alert

**`notify.py`** — a thin Gmail-SMTP-with-app-password sender, the same
pattern `lunch-notifier` already uses on this box (reused deliberately
rather than adding a second notification channel). `GMAIL_USER`/
`GMAIL_APP_PASSWORD` (a Gmail **app password**, not the real account
password — Gmail blocks plain SMTP auth otherwise) and optional
`ALERT_EMAIL` (defaults to `GMAIL_USER`). `send_alert()` is a best-effort
no-op — returns `False` and just logs — if those env vars aren't set or
the send fails, since a broken mail config shouldn't break automation.

Fires exactly **once** per expiry, not on every failed scheduled run
while it's down: `automation._notify_session_expired_once()` is called
from `_ensure_authenticated()` right before it raises, checks/sets a
`session_expired_notified` flag in `notify_state.json`, and only sends
if that flag isn't already set. The flag resets to `False` the moment
`import_cookies()` successfully saves a fresh session, so the next
expiry gets its own alert. The flag is only set `True` after a
confirmed-sent email — if mail is misconfigured, every failing run keeps
retrying the send rather than silently giving up after the first one.

## Local development

```bash
pip install -r requirements.txt
playwright install chromium   # only needed if testing automation.py directly
python3 app.py
```

Runs on port 5000 (`PORT` env to change). Without Chromium installed and
without a saved `auth_state.json`, the Dashboard still boots —
`_bootstrap()` in `app.py` best-effort-catches the initial rule sync, so
local UI/schedule-editing work doesn't require Docker or a real MassPrice
session at all. `DATA_DIR` env var overrides where JSON state is
read/written (defaults to `/data`, which won't exist locally — point it
at a scratch directory).

**Every commit should be matched with a version bump** —
`APP_VERSION`/`APP_VERSION_DATE` near the top of `app.py` — the same
"does the footer prove this deploy landed" signal used on Arr Summary.

## Deployment

See the `deploy-massprice` skill for the Forgejo push-to-deploy
pipeline and manual container restart steps.

**Never put real credentials in the repo.** `.env` and `data/` are
gitignored and excluded from the deploy rsync precisely so
`LOGIN_PASSWORD`/`SECRET_KEY` and the saved
MassPrice session (`data/auth_state.json`) have somewhere to live
outside git.