Massprice/CLAUDE.md

# MassPrice Scheduler

A Flask app that drives a real headless-Chromium session against
`massprice.tcgplayer.com` (TCGplayer's bulk pricing-rule tool) to click
"Run Rule" for each of the user's pricing rules on a schedule set through
this app's own web UI. Runs as one Docker container on the user's Unraid
home server, deployed the same way as the other projects on that box
(`arr-summary`, `card-grader`, `lunch-notifier`, `n64-tracker`).

## Why this exists

MassPrice has its own native per-rule scheduler (day-of-week + one
time-of-day), but it's capped at 15 active schedules account-wide and
only allows **one run per day per rule** — no multiple runs/day. Native
runs also showed real clock drift during discovery (6–22 minutes late
against their scheduled time). This tool **replaces** native scheduling
entirely: each rule's native schedule should be set to Inactive once this
tool is confirmed working, so there's exactly one thing triggering runs.

## Architecture

- **`app.py`** — Flask app, single shared-password session auth
  (`LOGIN_PASSWORD` env var, `hmac.compare_digest`) — this protects the
  *admin UI itself*, unrelated to the MassPrice/TCGplayer login below.
  Routes: `/` (Dashboard — rule cards with schedule editor), `/activity`
  (run history), `/session` (login-session import), and a small JSON API
  (`/api/rules/refresh`, `/api/rules/<id>/schedule`,
  `/api/rules/<id>/run`, `/api/session/import`) that the frontend JS
  calls without full page reloads.
- **`scheduler.py`** — an in-process APScheduler `BackgroundScheduler`
  holding one `CronTrigger` job per `(rule, schedule entry)` pair,
  rebuilt from `schedules.json` on startup and whenever the webui saves
  a change. Triggered jobs (and manual "Run Now" clicks) are pushed onto
  one `queue.Queue`, drained by a single dedicated worker thread — this
  guarantees only one Playwright browser session is ever active at a
  time, matching how a human would actually operate the site and
  avoiding concurrent-session weirdness.
  - **Must run in exactly one process.** `gunicorn.conf.py` pins
    `workers = 1` for this reason — a second worker would instantiate a
    second scheduler and duplicate every run. Fine for a low-traffic
    single-user tool.
  - Also schedules **one-off** `verify-<entry_id>` jobs (APScheduler
    `date` trigger, not recurring) `VERIFY_DELAY_MINUTES` after a run is
    confirmed sent — see `automation.verify_run()` below.
    `rebuild_jobs()` only ever removes `rule-*` jobs, deliberately, so
    editing a schedule mid-flight can't cancel a verification that's
    already in progress for an earlier run.
- **`automation.py`** — all Playwright logic. `list_rules()` scrapes
  `https://massprice.tcgplayer.com/Rule/List` for the live rule
  id/name pairs (cached to `rules_cache.json`, refreshed via the
  Dashboard's "Refresh Rules" button or on first boot). `run_rule(id)`
  finds that rule's `li.rule.widget-rule` row, clicks its
  `.dropdown-ellipsis` three-dot toggle, clicks the row's
  `a.run-rule[data-rule-id=...]` item — mirroring exactly what a human
  does in the UI — **then must click through a confirmation modal**
  (`#ConfirmationModal` → `#Confirmation_Run`; the same generic modal
  is reused for Delete Rule etc., so the confirm click is scoped to
  that specific modal instance). That confirmation step is not
  optional or cosmetic: confirmed by live testing that skipping it
  fires **zero** network requests and the rule silently never runs,
  even though the initial dropdown click "succeeds" with no error —
  an earlier version of this code did exactly that and every reported
  "success" was a false positive for a real stretch of production use.
  If a future site redesign changes this modal, watch for the same
  symptom: local history says success, MassPrice's own Activity Log
  never gets a matching entry. **No public API was found or used** —
  the "Run Rule" click handler is JS/AJAX-bound (no plain href, a
  delegated jQuery handler on `document.body`, not the element itself)
  and its request (`POST /Rule/Run`, confirmed by live network capture)
  is almost certainly anti-forgery-token protected, so automation
  drives the real page rather than replaying a raw HTTP POST.
- **Run verification** — confirming the modal only means MassPrice
  *accepted* the run; it still processes asynchronously on their side
  (queued → running → Complete), taking anywhere from ~1 to ~22 minutes
  in observed practice. `run_rule()` captures a "watermark" — the
  `data-activitylogid` of that rule's current top entry on
  `/Activity/List` (a stable, monotonically increasing ID; deliberately
  *not* parsing the displayed timestamp string, which would need
  fragile date/timezone handling) — **before** submitting, logs the run
  as `status: "sent"` with `verification: "pending"`, and the caller in
  `scheduler.py` schedules `verify_run()` to run `VERIFY_DELAY_MINUTES`
  later. `verify_run()` re-checks `/Activity/List`: a new entry (higher
  `data-activitylogid` than the watermark) whose status text contains
  "Complete" → `verification: "complete"`; otherwise
  `verification: "not_confirmed"` (which can mean "still processing",
  not necessarily "failed" — the delay is a reasonable middle ground,
  not a guarantee it's had enough time). `_find_latest_activity()` is
  shared by the watermark capture and the follow-up check. The Activity
  page's "Complete" column surfaces this; "Status" shows whether the
  click itself was sent, independent of whether MassPrice finished it.
- **Session persistence** — a Playwright `storage_state` (cookies +
  localStorage) is saved to `/data/auth_state.json` and reused across
  runs: scheduled runs never touch the login form at all, they just load
  this saved session. When it's missing or has expired (detected by an
  unauthenticated visit to `/Rule/List` silently redirecting to the
  public marketing homepage `/` — there's no `/login` URL to check
  against), the run fails with a clear "session expired or missing"
  error rather than attempting to log in — see below for why.
- **`storage.py`** — tiny JSON read/write helpers (temp-file +
  `os.replace`, so a crash mid-write can't corrupt state) for
  everything under `/data`: `schedules.json`, `auth_state.json`,
  `run_history.json` (bounded to the last 200 entries), `rules_cache.json`,
  `notify_state.json` (the session-expiry alert's send-once dedupe flag).
- **`templates/` / `static/`** — a small, plain dark dashboard. This is
  a deliberately different, simpler visual identity than Arr Summary's
  teletext system — that design language is specific to that app, not a
  house style to reuse here.

## Login — why it's manual, and how it works

MassPrice's login (`store.tcgplayer.com/oauth/login`, TCGplayer's shared
OAuth identity provider) is protected by **hCaptcha**. Two automated
approaches were tried and confirmed broken by live testing during
development — there is no selector fix for either, **fully automated
login is not achievable here**, full stop:
1. Filling in the real username/password and clicking Submit — the
   invisible captcha check silently never resolves for a headless/
   automated browser, and the form just refuses to submit. An earlier
   version of this app tried `MASSPRICE_USERNAME`/`MASSPRICE_PASSWORD`
   this way; removed once confirmed broken.
2. Driving a real human's clicks through a headless Chromium exposed via
   Chrome remote debugging (`chrome://inspect`) — also removed after
   testing. Still fails hCaptcha, because the browser itself (a
   `HeadlessChrome` user-agent, `navigator.webdriver = true`) is what
   gets flagged, independent of who's actually clicking. Deliberately
   not pursued further than that one test: disguising that browser to
   get past hCaptcha's bot detection would be captcha evasion, not a
   session-handling technique, regardless of how legitimate running
   automation on your own account is.

What actually works: log in with an ordinary, non-automated browser —
hCaptcha passes the way it always does for a real person, because
nothing about that browser looks automated — then copy that session's
cookies over. The **Session** page in the webui drives this:
1. The user logs into massprice.tcgplayer.com in their own regular
   Chrome and exports its cookies with a browser extension (Cookie-Editor,
   "Export as JSON").
2. They paste that JSON into the Session page and click "Import Session".
3. `automation.import_cookies()` parses it, converts each cookie from
   Cookie-Editor's shape into Playwright's `storage_state` cookie shape
   (`_convert_cookie()` — mainly the `sameSite` enum values and the
   `expirationDate` → `expires` field rename), and — critically —
   **verifies it actually authenticates** (a real headless request to
   `/Rule/List`, checked with the same `_is_logged_out()` used
   everywhere else) before saving it to `/data/auth_state.json`. A bad
   paste never overwrites a working session.

This session is a real, live bearer credential for the user's MassPrice
account for as long as it's valid — same trust model as `LOGIN_PASSWORD`/
`SECRET_KEY` in `.env` (server-only, gitignored, never in git), not
something to treat as inert config. It isn't permanent: every successful
automated run re-saves it (`_save_state()`), which tends to slide a
"remember me"-style cookie's expiration forward, but it will eventually
expire (inactivity, a password change, TCGplayer forcing re-auth) and
need re-importing.

The Dashboard shows a red banner (`_session_warning()` in `app.py`) when
the most recent run/refresh failed specifically because of a missing/
expired session, so this doesn't just look like "no rules found."

## Session-expiry email alert

**`notify.py`** — a thin Gmail-SMTP-with-app-password sender, the same
pattern `lunch-notifier` already uses on this box (reused deliberately
rather than adding a second notification channel). `GMAIL_USER`/
`GMAIL_APP_PASSWORD` (a Gmail **app password**, not the real account
password — Gmail blocks plain SMTP auth otherwise) and optional
`ALERT_EMAIL` (defaults to `GMAIL_USER`). `send_alert()` is a best-effort
no-op — returns `False` and just logs — if those env vars aren't set or
the send fails, since a broken mail config shouldn't break automation.

Fires exactly **once** per expiry, not on every failed scheduled run
while it's down: `automation._notify_session_expired_once()` is called
from `_ensure_authenticated()` right before it raises, checks/sets a
`session_expired_notified` flag in `notify_state.json`, and only sends
if that flag isn't already set. The flag resets to `False` the moment
`import_cookies()` successfully saves a fresh session, so the next
expiry gets its own alert. The flag is only set `True` after a
confirmed-sent email — if mail is misconfigured, every failing run keeps
retrying the send rather than silently giving up after the first one.

## Local development

```bash
pip install -r requirements.txt
playwright install chromium   # only needed if testing automation.py directly
python3 app.py
```

Runs on port 5000 (`PORT` env to change). Without Chromium installed and
without a saved `auth_state.json`, the Dashboard still boots —
`_bootstrap()` in `app.py` best-effort-catches the initial rule sync, so
local UI/schedule-editing work doesn't require Docker or a real MassPrice
session at all. `DATA_DIR` env var overrides where JSON state is
read/written (defaults to `/data`, which won't exist locally — point it
at a scratch directory).

**Every commit should be matched with a version bump** —
`APP_VERSION`/`APP_VERSION_DATE` near the top of `app.py` — the same
"does the footer prove this deploy landed" signal used on Arr Summary.

## Deployment

See the `deploy-massprice` skill for the Forgejo push-to-deploy
pipeline and manual container restart steps.

**Never put real credentials in the repo.** `.env` and `data/` are
gitignored and excluded from the deploy rsync precisely so
`LOGIN_PASSWORD`/`SECRET_KEY` and the saved
MassPrice session (`data/auth_state.json`) have somewhere to live
outside git.