Massprice/CLAUDE.md

MassPrice Scheduler

A Flask app that drives a real headless-Chromium session against massprice.tcgplayer.com (TCGplayer's bulk pricing-rule tool) to click "Run Rule" for each of the user's pricing rules on a schedule set through this app's own web UI. Runs as one Docker container on the user's Unraid home server, deployed the same way as the other projects on that box (arr-summary, card-grader, lunch-notifier, n64-tracker).

Why this exists

MassPrice has its own native per-rule scheduler (day-of-week + one time-of-day), but it's capped at 15 active schedules account-wide and only allows one run per day per rule — no multiple runs/day. Native runs also showed real clock drift during discovery (6–22 minutes late against their scheduled time). This tool replaces native scheduling entirely: each rule's native schedule should be set to Inactive once this tool is confirmed working, so there's exactly one thing triggering runs.

Architecture

• app.py — Flask app, single shared-password session auth (LOGIN_PASSWORD env var, hmac.compare_digest) — this protects the admin UI itself, unrelated to the MassPrice/TCGplayer login below. Routes: / (Dashboard — rule cards with schedule editor), /activity (run history), /session (login-session import), and a small JSON API (/api/rules/refresh, /api/rules/<id>/schedule, /api/rules/<id>/run, /api/session/import) that the frontend JS calls without full page reloads.
• scheduler.py — an in-process APScheduler BackgroundScheduler holding one CronTrigger job per (rule, schedule entry) pair, rebuilt from schedules.json on startup and whenever the webui saves a change. Triggered jobs (and manual "Run Now" clicks) are pushed onto one queue.Queue, drained by a single dedicated worker thread — this guarantees only one Playwright browser session is ever active at a time, matching how a human would actually operate the site and avoiding concurrent-session weirdness.
  • Must run in exactly one process. gunicorn.conf.py pins workers = 1 for this reason — a second worker would instantiate a second scheduler and duplicate every run. Fine for a low-traffic single-user tool.
  • Also schedules one-off verify-<entry_id> jobs (APScheduler date trigger, not recurring) VERIFY_DELAY_MINUTES after a run is confirmed sent — see automation.verify_run() below. rebuild_jobs() only ever removes rule-* jobs, deliberately, so editing a schedule mid-flight can't cancel a verification that's already in progress for an earlier run.
• automation.py — all Playwright logic. list_rules() scrapes https://massprice.tcgplayer.com/Rule/List for the live rule id/name pairs (cached to rules_cache.json, refreshed via the Dashboard's "Refresh Rules" button or on first boot). run_rule(id) finds that rule's li.rule.widget-rule row, clicks its .dropdown-ellipsis three-dot toggle, clicks the row's a.run-rule[data-rule-id=...] item — mirroring exactly what a human does in the UI — then must click through a confirmation modal (#ConfirmationModal → #Confirmation_Run; the same generic modal is reused for Delete Rule etc., so the confirm click is scoped to that specific modal instance). That confirmation step is not optional or cosmetic: confirmed by live testing that skipping it fires zero network requests and the rule silently never runs, even though the initial dropdown click "succeeds" with no error — an earlier version of this code did exactly that and every reported "success" was a false positive for a real stretch of production use. If a future site redesign changes this modal, watch for the same symptom: local history says success, MassPrice's own Activity Log never gets a matching entry. No public API was found or used — the "Run Rule" click handler is JS/AJAX-bound (no plain href, a delegated jQuery handler on document.body, not the element itself) and its request (POST /Rule/Run, confirmed by live network capture) is almost certainly anti-forgery-token protected, so automation drives the real page rather than replaying a raw HTTP POST.
• Run verification — confirming the modal only means MassPrice accepted the run; it still processes asynchronously on their side (queued → running → Complete), taking anywhere from ~1 to ~22 minutes in observed practice. run_rule() captures a "watermark" — the data-activitylogid of that rule's current top entry on /Activity/List (a stable, monotonically increasing ID; deliberately not parsing the displayed timestamp string, which would need fragile date/timezone handling) — before submitting, logs the run as status: "sent" with verification: "pending", and the caller in scheduler.py schedules verify_run() to run VERIFY_DELAY_MINUTES later. verify_run() re-checks /Activity/List: a new entry (higher data-activitylogid than the watermark) whose status text contains "Complete" → verification: "complete"; otherwise verification: "not_confirmed" (which can mean "still processing", not necessarily "failed" — the delay is a reasonable middle ground, not a guarantee it's had enough time). _find_latest_activity() is shared by the watermark capture and the follow-up check. The Activity page's "Complete" column surfaces this; "Status" shows whether the click itself was sent, independent of whether MassPrice finished it.
• Session persistence — a Playwright storage_state (cookies + localStorage) is saved to /data/auth_state.json and reused across runs: scheduled runs never touch the login form at all, they just load this saved session. When it's missing or has expired (detected by an unauthenticated visit to /Rule/List silently redirecting to the public marketing homepage / — there's no /login URL to check against), the run fails with a clear "session expired or missing" error rather than attempting to log in — see below for why.
• storage.py — tiny JSON read/write helpers (temp-file + os.replace, so a crash mid-write can't corrupt state) for everything under /data: schedules.json, auth_state.json, run_history.json (bounded to the last 200 entries), rules_cache.json, notify_state.json (the session-expiry alert's send-once dedupe flag).
• templates/ / static/ — a small, plain dark dashboard. This is a deliberately different, simpler visual identity than Arr Summary's teletext system — that design language is specific to that app, not a house style to reuse here.

Login — why it's manual, and how it works

MassPrice's login (store.tcgplayer.com/oauth/login, TCGplayer's shared OAuth identity provider) is protected by hCaptcha. Two automated approaches were tried and confirmed broken by live testing during development — there is no selector fix for either, fully automated login is not achievable here, full stop:

1. Filling in the real username/password and clicking Submit — the invisible captcha check silently never resolves for a headless/ automated browser, and the form just refuses to submit. An earlier version of this app tried MASSPRICE_USERNAME/MASSPRICE_PASSWORD this way; removed once confirmed broken.
2. Driving a real human's clicks through a headless Chromium exposed via Chrome remote debugging (chrome://inspect) — also removed after testing. Still fails hCaptcha, because the browser itself (a HeadlessChrome user-agent, navigator.webdriver = true) is what gets flagged, independent of who's actually clicking. Deliberately not pursued further than that one test: disguising that browser to get past hCaptcha's bot detection would be captcha evasion, not a session-handling technique, regardless of how legitimate running automation on your own account is.

What actually works: log in with an ordinary, non-automated browser — hCaptcha passes the way it always does for a real person, because nothing about that browser looks automated — then copy that session's cookies over. The Session page in the webui drives this:

1. The user logs into massprice.tcgplayer.com in their own regular Chrome and exports its cookies with a browser extension (Cookie-Editor, "Export as JSON").
2. They paste that JSON into the Session page and click "Import Session".
3. automation.import_cookies() parses it, converts each cookie from Cookie-Editor's shape into Playwright's storage_state cookie shape (_convert_cookie() — mainly the sameSite enum values and the expirationDate → expires field rename), and — critically — verifies it actually authenticates (a real headless request to /Rule/List, checked with the same _is_logged_out() used everywhere else) before saving it to /data/auth_state.json. A bad paste never overwrites a working session.

This session is a real, live bearer credential for the user's MassPrice account for as long as it's valid — same trust model as LOGIN_PASSWORD/ SECRET_KEY in .env (server-only, gitignored, never in git), not something to treat as inert config. It isn't permanent: every successful automated run re-saves it (_save_state()), which tends to slide a "remember me"-style cookie's expiration forward, but it will eventually expire (inactivity, a password change, TCGplayer forcing re-auth) and need re-importing.

The Dashboard shows a red banner (_session_warning() in app.py) when the most recent run/refresh failed specifically because of a missing/ expired session, so this doesn't just look like "no rules found."

Session-expiry email alert

notify.py — a thin Gmail-SMTP-with-app-password sender, the same pattern lunch-notifier already uses on this box (reused deliberately rather than adding a second notification channel). GMAIL_USER/ GMAIL_APP_PASSWORD (a Gmail app password, not the real account password — Gmail blocks plain SMTP auth otherwise) and optional ALERT_EMAIL (defaults to GMAIL_USER). send_alert() is a best-effort no-op — returns False and just logs — if those env vars aren't set or the send fails, since a broken mail config shouldn't break automation.

Fires exactly once per expiry, not on every failed scheduled run while it's down: automation._notify_session_expired_once() is called from _ensure_authenticated() right before it raises, checks/sets a session_expired_notified flag in notify_state.json, and only sends if that flag isn't already set. The flag resets to False the moment import_cookies() successfully saves a fresh session, so the next expiry gets its own alert. The flag is only set True after a confirmed-sent email — if mail is misconfigured, every failing run keeps retrying the send rather than silently giving up after the first one.

Local development

pip install -r requirements.txt
playwright install chromium   # only needed if testing automation.py directly
python3 app.py

Runs on port 5000 (PORT env to change). Without Chromium installed and without a saved auth_state.json, the Dashboard still boots — _bootstrap() in app.py best-effort-catches the initial rule sync, so local UI/schedule-editing work doesn't require Docker or a real MassPrice session at all. DATA_DIR env var overrides where JSON state is read/written (defaults to /data, which won't exist locally — point it at a scratch directory).

Every commit should be matched with a version bump — APP_VERSION/APP_VERSION_DATE near the top of app.py — the same "does the footer prove this deploy landed" signal used on Arr Summary.

Deployment

See the deploy-massprice skill for the Forgejo push-to-deploy pipeline and manual container restart steps.

Never put real credentials in the repo. .env and data/ are gitignored and excluded from the deploy rsync precisely so LOGIN_PASSWORD/SECRET_KEY and the saved MassPrice session (data/auth_state.json) have somewhere to live outside git.