Massprice/CLAUDE.md

MassPrice Scheduler

A Flask app that drives a real headless-Chromium session against massprice.tcgplayer.com (TCGplayer's bulk pricing-rule tool) to click "Run Rule" for each of the user's pricing rules on a schedule set through this app's own web UI. Runs as one Docker container on the user's Unraid home server, deployed the same way as the other projects on that box (arr-summary, card-grader, lunch-notifier, n64-tracker).

Why this exists

MassPrice has its own native per-rule scheduler (day-of-week + one time-of-day), but it's capped at 15 active schedules account-wide and only allows one run per day per rule — no multiple runs/day. Native runs also showed real clock drift during discovery (6–22 minutes late against their scheduled time). This tool replaces native scheduling entirely: each rule's native schedule should be set to Inactive once this tool is confirmed working, so there's exactly one thing triggering runs.

Architecture

• app.py — Flask app, single shared-password session auth (LOGIN_PASSWORD env var, hmac.compare_digest) — this protects the admin UI itself, unrelated to the MassPrice/TCGplayer login below. Routes: / (Dashboard — rule cards with schedule editor), /activity (run history), /session (login-session import), and a small JSON API (/api/rules/refresh, /api/rules/<id>/schedule, /api/rules/<id>/run, /api/session/import) that the frontend JS calls without full page reloads.
• scheduler.py — an in-process APScheduler BackgroundScheduler holding one CronTrigger job per (rule, schedule entry) pair, rebuilt from schedules.json on startup and whenever the webui saves a change. Every Playwright session goes through submit() onto one queue.Queue, drained by a single dedicated worker thread — scheduled runs, "Run Now", "Refresh Rules" and cookie imports alike. That guarantees only one browser is ever active at a time, matching how a human would actually operate the site and avoiding concurrent-session weirdness. Nothing may call automation.py directly from a Flask request thread: with threads = 4 that opens a second Chromium alongside a scheduled run, and both write auth_state.json — which Playwright writes itself, non-atomically — so a torn write reads back as "session expired" and costs a pointless cookie re-import.
  • Request handlers that need an answer (refresh, import) use run_browser_task(), which waits REQUEST_WAIT_SECONDS (90) and then raises BrowserBusy → HTTP 503, without cancelling the work. The bound is the point: gunicorn kills any worker whose request exceeds its timeout (120s), and since the jobstore and queue live in that worker's memory, being killed drops every pending verification poll. Waiting behind a slow rule run must never be able to take the scheduler down.
  • Must run in exactly one process. gunicorn.conf.py pins workers = 1 for this reason — a second worker would instantiate a second scheduler and duplicate every run. Fine for a low-traffic single-user tool.
  • Also drives the completion poll loop: after a run is confirmed sent, a one-off verify-<entry_id>-<attempt> job (APScheduler date trigger, not recurring) fires VERIFY_POLL_MINUTES later and calls automation.check_completion(). Not complete yet? Another one-off job reschedules the next check the same interval later — _run_verification_check() in scheduler.py owns this loop — up to VERIFY_MAX_ATTEMPTS times, after which it's marked "not_confirmed" for good. rebuild_jobs() only ever removes rule-* jobs, deliberately, so editing a schedule mid-flight can't cancel a poll loop that's already in progress for an earlier run. These one-off jobs live only in the scheduler's in-memory jobstore (no persistent jobstore is configured) — a container restart mid-poll silently drops that one entry's remaining checks, leaving it stuck on whatever state it was last in (usually "pending"). Low-impact (an hour-long window, restarts are infrequent) but worth knowing if an entry is ever seen stuck on "Pending" long after the fact.
• automation.py — all Playwright logic. list_rules() scrapes https://massprice.tcgplayer.com/Rule/List for the live rule id/name pairs (cached to rules_cache.json, refreshed via the Dashboard's "Refresh Rules" button or on first boot). A refresh also runs _reconcile_schedules() in app.py, which re-syncs schedules.json against that live list: rule ids are stable but names aren't, and a stale cached name silently breaks run verification (it's what the Activity-page match keys on), while a rule deleted upstream would otherwise keep firing cron jobs with no card left in the UI to disable them. Deliberately skipped when the scrape returns an empty list, so a MassPrice redesign that breaks the scrape can't delete every saved schedule. run_rule(id) finds that rule's li.rule.widget-rule row, clicks its .dropdown-ellipsis three-dot toggle, clicks the row's a.run-rule[data-rule-id=...] item — mirroring exactly what a human does in the UI — then must click through a confirmation modal (#ConfirmationModal → #Confirmation_Run; the same generic modal is reused for Delete Rule etc., so the confirm click is scoped to that specific modal instance). That confirmation step is not optional or cosmetic: confirmed by live testing that skipping it fires zero network requests and the rule silently never runs, even though the initial dropdown click "succeeds" with no error — an earlier version of this code did exactly that and every reported "success" was a false positive for a real stretch of production use. If a future site redesign changes this modal, watch for the same symptom: local history says success, MassPrice's own Activity Log never gets a matching entry. No public API was found or used — the "Run Rule" click handler is JS/AJAX-bound (no plain href, a delegated jQuery handler on document.body, not the element itself) and its request (POST /Rule/Run, confirmed by live network capture) is almost certainly anti-forgery-token protected, so automation drives the real page rather than replaying a raw HTTP POST.
• Run verification — confirming the modal only means MassPrice accepted the run; it still processes asynchronously on their side (queued → running → Complete), taking anywhere from ~1 minute to over 20 in observed practice, and its queue can back up further under load (observed directly: a run still sitting in "Queued" 15+ minutes in). run_rule() captures a "watermark" — the data-activitylogid of that rule's current top entry on /Activity/List (a stable, monotonically increasing ID; deliberately not parsing the displayed timestamp string, which would need fragile date/timezone handling) — before submitting, logs the run as status: "sent" with verification: "pending", and the caller in scheduler.py starts a poll loop (automation.check_completion(), every VERIFY_POLL_MINUTES, up to VERIFY_MAX_ATTEMPTS times — see scheduler.py's own section above for the mechanics). Each check re-scans /Activity/List: a new entry (higher data-activitylogid than the watermark) whose status text contains "Complete" → verification: "complete", loop stops; otherwise the loop keeps verification: "pending" and just updates verify_message with the latest note, until attempts run out and it's marked verification: "not_confirmed" for good — which can mean "still processing somewhere past our patience", not necessarily "failed". Also worth knowing: a still-"Queued" MassPrice entry has no data-activitylogid yet (no "View" link exists until it finishes), so _find_latest_activity() — shared by the watermark capture and every poll — correctly falls through to the previous completed entry rather than matching the in-flight one; that's why "no ID higher than the watermark" is the right "not done yet" signal, not a bug. The Activity page's "Complete" column surfaces all of this; "Status" shows whether the click itself was sent, independent of whether MassPrice finished it.
• Session persistence — a Playwright storage_state (cookies + localStorage) is saved to /data/auth_state.json and reused across runs: scheduled runs never touch the login form at all, they just load this saved session. When it's missing or has expired (detected by an unauthenticated visit to /Rule/List silently redirecting to the public marketing homepage / — there's no /login URL to check against), the run fails with a clear "session expired or missing" error rather than attempting to log in — see below for why.
• storage.py — tiny JSON read/write helpers (temp-file + os.replace, so a crash mid-write can't corrupt state) for the state files under /data: schedules.json, run_history.json (bounded to the last 200 entries), rules_cache.json, session_state.json (the "is the saved session known-bad" flag behind the dashboard banner) and notify_state.json (the session-expiry alert's send-once dedupe flag). auth_state.json lives in the same directory but is not written through here — Playwright writes it itself via context.storage_state(path=...) in automation._save_state(), so it gets none of that atomic-write protection. Worth knowing: it's also the one file two concurrent Playwright sessions can both write.
• templates/ / static/ — a small, plain dark dashboard. This is a deliberately different, simpler visual identity than Arr Summary's teletext system — that design language is specific to that app, not a house style to reuse here.

Login — why it's manual, and how it works

MassPrice's login (store.tcgplayer.com/oauth/login, TCGplayer's shared OAuth identity provider) is protected by hCaptcha. Two automated approaches were tried and confirmed broken by live testing during development — there is no selector fix for either, fully automated login is not achievable here, full stop:

1. Filling in the real username/password and clicking Submit — the invisible captcha check silently never resolves for a headless/ automated browser, and the form just refuses to submit. An earlier version of this app tried MASSPRICE_USERNAME/MASSPRICE_PASSWORD this way; removed once confirmed broken.
2. Driving a real human's clicks through a headless Chromium exposed via Chrome remote debugging (chrome://inspect) — also removed after testing. Still fails hCaptcha, because the browser itself (a HeadlessChrome user-agent, navigator.webdriver = true) is what gets flagged, independent of who's actually clicking. Deliberately not pursued further than that one test: disguising that browser to get past hCaptcha's bot detection would be captcha evasion, not a session-handling technique, regardless of how legitimate running automation on your own account is.

What actually works: log in with an ordinary, non-automated browser — hCaptcha passes the way it always does for a real person, because nothing about that browser looks automated — then copy that session's cookies over. The Session page in the webui drives this:

1. The user logs into massprice.tcgplayer.com in their own regular Chrome and exports its cookies with a browser extension (Cookie-Editor, "Export as JSON").
2. They paste that JSON into the Session page and click "Import Session".
3. automation.import_cookies() parses it, converts each cookie from Cookie-Editor's shape into Playwright's storage_state cookie shape (_convert_cookie() — mainly the sameSite enum values and the expirationDate → expires field rename), and — critically — verifies it actually authenticates (a real headless request to /Rule/List, checked with the same _is_logged_out() used everywhere else) before saving it to /data/auth_state.json. A bad paste never overwrites a working session.

This session is a real, live bearer credential for the user's MassPrice account for as long as it's valid — same trust model as LOGIN_PASSWORD/ SECRET_KEY in .env (server-only, gitignored, never in git), not something to treat as inert config. It isn't permanent: every successful automated run re-saves it (_save_state()), which tends to slide a "remember me"-style cookie's expiration forward, but it will eventually expire (inactivity, a password change, TCGplayer forcing re-auth) and need re-importing.

The Dashboard shows a red banner (_session_warning() in app.py) whenever the saved session is known to be missing/expired, so this doesn't just look like "no rules found." The flag behind it lives in session_state.json and is set by automation._ensure_authenticated() on any failed page load — a scheduled run, a "Refresh Rules", or a verification poll — and cleared the moment one authenticates again (or a fresh export is imported). It's deliberately not inferred from run_history.json: list_rules() writes no history entry, so a failed refresh — often the first symptom the user sees — would never raise the banner.

Session-expiry email alert

notify.py — a thin Gmail-SMTP-with-app-password sender, the same pattern lunch-notifier already uses on this box (reused deliberately rather than adding a second notification channel). GMAIL_USER/ GMAIL_APP_PASSWORD (a Gmail app password, not the real account password — Gmail blocks plain SMTP auth otherwise) and optional ALERT_EMAIL (defaults to GMAIL_USER). send_alert() is a best-effort no-op — returns False and just logs — if those env vars aren't set or the send fails, since a broken mail config shouldn't break automation.

Fires exactly once per expiry, not on every failed scheduled run while it's down: automation._notify_session_expired_once() is called from _ensure_authenticated() right before it raises, checks/sets a session_expired_notified flag in notify_state.json, and only sends if that flag isn't already set. The flag resets to False the moment import_cookies() successfully saves a fresh session, so the next expiry gets its own alert. The flag is only set True after a confirmed-sent email — if mail is misconfigured, every failing run keeps retrying the send rather than silently giving up after the first one.

Local development

pip install -r requirements.txt
playwright install chromium   # only needed if testing automation.py directly
python3 app.py

Runs on port 5000 (PORT env to change). Without Chromium installed and without a saved auth_state.json, the Dashboard still boots — _bootstrap() in app.py queues the initial rule sync instead of running it inline, so the failure lands in the worker thread's log and local UI/schedule-editing work doesn't require Docker or a real MassPrice session at all. DATA_DIR env var overrides where JSON state is read/written (defaults to /data, which won't exist locally — point it at a scratch directory).

Every commit should be matched with a version bump — APP_VERSION/APP_VERSION_DATE near the top of app.py — the same "does the footer prove this deploy landed" signal used on Arr Summary.

Deployment

See the deploy-massprice skill for the Forgejo push-to-deploy pipeline and manual container restart steps.

Never put real credentials in the repo. .env and data/ are gitignored and excluded from the deploy rsync precisely so LOGIN_PASSWORD/SECRET_KEY and the saved MassPrice session (data/auth_state.json) have somewhere to live outside git.