e2eaabca06
All checks were successful
Deploy / deploy (push) Successful in 30s
Full read-through of the project. Grouped by what actually breaks:
Browser concurrency (the invariant the design rests on)
- Every Playwright session now goes through scheduler.submit() onto the
single worker queue. "Refresh Rules", cookie import and the boot sync
were calling automation.py directly from Flask request threads, so
with gunicorn's threads=4 a refresh during a scheduled run opened a
second Chromium and both wrote /data/auth_state.json — which
Playwright writes itself, non-atomically. A torn write there reads
back as "session expired" and costs a pointless cookie re-import.
- Request handlers that need a result use run_browser_task(), which
waits REQUEST_WAIT_SECONDS (90) then raises BrowserBusy -> 503
without cancelling the work. Previously a slow in-request scrape could
exceed gunicorn's 120s timeout; killing that worker takes the
APScheduler jobstore and run queue with it, silently dropping every
pending verification poll.
Verification correctness (the feature that exists to catch silent failures)
- run_rule() no longer starts a poll loop when the watermark capture
itself failed. watermark_id was None both for "capture failed" and
"rule has no prior activity", and check_completion() skips its
comparison on None — so a failed capture matched the *previous* run's
Complete entry and reported a false "verified", the exact class of bug
the verification step was added to catch.
- A refresh now reconciles schedules.json against the live rule list.
Rule ids are stable, names are not, and the cached name is what the
Activity-page match keys on: a renamed rule kept running while
reporting "Not Confirmed" forever, training you to ignore the signal.
A deleted rule was worse — its card disappears while its cron jobs
keep firing, with no UI left to disable them. Skipped on an empty
scrape so a MassPrice redesign can't delete every saved schedule.
- The dashboard's session banner reads a session_state.json flag set by
_ensure_authenticated() instead of guessing from run_history.json.
list_rules() writes no history entry, so a failed refresh — often the
first symptom — never raised the banner, despite the docs saying it
covered "run/refresh".
Scheduler robustness
- rebuild_jobs() parses and adds each entry under its own guard. It runs
*after* clearing existing jobs, so one malformed entry escaping as an
exception left the scheduler holding no jobs at all — every schedule
silently dead until the next restart. _parse_entry() also accepts
HH:MM:SS (some browsers emit seconds, which the old two-value unpack
crashed on) and range-checks hour/minute.
- The API validates entries at save time rather than storing something
every rebuild skips, and the frontend surfaces save failures instead
of discarding them.
- An unknown TZ logs and falls back to UTC instead of crash-looping the
container with a traceback that doesn't obviously point at .env.
Auth and web hygiene
- Empty-but-set LOGIN_PASSWORD/SECRET_KEY now fall back to their
defaults. docker-compose always sets these, so a blank .env value
arrived as "" — and compare_digest("", "") is True, i.e. no login at
all. An empty submitted password is rejected outright too.
- Password compared as bytes: compare_digest raises TypeError on
non-ASCII str, turning any accented character into a 500.
- CSRF token per session, in a meta tag, echoed as X-CSRF-Token on all
four POST endpoints; explicit SameSite=Lax/HttpOnly.
- /api/* answers 401 instead of redirecting. fetch() follows a 302 to
/login, gets 200 HTML, and reads it as success — "Run Now" reported
"Queued" for runs that were never queued.
- ?next= is restricted to local paths (open redirect).
Smaller correctness fixes
- Timestamps render in TZ via a localtime filter; they were shown as raw
UTC while schedules run in America/Phoenix.
- Activity's 24h filter excludes unparseable timestamps rather than
including them, and treats naive values as UTC instead of raising.
- SameSite=None cookies are forced Secure — Chromium drops them
otherwise, silently, possibly losing the one cookie carrying the login.
- The Activity selector escapes the rule name, so a name containing a
quote no longer produces an unparseable selector.
- pytz pinned explicitly; it only ever resolved as an apscheduler 3.x
transitive dep, which 4.x drops.
Docs corrected where they disagreed with the code: auth_state.json is
written by Playwright, not through storage.py's atomic helper; the
banner's real trigger; and the deploy skill's claim that a rename causes
"rule id not found" (it can't — schedules are keyed by the stable id).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
268 lines
16 KiB
Markdown
268 lines
16 KiB
Markdown
# MassPrice Scheduler
|
||
|
||
A Flask app that drives a real headless-Chromium session against
|
||
`massprice.tcgplayer.com` (TCGplayer's bulk pricing-rule tool) to click
|
||
"Run Rule" for each of the user's pricing rules on a schedule set through
|
||
this app's own web UI. Runs as one Docker container on the user's Unraid
|
||
home server, deployed the same way as the other projects on that box
|
||
(`arr-summary`, `card-grader`, `lunch-notifier`, `n64-tracker`).
|
||
|
||
## Why this exists
|
||
|
||
MassPrice has its own native per-rule scheduler (day-of-week + one
|
||
time-of-day), but it's capped at 15 active schedules account-wide and
|
||
only allows **one run per day per rule** — no multiple runs/day. Native
|
||
runs also showed real clock drift during discovery (6–22 minutes late
|
||
against their scheduled time). This tool **replaces** native scheduling
|
||
entirely: each rule's native schedule should be set to Inactive once this
|
||
tool is confirmed working, so there's exactly one thing triggering runs.
|
||
|
||
## Architecture
|
||
|
||
- **`app.py`** — Flask app, single shared-password session auth
|
||
(`LOGIN_PASSWORD` env var, `hmac.compare_digest`) — this protects the
|
||
*admin UI itself*, unrelated to the MassPrice/TCGplayer login below.
|
||
Routes: `/` (Dashboard — rule cards with schedule editor), `/activity`
|
||
(run history), `/session` (login-session import), and a small JSON API
|
||
(`/api/rules/refresh`, `/api/rules/<id>/schedule`,
|
||
`/api/rules/<id>/run`, `/api/session/import`) that the frontend JS
|
||
calls without full page reloads.
|
||
- **`scheduler.py`** — an in-process APScheduler `BackgroundScheduler`
|
||
holding one `CronTrigger` job per `(rule, schedule entry)` pair,
|
||
rebuilt from `schedules.json` on startup and whenever the webui saves
|
||
a change. **Every** Playwright session goes through `submit()` onto
|
||
one `queue.Queue`, drained by a single dedicated worker thread —
|
||
scheduled runs, "Run Now", "Refresh Rules" and cookie imports alike.
|
||
That guarantees only one browser is ever active at a time, matching
|
||
how a human would actually operate the site and avoiding
|
||
concurrent-session weirdness. Nothing may call `automation.py`
|
||
directly from a Flask request thread: with `threads = 4` that opens a
|
||
second Chromium alongside a scheduled run, and both write
|
||
`auth_state.json` — which Playwright writes itself, non-atomically —
|
||
so a torn write reads back as "session expired" and costs a pointless
|
||
cookie re-import.
|
||
- Request handlers that need an answer (refresh, import) use
|
||
`run_browser_task()`, which waits `REQUEST_WAIT_SECONDS` (90) and
|
||
then raises `BrowserBusy` → HTTP 503, *without* cancelling the work.
|
||
The bound is the point: gunicorn kills any worker whose request
|
||
exceeds its `timeout` (120s), and since the jobstore and queue live
|
||
in that worker's memory, being killed drops every pending
|
||
verification poll. Waiting behind a slow rule run must never be able
|
||
to take the scheduler down.
|
||
- **Must run in exactly one process.** `gunicorn.conf.py` pins
|
||
`workers = 1` for this reason — a second worker would instantiate a
|
||
second scheduler and duplicate every run. Fine for a low-traffic
|
||
single-user tool.
|
||
- Also drives the **completion poll loop**: after a run is confirmed
|
||
sent, a one-off `verify-<entry_id>-<attempt>` job (APScheduler
|
||
`date` trigger, not recurring) fires `VERIFY_POLL_MINUTES` later and
|
||
calls `automation.check_completion()`. Not complete yet? Another
|
||
one-off job reschedules the next check the same interval later —
|
||
`_run_verification_check()` in `scheduler.py` owns this loop — up to
|
||
`VERIFY_MAX_ATTEMPTS` times, after which it's marked
|
||
`"not_confirmed"` for good. `rebuild_jobs()` only ever removes
|
||
`rule-*` jobs, deliberately, so editing a schedule mid-flight can't
|
||
cancel a poll loop that's already in progress for an earlier run.
|
||
These one-off jobs live only in the scheduler's in-memory jobstore
|
||
(no persistent jobstore is configured) — a container restart mid-poll
|
||
silently drops that one entry's remaining checks, leaving it stuck
|
||
on whatever state it was last in (usually "pending"). Low-impact
|
||
(an hour-long window, restarts are infrequent) but worth knowing if
|
||
an entry is ever seen stuck on "Pending" long after the fact.
|
||
- **`automation.py`** — all Playwright logic. `list_rules()` scrapes
|
||
`https://massprice.tcgplayer.com/Rule/List` for the live rule
|
||
id/name pairs (cached to `rules_cache.json`, refreshed via the
|
||
Dashboard's "Refresh Rules" button or on first boot). A refresh also
|
||
runs `_reconcile_schedules()` in `app.py`, which re-syncs
|
||
`schedules.json` against that live list: rule **ids** are stable but
|
||
names aren't, and a stale cached name silently breaks run
|
||
verification (it's what the Activity-page match keys on), while a
|
||
rule deleted upstream would otherwise keep firing cron jobs with no
|
||
card left in the UI to disable them. Deliberately skipped when the
|
||
scrape returns an empty list, so a MassPrice redesign that breaks
|
||
the scrape can't delete every saved schedule. `run_rule(id)`
|
||
finds that rule's `li.rule.widget-rule` row, clicks its
|
||
`.dropdown-ellipsis` three-dot toggle, clicks the row's
|
||
`a.run-rule[data-rule-id=...]` item — mirroring exactly what a human
|
||
does in the UI — **then must click through a confirmation modal**
|
||
(`#ConfirmationModal` → `#Confirmation_Run`; the same generic modal
|
||
is reused for Delete Rule etc., so the confirm click is scoped to
|
||
that specific modal instance). That confirmation step is not
|
||
optional or cosmetic: confirmed by live testing that skipping it
|
||
fires **zero** network requests and the rule silently never runs,
|
||
even though the initial dropdown click "succeeds" with no error —
|
||
an earlier version of this code did exactly that and every reported
|
||
"success" was a false positive for a real stretch of production use.
|
||
If a future site redesign changes this modal, watch for the same
|
||
symptom: local history says success, MassPrice's own Activity Log
|
||
never gets a matching entry. **No public API was found or used** —
|
||
the "Run Rule" click handler is JS/AJAX-bound (no plain href, a
|
||
delegated jQuery handler on `document.body`, not the element itself)
|
||
and its request (`POST /Rule/Run`, confirmed by live network capture)
|
||
is almost certainly anti-forgery-token protected, so automation
|
||
drives the real page rather than replaying a raw HTTP POST.
|
||
- **Run verification** — confirming the modal only means MassPrice
|
||
*accepted* the run; it still processes asynchronously on their side
|
||
(queued → running → Complete), taking anywhere from ~1 minute to over
|
||
20 in observed practice, and its queue can back up further under
|
||
load (observed directly: a run still sitting in "Queued" 15+ minutes
|
||
in). `run_rule()` captures a "watermark" — the `data-activitylogid` of
|
||
that rule's current top entry on `/Activity/List` (a stable,
|
||
monotonically increasing ID; deliberately *not* parsing the displayed
|
||
timestamp string, which would need fragile date/timezone handling) —
|
||
**before** submitting, logs the run as `status: "sent"` with
|
||
`verification: "pending"`, and the caller in `scheduler.py` starts a
|
||
poll loop (`automation.check_completion()`, every
|
||
`VERIFY_POLL_MINUTES`, up to `VERIFY_MAX_ATTEMPTS` times — see
|
||
`scheduler.py`'s own section above for the mechanics). Each check
|
||
re-scans `/Activity/List`: a new entry (higher `data-activitylogid`
|
||
than the watermark) whose status text contains "Complete" →
|
||
`verification: "complete"`, loop stops; otherwise the loop keeps
|
||
`verification: "pending"` and just updates `verify_message` with the
|
||
latest note, until attempts run out and it's marked
|
||
`verification: "not_confirmed"` for good — which can mean "still
|
||
processing somewhere past our patience", not necessarily "failed".
|
||
Also worth knowing: a still-"Queued" MassPrice entry has no
|
||
`data-activitylogid` yet (no "View" link exists until it finishes),
|
||
so `_find_latest_activity()` — shared by the watermark capture and
|
||
every poll — correctly falls through to the *previous* completed
|
||
entry rather than matching the in-flight one; that's why "no ID
|
||
higher than the watermark" is the right "not done yet" signal, not a
|
||
bug. The Activity page's "Complete" column surfaces all of this;
|
||
"Status" shows whether the click itself was sent, independent of
|
||
whether MassPrice finished it.
|
||
- **Session persistence** — a Playwright `storage_state` (cookies +
|
||
localStorage) is saved to `/data/auth_state.json` and reused across
|
||
runs: scheduled runs never touch the login form at all, they just load
|
||
this saved session. When it's missing or has expired (detected by an
|
||
unauthenticated visit to `/Rule/List` silently redirecting to the
|
||
public marketing homepage `/` — there's no `/login` URL to check
|
||
against), the run fails with a clear "session expired or missing"
|
||
error rather than attempting to log in — see below for why.
|
||
- **`storage.py`** — tiny JSON read/write helpers (temp-file +
|
||
`os.replace`, so a crash mid-write can't corrupt state) for the state
|
||
files under `/data`: `schedules.json`, `run_history.json` (bounded to
|
||
the last 200 entries), `rules_cache.json`, `session_state.json` (the
|
||
"is the saved session known-bad" flag behind the dashboard banner) and
|
||
`notify_state.json` (the session-expiry alert's send-once dedupe flag).
|
||
`auth_state.json` lives in the same directory but is **not** written
|
||
through here — Playwright writes it itself via
|
||
`context.storage_state(path=...)` in `automation._save_state()`, so it
|
||
gets none of that atomic-write protection. Worth knowing: it's also
|
||
the one file two concurrent Playwright sessions can both write.
|
||
- **`templates/` / `static/`** — a small, plain dark dashboard. This is
|
||
a deliberately different, simpler visual identity than Arr Summary's
|
||
teletext system — that design language is specific to that app, not a
|
||
house style to reuse here.
|
||
|
||
## Login — why it's manual, and how it works
|
||
|
||
MassPrice's login (`store.tcgplayer.com/oauth/login`, TCGplayer's shared
|
||
OAuth identity provider) is protected by **hCaptcha**. Two automated
|
||
approaches were tried and confirmed broken by live testing during
|
||
development — there is no selector fix for either, **fully automated
|
||
login is not achievable here**, full stop:
|
||
1. Filling in the real username/password and clicking Submit — the
|
||
invisible captcha check silently never resolves for a headless/
|
||
automated browser, and the form just refuses to submit. An earlier
|
||
version of this app tried `MASSPRICE_USERNAME`/`MASSPRICE_PASSWORD`
|
||
this way; removed once confirmed broken.
|
||
2. Driving a real human's clicks through a headless Chromium exposed via
|
||
Chrome remote debugging (`chrome://inspect`) — also removed after
|
||
testing. Still fails hCaptcha, because the browser itself (a
|
||
`HeadlessChrome` user-agent, `navigator.webdriver = true`) is what
|
||
gets flagged, independent of who's actually clicking. Deliberately
|
||
not pursued further than that one test: disguising that browser to
|
||
get past hCaptcha's bot detection would be captcha evasion, not a
|
||
session-handling technique, regardless of how legitimate running
|
||
automation on your own account is.
|
||
|
||
What actually works: log in with an ordinary, non-automated browser —
|
||
hCaptcha passes the way it always does for a real person, because
|
||
nothing about that browser looks automated — then copy that session's
|
||
cookies over. The **Session** page in the webui drives this:
|
||
1. The user logs into massprice.tcgplayer.com in their own regular
|
||
Chrome and exports its cookies with a browser extension (Cookie-Editor,
|
||
"Export as JSON").
|
||
2. They paste that JSON into the Session page and click "Import Session".
|
||
3. `automation.import_cookies()` parses it, converts each cookie from
|
||
Cookie-Editor's shape into Playwright's `storage_state` cookie shape
|
||
(`_convert_cookie()` — mainly the `sameSite` enum values and the
|
||
`expirationDate` → `expires` field rename), and — critically —
|
||
**verifies it actually authenticates** (a real headless request to
|
||
`/Rule/List`, checked with the same `_is_logged_out()` used
|
||
everywhere else) before saving it to `/data/auth_state.json`. A bad
|
||
paste never overwrites a working session.
|
||
|
||
This session is a real, live bearer credential for the user's MassPrice
|
||
account for as long as it's valid — same trust model as `LOGIN_PASSWORD`/
|
||
`SECRET_KEY` in `.env` (server-only, gitignored, never in git), not
|
||
something to treat as inert config. It isn't permanent: every successful
|
||
automated run re-saves it (`_save_state()`), which tends to slide a
|
||
"remember me"-style cookie's expiration forward, but it will eventually
|
||
expire (inactivity, a password change, TCGplayer forcing re-auth) and
|
||
need re-importing.
|
||
|
||
The Dashboard shows a red banner (`_session_warning()` in `app.py`)
|
||
whenever the saved session is known to be missing/expired, so this
|
||
doesn't just look like "no rules found." The flag behind it lives in
|
||
`session_state.json` and is set by `automation._ensure_authenticated()`
|
||
on *any* failed page load — a scheduled run, a "Refresh Rules", or a
|
||
verification poll — and cleared the moment one authenticates again (or a
|
||
fresh export is imported). It's deliberately not inferred from
|
||
`run_history.json`: `list_rules()` writes no history entry, so a failed
|
||
refresh — often the first symptom the user sees — would never raise the
|
||
banner.
|
||
|
||
## Session-expiry email alert
|
||
|
||
**`notify.py`** — a thin Gmail-SMTP-with-app-password sender, the same
|
||
pattern `lunch-notifier` already uses on this box (reused deliberately
|
||
rather than adding a second notification channel). `GMAIL_USER`/
|
||
`GMAIL_APP_PASSWORD` (a Gmail **app password**, not the real account
|
||
password — Gmail blocks plain SMTP auth otherwise) and optional
|
||
`ALERT_EMAIL` (defaults to `GMAIL_USER`). `send_alert()` is a best-effort
|
||
no-op — returns `False` and just logs — if those env vars aren't set or
|
||
the send fails, since a broken mail config shouldn't break automation.
|
||
|
||
Fires exactly **once** per expiry, not on every failed scheduled run
|
||
while it's down: `automation._notify_session_expired_once()` is called
|
||
from `_ensure_authenticated()` right before it raises, checks/sets a
|
||
`session_expired_notified` flag in `notify_state.json`, and only sends
|
||
if that flag isn't already set. The flag resets to `False` the moment
|
||
`import_cookies()` successfully saves a fresh session, so the next
|
||
expiry gets its own alert. The flag is only set `True` after a
|
||
confirmed-sent email — if mail is misconfigured, every failing run keeps
|
||
retrying the send rather than silently giving up after the first one.
|
||
|
||
## Local development
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
playwright install chromium # only needed if testing automation.py directly
|
||
python3 app.py
|
||
```
|
||
|
||
Runs on port 5000 (`PORT` env to change). Without Chromium installed and
|
||
without a saved `auth_state.json`, the Dashboard still boots —
|
||
`_bootstrap()` in `app.py` queues the initial rule sync instead of
|
||
running it inline, so the failure lands in the worker thread's log and
|
||
local UI/schedule-editing work doesn't require Docker or a real
|
||
MassPrice session at all. `DATA_DIR` env var overrides where JSON state is
|
||
read/written (defaults to `/data`, which won't exist locally — point it
|
||
at a scratch directory).
|
||
|
||
**Every commit should be matched with a version bump** —
|
||
`APP_VERSION`/`APP_VERSION_DATE` near the top of `app.py` — the same
|
||
"does the footer prove this deploy landed" signal used on Arr Summary.
|
||
|
||
## Deployment
|
||
|
||
See the `deploy-massprice` skill for the Forgejo push-to-deploy
|
||
pipeline and manual container restart steps.
|
||
|
||
**Never put real credentials in the repo.** `.env` and `data/` are
|
||
gitignored and excluded from the deploy rsync precisely so
|
||
`LOGIN_PASSWORD`/`SECRET_KEY` and the saved
|
||
MassPrice session (`data/auth_state.json`) have somewhere to live
|
||
outside git.
|