48d8bac332
All checks were successful
Deploy / deploy (push) Successful in 25s
Live testing showed the CDP approach still fails hCaptcha even with a real human clicking through chrome://inspect: the headless Chromium's own fingerprint (HeadlessChrome user-agent, navigator.webdriver=true) is what gets flagged, independent of who's driving it. Trying to disguise that browser to get past bot detection would be captcha evasion, not a session-handling technique, so that path is dropped entirely rather than patched further. New flow: log into MassPrice with an ordinary, non-automated browser (hCaptcha passes normally), export the session's cookies with a extension, and paste them into the Session page. import_cookies() converts the export into Playwright's storage_state shape and verifies it actually authenticates against the live site before saving it, so a bad paste never clobbers a working session. This also removes the now-unnecessary socat proxy / debug port exposure — Chromium's own CDP port was only ever needed for the dropped approach. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
151 lines
8.1 KiB
Markdown
151 lines
8.1 KiB
Markdown
# MassPrice Scheduler
|
||
|
||
A Flask app that drives a real headless-Chromium session against
|
||
`massprice.tcgplayer.com` (TCGplayer's bulk pricing-rule tool) to click
|
||
"Run Rule" for each of the user's pricing rules on a schedule set through
|
||
this app's own web UI. Runs as one Docker container on the user's Unraid
|
||
home server, deployed the same way as the other projects on that box
|
||
(`arr-summary`, `card-grader`, `lunch-notifier`, `n64-tracker`).
|
||
|
||
## Why this exists
|
||
|
||
MassPrice has its own native per-rule scheduler (day-of-week + one
|
||
time-of-day), but it's capped at 15 active schedules account-wide and
|
||
only allows **one run per day per rule** — no multiple runs/day. Native
|
||
runs also showed real clock drift during discovery (6–22 minutes late
|
||
against their scheduled time). This tool **replaces** native scheduling
|
||
entirely: each rule's native schedule should be set to Inactive once this
|
||
tool is confirmed working, so there's exactly one thing triggering runs.
|
||
|
||
## Architecture
|
||
|
||
- **`app.py`** — Flask app, single shared-password session auth
|
||
(`LOGIN_PASSWORD` env var, `hmac.compare_digest`) — this protects the
|
||
*admin UI itself*, unrelated to the MassPrice/TCGplayer login below.
|
||
Routes: `/` (Dashboard — rule cards with schedule editor), `/activity`
|
||
(run history), and a small JSON API
|
||
(`/api/rules/refresh`, `/api/rules/<id>/schedule`,
|
||
`/api/rules/<id>/run`) that the Dashboard's JS calls without full page
|
||
reloads.
|
||
- **`scheduler.py`** — an in-process APScheduler `BackgroundScheduler`
|
||
holding one `CronTrigger` job per `(rule, schedule entry)` pair,
|
||
rebuilt from `schedules.json` on startup and whenever the webui saves
|
||
a change. Triggered jobs (and manual "Run Now" clicks) are pushed onto
|
||
one `queue.Queue`, drained by a single dedicated worker thread — this
|
||
guarantees only one Playwright browser session is ever active at a
|
||
time, matching how a human would actually operate the site and
|
||
avoiding concurrent-session weirdness.
|
||
- **Must run in exactly one process.** `gunicorn.conf.py` pins
|
||
`workers = 1` for this reason — a second worker would instantiate a
|
||
second scheduler and duplicate every run. Fine for a low-traffic
|
||
single-user tool.
|
||
- **`automation.py`** — all Playwright logic. `list_rules()` scrapes
|
||
`https://massprice.tcgplayer.com/Rule/List` for the live rule
|
||
id/name pairs (cached to `rules_cache.json`, refreshed via the
|
||
Dashboard's "Refresh Rules" button or on first boot). `run_rule(id)`
|
||
finds that rule's `li.rule.widget-rule` row, clicks its
|
||
`.dropdown-ellipsis` three-dot toggle, then clicks the row's
|
||
`a.run-rule[data-rule-id=...]` item — mirroring exactly what a human
|
||
does in the UI. **No public API was found or used** — the "Run Rule"
|
||
click handler is JS/AJAX-bound (no plain href) and its request is
|
||
almost certainly anti-forgery-token protected, so automation drives
|
||
the real page rather than replaying a raw HTTP POST.
|
||
- **Session persistence** — a Playwright `storage_state` (cookies +
|
||
localStorage) is saved to `/data/auth_state.json` and reused across
|
||
runs: scheduled runs never touch the login form at all, they just load
|
||
this saved session. When it's missing or has expired (detected by an
|
||
unauthenticated visit to `/Rule/List` silently redirecting to the
|
||
public marketing homepage `/` — there's no `/login` URL to check
|
||
against), the run fails with a clear "session expired or missing"
|
||
error rather than attempting to log in — see below for why.
|
||
- **`storage.py`** — tiny JSON read/write helpers (temp-file +
|
||
`os.replace`, so a crash mid-write can't corrupt state) for
|
||
everything under `/data`: `schedules.json`, `auth_state.json`,
|
||
`run_history.json` (bounded to the last 200 entries), `rules_cache.json`.
|
||
- **`templates/` / `static/`** — a small, plain dark dashboard. This is
|
||
a deliberately different, simpler visual identity than Arr Summary's
|
||
teletext system — that design language is specific to that app, not a
|
||
house style to reuse here.
|
||
|
||
## Login — why it's manual, and how it works
|
||
|
||
MassPrice's login (`store.tcgplayer.com/oauth/login`, TCGplayer's shared
|
||
OAuth identity provider) is protected by **hCaptcha**. Two automated
|
||
approaches were tried and confirmed broken by live testing during
|
||
development — there is no selector fix for either, **fully automated
|
||
login is not achievable here**, full stop:
|
||
1. Filling in the real username/password and clicking Submit — the
|
||
invisible captcha check silently never resolves for a headless/
|
||
automated browser, and the form just refuses to submit. An earlier
|
||
version of this app tried `MASSPRICE_USERNAME`/`MASSPRICE_PASSWORD`
|
||
this way; removed once confirmed broken.
|
||
2. Driving a real human's clicks through a headless Chromium exposed via
|
||
Chrome remote debugging (`chrome://inspect`) — also removed after
|
||
testing. Still fails hCaptcha, because the browser itself (a
|
||
`HeadlessChrome` user-agent, `navigator.webdriver = true`) is what
|
||
gets flagged, independent of who's actually clicking. Deliberately
|
||
not pursued further than that one test: disguising that browser to
|
||
get past hCaptcha's bot detection would be captcha evasion, not a
|
||
session-handling technique, regardless of how legitimate running
|
||
automation on your own account is.
|
||
|
||
What actually works: log in with an ordinary, non-automated browser —
|
||
hCaptcha passes the way it always does for a real person, because
|
||
nothing about that browser looks automated — then copy that session's
|
||
cookies over. The **Session** page in the webui drives this:
|
||
1. The user logs into massprice.tcgplayer.com in their own regular
|
||
Chrome and exports its cookies with a browser extension (Cookie-Editor,
|
||
"Export as JSON").
|
||
2. They paste that JSON into the Session page and click "Import Session".
|
||
3. `automation.import_cookies()` parses it, converts each cookie from
|
||
Cookie-Editor's shape into Playwright's `storage_state` cookie shape
|
||
(`_convert_cookie()` — mainly the `sameSite` enum values and the
|
||
`expirationDate` → `expires` field rename), and — critically —
|
||
**verifies it actually authenticates** (a real headless request to
|
||
`/Rule/List`, checked with the same `_is_logged_out()` used
|
||
everywhere else) before saving it to `/data/auth_state.json`. A bad
|
||
paste never overwrites a working session.
|
||
|
||
This session is a real, live bearer credential for the user's MassPrice
|
||
account for as long as it's valid — same trust model as `LOGIN_PASSWORD`/
|
||
`SECRET_KEY` in `.env` (server-only, gitignored, never in git), not
|
||
something to treat as inert config. It isn't permanent: every successful
|
||
automated run re-saves it (`_save_state()`), which tends to slide a
|
||
"remember me"-style cookie's expiration forward, but it will eventually
|
||
expire (inactivity, a password change, TCGplayer forcing re-auth) and
|
||
need re-importing.
|
||
|
||
The Dashboard shows a red banner (`_session_warning()` in `app.py`) when
|
||
the most recent run/refresh failed specifically because of a missing/
|
||
expired session, so this doesn't just look like "no rules found."
|
||
|
||
## Local development
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
playwright install chromium # only needed if testing automation.py directly
|
||
python3 app.py
|
||
```
|
||
|
||
Runs on port 5000 (`PORT` env to change). Without Chromium installed and
|
||
without a saved `auth_state.json`, the Dashboard still boots —
|
||
`_bootstrap()` in `app.py` best-effort-catches the initial rule sync, so
|
||
local UI/schedule-editing work doesn't require Docker or a real MassPrice
|
||
session at all. `DATA_DIR` env var overrides where JSON state is
|
||
read/written (defaults to `/data`, which won't exist locally — point it
|
||
at a scratch directory).
|
||
|
||
**Every commit should be matched with a version bump** —
|
||
`APP_VERSION`/`APP_VERSION_DATE` near the top of `app.py` — the same
|
||
"does the footer prove this deploy landed" signal used on Arr Summary.
|
||
|
||
## Deployment
|
||
|
||
See the `deploy-massprice` skill for the Forgejo push-to-deploy
|
||
pipeline and manual container restart steps.
|
||
|
||
**Never put real credentials in the repo.** `.env` and `data/` are
|
||
gitignored and excluded from the deploy rsync precisely so
|
||
`LOGIN_PASSWORD`/`SECRET_KEY` and the saved
|
||
MassPrice session (`data/auth_state.json`) have somewhere to live
|
||
outside git.
|