bec399dd11
All checks were successful
Deploy / deploy (push) Successful in 38s
Live testing against the real site found MassPrice's login is behind hCaptcha, which silently blocks fully automated username/password login outright (the form never submits) — no selector fix could work around this. Also fixed the broken _is_logged_out() heuristic: an unauthenticated visit to /Rule/List redirects to the public marketing homepage, not a /login URL, so run_rule/list_rules were silently scraping the wrong page instead of erroring. New flow: a Session page lets the user start a headless-but-CDP-exposed Chromium instance, log in themselves via chrome://inspect (solving the captcha as a real click), then capture that browser's cookies into the same auth_state.json scheduled runs already use. Required a socat proxy in the container image, since Chromium's debug port binds loopback-only regardless of --remote-debugging-address (confirmed by live testing), unreachable through Docker's port mapping without it. All Playwright objects for this now live on one dedicated thread (automation.py's _debug_worker), since the sync API isn't safe to call across the arbitrary threads gunicorn's gthread workers hand out per request. Dashboard now shows a banner when the latest run failed specifically due to a missing/expired session, rather than just looking like an empty rule list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
141 lines
7.4 KiB
Markdown
141 lines
7.4 KiB
Markdown
# MassPrice Scheduler
|
||
|
||
A Flask app that drives a real headless-Chromium session against
|
||
`massprice.tcgplayer.com` (TCGplayer's bulk pricing-rule tool) to click
|
||
"Run Rule" for each of the user's pricing rules on a schedule set through
|
||
this app's own web UI. Runs as one Docker container on the user's Unraid
|
||
home server, deployed the same way as the other projects on that box
|
||
(`arr-summary`, `card-grader`, `lunch-notifier`, `n64-tracker`).
|
||
|
||
## Why this exists
|
||
|
||
MassPrice has its own native per-rule scheduler (day-of-week + one
|
||
time-of-day), but it's capped at 15 active schedules account-wide and
|
||
only allows **one run per day per rule** — no multiple runs/day. Native
|
||
runs also showed real clock drift during discovery (6–22 minutes late
|
||
against their scheduled time). This tool **replaces** native scheduling
|
||
entirely: each rule's native schedule should be set to Inactive once this
|
||
tool is confirmed working, so there's exactly one thing triggering runs.
|
||
|
||
## Architecture
|
||
|
||
- **`app.py`** — Flask app, single shared-password session auth
|
||
(`LOGIN_PASSWORD` env var, `hmac.compare_digest`) — this protects the
|
||
*admin UI itself*, unrelated to the MassPrice/TCGplayer login below.
|
||
Routes: `/` (Dashboard — rule cards with schedule editor), `/activity`
|
||
(run history), and a small JSON API
|
||
(`/api/rules/refresh`, `/api/rules/<id>/schedule`,
|
||
`/api/rules/<id>/run`) that the Dashboard's JS calls without full page
|
||
reloads.
|
||
- **`scheduler.py`** — an in-process APScheduler `BackgroundScheduler`
|
||
holding one `CronTrigger` job per `(rule, schedule entry)` pair,
|
||
rebuilt from `schedules.json` on startup and whenever the webui saves
|
||
a change. Triggered jobs (and manual "Run Now" clicks) are pushed onto
|
||
one `queue.Queue`, drained by a single dedicated worker thread — this
|
||
guarantees only one Playwright browser session is ever active at a
|
||
time, matching how a human would actually operate the site and
|
||
avoiding concurrent-session weirdness.
|
||
- **Must run in exactly one process.** `gunicorn.conf.py` pins
|
||
`workers = 1` for this reason — a second worker would instantiate a
|
||
second scheduler and duplicate every run. Fine for a low-traffic
|
||
single-user tool.
|
||
- **`automation.py`** — all Playwright logic. `list_rules()` scrapes
|
||
`https://massprice.tcgplayer.com/Rule/List` for the live rule
|
||
id/name pairs (cached to `rules_cache.json`, refreshed via the
|
||
Dashboard's "Refresh Rules" button or on first boot). `run_rule(id)`
|
||
finds that rule's `li.rule.widget-rule` row, clicks its
|
||
`.dropdown-ellipsis` three-dot toggle, then clicks the row's
|
||
`a.run-rule[data-rule-id=...]` item — mirroring exactly what a human
|
||
does in the UI. **No public API was found or used** — the "Run Rule"
|
||
click handler is JS/AJAX-bound (no plain href) and its request is
|
||
almost certainly anti-forgery-token protected, so automation drives
|
||
the real page rather than replaying a raw HTTP POST.
|
||
- **Session persistence** — a Playwright `storage_state` (cookies +
|
||
localStorage) is saved to `/data/auth_state.json` and reused across
|
||
runs: scheduled runs never touch the login form at all, they just load
|
||
this saved session. When it's missing or has expired (detected by an
|
||
unauthenticated visit to `/Rule/List` silently redirecting to the
|
||
public marketing homepage `/` — there's no `/login` URL to check
|
||
against), the run fails with a clear "session expired or missing"
|
||
error rather than attempting to log in — see below for why.
|
||
- **`storage.py`** — tiny JSON read/write helpers (temp-file +
|
||
`os.replace`, so a crash mid-write can't corrupt state) for
|
||
everything under `/data`: `schedules.json`, `auth_state.json`,
|
||
`run_history.json` (bounded to the last 200 entries), `rules_cache.json`.
|
||
- **`templates/` / `static/`** — a small, plain dark dashboard. This is
|
||
a deliberately different, simpler visual identity than Arr Summary's
|
||
teletext system — that design language is specific to that app, not a
|
||
house style to reuse here.
|
||
|
||
## Login — why it's manual, and how it works
|
||
|
||
MassPrice's login (`store.tcgplayer.com/oauth/login`, TCGplayer's shared
|
||
OAuth identity provider) is protected by **hCaptcha**. This was
|
||
confirmed by live testing during development: filling in the real
|
||
username/password and clicking Submit never navigates anywhere — the
|
||
invisible captcha check silently never resolves for a headless/automated
|
||
browser, and the form just refuses to submit. There is no selector fix
|
||
for this; **fully automated login is not achievable here**, full stop.
|
||
An earlier version of this app tried `MASSPRICE_USERNAME`/
|
||
`MASSPRICE_PASSWORD` credential-based auto-login — removed once this was
|
||
confirmed, since keeping dead code that can never succeed is worse than
|
||
having none.
|
||
|
||
What actually works: a human only needs to pass the hCaptcha, not
|
||
automate around it, and that's a one-time (well, periodic — however long
|
||
the session cookie lasts) cost, not a per-run one. The **Session** page
|
||
in the webui drives this:
|
||
1. "Start Login Session" launches headless Chromium with its CDP debug
|
||
port exposed (`DEBUG_PORT`, default `9222`).
|
||
2. The user connects their own Chrome to it via `chrome://inspect`,
|
||
which renders and accepts real input against that same running
|
||
browser instance — no VNC/Xvfb needed, headless Chromium's debug
|
||
protocol already supports this.
|
||
3. The user logs in normally in that inspected tab, solving the hCaptcha
|
||
themselves since it's a real click from a real browser.
|
||
4. "Capture Session" saves that browser's cookies to
|
||
`/data/auth_state.json` — the same file scheduled runs read.
|
||
|
||
**All of this lives on one dedicated thread** (`automation._debug_worker`),
|
||
not arbitrary Flask request threads — Playwright's sync API isn't safe
|
||
to call across threads, and gunicorn's `gthread` workers hand each
|
||
request to a different one. Flask routes only ever put a command on
|
||
`_debug_cmd_queue` and block for the result; they never touch the
|
||
Playwright browser/context/page objects directly. An un-captured debug
|
||
session auto-closes after 10 minutes so an open, unauthenticated CDP
|
||
port doesn't sit exposed on the LAN indefinitely.
|
||
|
||
The Dashboard shows a red banner (`_session_warning()` in `app.py`) when
|
||
the most recent run/refresh failed specifically because of a missing/
|
||
expired session, so this doesn't just look like "no rules found."
|
||
|
||
## Local development
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
playwright install chromium # only needed if testing automation.py directly
|
||
python3 app.py
|
||
```
|
||
|
||
Runs on port 5000 (`PORT` env to change). Without Chromium installed and
|
||
without a saved `auth_state.json`, the Dashboard still boots —
|
||
`_bootstrap()` in `app.py` best-effort-catches the initial rule sync, so
|
||
local UI/schedule-editing work doesn't require Docker or a real MassPrice
|
||
session at all. `DATA_DIR` env var overrides where JSON state is
|
||
read/written (defaults to `/data`, which won't exist locally — point it
|
||
at a scratch directory).
|
||
|
||
**Every commit should be matched with a version bump** —
|
||
`APP_VERSION`/`APP_VERSION_DATE` near the top of `app.py` — the same
|
||
"does the footer prove this deploy landed" signal used on Arr Summary.
|
||
|
||
## Deployment
|
||
|
||
See the `deploy-massprice` skill for the Forgejo push-to-deploy
|
||
pipeline and manual container restart steps.
|
||
|
||
**Never put real credentials in the repo.** `.env` and `data/` are
|
||
gitignored and excluded from the deploy rsync precisely so
|
||
`LOGIN_PASSWORD`/`SECRET_KEY` and the saved
|
||
MassPrice session (`data/auth_state.json`) have somewhere to live
|
||
outside git.
|