Full read-through of the project. Grouped by what actually breaks:
Browser concurrency (the invariant the design rests on)
- Every Playwright session now goes through scheduler.submit() onto the
single worker queue. "Refresh Rules", cookie import and the boot sync
were calling automation.py directly from Flask request threads, so
with gunicorn's threads=4 a refresh during a scheduled run opened a
second Chromium and both wrote /data/auth_state.json — which
Playwright writes itself, non-atomically. A torn write there reads
back as "session expired" and costs a pointless cookie re-import.
- Request handlers that need a result use run_browser_task(), which
waits REQUEST_WAIT_SECONDS (90) then raises BrowserBusy -> 503
without cancelling the work. Previously a slow in-request scrape could
exceed gunicorn's 120s timeout; killing that worker takes the
APScheduler jobstore and run queue with it, silently dropping every
pending verification poll.
Verification correctness (the feature that exists to catch silent failures)
- run_rule() no longer starts a poll loop when the watermark capture
itself failed. watermark_id was None both for "capture failed" and
"rule has no prior activity", and check_completion() skips its
comparison on None — so a failed capture matched the *previous* run's
Complete entry and reported a false "verified", the exact class of bug
the verification step was added to catch.
- A refresh now reconciles schedules.json against the live rule list.
Rule ids are stable, names are not, and the cached name is what the
Activity-page match keys on: a renamed rule kept running while
reporting "Not Confirmed" forever, training you to ignore the signal.
A deleted rule was worse — its card disappears while its cron jobs
keep firing, with no UI left to disable them. Skipped on an empty
scrape so a MassPrice redesign can't delete every saved schedule.
- The dashboard's session banner reads a session_state.json flag set by
_ensure_authenticated() instead of guessing from run_history.json.
list_rules() writes no history entry, so a failed refresh — often the
first symptom — never raised the banner, despite the docs saying it
covered "run/refresh".
Scheduler robustness
- rebuild_jobs() parses and adds each entry under its own guard. It runs
*after* clearing existing jobs, so one malformed entry escaping as an
exception left the scheduler holding no jobs at all — every schedule
silently dead until the next restart. _parse_entry() also accepts
HH:MM:SS (some browsers emit seconds, which the old two-value unpack
crashed on) and range-checks hour/minute.
- The API validates entries at save time rather than storing something
every rebuild skips, and the frontend surfaces save failures instead
of discarding them.
- An unknown TZ logs and falls back to UTC instead of crash-looping the
container with a traceback that doesn't obviously point at .env.
Auth and web hygiene
- Empty-but-set LOGIN_PASSWORD/SECRET_KEY now fall back to their
defaults. docker-compose always sets these, so a blank .env value
arrived as "" — and compare_digest("", "") is True, i.e. no login at
all. An empty submitted password is rejected outright too.
- Password compared as bytes: compare_digest raises TypeError on
non-ASCII str, turning any accented character into a 500.
- CSRF token per session, in a meta tag, echoed as X-CSRF-Token on all
four POST endpoints; explicit SameSite=Lax/HttpOnly.
- /api/* answers 401 instead of redirecting. fetch() follows a 302 to
/login, gets 200 HTML, and reads it as success — "Run Now" reported
"Queued" for runs that were never queued.
- ?next= is restricted to local paths (open redirect).
Smaller correctness fixes
- Timestamps render in TZ via a localtime filter; they were shown as raw
UTC while schedules run in America/Phoenix.
- Activity's 24h filter excludes unparseable timestamps rather than
including them, and treats naive values as UTC instead of raising.
- SameSite=None cookies are forced Secure — Chromium drops them
otherwise, silently, possibly losing the one cookie carrying the login.
- The Activity selector escapes the rule name, so a name containing a
quote no longer produces an unparseable selector.
- pytz pinned explicitly; it only ever resolved as an apscheduler 3.x
transitive dep, which 4.x drops.
Docs corrected where they disagreed with the code: auth_state.json is
written by Playwright, not through storage.py's atomic helper; the
banner's real trigger; and the deploy skill's claim that a rename causes
"rule id not found" (it can't — schedules are keyed by the stable id).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
The one-shot VERIFY_DELAY_MINUTES check was too rigid: MassPrice's
processing time varies widely in practice (observed ~1 to over 20
minutes, and its queue can back up further under load — confirmed
live, a run still sitting in "Queued" 15+ minutes in), so a single
check window either misses slow-but-fine completions or has to be set
needlessly long.
Replaced with a poll loop: scheduler.py reschedules another one-off
check every VERIFY_POLL_MINUTES (default 5) until automation's
check_completion() reports "complete", up to VERIFY_MAX_ATTEMPTS
(default 12, ~1 hour) before giving up and marking "not_confirmed" for
good. verify_run() is split into a pure check_completion() (no storage
side effects) so scheduler.py's loop owns the retry/give-up decision;
automation.py just answers "is it done yet".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
Status now reads "Sent" when the confirmation click lands, since that
only means MassPrice accepted the run request, not that it finished —
their own processing is async and takes anywhere from ~1 to ~22
minutes in observed practice. A new "Complete" column tracks that
separately: run_rule() captures a watermark (the target rule's current
top data-activitylogid on /Activity/List, not the displayed timestamp
string, to sidestep fragile date/timezone parsing) before submitting,
and scheduler.py schedules a one-off verify_run() job
VERIFY_DELAY_MINUTES later to check whether a new "Complete" entry
appeared past that watermark.
rebuild_jobs() now only clears recurring "rule-*" jobs, not the one-off
"verify-*" jobs, so editing a schedule can't cancel a verification
that's already in flight for an earlier run.
Also scopes the Activity page to the last 24 hours per request.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
The user reported runs logged as "success" locally weren't showing up
in MassPrice's own Activity Log. Live-debugged directly against
production: clicking the dropdown's "Run Rule" item opens a
confirmation modal ("Are you sure you want to continue?") rather than
running the rule immediately. Confirmed by network capture — without
clicking the modal's #Confirmation_Run button, zero requests fire, no
error is thrown, and the rule silently never runs. Every prior
"success" in run_history.json for real production traffic was
therefore a false positive.
Verified the fix live: clicking through #ConfirmationModal's
#Confirmation_Run now fires a real POST /Rule/Run request.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
Selects all 7 day toggles in one click instead of clicking each one.
Scoped selectors to .day-toggle[data-day] throughout so the new button
(which has no data-day) can't accidentally get read as a day or toggled
by the generic day-button click handler.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
The America/Phoenix TZ fix earlier only passed timezone= to
BackgroundScheduler's constructor. That wasn't enough: APScheduler's
CronTrigger independently calls tzlocal's get_localzone() in its own
__init__ when not given an explicit timezone, regardless of what the
scheduler itself is configured with. This never surfaced in testing
because schedules.json was always empty — the rebuild_jobs() loop that
constructs CronTrigger never executed. It crashed for real once a real
schedule existed (confirmed live: the already-configured "Pokemon
Singles >10" 4x/day schedule crash-looped the container on every boot).
Fixed by passing the same resolved timezone to every CronTrigger call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
Phoenix doesn't observe DST, so schedule times stay meaningful
year-round without needing a seasonal adjustment.
notify.py reuses lunch-notifier's existing Gmail-SMTP-with-app-password
pattern rather than adding a new notification channel. The alert fires
exactly once per expiry (not on every failed scheduled run while it's
down) via a dedupe flag in notify_state.json, reset the moment a fresh
session is successfully imported.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
Live testing showed the CDP approach still fails hCaptcha even with a
real human clicking through chrome://inspect: the headless Chromium's
own fingerprint (HeadlessChrome user-agent, navigator.webdriver=true)
is what gets flagged, independent of who's driving it. Trying to
disguise that browser to get past bot detection would be captcha
evasion, not a session-handling technique, so that path is dropped
entirely rather than patched further.
New flow: log into MassPrice with an ordinary, non-automated browser
(hCaptcha passes normally), export the session's cookies with a
extension, and paste them into the Session page. import_cookies()
converts the export into Playwright's storage_state shape and verifies
it actually authenticates against the live site before saving it,
so a bad paste never clobbers a working session. This also removes the
now-unnecessary socat proxy / debug port exposure — Chromium's own
CDP port was only ever needed for the dropped approach.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
Live testing against the real site found MassPrice's login is behind
hCaptcha, which silently blocks fully automated username/password
login outright (the form never submits) — no selector fix could work
around this. Also fixed the broken _is_logged_out() heuristic: an
unauthenticated visit to /Rule/List redirects to the public marketing
homepage, not a /login URL, so run_rule/list_rules were silently
scraping the wrong page instead of erroring.
New flow: a Session page lets the user start a headless-but-CDP-exposed
Chromium instance, log in themselves via chrome://inspect (solving the
captcha as a real click), then capture that browser's cookies into the
same auth_state.json scheduled runs already use. Required a socat proxy
in the container image, since Chromium's debug port binds loopback-only
regardless of --remote-debugging-address (confirmed by live testing),
unreachable through Docker's port mapping without it. All Playwright
objects for this now live on one dedicated thread (automation.py's
_debug_worker), since the sync API isn't safe to call across the
arbitrary threads gunicorn's gthread workers hand out per request.
Dashboard now shows a banner when the latest run failed specifically
due to a missing/expired session, rather than just looking like an
empty rule list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
arr-summary's own docs say 5055, but it's really deployed on 5056
(confirmed via docker ps on the Unraid box), which collided with
massprice's original default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
Drives massprice.tcgplayer.com's Run Rule action per-rule on a
user-configured schedule (multiple times/day, day-of-week toggles),
replacing MassPrice's native scheduler (capped at 15 schedules
account-wide, one run/day, with observed clock drift). Deployed via
Forgejo push-to-deploy on Unraid, matching arr-summary/card-grader.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
