ninja_hippo/Massprice
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Replace credential auto-login with a Chrome-remote-debugging session capture flow
All checks were successful
Deploy / deploy (push) Successful in 38s
Details

Browse source 
Live testing against the real site found MassPrice's login is behind
hCaptcha, which silently blocks fully automated username/password
login outright (the form never submits) — no selector fix could work
around this. Also fixed the broken _is_logged_out() heuristic: an
unauthenticated visit to /Rule/List redirects to the public marketing
homepage, not a /login URL, so run_rule/list_rules were silently
scraping the wrong page instead of erroring.

New flow: a Session page lets the user start a headless-but-CDP-exposed
Chromium instance, log in themselves via chrome://inspect (solving the
captcha as a real click), then capture that browser's cookies into the
same auth_state.json scheduled runs already use. Required a socat proxy
in the container image, since Chromium's debug port binds loopback-only
regardless of --remote-debugging-address (confirmed by live testing),
unreachable through Docker's port mapping without it. All Playwright
objects for this now live on one dedicated thread (automation.py's
_debug_worker), since the sync API isn't safe to call across the
arbitrary threads gunicorn's gthread workers hand out per request.

Dashboard now shows a banner when the latest run failed specifically
due to a missing/expired session, rather than just looking like an
empty rule list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
This commit is contained in:
Barely Removable 2026-08-31 19:55:08 -07:00
parent cb027167a7
commit bec399dd11
12 changed files with 473 additions and 121 deletions
Download patch file Download diff file Expand all files Collapse all files

251
automation.py
View file

@ -5,21 +5,32 @@ is bound to a JS/AJAX click handler with no plain href, and the request
it fires is almost certainly anti-forgery-token protected.
Session handling: a Playwright `storage_state` (cookies + localStorage)
is persisted to AUTH_STATE_PATH and reused across runs, so the tool logs
in once and slides the session forward on every successful run rather
than hitting the login form every time. A fresh login (via
MASSPRICE_USERNAME/MASSPRICE_PASSWORD) only happens when the saved state
is missing or has expired.
is persisted to AUTH_STATE_PATH and reused across runs, so scheduled runs
never touch the login form at all. That's deliberate, not just an
optimization — MassPrice's real login (on store.tcgplayer.com, TCGplayer's
shared OAuth identity provider) is behind hCaptcha, confirmed by live
testing: filling the real username/password and submitting silently
never navigates away, because the invisible captcha check never resolves
for a headless/automated browser. **There is no way to fully automate
logging in.** A human has to solve that captcha at least once per
session lifetime, via the "Chrome remote debugging" capture flow below.
NOTE — the real logged-out login form was not inspected during
development (the account was already signed in during discovery, and
forcing a logout to inspect it would have disrupted the user's real
MassPrice session). The locators in _login() are written to be resilient
(role/label/placeholder based, not brittle CSS) but may need a one-time
adjustment against the live form on first deploy. See CLAUDE.md.
Capture flow: `start_debug_session()` launches a headless-but-CDP-exposed
Chromium instance (`--remote-debugging-port`), which the user connects to
from their own Chrome via chrome://inspect to log in interactively —
solving the hCaptcha themselves, since it's a real click. Once logged in,
`capture_debug_session()` saves that browser's cookies to
AUTH_STATE_PATH. All of this must happen on a single dedicated thread —
Playwright's sync API is not safe to call from arbitrary threads, and
Flask/gunicorn's gthread workers hand each request to a different one —
so `_debug_worker()` owns every Playwright object here and talks to
Flask request threads only through `_debug_cmd_queue`.
"""
import logging
import os
import queue
import subprocess
import threading
from datetime import datetime, timezone
from playwright.sync_api import sync_playwright
@ -31,8 +42,19 @@ logger = logging.getLogger("massprice.automation")
BASE_URL = "https://massprice.tcgplayer.com"
AUTH_STATE_PATH = os.path.join(storage.DATA_DIR, "auth_state.json")
MASSPRICE_USERNAME = os.environ.get("MASSPRICE_USERNAME")
MASSPRICE_PASSWORD = os.environ.get("MASSPRICE_PASSWORD")
# Chromium's remote-debugging port binds to 127.0.0.1 only inside the
# container, no matter what --remote-debugging-address is set to —
# deliberate Chromium security hardening (confirmed by live testing, not
# a flag mistake). A loopback-only listener is unreachable through
# Docker's port mapping, so `socat` re-exposes it on all interfaces at a
# second, different internal port (they can't share one — Chromium's
# 127.0.0.1 bind and a 0.0.0.0 bind on the same port conflict). Neither
# of these is the DEBUG_PORT env var app.py reads for display — that one
# is the user-facing HOST port for chrome://inspect instructions, mapped
# in docker-compose.yml to _PROXY_PORT, not to Chromium's own port.
_INTERNAL_DEBUG_PORT = 9222
_PROXY_PORT = 9333
DEBUG_SESSION_TIMEOUT = 600 # auto-close an un-captured debug session after 10 minutes
# Standard, necessary mitigations for headless Chromium inside Docker.
LAUNCH_ARGS = ["--no-sandbox", "--disable-dev-shm-usage"]
@ -54,54 +76,28 @@ def _save_state(context):
def _is_logged_out(page):
if "login" in page.url.lower() or "signin" in page.url.lower():
return True
return page.locator('input[type="password"]').count() > 0
def _login(page):
if not MASSPRICE_USERNAME or not MASSPRICE_PASSWORD:
raise AutomationError(
"MASSPRICE_USERNAME/MASSPRICE_PASSWORD not set — cannot log in"
)
page.goto(f"{BASE_URL}/Dashboard")
if not _is_logged_out(page):
return
email_field = (
page.get_by_label("Email", exact=False)
.or_(page.get_by_placeholder("Email", exact=False))
.or_(page.locator('input[type="email"]'))
.or_(page.locator('input[name="username" i]'))
)
password_field = page.locator('input[type="password"]')
email_field.first.fill(MASSPRICE_USERNAME)
password_field.first.fill(MASSPRICE_PASSWORD)
submit = (
page.get_by_role("button", name="Log In", exact=False)
.or_(page.get_by_role("button", name="Sign In", exact=False))
.or_(page.locator('button[type="submit"]'))
)
submit.first.click()
page.wait_for_load_state("networkidle")
if _is_logged_out(page):
raise AutomationError(
"Login failed — still on a login page after submit "
"(bad credentials, or a CAPTCHA/2FA challenge that automated "
"login cannot get past — see CLAUDE.md)"
)
"""An unauthenticated visit to any /Rule/* page silently redirects to
the public marketing homepage (`/`) — there's no `/login` URL and no
password field to detect, so "did we get bounced off of /Rule" is the
only reliable signal (verified by live inspection)."""
return "/Rule" not in page.url
def _ensure_authenticated(page):
page.goto(f"{BASE_URL}/Rule/List")
if _is_logged_out(page):
_login(page)
page.goto(f"{BASE_URL}/Rule/List")
if _is_logged_out(page):
raise AutomationError("Still logged out after a login attempt")
raise AutomationError(
"MassPrice session expired or missing — log in via the "
"Session page (chrome://inspect) and click Capture Session."
)
def session_status():
if not os.path.exists(AUTH_STATE_PATH):
return {"state": "missing", "saved_at": None}
mtime = os.path.getmtime(AUTH_STATE_PATH)
saved_at = datetime.fromtimestamp(mtime, tz=timezone.utc).isoformat()
return {"state": "saved", "saved_at": saved_at}
def list_rules():
@ -200,3 +196,146 @@ def _log_history(rule_id, rule_name, status, message):
},
)
storage.save("run_history.json", history[:200])
# ── Interactive login-session capture (Chrome remote debugging) ─────────
#
# Everything below runs exclusively on `_debug_worker`'s thread. Flask
# routes call start_debug_session()/capture_debug_session()/
# stop_debug_session(), which just enqueue a command and block for the
# result — never touch the Playwright objects directly.
_debug_cmd_queue: "queue.Queue" = queue.Queue()
_debug_thread = None
_debug_active = False # read from other threads; plain bool assignment is atomic in CPython
def _debug_worker():
global _debug_active
playwright = browser = context = page = timer = socat_proc = None
def do_stop():
nonlocal playwright, browser, context, page, timer, socat_proc
global _debug_active
if timer:
timer.cancel()
timer = None
if socat_proc:
try:
socat_proc.terminate()
except Exception:
pass
socat_proc = None
if browser:
try:
browser.close()
except Exception:
pass
if playwright:
try:
playwright.stop()
except Exception:
pass
playwright = browser = context = page = None
_debug_active = False
while True:
cmd, result_q = _debug_cmd_queue.get()
try:
if cmd == "start":
if browser is None:
playwright = sync_playwright().start()
browser = playwright.chromium.launch(
headless=True,
args=LAUNCH_ARGS
+ [
f"--remote-debugging-port={_INTERNAL_DEBUG_PORT}",
"--remote-allow-origins=*",
],
)
# Re-expose Chromium's loopback-only debug port on
# all interfaces so Docker's port mapping (which
# forwards to _PROXY_PORT, not Chromium's own port —
# see the comment above) can actually reach it.
socat_proc = subprocess.Popen(
[
"socat",
f"TCP-LISTEN:{_PROXY_PORT},fork,reuseaddr",
f"TCP:127.0.0.1:{_INTERNAL_DEBUG_PORT}",
]
)
context = _new_context(browser)
page = context.new_page()
page.goto(f"{BASE_URL}/")
_debug_active = True
timer = threading.Timer(
DEBUG_SESSION_TIMEOUT, lambda: _debug_cmd_queue.put(("stop", None))
)
timer.daemon = True
timer.start()
if result_q:
result_q.put(("ok", None))
elif cmd == "capture":
if context is None:
if result_q:
result_q.put(
("error", "No active login session — click Start Login Session first")
)
else:
page.goto(f"{BASE_URL}/Rule/List")
page.wait_for_timeout(1000)
if _is_logged_out(page):
if result_q:
result_q.put(
(
"error",
"Still not logged in — finish logging in via "
"chrome://inspect, then try Capture Session again",
)
)
else:
_save_state(context)
do_stop()
if result_q:
result_q.put(("ok", None))
elif cmd == "stop":
do_stop()
if result_q:
result_q.put(("ok", None))
except Exception as exc:
logger.exception("debug session command %r failed", cmd)
if result_q:
result_q.put(("error", str(exc)))
def _ensure_debug_thread():
global _debug_thread
if _debug_thread is None or not _debug_thread.is_alive():
_debug_thread = threading.Thread(target=_debug_worker, daemon=True)
_debug_thread.start()
def _debug_call(cmd, timeout=30):
result_q: "queue.Queue" = queue.Queue()
_debug_cmd_queue.put((cmd, result_q))
status, payload = result_q.get(timeout=timeout)
if status == "error":
raise AutomationError(payload)
def start_debug_session():
_ensure_debug_thread()
_debug_call("start")
def capture_debug_session():
_debug_call("capture")
def stop_debug_session():
_ensure_debug_thread()
_debug_call("stop")
def debug_session_active():
return _debug_active