ninja_hippo/Massprice
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Replace Chrome-remote-debugging login capture with a cookie-paste import flow
All checks were successful
Deploy / deploy (push) Successful in 25s
Details

Browse source 
Live testing showed the CDP approach still fails hCaptcha even with a
real human clicking through chrome://inspect: the headless Chromium's
own fingerprint (HeadlessChrome user-agent, navigator.webdriver=true)
is what gets flagged, independent of who's driving it. Trying to
disguise that browser to get past bot detection would be captcha
evasion, not a session-handling technique, so that path is dropped
entirely rather than patched further.

New flow: log into MassPrice with an ordinary, non-automated browser
(hCaptcha passes normally), export the session's cookies with a
extension, and paste them into the Session page. import_cookies()
converts the export into Playwright's storage_state shape and verifies
it actually authenticates against the live site before saving it,
so a bad paste never clobbers a working session. This also removes the
now-unnecessary socat proxy / debug port exposure — Chromium's own
CDP port was only ever needed for the dropped approach.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
This commit is contained in:
Barely Removable 2026-08-31 20:10:51 -07:00
parent bec399dd11
commit 48d8bac332
10 changed files with 214 additions and 311 deletions
Download patch file Download diff file Expand all files Collapse all files

269
automation.py
View file

@ -6,31 +6,32 @@ it fires is almost certainly anti-forgery-token protected.
Session handling: a Playwright `storage_state` (cookies + localStorage)
is persisted to AUTH_STATE_PATH and reused across runs, so scheduled runs
never touch the login form at all. That's deliberate, not just an
never touch the login form at all. That's a hard requirement, not just an
optimization — MassPrice's real login (on store.tcgplayer.com, TCGplayer's
shared OAuth identity provider) is behind hCaptcha, confirmed by live
testing: filling the real username/password and submitting silently
never navigates away, because the invisible captcha check never resolves
for a headless/automated browser. **There is no way to fully automate
logging in.** A human has to solve that captcha at least once per
session lifetime, via the "Chrome remote debugging" capture flow below.
shared OAuth identity provider) is behind hCaptcha. Two automated
approaches were tried and confirmed broken during development:
1. Filling the real username/password and submitting — the invisible
captcha check never resolves for a headless/automated browser, so
the form silently never navigates.
2. Driving a real human's clicks through a headless Chromium exposed
via Chrome remote debugging (chrome://inspect) — still fails, because
that Chromium's fingerprint (a `HeadlessChrome` user-agent,
`navigator.webdriver = true`) is itself what hCaptcha flags,
independent of who's actually clicking.
Deliberately not pursued further: trying to disguise that browser to get
past hCaptcha's bot detection is captcha-evasion, not a session-handling
technique, regardless of how legitimate the underlying goal is.
Capture flow: `start_debug_session()` launches a headless-but-CDP-exposed
Chromium instance (`--remote-debugging-port`), which the user connects to
from their own Chrome via chrome://inspect to log in interactively —
solving the hCaptcha themselves, since it's a real click. Once logged in,
`capture_debug_session()` saves that browser's cookies to
AUTH_STATE_PATH. All of this must happen on a single dedicated thread —
Playwright's sync API is not safe to call from arbitrary threads, and
Flask/gunicorn's gthread workers hand each request to a different one —
so `_debug_worker()` owns every Playwright object here and talks to
Flask request threads only through `_debug_cmd_queue`.
What actually works: the user logs into MassPrice in their own ordinary,
non-automated Chrome (hCaptcha passes the way it always does for a real
person), exports that session's cookies with a browser extension, and
pastes them into the Session page. `import_cookies()` below converts that
paste into Playwright's storage_state format and verifies it actually
authenticates before accepting it.
"""
import json
import logging
import os
import queue
import subprocess
import threading
from datetime import datetime, timezone
from playwright.sync_api import sync_playwright
@ -42,23 +43,18 @@ logger = logging.getLogger("massprice.automation")
BASE_URL = "https://massprice.tcgplayer.com"
AUTH_STATE_PATH = os.path.join(storage.DATA_DIR, "auth_state.json")
# Chromium's remote-debugging port binds to 127.0.0.1 only inside the
# container, no matter what --remote-debugging-address is set to —
# deliberate Chromium security hardening (confirmed by live testing, not
# a flag mistake). A loopback-only listener is unreachable through
# Docker's port mapping, so `socat` re-exposes it on all interfaces at a
# second, different internal port (they can't share one — Chromium's
# 127.0.0.1 bind and a 0.0.0.0 bind on the same port conflict). Neither
# of these is the DEBUG_PORT env var app.py reads for display — that one
# is the user-facing HOST port for chrome://inspect instructions, mapped
# in docker-compose.yml to _PROXY_PORT, not to Chromium's own port.
_INTERNAL_DEBUG_PORT = 9222
_PROXY_PORT = 9333
DEBUG_SESSION_TIMEOUT = 600 # auto-close an un-captured debug session after 10 minutes
# Standard, necessary mitigations for headless Chromium inside Docker.
LAUNCH_ARGS = ["--no-sandbox", "--disable-dev-shm-usage"]
# Chrome's internal sameSite cookie values (what export extensions like
# Cookie-Editor report) mapped to what Playwright's storage_state expects.
_SAME_SITE_MAP = {
"no_restriction": "None",
"lax": "Lax",
"strict": "Strict",
"unspecified": "Lax",
}
class AutomationError(Exception):
pass
@ -88,7 +84,7 @@ def _ensure_authenticated(page):
if _is_logged_out(page):
raise AutomationError(
"MassPrice session expired or missing — log in via the "
"Session page (chrome://inspect) and click Capture Session."
"Session page and paste in a fresh cookie export."
)
@ -198,144 +194,75 @@ def _log_history(rule_id, rule_name, status, message):
storage.save("run_history.json", history[:200])
# ── Interactive login-session capture (Chrome remote debugging) ─────────
#
# Everything below runs exclusively on `_debug_worker`'s thread. Flask
# routes call start_debug_session()/capture_debug_session()/
# stop_debug_session(), which just enqueue a command and block for the
# result — never touch the Playwright objects directly.
_debug_cmd_queue: "queue.Queue" = queue.Queue()
_debug_thread = None
_debug_active = False # read from other threads; plain bool assignment is atomic in CPython
# ── Login-session import (paste cookies exported from a real browser) ───
def _debug_worker():
global _debug_active
playwright = browser = context = page = timer = socat_proc = None
def _convert_cookie(raw):
"""Convert one cookie from a Cookie-Editor-style export into
Playwright's storage_state cookie shape."""
expires = raw.get("expirationDate")
same_site_raw = (raw.get("sameSite") or "unspecified").lower()
return {
"name": raw["name"],
"value": raw["value"],
"domain": raw["domain"],
"path": raw.get("path", "/"),
"expires": expires if expires is not None else -1,
"httpOnly": bool(raw.get("httpOnly", False)),
"secure": bool(raw.get("secure", False)),
"sameSite": _SAME_SITE_MAP.get(same_site_raw, "Lax"),
}
def do_stop():
nonlocal playwright, browser, context, page, timer, socat_proc
global _debug_active
if timer:
timer.cancel()
timer = None
if socat_proc:
try:
socat_proc.terminate()
except Exception:
pass
socat_proc = None
if browser:
try:
browser.close()
except Exception:
pass
if playwright:
try:
playwright.stop()
except Exception:
pass
playwright = browser = context = page = None
_debug_active = False
while True:
cmd, result_q = _debug_cmd_queue.get()
def import_cookies(raw_text):
"""Parse a pasted cookie export, convert it, verify it actually
authenticates against MassPrice, and only then save it as the new
session. Raises AutomationError with a user-facing message on any
failure; never overwrites a working session with a bad paste."""
raw_text = (raw_text or "").strip()
if not raw_text:
raise AutomationError("Paste your exported cookies first — the box was empty")
try:
parsed = json.loads(raw_text)
except json.JSONDecodeError:
raise AutomationError(
"Couldn't parse that as JSON — make sure you copied the full "
"export from the cookie extension"
)
if not isinstance(parsed, list) or not parsed:
raise AutomationError(
"Expected a JSON array of cookies (Cookie-Editor's export format) — "
"got something else"
)
relevant = [c for c in parsed if "tcgplayer.com" in (c.get("domain") or "")]
if not relevant:
raise AutomationError(
"None of the pasted cookies are for tcgplayer.com — make sure you "
"exported while on the massprice.tcgplayer.com tab, after logging in"
)
try:
cookies = [_convert_cookie(c) for c in relevant]
except KeyError as exc:
raise AutomationError(f"A pasted cookie was missing its {exc} field")
candidate_state = {"cookies": cookies, "origins": []}
with sync_playwright() as p:
browser = p.chromium.launch(headless=True, args=LAUNCH_ARGS)
try:
if cmd == "start":
if browser is None:
playwright = sync_playwright().start()
browser = playwright.chromium.launch(
headless=True,
args=LAUNCH_ARGS
+ [
f"--remote-debugging-port={_INTERNAL_DEBUG_PORT}",
"--remote-allow-origins=*",
],
)
# Re-expose Chromium's loopback-only debug port on
# all interfaces so Docker's port mapping (which
# forwards to _PROXY_PORT, not Chromium's own port —
# see the comment above) can actually reach it.
socat_proc = subprocess.Popen(
[
"socat",
f"TCP-LISTEN:{_PROXY_PORT},fork,reuseaddr",
f"TCP:127.0.0.1:{_INTERNAL_DEBUG_PORT}",
]
)
context = _new_context(browser)
page = context.new_page()
page.goto(f"{BASE_URL}/")
_debug_active = True
timer = threading.Timer(
DEBUG_SESSION_TIMEOUT, lambda: _debug_cmd_queue.put(("stop", None))
)
timer.daemon = True
timer.start()
if result_q:
result_q.put(("ok", None))
elif cmd == "capture":
if context is None:
if result_q:
result_q.put(
("error", "No active login session — click Start Login Session first")
)
else:
page.goto(f"{BASE_URL}/Rule/List")
page.wait_for_timeout(1000)
if _is_logged_out(page):
if result_q:
result_q.put(
(
"error",
"Still not logged in — finish logging in via "
"chrome://inspect, then try Capture Session again",
)
)
else:
_save_state(context)
do_stop()
if result_q:
result_q.put(("ok", None))
elif cmd == "stop":
do_stop()
if result_q:
result_q.put(("ok", None))
except Exception as exc:
logger.exception("debug session command %r failed", cmd)
if result_q:
result_q.put(("error", str(exc)))
def _ensure_debug_thread():
global _debug_thread
if _debug_thread is None or not _debug_thread.is_alive():
_debug_thread = threading.Thread(target=_debug_worker, daemon=True)
_debug_thread.start()
def _debug_call(cmd, timeout=30):
result_q: "queue.Queue" = queue.Queue()
_debug_cmd_queue.put((cmd, result_q))
status, payload = result_q.get(timeout=timeout)
if status == "error":
raise AutomationError(payload)
def start_debug_session():
_ensure_debug_thread()
_debug_call("start")
def capture_debug_session():
_debug_call("capture")
def stop_debug_session():
_ensure_debug_thread()
_debug_call("stop")
def debug_session_active():
return _debug_active
context = browser.new_context(storage_state=candidate_state)
page = context.new_page()
page.goto(f"{BASE_URL}/Rule/List")
if _is_logged_out(page):
raise AutomationError(
"Imported cookies didn't authenticate — still logged out. "
"Make sure you exported them from massprice.tcgplayer.com "
"itself, after finishing login, not from the login page."
)
_save_state(context)
finally:
browser.close()