Replace credential auto-login with a Chrome-remote-debugging session capture flow
All checks were successful
Deploy / deploy (push) Successful in 38s
All checks were successful
Deploy / deploy (push) Successful in 38s
Live testing against the real site found MassPrice's login is behind hCaptcha, which silently blocks fully automated username/password login outright (the form never submits) — no selector fix could work around this. Also fixed the broken _is_logged_out() heuristic: an unauthenticated visit to /Rule/List redirects to the public marketing homepage, not a /login URL, so run_rule/list_rules were silently scraping the wrong page instead of erroring. New flow: a Session page lets the user start a headless-but-CDP-exposed Chromium instance, log in themselves via chrome://inspect (solving the captcha as a real click), then capture that browser's cookies into the same auth_state.json scheduled runs already use. Required a socat proxy in the container image, since Chromium's debug port binds loopback-only regardless of --remote-debugging-address (confirmed by live testing), unreachable through Docker's port mapping without it. All Playwright objects for this now live on one dedicated thread (automation.py's _debug_worker), since the sync API isn't safe to call across the arbitrary threads gunicorn's gthread workers hand out per request. Dashboard now shows a banner when the latest run failed specifically due to a missing/expired session, rather than just looking like an empty rule list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
This commit is contained in:
Barely Removable
parent
cb027167a7
commit
bec399dd11
12 changed files with 473 additions and 121 deletions
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: deploy-massprice
|
||||
description: Use when deploying MassPrice Scheduler, restarting the massprice container on Unraid, confirming a deploy landed, or troubleshooting its scheduled-rule automation.
|
||||
description: Use when deploying MassPrice Scheduler, restarting the massprice container on Unraid, confirming a deploy landed, logging into MassPrice via the Session page, or troubleshooting its scheduled-rule automation.
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
|
@ -9,15 +9,14 @@ description: Use when deploying MassPrice Scheduler, restarting the massprice co
|
|||
built from `Dockerfile`/`docker-compose.yml` in this repo. Container name
|
||||
`massprice`, port `5057` (overridable via `HOST_PORT` — `5056` was
|
||||
already taken by `arr-summary`, despite its own docs saying `5055`) →
|
||||
container port
|
||||
`5000`.
|
||||
container port `5000`. Also exposes `DEBUG_PORT` (default `9222`) for the
|
||||
Chrome-remote-debugging login flow — see below.
|
||||
|
||||
**Deploy mechanism: Forgejo push-to-deploy**, the same pattern as
|
||||
`arr-summary`, `card-grader`, `lunch-notifier`, `n64-tracker` — see
|
||||
`arr-summary`'s `deploy-arr-summary` skill or `card-grader`'s `CLAUDE.md`
|
||||
for the full runner mechanics. Push to `main` on the `forgejo` remote
|
||||
(`https://git.hippofam.com/ninja_hippo/massprice.git` — create this repo
|
||||
on the Forgejo instance first if it doesn't exist yet) triggers
|
||||
(`https://git.hippofam.com/ninja_hippo/massprice.git`) triggers
|
||||
`.forgejo/workflows/deploy.yml`, which syncs into
|
||||
`/mnt/user/appdata/massprice` (excluding `.git`, `.env`, `data/`) and
|
||||
runs `docker compose up -d --build`.
|
||||
|
|
@ -40,41 +39,41 @@ cd /mnt/user/appdata/massprice && docker compose restart
|
|||
|
||||
1. Create `/mnt/user/appdata/massprice` on Unraid and a real `.env`
|
||||
there (copy `.env.example`, fill in `LOGIN_PASSWORD`, `SECRET_KEY`,
|
||||
`MASSPRICE_USERNAME`, `MASSPRICE_PASSWORD`, `TZ`).
|
||||
`HOST_PORT`, `DEBUG_PORT`, `TZ`).
|
||||
2. First `docker compose up -d --build` — confirm the container starts
|
||||
and `/login` responds on `HOST_PORT`.
|
||||
3. Log into the webui, click "Refresh Rules" — confirms Playwright can
|
||||
reach massprice.tcgplayer.com and log in automatically. **If this
|
||||
fails**, check `data/run_history.json`/container logs for the login
|
||||
error — the login form locators in `automation._login()` were written
|
||||
generically (never verified against the real logged-out form, see
|
||||
`CLAUDE.md`) and may need adjusting to match TCGplayer's actual field
|
||||
names/labels.
|
||||
4. Set up at least one schedule and use "Run Now" once to confirm a
|
||||
3. Log into the webui, go to the **Session** page, and follow its
|
||||
instructions: click "Start Login Session", connect your own Chrome to
|
||||
`<unraid-ip>:<DEBUG_PORT>` via `chrome://inspect`, log in (solving the
|
||||
hCaptcha yourself — this cannot be automated, see `CLAUDE.md`), then
|
||||
click "Capture Session" back in the webui.
|
||||
4. Click "Refresh Rules" on the Dashboard — confirms the saved session
|
||||
actually works and pulls the real rule list.
|
||||
5. Set up at least one schedule and use "Run Now" once to confirm a
|
||||
click actually lands — cross-check by looking at MassPrice's own
|
||||
Activity Log (`https://massprice.tcgplayer.com/Activity/List`) for a
|
||||
fresh timestamp on that rule.
|
||||
5. **Only after step 4 succeeds**, go to each of the 5 existing rules on
|
||||
6. **Only after step 5 succeeds**, go to each existing rule on
|
||||
massprice.tcgplayer.com (three-dot menu → Edit Schedule) and set
|
||||
Status to Inactive, so native scheduling stops double-triggering runs
|
||||
and this tool becomes the sole source of scheduled runs.
|
||||
|
||||
## Persistent state, never touched by deploys
|
||||
|
||||
`.env` holds `LOGIN_PASSWORD`, `SECRET_KEY`, and the MassPrice
|
||||
credentials. `data/` (bind-mounted to `/data`) holds `schedules.json`
|
||||
(the schedules set from the webui), `auth_state.json` (the saved
|
||||
MassPrice browser session — losing this just costs one re-login, not
|
||||
data), `run_history.json`, and `rules_cache.json`. Both are excluded
|
||||
from the deploy rsync (which runs `--delete`) and from git.
|
||||
`.env` holds `LOGIN_PASSWORD` and `SECRET_KEY`. `data/` (bind-mounted to
|
||||
`/data`) holds `schedules.json` (the schedules set from the webui),
|
||||
`auth_state.json` (the saved MassPrice browser session captured via the
|
||||
Session page — losing this just costs one re-login, not data),
|
||||
`run_history.json`, and `rules_cache.json`. Both are excluded from the
|
||||
deploy rsync (which runs `--delete`) and from git.
|
||||
|
||||
## If scheduled runs stop working
|
||||
|
||||
Check, in order:
|
||||
1. **Activity page in the webui** — does the failure message say
|
||||
"still on a login page" (credentials/CAPTCHA problem) or "rule id not
|
||||
found" (a rule was renamed/deleted on MassPrice — click "Refresh
|
||||
Rules")?
|
||||
1. **Dashboard banner / Activity page** — a "session expired or missing"
|
||||
message means go log in again via the Session page. "rule id not
|
||||
found" means a rule was renamed/deleted on MassPrice — click "Refresh
|
||||
Rules".
|
||||
2. **Is the container still running** — `docker ps --filter
|
||||
name=massprice` — a Chromium crash inside the container (rare, but
|
||||
possible if `shm_size` ever gets dropped from `docker-compose.yml`)
|
||||
|
|
@ -82,3 +81,8 @@ Check, in order:
|
|||
3. **Timezone** — `TZ` in `.env` controls what "6:00 AM" means for every
|
||||
schedule; confirm it matches what the user actually expects before
|
||||
assuming a schedule itself is wrong.
|
||||
4. **Session capture stuck at "Still not logged in"** — the debug
|
||||
session's page needs to actually be on `/Rule/List` or somewhere under
|
||||
`/Rule` after login for the capture check to pass; if the inspected
|
||||
tab is still sitting on the marketing homepage or the OAuth login
|
||||
page, finish clicking through to the dashboard/rules first.
|
||||
|
|
|
|||
13
.env.example
13
.env.example
|
|
@ -10,17 +10,16 @@ LOGIN_PASSWORD=changeme
|
|||
# python3 -c "import secrets; print(secrets.token_hex(32))"
|
||||
SECRET_KEY=change_this_to_a_random_secret_key
|
||||
|
||||
# ── MassPrice (TCGplayer) login ─────────────────────────────────────
|
||||
# Used by Playwright to log in automatically the first time, and again
|
||||
# any time the saved session expires. Stored here only — gitignored,
|
||||
# never committed, never logged.
|
||||
MASSPRICE_USERNAME=your_tcgplayer_email@example.com
|
||||
MASSPRICE_PASSWORD=your_tcgplayer_password
|
||||
|
||||
# ── Host port (what port Unraid exposes) ────────────────────────────
|
||||
# Default is 5057 (arr-summary already uses 5056) — change if in use
|
||||
HOST_PORT=5057
|
||||
|
||||
# ── Debug port (Chrome remote debugging, for logging in) ───────────
|
||||
# MassPrice's login is behind hCaptcha, so there's no way to fully
|
||||
# automate signing in. Log in via the webui's Session page instead,
|
||||
# which connects your own Chrome (chrome://inspect) to this port.
|
||||
DEBUG_PORT=9222
|
||||
|
||||
# ── Timezone ─────────────────────────────────────────────────────────
|
||||
# Controls what "6:00 AM" means for your schedules
|
||||
TZ=America/New_York
|
||||
|
|
|
|||
82
CLAUDE.md
82
CLAUDE.md
|
|
@ -52,11 +52,12 @@ tool is confirmed working, so there's exactly one thing triggering runs.
|
|||
the real page rather than replaying a raw HTTP POST.
|
||||
- **Session persistence** — a Playwright `storage_state` (cookies +
|
||||
localStorage) is saved to `/data/auth_state.json` and reused across
|
||||
runs: the tool logs in **once**, then slides the session forward after
|
||||
every successful run, rather than hitting the login form on every
|
||||
scheduled click. A fresh login (via `MASSPRICE_USERNAME` /
|
||||
`MASSPRICE_PASSWORD`) only happens when the saved state is missing or
|
||||
has expired (detected by landing back on a login page).
|
||||
runs: scheduled runs never touch the login form at all, they just load
|
||||
this saved session. When it's missing or has expired (detected by an
|
||||
unauthenticated visit to `/Rule/List` silently redirecting to the
|
||||
public marketing homepage `/` — there's no `/login` URL to check
|
||||
against), the run fails with a clear "session expired or missing"
|
||||
error rather than attempting to log in — see below for why.
|
||||
- **`storage.py`** — tiny JSON read/write helpers (temp-file +
|
||||
`os.replace`, so a crash mid-write can't corrupt state) for
|
||||
everything under `/data`: `schedules.json`, `auth_state.json`,
|
||||
|
|
@ -66,24 +67,47 @@ tool is confirmed working, so there's exactly one thing triggering runs.
|
|||
teletext system — that design language is specific to that app, not a
|
||||
house style to reuse here.
|
||||
|
||||
## Known limitation — automated login
|
||||
## Login — why it's manual, and how it works
|
||||
|
||||
`MASSPRICE_USERNAME`/`MASSPRICE_PASSWORD` drive a fully automated login
|
||||
(the user's explicit choice over a manual hand-off flow). **If
|
||||
TCGplayer's login ever presents a CAPTCHA or a 2FA challenge, automated
|
||||
login will fail outright** — there is deliberately no fallback UI for
|
||||
this yet; a failed login just shows up as a `failure` entry in Activity
|
||||
with the error message. If this starts happening routinely, it needs a
|
||||
design change (e.g. a Chrome-remote-debugging-based manual re-auth flow),
|
||||
not a bigger retry loop.
|
||||
MassPrice's login (`store.tcgplayer.com/oauth/login`, TCGplayer's shared
|
||||
OAuth identity provider) is protected by **hCaptcha**. This was
|
||||
confirmed by live testing during development: filling in the real
|
||||
username/password and clicking Submit never navigates anywhere — the
|
||||
invisible captcha check silently never resolves for a headless/automated
|
||||
browser, and the form just refuses to submit. There is no selector fix
|
||||
for this; **fully automated login is not achievable here**, full stop.
|
||||
An earlier version of this app tried `MASSPRICE_USERNAME`/
|
||||
`MASSPRICE_PASSWORD` credential-based auto-login — removed once this was
|
||||
confirmed, since keeping dead code that can never succeed is worse than
|
||||
having none.
|
||||
|
||||
**The real logged-out MassPrice login form was never inspected** during
|
||||
development — the account was already signed in when the site was
|
||||
explored, and logging it out to look at the form would have disrupted
|
||||
the user's real, live MassPrice session. `automation._login()` uses
|
||||
resilient role/label/placeholder-based Playwright locators rather than
|
||||
brittle CSS, but **expect to need a one-time adjustment against the real
|
||||
form on first deploy** if the generic locators don't match.
|
||||
What actually works: a human only needs to pass the hCaptcha, not
|
||||
automate around it, and that's a one-time (well, periodic — however long
|
||||
the session cookie lasts) cost, not a per-run one. The **Session** page
|
||||
in the webui drives this:
|
||||
1. "Start Login Session" launches headless Chromium with its CDP debug
|
||||
port exposed (`DEBUG_PORT`, default `9222`).
|
||||
2. The user connects their own Chrome to it via `chrome://inspect`,
|
||||
which renders and accepts real input against that same running
|
||||
browser instance — no VNC/Xvfb needed, headless Chromium's debug
|
||||
protocol already supports this.
|
||||
3. The user logs in normally in that inspected tab, solving the hCaptcha
|
||||
themselves since it's a real click from a real browser.
|
||||
4. "Capture Session" saves that browser's cookies to
|
||||
`/data/auth_state.json` — the same file scheduled runs read.
|
||||
|
||||
**All of this lives on one dedicated thread** (`automation._debug_worker`),
|
||||
not arbitrary Flask request threads — Playwright's sync API isn't safe
|
||||
to call across threads, and gunicorn's `gthread` workers hand each
|
||||
request to a different one. Flask routes only ever put a command on
|
||||
`_debug_cmd_queue` and block for the result; they never touch the
|
||||
Playwright browser/context/page objects directly. An un-captured debug
|
||||
session auto-closes after 10 minutes so an open, unauthenticated CDP
|
||||
port doesn't sit exposed on the LAN indefinitely.
|
||||
|
||||
The Dashboard shows a red banner (`_session_warning()` in `app.py`) when
|
||||
the most recent run/refresh failed specifically because of a missing/
|
||||
expired session, so this doesn't just look like "no rules found."
|
||||
|
||||
## Local development
|
||||
|
||||
|
|
@ -93,13 +117,13 @@ playwright install chromium # only needed if testing automation.py directly
|
|||
python3 app.py
|
||||
```
|
||||
|
||||
Runs on port 5000 (`PORT` env to change). Without `MASSPRICE_USERNAME`/
|
||||
`MASSPRICE_PASSWORD` set and without Chromium installed, the Dashboard
|
||||
still boots — `_bootstrap()` in `app.py` best-effort-catches the initial
|
||||
rule sync, so local UI/schedule-editing work doesn't require Docker or
|
||||
real MassPrice credentials at all. `DATA_DIR` env var overrides where
|
||||
JSON state is read/written (defaults to `/data`, which won't exist
|
||||
locally — point it at a scratch directory).
|
||||
Runs on port 5000 (`PORT` env to change). Without Chromium installed and
|
||||
without a saved `auth_state.json`, the Dashboard still boots —
|
||||
`_bootstrap()` in `app.py` best-effort-catches the initial rule sync, so
|
||||
local UI/schedule-editing work doesn't require Docker or a real MassPrice
|
||||
session at all. `DATA_DIR` env var overrides where JSON state is
|
||||
read/written (defaults to `/data`, which won't exist locally — point it
|
||||
at a scratch directory).
|
||||
|
||||
**Every commit should be matched with a version bump** —
|
||||
`APP_VERSION`/`APP_VERSION_DATE` near the top of `app.py` — the same
|
||||
|
|
@ -112,6 +136,6 @@ pipeline and manual container restart steps.
|
|||
|
||||
**Never put real credentials in the repo.** `.env` and `data/` are
|
||||
gitignored and excluded from the deploy rsync precisely so
|
||||
`MASSPRICE_USERNAME`/`MASSPRICE_PASSWORD`/`LOGIN_PASSWORD` and the saved
|
||||
`LOGIN_PASSWORD`/`SECRET_KEY` and the saved
|
||||
MassPrice session (`data/auth_state.json`) have somewhere to live
|
||||
outside git.
|
||||
|
|
|
|||
|
|
@ -5,6 +5,13 @@ LABEL maintainer="MassPrice Scheduler"
|
|||
|
||||
WORKDIR /app
|
||||
|
||||
# socat proxies the login-session debug port: Chromium's remote-debugging
|
||||
# port binds to 127.0.0.1 only, regardless of --remote-debugging-address
|
||||
# (deliberate Chromium security hardening, confirmed by testing) — a
|
||||
# loopback-only listener is unreachable through Docker's port mapping,
|
||||
# so socat re-exposes it on all interfaces. See automation.py.
|
||||
RUN apt-get update -qq && apt-get install -y -qq socat && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
|
|
|
|||
59
app.py
59
app.py
|
|
@ -8,8 +8,8 @@ import automation
|
|||
import scheduler
|
||||
import storage
|
||||
|
||||
APP_VERSION = "0.1.0"
|
||||
APP_VERSION_DATE = "2026-08-31"
|
||||
APP_VERSION = "0.2.0"
|
||||
APP_VERSION_DATE = "2026-09-01"
|
||||
|
||||
app = Flask(__name__)
|
||||
app.secret_key = os.environ.get("SECRET_KEY", "dev-secret-change-me")
|
||||
|
|
@ -81,6 +81,19 @@ def _merged_rules():
|
|||
return rules, cache.get("updated_at")
|
||||
|
||||
|
||||
def _session_warning():
|
||||
"""True if the most recent run/refresh failed specifically because
|
||||
the saved MassPrice session is missing/expired — surfaced as a
|
||||
dashboard banner so a dead session doesn't just look like "no rules"."""
|
||||
history = storage.load("run_history.json", [])
|
||||
if not history:
|
||||
return False
|
||||
latest = history[0]
|
||||
return latest.get("status") == "failure" and "session expired or missing" in (
|
||||
latest.get("message") or ""
|
||||
)
|
||||
|
||||
|
||||
@app.route("/")
|
||||
@login_required
|
||||
def dashboard():
|
||||
|
|
@ -91,9 +104,51 @@ def dashboard():
|
|||
rules_updated_at=rules_updated_at,
|
||||
days=DAYS,
|
||||
day_labels=DAY_LABELS,
|
||||
session_warning=_session_warning(),
|
||||
)
|
||||
|
||||
|
||||
@app.route("/session")
|
||||
@login_required
|
||||
def session_page():
|
||||
return render_template(
|
||||
"session.html",
|
||||
status=automation.session_status(),
|
||||
debug_active=automation.debug_session_active(),
|
||||
debug_port=os.environ.get("DEBUG_PORT", "9222"),
|
||||
host=request.host.split(":")[0],
|
||||
)
|
||||
|
||||
|
||||
@app.route("/api/session/start", methods=["POST"])
|
||||
@login_required
|
||||
def api_session_start():
|
||||
try:
|
||||
automation.start_debug_session()
|
||||
except Exception as exc:
|
||||
return jsonify({"error": str(exc)}), 502
|
||||
return jsonify({"ok": True})
|
||||
|
||||
|
||||
@app.route("/api/session/capture", methods=["POST"])
|
||||
@login_required
|
||||
def api_session_capture():
|
||||
try:
|
||||
automation.capture_debug_session()
|
||||
except automation.AutomationError as exc:
|
||||
return jsonify({"error": str(exc)}), 409
|
||||
except Exception as exc:
|
||||
return jsonify({"error": str(exc)}), 502
|
||||
return jsonify({"ok": True})
|
||||
|
||||
|
||||
@app.route("/api/session/stop", methods=["POST"])
|
||||
@login_required
|
||||
def api_session_stop():
|
||||
automation.stop_debug_session()
|
||||
return jsonify({"ok": True})
|
||||
|
||||
|
||||
@app.route("/activity")
|
||||
@login_required
|
||||
def activity():
|
||||
|
|
|
|||
251
automation.py
251
automation.py
|
|
@ -5,21 +5,32 @@ is bound to a JS/AJAX click handler with no plain href, and the request
|
|||
it fires is almost certainly anti-forgery-token protected.
|
||||
|
||||
Session handling: a Playwright `storage_state` (cookies + localStorage)
|
||||
is persisted to AUTH_STATE_PATH and reused across runs, so the tool logs
|
||||
in once and slides the session forward on every successful run rather
|
||||
than hitting the login form every time. A fresh login (via
|
||||
MASSPRICE_USERNAME/MASSPRICE_PASSWORD) only happens when the saved state
|
||||
is missing or has expired.
|
||||
is persisted to AUTH_STATE_PATH and reused across runs, so scheduled runs
|
||||
never touch the login form at all. That's deliberate, not just an
|
||||
optimization — MassPrice's real login (on store.tcgplayer.com, TCGplayer's
|
||||
shared OAuth identity provider) is behind hCaptcha, confirmed by live
|
||||
testing: filling the real username/password and submitting silently
|
||||
never navigates away, because the invisible captcha check never resolves
|
||||
for a headless/automated browser. **There is no way to fully automate
|
||||
logging in.** A human has to solve that captcha at least once per
|
||||
session lifetime, via the "Chrome remote debugging" capture flow below.
|
||||
|
||||
NOTE — the real logged-out login form was not inspected during
|
||||
development (the account was already signed in during discovery, and
|
||||
forcing a logout to inspect it would have disrupted the user's real
|
||||
MassPrice session). The locators in _login() are written to be resilient
|
||||
(role/label/placeholder based, not brittle CSS) but may need a one-time
|
||||
adjustment against the live form on first deploy. See CLAUDE.md.
|
||||
Capture flow: `start_debug_session()` launches a headless-but-CDP-exposed
|
||||
Chromium instance (`--remote-debugging-port`), which the user connects to
|
||||
from their own Chrome via chrome://inspect to log in interactively —
|
||||
solving the hCaptcha themselves, since it's a real click. Once logged in,
|
||||
`capture_debug_session()` saves that browser's cookies to
|
||||
AUTH_STATE_PATH. All of this must happen on a single dedicated thread —
|
||||
Playwright's sync API is not safe to call from arbitrary threads, and
|
||||
Flask/gunicorn's gthread workers hand each request to a different one —
|
||||
so `_debug_worker()` owns every Playwright object here and talks to
|
||||
Flask request threads only through `_debug_cmd_queue`.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import subprocess
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
|
@ -31,8 +42,19 @@ logger = logging.getLogger("massprice.automation")
|
|||
BASE_URL = "https://massprice.tcgplayer.com"
|
||||
AUTH_STATE_PATH = os.path.join(storage.DATA_DIR, "auth_state.json")
|
||||
|
||||
MASSPRICE_USERNAME = os.environ.get("MASSPRICE_USERNAME")
|
||||
MASSPRICE_PASSWORD = os.environ.get("MASSPRICE_PASSWORD")
|
||||
# Chromium's remote-debugging port binds to 127.0.0.1 only inside the
|
||||
# container, no matter what --remote-debugging-address is set to —
|
||||
# deliberate Chromium security hardening (confirmed by live testing, not
|
||||
# a flag mistake). A loopback-only listener is unreachable through
|
||||
# Docker's port mapping, so `socat` re-exposes it on all interfaces at a
|
||||
# second, different internal port (they can't share one — Chromium's
|
||||
# 127.0.0.1 bind and a 0.0.0.0 bind on the same port conflict). Neither
|
||||
# of these is the DEBUG_PORT env var app.py reads for display — that one
|
||||
# is the user-facing HOST port for chrome://inspect instructions, mapped
|
||||
# in docker-compose.yml to _PROXY_PORT, not to Chromium's own port.
|
||||
_INTERNAL_DEBUG_PORT = 9222
|
||||
_PROXY_PORT = 9333
|
||||
DEBUG_SESSION_TIMEOUT = 600 # auto-close an un-captured debug session after 10 minutes
|
||||
|
||||
# Standard, necessary mitigations for headless Chromium inside Docker.
|
||||
LAUNCH_ARGS = ["--no-sandbox", "--disable-dev-shm-usage"]
|
||||
|
|
@ -54,54 +76,28 @@ def _save_state(context):
|
|||
|
||||
|
||||
def _is_logged_out(page):
|
||||
if "login" in page.url.lower() or "signin" in page.url.lower():
|
||||
return True
|
||||
return page.locator('input[type="password"]').count() > 0
|
||||
|
||||
|
||||
def _login(page):
|
||||
if not MASSPRICE_USERNAME or not MASSPRICE_PASSWORD:
|
||||
raise AutomationError(
|
||||
"MASSPRICE_USERNAME/MASSPRICE_PASSWORD not set — cannot log in"
|
||||
)
|
||||
page.goto(f"{BASE_URL}/Dashboard")
|
||||
if not _is_logged_out(page):
|
||||
return
|
||||
|
||||
email_field = (
|
||||
page.get_by_label("Email", exact=False)
|
||||
.or_(page.get_by_placeholder("Email", exact=False))
|
||||
.or_(page.locator('input[type="email"]'))
|
||||
.or_(page.locator('input[name="username" i]'))
|
||||
)
|
||||
password_field = page.locator('input[type="password"]')
|
||||
|
||||
email_field.first.fill(MASSPRICE_USERNAME)
|
||||
password_field.first.fill(MASSPRICE_PASSWORD)
|
||||
|
||||
submit = (
|
||||
page.get_by_role("button", name="Log In", exact=False)
|
||||
.or_(page.get_by_role("button", name="Sign In", exact=False))
|
||||
.or_(page.locator('button[type="submit"]'))
|
||||
)
|
||||
submit.first.click()
|
||||
page.wait_for_load_state("networkidle")
|
||||
|
||||
if _is_logged_out(page):
|
||||
raise AutomationError(
|
||||
"Login failed — still on a login page after submit "
|
||||
"(bad credentials, or a CAPTCHA/2FA challenge that automated "
|
||||
"login cannot get past — see CLAUDE.md)"
|
||||
)
|
||||
"""An unauthenticated visit to any /Rule/* page silently redirects to
|
||||
the public marketing homepage (`/`) — there's no `/login` URL and no
|
||||
password field to detect, so "did we get bounced off of /Rule" is the
|
||||
only reliable signal (verified by live inspection)."""
|
||||
return "/Rule" not in page.url
|
||||
|
||||
|
||||
def _ensure_authenticated(page):
|
||||
page.goto(f"{BASE_URL}/Rule/List")
|
||||
if _is_logged_out(page):
|
||||
_login(page)
|
||||
page.goto(f"{BASE_URL}/Rule/List")
|
||||
if _is_logged_out(page):
|
||||
raise AutomationError("Still logged out after a login attempt")
|
||||
raise AutomationError(
|
||||
"MassPrice session expired or missing — log in via the "
|
||||
"Session page (chrome://inspect) and click Capture Session."
|
||||
)
|
||||
|
||||
|
||||
def session_status():
|
||||
if not os.path.exists(AUTH_STATE_PATH):
|
||||
return {"state": "missing", "saved_at": None}
|
||||
mtime = os.path.getmtime(AUTH_STATE_PATH)
|
||||
saved_at = datetime.fromtimestamp(mtime, tz=timezone.utc).isoformat()
|
||||
return {"state": "saved", "saved_at": saved_at}
|
||||
|
||||
|
||||
def list_rules():
|
||||
|
|
@ -200,3 +196,146 @@ def _log_history(rule_id, rule_name, status, message):
|
|||
},
|
||||
)
|
||||
storage.save("run_history.json", history[:200])
|
||||
|
||||
|
||||
# ── Interactive login-session capture (Chrome remote debugging) ─────────
|
||||
#
|
||||
# Everything below runs exclusively on `_debug_worker`'s thread. Flask
|
||||
# routes call start_debug_session()/capture_debug_session()/
|
||||
# stop_debug_session(), which just enqueue a command and block for the
|
||||
# result — never touch the Playwright objects directly.
|
||||
|
||||
_debug_cmd_queue: "queue.Queue" = queue.Queue()
|
||||
_debug_thread = None
|
||||
_debug_active = False # read from other threads; plain bool assignment is atomic in CPython
|
||||
|
||||
|
||||
def _debug_worker():
|
||||
global _debug_active
|
||||
playwright = browser = context = page = timer = socat_proc = None
|
||||
|
||||
def do_stop():
|
||||
nonlocal playwright, browser, context, page, timer, socat_proc
|
||||
global _debug_active
|
||||
if timer:
|
||||
timer.cancel()
|
||||
timer = None
|
||||
if socat_proc:
|
||||
try:
|
||||
socat_proc.terminate()
|
||||
except Exception:
|
||||
pass
|
||||
socat_proc = None
|
||||
if browser:
|
||||
try:
|
||||
browser.close()
|
||||
except Exception:
|
||||
pass
|
||||
if playwright:
|
||||
try:
|
||||
playwright.stop()
|
||||
except Exception:
|
||||
pass
|
||||
playwright = browser = context = page = None
|
||||
_debug_active = False
|
||||
|
||||
while True:
|
||||
cmd, result_q = _debug_cmd_queue.get()
|
||||
try:
|
||||
if cmd == "start":
|
||||
if browser is None:
|
||||
playwright = sync_playwright().start()
|
||||
browser = playwright.chromium.launch(
|
||||
headless=True,
|
||||
args=LAUNCH_ARGS
|
||||
+ [
|
||||
f"--remote-debugging-port={_INTERNAL_DEBUG_PORT}",
|
||||
"--remote-allow-origins=*",
|
||||
],
|
||||
)
|
||||
# Re-expose Chromium's loopback-only debug port on
|
||||
# all interfaces so Docker's port mapping (which
|
||||
# forwards to _PROXY_PORT, not Chromium's own port —
|
||||
# see the comment above) can actually reach it.
|
||||
socat_proc = subprocess.Popen(
|
||||
[
|
||||
"socat",
|
||||
f"TCP-LISTEN:{_PROXY_PORT},fork,reuseaddr",
|
||||
f"TCP:127.0.0.1:{_INTERNAL_DEBUG_PORT}",
|
||||
]
|
||||
)
|
||||
context = _new_context(browser)
|
||||
page = context.new_page()
|
||||
page.goto(f"{BASE_URL}/")
|
||||
_debug_active = True
|
||||
timer = threading.Timer(
|
||||
DEBUG_SESSION_TIMEOUT, lambda: _debug_cmd_queue.put(("stop", None))
|
||||
)
|
||||
timer.daemon = True
|
||||
timer.start()
|
||||
if result_q:
|
||||
result_q.put(("ok", None))
|
||||
elif cmd == "capture":
|
||||
if context is None:
|
||||
if result_q:
|
||||
result_q.put(
|
||||
("error", "No active login session — click Start Login Session first")
|
||||
)
|
||||
else:
|
||||
page.goto(f"{BASE_URL}/Rule/List")
|
||||
page.wait_for_timeout(1000)
|
||||
if _is_logged_out(page):
|
||||
if result_q:
|
||||
result_q.put(
|
||||
(
|
||||
"error",
|
||||
"Still not logged in — finish logging in via "
|
||||
"chrome://inspect, then try Capture Session again",
|
||||
)
|
||||
)
|
||||
else:
|
||||
_save_state(context)
|
||||
do_stop()
|
||||
if result_q:
|
||||
result_q.put(("ok", None))
|
||||
elif cmd == "stop":
|
||||
do_stop()
|
||||
if result_q:
|
||||
result_q.put(("ok", None))
|
||||
except Exception as exc:
|
||||
logger.exception("debug session command %r failed", cmd)
|
||||
if result_q:
|
||||
result_q.put(("error", str(exc)))
|
||||
|
||||
|
||||
def _ensure_debug_thread():
|
||||
global _debug_thread
|
||||
if _debug_thread is None or not _debug_thread.is_alive():
|
||||
_debug_thread = threading.Thread(target=_debug_worker, daemon=True)
|
||||
_debug_thread.start()
|
||||
|
||||
|
||||
def _debug_call(cmd, timeout=30):
|
||||
result_q: "queue.Queue" = queue.Queue()
|
||||
_debug_cmd_queue.put((cmd, result_q))
|
||||
status, payload = result_q.get(timeout=timeout)
|
||||
if status == "error":
|
||||
raise AutomationError(payload)
|
||||
|
||||
|
||||
def start_debug_session():
|
||||
_ensure_debug_thread()
|
||||
_debug_call("start")
|
||||
|
||||
|
||||
def capture_debug_session():
|
||||
_debug_call("capture")
|
||||
|
||||
|
||||
def stop_debug_session():
|
||||
_ensure_debug_thread()
|
||||
_debug_call("stop")
|
||||
|
||||
|
||||
def debug_session_active():
|
||||
return _debug_active
|
||||
|
|
|
|||
|
|
@ -7,6 +7,16 @@ services:
|
|||
shm_size: "1gb"
|
||||
ports:
|
||||
- "${HOST_PORT:-5057}:5000"
|
||||
# MassPrice's login is behind hCaptcha, which blocks fully
|
||||
# automated sign-in — this port reaches (via an in-container socat
|
||||
# proxy, see automation.py) headless Chromium's CDP debug endpoint,
|
||||
# so a human can log in via chrome://inspect from their own machine
|
||||
# (see the Session page / deploy-massprice skill). Maps to 9333,
|
||||
# NOT Chromium's own 9222 — Chromium's debug port binds loopback-
|
||||
# only regardless of config, so socat re-exposes it on a different
|
||||
# internal port. Only listens while a login session is actively
|
||||
# started from the webui, and only reachable on the LAN.
|
||||
- "${DEBUG_PORT:-9222}:9333"
|
||||
volumes:
|
||||
# Schedules, run history, the rule-name cache, and the saved
|
||||
# MassPrice login session must outlive `up -d --build`, which
|
||||
|
|
@ -15,7 +25,6 @@ services:
|
|||
environment:
|
||||
- LOGIN_PASSWORD=${LOGIN_PASSWORD}
|
||||
- SECRET_KEY=${SECRET_KEY}
|
||||
- MASSPRICE_USERNAME=${MASSPRICE_USERNAME}
|
||||
- MASSPRICE_PASSWORD=${MASSPRICE_PASSWORD}
|
||||
- DEBUG_PORT=${DEBUG_PORT:-9222}
|
||||
- TZ=${TZ:-America/New_York}
|
||||
restart: unless-stopped
|
||||
|
|
|
|||
|
|
@ -154,6 +154,33 @@ footer {
|
|||
|
||||
.empty-state { color: var(--text-dim); }
|
||||
|
||||
.banner {
|
||||
border-radius: var(--radius);
|
||||
padding: 12px 16px;
|
||||
margin-bottom: 20px;
|
||||
font-size: 0.88rem;
|
||||
}
|
||||
.banner-warn {
|
||||
background: rgba(248, 113, 113, 0.12);
|
||||
border: 1px solid var(--failure);
|
||||
color: #f2c2c2;
|
||||
}
|
||||
.banner a { color: var(--text); }
|
||||
|
||||
.panel {
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
.panel h2 { font-size: 1rem; margin: 0 0 12px; }
|
||||
.panel-narrow { max-width: 560px; }
|
||||
|
||||
.session-steps { padding-left: 20px; display: flex; flex-direction: column; gap: 6px; font-size: 0.88rem; }
|
||||
.session-steps code { background: var(--panel-alt); border: 1px solid var(--border); border-radius: 4px; padding: 1px 6px; }
|
||||
.session-actions { display: flex; gap: 8px; margin-top: 16px; }
|
||||
|
||||
.activity-table { width: 100%; border-collapse: collapse; font-size: 0.85rem; }
|
||||
.activity-table th, .activity-table td { text-align: left; padding: 8px 12px; border-bottom: 1px solid var(--border); }
|
||||
.activity-table th { color: var(--text-dim); font-weight: 500; font-size: 0.75rem; text-transform: uppercase; }
|
||||
|
|
|
|||
|
|
@ -21,8 +21,40 @@ document.addEventListener("DOMContentLoaded", () => {
|
|||
}
|
||||
|
||||
document.querySelectorAll(".rule-card").forEach(initRuleCard);
|
||||
initSessionControls();
|
||||
});
|
||||
|
||||
function initSessionControls() {
|
||||
const startBtn = document.getElementById("session-start");
|
||||
const captureBtn = document.getElementById("session-capture");
|
||||
const stopBtn = document.getElementById("session-stop");
|
||||
if (!startBtn) return; // not on the Session page
|
||||
|
||||
async function call(url, btn) {
|
||||
btn.disabled = true;
|
||||
const original = btn.textContent;
|
||||
btn.textContent = "Working…";
|
||||
try {
|
||||
const res = await fetch(url, { method: "POST" });
|
||||
const data = await res.json().catch(() => ({}));
|
||||
if (!res.ok) {
|
||||
alert(data.error || "Request failed");
|
||||
btn.disabled = false;
|
||||
btn.textContent = original;
|
||||
return;
|
||||
}
|
||||
window.location.reload();
|
||||
} catch (err) {
|
||||
btn.disabled = false;
|
||||
btn.textContent = original;
|
||||
}
|
||||
}
|
||||
|
||||
startBtn.addEventListener("click", () => call("/api/session/start", startBtn));
|
||||
captureBtn.addEventListener("click", () => call("/api/session/capture", captureBtn));
|
||||
stopBtn.addEventListener("click", () => call("/api/session/stop", stopBtn));
|
||||
}
|
||||
|
||||
function initRuleCard(card) {
|
||||
const ruleId = card.dataset.ruleId;
|
||||
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@
|
|||
<nav>
|
||||
<a href="{{ url_for('dashboard') }}" class="{{ 'active' if request.endpoint == 'dashboard' else '' }}">Dashboard</a>
|
||||
<a href="{{ url_for('activity') }}" class="{{ 'active' if request.endpoint == 'activity' else '' }}">Activity</a>
|
||||
<a href="{{ url_for('session_page') }}" class="{{ 'active' if request.endpoint == 'session_page' else '' }}">Session</a>
|
||||
<a href="{{ url_for('logout') }}">Log Out</a>
|
||||
</nav>
|
||||
{% endif %}
|
||||
|
|
|
|||
|
|
@ -9,6 +9,13 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
{% if session_warning %}
|
||||
<div class="banner banner-warn">
|
||||
Your MassPrice login session has expired or was never captured — scheduled runs and rule
|
||||
refreshes will fail until you log in again. <a href="{{ url_for('session_page') }}">Go to the Session page</a>.
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if not rules %}
|
||||
<p class="empty-state">No rules loaded yet. Click "Refresh Rules" to pull the current rule list from MassPrice.</p>
|
||||
{% endif %}
|
||||
|
|
|
|||
48
templates/session.html
Normal file
48
templates/session.html
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
{% extends "_shell.html" %}
|
||||
{% block title %}Session · MassPrice Scheduler{% endblock %}
|
||||
{% block content %}
|
||||
<h1>MassPrice Login Session</h1>
|
||||
|
||||
<div class="panel panel-narrow">
|
||||
<h2>Status</h2>
|
||||
{% if status.state == 'saved' %}
|
||||
<p><span class="status-dot status-success"></span>Session saved {{ status.saved_at }}</p>
|
||||
<p class="muted small">If scheduled runs start failing with a "session expired" error, come back here and log in again.</p>
|
||||
{% else %}
|
||||
<p><span class="status-dot status-failure"></span>No session saved yet — scheduled runs will fail until you log in.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<div class="panel panel-narrow">
|
||||
<h2>Log in</h2>
|
||||
<p class="muted small">
|
||||
MassPrice's login is protected by hCaptcha, which blocks fully automated
|
||||
sign-in. You need to log in yourself, at least once per session lifetime,
|
||||
using your own Chrome browser connected to a debug browser running in
|
||||
this container.
|
||||
</p>
|
||||
<ol class="session-steps">
|
||||
<li>Click <strong>Start Login Session</strong> below.</li>
|
||||
<li>On your own computer, open Chrome and go to <code>chrome://inspect/#devices</code>.</li>
|
||||
<li>Under "Discover network targets", click <strong>Configure…</strong> and add
|
||||
<code>{{ host }}:{{ debug_port }}</code>, then close the dialog.</li>
|
||||
<li>A MassPrice tab should appear under "Remote Target" — click
|
||||
<strong>inspect</strong> to open it, then log in normally (including solving the
|
||||
hCaptcha challenge yourself).</li>
|
||||
<li>Once you're looking at the MassPrice dashboard or rules list in that inspected
|
||||
tab, come back here and click <strong>Capture Session</strong>.</li>
|
||||
</ol>
|
||||
|
||||
<div class="session-actions">
|
||||
<button id="session-start" type="button" class="btn btn-primary" {{ 'disabled' if debug_active else '' }}>Start Login Session</button>
|
||||
<button id="session-capture" type="button" class="btn" {{ 'disabled' if not debug_active else '' }}>Capture Session</button>
|
||||
<button id="session-stop" type="button" class="btn" {{ 'disabled' if not debug_active else '' }}>Stop</button>
|
||||
</div>
|
||||
{% if debug_active %}
|
||||
<p class="muted small">Login session is active — it auto-closes after 10 minutes if not captured.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
{% block scripts %}
|
||||
<script src="{{ url_for('static', filename='js/app.js') }}"></script>
|
||||
{% endblock %}
|
||||
Loading…
Add table
Add a link
Reference in a new issue