All checks were successful
Deploy / deploy (push) Successful in 30s
Full read-through of the project. Grouped by what actually breaks:
Browser concurrency (the invariant the design rests on)
- Every Playwright session now goes through scheduler.submit() onto the
single worker queue. "Refresh Rules", cookie import and the boot sync
were calling automation.py directly from Flask request threads, so
with gunicorn's threads=4 a refresh during a scheduled run opened a
second Chromium and both wrote /data/auth_state.json — which
Playwright writes itself, non-atomically. A torn write there reads
back as "session expired" and costs a pointless cookie re-import.
- Request handlers that need a result use run_browser_task(), which
waits REQUEST_WAIT_SECONDS (90) then raises BrowserBusy -> 503
without cancelling the work. Previously a slow in-request scrape could
exceed gunicorn's 120s timeout; killing that worker takes the
APScheduler jobstore and run queue with it, silently dropping every
pending verification poll.
Verification correctness (the feature that exists to catch silent failures)
- run_rule() no longer starts a poll loop when the watermark capture
itself failed. watermark_id was None both for "capture failed" and
"rule has no prior activity", and check_completion() skips its
comparison on None — so a failed capture matched the *previous* run's
Complete entry and reported a false "verified", the exact class of bug
the verification step was added to catch.
- A refresh now reconciles schedules.json against the live rule list.
Rule ids are stable, names are not, and the cached name is what the
Activity-page match keys on: a renamed rule kept running while
reporting "Not Confirmed" forever, training you to ignore the signal.
A deleted rule was worse — its card disappears while its cron jobs
keep firing, with no UI left to disable them. Skipped on an empty
scrape so a MassPrice redesign can't delete every saved schedule.
- The dashboard's session banner reads a session_state.json flag set by
_ensure_authenticated() instead of guessing from run_history.json.
list_rules() writes no history entry, so a failed refresh — often the
first symptom — never raised the banner, despite the docs saying it
covered "run/refresh".
Scheduler robustness
- rebuild_jobs() parses and adds each entry under its own guard. It runs
*after* clearing existing jobs, so one malformed entry escaping as an
exception left the scheduler holding no jobs at all — every schedule
silently dead until the next restart. _parse_entry() also accepts
HH:MM:SS (some browsers emit seconds, which the old two-value unpack
crashed on) and range-checks hour/minute.
- The API validates entries at save time rather than storing something
every rebuild skips, and the frontend surfaces save failures instead
of discarding them.
- An unknown TZ logs and falls back to UTC instead of crash-looping the
container with a traceback that doesn't obviously point at .env.
Auth and web hygiene
- Empty-but-set LOGIN_PASSWORD/SECRET_KEY now fall back to their
defaults. docker-compose always sets these, so a blank .env value
arrived as "" — and compare_digest("", "") is True, i.e. no login at
all. An empty submitted password is rejected outright too.
- Password compared as bytes: compare_digest raises TypeError on
non-ASCII str, turning any accented character into a 500.
- CSRF token per session, in a meta tag, echoed as X-CSRF-Token on all
four POST endpoints; explicit SameSite=Lax/HttpOnly.
- /api/* answers 401 instead of redirecting. fetch() follows a 302 to
/login, gets 200 HTML, and reads it as success — "Run Now" reported
"Queued" for runs that were never queued.
- ?next= is restricted to local paths (open redirect).
Smaller correctness fixes
- Timestamps render in TZ via a localtime filter; they were shown as raw
UTC while schedules run in America/Phoenix.
- Activity's 24h filter excludes unparseable timestamps rather than
including them, and treats naive values as UTC instead of raising.
- SameSite=None cookies are forced Secure — Chromium drops them
otherwise, silently, possibly losing the one cookie carrying the login.
- The Activity selector escapes the rule name, so a name containing a
quote no longer produces an unparseable selector.
- pytz pinned explicitly; it only ever resolved as an apscheduler 3.x
transitive dep, which 4.x drops.
Docs corrected where they disagreed with the code: auth_state.json is
written by Playwright, not through storage.py's atomic helper; the
banner's real trigger; and the deploy skill's claim that a rename causes
"rule id not found" (it can't — schedules are keyed by the stable id).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6TgffxTvYhqxykAbVQg2v
355 lines
12 KiB
Python
355 lines
12 KiB
Python
import hmac
|
|
import os
|
|
import secrets
|
|
from datetime import datetime, timedelta, timezone
|
|
from functools import wraps
|
|
|
|
from flask import Flask, jsonify, redirect, render_template, request, session, url_for
|
|
|
|
import automation
|
|
import scheduler
|
|
import storage
|
|
|
|
APP_VERSION = "0.10.0"
|
|
APP_VERSION_DATE = "2026-09-01"
|
|
|
|
app = Flask(__name__)
|
|
# `or default`, not `get(name, default)`: docker-compose.yml always *sets*
|
|
# these vars, so a missing/blank value in .env arrives as an empty string
|
|
# and the two-arg default never applies. An empty secret_key makes Flask
|
|
# raise on every session write, and an empty LOGIN_PASSWORD would make
|
|
# compare_digest("", "") return True — i.e. no login at all.
|
|
app.secret_key = os.environ.get("SECRET_KEY") or "dev-secret-change-me"
|
|
app.config["SESSION_COOKIE_HTTPONLY"] = True
|
|
app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
|
|
|
|
LOGIN_PASSWORD = os.environ.get("LOGIN_PASSWORD") or "changeme"
|
|
|
|
DAYS = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"]
|
|
DAY_LABELS = {"mon": "Mon", "tue": "Tue", "wed": "Wed", "thu": "Thu", "fri": "Fri", "sat": "Sat", "sun": "Sun"}
|
|
|
|
|
|
def login_required(view):
|
|
@wraps(view)
|
|
def wrapped(*args, **kwargs):
|
|
if not session.get("authenticated"):
|
|
# The JSON API answers 401 instead of redirecting: fetch()
|
|
# follows a 302 to /login, gets 200 HTML back, and the caller
|
|
# reads res.ok as success — "Run Now" would report "Queued"
|
|
# for a run that was never queued.
|
|
if request.path.startswith("/api/"):
|
|
return jsonify({"error": "Signed out — reload the page to log in again."}), 401
|
|
return redirect(url_for("login", next=request.path))
|
|
return view(*args, **kwargs)
|
|
|
|
return wrapped
|
|
|
|
|
|
def _csrf_token():
|
|
token = session.get("csrf_token")
|
|
if not token:
|
|
token = secrets.token_urlsafe(32)
|
|
session["csrf_token"] = token
|
|
return token
|
|
|
|
|
|
def csrf_protect(view):
|
|
"""Header-based CSRF check for the JSON API. The token is minted per
|
|
browser session, handed to the page in a <meta> tag, and echoed back
|
|
by app.js in X-CSRF-Token — a cross-origin page can read neither the
|
|
meta tag nor the session cookie's contents, so it can't forge the
|
|
header, and a plain cross-site <form> POST can't set one at all."""
|
|
|
|
@wraps(view)
|
|
def wrapped(*args, **kwargs):
|
|
sent = request.headers.get("X-CSRF-Token", "")
|
|
expected = session.get("csrf_token", "")
|
|
# Compared as bytes: `sent` is attacker-controlled, and
|
|
# compare_digest raises TypeError on non-ASCII str input.
|
|
if not expected or not hmac.compare_digest(sent.encode(), expected.encode()):
|
|
return jsonify({"error": "Invalid or missing CSRF token — reload the page."}), 403
|
|
return view(*args, **kwargs)
|
|
|
|
return wrapped
|
|
|
|
|
|
@app.context_processor
|
|
def inject_version():
|
|
return {
|
|
"app_version": APP_VERSION,
|
|
"app_version_date": APP_VERSION_DATE,
|
|
"csrf_token": _csrf_token(),
|
|
}
|
|
|
|
|
|
def _safe_next(target):
|
|
"""Only ever redirect to a path on this app. Without this, a crafted
|
|
/login?next=https://evil.com turns the login form into an open
|
|
redirect. `//host` and `/\\host` are protocol-relative to a browser,
|
|
so they're rejected too."""
|
|
if not target or not target.startswith("/"):
|
|
return None
|
|
if target.startswith("//") or target.startswith("/\\"):
|
|
return None
|
|
return target
|
|
|
|
|
|
@app.route("/login", methods=["GET", "POST"])
|
|
def login():
|
|
error = None
|
|
if request.method == "POST":
|
|
password = request.form.get("password", "")
|
|
# Compared as bytes: compare_digest raises TypeError on
|
|
# non-ASCII str, which turned any accented character typed into
|
|
# the password box into a 500 instead of "Incorrect password".
|
|
# An empty password never authenticates, even if LOGIN_PASSWORD
|
|
# somehow ends up empty too.
|
|
if password and hmac.compare_digest(password.encode(), LOGIN_PASSWORD.encode()):
|
|
session["authenticated"] = True
|
|
return redirect(_safe_next(request.args.get("next")) or url_for("dashboard"))
|
|
error = "Incorrect password"
|
|
return render_template("login.html", error=error)
|
|
|
|
|
|
@app.route("/logout")
|
|
def logout():
|
|
session.clear()
|
|
return redirect(url_for("login"))
|
|
|
|
|
|
def _merged_rules():
|
|
"""Combine the cached rule list with saved schedules and last-run status."""
|
|
cache = storage.load("rules_cache.json", {"updated_at": None, "rules": []})
|
|
schedules = storage.load("schedules.json", {})
|
|
history = storage.load("run_history.json", [])
|
|
|
|
last_run_by_rule = {}
|
|
for entry in history:
|
|
rid = entry.get("rule_id")
|
|
if rid and rid not in last_run_by_rule:
|
|
last_run_by_rule[rid] = entry
|
|
|
|
rules = []
|
|
for r in cache.get("rules", []):
|
|
rid = r["id"]
|
|
sched = schedules.get(rid, {})
|
|
rules.append(
|
|
{
|
|
"id": rid,
|
|
"name": r["name"],
|
|
"enabled": sched.get("enabled", True),
|
|
"schedules": sched.get("schedules", []),
|
|
"last_run": last_run_by_rule.get(rid),
|
|
}
|
|
)
|
|
return rules, cache.get("updated_at")
|
|
|
|
|
|
def _session_warning():
|
|
"""True while the saved MassPrice session is known to be missing or
|
|
expired — surfaced as a dashboard banner so a dead session doesn't
|
|
just look like "no rules".
|
|
|
|
Reads the flag that automation._ensure_authenticated() sets on *any*
|
|
failed page load (scheduled run, "Refresh Rules", or a verification
|
|
poll) and clears as soon as one authenticates again. It deliberately
|
|
doesn't infer this from run_history.json: list_rules() never writes
|
|
a history entry, so a refresh failure — often the first thing the
|
|
user notices — would never have raised the banner."""
|
|
return bool(storage.load("session_state.json", {}).get("expired"))
|
|
|
|
|
|
@app.route("/")
|
|
@login_required
|
|
def dashboard():
|
|
rules, rules_updated_at = _merged_rules()
|
|
return render_template(
|
|
"dashboard.html",
|
|
rules=rules,
|
|
rules_updated_at=rules_updated_at,
|
|
days=DAYS,
|
|
day_labels=DAY_LABELS,
|
|
session_warning=_session_warning(),
|
|
)
|
|
|
|
|
|
@app.route("/session")
|
|
@login_required
|
|
def session_page():
|
|
return render_template("session.html", status=automation.session_status())
|
|
|
|
|
|
@app.route("/api/session/import", methods=["POST"])
|
|
@login_required
|
|
@csrf_protect
|
|
def api_session_import():
|
|
body = request.get_json(force=True, silent=True) or {}
|
|
try:
|
|
scheduler.import_session(body.get("cookies", ""))
|
|
except scheduler.BrowserBusy as exc:
|
|
return jsonify({"error": str(exc)}), 503
|
|
except automation.AutomationError as exc:
|
|
return jsonify({"error": str(exc)}), 409
|
|
except Exception as exc:
|
|
return jsonify({"error": str(exc)}), 502
|
|
return jsonify({"ok": True})
|
|
|
|
|
|
def _parse_ts(value):
|
|
"""Parse a stored ISO timestamp into an aware datetime, or None.
|
|
Everything written by _log_history() is UTC-aware; a naive value
|
|
(hand-edited or from an older build) is assumed to be UTC so it can
|
|
still be compared against an aware cutoff without raising."""
|
|
try:
|
|
dt = datetime.fromisoformat(value)
|
|
except (TypeError, ValueError):
|
|
return None
|
|
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
|
|
|
|
|
|
@app.template_filter("localtime")
|
|
def localtime(value):
|
|
"""Render a stored UTC timestamp in the configured display timezone.
|
|
Schedules are set in TZ, so showing run times in UTC reads as wrong
|
|
even when the run fired at exactly the right moment."""
|
|
dt = _parse_ts(value)
|
|
if dt is None:
|
|
return value or "—"
|
|
return dt.astimezone(scheduler.TZ).strftime("%b %-d, %-I:%M %p %Z")
|
|
|
|
|
|
@app.route("/activity")
|
|
@login_required
|
|
def activity():
|
|
history = storage.load("run_history.json", [])
|
|
cutoff = datetime.now(timezone.utc) - timedelta(hours=24)
|
|
# An entry whose timestamp can't be parsed can't be shown to be
|
|
# recent, so it's excluded rather than silently kept.
|
|
recent = [
|
|
e for e in history if (ts := _parse_ts(e.get("timestamp"))) is not None and ts >= cutoff
|
|
]
|
|
return render_template("activity.html", history=recent)
|
|
|
|
|
|
def _reconcile_schedules(rules):
|
|
"""Bring schedules.json back in line with the live rule list.
|
|
|
|
Rule ids are stable; names are not, and schedules.json caches the
|
|
name. Two things went wrong without this:
|
|
|
|
- **Renamed rule.** The stale name is what run verification matches
|
|
against on MassPrice's Activity page, so every run kept working
|
|
while reporting "Not Confirmed" forever — training you to ignore
|
|
the one signal that catches a silently-failing run.
|
|
- **Deleted rule.** Its card vanishes from the dashboard (those come
|
|
from rules_cache.json) while its cron jobs keep firing and failing
|
|
from schedules.json, with no UI left to switch them off.
|
|
|
|
Skipped entirely on an empty rule list: a scrape that breaks after a
|
|
MassPrice redesign shouldn't be able to delete every saved schedule.
|
|
"""
|
|
if not rules:
|
|
return
|
|
schedules = storage.load("schedules.json", {})
|
|
if not schedules:
|
|
return
|
|
|
|
live = {r["id"]: r["name"] for r in rules}
|
|
kept = {}
|
|
changed = False
|
|
for rule_id, entry in schedules.items():
|
|
if rule_id not in live:
|
|
app.logger.info("Dropping schedule for rule %s — gone from MassPrice", rule_id)
|
|
changed = True
|
|
continue
|
|
if entry.get("name") != live[rule_id]:
|
|
app.logger.info(
|
|
"Rule %s renamed %r -> %r, updating stored schedule",
|
|
rule_id,
|
|
entry.get("name"),
|
|
live[rule_id],
|
|
)
|
|
entry["name"] = live[rule_id]
|
|
changed = True
|
|
kept[rule_id] = entry
|
|
|
|
if changed:
|
|
storage.save("schedules.json", kept)
|
|
scheduler.rebuild_jobs()
|
|
|
|
|
|
@app.route("/api/rules/refresh", methods=["POST"])
|
|
@login_required
|
|
@csrf_protect
|
|
def api_refresh_rules():
|
|
try:
|
|
rules = scheduler.refresh_rules()
|
|
except scheduler.BrowserBusy as exc:
|
|
return jsonify({"error": str(exc)}), 503
|
|
except Exception as exc:
|
|
return jsonify({"error": str(exc)}), 502
|
|
_reconcile_schedules(rules)
|
|
return jsonify({"rules": rules})
|
|
|
|
|
|
@app.route("/api/rules/<rule_id>/schedule", methods=["POST"])
|
|
@login_required
|
|
@csrf_protect
|
|
def api_save_schedule(rule_id):
|
|
body = request.get_json(force=True, silent=True) or {}
|
|
|
|
# Validate before persisting: an entry rebuild_jobs() can't parse
|
|
# would otherwise be stored, skipped on every rebuild, and quietly
|
|
# never run.
|
|
raw_schedules = body.get("schedules", [])
|
|
if not isinstance(raw_schedules, list):
|
|
return jsonify({"error": "schedules must be a list"}), 400
|
|
bad = next((e for e in raw_schedules if not scheduler.is_valid_entry(e)), None)
|
|
if bad is not None:
|
|
return jsonify(
|
|
{
|
|
"error": "Unusable schedule entry — each one needs an HH:MM time and "
|
|
f"at least one valid day. Got: {bad!r}"
|
|
}
|
|
), 400
|
|
|
|
schedules = storage.load("schedules.json", {})
|
|
cache = storage.load("rules_cache.json", {"rules": []})
|
|
name = next((r["name"] for r in cache.get("rules", []) if r["id"] == rule_id), None)
|
|
|
|
schedules[rule_id] = {
|
|
"name": name,
|
|
"enabled": bool(body.get("enabled", True)),
|
|
"schedules": raw_schedules,
|
|
}
|
|
storage.save("schedules.json", schedules)
|
|
scheduler.rebuild_jobs()
|
|
return jsonify({"ok": True})
|
|
|
|
|
|
@app.route("/api/rules/<rule_id>/run", methods=["POST"])
|
|
@login_required
|
|
@csrf_protect
|
|
def api_run_rule(rule_id):
|
|
cache = storage.load("rules_cache.json", {"rules": []})
|
|
name = next((r["name"] for r in cache.get("rules", []) if r["id"] == rule_id), None)
|
|
scheduler.enqueue_run(rule_id, name)
|
|
return jsonify({"ok": True})
|
|
|
|
|
|
def _bootstrap():
|
|
scheduler.start()
|
|
# Best-effort initial rule sync so the dashboard isn't blank before
|
|
# the user ever clicks "Refresh Rules". Queued rather than run inline
|
|
# so boot doesn't block on Chromium, and so it can't race a run; a
|
|
# failure (no Playwright or no session yet, e.g. local dev) is logged
|
|
# by the worker and otherwise ignored.
|
|
if storage.load("rules_cache.json", None) is None:
|
|
scheduler.submit(automation.list_rules)
|
|
|
|
|
|
_bootstrap()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
app.run(host="0.0.0.0", port=int(os.environ.get("PORT", 5000)), debug=False)
|