From 3d374c4417e6bc82c224945f7e7b4df95722f37c Mon Sep 17 00:00:00 2001 From: Barely Removable Date: Tue, 29 Sep 2026 10:49:59 -0700 Subject: [PATCH] Pull daily instead of on day 25; skip past dates and weekend misreads The monthly day-25 pull pushed whatever month the district's permalink served. September's PDF was still live on 9/29, so the 9/25 pull re-pushed September and October would not have been pulled until 10/25. Home Assistant now dedups lunch events per date, so pull every day and pick up a new month the evening it's posted. - push only today onward; skip weekend dates that have items (a misread date: Fri 9/25's menu once landed on Sat 9/26) and log a warning - settings schema is just calendar_run_time; load_settings keeps only known keys and startup rewrites the file, dropping stale SMS-era data - UI: remove day-of-month field and the "no dedup" warning, fix buttons that reverted to "This Week" labels - docs: daily scheduler, HA queued mode + dedup, calendar is iCloud Co-Authored-By: Claude Opus 5.5 --- .env.example | 10 ++--- CLAUDE.md | 104 +++++++++++++++++++++++++------------------------ app.py | 107 +++++++++++++++++++++++---------------------------- 3 files changed, 105 insertions(+), 116 deletions(-) diff --git a/.env.example b/.env.example index d0bd1b7..3db6397 100644 --- a/.env.example +++ b/.env.example @@ -12,13 +12,9 @@ HA_WEBHOOK_URL= ANTHROPIC_API_KEY= # --- Optional --- -# Day of the month (1-28) to pull whichever month is currently published -# and push it to the calendar. Re-pushing the same month creates duplicate -# events, so pick one day near month-end (the district tends to post next -# month's menu a few days early). -CALENDAR_RUN_DAY_OF_MONTH=25 - -# Time to run the pull (24-hour HH:MM, container timezone) +# Time of the daily pull (24-hour HH:MM, container timezone). Each day it +# pushes today onward from whichever month is published; Home Assistant +# skips dates that already have a lunch event, so daily pulls are safe. CALENDAR_RUN_TIME=18:00 # HTTP server port diff --git a/CLAUDE.md b/CLAUDE.md index e620c6d..8a93c8f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,12 +11,17 @@ user's request in favor of Home Assistant calendar events exclusively. If you find references to texting/SMS/phone numbers anywhere (docs, old commits), they describe the pre-2026-09-03 app, not the current one. +**2026-09-29: monthly → daily pulls, HA-side dedup.** See "Pull scheduler" +and "Home Assistant integration" below. Anything describing a day-25 +monthly pull or "no dedup, re-pushing creates duplicates" describes the +pre-2026-09-29 app. + **2026-09-08: pivoted from Nutrislice to a Claude-read PDF, and from weekly to monthly pulls.** The district discontinued Nutrislice (the old scraper hit `kyrene.api.nutrislice.com`, a clean per-day JSON API) in favor of publishing one monthly menu PDF per grade band on `kyrene.org/family-student-resources/food-services/school-menus`. See -"Menu source" and "Monthly pull scheduler" below for how that's handled +"Menu source" and "Daily pull scheduler" below for how that's handled now. Anything describing a weekly Nutrislice pull describes the pre-2026-09-08 app. @@ -54,16 +59,15 @@ pre-2026-09-08 app. fail-soft-into-`_status`-error behavior as a Home Assistant push failure. `GET /api/integrations` reports both keys' configured/not-configured state. -- Settings (calendar pull schedule) persist to `/data/settings.json`, not - environment variables — env vars (`CALENDAR_RUN_TIME`, - `CALENDAR_RUN_DAY_OF_MONTH`) are defaults used only the very first time, - before a settings file exists. `load_settings()` merges the file's - contents *over* `_DEFAULTS` rather than returning the file verbatim — - this matters because the server's settings.json predates several schema - changes (SMS era, then the weekly-schedule era) and is missing whatever - keys the current schema expects; without the merge, the scheduler would - silently come up with zero jobs after every deploy until someone happened - to re-save the Schedule card in the UI. +- Settings (just `calendar_run_time`) persist to `/data/settings.json`, not + environment variables — the env var `CALENDAR_RUN_TIME` is a default used + only before a settings file exists. `_DEFAULTS` is the complete schema: + `load_settings()` merges the file *over* `_DEFAULTS` (so an old file + missing a key still yields a valid schedule instead of zero jobs) and + **keeps only keys in `_DEFAULTS`**; startup rewrites the file with that + result. That pruning (2026-09-29) removed the SMS era's phone numbers / + carrier gateway / vacations and the old `calendar_run_day_of_month` from + the server's settings.json — they had lingered, unread, since the pivots. - **No vacation/no-push date ranges (removed 2026-09-03, briefly existed).** A day with no posted menu already produces an empty `items` list from `get_month_menu()`, and `_push_calendar()` already skips pushing when @@ -77,53 +81,51 @@ pre-2026-09-08 app. does anything externally-visible. On the HA side (`automation.lunch_menu_family_calendar`, `http://192.168.86.207:8123`), a `local_only` webhook trigger runs `calendar.create_event` against - `calendar.family_2` (the "Family" Google Calendar entity — there are two - entities both named "Family" in this HA instance, `calendar.family` and - `calendar.family_2`; this integration deliberately targets `_2` per user - choice). `calendar.family_2` only supports `create_event`, not - `delete_event` — a bad push can't be cleaned up via the HA API, only - manually in Google Calendar. No duplicate-prevention: re-pushing the - same date's menu creates a second event — this is why the default - schedule is a single day/month, not daily (see below). - **`mode: parallel`, not the default `single` (fixed 2026-09-03 after a - real bug):** a push fires one webhook POST per school day back-to-back - with no delay. Under `mode: single`, HA rejects any webhook trigger that - arrives while a previous run is still executing (`calendar.create_event` - itself takes ~600-700ms) — most days in a batch were silently dropped as - `failed_single`, so only one day's event ever actually got created per - push. Found via `ha_get_automation_traces` after the user reported a - manual test push not showing up on the calendar. Each invocation is - independent (one `calendar.create_event` call, no shared state), so - `parallel` is the correct mode per HA's own guidance ("independent - per-entity actions → parallel"), not just a workaround. The webhook id is + `calendar.family_2` (the shared iCloud "Family" calendar, via CalDAV — + there are two entities both named "Family" in this HA instance, + `calendar.family` and `calendar.family_2`; this integration deliberately + targets `_2` per user choice). `calendar.family_2` supports + `create_event` and `get_events` but not `delete_event` — a bad push can't + be cleaned up via the HA API, only by hand in the Calendar app. + **Automation mode history — `queued` since 2026-09-29.** The app fires + one webhook POST per school day back-to-back (HA answers each POST + immediately, so the app never waits on the calendar write). `single` + (default) dropped all but the first as `failed_single` (fixed 2026-09-03 + → `parallel`); `parallel` then sent ~20 simultaneous CalDAV PUTs on a + month push, overflowing iCloud (connection pool full + ~10× `500 + Internal Server Error` on the 2026-09-25 push). Now `queued` (max 40): + one write at a time, nothing dropped. **Dedup (2026-09-29):** before + creating, the automation reads the date with `calendar.get_events` and + stops if a `🍱 Mariposa Lunch` event already exists — that's what makes + daily pulls safe. It verifies the event saved and retries once after 15s. + Pre-dedup pushes left 2-3 copies on most early-September days; those + have to be deleted by hand. The webhook id is a random token that lives only in the server's `.env` (`HA_WEBHOOK_URL`) and in the automation's trigger config — never committed here. If the calendar push stops working, check both ends: `GET /api/integrations` on this app reports whether `HA_WEBHOOK_URL` is set at all, and `ha_get_automation_traces` on `automation.lunch_menu_family_calendar` shows whether HA is even receiving the webhook. -- **Monthly pull scheduler (weekly → monthly, 2026-09-08):** calendar pulls - run on their own schedule (`calendar_run_time`/`calendar_run_day_of_month` - in settings, `schedule` jobs tagged `"calendar"`), completely decoupled - from anything else — the tag/isolation pattern - (`schedule.clear("calendar")` instead of a bare `schedule.clear()`) is - deliberate so a future second schedule wouldn't stomp this one. The - `schedule` library has no native day-of-month trigger, so - `apply_calendar_schedule()` registers a *daily* job - (`_maybe_push_month`) that no-ops unless `datetime.now().day` matches - the configured day — `push_month_to_calendar()` only actually runs on - that one day per month. `next_calendar_run()` computes the real next - firing directly from settings (handling month/year rollover and clamping - the configured day to short months, e.g. day 30 in February → the 28th) - for the UI's "Next Scheduled" field, since the underlying `schedule` job's - own `next_run` is meaningless here (it's always just "tomorrow"). Default - is day 25 at 18:00 — picked because the district tends to post next - month's PDF a few days before month-end, so pulling then usually catches - the *upcoming* month rather than the one ending. Whatever month is live - at the permalink when the job fires is what gets pushed — there's no - logic forcing "next month specifically." +- **Daily pull scheduler (monthly → daily, 2026-09-29):** `push_month_to_calendar()` + runs every day at `calendar_run_time` (default 18:00; `schedule` job + tagged `"calendar"` — `schedule.clear("calendar")`, not a bare + `schedule.clear()`, so a future second schedule wouldn't stomp it). + **Why daily:** the old once-a-month day-25 pull pushed whatever month was + live at the permalink; when the district posted late (September 2026: the + permalink still served September on 09-29), it silently re-pushed the + current month and the next month's menu wouldn't be pulled until the + following 25th — most of that month gone. Daily pulls pick up a new month + the evening it's posted; HA-side dedup makes the repeats harmless, and the + Claude read costs well under a cent a day. Each pull pushes **only today + onward** (past dates are skipped — they're already on the calendar or no + longer matter) and **skips weekend dates that have items**, logging a + warning and noting it in the status result: there's never school lunch on + a weekend, so that's a misread date (seen once: Fri 9/25's menu landed on + Sat 9/26). Whatever month is live at the permalink is what gets read — + there's no logic forcing "next month specifically." - **Manual actions:** "Push This Month to Calendar" (`POST /api/push-calendar` - → `push_month_to_calendar()`, same function the schedule calls) and + → `push_month_to_calendar()`, same function the schedule calls — so it + also pushes only today onward and skips weekend misreads) and "Preview This Month's Menu" (`GET /api/preview` → `get_month_menu()` directly, pure dry run, no Home Assistant push, no side effects at all — use this to check the scraper before trusting a scheduled push). diff --git a/app.py b/app.py index b50f547..279f9ad 100644 --- a/app.py +++ b/app.py @@ -2,13 +2,12 @@ Lunch Notifier — pulls the Mariposa lunch menu and pushes it to a Home Assistant calendar. Visit http://localhost:2323 to manage settings. """ -import calendar import json import logging import os import threading import time -from datetime import datetime +from datetime import date, datetime, timedelta from pathlib import Path import requests @@ -25,10 +24,10 @@ app = Flask(__name__) SETTINGS_FILE = Path(os.environ.get("SETTINGS_FILE", "/data/settings.json")) PORT = int(os.environ.get("PORT", "2323")) -# Env-var defaults (used only when no settings file exists yet) +# Env-var defaults (used only when no settings file exists yet). These are +# also the complete settings schema: load_settings() drops any other key. _DEFAULTS = { "calendar_run_time": os.environ.get("CALENDAR_RUN_TIME", "18:00"), - "calendar_run_day_of_month": int(os.environ.get("CALENDAR_RUN_DAY_OF_MONTH", "25")), } _status = {"last_calendar_run": None, "calendar_result": None} @@ -42,11 +41,12 @@ def load_settings() -> dict: try: with open(SETTINGS_FILE) as f: # Merge over _DEFAULTS so a settings.json from before a schema - # change (e.g. the pre-calendar SMS-era file, or the - # weekly-schedule era before calendar_run_day_of_month - # existed) still gets a valid schedule rather than silently - # scheduling nothing. - return {**_DEFAULTS, **json.load(f)} + # change still gets a valid schedule rather than silently + # scheduling nothing, and keep only current keys so stale + # ones (SMS-era phone numbers, the old day-of-month pull) + # don't linger. Startup rewrites the file with this result. + stored = json.load(f) + return {**_DEFAULTS, **{k: v for k, v in stored.items() if k in _DEFAULTS}} except Exception: pass return dict(_DEFAULTS) @@ -83,11 +83,15 @@ def _push_calendar(items: list[str], date_str: str | None) -> str: def push_month_to_calendar() -> None: - """Fetch whichever month is currently published and push every day that - has a posted menu — days with nothing posted (weekends, breaks, - holidays) are skipped automatically since there's nothing to create an - event from. Runs on its own schedule (calendar_run_time/ - calendar_run_day_of_month), independent of anything else.""" + """Fetch whichever month is currently published and push every day from + today onward that has a posted menu. Days with nothing posted (breaks, + holidays) are skipped since there's nothing to create an event from; + past days are skipped since they're already on the calendar or no + longer matter; weekend days are skipped even if the PDF read put a menu + on one (there's never school lunch on a weekend, so that's a misread + date — seen once as Fri 9/25's menu landing on Sat 9/26). Runs daily + (calendar_run_time); Home Assistant skips dates that already have a + lunch event, so repeated pulls don't create duplicates.""" global _status try: @@ -97,8 +101,20 @@ def push_month_to_calendar() -> None: _status = {**_status, "last_calendar_run": datetime.now().isoformat(), "calendar_result": f"error: {e}"} return - pushed = errors = no_items = 0 + today = date.today() + pushed = errors = no_items = past = weekend = 0 for day in month: + d = date.fromisoformat(day["date"]) + if d < today: + past += 1 + continue + if d.weekday() >= 5: + if day["items"]: + log.warning(f"Skipping {day['date']} ({d:%a}): menu on a weekend is a misread date") + weekend += 1 + else: + no_items += 1 + continue result = _push_calendar(day["items"], day["date"]) if result == "ok": pushed += 1 @@ -107,7 +123,8 @@ def push_month_to_calendar() -> None: else: errors += 1 - log.info(f"Month push: {pushed} pushed, {errors} errors, {no_items} no menu") + log.info(f"Month push: {pushed} pushed, {errors} errors, {no_items} no menu, " + f"{past} past, {weekend} weekend misreads skipped") if errors: summary = f"{pushed} ok, {errors} error" + ("s" if errors != 1 else "") @@ -115,6 +132,8 @@ def push_month_to_calendar() -> None: summary = f"ok ({pushed} day{'s' if pushed != 1 else ''})" else: summary = "no items" + if weekend: + summary += f", {weekend} weekend misread{'s' if weekend != 1 else ''} skipped" _status = { **_status, @@ -125,43 +144,20 @@ def push_month_to_calendar() -> None: # ── Scheduler ───────────────────────────────────────────────────────────────── -def _maybe_push_month(day_of_month: int) -> None: - """schedule fires this daily at run_time; only actually pushes on the - configured day of the month (the `schedule` library has no native - day-of-month trigger).""" - if datetime.now().day == day_of_month: - push_month_to_calendar() - - def apply_calendar_schedule(settings: dict) -> None: with _lock: schedule.clear("calendar") run_time = settings.get("calendar_run_time", "18:00") - day_of_month = settings.get("calendar_run_day_of_month", 25) - schedule.every().day.at(run_time).do(_maybe_push_month, day_of_month).tag("calendar") - log.info(f"Scheduled calendar pull: day {day_of_month} of each month at {run_time}") + schedule.every().day.at(run_time).do(push_month_to_calendar).tag("calendar") + log.info(f"Scheduled calendar pull: daily at {run_time}") def next_calendar_run(settings: dict) -> datetime: - """The `schedule` job above fires daily (it has no native day-of-month - trigger — see _maybe_push_month), so its own next_run is just - "tomorrow", not the actual next push. Compute the real one directly - from settings for display in the UI.""" run_time = settings.get("calendar_run_time", "18:00") hour, minute = (int(p) for p in run_time.split(":")) - day_of_month = settings.get("calendar_run_day_of_month", 25) - now = datetime.now() - candidate = now.replace( - day=min(day_of_month, calendar.monthrange(now.year, now.month)[1]), - hour=hour, minute=minute, second=0, microsecond=0, - ) - if candidate <= now: - y, m = (now.year, now.month + 1) if now.month < 12 else (now.year + 1, 1) - candidate = candidate.replace( - year=y, month=m, day=min(day_of_month, calendar.monthrange(y, m)[1]), - ) - return candidate + candidate = now.replace(hour=hour, minute=minute, second=0, microsecond=0) + return candidate if candidate > now else candidate + timedelta(days=1) def _schedule_loop() -> None: @@ -195,7 +191,7 @@ def api_get_settings(): def api_post_settings(): data = request.get_json(force=True) s = load_settings() - for key in ("calendar_run_time", "calendar_run_day_of_month"): + for key in _DEFAULTS: if key in data: s[key] = data[key] save_settings(s) @@ -369,11 +365,7 @@ input:focus { outline: 2px solid var(--primary); outline-offset: -1px; border-co -
- - -
Fires once a month and pushes whichever month is currently published — e.g. day 25 catches next month's menu once the district posts it near month-end. There's no dedup, so re-pushing the same month creates duplicate calendar events.
-
+
Pulls every day and pushes today onward from whichever month the district has published, so next month's menu lands on the calendar as soon as it's posted. Home Assistant skips dates that already have a lunch event, so daily pulls don't create duplicates.
@@ -413,7 +405,6 @@ function applyIntegrations(ig) { function applySettings(s) { document.getElementById('f-time').value = s.calendar_run_time || '18:00'; - document.getElementById('f-day').value = s.calendar_run_day_of_month || 25; } function applyStatus(st) { @@ -428,8 +419,7 @@ function applyStatus(st) { async function saveSchedule() { const calendar_run_time = document.getElementById('f-time').value; - const calendar_run_day_of_month = parseInt(document.getElementById('f-day').value, 10); - await post('/api/settings', {calendar_run_time, calendar_run_day_of_month}); + await post('/api/settings', {calendar_run_time}); toast('Schedule saved'); load(); } @@ -440,7 +430,7 @@ async function pushCalendar() { btn.textContent = 'Pushing…'; await post('/api/push-calendar', {}); toast('Triggered! Check status in a moment.'); - setTimeout(() => { btn.disabled=false; btn.textContent='Push This Week to Calendar'; load(); }, 6000); + setTimeout(() => { btn.disabled=false; btn.textContent='Push This Month to Calendar'; load(); }, 6000); } async function previewMenu() { @@ -465,7 +455,7 @@ async function previewMenu() { box.style.display = ''; } finally { btn.disabled = false; - btn.textContent = "Preview This Week's Menu"; + btn.textContent = "Preview This Month's Menu"; } } @@ -492,11 +482,12 @@ setInterval(load, 30000); # ── Entrypoint ──────────────────────────────────────────────────────────────── if __name__ == "__main__": - if not SETTINGS_FILE.exists(): - log.info(f"Creating default settings at {SETTINGS_FILE}") - save_settings(_DEFAULTS) + # Always rewrite: creates the file on first run, and prunes keys the + # current schema no longer uses (see load_settings). + settings = load_settings() + save_settings(settings) - apply_calendar_schedule(load_settings()) + apply_calendar_schedule(settings) threading.Thread(target=_schedule_loop, daemon=True).start() log.info(f"Web UI → http://localhost:{PORT}") app.run(host="0.0.0.0", port=PORT, debug=False, use_reloader=False)