cron-lane-watchdog

Cron-Lane Watchdog (2026-08-08)

Born from the 2026-08-08 overnight incident: the ARQ worker/scheduler lane went dark ~02:15–07:15 ET (shepherd + all morning crons lost, zero SMS) with every container “Up” and every RUBRIC canary green. Self-recovered at 07:15:01 ET, before the morning rebuild. Mechanism unconfirmed (container logs destroyed by the rebuild) — evidence shape fits wedged TCP connections that ARQ’s 600s job_timeout couldn’t reap.

Defenses (entirely-api 7022048, /opt/scripts c52e678)

• loop_watchdog (services/loop_watchdog.py) — daemon-thread watchdog in worker + scheduler. Exit 75 = event loop stopped beating (300s); exit 76 = worker queue non-empty with zero sched-job completions for 15 min (2-strike). restart: unless-stopped restarts the process with fresh connections. Liveness keys: ops:worker_loop_alive, ops:scheduler_loop_alive (5-min TTL).
• Scheduler hardening — every enqueue-path redis op bounded at 30s; a cron fire whose enqueue errors retries every 60s for up to 30 min (previously: silently lost until tomorrow).
• pipeline-shepherd phase bounding — PHASE_TIMEOUTS (fetch 120 / rules 150 / actions 240 / enrichment 30 / summary 45), queries in threads, failures recorded in report["phase_errors"], report always saves, every send wrapped in _safe_action.
• RUBRIC canaries — cron_lane (dead_letter_processor heartbeat in ops:task_heartbeat; >30 min = alert + portal push-alert to owners/admins), worker_loop / scheduler_loop (alive keys).

Incident forensics playbook

Sources that survive a rebuild: ops:task_heartbeat (LATEST completion per sched task — read it EARLY, fast tasks overwrite), Anthropic Batches API created_at (extraction ticks at :40), fleet-history.jsonl, portal_notifications timestamps, Supabase project_status_events, Sentry (DSN configured; UI has the ERROR events).