Automation Fleet Audit — Kilroy, Pipeline Shepherd, Auto-Validation, Recon Scanner, Document Extraction
Date: 2026-08-06 · Auditor: Claude Code (read-only — no changes made) Method: parallel code exploration + live data queries (shepherd reports since 7/28 host-mount, recon SQLite DBs, Supabase document_extractions, Kilroy cron registry/journal). Companion to FRIDAY_CAPABILITY_AUDIT_2026-08-06.md.
0. Cross-cutting verdict
The same pattern found in Friday repeats across the whole fleet: each system’s engine works, but its output funnels to Dustin (or to nobody) instead of to the team member who could act on it. Five specifics:
- Silent delivery failures. Shepherd calls
notify_role("admin")andnotify_role("owner")— roles that don’t exist in portal_users (real roles areup_admin/up_owner). Four rule categories have delivered to no one, ever: cert drafting overdue, certified-not-invoiced (the revenue-leakage rule), unpaid-30, on-hold-stale. Verified empirically: zero matching notifications in the DB. - Everything routes to Dustin. 12 of Kilroy’s 13 crons deliver only to Dustin. All Shepherd WhatsApps go to Dustin. Recon parking fires no notification at all. Darius gets exactly one message a day; schedulers/triage get only 2 of ~19 signal types.
- Data produced but unconsumed. Extraction has 553 permit numbers + 779 jurisdictions that projects are missing — no writer exists. Payment-lag benchmarks are computed by an unscheduled script nobody surfaces. Shepherd’s recon/financial enrichment is attached after the report is saved, so it’s null in every persisted report. CommandCenter’s Shepherd “Alerts” lane is permanently empty (schema mismatch — it reads a key the service never emits).
- Noise drowning signal. Shepherd sends 984 RFI-aging messages per run to Dustin’s Telegram (no cap on the Kilroy tier), from a query truncated at 1,000 rows — a frozen slice that never changes. This is the single worst signal-to-noise offender in the fleet.
- Stale identity/config debt from the Rob cutover. Kilroy’s most-used skills still point at Rob’s dead Docker IP; its cron prompts still say “trial shadow”; quiet hours are asserted but defined nowhere.
Live bugs found in passing (not just leverage gaps): write-off rows counted as real cash in the portal; a remittance ingested twice doubling paid amounts on 9 invoices; a NameError in auto-validation Check 6’s error path; Shepherd’s auto-hold creating permanently invisible projects; a stuck extraction batch would halt that pipeline forever with no alert.
1. KILROY (Hermes agent, Telegram)
What it is today
- 13 internal cron jobs, all healthy: 6 agent-invoking (Sonnet-pinned — field schedule 6 AM, morning briefing, EOD summary, tomorrow preview, 2 weekly rollups) + 7 zero-token Python scripts (backup health, triage alert, cert readiness, invoice aging, stalled projects, permit expiry, RFI aging).
- Conversational: full localhost API access, vault MCP (read-mostly), brain.js recall, vision, browser, STT, 88 skills (11 Unlikely-specific). Allowlist = Dustin + Darius only; standing outbound grant to those two only.
- API integration: HMAC webhook routes (ops-notify → Dustin, field-proposal → Darius, both
deliver_only), tier #1 fallback in the WhatsApp chain,/field-media/uploadwith the full filing pipeline (dedup, RFI auto-resolve, pier-log OCR). - Guardrails: daily 4 AM session reset, Opus-conversation/Sonnet-cron split, execute_code blocked in crons, ET-display rules, one-upload-path policy.
Gaps / unleveraged
| Finding | Detail |
|---|---|
| Dead Rob IP in pinned live skills | unlikely-field (18 uses — the most-used skill), unlikely-calendar, unlikely-status all hardcode 172.17.0.1:8000 (Rob’s Docker bridge). Correct is localhost:8000. The fix is even self-documented in unlikely-morning-briefing/SKILL.md:253 but never executed (curator refuses to patch pinned skills). This is the port-8000-exposure fix’s remaining scope too (memory project_port_8000_exposure). |
| ”Trial shadow” identity debt | All agent-cron prompts still open with “TRIAL NOTE… shadowing Rob”; morning briefing and EOD still label output “(Kilroy trial shadow)”; several prompts say “must NOT message Darius” — contradicting the production promotion and standing grant. |
| Darius gets one message a day | Only Daily Field Schedule reaches him. Tomorrow Preview (4 PM) is Dustin-only — Darius has no advance notice of his own next day. Permit-expiry and RFI-aging findings about his stops don’t reach him. The Darius webhook route exists and is idle except as a Vision-proposal fallback. |
| Portal alert push: specified, never called | SOUL.md fully documents POST /whatsapp/push-alert with a 9-row cron→alert mapping (“ALWAYS push an alert after a cron scan finds actionable items”); the endpoint exists with per-day dedup; zero scripts call it. Every cron finding is ephemeral Telegram text — nothing lands in the portal feed where Jacob/staff could see it. |
| Quiet hours undefined | SOUL asserts quiet-hours rules apply but no window is defined anywhere. 3:00/3:30 AM scans and a 9 PM Sunday check run unguarded. |
| Monday collision | Weekly Pipeline Summary and Weekly Health Digest both fire 0 7 * * 1 with overlapping content; both show completed=1 vs ~5 Mondays elapsed — worth checking why. |
| Unused machinery | _opslib.write_xlsx + MEDIA attachment contract implemented, never used (aging scans could ship spreadsheets at zero token cost); exif-compare.py orphaned; 3 zero-use skills incl. pinned tufte-design (prompt-budget waste) and unlikely-eod-closeout (0 uses despite EOD being “the most important thing you do” — suggests the skill never triggers and SOUL carries the workflow); WhatsApp platform still configured, fatal whatsapp_not_paired since 7/23. |
notify_ops underused | Only whatsapp_service calls it. Deploy failures, cert-gen completions, portal escalations could all route through the existing signed webhook. |
Recommendations
- Fix the skill IPs (
localhost:8000) — also a prerequisite of the approved port-8000 lockdown. Small, immediate. - De-trial the cron prompts — remove “trial shadow” labels and the “don’t message Darius” contradictions.
- Give Darius his day: send Tomorrow Preview to Darius too; route permit-expiry/RFI findings for his stops to him. All within the existing standing grant.
- Wire the 7 scan scripts to
push-alertso findings persist in the portal feed (the mapping is already written in SOUL). - Define quiet hours in one place; merge the two Monday rollups; unpin
tufte-design; retire the WhatsApp platform config.
2. PIPELINE SHEPHERD
What it is today
2:15 AM ET weekday scan of ~630 active projects; 16 live rules covering intake staleness, scheduling gaps, cert/invoice lag, payment aging (branch-aware p90 thresholds), holds, return visits, triage auto-transfer/auto-return, RFI aging, chat-stale assignments. Three auto-actions (trigger validation, auto-hold stalled In Progress, triage transfer). Daily WhatsApp digest to Dustin. Reports persist to /data/shepherd (host-mounted since 7/28).
Broken (fires but delivers to nobody, or misfires)
| Finding | Detail |
|---|---|
notify_role("admin"/"owner") match zero users | Real roles are up_admin/up_owner. Over just 8 runs: 5 cert-drafting-overdue, 5 unpaid-30, 55 unpaid-60 (notification leg), 37 on-hold-stale findings → delivered to nobody. The 60-day tier survives only via its parallel WhatsApp to Dustin. One-line fix per call site. |
| RFI query capped at 1,000 rows, no pagination | Every report: exactly 1000 open RFIs, exactly 984 aged — a frozen truncated slice. RFIs beyond row 1000 invisible. |
| 984 messages/run to Dustin | One Telegram message per aged RFI, every weekday. Branch is already in the data; nothing routes to branch schedulers. This is duplicate noise on top of rfi_escalation_service. |
| P20 auto-hold black hole | Auto-hold sets on_hold=True but not hold_since → the 30-day on-hold rule can never see it → shepherd-held projects fall out of every rule permanently, no release path. |
| Report saved before enrichment | recon/smart-recon/financial/chat sections + final action counts are attached after _save_report — null in every persisted report, so Friday’s Reconciliation section and any report reader see nothing. |
| CommandCenter alerts lane permanently empty | Frontend reads report.alerts; service never emits that key. Only surface, only up_owner, and it shows nothing. |
| Check-6 error path crashes | cert_validate.py:551 references undefined display_id — any Vision exception becomes a NameError that aborts the whole validation, then degrades to a notify-nobody “admin” notification. |
| Never-fired rules | hold_follow_up_due (0 ever, despite migration 030 + portal band + tests), intake_stale, certified_not_invoiced, rfi_aging_3d (logged_only — no action by design). Worth confirming whether conditions are genuinely absent or detection is wrong. |
Unleveraged / missing rules the data already supports
- Permit-expiration early warning (T-30/T-14) — today the first signal is a hard cert blocker after expiry.
- Cert Approved-but-never-delivered — cert_delivery_log exists; no rule reads it.
- Return-visit follow-through — P14 silently clears the flag at 14 days (which also unblocks auto-validation) instead of scheduling the visit or telling the branch scheduler.
- Repeat-validation-failure escalation — a project can fail and regenerate the same RFI forever; audit rows exist, nothing counts them.
- Branch-level SLA aggregates — per-branch RFI grouping computed and used only for per-RFI spam.
- ~13 hardcoded thresholds should move to config (only payment-overdue ever was).
Recommendations (ordered)
- Fix the two role strings (
admin→up_admin,owner→up_owner) — instantly revives 4 rule categories. - Paginate the RFI query; collapse RFI aging to one per-branch digest routed to that branch’s schedulers (kills the 984-message flood and puts the signal where action happens).
- Set
hold_sinceon auto-hold; fix the Check-6 NameError; persist enrichment before save. - Fix or remove the CommandCenter alerts contract; consider surfacing shepherd findings to schedulers/triage in-portal (they already get the bell for 2 rule types — extend it).
- Add the permit-expiry and cert-undelivered rules; then threshold config.
3. AUTO-VALIDATION
What it is today
Guarded 13-check gate (docs still say “10-point”) on Field Complete projects: SOW inspected, RFIs closed, permit valid/present, pier logs vs SoW targets, Vision-verified in-situ photos, inspection date, account, scope resolved, production days populated/confirmed, scope-change reconciled. Pass → auto-advance to Ready for Cert + Draft, auto-calculating production days. Fail → auto-RFI (5-day due) assigned to branch schedulers. Reachable from Shepherd, 8 endpoints, chat photo-upload re-validation, and ARQ — but no direct scheduler entry of its own.
Gaps
- Check-6 crash path (shared with Shepherd above) — Vision exception → NameError → whole validation errors → notification to nonexistent “admin” role. The Vision gate otherwise fails open (no API key → pass with “content not verified”), which is by design but worth remembering during credit outages like 8/5’s.
- Vision model pinned to
claude-sonnet-4-20250514(config default) — a retired-generation model ID; verify it still resolves. - Assignment fallback
["scheduler1"]— silent misroute for any branch whose scheduler lookup fails. - No outcome rollup —
auto_validationaudit category is write-only; nobody sees pass/fail rates, repeat offenders, or which checks block most often. That’s exactly the operational intelligence Jacob/schedulers could act on. - “Validation passed (13/10)” cosmetic bug in the audit message; docstrings say 10-point.
Recommendations
- Fix the NameError + the role string on its failure path (same PR as Shepherd fixes).
- Add a weekly validation-outcomes digest (which checks fail most, repeat failures) — the audit rows already exist; this is a read-and-summarize job perfect for a zero-token Kilroy script or a Shepherd section.
- Verify/update the Vision model constant; make the scheduler fallback loud (log ERROR + notify) instead of silent.
4. RECON SCANNER (payment reconciliation)
What it is today
Gmail remittance polling (Tue/Thu 2–8 PM) → PDF parse → 5-method invoice matching → 3-tier confidence scoring → auto-execute (Paid→Closed, 0.01 tolerance) or park in pending_matches. Daily scans: recon 7:15, financial recon 8:00, smart-recon catchup 10:00, missed-remittance alert Tue/Thu 8 PM. Portal: /payments + /smart-recon (owner/admin only). Zero auto-execute errors since the April two-stage-write fix; 173 clean auto-executions.
Live bugs
| Finding | Detail |
|---|---|
| Write-offs count as cash | The 5 MANA write-off rows ($4,050, 8/4) are excluded nowhere in the API — only in the offline payment-lag script. They render as reconciled/paid with phantom cash in OwnerPayments, can push invoice_status=Paid, and are live candidates on every 10 AM catchup run (currently skipped only because they score LOW — safe by accident). Violates the convention in memory reference_invoice_writeoff. |
| Duplicate remittance → doubled paid_amount | The 7/7 EFT arrived as two Gmail messages → ingested as remittances 204 AND 205 → 9 invoices (MANA-INV-9137…9147) each carry 2 payment rows. Dedup keys on gmail_message_id and revolut_txn_id (NULL on this path); nothing dedupes on (payer, date, total). |
| Negative amounts parked forever | 3 reversal rows (-$6,600) scored as MEDIUM “payments” with nonsense reasons; no refund/chargeback branch in scoring. They also net their invoices’ SUM(amount) to 0. |
| ”Unmatched payments” metric is vacuous | Always returns 0 (filters on a column that’s always populated); real unmatched volume lives in pending_matches. |
| Mixed date formats | payment_date is M/D/YYYY on remittance rows, ISO on manual rows — string comparisons and MAX() are wrong. |
Unleveraged
- Nobody is told when a match parks. Parking fires no notification at any tier — the only discovery path is opening /smart-recon. Direct cause of the stale queue: 14 pending, including 3 at 90–118 days and 4 at 78 days.
- JES remittances: zero coverage. Parser and Gmail filter are hardcoded Groundworks-only (all 204 remittances ever = ‘Groundworks Companies’). JES/dSilo remittances are never even fetched; the documented address-based matching for JES memos is unimplemented.
- Aging exists but isn’t shown.
days_outstandingis computed for the AP-email payload but never returned by GET /invoices — the portal AR view has no aging column despite the logic existing. - Payment-lag benchmarks unscheduled/unsurfaced —
/opt/scripts/payment-lag-analysis.py(median/p90 by branch, write-off-aware) runs only by hand; no slow-payer alerts despite BWI’s known p90-90d tail. - Dead code:
_require_owner,remittance_inbox_loop(contradicts the real schedule window).
Recommendations (ordered)
- Fix the money bugs first: write-off exclusion at the 3 query sites (+ score guard), remittance dedupe on (payer, date, total) + clean up the 9 doubled invoices, negative-amount branch.
- Notify on park: a pending match should ping Jacob/up_admin (portal + Kilroy ops-notify), and a weekly stale-queue digest should exist. This alone likely ends the 118-day parked rows.
- Surface aging in GET /invoices (already computed); schedule payment-lag-analysis monthly and route slow-payer flags to the owner digest.
- JES remittance support when JES payment volume warrants — the parser refactor (payer profiles instead of hardcoded strings) is the enabler.
5. DOCUMENT EXTRACTION (“dock extraction”)
What it is today
Hourly cycle in the worker: discover CERTS/PERMITS PDFs → PyMuPDF text → Haiku 4.5 via Batches API (~8. Rich corpus: 18,609 piers, 3,637 projects, jurisdiction/scope/certifier coverage 95%+.
The core problem: nobody consumes it
Only two references outside the service exist — both read-only schema docs for Friday. No backfill writer, no portal surface, no search/embedding integration. Meanwhile the corpus is sitting on:
- 553 projects that would gain a permit_number they currently lack
- 779 projects that would gain a jurisdiction
- (Calibration already proved this on a 100-doc sample; the full-corpus numbers are now measured.)
Gaps
| Finding | Detail |
|---|---|
| Stuck-batch halt, no alert | Any batch that never reaches “ended” (expired/canceled/lost) keeps batches_pending > 0 forever, and the cycle gates discovery+submission on that — one stuck batch silently stops the whole pipeline permanently. No max-age, no retry, no dead-letter (despite dead_letter.py existing). |
| 46% orphan rows | 1,672 of 3,637 project_ss_ids have no projects row (legacy folders). Positionally derived, no validation — these extractions can’t join to anything. |
| No conflict handling | 215 projects have contradictory extracted permit numbers across docs; 36 disagree with Supabase permit_number, 28 with jurisdiction. Any backfill needs a winner-picking/confidence/review story first. |
| doc_type reliability unre-measured | Calibration showed 73/100 folder agreement; the prose-enum fix was applied but never re-tested. Live data suggests residual drift. |
| Jurisdictions unnormalized | ”Arlington County” (440) vs “Arlington” (117) etc. — aggregates double-count, including the example query advertised in Friday’s schema ref. |
| OCR queue named, never built | 430 skipped_no_text rows waiting; the 83 GB archives (largely scanned) will skew far higher. |
| Phase B not started | 30,590 archive PDFs (~$38 at batch pricing) untouched — including the bulk of the permit corpus (project folders only held 160 permit PDFs), which is exactly what permit-timeline intelligence needs. |
| Minor | No token/cost telemetry captured; no updated_at trigger; hourly full-cycle is sluggish for the 1–2 new docs/day trickle. |
Recommendations (ordered)
- Add a batch timeout/dead-letter + alert — cheap insurance against the silent-halt failure mode.
- Build the backfill writer (through supabase_write_service, per standing rule; review UI or at least a dry-run report per
feedback_no_batch_mutations): normalize jurisdictions, resolve conflicts (multi-doc majority + newest-doc tiebreak, park disagreements for review), then fill the 553 permits + 779 jurisdictions. This is the single highest-value unlock in the fleet — it feeds certcheck, jurisdiction forms routing, triage, and Friday answers. - Phase B archives (~$38) — especially for the permit corpus — with the OCR queue for scanned docs.
- Re-measure doc_type accuracy after the enum fix before anything downstream trusts it; capture usage/cost telemetry at ingest.
6. Priority matrix (fleet-wide)
| # | Item | System | Type | Effort |
|---|---|---|---|---|
| 1 | notify_role role-string fix (admin→up_admin, owner→up_owner) | Shepherd | broken delivery | trivial |
| 2 | Write-off exclusion in cash queries + catchup guard | Recon | money bug | small |
| 3 | Remittance dedupe + repair 9 doubled invoices | Recon | money bug | small |
| 4 | RFI pagination + per-branch digest (kill the 984-message flood) | Shepherd | noise→signal | small-med |
| 5 | Check-6 NameError + auto-hold hold_since | Shepherd/AutoVal | bug | trivial |
| 6 | Batch timeout + alert | Extraction | resilience | small |
| 7 | Kilroy skill IP fix + de-trial prompts | Kilroy | debt | small |
| 8 | Pending-match park notifications + stale digest | Recon | leverage | small |
| 9 | Extraction backfill writer (553 permits / 779 jurisdictions) | Extraction | leverage | medium |
| 10 | Kilroy → portal push-alert wiring; Darius preview + findings | Kilroy | leverage | small-med |
| 11 | Shepherd findings to schedulers/triage in-portal; fix CommandCenter contract | Shepherd | leverage | medium |
| 12 | Validation-outcomes digest; aging column in /invoices; payment-lag on a schedule | AutoVal/Recon | leverage | small each |
| 13 | Phase B archive extraction + OCR queue | Extraction | expansion | medium |
| 14 | Permit-expiry early warning + cert-undelivered rules | Shepherd | new rules | medium |
| 15 | JES remittance parsing (payer profiles) | Recon | expansion | medium |
Items 1–6 are correctness/money fixes and belong before any of the leverage work.