Doc extraction: Sonnet 5 switch + backfill adjudication (2026-08-10)
The archive-extraction pipeline (document-extraction-pipeline) moved from claude-haiku-4-5 to claude-sonnet-5 (entirely-api 918b53a). Decision was measured, not assumed: same-100-doc A/B against the 8/05 calibration ground truth showed Sonnet fixing the dominant error class (town extracted as jurisdiction where the county is the AHJ - Glen Burnie/Pasadena/Edgewater/ Annapolis → Anne Arundel County) at ~$0.008/doc; Opus 5 added nothing over Sonnet at 2.4x. An AHJ rule was also added to the extraction prompt (jurisdiction = permit-issuing authority, per Dustin’s 8/10 ruling), and max_tokens went 1024 → 4096 because Sonnet’s adaptive thinking bills against the cap.
Same day, a Sonnet adjudication pass judged the entire backfill review queue (1,162 projects / 1,658 fields, one judge call per project re-reading its documents, $11.65 via Batch API). Key insight: the judge catches wrong-property document linkage, which the 2-doc corroboration rule cannot see (a mis-linked doc agrees with itself). Applied on Dustin’s go: 938 fields accepted + 192 rejects cleared; queue fell 1,866 → 343 in one day. Jacob’s 239 parked ties were excluded from bulk actions and instead got advisory AI-check chips in the portal Data Backfill page (backfill_service merges /data/shepherd/backfill_adjudication.json into /candidates, stale-guarded; BackfillReview.tsx renders verdict + reason).
Ops gotchas recorded: PostgREST count queries silently truncate at the
default 1,000-row page (use Prefer: count=exact); backfill_service
_record_decisions swallows exceptions with only a log line, and docker-exec’d
processes don’t write to the container log.
Scripts: /opt/scripts/{sonnet5_calibration,adjudicate_backfill,apply_adjudication}_20260810.py