document-extraction-pipeline

Document Extraction Pipeline (Phase 1 fleet)

Shipped: 2026-08-05 (entirely-api 04e0fec + 8397586, migration 033, deployed; first live batch submitted same session). The fleet stage of the ML/workflow-intelligence plan, following the passed calibration (memory project_phase1_extraction_calibration).

Shape

Hourly INTERVAL_TASKS job document_extraction → services/document_extraction_service.run_extraction_cycle in the worker:

1. poll_batches — retrieve every Anthropic Message Batch referenced by submitted rows; when ended, ingest results into document_extractions.extracted (jsonb) or error.
2. When nothing is in flight: discover_documents (glob /data/projects/*/CERTS|PERMITS/*.pdf, paginated known-path diff — NB an IN-filter on hundreds of paths 400s at the PostgREST URL limit) → submit_batch (400 docs; PyMuPDF text, <300 chars → skipped_no_text for the future OCR queue; claude-haiku-4-5 + structured outputs at 50% batch pricing).

Cycle never raises — a bad hour logs with traceback and self-heals next hour. ~27k text-layer docs drain in a few days at ~$0.0013/doc batch-priced.

Fleet schema (calibration fixes applied)

visit_dates is an ARRAY (multi-visit certs made a scalar ambiguous); doc_type enum definitions spelled out (cert packages titled “Field Inspection Certification Report” were misread as field_report); no enum on nullable unions (API 400).

Table: document_extractions (migration 033)

file_path (unique), project_ss_id (from path), doc_kind, status (pending | submitted | extracted | error | skipped_no_text), batch_id, model, extracted jsonb, error, timestamps.

Known benign race

Running a manual cycle while the scheduler’s own firing is mid-flight can double-submit the same 400 docs (both select pending before either marks submitted); happened once at launch — duplicate batch canceled, ~$0.25 waste, rows kept the second batch_id. Steady state is safe (ARQ dedups the scheduler job). Don’t run manual cycles while the hourly job is live unless checking status='submitted' is empty first.

Next

Phase B: extend DISCOVERY_ROOTS to the archives (path→project via project_files linkage) once project folders drain; OCR queue for skipped_no_text; downstream consumers = permit-timeline intelligence, cert QA, Supabase gap backfill.

Related: date-parser-consolidation, status-event-log.

OCR queue (2026-08-07, migration 036)

tesseract 5.5.0 baked into the shared image; document_ocr INTERVAL_TASK (30 min, 25 docs, 10-min budget) recovers skipped_no_text PDFs: PyMuPDF rasterizes 6 pages @200dpi → tesseract; native text layer + OCR output COMBINED with a 150-char acceptance floor (photo cover sheets hold permit#/county/address at ~250-500 combined chars — the 300-char native floor alone terminalized them; live fix took recovery from 1/6 to 5/6). Recovered rows → status pending + ocr_used + ocr_text (submit_batch prefers ocr_text); hopeless → no_text_final (terminal).