DealDocumentScreening/docs/PHASE1_HANDOFF.md

Phase 1 Handoff — Extraction Layer Refactor

Scope: implement the structured-extraction port from document-extraction-spec.md, preserving the hexagonal architecture and keeping the existing pipeline intact. Completed: 2026-08-16.

Goal

Replace flat-text extraction (core.analysis.extractor.extract_text) with a format-aware extraction port that returns Markdown with preserved structure (headings from DOCX, tables from PDF). The rest of the pipeline (chunker, worker-analyze, RabbitMQ contracts) remains unchanged except for Markdown-aware chunking and additive metadata in DocumentExtracted.

New / modified files

New: core/extraction/ hexagonal package

File	Purpose
src/contract_check/core/extraction/port.py	DocumentExtractor Protocol, ExtractedDocument DTO, UnsupportedFormatError, ExtractionFailedError, MIN_TEXT_CHARS
src/contract_check/core/extraction/factory.py	ExtractorFactory, detect_format(), SUPPORTED_SUFFIXES; routes bytes/mime/suffix → adapter. Uses python-magic for content sniffing, suffix as fallback.
src/contract_check/core/extraction/adapters/pdf_pymupdf.py	PyMuPDFExtractor — text + page.find_tables() → Markdown pipes. Interleaves tables and text blocks by bbox so cell text is not duplicated. is_structured=True iff tables found.
src/contract_check/core/extraction/adapters/docx_mammoth.py	MammothDocxExtractor — mammoth.convert_to_markdown(). Headings/lists/tables become Markdown; is_structured=True on headings/tables/lists.
src/contract_check/core/extraction/adapters/rtf_striprtf.py	RtfExtractor — striprtf to plaintext. is_structured=False.
src/contract_check/core/extraction/adapters/txt_chardet.py	TxtExtractor — chardet encoding detection, decode, metadata includes encoding.
src/contract_check/core/extraction/adapters/ocr_tesseract.py	TesseractOcrExtractor — OCR for scanned PDFs and raster images (PNG/JPG/TIFF), reusing the same pytesseract/pymupdf logic as core.analysis.ocr.
src/contract_check/core/extraction/__init__.py	extract_document() orchestrator: adapter → extract → OCR fallback for short PDFs.
src/contract_check/core/extraction/adapters/__init__.py	Re-exports of all adapter classes.

Modified worker / pipeline

File	Change
src/contract_check/worker_extract/handler.py	Bytes-based extraction via core.extraction.extract_document(). No temp files. Uploaded text now text/markdown; charset=utf-8. DocumentExtracted includes is_structured/has_tables. Duration histogram labeled by format. Classification updated for new errors.
src/contract_check/worker_extract/consumer.py	Removed @extract_duration.time() decorator — labeled timing moved into the handler where the format is known.
src/contract_check/core/mq/messages.py	DocumentExtracted added is_structured: bool = False and has_tables: bool = False (backward-compatible defaults).
src/contract_check/core/metrics.py	extract_duration now has format label. Added extraction_total{format,structured} counter.
src/contract_check/core/analysis/chunker.py	Added chunk_markdown() — splits on ATX headings, keeps headings attached to their body, falls back to chunk_text() for unstructured text.
src/contract_check/core/llm/ollama_cloud.py	Analyzer now uses chunk_markdown(text) instead of chunk_text(text) so heading boundaries are preserved when the extractor produced structured Markdown.
src/contract_check/api/services.py	Upload gate now imports SUPPORTED_SUFFIXES from core.extraction. _content_type_from_suffix() extended for RTF/TXT/CSV/images.
src/contract_check/bot/handlers.py	Bot extension gate now uses the same SUPPORTED_SUFFIXES set and content-type map.
src/contract_check/worker_extract/extract_document.py	Deleted. Superseded by core.extraction.extract_document().

Dependencies / Docker

File	Change
pyproject.toml	extract group adds mammoth, striprtf, chardet, python-magic. Mypy ignore list extended for these libs.
srv/worker-extract/Dockerfile	Adds libmagic1 apt package (required by python-magic).
uv.lock	Regenerated.

Documentation

File	Change
docs/ARCHITECTURE.md	Layout updated (core/extraction/ added, worker_extract/extract_document.py removed, new tests listed). Pipeline notes that extracted Markdown is uploaded. DocumentExtracted schema updated. Metrics table updated. pyproject extract group updated. worker-extract Dockerfile apt row updated.
src/contract_check/api/routes/README.md	Upload endpoint extension list updated.

Tests

File	Change
tests/unit/test_extraction_adapters.py	New: PDF text-only/table, DOCX heading, RTF, TXT chardet, OCR error paths, DTO defaults.
tests/unit/test_extraction_factory.py	New: suffix/mime/magic routing, unsupported formats, end-to-end extract_document(), OCR fallback behavior.
tests/unit/test_chunker.py	New: chunk_markdown heading preservation, packing, oversized-section split, preamble handling, fallback.
tests/integration/test_extract_worker.py	Expected exceptions updated: ExtractionFailedError is now a possible raised error alongside OCRError (the failure-class mapping remains extraction_failed / ocr_failed).

Behavior changes

• Format support (upload-gate accepted): .pdf, .docx, .rtf, .txt, .csv, .png, .jpg, .jpeg, .tif, .tiff.
• Output format: extracted text is now Markdown; stored with content-type text/markdown; charset=utf-8.
• Backward compatibility: DocumentExtracted new fields have defaults, so in-flight messages during deploy are valid.
• Refund policy unchanged: ExtractionFailedError → extraction_failed (not refundable under infra_only); OCRError → ocr_failed (refundable).
• Chunking: worker-analyze uses heading-aware chunk_markdown() when structure exists; otherwise identical to previous chunk_text().

Verification

make lint       # ruff check + format check — passed
make typecheck  # mypy src — passed (92 files)
make test-unit  # 132 passed, 32 deselected integration tests

Integration tests (tests/integration/test_extract_worker.py and tests/integration/test_upload_pipeline.py) passed with the live Docker stack. The full make test-integration suite is slow because it exercises the real LLM path; the extraction-specific integration subset is green.

Deployment notes

1. The worker-extract Docker image must be rebuilt because:
   • new apt package libmagic1
   • new Python deps mammoth, striprtf, chardet, python-magic
2. Existing running workers should be recreated (docker compose --profile services up -d --build worker-extract).
3. No DB migration required — changes are additive to message schema and object storage content.

Rollback

• The new code is additive; old messages with fewer DocumentExtracted fields still validate. If rollback is needed, the previous worker-extract image continues to work, but extracted objects would be stored as text/plain instead of text/markdown.

Deviations from the spec

• ExtractedDocument.attachments: list[bytes] was omitted — there is no consumer or storage design for extracted images yet.
• Heavy adapters (marker, paddleocr, easyocr, img2table, email) remain deferred per the spec.

Next step

Phase 2: prescreen stage between worker-extract and worker-analyze. Based on SPIKE_PHASE0.md, this will use a deterministic regex+pydantic extractor for RU/BY contracts instead of the Needle 2 model (English-only in testing).