DealDocumentScreening/docs/PHASE1_HANDOFF.md

# Phase 1 Handoff — Extraction Layer Refactor

> Scope: implement the structured-extraction port from `document-extraction-spec.md`,
> preserving the hexagonal architecture and keeping the existing pipeline intact.
> Completed: 2026-08-16.

## Goal

Replace flat-text extraction (`core.analysis.extractor.extract_text`) with a
format-aware extraction **port** that returns Markdown with preserved structure
(headings from DOCX, tables from PDF). The rest of the pipeline (chunker,
worker-analyze, RabbitMQ contracts) remains unchanged except for Markdown-aware
chunking and additive metadata in `DocumentExtracted`.

## New / modified files

### New: `core/extraction/` hexagonal package

| File | Purpose |
|---|---|
| `src/contract_check/core/extraction/port.py` | `DocumentExtractor` Protocol, `ExtractedDocument` DTO, `UnsupportedFormatError`, `ExtractionFailedError`, `MIN_TEXT_CHARS` |
| `src/contract_check/core/extraction/factory.py` | `ExtractorFactory`, `detect_format()`, `SUPPORTED_SUFFIXES`; routes bytes/mime/suffix → adapter. Uses `python-magic` for content sniffing, suffix as fallback. |
| `src/contract_check/core/extraction/adapters/pdf_pymupdf.py` | `PyMuPDFExtractor` — text + `page.find_tables()` → Markdown pipes. Interleaves tables and text blocks by bbox so cell text is not duplicated. `is_structured=True` iff tables found. |
| `src/contract_check/core/extraction/adapters/docx_mammoth.py` | `MammothDocxExtractor` — `mammoth.convert_to_markdown()`. Headings/lists/tables become Markdown; `is_structured=True` on headings/tables/lists. |
| `src/contract_check/core/extraction/adapters/rtf_striprtf.py` | `RtfExtractor` — `striprtf` to plaintext. `is_structured=False`. |
| `src/contract_check/core/extraction/adapters/txt_chardet.py` | `TxtExtractor` — `chardet` encoding detection, decode, metadata includes `encoding`. |
| `src/contract_check/core/extraction/adapters/ocr_tesseract.py` | `TesseractOcrExtractor` — OCR for scanned PDFs and raster images (PNG/JPG/TIFF), reusing the same pytesseract/pymupdf logic as `core.analysis.ocr`. |
| `src/contract_check/core/extraction/__init__.py` | `extract_document()` orchestrator: adapter → extract → OCR fallback for short PDFs. |
| `src/contract_check/core/extraction/adapters/__init__.py` | Re-exports of all adapter classes. |

### Modified worker / pipeline

| File | Change |
|---|---|
| `src/contract_check/worker_extract/handler.py` | Bytes-based extraction via `core.extraction.extract_document()`. No temp files. Uploaded text now `text/markdown; charset=utf-8`. `DocumentExtracted` includes `is_structured`/`has_tables`. Duration histogram labeled by `format`. Classification updated for new errors. |
| `src/contract_check/worker_extract/consumer.py` | Removed `@extract_duration.time()` decorator — labeled timing moved into the handler where the format is known. |
| `src/contract_check/core/mq/messages.py` | `DocumentExtracted` added `is_structured: bool = False` and `has_tables: bool = False` (backward-compatible defaults). |
| `src/contract_check/core/metrics.py` | `extract_duration` now has `format` label. Added `extraction_total{format,structured}` counter. |
| `src/contract_check/core/analysis/chunker.py` | Added `chunk_markdown()` — splits on ATX headings, keeps headings attached to their body, falls back to `chunk_text()` for unstructured text. |
| `src/contract_check/core/llm/ollama_cloud.py` | Analyzer now uses `chunk_markdown(text)` instead of `chunk_text(text)` so heading boundaries are preserved when the extractor produced structured Markdown. |
| `src/contract_check/api/services.py` | Upload gate now imports `SUPPORTED_SUFFIXES` from `core.extraction`. `_content_type_from_suffix()` extended for RTF/TXT/CSV/images. |
| `src/contract_check/bot/handlers.py` | Bot extension gate now uses the same `SUPPORTED_SUFFIXES` set and content-type map. |
| `src/contract_check/worker_extract/extract_document.py` | **Deleted**. Superseded by `core.extraction.extract_document()`. |

### Dependencies / Docker

| File | Change |
|---|---|
| `pyproject.toml` | `extract` group adds `mammoth`, `striprtf`, `chardet`, `python-magic`. Mypy ignore list extended for these libs. |
| `srv/worker-extract/Dockerfile` | Adds `libmagic1` apt package (required by `python-magic`). |
| `uv.lock` | Regenerated. |

### Documentation

| File | Change |
|---|---|
| `docs/ARCHITECTURE.md` | Layout updated (`core/extraction/` added, `worker_extract/extract_document.py` removed, new tests listed). Pipeline notes that extracted Markdown is uploaded. `DocumentExtracted` schema updated. Metrics table updated. pyproject `extract` group updated. worker-extract Dockerfile apt row updated. |
| `src/contract_check/api/routes/README.md` | Upload endpoint extension list updated. |

### Tests

| File | Change |
|---|---|
| `tests/unit/test_extraction_adapters.py` | New: PDF text-only/table, DOCX heading, RTF, TXT chardet, OCR error paths, DTO defaults. |
| `tests/unit/test_extraction_factory.py` | New: suffix/mime/magic routing, unsupported formats, end-to-end `extract_document()`, OCR fallback behavior. |
| `tests/unit/test_chunker.py` | New: `chunk_markdown` heading preservation, packing, oversized-section split, preamble handling, fallback. |
| `tests/integration/test_extract_worker.py` | Expected exceptions updated: `ExtractionFailedError` is now a possible raised error alongside `OCRError` (the failure-class mapping remains `extraction_failed` / `ocr_failed`). |

## Behavior changes

- **Format support** (upload-gate accepted): `.pdf`, `.docx`, `.rtf`, `.txt`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`.
- **Output format**: extracted text is now Markdown; stored with content-type `text/markdown; charset=utf-8`.
- **Backward compatibility**: `DocumentExtracted` new fields have defaults, so in-flight messages during deploy are valid.
- **Refund policy unchanged**: `ExtractionFailedError` → `extraction_failed` (not refundable under `infra_only`); `OCRError` → `ocr_failed` (refundable).
- **Chunking**: `worker-analyze` uses heading-aware `chunk_markdown()` when structure exists; otherwise identical to previous `chunk_text()`.

## Verification

```bash
make lint       # ruff check + format check — passed
make typecheck  # mypy src — passed (92 files)
make test-unit  # 132 passed, 32 deselected integration tests
```

Integration tests (`tests/integration/test_extract_worker.py` and
`tests/integration/test_upload_pipeline.py`) passed with the live Docker stack.
The full `make test-integration` suite is slow because it exercises the real LLM
path; the extraction-specific integration subset is green.

## Deployment notes

1. The `worker-extract` Docker image must be rebuilt because:
   - new apt package `libmagic1`
   - new Python deps `mammoth`, `striprtf`, `chardet`, `python-magic`
2. Existing running workers should be recreated (`docker compose --profile services up -d --build worker-extract`).
3. No DB migration required — changes are additive to message schema and object storage content.

## Rollback

- The new code is additive; old messages with fewer `DocumentExtracted` fields
  still validate. If rollback is needed, the previous worker-extract image
  continues to work, but extracted objects would be stored as `text/plain`
  instead of `text/markdown`.

## Deviations from the spec

- `ExtractedDocument.attachments: list[bytes]` was omitted — there is no consumer
  or storage design for extracted images yet.
- Heavy adapters (`marker`, `paddleocr`, `easyocr`, `img2table`, `email`) remain
  deferred per the spec.

## Next step

Phase 2: prescreen stage between `worker-extract` and `worker-analyze`. Based
on `SPIKE_PHASE0.md`, this will use a deterministic regex+pydantic extractor for
RU/BY contracts instead of the Needle 2 model (English-only in testing).