DealDocumentScreening/docs/PHASE2_HANDOFF.md

# Phase 2 Handoff — Prescreen Stage

> Scope: insert a deterministic prescreen stage between extraction and LLM
> analysis for Russian/Belarusian contracts.
> Completed: 2026-08-16.

## Goal

Short-circuit obvious low-risk contracts and protect the expensive LLM worker
from trivial documents. Use a deterministic regex+pydantic extractor because the
Needle 2 (cactus-needle) model is English-only and unusable for RU/BY contracts
(see `docs/SPIKE_PHASE0.md`).

## What changed

### DB schema (`migrations/versions/0006_prescreen.py`)

- New `prescreen_results` table with contract metadata, field-coverage score,
  routing decision, and lightweight auto-approve output.
- `documents.status` adds `'prescreening'`.
- `jobs.queue` adds `'prescreen'`.
- `reports` gains `prescreen_result_id` FK and `prescreen_meta` JSONB.

### Models (`src/contract_check/core/db/`)

- `models.py`: added `PrescreenResult` model and relationships.
- `enums.py`: `DocStatus` and `QueueName` literals updated.

### RabbitMQ topology (`core/mq/topology.py`)

- Added `prescreen.q`, `prescreen.retry.q`, `prescreen.dlq`.
- Added routing keys `prescreen` / `retry.prescreen`.

### Messages (`core/mq/messages.py`)

- `PrescreenRequested` — worker-extract → prescreen.q.
- `PrescreenCompleted` — prescreen result, persisted and forwarded.
- `AnalyzeRequested` — worker-prescreen → analyze.q, carries `prescreen_meta`.
- `DocumentExtracted` kept for backward compatibility but no longer published.

### New service: `worker_prescreen/`

| File | Purpose |
|---|---|
| `extractor.py` | Regex extractor for RU/BY contracts; returns `PrescreenContractMeta` + confidence score. |
| `router.py` | Routing decision: `auto_approve` / `manual_review` / `deep_analysis`. `auto_approve` is disabled by default. |
| `handler.py` | Download Markdown, run extractor, persist `prescreen_results`, publish next message. |
| `consumer.py` | RabbitMQ consumer for `prescreen.q`. |
| `config.py` | `PrescreenSettings` with prefetch tunable. |
| `__main__.py` | Entrypoint, metrics server on `:9104`. |

### Wired existing services

- `worker_extract/handler.py`: now publishes `PrescreenRequested` to `prescreen.q`
  and sets `documents.status = 'prescreening'`.
- `worker_analyze/consumer.py` + `handler.py`: now consumes `AnalyzeRequested`,
  receives `prescreen_meta`, appends it to the LLM prompt as known contract
  fields.
- `core/llm/port.py` + `ollama_cloud.py`: `analyze()` accepts `extra_context: str`
  and injects it into each chunk's user prompt.

### Configuration (`core/config.py`)

New envs:

- `PRESCREEN_ENABLED` (default `true`)
- `PRESCREEN_AUTO_APPROVE` (default `false`)
- `PRESCREEN_CONFIDENCE_THRESHOLD` (default `0.75`)
- `PRESCREEN_HIGH_VALUE_THRESHOLD` (default `100000`)

### Metrics (`core/metrics.py`)

- `prescreen_duration_seconds{decision}`
- `prescreen_runs_total{decision,contract_type}`
- `prescreen_confidence` histogram

### Docker / compose

- New `srv/worker-prescreen/Dockerfile` (lean, no tesseract).
- Added `worker-prescreen` service to `docker-compose.yml` on port `9104`.

### Dependencies (`pyproject.toml`)

- Added `prescreen` dependency group (core + db/mq/s3/obs).
- Added to `dev` group.

### Tests

- Unit: `tests/unit/test_prescreen_extractor.py`, `tests/unit/test_prescreen_router.py`.
- Integration: updated `test_extract_worker.py`, `test_analyze_worker.py`,
  `test_upload_pipeline.py`, `test_b2b_api.py` for the new pipeline.

### Docs

- Updated `docs/ARCHITECTURE.md` with prescreen topology, messages, schema,
  metrics, and Docker/compose rows.

## Behavior

1. Upload → `extract.q` (unchanged).
2. worker-extract uploads Markdown to MinIO, publishes `PrescreenRequested`.
3. worker-prescreen:
   - extracts contract type, parties, amount, currency, dates, penalty/
     termination/arbitration clauses via regex
   - `confidence_score = matched_fields / total_fields`
   - routes:
     - `deep_analysis` if high value, penalty clause, or arbitration
     - `manual_review` if low confidence, missing parties, or auto_approve disabled
     - `auto_approve` only when `PRESCREEN_AUTO_APPROVE=true` and low risk
4. `deep_analysis` → `AnalyzeRequested` → worker-analyze with prescreen context.
5. `manual_review` → terminal DB state, no LLM call, credit **not** refunded
   (this is an intentional routing outcome, not a failure).
6. `auto_approve` → lightweight report row, `status=done`.

## Verification

```bash
make lint       # passed
make typecheck  # passed (99 files)
make test-unit  # 147 passed, 32 deselected
make test-integration  # 32 passed, 147 deselected (with live workers stopped to avoid races)
```

## Deployment notes

1. Run migration: `make migrate` (or `docker compose --profile services run --rm api alembic upgrade head`).
2. Rebuild images: `docker compose --profile services up -d --build`.
3. New worker-prescreen must be started; worker-extract and worker-analyze
   images also changed.
4. No application code outside core/workers was changed; the API just returns
   the new `prescreening` status string.

## Deviations from the original prescreen spec

- **Extractor:** regex instead of Needle 2. Rationale in `docs/SPIKE_PHASE0.md`.
- **No separate credits split:** the credit is still reserved once on upload;
  prescreen is treated as part of the same paid job.
- **No admin/web SPA yet:** `manual_review` documents are terminal in the DB
  until the admin panel lands.

## Next step

Phase 3 is storage modernization evaluation (RustFS watch). The current stack
keeps MinIO as the production default; RustFS is tracked as a future option.
Alternatively, continue with the backlog: admin/web SPA, ЮKassa payments, heavy
OCR adapters.

---

**Recommended immediate follow-up:** end-to-end smoke test in Docker with all
services running (`make services-up`) and a real PDF/DOCX upload through the
API or bot, verifying that `worker-prescreen` routes to `analyze.q` or
`manual_review` correctly.