DealDocumentScreening/docs/PHASE2_HANDOFF.md

Phase 2 Handoff — Prescreen Stage

Scope: insert a deterministic prescreen stage between extraction and LLM analysis for Russian/Belarusian contracts. Completed: 2026-08-16.

Goal

Short-circuit obvious low-risk contracts and protect the expensive LLM worker from trivial documents. Use a deterministic regex+pydantic extractor because the Needle 2 (cactus-needle) model is English-only and unusable for RU/BY contracts (see docs/SPIKE_PHASE0.md).

What changed

DB schema (migrations/versions/0006_prescreen.py)

• New prescreen_results table with contract metadata, field-coverage score, routing decision, and lightweight auto-approve output.
• documents.status adds 'prescreening'.
• jobs.queue adds 'prescreen'.
• reports gains prescreen_result_id FK and prescreen_meta JSONB.

Models (src/contract_check/core/db/)

• models.py: added PrescreenResult model and relationships.
• enums.py: DocStatus and QueueName literals updated.

RabbitMQ topology (core/mq/topology.py)

• Added prescreen.q, prescreen.retry.q, prescreen.dlq.
• Added routing keys prescreen / retry.prescreen.

Messages (core/mq/messages.py)

• PrescreenRequested — worker-extract → prescreen.q.
• PrescreenCompleted — prescreen result, persisted and forwarded.
• AnalyzeRequested — worker-prescreen → analyze.q, carries prescreen_meta.
• DocumentExtracted kept for backward compatibility but no longer published.

New service: worker_prescreen/

File	Purpose
extractor.py	Regex extractor for RU/BY contracts; returns PrescreenContractMeta + confidence score.
router.py	Routing decision: auto_approve / manual_review / deep_analysis. auto_approve is disabled by default.
handler.py	Download Markdown, run extractor, persist prescreen_results, publish next message.
consumer.py	RabbitMQ consumer for prescreen.q.
config.py	PrescreenSettings with prefetch tunable.
__main__.py	Entrypoint, metrics server on :9104.

Wired existing services

• worker_extract/handler.py: now publishes PrescreenRequested to prescreen.q and sets documents.status = 'prescreening'.
• worker_analyze/consumer.py + handler.py: now consumes AnalyzeRequested, receives prescreen_meta, appends it to the LLM prompt as known contract fields.
• core/llm/port.py + ollama_cloud.py: analyze() accepts extra_context: str and injects it into each chunk's user prompt.

Configuration (core/config.py)

New envs:

• PRESCREEN_ENABLED (default true)
• PRESCREEN_AUTO_APPROVE (default false)
• PRESCREEN_CONFIDENCE_THRESHOLD (default 0.75)
• PRESCREEN_HIGH_VALUE_THRESHOLD (default 100000)

Metrics (core/metrics.py)

• prescreen_duration_seconds{decision}
• prescreen_runs_total{decision,contract_type}
• prescreen_confidence histogram

Docker / compose

• New srv/worker-prescreen/Dockerfile (lean, no tesseract).
• Added worker-prescreen service to docker-compose.yml on port 9104.

Dependencies (pyproject.toml)

• Added prescreen dependency group (core + db/mq/s3/obs).
• Added to dev group.

Tests

• Unit: tests/unit/test_prescreen_extractor.py, tests/unit/test_prescreen_router.py.
• Integration: updated test_extract_worker.py, test_analyze_worker.py, test_upload_pipeline.py, test_b2b_api.py for the new pipeline.

Docs

• Updated docs/ARCHITECTURE.md with prescreen topology, messages, schema, metrics, and Docker/compose rows.

Behavior

1. Upload → extract.q (unchanged).
2. worker-extract uploads Markdown to MinIO, publishes PrescreenRequested.
3. worker-prescreen:
   • extracts contract type, parties, amount, currency, dates, penalty/ termination/arbitration clauses via regex
   • confidence_score = matched_fields / total_fields
   • routes:
     • deep_analysis if high value, penalty clause, or arbitration
     • manual_review if low confidence, missing parties, or auto_approve disabled
     • auto_approve only when PRESCREEN_AUTO_APPROVE=true and low risk
4. deep_analysis → AnalyzeRequested → worker-analyze with prescreen context.
5. manual_review → terminal DB state, no LLM call, credit not refunded (this is an intentional routing outcome, not a failure).
6. auto_approve → lightweight report row, status=done.

Verification

make lint       # passed
make typecheck  # passed (99 files)
make test-unit  # 147 passed, 32 deselected
make test-integration  # 32 passed, 147 deselected (with live workers stopped to avoid races)

Deployment notes

1. Run migration: make migrate (or docker compose --profile services run --rm api alembic upgrade head).
2. Rebuild images: docker compose --profile services up -d --build.
3. New worker-prescreen must be started; worker-extract and worker-analyze images also changed.
4. No application code outside core/workers was changed; the API just returns the new prescreening status string.

Deviations from the original prescreen spec

• Extractor: regex instead of Needle 2. Rationale in docs/SPIKE_PHASE0.md.
• No separate credits split: the credit is still reserved once on upload; prescreen is treated as part of the same paid job.
• No admin/web SPA yet: manual_review documents are terminal in the DB until the admin panel lands.

Next step

Phase 3 is storage modernization evaluation (RustFS watch). The current stack keeps MinIO as the production default; RustFS is tracked as a future option. Alternatively, continue with the backlog: admin/web SPA, ЮKassa payments, heavy OCR adapters.

---

Recommended immediate follow-up: end-to-end smoke test in Docker with all services running (make services-up) and a real PDF/DOCX upload through the API or bot, verifying that worker-prescreen routes to analyze.q or manual_review correctly.