admin-git/DealDocumentScreening
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Sync docs and clean up unused.

Browse source
This commit is contained in:
febux 2026-09-06 15:03:51 +03:00
parent a1e119ce6e
commit 2d5c2b50c0
9 changed files with 43 additions and 1396 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
README.md
View file

@ -13,10 +13,7 @@ Pipeline: `PDF/DOCX/RTF/TXT/CSV/изображения (pymupdf / mammoth / tess
- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — **единственный источник истины** для production-рефакторинга. Hexagonal-архитектура, RabbitMQ-конвейер, схемы БД, очередь/ретраи/DLQ, конфиг, deploy, observability.
- [`docs/DEPLOY.md`](docs/DEPLOY.md) — руководство по развёртыванию: локально, Docker Compose, VPS, seed-token, backup, troubleshooting.
- [`docs/BUSINESS_IDEA.md`](docs/BUSINESS_IDEA.md) — идея и бизнес-модель.
- [`docs/IMPLEMENTATION_PLAN.md`](docs/IMPLEMENTATION_PLAN.md) — исходный «ленивый» план. Этап 0 (прототип) актуален; этапы 1+ superseded в `docs/ARCHITECTURE.md`.
- [`docs/TICKETS.md`](docs/TICKETS.md) — тикеты реализации со статусами.
- [`docs/PHASE2_HANDOFF.md`](docs/PHASE2_HANDOFF.md), [`docs/PHASES_2_PLUS_ROADMAP.md`](docs/PHASES_2_PLUS_ROADMAP.md), [`docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md`](docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md) — этап 2: prescreen-стейдж и дальнейший роадмап.
- [`docs/SPIKE_PHASE0.md`](docs/SPIKE_PHASE0.md) — спайки: prescreen-библиотеки, RustFS (артефакты в `rustfs-spike/`).
## Архитектура (одна строка)
@ -110,7 +107,6 @@ Makefile # повседневные команды (make h
docker-compose.yml # default = инфра; --profile services = стек; --profile edge = nginx+certbot
pyproject.toml # hatchling + PEP 735 dependency-groups (db/mq/s3/obs/api/extract/prescreen/analyze/notify/billing/bot/dev)
.env.example # полный список env (см. docs/ARCHITECTURE.md §11)
rustfs-spike/ # артефакты спайка RustFS (docs/SPIKE_PHASE0.md)
```
## Быстрый старт

339
docs/IMPLEMENTATION_PLAN.md
View file

@ -1,339 +0,0 @@
# План реализации «Контракт-чек» (Ollama Cloud / hosted LLM)
> **Примечание:** этот документ описывает исходную «ленивую» архитектуру с arq+Redis
> (очередь), Selectel S3 (хранилище) и единым Docker-образом с `MODE=api|worker|bot`.
> **Этап 0 (прототип) актуален.** Этапы 1+ superseded [`ARCHITECTURE.md`](ARCHITECTURE.md),
> где реализована production-архитектура: RabbitMQ-конвейер, MinIO, 4 worker-а
> (extract, prescreen, analyze, notify), aiogram-бот, B2B API, 7 Dockerfile-ов в `srv/`,
> dependency-groups (PEP 735).
> Принцип: ленивая архитектура. Каждый этап — минимальный работающий
> срез. Никаких абстракций «на потом». K8s, RabbitMQ, микросервисы —
> не нужны до $10k MRR.
>
> **LLM:** [Ollama Cloud](https://ollama.com/cloud) — hosted-инференс
> открытых моделей. Те же модели и API (`/api/chat`, `format: json`),
> что и у локального Ollama, но без своего железа: платим подписку +
> metered overage, инференс бежит у Ollama (NCP-партнёры, в основном США).
---
## О выборе Ollama Cloud
- **API:** стандартный Ollama HTTP API на cloud-эндпоинте. Авторизация —
API-ключ (bearer) из настроек аккаунта Ollama. Структурный вывод через
`format: json` (или JSON-schema) для парсимого отчёта.
- **Модели:** только cloud-enabled (см. `ollama.com/search?c=cloud`).
Рекомендация — **`qwen2.5:14b`** (сильный русский + reasoning).
Fallback — более лёгкая **`qwen2.5:7b`** при 429/квоте.
Теги БЕЗ суффикса `-instruct` (его нет в Ollama): `qwen2.5:14b`,
`llama3.1:8b`, `gemma2:9b` и т.п.
- **`format: json` ≠ соответствие схеме.** Ollama гарантирует синтаксис
JSON; соответствие вашей pydantic-схеме проверяем на клиенте + цикл
repair/retry. Не полагаемся на модель.
## Стоимость и лимиты Ollama Cloud (актуально на момент планирования)
| План | Цена | Конкарренси | Назначение | Статус |
|-------|-------------|-------------|-------------------------|------------------|
| Free | $0 | 1 | тесты | открыт |
| Pro | $20/мес | 3 | рабочая лошадка MVP | открыт |
| Max | $100/мес | 10 | тяжёлый поток | **приостановлен** |
| Team | $25/seat | — | команда (5 seat min) | waitlist |
| Enterprise | custom | custom | продакшн SaaS | по запросу |
- **Usage:** rolling-лимиты — сессия 5 ч + недельный 7 дн. Pro = 50× Free.
При превышении — **metered overage** (добиваем баланс, оплата по
токенам модели). Жёсткой стены нет, но конкарренси = 3 на Pro.
- **Хостинг: в основном США** (подключаются EU/Singapore). Это **ломает
152-ФЗ data-residency**, см. раздел «Риски».
## Экономика проекта (честно)
- Опекс фиксированный до квоты: VPS ~4 €/мес + Ollama Cloud Pro $20/мес +
Selectel S3 (копейки). Итого ~$25–30/мес на старте.
- Рост объёма → metered overage Ollama Cloud по токенам. Маржа на
pay-per-doc (199 ₽) остаётся, **если** один договор съедает мало
квоты. Это **обязательно измерить на этапе 0** (квота/документ).
- В отличие от локального Ollama — нет потолка по железу, но есть потолок
по конкарренси (3) и плавающий cost при overage. В отличие от GigaChat —
cost в $, не в ₽, и нет РФ-локализации данных.
---
## Архитектура — hexagonal (ports & adapters)
Ядро и адаптеры разделены. Это снимает связность «бот знает про БД/S3/LLM» и
позволяет добавлять каналы доставки (CLI, веб, B2B API) без дублирования
бизнес-логики.
- **Ядро (application core):**
- `api` (FastAPI/uvicorn) — владеет БД, S3, кредитами, оплатой; принимает
документы, **резервирует кредит на enqueue**, ставит arq-задачи в redis,
отдаёт отчёты/профиль. Аутентификация адаптеров — общий `SERVICE_TOKEN`.
- `worker` (arq) — разбирает очередь: S3 → экстракт → (OCR) → chunker →
analyzer → Ollama Cloud → Report. Idempotency + refund при ошибке.
- Общий доменный пакет `contract_check/` (`extractor`, `chunker`,
`llm_client`, `analyzer`, `checklist`, `report_schema`, `ocr`, `storage`,
`db`, `models`, `payments`, `quota`) — используется только ядром.
- **Адаптеры (delivery = HTTP-клиенты к `api`):**
- `bot` (aiogram) — Telegram: приём документа → `POST /documents` (multipart)
→ отдача отчёта. **Не трогает БД/S3/redis/LLM.**
- `cli` — пользовательский/отладочный CLI поверх api (новый, отдельный от
прототипа).
- `web` (React SPA) — этап 2, тоже HTTP-клиент к api.
- **Прототип `prototype.py` (stage 0)** — standalone, in-process LLM; артефакт
go/no-go, **не адаптер и не ядро**.
> Следствие для планирования: **api + worker строятся на этапе 1** (боту-адаптеру
> нужен api). Этап 2 = добавление адаптера `web` + подписок (api уже есть).
---
## Этап 0 — Прототип (1 выходной)
**Цель:** доказать, что модель через Ollama Cloud реально находит риски,
и измерить задержку + **квоту на один договор** (для юнит-экономики).
**Деливерэбл:** один файл `prototype.py` — end-to-end.
```
PDF/DOCX → pymupdf/python-docx → текст →
prompt с чек-листом → Ollama Cloud (format: json) → отчёт в markdown
```
**Что делаем:**
1. Завести аккаунт Ollama, взять API-ключ, положить в `.env`.
2. Поставить `pymupdf`, `httpx` в venv.
3. Один скрипт: читает PDF → достаёт текст → промпт с чек-листом из 10
пунктов → дёргает Ollama Cloud → печатает отчёт.
4. Прогнать 3–5 реальных договоров (NDA, оказание услуг, поставка).
5. Зафиксировать: качество находок, задержка (сек), % битых JSON,
**сколько квоты/токенов на договор**.
**Чего НЕ делаем:** база, веб, бот, оплату, деплой. Один скрипт локально.
**Критерий успеха:** отчёт по реальному договору содержит хотя бы 3
осмысленные находки с цитатами; задержка < 60 сек; квота/договор даёт
понятную маржинальность при 199 ₽.
---
## Этап 1 — Telegram-бот MVP (2–3 недели)
**Цель:** первые платящие пользователи. Бот = самый быстрый путь до ЦА.
### 1.1 Структура проекта
```
contract_check/
├── pyproject.toml # hatchling, deps (core + adapters)
├── docker-compose.yml # postgres, redis, api, workers, bot (profiles: services, bot, edge)
├── .env.example
├── src/contract_check/
│ ├── main.py # диспетчер: MODE=api|worker|bot|cli
│ ├── prototype.py # stage-0 standalone (in-process LLM) — удалён, больше не нужен
│ │ # ── ЯДРО (application core): владеет БД/S3/кредитами/LLM ──
│ ├── api.py # FastAPI: /documents, /reports/{id}, /me, /healthz
│ ├── worker.py # arq analyze_document(doc_id): S3→extract→OCR→analyze→Report
│ ├── analyzer.py extractor.py chunker.py ocr.py
│ ├── llm_client.py checklist.py report_schema.py
│ ├── storage.py db.py models.py payments.py quota.py
│ │ # ── АДАПТЕРЫ (delivery): HTTP-клиенты к api, без БД/S3/LLM ──
│ ├── bot.py # aiogram: приём документа → POST /documents → отчёт
│ └── cli.py # CLI поверх api (новый, не прототип)
├── migrations/ # alembic
└── tests/
```
### 1.2 База данных — 3 таблицы
```sql
users (id, telegram_id, created_at, credits_left)
documents (id, user_id, s3_key, status, created_at)
reports (id, document_id, content_json, created_at)
```
`credits_left` — prepaid-кредиты (pay-per-doc). Подписок в v1 нет.
`status` хранить как `TEXT` + `CHECK`, не Postgres-ENUM (миграции проще).
### 1.3 Поток (резервирование кредита на enqueue)
```
Юзер кидает PDF в бот
→ бот (adapter): multipart'ом шлёт файл в api: POST /documents (+ SERVICE_TOKEN)
→ api (core): get_or_create_user, проверяет credits_left > 0, РЕЗЕРВИРУЕТ (-= 1),
грузит файл в S3, создаёт Document(status=queued), ставит arq-таску → 202 + job_id
→ worker (core): idempotency guard по status, достаёт из S3 → текст → (OCR если скан)
→ worker: chunker при необходимости → analyzer → Ollama Cloud → отчёт
→ worker: pydantic-валидация + repair-loop; пишет Report, status=done
→ при ошибке: status=failed, ВОЗВРАТ кредита (+1)
→ бот: опрашивает GET /reports/{id} (или push), отправляет отчёт
(с разбивкой/файлом при >4096 симв.), футер-disclaimer
```
### 1.4 Оплата (минимум)
- ЮKassa: бот генерирует ссылку на оплату N ₽ → webhook пополняет
`credits_left`.
- Один тариф: 199 ₽ = 1 документ. Без подписок.
### 1.5 Деплой
- **Один VPS** (Hetzner CX22 ~4 €/мес или Selectel под РФ-локацию БД).
- Docker Compose: `postgres`, `redis`, `api`, `worker`, `bot`. **Без GPU.**
(Бот — адаптер к api; веб-адаптер `web` добавится на этапе 2.)
- S3 — Selectel Object Storage.
- Домен + HTTPS только для webhook ЮKassa; на старте ngrok (с оговоркой:
URL на free-ngrok меняется → лучше сразу дешёвый домен + Caddy).
### 1.6 Критерий успеха
10 платящих. MRR ~2 000 ₽. Отчёты не вызывают жалоб «ничего не нашёл».
Ollama Cloud Pro покрывает нагрузку без ухода в overage.
---
## Этап 2 — Веб + подписки (3–4 недели)
**Цель:** B2B-веб-интерфейс и подписки.
### 2.1 Что добавляем
- **`api` уже построен на этапе 1** (T-E1-015). На этапе 2 добавляем: роуты
подписок/счетов, Telegram Login auth (сессия/JWT) для веб-адаптера, и сам
веб-адаптер `web` (React SPA) — ещё один HTTP-клиент к api.
- React SPA (Vite): загрузка, история, профиль, подписка. Отдаётся статикой
позже (отдельный `web`-контейнер / Nginx).
- Auth: Telegram Login Widget (нужен публичный HTTPS-домен в BotFather) →
сессия/JWT.
- Подписки: solo (1 490 ₽), team (3 990 ₽). ЮKassa recurring.
### 2.2 Что меняем в базе
```sql
ALTER TABLE users ADD COLUMN plan TEXT DEFAULT 'free';
ALTER TABLE users ADD COLUMN plan_renews_at TIMESTAMP;
CREATE TABLE invoices (id, user_id, amount, status, provider, external_id, created_at);
```
`credits_left` остаётся для pay-per-doc; подписка = безлимит с monthly
reset через cron-arq-таску.
### 2.3 Архитектура — без изменений в ядре
`api` и `worker` работают с этапа 1; на этапе 2 добавляется только адаптер
`web` (React, статика) перед Nginx → HTTPS (Let's Encrypt). Hexagonal-граница
сохранена: `web` — такой же HTTP-клиент к api, как `bot`. LLM по-прежнему
Ollama Cloud; при росте — Enterprise тариф или переход на свой GPU.
### 2.4 Критерий успеха
50 платящих. MRR ~30 000 ₽. Есть хотя бы один team-клиент.
---
## Этап 3 — B2B API (2 недели, только если есть спрос)
**Цель:** сторонние сервисы дёргают анализ через API.
### 3.1 Что добавляем
- API-ключи: таблица `api_keys`, header `X-API-Key`.
- Rate-limit: Redis (token bucket). Лимит = **конкарренси/квота Ollama
Cloud** (3 на Pro), а не ₽.
- `POST /api/v1/analyze` (multipart) → 202 + job_id →
`GET /api/v1/reports/{id}`.
- Дашборд: ключи, usage, счета.
### 3.2 Чего НЕ делаем
- Нет SDK, нет вебхуков (клиент поллит), нет OAuth2.
- Один тариф API: 9 900 ₽/мес за 100 запросов.
### 3.3 Критерий успеха
3 API-клиента. MRR +30 000 ₽.
---
## Инфраструктура — сводка
| Компонент | Этап 0 | Этап 1 | Этап 2–3 |
|----------------|------------|------------------------------|--------------------------------|
| Compute | ноутбук | 1 VPS (Docker Compose) | 1 VPS (апгрейд RAM) |
| LLM | Ollama Cloud (Free/Pro) | Ollama Cloud Pro | Ollama Cloud Pro/Enterprise |
| DB | — | Postgres (в compose) | Postgres (+ backup cron) |
| Queue | — | Redis (в compose) | Redis (тот же) |
| Object storage | — | Selectel S3 | Selectel S3 |
| OCR | — | Tesseract (локально) | Tesseract (+ Yandex Vision) |
| Payments | — | ЮKassa | ЮKassa + CloudPayments |
| Monitoring | — | Docker logs | Uptime Kuma + Sentry (free) |
| CI/CD | — | git push → ssh deploy | GitHub Actions → build → deploy|
**K8s / RabbitMQ / Kafka / Elasticsearch / vLLM / TGI — НЕ НУЖНЫ.** Один
VPS + Ollama Cloud держит всё до заметного объёма. Свой GPU — только если
Ollama Cloud overage станет дороже self-host (отдельное решение позже).
---
## Чек-лист пунктов анализа (v1)
Содержимое `checklist.py` — один список, без БД:
1. Неустойки / штрафы (размер, односторонний)
2. Подсудность (чужой регион)
3. Сроки оплаты (условия, просрочка)
4. IP-права (кому отходят результаты)
5. Одностороннее изменение условий
6. Гарантии и их срок
7. Форс-мажор (формулировки)
8. НДС (включён / сверх)
9. Ответственность сторон (cap, исключения)
10. Расторжение (условия, уведомление)
---
## Сроки (реалистично, соло, вечера/выходные)
| Этап | Что | Время |
|------|------------------|-----------|
| 0 | Прототип | 1 выходной|
| 1 | Telegram-бот MVP | 2–3 недели|
| 2 | Веб + подписки | 3–4 недели|
| 3 | B2B API | 2 недели |
**До первого платящего (0+1): ~3–4 недели.** Заложить ~1–2 дня на подбор
cloud-модели и тюнинг промпта (локальные/open модели капризнее GPT-4o).
---
## Риски Ollama Cloud (честно)
1. **152-ФЗ / data-residency.** Контракт улетает в Ollama Cloud (США).
Это тот же класс риска, что и GPT-4o. Митигация: disclaimer,
обезличивание ПДн перед отправкой, либо при необходимости —
отказ от Ollama Cloud в пользу РФ-LLM (GigaChat/YandexGPT) или своего
GPU-бокса. **Не позиционировать продукт как «данные не покидают РФ».**
2. **Квота/конкарренси.** Pro = 3 одновременных модели + rolling usage.
Бурст платящих юзеров упрётся в очередь/429. Трекать usage,
алертить у лимита, при росте — overage-баланс или Enterprise.
3. **Качество open-моделей.** 14B галлюцинирует/пересказывает цитаты
сильнее GPT-4o/GigaChat. Всегда: цитата + номер пункта + ремонт-цикл
валидации JSON. Disclaimer «не заменяет юриста» — в каждый отчёт.
4. **Зависимость от одного провайдера.** Один аккаунт Ollama (нельзя
несколько). Иметь готовый план Б: GigaChat/YandexGPT-фолбэк или свой
GPU при блокировке/превышении квоты.
---
## Что сознательно отложено (YAGNI)
- Multi-tenant / организации / роли — пока все юзеры = solo.
- Шаблоны договоров (генерация) — другой продукт.
- ЭЦП / Госуслуги — чужой регуляторный ад.
- Команда юристов (human-in-the-loop) — только если попросят.
- Mobile app — веб + бот закрывают 100%.
- White-label — один продукт, один бренд.
- Свой GPU / vLLM / TGI / RAG-над-векторной-базой — пока Ollama Cloud
дешевле; пересмотрим при выходе overage в минус.

116
docs/PHASE1_HANDOFF.md
View file

@ -1,116 +0,0 @@
# Phase 1 Handoff — Extraction Layer Refactor
> Scope: implement the structured-extraction port from `document-extraction-spec.md`,
> preserving the hexagonal architecture and keeping the existing pipeline intact.
> Completed: 2026-08-16.
## Goal
Replace flat-text extraction (`core.analysis.extractor.extract_text`) with a
format-aware extraction **port** that returns Markdown with preserved structure
(headings from DOCX, tables from PDF). The rest of the pipeline (chunker,
worker-analyze, RabbitMQ contracts) remains unchanged except for Markdown-aware
chunking and additive metadata in `DocumentExtracted`.
## New / modified files
### New: `core/extraction/` hexagonal package
| File | Purpose |
|---|---|
| `src/contract_check/core/extraction/port.py` | `DocumentExtractor` Protocol, `ExtractedDocument` DTO, `UnsupportedFormatError`, `ExtractionFailedError`, `MIN_TEXT_CHARS` |
| `src/contract_check/core/extraction/factory.py` | `ExtractorFactory`, `detect_format()`, `SUPPORTED_SUFFIXES`; routes bytes/mime/suffix → adapter. Uses `python-magic` for content sniffing, suffix as fallback. |
| `src/contract_check/core/extraction/adapters/pdf_pymupdf.py` | `PyMuPDFExtractor` — text + `page.find_tables()` → Markdown pipes. Interleaves tables and text blocks by bbox so cell text is not duplicated. `is_structured=True` iff tables found. |
| `src/contract_check/core/extraction/adapters/docx_mammoth.py` | `MammothDocxExtractor` — `mammoth.convert_to_markdown()`. Headings/lists/tables become Markdown; `is_structured=True` on headings/tables/lists. |
| `src/contract_check/core/extraction/adapters/rtf_striprtf.py` | `RtfExtractor` — `striprtf` to plaintext. `is_structured=False`. |
| `src/contract_check/core/extraction/adapters/txt_chardet.py` | `TxtExtractor` — `chardet` encoding detection, decode, metadata includes `encoding`. |
| `src/contract_check/core/extraction/adapters/ocr_tesseract.py` | `TesseractOcrExtractor` — OCR for scanned PDFs and raster images (PNG/JPG/TIFF), reusing the same pytesseract/pymupdf logic as `core.analysis.ocr`. |
| `src/contract_check/core/extraction/__init__.py` | `extract_document()` orchestrator: adapter → extract → OCR fallback for short PDFs. |
| `src/contract_check/core/extraction/adapters/__init__.py` | Re-exports of all adapter classes. |
### Modified worker / pipeline
| File | Change |
|---|---|
| `src/contract_check/worker_extract/handler.py` | Bytes-based extraction via `core.extraction.extract_document()`. No temp files. Uploaded text now `text/markdown; charset=utf-8`. `DocumentExtracted` includes `is_structured`/`has_tables`. Duration histogram labeled by `format`. Classification updated for new errors. |
| `src/contract_check/worker_extract/consumer.py` | Removed `@extract_duration.time()` decorator — labeled timing moved into the handler where the format is known. |
| `src/contract_check/core/mq/messages.py` | `DocumentExtracted` added `is_structured: bool = False` and `has_tables: bool = False` (backward-compatible defaults). |
| `src/contract_check/core/metrics.py` | `extract_duration` now has `format` label. Added `extraction_total{format,structured}` counter. |
| `src/contract_check/core/analysis/chunker.py` | Added `chunk_markdown()` — splits on ATX headings, keeps headings attached to their body, falls back to `chunk_text()` for unstructured text. |
| `src/contract_check/core/llm/ollama_cloud.py` | Analyzer now uses `chunk_markdown(text)` instead of `chunk_text(text)` so heading boundaries are preserved when the extractor produced structured Markdown. |
| `src/contract_check/api/services.py` | Upload gate now imports `SUPPORTED_SUFFIXES` from `core.extraction`. `_content_type_from_suffix()` extended for RTF/TXT/CSV/images. |
| `src/contract_check/bot/handlers.py` | Bot extension gate now uses the same `SUPPORTED_SUFFIXES` set and content-type map. |
| `src/contract_check/worker_extract/extract_document.py` | **Deleted**. Superseded by `core.extraction.extract_document()`. |
### Dependencies / Docker
| File | Change |
|---|---|
| `pyproject.toml` | `extract` group adds `mammoth`, `striprtf`, `chardet`, `python-magic`. Mypy ignore list extended for these libs. |
| `srv/worker-extract/Dockerfile` | Adds `libmagic1` apt package (required by `python-magic`). |
| `uv.lock` | Regenerated. |
### Documentation
| File | Change |
|---|---|
| `docs/ARCHITECTURE.md` | Layout updated (`core/extraction/` added, `worker_extract/extract_document.py` removed, new tests listed). Pipeline notes that extracted Markdown is uploaded. `DocumentExtracted` schema updated. Metrics table updated. pyproject `extract` group updated. worker-extract Dockerfile apt row updated. |
| `src/contract_check/api/routes/README.md` | Upload endpoint extension list updated. |
### Tests
| File | Change |
|---|---|
| `tests/unit/test_extraction_adapters.py` | New: PDF text-only/table, DOCX heading, RTF, TXT chardet, OCR error paths, DTO defaults. |
| `tests/unit/test_extraction_factory.py` | New: suffix/mime/magic routing, unsupported formats, end-to-end `extract_document()`, OCR fallback behavior. |
| `tests/unit/test_chunker.py` | New: `chunk_markdown` heading preservation, packing, oversized-section split, preamble handling, fallback. |
| `tests/integration/test_extract_worker.py` | Expected exceptions updated: `ExtractionFailedError` is now a possible raised error alongside `OCRError` (the failure-class mapping remains `extraction_failed` / `ocr_failed`). |
## Behavior changes
- **Format support** (upload-gate accepted): `.pdf`, `.docx`, `.rtf`, `.txt`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`.
- **Output format**: extracted text is now Markdown; stored with content-type `text/markdown; charset=utf-8`.
- **Backward compatibility**: `DocumentExtracted` new fields have defaults, so in-flight messages during deploy are valid.
- **Refund policy unchanged**: `ExtractionFailedError` → `extraction_failed` (not refundable under `infra_only`); `OCRError` → `ocr_failed` (refundable).
- **Chunking**: `worker-analyze` uses heading-aware `chunk_markdown()` when structure exists; otherwise identical to previous `chunk_text()`.
## Verification
```bash
make lint # ruff check + format check — passed
make typecheck # mypy src — passed (92 files)
make test-unit # 132 passed, 32 deselected integration tests
```
Integration tests (`tests/integration/test_extract_worker.py` and
`tests/integration/test_upload_pipeline.py`) passed with the live Docker stack.
The full `make test-integration` suite is slow because it exercises the real LLM
path; the extraction-specific integration subset is green.
## Deployment notes
1. The `worker-extract` Docker image must be rebuilt because:
- new apt package `libmagic1`
- new Python deps `mammoth`, `striprtf`, `chardet`, `python-magic`
2. Existing running workers should be recreated (`docker compose --profile services up -d --build worker-extract`).
3. No DB migration required — changes are additive to message schema and object storage content.
## Rollback
- The new code is additive; old messages with fewer `DocumentExtracted` fields
still validate. If rollback is needed, the previous worker-extract image
continues to work, but extracted objects would be stored as `text/plain`
instead of `text/markdown`.
## Deviations from the spec
- `ExtractedDocument.attachments: list[bytes]` was omitted — there is no consumer
or storage design for extracted images yet.
- Heavy adapters (`marker`, `paddleocr`, `easyocr`, `img2table`, `email`) remain
deferred per the spec.
## Next step
Phase 2: prescreen stage between `worker-extract` and `worker-analyze`. Based
on `SPIKE_PHASE0.md`, this will use a deterministic regex+pydantic extractor for
RU/BY contracts instead of the Needle 2 model (English-only in testing).

153
docs/PHASE2_HANDOFF.md
View file

@ -1,153 +0,0 @@
# Phase 2 Handoff — Prescreen Stage
> Scope: insert a deterministic prescreen stage between extraction and LLM
> analysis for Russian/Belarusian contracts.
> Completed: 2026-08-16.
## Goal
Short-circuit obvious low-risk contracts and protect the expensive LLM worker
from trivial documents. Use a deterministic regex+pydantic extractor because the
Needle 2 (cactus-needle) model is English-only and unusable for RU/BY contracts
(see `docs/SPIKE_PHASE0.md`).
## What changed
### DB schema (`migrations/versions/0006_prescreen.py`)
- New `prescreen_results` table with contract metadata, field-coverage score,
routing decision, and lightweight auto-approve output.
- `documents.status` adds `'prescreening'`.
- `jobs.queue` adds `'prescreen'`.
- `reports` gains `prescreen_result_id` FK and `prescreen_meta` JSONB.
### Models (`src/contract_check/core/db/`)
- `models.py`: added `PrescreenResult` model and relationships.
- `enums.py`: `DocStatus` and `QueueName` literals updated.
### RabbitMQ topology (`core/mq/topology.py`)
- Added `prescreen.q`, `prescreen.retry.q`, `prescreen.dlq`.
- Added routing keys `prescreen` / `retry.prescreen`.
### Messages (`core/mq/messages.py`)
- `PrescreenRequested` — worker-extract → prescreen.q.
- `PrescreenCompleted` — prescreen result, persisted and forwarded.
- `AnalyzeRequested` — worker-prescreen → analyze.q, carries `prescreen_meta`.
- `DocumentExtracted` kept for backward compatibility but no longer published.
### New service: `worker_prescreen/`
| File | Purpose |
|---|---|
| `extractor.py` | Regex extractor for RU/BY contracts; returns `PrescreenContractMeta` + confidence score. |
| `router.py` | Routing decision: `auto_approve` / `manual_review` / `deep_analysis`. `auto_approve` is disabled by default. |
| `handler.py` | Download Markdown, run extractor, persist `prescreen_results`, publish next message. |
| `consumer.py` | RabbitMQ consumer for `prescreen.q`. |
| `config.py` | `PrescreenSettings` with prefetch tunable. |
| `__main__.py` | Entrypoint, metrics server on `:9104`. |
### Wired existing services
- `worker_extract/handler.py`: now publishes `PrescreenRequested` to `prescreen.q`
and sets `documents.status = 'prescreening'`.
- `worker_analyze/consumer.py` + `handler.py`: now consumes `AnalyzeRequested`,
receives `prescreen_meta`, appends it to the LLM prompt as known contract
fields.
- `core/llm/port.py` + `ollama_cloud.py`: `analyze()` accepts `extra_context: str`
and injects it into each chunk's user prompt.
### Configuration (`core/config.py`)
New envs:
- `PRESCREEN_ENABLED` (default `true`)
- `PRESCREEN_AUTO_APPROVE` (default `false`)
- `PRESCREEN_CONFIDENCE_THRESHOLD` (default `0.75`)
- `PRESCREEN_HIGH_VALUE_THRESHOLD` (default `100000`)
### Metrics (`core/metrics.py`)
- `prescreen_duration_seconds{decision}`
- `prescreen_runs_total{decision,contract_type}`
- `prescreen_confidence` histogram
### Docker / compose
- New `srv/worker-prescreen/Dockerfile` (lean, no tesseract).
- Added `worker-prescreen` service to `docker-compose.yml` on port `9104`.
### Dependencies (`pyproject.toml`)
- Added `prescreen` dependency group (core + db/mq/s3/obs).
- Added to `dev` group.
### Tests
- Unit: `tests/unit/test_prescreen_extractor.py`, `tests/unit/test_prescreen_router.py`.
- Integration: updated `test_extract_worker.py`, `test_analyze_worker.py`,
`test_upload_pipeline.py`, `test_b2b_api.py` for the new pipeline.
### Docs
- Updated `docs/ARCHITECTURE.md` with prescreen topology, messages, schema,
metrics, and Docker/compose rows.
## Behavior
1. Upload → `extract.q` (unchanged).
2. worker-extract uploads Markdown to MinIO, publishes `PrescreenRequested`.
3. worker-prescreen:
- extracts contract type, parties, amount, currency, dates, penalty/
termination/arbitration clauses via regex
- `confidence_score = matched_fields / total_fields`
- routes:
- `deep_analysis` if high value, penalty clause, or arbitration
- `manual_review` if low confidence, missing parties, or auto_approve disabled
- `auto_approve` only when `PRESCREEN_AUTO_APPROVE=true` and low risk
4. `deep_analysis` → `AnalyzeRequested` → worker-analyze with prescreen context.
5. `manual_review` → terminal DB state, no LLM call, credit **not** refunded
(this is an intentional routing outcome, not a failure).
6. `auto_approve` → lightweight report row, `status=done`.
## Verification
```bash
make lint # passed
make typecheck # passed (99 files)
make test-unit # 147 passed, 32 deselected
make test-integration # 32 passed, 147 deselected (with live workers stopped to avoid races)
```
## Deployment notes
1. Run migration: `make migrate` (or `docker compose --profile services run --rm api alembic upgrade head`).
2. Rebuild images: `docker compose --profile services up -d --build`.
3. New worker-prescreen must be started; worker-extract and worker-analyze
images also changed.
4. No application code outside core/workers was changed; the API just returns
the new `prescreening` status string.
## Deviations from the original prescreen spec
- **Extractor:** regex instead of Needle 2. Rationale in `docs/SPIKE_PHASE0.md`.
- **No separate credits split:** the credit is still reserved once on upload;
prescreen is treated as part of the same paid job.
- **No admin/web SPA yet:** `manual_review` documents are terminal in the DB
until the admin panel lands.
## Next step
Phase 3 is storage modernization evaluation (RustFS watch). The current stack
keeps MinIO as the production default; RustFS is tracked as a future option.
Alternatively, continue with the backlog: admin/web SPA, ЮKassa payments, heavy
OCR adapters.
---
**Recommended immediate follow-up:** end-to-end smoke test in Docker with all
services running (`make services-up`) and a real PDF/DOCX upload through the
API or bot, verifying that `worker-prescreen` routes to `analyze.q` or
`manual_review` correctly.

439
docs/PHASES_2_PLUS_ROADMAP.md
View file

@ -1,439 +0,0 @@
# Implementation Roadmap — Phase 2 (Prescreen) & Remaining Work
> Based on `document-extraction-spec.md`, `needle-prescreen-integration.md`,
> and the Phase 0 spike report (`docs/SPIKE_PHASE0.md`).
> Current checkpoint: Phase 1 extraction refactor is complete (`docs/PHASE1_HANDOFF.md`).
## Status overview
| Phase | Scope | Status |
|---|---|---|
| Phase 0 | Needle/RustFS spikes | Done (`docs/SPIKE_PHASE0.md`) |
| Phase 1 | Extraction layer refactor | Done (`docs/PHASE1_HANDOFF.md`) |
| Phase 2 | Prescreen stage | **Done** (`worker_prescreen/`, migrations `0006_prescreen.py` … `0008_add_manual_review_status.py`) |
| Phase 3 | Storage modernization / RustFS watch | **Planned below** |
| Follow-up | Admin/web, payments, heavy OCR | **Backlog** |
## Phase 2 — Prescreen stage between extract and analyze
### Why
Insert a fast, local decision layer after extraction and before the expensive LLM
analysis so that:
- obvious low-risk contracts can short-circuit to a lightweight report
- high-risk / high-value contracts are routed to deep LLM analysis
- the heavy LLM worker is no longer the only path for every document
- per-stage queue metrics, retry, and DLQ are clean and independent
### Key correction from the original prescreen spec
The original spec assumed **Needle 2** (`cactus-needle`) as the prescreen
extractor. The Phase 0 spike proved the base model is English-only and fails on
Russian contracts with `confidence=0.0`. Fine-tuning would also disable the
calibrated confidence head the routing design depends on.
Therefore Phase 2 uses a **deterministic regex+pydantic extractor** for RU/BY
contracts. Contract boilerplate is highly templated, so regex gives:
- zero marginal cost
- no hallucination risk
- missing fields naturally map to `manual_review`
- `confidence_score` becomes a deterministic **field-coverage ratio**
A small-LLM fallback can be added later for fields regex misses.
### 2.1 RabbitMQ topology changes
Add to `core/mq/topology.py` using the existing direct-exchange + TTL retry/DLQ
pattern:
```text
contracts.x
├─ extract ─► extract.q (existing)
├─ prescreen ─► prescreen.q (NEW)
└─ analyze ─► analyze.q (existing)
contracts.retry.x
├─ retry.prescreen ─► prescreen.retry.q (NEW classic delay queue, DLX→contracts.x[prescreen])
prescreen.dlq (NEW quorum)
```
Constants: `QUEUE_PRESCREEN`, `RK_PRESCREEN`, update `_RETRY_QUEUE_FOR`,
`DLQ_FOR`, `declare_all()`.
### 2.2 Database migration `0006_prescreen.py`
New table `prescreen_results`:
```sql
CREATE TABLE prescreen_results (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
correlation_id UUID NOT NULL,
-- extracted metadata
contract_type VARCHAR(64),
party_a TEXT,
party_b TEXT,
total_amount DECIMAL(18, 2),
currency VARCHAR(8),
start_date DATE,
end_date DATE,
has_penalty_clause BOOLEAN,
has_termination_clause BOOLEAN,
has_arbitration BOOLEAN,
confidence_score DECIMAL(4, 3), -- coverage ratio 0.000–1.000
-- routing
routing_decision VARCHAR(32) NOT NULL, -- auto_approve | manual_review | deep_analysis
-- metrics / audit
prescreened_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processing_ms INTEGER,
extractor_version VARCHAR(32) DEFAULT 'regex-v1',
-- auto-approve lightweight output
auto_summary TEXT,
auto_findings JSONB DEFAULT '[]',
-- retry tracking
error_message TEXT,
retry_count INTEGER DEFAULT 0
);
CREATE INDEX idx_prescreen_document ON prescreen_results(document_id);
CREATE INDEX idx_prescreen_routing ON prescreen_results(routing_decision);
CREATE INDEX idx_prescreen_confidence ON prescreen_results(confidence_score);
```
Also:
```sql
ALTER TYPE document_status ADD VALUE 'prescreening' AFTER 'extracted';
ALTER TYPE jobs.queue ADD VALUE 'prescreen' AFTER 'analyze';
ALTER TABLE reports
ADD COLUMN prescreen_result_id UUID REFERENCES prescreen_results(id),
ADD COLUMN prescreen_meta JSONB DEFAULT NULL;
```
Use `TEXT+CHECK` instead of Postgres enums if the project convention prefers
additive migrations (the existing schema uses both). Decision to make during
implementation.
### 2.3 Message schema additions
Add to `core/mq/messages.py`:
```python
class PrescreenRequested(PipelineMessage):
"""worker-extract → contracts.x[prescreen] → worker-prescreen."""
text_s3_key: str
filename: str
class PrescreenCompleted(PipelineMessage):
"""worker-prescreen result; also the payload forwarded to analyze.q."""
text_s3_key: str
prescreened_at: datetime
contract_type: str | None
party_a: str | None
party_b: str | None
total_amount: float | None
currency: str | None
start_date: str | None
end_date: str | None
has_penalty_clause: bool | None
has_termination_clause: bool | None
has_arbitration: bool | None
confidence_score: float | None # 0..1 coverage ratio
routing_decision: Literal["auto_approve", "manual_review", "deep_analysis"]
auto_summary: str | None
auto_findings: list[dict] = Field(default_factory=list)
error_message: str | None
class AnalyzeRequested(PipelineMessage):
"""worker-prescreen → contracts.x[analyze] → worker-analyze."""
text_s3_key: str
filename: str
prescreen_meta: PrescreenCompleted
```
Keep `DocumentExtracted` for backward compatibility; `worker-extract` may still
publish it for observability, but the prescreen stage is the main forward path.
### 2.4 New service: `worker-prescreen`
Directory:
```text
src/contract_check/worker_prescreen/
__init__.py
__main__.py
consumer.py
handler.py
router.py
extractor.py # regex+pydantic RU/BY contract extractor
config.py
```
#### `extractor.py`
Deterministic extractor built with `re` + Pydantic model. Target fields:
```python
class PrescreenContractMeta(BaseModel):
contract_type: str | None
party_a: str | None
party_b: str | None
total_amount: float | None
currency: str | None
start_date: str | None
end_date: str | None
has_penalty_clause: bool | None
has_termination_clause: bool | None
has_arbitration: bool | None
```
Heuristics:
- `contract_type`: map first matched keyword (`договор поставки`, `договор оказания услуг`, `договор подряда`, `договор аренды`, `договор купли-продажи`, etc.)
- `party_a` / `party_b`: extract first two legal entities after patterns like
`Общество с ограниченной ответственностью «(.*?)»`, `Акционерное общество «(.*?)»`, `Индивидуальный предприниматель (.*?)`.
- `total_amount`: find `составляет ([\d\s,.]+) (рублей|руб|USD|EUR|€|\$)` and parse as float.
- `currency`: normalize to `RUB`, `USD`, `EUR`, `BYN`.
- `start_date` / `end_date`: match `с (\d{2}\.\d{2}\.\d{4})`, `от (\d{2}\.\d{2}\.\d{4})`,
`действует с (\d{2}\.\d{2}\.\d{4}) по (\d{2}\.\d{2}\.\d{4})`.
- `has_penalty_clause`: presence of `неустойка`, `штраф`, `пеня`, `0,1%`/`% за каждый день просрочки`.
- `has_termination_clause`: presence of `расторгнуть`, `расторжение`, `одностороннему порядке`.
- `has_arbitration`: presence of `арбитражный суд`, `Арбитражный суд`, `Международный коммерческий арбитражный суд`.
`confidence_score = matched_fields / total_fields`.
Test on synthetic and a few real contracts; tune false-positive/negative rate.
#### `router.py`
```python
class PrescreenRouter:
def __init__(
self,
confidence_threshold: float = 0.75,
high_value_threshold: float = 100_000,
) -> None:
...
def decide(self, meta: PrescreenContractMeta) -> str:
if confidence < threshold or missing mandatory fields:
return "manual_review"
if total_amount > high_value_threshold:
return "deep_analysis"
if has_penalty_clause or has_arbitration:
return "deep_analysis"
return "auto_approve"
```
`auto_approve` is **disabled by default** via env `PRESCREEN_AUTO_APPROVE=false`
until accuracy is proven; when disabled, `auto_approve` decisions are mapped
to `manual_review`.
#### `handler.py`
- idempotency check on `documents.status`
- set `status = 'prescreening'`, `jobs.status = 'running'`
- download extracted Markdown from MinIO
- run regex extractor in thread pool (`asyncio.to_thread`)
- route
- persist `prescreen_results`
- publish next message:
- `deep_analysis` → `contracts.x[analyze]` with `AnalyzeRequested`
- `auto_approve` → `contracts.x[report.completed]` with lightweight summary
(only when `PRESCREEN_AUTO_APPROVE=true`)
- `manual_review` → `contracts.x[review.manual]`; no further processing
- ack
#### `consumer.py`
`Consumer[PrescreenRequested]` for `prescreen.q`, wired like `AnalyzeConsumer`.
#### `config.py`
- `PRESCREEN_ENABLED` — if false, handler immediately publishes
`AnalyzeRequested` (passthrough, preserving analytics)
- `PRESCREEN_AUTO_APPROVE`
- `PRESCREEN_CONFIDENCE_THRESHOLD`
- `PRESCREEN_HIGH_VALUE_THRESHOLD`
### 2.5 Dockerfile + compose
- `srv/worker-prescreen/Dockerfile`: lean image, no tesseract/pymupdf, only core
deps + regex engine.
- Add `worker-prescreen` to `docker-compose.yml` profile `services`.
- Memory limit 128M (regex is tiny).
### 2.6 Changes to existing services
#### `worker-extract`
After publishing `DocumentExtracted` (backward compat), also publish
`PrescreenRequested`:
```python
await publisher.publish(
PrescreenRequested(
correlation_id=...,
document_id=...,
user_id=...,
text_s3_key=extracted_key,
filename=payload.filename,
),
routing_key=RK_PRESCREEN,
)
```
Gate behind `PRESCREEN_ENABLED` (default true after Phase 2).
#### `worker-analyze`
Change consumer to `AnalyzeRequested`:
```python
class AnalyzeConsumer(Consumer[AnalyzeRequested]):
queue = "analyze.q"
message_model = AnalyzeRequested
```
Enrich prompt with `prescreen_meta` (§6.3 of the original spec).
#### `api`
- Update document status endpoint to include prescreen block.
- Status lifecycle: `queued → extracting → prescreening → analyzing | manual_review | done`.
- Optional: expose `?mode=fast` vs `?mode=full` for B2B.
### 2.7 Metrics
Add to `core/metrics.py` and expose from `worker-prescreen`:
- `prescreen_requests_total{source}`
- `prescreen_duration_seconds{decision}`
- `prescreen_routing_decisions_total{decision,contract_type}`
- `prescreen_confidence_distribution`
- `prescreen_errors_total{error_type}`
- `prescreen_dlq_messages_total`
### 2.8 Tests
- Unit: `tests/unit/test_prescreen_extractor.py` — regex patterns on synthetic
contracts, coverage scoring.
- Unit: `tests/unit/test_prescreen_router.py` — routing matrix with all flag
combinations.
- Integration: `tests/integration/test_prescreen_worker.py`:
- `PrescreenRequested` → `AnalyzeRequested` for `deep_analysis`
- `PrescreenRequested` → `review.manual` for `manual_review`
- bypass mode (`PRESCREEN_ENABLED=false`) → passthrough `AnalyzeRequested`
- Update `test_extract_worker.py` to assert `PrescreenRequested` is also
published.
- Update `test_analyze_worker.py` to consume `AnalyzeRequested` and include
prescreen metadata.
### 2.9 DoD
```bash
make lint && make typecheck && make test-unit
make test-integration
```
## Phase 3 — Storage watch / RustFS evaluation
### Goal
Keep MinIO as the production default but maintain a credible migration path to
RustFS when its Lifecycle + KMS features reach GA. No application code changes
are required because S3 is already behind `core.s3.port.py`.
### Why not switch now
From the Phase 0 RustFS spike:
- Lifecycle/KMS/distributed mode are marked **"Under Testing"** upstream.
- SSE-S3 objects written by MinIO are **not readable by RustFS** today — a
migration would require re-putting objects through the app.
- RustFS is weeks old; MinIO is battle-tested for this exact compose setup.
- Real advantages exist: Apache 2.0 license (vs MinIO AGPL), no telemetry,
RF data-sovereignty friendly.
### Actions
1. Add an optional `docker-compose.rustfs.yml` overlay for local experiments.
2. Run the full integration suite against RustFS quarterly or when upstream
announces Lifecycle/KMS GA.
3. If/when adopted:
- pin exact RustFS image tag
- set `S3_SERVER_SIDE_ENCRYPTION=false` during transition
- re-encrypt objects via the app after cutover
- update `minio-init` service or replace with `rustfs-init`
## Backlog / follow-up work (not tied to Phase 2)
| Item | Rationale |
|---|---|
| Admin/web SPA | Deferred by original architecture scope. |
| ЮKassa payments | Stub `invoices` table exists; payment logic deferred. |
| Heavy OCR adapters (marker, paddleocr, easyocr, img2table) | Spec deferral. Consider separate `worker-extract-heavy` queue or cloud API. |
| Email extractor (MSG/EML) | Spec deferral; only relevant for B2B corporate email ingestion. |
| Multi-tenancy | Explicit non-goal of original refactor. Do not add `tenant_id` until orgs are a real requirement. |
| Fine-tuned RU Needle model | Revisit if `cactus-needle` ships a multilingual or RU-tuned base with calibrated confidence. |
## Appendix: original spec alignment
### `document-extraction-spec.md`
| Requirement | Phase 1 status |
|---|---|
| `DocumentExtractor` Protocol / ABC | Done (`core/extraction/port.py`) |
| `ExtractedDocument` dataclass / Pydantic | Done |
| `ExtractorFactory` with MIME detection | Done |
| `DocxExtractor` (`mammoth`) | Done |
| `RtfExtractor` (`striprtf`) | Done |
| `EncodingDetector` (`chardet`) | Done |
| `PyMuPDFExtractor` + `has_tables` | Done |
| `TesseractOCRAdapter` + `is_structured=False` | Done |
| Metrics by format | Done |
| Update architecture docs | Done |
| Heavy adapters deferred | Backlog |
### `needle-prescreen-integration.md`
| Requirement | Phase 2 plan above |
|---|---|
| Separate `prescreen.q` stage | Yes |
| `PrescreenRequested` / `PrescreenCompleted` messages | Yes, corrected to extend `PipelineMessage` |
| `prescreen_results` table | Yes, adjusted columns |
| `worker-prescreen` service | Yes, regex-based instead of Needle |
| Routing decisions | Yes, conservative default |
| `worker-extract` publishes prescreen | Yes |
| `worker-analyze` consumes `AnalyzeRequested` | Yes |
| API status endpoint enriched | Yes |
| Dockerfile/compose for prescreen worker | Yes |
| Rollback / bypass flag | Yes |
| Multi-tenant / credit split | **Rejected** — not in current scope; integer credit model stays |
## Ordering recommendation
1. Phase 2 DB migration + RabbitMQ topology.
2. Phase 2 messages + `worker_prescreen` skeleton (no-op passthrough first).
3. Phase 2 regex extractor + router + tests.
4. Wire `worker-extract` → `prescreen.q` and `worker-analyze` → `analyze.q`.
5. End-to-end integration test.
6. Phase 3 RustFS spike only after upstream GA announcement.
---
**Next action:** implement Phase 2 step 1 (migration + topology) if approved.

242
docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md
View file

@ -1,242 +0,0 @@
# Refactor Plan: Prescreen Worker → Hybrid Extraction (Heuristic + LLM Fallback)
> Status: delivered through Phase 4; Phase 5 rollout pending merge
> Scope: `src/contract_check/worker_prescreen/`, `src/contract_check/core/llm/`, `src/contract_check/core/config.py`, `src/contract_check/core/metrics.py`
> Supersedes: the regex-only extractor described in `worker_prescreen/extractor.py` (regex-v1)
---
## 1. Goal
Replace the monolithic regex extractor (`extractor.py`, ~260 lines of Russian/BY legal
regexes) with a **hybrid two-stage extractor**:
1. **Stage 1 — Heuristic (deterministic, zero-cost):** keyword dictionaries,
positional windows, sentence scanning, per-field confidence weights.
No regex — plain string operations only.
2. **Stage 2 — LLM fallback (only when Stage 1 confidence is low):** reuse the
existing `core/llm` provider infrastructure (Ollama Cloud / YandexGPT) with a
JSON-schema-constrained extraction prompt.
Non-goals:
- No changes to routing semantics (`router.py` thresholds/decisions stay as-is).
- No DB schema migration (`prescreen_results.extractor_version` already exists,
`String(32)` fits `"heuristic-v2"` / `"llm-v1"`).
- No changes to MQ topology, messages (`PrescreenCompleted` shape unchanged),
or downstream `worker_analyze`.
- Auto-approve stays disabled by default.
## 2. Current State
```
worker_prescreen/
├── consumer.py # consumes prescreen.q → handler
├── handler.py # Stage 4: asyncio.to_thread(extract_contract_meta, text)
├── extractor.py # PrescreenContractMeta + 10 regex patterns ← TO REPLACE
├── router.py # decides auto_approve|manual_review|deep_analysis
└── config.py
```
Problems:
- Regexes are brittle (line-noise, OCR artifacts, whitespace variants) and hard
to extend (every new contract type = new regex).
- No recovery path: low-confidence extraction always ends in `manual_review`.
- `prescreen_results.extractor_version` is never written by the handler INSERT —
it silently relies on the DB default `'regex-v1'`.
## 3. Target Architecture
```
┌────────────────────────────────────────────┐
│ handler.py │
│ Stage 4: meta = await extractor.extract() │
└───────────────┬────────────────────────────┘
│
HybridMetaExtractor (orchestrator)
│
┌────────────────────┴──────────────────────┐
▼ ▼
HeuristicExtractor (sync, to_thread) LLMPrescreenExtractor
keyword + positional + sentence wraps core/llm provider
scanning, confidence-weighted extract_prescreen(text)
│ │
confidence ≥ threshold? ──── no ────► run LLM, validate JSON,
│ yes merge over heuristic meta
▼
PrescreenContractMeta → router.decide() → persist (with extractor_version)
```
### 3.1 New module layout (flat, matches existing package conventions)
```
worker_prescreen/
├── extractor.py # PrescreenContractMeta (model, unchanged shape)
│ # + MetadataExtractor protocol + re-export shim
├── extractor_heuristic.py# HeuristicExtractor (Stage 1)
├── extractor_llm.py # LLMPrescreenExtractor (Stage 2, dict → pydantic validation)
├── extractor_hybrid.py # HybridMetaExtractor (threshold + merge + kill-switch)
├── handler.py # Stage 4 rewired; INSERT gains extractor_version
└── router.py # unchanged
```
### 3.2 Heuristic stage design (regex-free)
| Field | Method | Notes |
|---|---|---|
| `contract_type` | Phrase dictionary match on normalized text | `{"supply": ["договор поставки", "договор купли-продажи", ...], ...}` — lowercase + whitespace-collapse once, then `in` checks |
| `party_a/b` | Positional: scan only the header window (text up to first `1.` / `ПРЕДМЕТ` heading, capped ~1500 chars); token-scan for entity-form tokens (ООО, ИП, АО, …) then capture until quote `»`/`"` close or line end | Replaces `_ENTITY_RE` |
| `total_amount` + `currency` | Trigger-word scan (`составляет`, `стоимость`, `цена`, …) then manual digit-window parse (`_scan_number` helper walking chars, handling spaces/commas); currency via token lookup in the following ~50 chars | Replaces `_AMOUNT_RE` |
| `start/end_date` | Trigger-word scan (`действует с`, `с … по …`) then `_parse_ddmmyyyy` manual splitter (already regex-free in v1 — keep) | Replaces `_DATE_RE`/`_END_DATE_RE` |
| boolean clauses | Sentence segmentation via `str.split` on `. ` / `\n`, keyword membership per sentence (`неустойка`, `штраф`, `расторгнуть`, `арбитражн`, …) | Replaces `_PENALTY_RE` etc. |
Confidence scoring becomes **weighted** instead of flat field coverage:
```python
FIELD_WEIGHTS = {
"contract_type": 0.25,
"party_a": 0.15,
"party_b": 0.15,
"total_amount": 0.15,
"currency": 0.05,
"start_date": 0.10,
"end_date": 0.05,
"has_penalty_clause": 0.05,
"has_termination_clause": 0.05,
"has_arbitration": 0.05,
} # sums to 1.0
```
Optional per-field method bonus (exact phrase match = full weight, positional
window hit = full weight, fuzzy tail hit = ×0.7) — start simple, weights
constant, tune later with real data.
### 3.3 LLM fallback design
- **Protocol extension** (`core/llm/port.py`):
`async def extract_prescreen(self, text: str) -> dict[str, Any]` — returns raw
JSON dict; **no import of worker_prescreen** (keeps layering clean).
Implemented by both `OllamaCloudProvider` and `YandexGPTProvider` as a thin
wrapper over their existing JSON-chat + repair-loop machinery
(`_chat_json` / `responseFormat=json_schema`) with a dedicated
`PRESCREEN_SYSTEM` prompt: extract the 10 meta fields, cite nothing, JSON only.
- **Worker-side wrapper** (`extractor_llm.py`):
`dict` → `PrescreenContractMeta` via pydantic (rejects hallucinated fields,
re-`None`s unknown enum values), then recomputes weighted confidence.
- **Input cap:** only the first `prescreen_llm_max_chars` (default 20 000) chars
are sent — meta lives in the header for templated contracts; cost control.
- **Merge rule:** LLM values override heuristic `None`s and low-confidence
fields; boolean clauses become OR(heuristic, llm) — both sources are
presence-checks, false positives are cheap, false negatives route wrong.
### 3.4 Failure semantics
- LLM fallback error (quota, timeout, invalid JSON after repair):
**do not fail the message.** Log + keep heuristic meta,
`extractor_version="heuristic-v2"`, record the error in
`prescreen_results.auto_findings` (`{"llm_fallback_error": "..."}`) and the
`prescreen_fallback_runs_total{outcome="failed"}` counter.
- Heuristic stage is pure string ops — its only failure mode is `None` fields,
which is already handled by low confidence → fallback / manual_review.
### 3.5 Config additions (`core/config.py`, `--- prescreen stage ---`)
```python
prescreen_llm_fallback_enabled: bool = Field(default=False, ...) # kill-switch; flip to True after burn-in
prescreen_llm_fallback_threshold: float = Field(default=0.75, ...) # ≤ router threshold
prescreen_llm_max_chars: int = Field(default=20_000, ...)
```
### 3.6 Metrics additions (`core/metrics.py`)
```python
prescreen_fallback_runs = Counter(
"contract_check_prescreen_fallback_runs_total",
"...", labelnames=["outcome"], # used | failed | skipped | disabled
)
prescreen_extraction_stage = Histogram(
"contract_check_prescreen_extraction_stage_seconds",
"...", labelnames=["stage"], # heuristic | llm
)
```
`extractor_version` values: `"regex-v1"` (legacy shim flag), `"heuristic-v2"`
(LLM not run / disabled / failed) and `"hybrid-llm-v1"` (LLM result merged).
## 4. Implementation Phases
Each phase is independently shippable and reverted by config/env flag.
### Phase 1 — Port + heuristic extractor (no behavior change for routing)
- [x] `extractor.py`: keep `PrescreenContractMeta` + weighted `_score_confidence`;
add `MetadataExtractor` protocol (`def extract(text: str) -> PrescreenContractMeta`);
keep `extract_contract_meta` as a delegating shim so
`tests/unit/test_prescreen_extractor.py` and `worker_prescreen/__init__.py`
keep importing it.
- [x] New `extractor_heuristic.py` implementing all helpers from §3.2
(`_scan_number`, `_find_after_trigger`, header-window splitter,
sentence splitter, dictionaries as module-level constants).
- [x] Port existing unit-test fixtures (`SIMPLE_SUPPLY`, `MINIMAL`, boolean-flag
parametrize) onto `HeuristicExtractor`; they must pass with identical
expected values.
- [x] `extractor_version` shim reports `"heuristic-v2"`.
### Phase 2 — LLM `extract_prescreen` on providers
- [x] Extend `LLMProvider` protocol + both adapters
(`ollama_cloud.py`, `yandex_gpt.py`): new `PRESCREEN_SYSTEM` prompt,
JSON schema for the 10 fields, reuse repair loop, truncate input to
`chunk_size`-independent small cap.
- [x] Unit tests with a fake transport (both adapters already have this pattern):
valid dict, invalid enum → `None`, malformed JSON → repair once → fail.
- [x] `extractor_llm.py`: dict → `PrescreenContractMeta` validation +
weighted confidence.
### Phase 3 — Orchestrator + handler wiring
- [x] `extractor_hybrid.py`: threshold check, LLM call (async), merge rules,
error swallowing, extractor_version selection, kill-switch.
- [x] `handler.py` Stage 4: replace `asyncio.to_thread(extract_contract_meta, …)`
with `await self._extractor.extract(contract_text)` where
`self._extractor` is injectable (constructor arg, defaults to hybrid) —
mirrors the existing `provider` injection pattern in `AnalyzeHandler`.
- [x] Handler Stage 6 INSERT: bind `extractor_version`.
- [x] Config + metrics from §3.5/§3.6.
### Phase 4 — Tests + verification
- [x] Unit: orchestrator matrix — high confidence skips LLM; low confidence
merges; LLM failure → heuristic result + `outcome="failed"`; disabled →
`outcome="disabled"`.
- [x] Integration (`tests/integration/test_prescreen_worker.py` pattern):
run handler in-process with a stub provider; assert
`prescreen_results.extractor_version` persisted, routing unchanged.
- [x] Backfill spot-check: replayed 8 RU/BY-style contract samples through
regex-v1 vs heuristic-v2. Confidence deltas: all neutral or better,
heuristic fixes regex under-parsing on NBSP/noise and captures fuller
party names. Record below in PR description.
- [x] `ruff check . && mypy src && pytest tests/unit tests/integration -k prescreen`.
(ruff/mypy findings outside the touched files are from pre-existing
uncommitted changes; prescreen tests pass.)
### Phase 5 — Rollout
- [ ] Merge with `prescreen_llm_fallback_enabled=false` (pure heuristic).
- [ ] Observe `prescreen_fallback_runs` / `prescreen_confidence` for a few days.
- [x] Document in `docs/ARCHITECTURE.md` prescreen section + `.env.example`
completed. The regex notes in `SPIKE_PHASE0.md` are no longer referenced
from the extractor docstring. Staging/prod enablement remains an
operational step after merge.
## 5. Risks & Mitigations
| Risk | Mitigation |
|---|---|
| Heuristic drops accuracy vs regex on edge templates | Port 100% of v1 unit tests before deleting regex; keep `extractor.py` regex code one release behind a flag (`PRESCREEN_KEEP_REGEX=true` env, temporary) |
| LLM hallucinates fields | pydantic validation whitelists contract_type enums; numeric/date parse checks; booleans only OR-merged |
| LLM latency blows up prescreen SLA | 20k char cap, existing provider timeouts, `prescreen_extraction_stage` histogram; fallback failures never block the pipeline |
| Cost creep on fallback rate | `prescreen_fallback_runs_total` alert; threshold tunable without deploy via env |
| Layering violation (core ← worker import) | Protocol returns plain `dict`; pydantic model stays in `worker_prescreen` |
## 6. Rollback
1. Config: `PRESCREEN_LLM_FALLBACK_ENABLED=false` → deterministic heuristic only.
2. Full: revert merge — no DB migration to undo; `extractor_version` strings are
informational only.

103
docs/SPIKE_PHASE0.md
View file

@ -1,103 +0,0 @@
# Phase 0 Spike Report — Needle 2 prescreen & RustFS storage
> Date: 2026-08-16. Evidence from live testing in isolated environments
> (`/tmp/opencode/needle-spike`, ephemeral `rustfs/rustfs:latest` container).
> Gates the Phase 1 (extraction refactor) and Phase 2 (prescreen stage) plans.
## Spike 1: cactus-needle (Needle 2) for RU contract prescreen — **NOT VIABLE**
### What was verified
- `cactus-needle==2.0.5` is real (Cactus Compute, Apache-2.0, first release
2026-08-10). Installs cleanly on Python 3.13. The spec's claims hold:
14MB engine (fetched once from HF, cached offline), `needle.extract(text,
PydanticModel)`, calibrated `confidence` on responses, `.cact` LoRA
fine-tuning pipeline. Note: PyPI package name is **`cactus-needle`**
(`import needle`); bare `needle` on PyPI is an unrelated dead 2017 CSS
testing tool.
- Engine session RAM measured at 37.8MB (simple schema) to 119.7MB (larger
schema) — fine for a 256M container. Python deps pull the JAX/Flax/optax
stack (~200MB+ image impact) because fine-tuning support ships in the core
dependency list.
### Results (base weights, `buffer_size` default)
| Case | Result |
|---|---|
| EN invoice (README example) | Perfect extraction, `confidence=0.9484` |
| EN off-topic sentence | Correctly refused: empty call `[]`, `confidence=0.7427` |
| RU contract, 2.5K chars | **Total hallucination**: parties → "Supplier"/"Supplier", amount → 12.0 USD (real: 1 234 567,80 RUB), dates → 2024 defaults. `confidence=0.0` |
| RU contract, 43K chars | Same failure mode, different hallucinations. `confidence=0.0` |
| RU garbage text | **Not refused** — fabricated "Party A/Party B/150 USD". `confidence=0.0` |
### Verdict
The base model is **English-centric; it does not understand Russian**. The
calibrated confidence head behaves correctly (0.0 = "I can't read this"), so a
naive deployment would have been *safe* (100% → manual_review) but useless.
Fine-tuning cannot rescue this for v1:
1. Calibrated confidence is **disabled on tuned weights** (reports `None`) —
and the spec's entire routing design hinges on confidence gating.
2. The model failed even trivial verbatim copy tasks (could not copy
«ООО «Ромашка»» from adjacent text) — this is tokenizer/training coverage,
not prompt tuning.
3. Building a RU legal extraction dataset + LoRA pipeline is its own project.
### Plan adjustment (Phase 2)
- `worker-prescreen` ships with a **deterministic regex+pydantic extractor**
instead of Needle: RU/BY contract boilerplate is highly templated
(«договор поставки», «именуемое в дальнейшем», «сумма договора составляет
N рублей», «неустойка», «арбитражный суд», DD.MM.YYYY ranges). Zero marginal
cost, no hallucination, p95 ≈ ms. Absence of a match = `None` →
`manual_review`, which matches the routing philosophy.
- `confidence_score` is redefined as a deterministic **field-coverage score**
(fraction of target fields extracted), not model calibration. Router
thresholds operate on coverage.
- `needle_client.py` / `model_weights` / `NEEDLE_*` env are dropped from the
plan; keep `prescreen_results.extractor_version` for future engine swaps.
- Optional fast-follow (not v1): small-LLM prescreen via the existing
`core/llm` port for fields regex missed.
- Revisit Needle only if/when a multilingual or RU-tuned base ships with
calibration intact.
## Spike 2: RustFS as MinIO alternative — **VIABLE, but stay on MinIO for now**
Tested with the project's own `MinioStorage` adapter (minio SDK) against
`rustfs/rustfs:latest`, single node, named volume.
| Check | Result |
|---|---|
| Health (`/minio/health/live`) | 200 — MinIO-compatible probe works |
| bucket create/exists, put/get/stat/delete | OK (incl. Cyrillic UTF-8 payloads) |
| `set/get/delete_bucket_lifecycle` (ILM config API) | OK — rule round-trips |
| **ILM expiration behavior** (`RUSTFS_ILM_DEBUG_DAY_SECS=2`, rule days=1) | **Object under `users/` purged in ~10s; control object outside prefix untouched** |
| Perf (4KB put+get ×20) | ~2.0ms/op |
| Docker ergonomics | Runs as UID 10001 — bind mounts must be chowned; named volumes are the easy path |
### Verdict
The critical blocker (152-ФЗ TTL purge via ILM) **works in practice** — the
"Under Testing" label upstream refers to CI gating, not absence. Remaining
reasons to keep MinIO as the default for production:
1. Project is weeks old (1.0.0-rc era); lifecycle, distributed mode, and KMS
are all still marked "Under Testing" upstream.
2. **SSE migration trap**: objects written by MinIO with SSE-S3/KMS/C are not
readable by RustFS — a later migration requires re-putting objects through
the app (we run SSE-S3=off in dev, on in prod per `S3_SERVER_SIDE_ENCRYPTION`).
3. No operational history; MinIO is battle-tested for this exact compose setup.
Because everything S3-shaped is already behind `core/s3/port.py`, switching
later is a compose + env change, not a code change. Action item: revisit when
RustFS marks Lifecycle + KMS GA (tracked in ARCHITECTURE notes, not code).
## Phase gate outcomes
- **Phase 1 (extraction refactor): GO** — unchanged.
- **Phase 2 (prescreen stage): GO with amendments** — regex extractor instead
of Needle; coverage-based confidence; no needle deps/env; conservative
routing (`PRESCREEN_AUTO_APPROVE=false` default) as already planned.
- **Storage: no change** — MinIO stays; RustFS is a credible fast-follow.

30
docs/agents/issue-tracker.md Normal file
View file

@ -0,0 +1,30 @@
# Issue tracker: Local Markdown
Issues and specs for this repo live as markdown files in `.scratch/`.
## Conventions
- One feature per directory: `.scratch/<feature-slug>/`
- The spec is `.scratch/<feature-slug>/spec.md`
- Implementation issues are one file per ticket at `.scratch/<feature-slug>/issues/<NN>-<slug>.md`, numbered from `01`, never a single combined tickets file
- Triage state is recorded as a `Status:` line near the top of each issue file (see `triage-labels.md` for the role strings)
- Comments and conversation history append to the bottom of the file under a `## Comments` heading
## When a skill says "publish to the issue tracker"
Create a new file under `.scratch/<feature-slug>/` (creating the directory if needed).
## When a skill says "fetch the relevant ticket"
Read the file at the referenced path. The user will normally pass the path or the issue number directly.
## Wayfinding operations
Used by `/wayfinder`. The **map** is a file with one **child** file per ticket.
- **Map**: `.scratch/<effort>/map.md` (the Notes / Decisions-so-far / Fog body).
- **Child ticket**: `.scratch/<effort>/issues/NN-<slug>.md`, numbered from `01`, with the question in the body. A `Type:` line records the ticket type (`research`/`prototype`/`grilling`/`task`); a `Status:` line records `claimed`/`resolved`.
- **Blocking**: a `Blocked by: NN, NN` line near the top. A ticket is unblocked when every file it lists is `resolved`.
- **Frontier**: scan `.scratch/<effort>/issues/` for files that are open, unblocked, and unclaimed; first by number wins.
- **Claim**: set `Status: claimed` and save before any work.
- **Resolve**: append the answer under an `## Answer` heading, set `Status: resolved`, then append a context pointer (gist + link) to the map's Decisions-so-far in `map.md`.

13
docs/agents/triage-labels.md Normal file
View file

@ -0,0 +1,13 @@
# Triage Labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker.
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.