From 2d5c2b50c0036632ed0b0e7d62deab38e39f2c9d Mon Sep 17 00:00:00 2001 From: febux Date: Sun, 6 Sep 2026 15:03:51 +0300 Subject: [PATCH] Sync docs and clean up unused. --- README.md | 4 - docs/IMPLEMENTATION_PLAN.md | 339 ------------------- docs/PHASE1_HANDOFF.md | 116 ------- docs/PHASE2_HANDOFF.md | 153 --------- docs/PHASES_2_PLUS_ROADMAP.md | 439 ------------------------- docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md | 242 -------------- docs/SPIKE_PHASE0.md | 103 ------ docs/agents/issue-tracker.md | 30 ++ docs/agents/triage-labels.md | 13 + 9 files changed, 43 insertions(+), 1396 deletions(-) delete mode 100644 docs/IMPLEMENTATION_PLAN.md delete mode 100644 docs/PHASE1_HANDOFF.md delete mode 100644 docs/PHASE2_HANDOFF.md delete mode 100644 docs/PHASES_2_PLUS_ROADMAP.md delete mode 100644 docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md delete mode 100644 docs/SPIKE_PHASE0.md create mode 100644 docs/agents/issue-tracker.md create mode 100644 docs/agents/triage-labels.md diff --git a/README.md b/README.md index afa6cee..b68bf8d 100644 --- a/README.md +++ b/README.md @@ -13,10 +13,7 @@ Pipeline: `PDF/DOCX/RTF/TXT/CSV/изображения (pymupdf / mammoth / tess - [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — **единственный источник истины** для production-рефакторинга. Hexagonal-архитектура, RabbitMQ-конвейер, схемы БД, очередь/ретраи/DLQ, конфиг, deploy, observability. - [`docs/DEPLOY.md`](docs/DEPLOY.md) — руководство по развёртыванию: локально, Docker Compose, VPS, seed-token, backup, troubleshooting. - [`docs/BUSINESS_IDEA.md`](docs/BUSINESS_IDEA.md) — идея и бизнес-модель. -- [`docs/IMPLEMENTATION_PLAN.md`](docs/IMPLEMENTATION_PLAN.md) — исходный «ленивый» план. Этап 0 (прототип) актуален; этапы 1+ superseded в `docs/ARCHITECTURE.md`. - [`docs/TICKETS.md`](docs/TICKETS.md) — тикеты реализации со статусами. -- [`docs/PHASE2_HANDOFF.md`](docs/PHASE2_HANDOFF.md), [`docs/PHASES_2_PLUS_ROADMAP.md`](docs/PHASES_2_PLUS_ROADMAP.md), [`docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md`](docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md) — этап 2: prescreen-стейдж и дальнейший роадмап. -- [`docs/SPIKE_PHASE0.md`](docs/SPIKE_PHASE0.md) — спайки: prescreen-библиотеки, RustFS (артефакты в `rustfs-spike/`). ## Архитектура (одна строка) @@ -110,7 +107,6 @@ Makefile # повседневные команды (make h docker-compose.yml # default = инфра; --profile services = стек; --profile edge = nginx+certbot pyproject.toml # hatchling + PEP 735 dependency-groups (db/mq/s3/obs/api/extract/prescreen/analyze/notify/billing/bot/dev) .env.example # полный список env (см. docs/ARCHITECTURE.md §11) -rustfs-spike/ # артефакты спайка RustFS (docs/SPIKE_PHASE0.md) ``` ## Быстрый старт diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md deleted file mode 100644 index 30b4fd6..0000000 --- a/docs/IMPLEMENTATION_PLAN.md +++ /dev/null @@ -1,339 +0,0 @@ -# План реализации «Контракт-чек» (Ollama Cloud / hosted LLM) - -> **Примечание:** этот документ описывает исходную «ленивую» архитектуру с arq+Redis -> (очередь), Selectel S3 (хранилище) и единым Docker-образом с `MODE=api|worker|bot`. -> **Этап 0 (прототип) актуален.** Этапы 1+ superseded [`ARCHITECTURE.md`](ARCHITECTURE.md), -> где реализована production-архитектура: RabbitMQ-конвейер, MinIO, 4 worker-а -> (extract, prescreen, analyze, notify), aiogram-бот, B2B API, 7 Dockerfile-ов в `srv/`, -> dependency-groups (PEP 735). - -> Принцип: ленивая архитектура. Каждый этап — минимальный работающий -> срез. Никаких абстракций «на потом». K8s, RabbitMQ, микросервисы — -> не нужны до $10k MRR. -> -> **LLM:** [Ollama Cloud](https://ollama.com/cloud) — hosted-инференс -> открытых моделей. Те же модели и API (`/api/chat`, `format: json`), -> что и у локального Ollama, но без своего железа: платим подписку + -> metered overage, инференс бежит у Ollama (NCP-партнёры, в основном США). - ---- - -## О выборе Ollama Cloud - -- **API:** стандартный Ollama HTTP API на cloud-эндпоинте. Авторизация — - API-ключ (bearer) из настроек аккаунта Ollama. Структурный вывод через - `format: json` (или JSON-schema) для парсимого отчёта. -- **Модели:** только cloud-enabled (см. `ollama.com/search?c=cloud`). - Рекомендация — **`qwen2.5:14b`** (сильный русский + reasoning). - Fallback — более лёгкая **`qwen2.5:7b`** при 429/квоте. - Теги БЕЗ суффикса `-instruct` (его нет в Ollama): `qwen2.5:14b`, - `llama3.1:8b`, `gemma2:9b` и т.п. -- **`format: json` ≠ соответствие схеме.** Ollama гарантирует синтаксис - JSON; соответствие вашей pydantic-схеме проверяем на клиенте + цикл - repair/retry. Не полагаемся на модель. - -## Стоимость и лимиты Ollama Cloud (актуально на момент планирования) - -| План | Цена | Конкарренси | Назначение | Статус | -|-------|-------------|-------------|-------------------------|------------------| -| Free | $0 | 1 | тесты | открыт | -| Pro | $20/мес | 3 | рабочая лошадка MVP | открыт | -| Max | $100/мес | 10 | тяжёлый поток | **приостановлен** | -| Team | $25/seat | — | команда (5 seat min) | waitlist | -| Enterprise | custom | custom | продакшн SaaS | по запросу | - -- **Usage:** rolling-лимиты — сессия 5 ч + недельный 7 дн. Pro = 50× Free. - При превышении — **metered overage** (добиваем баланс, оплата по - токенам модели). Жёсткой стены нет, но конкарренси = 3 на Pro. -- **Хостинг: в основном США** (подключаются EU/Singapore). Это **ломает - 152-ФЗ data-residency**, см. раздел «Риски». - -## Экономика проекта (честно) - -- Опекс фиксированный до квоты: VPS ~4 €/мес + Ollama Cloud Pro $20/мес + - Selectel S3 (копейки). Итого ~$25–30/мес на старте. -- Рост объёма → metered overage Ollama Cloud по токенам. Маржа на - pay-per-doc (199 ₽) остаётся, **если** один договор съедает мало - квоты. Это **обязательно измерить на этапе 0** (квота/документ). -- В отличие от локального Ollama — нет потолка по железу, но есть потолок - по конкарренси (3) и плавающий cost при overage. В отличие от GigaChat — - cost в $, не в ₽, и нет РФ-локализации данных. - ---- - -## Архитектура — hexagonal (ports & adapters) - -Ядро и адаптеры разделены. Это снимает связность «бот знает про БД/S3/LLM» и -позволяет добавлять каналы доставки (CLI, веб, B2B API) без дублирования -бизнес-логики. - -- **Ядро (application core):** - - `api` (FastAPI/uvicorn) — владеет БД, S3, кредитами, оплатой; принимает - документы, **резервирует кредит на enqueue**, ставит arq-задачи в redis, - отдаёт отчёты/профиль. Аутентификация адаптеров — общий `SERVICE_TOKEN`. - - `worker` (arq) — разбирает очередь: S3 → экстракт → (OCR) → chunker → - analyzer → Ollama Cloud → Report. Idempotency + refund при ошибке. - - Общий доменный пакет `contract_check/` (`extractor`, `chunker`, - `llm_client`, `analyzer`, `checklist`, `report_schema`, `ocr`, `storage`, - `db`, `models`, `payments`, `quota`) — используется только ядром. -- **Адаптеры (delivery = HTTP-клиенты к `api`):** - - `bot` (aiogram) — Telegram: приём документа → `POST /documents` (multipart) - → отдача отчёта. **Не трогает БД/S3/redis/LLM.** - - `cli` — пользовательский/отладочный CLI поверх api (новый, отдельный от - прототипа). - - `web` (React SPA) — этап 2, тоже HTTP-клиент к api. -- **Прототип `prototype.py` (stage 0)** — standalone, in-process LLM; артефакт - go/no-go, **не адаптер и не ядро**. - -> Следствие для планирования: **api + worker строятся на этапе 1** (боту-адаптеру -> нужен api). Этап 2 = добавление адаптера `web` + подписок (api уже есть). - ---- - -## Этап 0 — Прототип (1 выходной) - -**Цель:** доказать, что модель через Ollama Cloud реально находит риски, -и измерить задержку + **квоту на один договор** (для юнит-экономики). - -**Деливерэбл:** один файл `prototype.py` — end-to-end. - -``` -PDF/DOCX → pymupdf/python-docx → текст → -prompt с чек-листом → Ollama Cloud (format: json) → отчёт в markdown -``` - -**Что делаем:** -1. Завести аккаунт Ollama, взять API-ключ, положить в `.env`. -2. Поставить `pymupdf`, `httpx` в venv. -3. Один скрипт: читает PDF → достаёт текст → промпт с чек-листом из 10 - пунктов → дёргает Ollama Cloud → печатает отчёт. -4. Прогнать 3–5 реальных договоров (NDA, оказание услуг, поставка). -5. Зафиксировать: качество находок, задержка (сек), % битых JSON, - **сколько квоты/токенов на договор**. - -**Чего НЕ делаем:** база, веб, бот, оплату, деплой. Один скрипт локально. - -**Критерий успеха:** отчёт по реальному договору содержит хотя бы 3 -осмысленные находки с цитатами; задержка < 60 сек; квота/договор даёт -понятную маржинальность при 199 ₽. - ---- - -## Этап 1 — Telegram-бот MVP (2–3 недели) - -**Цель:** первые платящие пользователи. Бот = самый быстрый путь до ЦА. - -### 1.1 Структура проекта - -``` -contract_check/ -├── pyproject.toml # hatchling, deps (core + adapters) -├── docker-compose.yml # postgres, redis, api, workers, bot (profiles: services, bot, edge) -├── .env.example -├── src/contract_check/ -│ ├── main.py # диспетчер: MODE=api|worker|bot|cli -│ ├── prototype.py # stage-0 standalone (in-process LLM) — удалён, больше не нужен -│ │ # ── ЯДРО (application core): владеет БД/S3/кредитами/LLM ── -│ ├── api.py # FastAPI: /documents, /reports/{id}, /me, /healthz -│ ├── worker.py # arq analyze_document(doc_id): S3→extract→OCR→analyze→Report -│ ├── analyzer.py extractor.py chunker.py ocr.py -│ ├── llm_client.py checklist.py report_schema.py -│ ├── storage.py db.py models.py payments.py quota.py -│ │ # ── АДАПТЕРЫ (delivery): HTTP-клиенты к api, без БД/S3/LLM ── -│ ├── bot.py # aiogram: приём документа → POST /documents → отчёт -│ └── cli.py # CLI поверх api (новый, не прототип) -├── migrations/ # alembic -└── tests/ -``` - -### 1.2 База данных — 3 таблицы - -```sql -users (id, telegram_id, created_at, credits_left) -documents (id, user_id, s3_key, status, created_at) -reports (id, document_id, content_json, created_at) -``` - -`credits_left` — prepaid-кредиты (pay-per-doc). Подписок в v1 нет. -`status` хранить как `TEXT` + `CHECK`, не Postgres-ENUM (миграции проще). - -### 1.3 Поток (резервирование кредита на enqueue) - -``` -Юзер кидает PDF в бот - → бот (adapter): multipart'ом шлёт файл в api: POST /documents (+ SERVICE_TOKEN) - → api (core): get_or_create_user, проверяет credits_left > 0, РЕЗЕРВИРУЕТ (-= 1), - грузит файл в S3, создаёт Document(status=queued), ставит arq-таску → 202 + job_id - → worker (core): idempotency guard по status, достаёт из S3 → текст → (OCR если скан) - → worker: chunker при необходимости → analyzer → Ollama Cloud → отчёт - → worker: pydantic-валидация + repair-loop; пишет Report, status=done - → при ошибке: status=failed, ВОЗВРАТ кредита (+1) - → бот: опрашивает GET /reports/{id} (или push), отправляет отчёт - (с разбивкой/файлом при >4096 симв.), футер-disclaimer -``` - -### 1.4 Оплата (минимум) - -- ЮKassa: бот генерирует ссылку на оплату N ₽ → webhook пополняет - `credits_left`. -- Один тариф: 199 ₽ = 1 документ. Без подписок. - -### 1.5 Деплой - -- **Один VPS** (Hetzner CX22 ~4 €/мес или Selectel под РФ-локацию БД). -- Docker Compose: `postgres`, `redis`, `api`, `worker`, `bot`. **Без GPU.** - (Бот — адаптер к api; веб-адаптер `web` добавится на этапе 2.) -- S3 — Selectel Object Storage. -- Домен + HTTPS только для webhook ЮKassa; на старте ngrok (с оговоркой: - URL на free-ngrok меняется → лучше сразу дешёвый домен + Caddy). - -### 1.6 Критерий успеха - -10 платящих. MRR ~2 000 ₽. Отчёты не вызывают жалоб «ничего не нашёл». -Ollama Cloud Pro покрывает нагрузку без ухода в overage. - ---- - -## Этап 2 — Веб + подписки (3–4 недели) - -**Цель:** B2B-веб-интерфейс и подписки. - -### 2.1 Что добавляем - -- **`api` уже построен на этапе 1** (T-E1-015). На этапе 2 добавляем: роуты - подписок/счетов, Telegram Login auth (сессия/JWT) для веб-адаптера, и сам - веб-адаптер `web` (React SPA) — ещё один HTTP-клиент к api. -- React SPA (Vite): загрузка, история, профиль, подписка. Отдаётся статикой - позже (отдельный `web`-контейнер / Nginx). -- Auth: Telegram Login Widget (нужен публичный HTTPS-домен в BotFather) → - сессия/JWT. -- Подписки: solo (1 490 ₽), team (3 990 ₽). ЮKassa recurring. - -### 2.2 Что меняем в базе - -```sql -ALTER TABLE users ADD COLUMN plan TEXT DEFAULT 'free'; -ALTER TABLE users ADD COLUMN plan_renews_at TIMESTAMP; -CREATE TABLE invoices (id, user_id, amount, status, provider, external_id, created_at); -``` - -`credits_left` остаётся для pay-per-doc; подписка = безлимит с monthly -reset через cron-arq-таску. - -### 2.3 Архитектура — без изменений в ядре - -`api` и `worker` работают с этапа 1; на этапе 2 добавляется только адаптер -`web` (React, статика) перед Nginx → HTTPS (Let's Encrypt). Hexagonal-граница -сохранена: `web` — такой же HTTP-клиент к api, как `bot`. LLM по-прежнему -Ollama Cloud; при росте — Enterprise тариф или переход на свой GPU. - -### 2.4 Критерий успеха - -50 платящих. MRR ~30 000 ₽. Есть хотя бы один team-клиент. - ---- - -## Этап 3 — B2B API (2 недели, только если есть спрос) - -**Цель:** сторонние сервисы дёргают анализ через API. - -### 3.1 Что добавляем - -- API-ключи: таблица `api_keys`, header `X-API-Key`. -- Rate-limit: Redis (token bucket). Лимит = **конкарренси/квота Ollama - Cloud** (3 на Pro), а не ₽. -- `POST /api/v1/analyze` (multipart) → 202 + job_id → - `GET /api/v1/reports/{id}`. -- Дашборд: ключи, usage, счета. - -### 3.2 Чего НЕ делаем - -- Нет SDK, нет вебхуков (клиент поллит), нет OAuth2. -- Один тариф API: 9 900 ₽/мес за 100 запросов. - -### 3.3 Критерий успеха - -3 API-клиента. MRR +30 000 ₽. - ---- - -## Инфраструктура — сводка - -| Компонент | Этап 0 | Этап 1 | Этап 2–3 | -|----------------|------------|------------------------------|--------------------------------| -| Compute | ноутбук | 1 VPS (Docker Compose) | 1 VPS (апгрейд RAM) | -| LLM | Ollama Cloud (Free/Pro) | Ollama Cloud Pro | Ollama Cloud Pro/Enterprise | -| DB | — | Postgres (в compose) | Postgres (+ backup cron) | -| Queue | — | Redis (в compose) | Redis (тот же) | -| Object storage | — | Selectel S3 | Selectel S3 | -| OCR | — | Tesseract (локально) | Tesseract (+ Yandex Vision) | -| Payments | — | ЮKassa | ЮKassa + CloudPayments | -| Monitoring | — | Docker logs | Uptime Kuma + Sentry (free) | -| CI/CD | — | git push → ssh deploy | GitHub Actions → build → deploy| - -**K8s / RabbitMQ / Kafka / Elasticsearch / vLLM / TGI — НЕ НУЖНЫ.** Один -VPS + Ollama Cloud держит всё до заметного объёма. Свой GPU — только если -Ollama Cloud overage станет дороже self-host (отдельное решение позже). - ---- - -## Чек-лист пунктов анализа (v1) - -Содержимое `checklist.py` — один список, без БД: - -1. Неустойки / штрафы (размер, односторонний) -2. Подсудность (чужой регион) -3. Сроки оплаты (условия, просрочка) -4. IP-права (кому отходят результаты) -5. Одностороннее изменение условий -6. Гарантии и их срок -7. Форс-мажор (формулировки) -8. НДС (включён / сверх) -9. Ответственность сторон (cap, исключения) -10. Расторжение (условия, уведомление) - ---- - -## Сроки (реалистично, соло, вечера/выходные) - -| Этап | Что | Время | -|------|------------------|-----------| -| 0 | Прототип | 1 выходной| -| 1 | Telegram-бот MVP | 2–3 недели| -| 2 | Веб + подписки | 3–4 недели| -| 3 | B2B API | 2 недели | - -**До первого платящего (0+1): ~3–4 недели.** Заложить ~1–2 дня на подбор -cloud-модели и тюнинг промпта (локальные/open модели капризнее GPT-4o). - ---- - -## Риски Ollama Cloud (честно) - -1. **152-ФЗ / data-residency.** Контракт улетает в Ollama Cloud (США). - Это тот же класс риска, что и GPT-4o. Митигация: disclaimer, - обезличивание ПДн перед отправкой, либо при необходимости — - отказ от Ollama Cloud в пользу РФ-LLM (GigaChat/YandexGPT) или своего - GPU-бокса. **Не позиционировать продукт как «данные не покидают РФ».** -2. **Квота/конкарренси.** Pro = 3 одновременных модели + rolling usage. - Бурст платящих юзеров упрётся в очередь/429. Трекать usage, - алертить у лимита, при росте — overage-баланс или Enterprise. -3. **Качество open-моделей.** 14B галлюцинирует/пересказывает цитаты - сильнее GPT-4o/GigaChat. Всегда: цитата + номер пункта + ремонт-цикл - валидации JSON. Disclaimer «не заменяет юриста» — в каждый отчёт. -4. **Зависимость от одного провайдера.** Один аккаунт Ollama (нельзя - несколько). Иметь готовый план Б: GigaChat/YandexGPT-фолбэк или свой - GPU при блокировке/превышении квоты. - ---- - -## Что сознательно отложено (YAGNI) - -- Multi-tenant / организации / роли — пока все юзеры = solo. -- Шаблоны договоров (генерация) — другой продукт. -- ЭЦП / Госуслуги — чужой регуляторный ад. -- Команда юристов (human-in-the-loop) — только если попросят. -- Mobile app — веб + бот закрывают 100%. -- White-label — один продукт, один бренд. -- Свой GPU / vLLM / TGI / RAG-над-векторной-базой — пока Ollama Cloud - дешевле; пересмотрим при выходе overage в минус. diff --git a/docs/PHASE1_HANDOFF.md b/docs/PHASE1_HANDOFF.md deleted file mode 100644 index 02fe678..0000000 --- a/docs/PHASE1_HANDOFF.md +++ /dev/null @@ -1,116 +0,0 @@ -# Phase 1 Handoff — Extraction Layer Refactor - -> Scope: implement the structured-extraction port from `document-extraction-spec.md`, -> preserving the hexagonal architecture and keeping the existing pipeline intact. -> Completed: 2026-08-16. - -## Goal - -Replace flat-text extraction (`core.analysis.extractor.extract_text`) with a -format-aware extraction **port** that returns Markdown with preserved structure -(headings from DOCX, tables from PDF). The rest of the pipeline (chunker, -worker-analyze, RabbitMQ contracts) remains unchanged except for Markdown-aware -chunking and additive metadata in `DocumentExtracted`. - -## New / modified files - -### New: `core/extraction/` hexagonal package - -| File | Purpose | -|---|---| -| `src/contract_check/core/extraction/port.py` | `DocumentExtractor` Protocol, `ExtractedDocument` DTO, `UnsupportedFormatError`, `ExtractionFailedError`, `MIN_TEXT_CHARS` | -| `src/contract_check/core/extraction/factory.py` | `ExtractorFactory`, `detect_format()`, `SUPPORTED_SUFFIXES`; routes bytes/mime/suffix → adapter. Uses `python-magic` for content sniffing, suffix as fallback. | -| `src/contract_check/core/extraction/adapters/pdf_pymupdf.py` | `PyMuPDFExtractor` — text + `page.find_tables()` → Markdown pipes. Interleaves tables and text blocks by bbox so cell text is not duplicated. `is_structured=True` iff tables found. | -| `src/contract_check/core/extraction/adapters/docx_mammoth.py` | `MammothDocxExtractor` — `mammoth.convert_to_markdown()`. Headings/lists/tables become Markdown; `is_structured=True` on headings/tables/lists. | -| `src/contract_check/core/extraction/adapters/rtf_striprtf.py` | `RtfExtractor` — `striprtf` to plaintext. `is_structured=False`. | -| `src/contract_check/core/extraction/adapters/txt_chardet.py` | `TxtExtractor` — `chardet` encoding detection, decode, metadata includes `encoding`. | -| `src/contract_check/core/extraction/adapters/ocr_tesseract.py` | `TesseractOcrExtractor` — OCR for scanned PDFs and raster images (PNG/JPG/TIFF), reusing the same pytesseract/pymupdf logic as `core.analysis.ocr`. | -| `src/contract_check/core/extraction/__init__.py` | `extract_document()` orchestrator: adapter → extract → OCR fallback for short PDFs. | -| `src/contract_check/core/extraction/adapters/__init__.py` | Re-exports of all adapter classes. | - -### Modified worker / pipeline - -| File | Change | -|---|---| -| `src/contract_check/worker_extract/handler.py` | Bytes-based extraction via `core.extraction.extract_document()`. No temp files. Uploaded text now `text/markdown; charset=utf-8`. `DocumentExtracted` includes `is_structured`/`has_tables`. Duration histogram labeled by `format`. Classification updated for new errors. | -| `src/contract_check/worker_extract/consumer.py` | Removed `@extract_duration.time()` decorator — labeled timing moved into the handler where the format is known. | -| `src/contract_check/core/mq/messages.py` | `DocumentExtracted` added `is_structured: bool = False` and `has_tables: bool = False` (backward-compatible defaults). | -| `src/contract_check/core/metrics.py` | `extract_duration` now has `format` label. Added `extraction_total{format,structured}` counter. | -| `src/contract_check/core/analysis/chunker.py` | Added `chunk_markdown()` — splits on ATX headings, keeps headings attached to their body, falls back to `chunk_text()` for unstructured text. | -| `src/contract_check/core/llm/ollama_cloud.py` | Analyzer now uses `chunk_markdown(text)` instead of `chunk_text(text)` so heading boundaries are preserved when the extractor produced structured Markdown. | -| `src/contract_check/api/services.py` | Upload gate now imports `SUPPORTED_SUFFIXES` from `core.extraction`. `_content_type_from_suffix()` extended for RTF/TXT/CSV/images. | -| `src/contract_check/bot/handlers.py` | Bot extension gate now uses the same `SUPPORTED_SUFFIXES` set and content-type map. | -| `src/contract_check/worker_extract/extract_document.py` | **Deleted**. Superseded by `core.extraction.extract_document()`. | - -### Dependencies / Docker - -| File | Change | -|---|---| -| `pyproject.toml` | `extract` group adds `mammoth`, `striprtf`, `chardet`, `python-magic`. Mypy ignore list extended for these libs. | -| `srv/worker-extract/Dockerfile` | Adds `libmagic1` apt package (required by `python-magic`). | -| `uv.lock` | Regenerated. | - -### Documentation - -| File | Change | -|---|---| -| `docs/ARCHITECTURE.md` | Layout updated (`core/extraction/` added, `worker_extract/extract_document.py` removed, new tests listed). Pipeline notes that extracted Markdown is uploaded. `DocumentExtracted` schema updated. Metrics table updated. pyproject `extract` group updated. worker-extract Dockerfile apt row updated. | -| `src/contract_check/api/routes/README.md` | Upload endpoint extension list updated. | - -### Tests - -| File | Change | -|---|---| -| `tests/unit/test_extraction_adapters.py` | New: PDF text-only/table, DOCX heading, RTF, TXT chardet, OCR error paths, DTO defaults. | -| `tests/unit/test_extraction_factory.py` | New: suffix/mime/magic routing, unsupported formats, end-to-end `extract_document()`, OCR fallback behavior. | -| `tests/unit/test_chunker.py` | New: `chunk_markdown` heading preservation, packing, oversized-section split, preamble handling, fallback. | -| `tests/integration/test_extract_worker.py` | Expected exceptions updated: `ExtractionFailedError` is now a possible raised error alongside `OCRError` (the failure-class mapping remains `extraction_failed` / `ocr_failed`). | - -## Behavior changes - -- **Format support** (upload-gate accepted): `.pdf`, `.docx`, `.rtf`, `.txt`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`. -- **Output format**: extracted text is now Markdown; stored with content-type `text/markdown; charset=utf-8`. -- **Backward compatibility**: `DocumentExtracted` new fields have defaults, so in-flight messages during deploy are valid. -- **Refund policy unchanged**: `ExtractionFailedError` → `extraction_failed` (not refundable under `infra_only`); `OCRError` → `ocr_failed` (refundable). -- **Chunking**: `worker-analyze` uses heading-aware `chunk_markdown()` when structure exists; otherwise identical to previous `chunk_text()`. - -## Verification - -```bash -make lint # ruff check + format check — passed -make typecheck # mypy src — passed (92 files) -make test-unit # 132 passed, 32 deselected integration tests -``` - -Integration tests (`tests/integration/test_extract_worker.py` and -`tests/integration/test_upload_pipeline.py`) passed with the live Docker stack. -The full `make test-integration` suite is slow because it exercises the real LLM -path; the extraction-specific integration subset is green. - -## Deployment notes - -1. The `worker-extract` Docker image must be rebuilt because: - - new apt package `libmagic1` - - new Python deps `mammoth`, `striprtf`, `chardet`, `python-magic` -2. Existing running workers should be recreated (`docker compose --profile services up -d --build worker-extract`). -3. No DB migration required — changes are additive to message schema and object storage content. - -## Rollback - -- The new code is additive; old messages with fewer `DocumentExtracted` fields - still validate. If rollback is needed, the previous worker-extract image - continues to work, but extracted objects would be stored as `text/plain` - instead of `text/markdown`. - -## Deviations from the spec - -- `ExtractedDocument.attachments: list[bytes]` was omitted — there is no consumer - or storage design for extracted images yet. -- Heavy adapters (`marker`, `paddleocr`, `easyocr`, `img2table`, `email`) remain - deferred per the spec. - -## Next step - -Phase 2: prescreen stage between `worker-extract` and `worker-analyze`. Based -on `SPIKE_PHASE0.md`, this will use a deterministic regex+pydantic extractor for -RU/BY contracts instead of the Needle 2 model (English-only in testing). diff --git a/docs/PHASE2_HANDOFF.md b/docs/PHASE2_HANDOFF.md deleted file mode 100644 index ca3fa5b..0000000 --- a/docs/PHASE2_HANDOFF.md +++ /dev/null @@ -1,153 +0,0 @@ -# Phase 2 Handoff — Prescreen Stage - -> Scope: insert a deterministic prescreen stage between extraction and LLM -> analysis for Russian/Belarusian contracts. -> Completed: 2026-08-16. - -## Goal - -Short-circuit obvious low-risk contracts and protect the expensive LLM worker -from trivial documents. Use a deterministic regex+pydantic extractor because the -Needle 2 (cactus-needle) model is English-only and unusable for RU/BY contracts -(see `docs/SPIKE_PHASE0.md`). - -## What changed - -### DB schema (`migrations/versions/0006_prescreen.py`) - -- New `prescreen_results` table with contract metadata, field-coverage score, - routing decision, and lightweight auto-approve output. -- `documents.status` adds `'prescreening'`. -- `jobs.queue` adds `'prescreen'`. -- `reports` gains `prescreen_result_id` FK and `prescreen_meta` JSONB. - -### Models (`src/contract_check/core/db/`) - -- `models.py`: added `PrescreenResult` model and relationships. -- `enums.py`: `DocStatus` and `QueueName` literals updated. - -### RabbitMQ topology (`core/mq/topology.py`) - -- Added `prescreen.q`, `prescreen.retry.q`, `prescreen.dlq`. -- Added routing keys `prescreen` / `retry.prescreen`. - -### Messages (`core/mq/messages.py`) - -- `PrescreenRequested` — worker-extract → prescreen.q. -- `PrescreenCompleted` — prescreen result, persisted and forwarded. -- `AnalyzeRequested` — worker-prescreen → analyze.q, carries `prescreen_meta`. -- `DocumentExtracted` kept for backward compatibility but no longer published. - -### New service: `worker_prescreen/` - -| File | Purpose | -|---|---| -| `extractor.py` | Regex extractor for RU/BY contracts; returns `PrescreenContractMeta` + confidence score. | -| `router.py` | Routing decision: `auto_approve` / `manual_review` / `deep_analysis`. `auto_approve` is disabled by default. | -| `handler.py` | Download Markdown, run extractor, persist `prescreen_results`, publish next message. | -| `consumer.py` | RabbitMQ consumer for `prescreen.q`. | -| `config.py` | `PrescreenSettings` with prefetch tunable. | -| `__main__.py` | Entrypoint, metrics server on `:9104`. | - -### Wired existing services - -- `worker_extract/handler.py`: now publishes `PrescreenRequested` to `prescreen.q` - and sets `documents.status = 'prescreening'`. -- `worker_analyze/consumer.py` + `handler.py`: now consumes `AnalyzeRequested`, - receives `prescreen_meta`, appends it to the LLM prompt as known contract - fields. -- `core/llm/port.py` + `ollama_cloud.py`: `analyze()` accepts `extra_context: str` - and injects it into each chunk's user prompt. - -### Configuration (`core/config.py`) - -New envs: - -- `PRESCREEN_ENABLED` (default `true`) -- `PRESCREEN_AUTO_APPROVE` (default `false`) -- `PRESCREEN_CONFIDENCE_THRESHOLD` (default `0.75`) -- `PRESCREEN_HIGH_VALUE_THRESHOLD` (default `100000`) - -### Metrics (`core/metrics.py`) - -- `prescreen_duration_seconds{decision}` -- `prescreen_runs_total{decision,contract_type}` -- `prescreen_confidence` histogram - -### Docker / compose - -- New `srv/worker-prescreen/Dockerfile` (lean, no tesseract). -- Added `worker-prescreen` service to `docker-compose.yml` on port `9104`. - -### Dependencies (`pyproject.toml`) - -- Added `prescreen` dependency group (core + db/mq/s3/obs). -- Added to `dev` group. - -### Tests - -- Unit: `tests/unit/test_prescreen_extractor.py`, `tests/unit/test_prescreen_router.py`. -- Integration: updated `test_extract_worker.py`, `test_analyze_worker.py`, - `test_upload_pipeline.py`, `test_b2b_api.py` for the new pipeline. - -### Docs - -- Updated `docs/ARCHITECTURE.md` with prescreen topology, messages, schema, - metrics, and Docker/compose rows. - -## Behavior - -1. Upload → `extract.q` (unchanged). -2. worker-extract uploads Markdown to MinIO, publishes `PrescreenRequested`. -3. worker-prescreen: - - extracts contract type, parties, amount, currency, dates, penalty/ - termination/arbitration clauses via regex - - `confidence_score = matched_fields / total_fields` - - routes: - - `deep_analysis` if high value, penalty clause, or arbitration - - `manual_review` if low confidence, missing parties, or auto_approve disabled - - `auto_approve` only when `PRESCREEN_AUTO_APPROVE=true` and low risk -4. `deep_analysis` → `AnalyzeRequested` → worker-analyze with prescreen context. -5. `manual_review` → terminal DB state, no LLM call, credit **not** refunded - (this is an intentional routing outcome, not a failure). -6. `auto_approve` → lightweight report row, `status=done`. - -## Verification - -```bash -make lint # passed -make typecheck # passed (99 files) -make test-unit # 147 passed, 32 deselected -make test-integration # 32 passed, 147 deselected (with live workers stopped to avoid races) -``` - -## Deployment notes - -1. Run migration: `make migrate` (or `docker compose --profile services run --rm api alembic upgrade head`). -2. Rebuild images: `docker compose --profile services up -d --build`. -3. New worker-prescreen must be started; worker-extract and worker-analyze - images also changed. -4. No application code outside core/workers was changed; the API just returns - the new `prescreening` status string. - -## Deviations from the original prescreen spec - -- **Extractor:** regex instead of Needle 2. Rationale in `docs/SPIKE_PHASE0.md`. -- **No separate credits split:** the credit is still reserved once on upload; - prescreen is treated as part of the same paid job. -- **No admin/web SPA yet:** `manual_review` documents are terminal in the DB - until the admin panel lands. - -## Next step - -Phase 3 is storage modernization evaluation (RustFS watch). The current stack -keeps MinIO as the production default; RustFS is tracked as a future option. -Alternatively, continue with the backlog: admin/web SPA, ЮKassa payments, heavy -OCR adapters. - ---- - -**Recommended immediate follow-up:** end-to-end smoke test in Docker with all -services running (`make services-up`) and a real PDF/DOCX upload through the -API or bot, verifying that `worker-prescreen` routes to `analyze.q` or -`manual_review` correctly. diff --git a/docs/PHASES_2_PLUS_ROADMAP.md b/docs/PHASES_2_PLUS_ROADMAP.md deleted file mode 100644 index 546c587..0000000 --- a/docs/PHASES_2_PLUS_ROADMAP.md +++ /dev/null @@ -1,439 +0,0 @@ -# Implementation Roadmap — Phase 2 (Prescreen) & Remaining Work - -> Based on `document-extraction-spec.md`, `needle-prescreen-integration.md`, -> and the Phase 0 spike report (`docs/SPIKE_PHASE0.md`). -> Current checkpoint: Phase 1 extraction refactor is complete (`docs/PHASE1_HANDOFF.md`). - -## Status overview - -| Phase | Scope | Status | -|---|---|---| -| Phase 0 | Needle/RustFS spikes | Done (`docs/SPIKE_PHASE0.md`) | -| Phase 1 | Extraction layer refactor | Done (`docs/PHASE1_HANDOFF.md`) | -| Phase 2 | Prescreen stage | **Done** (`worker_prescreen/`, migrations `0006_prescreen.py` … `0008_add_manual_review_status.py`) | -| Phase 3 | Storage modernization / RustFS watch | **Planned below** | -| Follow-up | Admin/web, payments, heavy OCR | **Backlog** | - -## Phase 2 — Prescreen stage between extract and analyze - -### Why - -Insert a fast, local decision layer after extraction and before the expensive LLM -analysis so that: - -- obvious low-risk contracts can short-circuit to a lightweight report -- high-risk / high-value contracts are routed to deep LLM analysis -- the heavy LLM worker is no longer the only path for every document -- per-stage queue metrics, retry, and DLQ are clean and independent - -### Key correction from the original prescreen spec - -The original spec assumed **Needle 2** (`cactus-needle`) as the prescreen -extractor. The Phase 0 spike proved the base model is English-only and fails on -Russian contracts with `confidence=0.0`. Fine-tuning would also disable the -calibrated confidence head the routing design depends on. - -Therefore Phase 2 uses a **deterministic regex+pydantic extractor** for RU/BY -contracts. Contract boilerplate is highly templated, so regex gives: - -- zero marginal cost -- no hallucination risk -- missing fields naturally map to `manual_review` -- `confidence_score` becomes a deterministic **field-coverage ratio** - -A small-LLM fallback can be added later for fields regex misses. - -### 2.1 RabbitMQ topology changes - -Add to `core/mq/topology.py` using the existing direct-exchange + TTL retry/DLQ -pattern: - -```text -contracts.x - ├─ extract ─► extract.q (existing) - ├─ prescreen ─► prescreen.q (NEW) - └─ analyze ─► analyze.q (existing) - -contracts.retry.x - ├─ retry.prescreen ─► prescreen.retry.q (NEW classic delay queue, DLX→contracts.x[prescreen]) - -prescreen.dlq (NEW quorum) -``` - -Constants: `QUEUE_PRESCREEN`, `RK_PRESCREEN`, update `_RETRY_QUEUE_FOR`, -`DLQ_FOR`, `declare_all()`. - -### 2.2 Database migration `0006_prescreen.py` - -New table `prescreen_results`: - -```sql -CREATE TABLE prescreen_results ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - correlation_id UUID NOT NULL, - - -- extracted metadata - contract_type VARCHAR(64), - party_a TEXT, - party_b TEXT, - total_amount DECIMAL(18, 2), - currency VARCHAR(8), - start_date DATE, - end_date DATE, - has_penalty_clause BOOLEAN, - has_termination_clause BOOLEAN, - has_arbitration BOOLEAN, - confidence_score DECIMAL(4, 3), -- coverage ratio 0.000–1.000 - - -- routing - routing_decision VARCHAR(32) NOT NULL, -- auto_approve | manual_review | deep_analysis - - -- metrics / audit - prescreened_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - processing_ms INTEGER, - extractor_version VARCHAR(32) DEFAULT 'regex-v1', - - -- auto-approve lightweight output - auto_summary TEXT, - auto_findings JSONB DEFAULT '[]', - - -- retry tracking - error_message TEXT, - retry_count INTEGER DEFAULT 0 -); - -CREATE INDEX idx_prescreen_document ON prescreen_results(document_id); -CREATE INDEX idx_prescreen_routing ON prescreen_results(routing_decision); -CREATE INDEX idx_prescreen_confidence ON prescreen_results(confidence_score); -``` - -Also: - -```sql -ALTER TYPE document_status ADD VALUE 'prescreening' AFTER 'extracted'; -ALTER TYPE jobs.queue ADD VALUE 'prescreen' AFTER 'analyze'; - -ALTER TABLE reports - ADD COLUMN prescreen_result_id UUID REFERENCES prescreen_results(id), - ADD COLUMN prescreen_meta JSONB DEFAULT NULL; -``` - -Use `TEXT+CHECK` instead of Postgres enums if the project convention prefers -additive migrations (the existing schema uses both). Decision to make during -implementation. - -### 2.3 Message schema additions - -Add to `core/mq/messages.py`: - -```python -class PrescreenRequested(PipelineMessage): - """worker-extract → contracts.x[prescreen] → worker-prescreen.""" - - text_s3_key: str - filename: str - - -class PrescreenCompleted(PipelineMessage): - """worker-prescreen result; also the payload forwarded to analyze.q.""" - - text_s3_key: str - prescreened_at: datetime - contract_type: str | None - party_a: str | None - party_b: str | None - total_amount: float | None - currency: str | None - start_date: str | None - end_date: str | None - has_penalty_clause: bool | None - has_termination_clause: bool | None - has_arbitration: bool | None - confidence_score: float | None # 0..1 coverage ratio - routing_decision: Literal["auto_approve", "manual_review", "deep_analysis"] - auto_summary: str | None - auto_findings: list[dict] = Field(default_factory=list) - error_message: str | None - - -class AnalyzeRequested(PipelineMessage): - """worker-prescreen → contracts.x[analyze] → worker-analyze.""" - - text_s3_key: str - filename: str - prescreen_meta: PrescreenCompleted -``` - -Keep `DocumentExtracted` for backward compatibility; `worker-extract` may still -publish it for observability, but the prescreen stage is the main forward path. - -### 2.4 New service: `worker-prescreen` - -Directory: - -```text -src/contract_check/worker_prescreen/ - __init__.py - __main__.py - consumer.py - handler.py - router.py - extractor.py # regex+pydantic RU/BY contract extractor - config.py -``` - -#### `extractor.py` - -Deterministic extractor built with `re` + Pydantic model. Target fields: - -```python -class PrescreenContractMeta(BaseModel): - contract_type: str | None - party_a: str | None - party_b: str | None - total_amount: float | None - currency: str | None - start_date: str | None - end_date: str | None - has_penalty_clause: bool | None - has_termination_clause: bool | None - has_arbitration: bool | None -``` - -Heuristics: - -- `contract_type`: map first matched keyword (`договор поставки`, `договор оказания услуг`, `договор подряда`, `договор аренды`, `договор купли-продажи`, etc.) -- `party_a` / `party_b`: extract first two legal entities after patterns like - `Общество с ограниченной ответственностью «(.*?)»`, `Акционерное общество «(.*?)»`, `Индивидуальный предприниматель (.*?)`. -- `total_amount`: find `составляет ([\d\s,.]+) (рублей|руб|USD|EUR|€|\$)` and parse as float. -- `currency`: normalize to `RUB`, `USD`, `EUR`, `BYN`. -- `start_date` / `end_date`: match `с (\d{2}\.\d{2}\.\d{4})`, `от (\d{2}\.\d{2}\.\d{4})`, - `действует с (\d{2}\.\d{2}\.\d{4}) по (\d{2}\.\d{2}\.\d{4})`. -- `has_penalty_clause`: presence of `неустойка`, `штраф`, `пеня`, `0,1%`/`% за каждый день просрочки`. -- `has_termination_clause`: presence of `расторгнуть`, `расторжение`, `одностороннему порядке`. -- `has_arbitration`: presence of `арбитражный суд`, `Арбитражный суд`, `Международный коммерческий арбитражный суд`. - -`confidence_score = matched_fields / total_fields`. - -Test on synthetic and a few real contracts; tune false-positive/negative rate. - -#### `router.py` - -```python -class PrescreenRouter: - def __init__( - self, - confidence_threshold: float = 0.75, - high_value_threshold: float = 100_000, - ) -> None: - ... - - def decide(self, meta: PrescreenContractMeta) -> str: - if confidence < threshold or missing mandatory fields: - return "manual_review" - if total_amount > high_value_threshold: - return "deep_analysis" - if has_penalty_clause or has_arbitration: - return "deep_analysis" - return "auto_approve" -``` - -`auto_approve` is **disabled by default** via env `PRESCREEN_AUTO_APPROVE=false` -until accuracy is proven; when disabled, `auto_approve` decisions are mapped -to `manual_review`. - -#### `handler.py` - -- idempotency check on `documents.status` -- set `status = 'prescreening'`, `jobs.status = 'running'` -- download extracted Markdown from MinIO -- run regex extractor in thread pool (`asyncio.to_thread`) -- route -- persist `prescreen_results` -- publish next message: - - `deep_analysis` → `contracts.x[analyze]` with `AnalyzeRequested` - - `auto_approve` → `contracts.x[report.completed]` with lightweight summary - (only when `PRESCREEN_AUTO_APPROVE=true`) - - `manual_review` → `contracts.x[review.manual]`; no further processing -- ack - -#### `consumer.py` - -`Consumer[PrescreenRequested]` for `prescreen.q`, wired like `AnalyzeConsumer`. - -#### `config.py` - -- `PRESCREEN_ENABLED` — if false, handler immediately publishes - `AnalyzeRequested` (passthrough, preserving analytics) -- `PRESCREEN_AUTO_APPROVE` -- `PRESCREEN_CONFIDENCE_THRESHOLD` -- `PRESCREEN_HIGH_VALUE_THRESHOLD` - -### 2.5 Dockerfile + compose - -- `srv/worker-prescreen/Dockerfile`: lean image, no tesseract/pymupdf, only core - deps + regex engine. -- Add `worker-prescreen` to `docker-compose.yml` profile `services`. -- Memory limit 128M (regex is tiny). - -### 2.6 Changes to existing services - -#### `worker-extract` - -After publishing `DocumentExtracted` (backward compat), also publish -`PrescreenRequested`: - -```python -await publisher.publish( - PrescreenRequested( - correlation_id=..., - document_id=..., - user_id=..., - text_s3_key=extracted_key, - filename=payload.filename, - ), - routing_key=RK_PRESCREEN, -) -``` - -Gate behind `PRESCREEN_ENABLED` (default true after Phase 2). - -#### `worker-analyze` - -Change consumer to `AnalyzeRequested`: - -```python -class AnalyzeConsumer(Consumer[AnalyzeRequested]): - queue = "analyze.q" - message_model = AnalyzeRequested -``` - -Enrich prompt with `prescreen_meta` (§6.3 of the original spec). - -#### `api` - -- Update document status endpoint to include prescreen block. -- Status lifecycle: `queued → extracting → prescreening → analyzing | manual_review | done`. -- Optional: expose `?mode=fast` vs `?mode=full` for B2B. - -### 2.7 Metrics - -Add to `core/metrics.py` and expose from `worker-prescreen`: - -- `prescreen_requests_total{source}` -- `prescreen_duration_seconds{decision}` -- `prescreen_routing_decisions_total{decision,contract_type}` -- `prescreen_confidence_distribution` -- `prescreen_errors_total{error_type}` -- `prescreen_dlq_messages_total` - -### 2.8 Tests - -- Unit: `tests/unit/test_prescreen_extractor.py` — regex patterns on synthetic - contracts, coverage scoring. -- Unit: `tests/unit/test_prescreen_router.py` — routing matrix with all flag - combinations. -- Integration: `tests/integration/test_prescreen_worker.py`: - - `PrescreenRequested` → `AnalyzeRequested` for `deep_analysis` - - `PrescreenRequested` → `review.manual` for `manual_review` - - bypass mode (`PRESCREEN_ENABLED=false`) → passthrough `AnalyzeRequested` -- Update `test_extract_worker.py` to assert `PrescreenRequested` is also - published. -- Update `test_analyze_worker.py` to consume `AnalyzeRequested` and include - prescreen metadata. - -### 2.9 DoD - -```bash -make lint && make typecheck && make test-unit -make test-integration -``` - -## Phase 3 — Storage watch / RustFS evaluation - -### Goal - -Keep MinIO as the production default but maintain a credible migration path to -RustFS when its Lifecycle + KMS features reach GA. No application code changes -are required because S3 is already behind `core.s3.port.py`. - -### Why not switch now - -From the Phase 0 RustFS spike: - -- Lifecycle/KMS/distributed mode are marked **"Under Testing"** upstream. -- SSE-S3 objects written by MinIO are **not readable by RustFS** today — a - migration would require re-putting objects through the app. -- RustFS is weeks old; MinIO is battle-tested for this exact compose setup. -- Real advantages exist: Apache 2.0 license (vs MinIO AGPL), no telemetry, - RF data-sovereignty friendly. - -### Actions - -1. Add an optional `docker-compose.rustfs.yml` overlay for local experiments. -2. Run the full integration suite against RustFS quarterly or when upstream - announces Lifecycle/KMS GA. -3. If/when adopted: - - pin exact RustFS image tag - - set `S3_SERVER_SIDE_ENCRYPTION=false` during transition - - re-encrypt objects via the app after cutover - - update `minio-init` service or replace with `rustfs-init` - -## Backlog / follow-up work (not tied to Phase 2) - -| Item | Rationale | -|---|---| -| Admin/web SPA | Deferred by original architecture scope. | -| ЮKassa payments | Stub `invoices` table exists; payment logic deferred. | -| Heavy OCR adapters (marker, paddleocr, easyocr, img2table) | Spec deferral. Consider separate `worker-extract-heavy` queue or cloud API. | -| Email extractor (MSG/EML) | Spec deferral; only relevant for B2B corporate email ingestion. | -| Multi-tenancy | Explicit non-goal of original refactor. Do not add `tenant_id` until orgs are a real requirement. | -| Fine-tuned RU Needle model | Revisit if `cactus-needle` ships a multilingual or RU-tuned base with calibrated confidence. | - -## Appendix: original spec alignment - -### `document-extraction-spec.md` - -| Requirement | Phase 1 status | -|---|---| -| `DocumentExtractor` Protocol / ABC | Done (`core/extraction/port.py`) | -| `ExtractedDocument` dataclass / Pydantic | Done | -| `ExtractorFactory` with MIME detection | Done | -| `DocxExtractor` (`mammoth`) | Done | -| `RtfExtractor` (`striprtf`) | Done | -| `EncodingDetector` (`chardet`) | Done | -| `PyMuPDFExtractor` + `has_tables` | Done | -| `TesseractOCRAdapter` + `is_structured=False` | Done | -| Metrics by format | Done | -| Update architecture docs | Done | -| Heavy adapters deferred | Backlog | - -### `needle-prescreen-integration.md` - -| Requirement | Phase 2 plan above | -|---|---| -| Separate `prescreen.q` stage | Yes | -| `PrescreenRequested` / `PrescreenCompleted` messages | Yes, corrected to extend `PipelineMessage` | -| `prescreen_results` table | Yes, adjusted columns | -| `worker-prescreen` service | Yes, regex-based instead of Needle | -| Routing decisions | Yes, conservative default | -| `worker-extract` publishes prescreen | Yes | -| `worker-analyze` consumes `AnalyzeRequested` | Yes | -| API status endpoint enriched | Yes | -| Dockerfile/compose for prescreen worker | Yes | -| Rollback / bypass flag | Yes | -| Multi-tenant / credit split | **Rejected** — not in current scope; integer credit model stays | - -## Ordering recommendation - -1. Phase 2 DB migration + RabbitMQ topology. -2. Phase 2 messages + `worker_prescreen` skeleton (no-op passthrough first). -3. Phase 2 regex extractor + router + tests. -4. Wire `worker-extract` → `prescreen.q` and `worker-analyze` → `analyze.q`. -5. End-to-end integration test. -6. Phase 3 RustFS spike only after upstream GA announcement. - ---- - -**Next action:** implement Phase 2 step 1 (migration + topology) if approved. diff --git a/docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md b/docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md deleted file mode 100644 index 7f8c5f1..0000000 --- a/docs/PRESCREEN_HYBRID_REFACTOR_PLAN.md +++ /dev/null @@ -1,242 +0,0 @@ -# Refactor Plan: Prescreen Worker → Hybrid Extraction (Heuristic + LLM Fallback) - -> Status: delivered through Phase 4; Phase 5 rollout pending merge -> Scope: `src/contract_check/worker_prescreen/`, `src/contract_check/core/llm/`, `src/contract_check/core/config.py`, `src/contract_check/core/metrics.py` -> Supersedes: the regex-only extractor described in `worker_prescreen/extractor.py` (regex-v1) - ---- - -## 1. Goal - -Replace the monolithic regex extractor (`extractor.py`, ~260 lines of Russian/BY legal -regexes) with a **hybrid two-stage extractor**: - -1. **Stage 1 — Heuristic (deterministic, zero-cost):** keyword dictionaries, - positional windows, sentence scanning, per-field confidence weights. - No regex — plain string operations only. -2. **Stage 2 — LLM fallback (only when Stage 1 confidence is low):** reuse the - existing `core/llm` provider infrastructure (Ollama Cloud / YandexGPT) with a - JSON-schema-constrained extraction prompt. - -Non-goals: - -- No changes to routing semantics (`router.py` thresholds/decisions stay as-is). -- No DB schema migration (`prescreen_results.extractor_version` already exists, - `String(32)` fits `"heuristic-v2"` / `"llm-v1"`). -- No changes to MQ topology, messages (`PrescreenCompleted` shape unchanged), - or downstream `worker_analyze`. -- Auto-approve stays disabled by default. - -## 2. Current State - -``` -worker_prescreen/ -├── consumer.py # consumes prescreen.q → handler -├── handler.py # Stage 4: asyncio.to_thread(extract_contract_meta, text) -├── extractor.py # PrescreenContractMeta + 10 regex patterns ← TO REPLACE -├── router.py # decides auto_approve|manual_review|deep_analysis -└── config.py -``` - -Problems: - -- Regexes are brittle (line-noise, OCR artifacts, whitespace variants) and hard - to extend (every new contract type = new regex). -- No recovery path: low-confidence extraction always ends in `manual_review`. -- `prescreen_results.extractor_version` is never written by the handler INSERT — - it silently relies on the DB default `'regex-v1'`. - -## 3. Target Architecture - -``` - ┌────────────────────────────────────────────┐ - │ handler.py │ - │ Stage 4: meta = await extractor.extract() │ - └───────────────┬────────────────────────────┘ - │ - HybridMetaExtractor (orchestrator) - │ - ┌────────────────────┴──────────────────────┐ - ▼ ▼ - HeuristicExtractor (sync, to_thread) LLMPrescreenExtractor - keyword + positional + sentence wraps core/llm provider - scanning, confidence-weighted extract_prescreen(text) - │ │ - confidence ≥ threshold? ──── no ────► run LLM, validate JSON, - │ yes merge over heuristic meta - ▼ - PrescreenContractMeta → router.decide() → persist (with extractor_version) -``` - -### 3.1 New module layout (flat, matches existing package conventions) - -``` -worker_prescreen/ -├── extractor.py # PrescreenContractMeta (model, unchanged shape) -│ # + MetadataExtractor protocol + re-export shim -├── extractor_heuristic.py# HeuristicExtractor (Stage 1) -├── extractor_llm.py # LLMPrescreenExtractor (Stage 2, dict → pydantic validation) -├── extractor_hybrid.py # HybridMetaExtractor (threshold + merge + kill-switch) -├── handler.py # Stage 4 rewired; INSERT gains extractor_version -└── router.py # unchanged -``` - -### 3.2 Heuristic stage design (regex-free) - -| Field | Method | Notes | -|---|---|---| -| `contract_type` | Phrase dictionary match on normalized text | `{"supply": ["договор поставки", "договор купли-продажи", ...], ...}` — lowercase + whitespace-collapse once, then `in` checks | -| `party_a/b` | Positional: scan only the header window (text up to first `1.` / `ПРЕДМЕТ` heading, capped ~1500 chars); token-scan for entity-form tokens (ООО, ИП, АО, …) then capture until quote `»`/`"` close or line end | Replaces `_ENTITY_RE` | -| `total_amount` + `currency` | Trigger-word scan (`составляет`, `стоимость`, `цена`, …) then manual digit-window parse (`_scan_number` helper walking chars, handling spaces/commas); currency via token lookup in the following ~50 chars | Replaces `_AMOUNT_RE` | -| `start/end_date` | Trigger-word scan (`действует с`, `с … по …`) then `_parse_ddmmyyyy` manual splitter (already regex-free in v1 — keep) | Replaces `_DATE_RE`/`_END_DATE_RE` | -| boolean clauses | Sentence segmentation via `str.split` on `. ` / `\n`, keyword membership per sentence (`неустойка`, `штраф`, `расторгнуть`, `арбитражн`, …) | Replaces `_PENALTY_RE` etc. | - -Confidence scoring becomes **weighted** instead of flat field coverage: - -```python -FIELD_WEIGHTS = { - "contract_type": 0.25, - "party_a": 0.15, - "party_b": 0.15, - "total_amount": 0.15, - "currency": 0.05, - "start_date": 0.10, - "end_date": 0.05, - "has_penalty_clause": 0.05, - "has_termination_clause": 0.05, - "has_arbitration": 0.05, -} # sums to 1.0 -``` - -Optional per-field method bonus (exact phrase match = full weight, positional -window hit = full weight, fuzzy tail hit = ×0.7) — start simple, weights -constant, tune later with real data. - -### 3.3 LLM fallback design - -- **Protocol extension** (`core/llm/port.py`): - `async def extract_prescreen(self, text: str) -> dict[str, Any]` — returns raw - JSON dict; **no import of worker_prescreen** (keeps layering clean). - Implemented by both `OllamaCloudProvider` and `YandexGPTProvider` as a thin - wrapper over their existing JSON-chat + repair-loop machinery - (`_chat_json` / `responseFormat=json_schema`) with a dedicated - `PRESCREEN_SYSTEM` prompt: extract the 10 meta fields, cite nothing, JSON only. -- **Worker-side wrapper** (`extractor_llm.py`): - `dict` → `PrescreenContractMeta` via pydantic (rejects hallucinated fields, - re-`None`s unknown enum values), then recomputes weighted confidence. -- **Input cap:** only the first `prescreen_llm_max_chars` (default 20 000) chars - are sent — meta lives in the header for templated contracts; cost control. -- **Merge rule:** LLM values override heuristic `None`s and low-confidence - fields; boolean clauses become OR(heuristic, llm) — both sources are - presence-checks, false positives are cheap, false negatives route wrong. - -### 3.4 Failure semantics - -- LLM fallback error (quota, timeout, invalid JSON after repair): - **do not fail the message.** Log + keep heuristic meta, - `extractor_version="heuristic-v2"`, record the error in - `prescreen_results.auto_findings` (`{"llm_fallback_error": "..."}`) and the - `prescreen_fallback_runs_total{outcome="failed"}` counter. -- Heuristic stage is pure string ops — its only failure mode is `None` fields, - which is already handled by low confidence → fallback / manual_review. - -### 3.5 Config additions (`core/config.py`, `--- prescreen stage ---`) - -```python -prescreen_llm_fallback_enabled: bool = Field(default=False, ...) # kill-switch; flip to True after burn-in -prescreen_llm_fallback_threshold: float = Field(default=0.75, ...) # ≤ router threshold -prescreen_llm_max_chars: int = Field(default=20_000, ...) -``` - -### 3.6 Metrics additions (`core/metrics.py`) - -```python -prescreen_fallback_runs = Counter( - "contract_check_prescreen_fallback_runs_total", - "...", labelnames=["outcome"], # used | failed | skipped | disabled -) -prescreen_extraction_stage = Histogram( - "contract_check_prescreen_extraction_stage_seconds", - "...", labelnames=["stage"], # heuristic | llm -) -``` - -`extractor_version` values: `"regex-v1"` (legacy shim flag), `"heuristic-v2"` -(LLM not run / disabled / failed) and `"hybrid-llm-v1"` (LLM result merged). - -## 4. Implementation Phases - -Each phase is independently shippable and reverted by config/env flag. - -### Phase 1 — Port + heuristic extractor (no behavior change for routing) -- [x] `extractor.py`: keep `PrescreenContractMeta` + weighted `_score_confidence`; - add `MetadataExtractor` protocol (`def extract(text: str) -> PrescreenContractMeta`); - keep `extract_contract_meta` as a delegating shim so - `tests/unit/test_prescreen_extractor.py` and `worker_prescreen/__init__.py` - keep importing it. -- [x] New `extractor_heuristic.py` implementing all helpers from §3.2 - (`_scan_number`, `_find_after_trigger`, header-window splitter, - sentence splitter, dictionaries as module-level constants). -- [x] Port existing unit-test fixtures (`SIMPLE_SUPPLY`, `MINIMAL`, boolean-flag - parametrize) onto `HeuristicExtractor`; they must pass with identical - expected values. -- [x] `extractor_version` shim reports `"heuristic-v2"`. - -### Phase 2 — LLM `extract_prescreen` on providers -- [x] Extend `LLMProvider` protocol + both adapters - (`ollama_cloud.py`, `yandex_gpt.py`): new `PRESCREEN_SYSTEM` prompt, - JSON schema for the 10 fields, reuse repair loop, truncate input to - `chunk_size`-independent small cap. -- [x] Unit tests with a fake transport (both adapters already have this pattern): - valid dict, invalid enum → `None`, malformed JSON → repair once → fail. -- [x] `extractor_llm.py`: dict → `PrescreenContractMeta` validation + - weighted confidence. - -### Phase 3 — Orchestrator + handler wiring -- [x] `extractor_hybrid.py`: threshold check, LLM call (async), merge rules, - error swallowing, extractor_version selection, kill-switch. -- [x] `handler.py` Stage 4: replace `asyncio.to_thread(extract_contract_meta, …)` - with `await self._extractor.extract(contract_text)` where - `self._extractor` is injectable (constructor arg, defaults to hybrid) — - mirrors the existing `provider` injection pattern in `AnalyzeHandler`. -- [x] Handler Stage 6 INSERT: bind `extractor_version`. -- [x] Config + metrics from §3.5/§3.6. - -### Phase 4 — Tests + verification -- [x] Unit: orchestrator matrix — high confidence skips LLM; low confidence - merges; LLM failure → heuristic result + `outcome="failed"`; disabled → - `outcome="disabled"`. -- [x] Integration (`tests/integration/test_prescreen_worker.py` pattern): - run handler in-process with a stub provider; assert - `prescreen_results.extractor_version` persisted, routing unchanged. -- [x] Backfill spot-check: replayed 8 RU/BY-style contract samples through - regex-v1 vs heuristic-v2. Confidence deltas: all neutral or better, - heuristic fixes regex under-parsing on NBSP/noise and captures fuller - party names. Record below in PR description. -- [x] `ruff check . && mypy src && pytest tests/unit tests/integration -k prescreen`. - (ruff/mypy findings outside the touched files are from pre-existing - uncommitted changes; prescreen tests pass.) - -### Phase 5 — Rollout -- [ ] Merge with `prescreen_llm_fallback_enabled=false` (pure heuristic). -- [ ] Observe `prescreen_fallback_runs` / `prescreen_confidence` for a few days. -- [x] Document in `docs/ARCHITECTURE.md` prescreen section + `.env.example` - completed. The regex notes in `SPIKE_PHASE0.md` are no longer referenced - from the extractor docstring. Staging/prod enablement remains an - operational step after merge. - -## 5. Risks & Mitigations - -| Risk | Mitigation | -|---|---| -| Heuristic drops accuracy vs regex on edge templates | Port 100% of v1 unit tests before deleting regex; keep `extractor.py` regex code one release behind a flag (`PRESCREEN_KEEP_REGEX=true` env, temporary) | -| LLM hallucinates fields | pydantic validation whitelists contract_type enums; numeric/date parse checks; booleans only OR-merged | -| LLM latency blows up prescreen SLA | 20k char cap, existing provider timeouts, `prescreen_extraction_stage` histogram; fallback failures never block the pipeline | -| Cost creep on fallback rate | `prescreen_fallback_runs_total` alert; threshold tunable without deploy via env | -| Layering violation (core ← worker import) | Protocol returns plain `dict`; pydantic model stays in `worker_prescreen` | - -## 6. Rollback - -1. Config: `PRESCREEN_LLM_FALLBACK_ENABLED=false` → deterministic heuristic only. -2. Full: revert merge — no DB migration to undo; `extractor_version` strings are - informational only. diff --git a/docs/SPIKE_PHASE0.md b/docs/SPIKE_PHASE0.md deleted file mode 100644 index 0efca7b..0000000 --- a/docs/SPIKE_PHASE0.md +++ /dev/null @@ -1,103 +0,0 @@ -# Phase 0 Spike Report — Needle 2 prescreen & RustFS storage - -> Date: 2026-08-16. Evidence from live testing in isolated environments -> (`/tmp/opencode/needle-spike`, ephemeral `rustfs/rustfs:latest` container). -> Gates the Phase 1 (extraction refactor) and Phase 2 (prescreen stage) plans. - -## Spike 1: cactus-needle (Needle 2) for RU contract prescreen — **NOT VIABLE** - -### What was verified - -- `cactus-needle==2.0.5` is real (Cactus Compute, Apache-2.0, first release - 2026-08-10). Installs cleanly on Python 3.13. The spec's claims hold: - 14MB engine (fetched once from HF, cached offline), `needle.extract(text, - PydanticModel)`, calibrated `confidence` on responses, `.cact` LoRA - fine-tuning pipeline. Note: PyPI package name is **`cactus-needle`** - (`import needle`); bare `needle` on PyPI is an unrelated dead 2017 CSS - testing tool. -- Engine session RAM measured at 37.8MB (simple schema) to 119.7MB (larger - schema) — fine for a 256M container. Python deps pull the JAX/Flax/optax - stack (~200MB+ image impact) because fine-tuning support ships in the core - dependency list. - -### Results (base weights, `buffer_size` default) - -| Case | Result | -|---|---| -| EN invoice (README example) | Perfect extraction, `confidence=0.9484` | -| EN off-topic sentence | Correctly refused: empty call `[]`, `confidence=0.7427` | -| RU contract, 2.5K chars | **Total hallucination**: parties → "Supplier"/"Supplier", amount → 12.0 USD (real: 1 234 567,80 RUB), dates → 2024 defaults. `confidence=0.0` | -| RU contract, 43K chars | Same failure mode, different hallucinations. `confidence=0.0` | -| RU garbage text | **Not refused** — fabricated "Party A/Party B/150 USD". `confidence=0.0` | - -### Verdict - -The base model is **English-centric; it does not understand Russian**. The -calibrated confidence head behaves correctly (0.0 = "I can't read this"), so a -naive deployment would have been *safe* (100% → manual_review) but useless. - -Fine-tuning cannot rescue this for v1: - -1. Calibrated confidence is **disabled on tuned weights** (reports `None`) — - and the spec's entire routing design hinges on confidence gating. -2. The model failed even trivial verbatim copy tasks (could not copy - «ООО «Ромашка»» from adjacent text) — this is tokenizer/training coverage, - not prompt tuning. -3. Building a RU legal extraction dataset + LoRA pipeline is its own project. - -### Plan adjustment (Phase 2) - -- `worker-prescreen` ships with a **deterministic regex+pydantic extractor** - instead of Needle: RU/BY contract boilerplate is highly templated - («договор поставки», «именуемое в дальнейшем», «сумма договора составляет - N рублей», «неустойка», «арбитражный суд», DD.MM.YYYY ranges). Zero marginal - cost, no hallucination, p95 ≈ ms. Absence of a match = `None` → - `manual_review`, which matches the routing philosophy. -- `confidence_score` is redefined as a deterministic **field-coverage score** - (fraction of target fields extracted), not model calibration. Router - thresholds operate on coverage. -- `needle_client.py` / `model_weights` / `NEEDLE_*` env are dropped from the - plan; keep `prescreen_results.extractor_version` for future engine swaps. -- Optional fast-follow (not v1): small-LLM prescreen via the existing - `core/llm` port for fields regex missed. -- Revisit Needle only if/when a multilingual or RU-tuned base ships with - calibration intact. - -## Spike 2: RustFS as MinIO alternative — **VIABLE, but stay on MinIO for now** - -Tested with the project's own `MinioStorage` adapter (minio SDK) against -`rustfs/rustfs:latest`, single node, named volume. - -| Check | Result | -|---|---| -| Health (`/minio/health/live`) | 200 — MinIO-compatible probe works | -| bucket create/exists, put/get/stat/delete | OK (incl. Cyrillic UTF-8 payloads) | -| `set/get/delete_bucket_lifecycle` (ILM config API) | OK — rule round-trips | -| **ILM expiration behavior** (`RUSTFS_ILM_DEBUG_DAY_SECS=2`, rule days=1) | **Object under `users/` purged in ~10s; control object outside prefix untouched** | -| Perf (4KB put+get ×20) | ~2.0ms/op | -| Docker ergonomics | Runs as UID 10001 — bind mounts must be chowned; named volumes are the easy path | - -### Verdict - -The critical blocker (152-ФЗ TTL purge via ILM) **works in practice** — the -"Under Testing" label upstream refers to CI gating, not absence. Remaining -reasons to keep MinIO as the default for production: - -1. Project is weeks old (1.0.0-rc era); lifecycle, distributed mode, and KMS - are all still marked "Under Testing" upstream. -2. **SSE migration trap**: objects written by MinIO with SSE-S3/KMS/C are not - readable by RustFS — a later migration requires re-putting objects through - the app (we run SSE-S3=off in dev, on in prod per `S3_SERVER_SIDE_ENCRYPTION`). -3. No operational history; MinIO is battle-tested for this exact compose setup. - -Because everything S3-shaped is already behind `core/s3/port.py`, switching -later is a compose + env change, not a code change. Action item: revisit when -RustFS marks Lifecycle + KMS GA (tracked in ARCHITECTURE notes, not code). - -## Phase gate outcomes - -- **Phase 1 (extraction refactor): GO** — unchanged. -- **Phase 2 (prescreen stage): GO with amendments** — regex extractor instead - of Needle; coverage-based confidence; no needle deps/env; conservative - routing (`PRESCREEN_AUTO_APPROVE=false` default) as already planned. -- **Storage: no change** — MinIO stays; RustFS is a credible fast-follow. diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md new file mode 100644 index 0000000..0209a19 --- /dev/null +++ b/docs/agents/issue-tracker.md @@ -0,0 +1,30 @@ +# Issue tracker: Local Markdown + +Issues and specs for this repo live as markdown files in `.scratch/`. + +## Conventions + +- One feature per directory: `.scratch//` +- The spec is `.scratch//spec.md` +- Implementation issues are one file per ticket at `.scratch//issues/-.md`, numbered from `01`, never a single combined tickets file +- Triage state is recorded as a `Status:` line near the top of each issue file (see `triage-labels.md` for the role strings) +- Comments and conversation history append to the bottom of the file under a `## Comments` heading + +## When a skill says "publish to the issue tracker" + +Create a new file under `.scratch//` (creating the directory if needed). + +## When a skill says "fetch the relevant ticket" + +Read the file at the referenced path. The user will normally pass the path or the issue number directly. + +## Wayfinding operations + +Used by `/wayfinder`. The **map** is a file with one **child** file per ticket. + +- **Map**: `.scratch//map.md` (the Notes / Decisions-so-far / Fog body). +- **Child ticket**: `.scratch//issues/NN-.md`, numbered from `01`, with the question in the body. A `Type:` line records the ticket type (`research`/`prototype`/`grilling`/`task`); a `Status:` line records `claimed`/`resolved`. +- **Blocking**: a `Blocked by: NN, NN` line near the top. A ticket is unblocked when every file it lists is `resolved`. +- **Frontier**: scan `.scratch//issues/` for files that are open, unblocked, and unclaimed; first by number wins. +- **Claim**: set `Status: claimed` and save before any work. +- **Resolve**: append the answer under an `## Answer` heading, set `Status: resolved`, then append a context pointer (gist + link) to the map's Decisions-so-far in `map.md`. diff --git a/docs/agents/triage-labels.md b/docs/agents/triage-labels.md new file mode 100644 index 0000000..ad2d1ca --- /dev/null +++ b/docs/agents/triage-labels.md @@ -0,0 +1,13 @@ +# Triage Labels + +The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker. + +| Label in mattpocock/skills | Label in our tracker | Meaning | +| -------------------------- | -------------------- | ---------------------------------------- | +| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue | +| `needs-info` | `needs-info` | Waiting on reporter for more information | +| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent | +| `ready-for-human` | `ready-for-human` | Requires human implementation | +| `wontfix` | `wontfix` | Will not be actioned | + +When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.