Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

How We Cut Target Leakage from 92% to 0.1% in Court Outcome Prediction (and Built a Triple-A MCP Server)

Описание и применение

Мы сократили утечку цели в предсказании судебных решений с 92% до 0,1% путем очистки текста и использовали LightGBM для обучения модели, получив 78,2% точность. Модель развернута как сервер MCP без внешних зависимостей.

Задача

AI models for court outcome prediction suffer from target leakage, where 92% of raw texts contain the outcome verbatim, causing models to simply read back the answer instead of learning legal reasoning.

Результат

Target leakage dropped from 92% to 0.1%; model achieved 78.2% accuracy (vs 43.7% baseline) with macro‑F1 of 77.1% across 609,715 cases. MCP server received Triple‑A audit rating.

Рабочий процесс

  • 1. Implement strict sanitization: strip dictum, summary lines, and outcome‑announcing phrases from raw texts.
  • 2. Continuously measure residual outcome markers to confirm leakage reduction.
  • 3. Train LightGBM on the sanitized dataset.
  • 4. Evaluate with 5‑fold cross‑validation on out‑of‑fold data.
  • 5. Wrap the pipeline into a zero‑dependency MCP server exposing three tools: rechtspraak_cijfers, lekkage_check, voorspel_uitkomst.

Доказательства

Leakage dropped from 92% to 0.1% (around 1 in 1,000 texts).

Источник

Overall Accuracy: 78.2% (against a 43.7% majority baseline)

Источник

Zero external dependencies: Single Python file using only standard library

Источник

  • Дата записи: 2026-09-15T21:00:45.174Z