A reranker made my legal search worse

说明与用途

添加多语言重排序器导致DPOLens搜索召回率下降并延长响应时间。

问题

Adding a multilingual reranker to DPOLens’ search pipeline degraded recall and increased latency.

结果

The reranker consistently lowered recall@5 from 0.77 to 0.57–0.43 while increasing query time up to 40×, demonstrating that a general‑purpose multilingual reranker can harm legal search performance.

工作流

• 1. Write 50 developer‑style GDPR questions. 2. Split into 20 practice and 30 held‑out sets. 3. Measure recall@5 for keyword only, meaning only, meaning without recitals, and combined. 4. Add multilingual reranker, reorder top 10/25/50. 5. Measure recall@5 and latency for each reranker depth. 6. Verify reranker wiring (output, direction, input format). 7. Analyze score distributions and identify why reranker harms performance. 8. Document findings and keep code for future experiments.

证据

DPOLens runs two kinds of search over the same clauses.

来源

Combining both searches took it to 0.77.

来源

Worse at every depth, and steadily worse the deeper it went, at up to 40 times the time per question.

来源

The answer was in the scores. For a question about reporting a data breach, the reranker scored the clause that answers it at -2.11. It scored boilerplate about the date the regulation enters into force at -2.20.

来源