WeirdClauses: Skimming the Boring 90% of Contracts with Open-Weight Surprisal

Description and use

WeirdClauses enables rapid detection of anomalous clauses in contracts via local surprisal calculation and standard-clause conditioning, cutting review time to 3–4 minutes.

Problem

Reading long contracts is mentally exhausting; most clauses are boilerplate, but the few anomalous clauses can hide significant risks.

Outcome

On a benchmark of 100 synthetic leases, WeirdClauses achieved Recall@10% of 0.517, Mean Percentile Rank of 0.820, and MRR of 0.916, demonstrating that reviewing the top 10% of flagged clauses captures the majority of anomalous terms.

Workflow

• Ingest contract file (.txt, .md, .docx, .pdf)
• Segment into clauses with exact character spans
• Compute per-token linguistic surprisal using Qwen/Qwen2.5-0.5B
• Retrieve nearest standard clauses via sentence-transformers/all-MiniLM-L6-v2
• Condition surprisal on standard clauses
• Aggregate token surprisals (top-quartile mean, Z-score)
• Rank clauses by unusualness
• Apply rule-based regex to extract key facts
• Generate standalone HTML heat-map report
• Present interactive web explorer or CLI ranking table

Evidence

By reading just the top 10% highest-ranked clauses, you capture over 51.7% of all anomalous terms.

Source

Zero document text, tokens, or metadata ever leave your machine. No API keys, no telemetry, no recurring SaaS subscriptions.

Source

The very first unusual clause in the document appears at rank 1.09 on average—meaning the top-ranked clause is almost guaranteed to be an anomaly.

Source