I Tried to Make Natural Language a SQL Predicate. The Numbers Said No.

Описание и применение

Разработан прототип SemPred, позволяющий использовать естественный язык как предикат в SQL без вызовов LLM‑API. Используется фиксированный энкодер MiniLM, логистические головы, кэш эмбеддингов и порог с абстенцией. В тестах на Banking77 точность 72 %, но покрытие 44 % при 0,1‑м пороге, что не удовлетворяет заданному порогу 90 % точности и 50 % покрытия.

Задача

Integrating natural‑language predicates into SQL without per‑row API calls, while keeping deterministic, batched inference and native database integration.

Результат

The prototype met accuracy but failed the coverage gate; overall accuracy 72% vs 85% TF‑IDF baseline, coverage 44% with abstention, and 21% at 90% accuracy. The architecture was deemed not production‑ready.

Рабочий процесс

• 1. Encode ticket_text with frozen all‑MiniLM‑L6‑v2 encoder.
• 2. Cache embeddings to avoid re‑encoding identical text.
• 3. For each predicate, run a logistic‑regression head on the shared embedding.
• 4. Apply abstention margin (0.1) around threshold 0.5 to produce UNKNOWN when confidence is low.
• 5. Expose SEM_SCORE and SEM_PREDICT as DuckDB Arrow UDFs for batched execution.
• 6. Benchmark on Banking77 dataset using few‑shot examples per predicate.
• 7. Evaluate accuracy, coverage, and gate conditions (≥90% accuracy & ≥50% coverage).

Доказательства

I built a prototype called SemPred, benchmarked it, and stopped when it missed the bar I'd set beforehand.

Источник

The constraints I set up front: no external API call per row, batched inference, reusable embeddings, deterministic output, model serialization, explicit abstention, native DuckDB integration, a reproducible benchmark.

Источник

At 16 examples, MiniLM beats the matched TF-IDF setup by about 11.9 points.

Источник

At an abstention margin of 0.10, the 16-shot system committed to a decision on about 44.1% of ticket-predicate pairs.

Источник

No margin I evaluated met both ≥90% committed-ticket accuracy AND ≥50% ticket coverage.

Источник