I Tried to Make Natural Language a SQL Predicate. The Numbers Said No.

说明与用途

SemPred 原型实现了在 SQL 中使用自然语言谓词，而无需逐行调用 LLM。采用冻结的 MiniLM 编码器、逻辑回归头、嵌入缓存和弃权阈值。在 Banking77 上，准确率为 72%，但覆盖率仅为 44%（0.1 阈值），未达到 90% 准确率/50% 覆盖率门槛。

问题

Integrating natural‑language predicates into SQL without per‑row API calls, while keeping deterministic, batched inference and native database integration.

结果

The prototype met accuracy but failed the coverage gate; overall accuracy 72% vs 85% TF‑IDF baseline, coverage 44% with abstention, and 21% at 90% accuracy. The architecture was deemed not production‑ready.

工作流

• 1. Encode ticket_text with frozen all‑MiniLM‑L6‑v2 encoder.
• 2. Cache embeddings to avoid re‑encoding identical text.
• 3. For each predicate, run a logistic‑regression head on the shared embedding.
• 4. Apply abstention margin (0.1) around threshold 0.5 to produce UNKNOWN when confidence is low.
• 5. Expose SEM_SCORE and SEM_PREDICT as DuckDB Arrow UDFs for batched execution.
• 6. Benchmark on Banking77 dataset using few‑shot examples per predicate.
• 7. Evaluate accuracy, coverage, and gate conditions (≥90% accuracy & ≥50% coverage).

证据

I built a prototype called SemPred, benchmarked it, and stopped when it missed the bar I'd set beforehand.

来源

The constraints I set up front: no external API call per row, batched inference, reusable embeddings, deterministic output, model serialization, explicit abstention, native DuckDB integration, a reproducible benchmark.

来源

At 16 examples, MiniLM beats the matched TF-IDF setup by about 11.9 points.

来源

At an abstention margin of 0.10, the 16-shot system committed to a decision on about 44.1% of ticket-predicate pairs.

来源

No margin I evaluated met both ≥90% committed-ticket accuracy AND ≥50% ticket coverage.

来源