I Tried to Make Natural Language a SQL Predicate. The Numbers Said No.

Description and use

SemPred prototype enables natural‑language predicates in SQL without per‑row LLM calls. It uses a frozen MiniLM encoder, logistic heads, embedding cache, and an abstention margin. On Banking77, accuracy is 72 % but coverage is 44 % at a 0.1 margin, failing the 90 % accuracy/50 % coverage gate.

Problem

Integrating natural‑language predicates into SQL without per‑row API calls, while keeping deterministic, batched inference and native database integration.

Outcome

The prototype met accuracy but failed the coverage gate; overall accuracy 72% vs 85% TF‑IDF baseline, coverage 44% with abstention, and 21% at 90% accuracy. The architecture was deemed not production‑ready.

Workflow

• 1. Encode ticket_text with frozen all‑MiniLM‑L6‑v2 encoder.
• 2. Cache embeddings to avoid re‑encoding identical text.
• 3. For each predicate, run a logistic‑regression head on the shared embedding.
• 4. Apply abstention margin (0.1) around threshold 0.5 to produce UNKNOWN when confidence is low.
• 5. Expose SEM_SCORE and SEM_PREDICT as DuckDB Arrow UDFs for batched execution.
• 6. Benchmark on Banking77 dataset using few‑shot examples per predicate.
• 7. Evaluate accuracy, coverage, and gate conditions (≥90% accuracy & ≥50% coverage).

Evidence

I built a prototype called SemPred, benchmarked it, and stopped when it missed the bar I'd set beforehand.

Source

The constraints I set up front: no external API call per row, batched inference, reusable embeddings, deterministic output, model serialization, explicit abstention, native DuckDB integration, a reproducible benchmark.

Source

At 16 examples, MiniLM beats the matched TF-IDF setup by about 11.9 points.

Source

At an abstention margin of 0.10, the 16-shot system committed to a decision on about 44.1% of ticket-predicate pairs.

Source

No margin I evaluated met both ≥90% committed-ticket accuracy AND ≥50% ticket coverage.

Source