I Beat Meta's LLM Guardrail With No GPU and No Team - Here's How

Description and use

A solo developer built the FIE local guardrail, detecting prompt attacks with 98.6% recall and 8% false positives, outperforming Meta’s guardrail recall while requiring no GPU or team.

Problem

A single developer needed a way to detect and mitigate prompt attacks on an LLM-powered tool without relying on GPU, large teams, or external APIs.

Outcome

FIE achieved 98.6% recall and 8% false positive rate on 282 real adversarial prompts, outperforming Meta Prompt Guard 2-86M’s recall while maintaining a competitive false positive rate.

Workflow

• 1. Identify prompt injection vulnerability in the LLM app.
• 2. Design FIE to sit between the app and the LLM, inspecting prompts and outputs.
• 3. Build 13 detection layers (regex, semantic scoring, FAISS search, encoding detection, escalation tracking).
• 4. Add Shadow Jury: three independent models cross-check outputs for hallucinations.
• 5. Define failure archetypes (e.g., HALLUCINATION_RISK, OVERCONFIDENT_FAILURE).
• 6. Implement auto‑correction for high‑confidence fixes.
• 7. Integrate via a single decorator: @monitor(mode="local").
• 8. Evaluate against JailbreakBench (282 prompts) to measure recall and false positives.
• 9. Iterate on semantic layer using PAIR classifier to reduce false positives.
• 10. Publish SDK (fie-sdk) and GitHub repo for community use.

Evidence

FIE hit 98.6% recall. Prompt Guard hit 64.9%.

Source

Evaluated against 282 real adversarial prompts from JailbreakBench

Source