How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)

Description and use

A local LLM pipeline (CreativeAudit) was built to automatically evaluate ad creatives against brand briefs using strict Pydantic contracts, reducing first‑pass review time from minutes to seconds.

Problem

Art directors spend 20–40 minutes per creative reviewing brand guidelines, mandatory elements, and forbidden techniques, making first‑pass review slow, expensive and inconsistent.

Outcome

First‑pass review time dropped from tens of minutes to seconds; output became consistent, machine‑readable, and critical brand violations were reliably caught.

Workflow

• Collect brief and creatives
• Render Jinja2 prompt from brief and creatives
• Send prompt to local LLM (Ollama)
• Validate LLM output against strict Pydantic v2 schema
• If validation fails, reject and log error
• Score brand_alignment, constraint_compliance, message_clarity
• Compute total weighted score
• Derive verdict based on score and critical failures
• Return verdict, score, and explanation to UI or API

Evidence

First-pass review time dropped from tens of minutes to seconds

Source

The model is non-deterministic. I treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score.

Source

I wrote 38 unit tests. The external LLM is fully mocked, so tests are deterministic and run in under 2 seconds.

Source