How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)

Описание и применение

Создана локальная LLM‑платформа CreativeAudit, которая автоматически оценивает рекламные креативы по строгим контрактам Pydantic, сокращая время первой проверки с десятков минут до секунд.

Задача

Art directors spend 20–40 minutes per creative reviewing brand guidelines, mandatory elements, and forbidden techniques, making first‑pass review slow, expensive and inconsistent.

Результат

First‑pass review time dropped from tens of minutes to seconds; output became consistent, machine‑readable, and critical brand violations were reliably caught.

Рабочий процесс

• Collect brief and creatives
• Render Jinja2 prompt from brief and creatives
• Send prompt to local LLM (Ollama)
• Validate LLM output against strict Pydantic v2 schema
• If validation fails, reject and log error
• Score brand_alignment, constraint_compliance, message_clarity
• Compute total weighted score
• Derive verdict based on score and critical failures
• Return verdict, score, and explanation to UI or API

Доказательства

First-pass review time dropped from tens of minutes to seconds

Источник

The model is non-deterministic. I treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score.

Источник

I wrote 38 unit tests. The external LLM is fully mocked, so tests are deterministic and run in under 2 seconds.

Источник