LLMs are great writers and terrible referees: why my PR roaster has a strict judge
Description and use
RoastMyPR now reliably produces jokes that roast code, not people, and rejects hallucinated or unsafe content while remaining a lightweight, fail‑open tool for developers.
Problem
An LLM that writes PR roasts also decides if the roast is acceptable, leading to inconsistent, inappropriate, and hallucinated content.
Outcome
RoastMyPR now reliably produces jokes that roast code, not people, and rejects hallucinated or unsafe content while remaining a lightweight, fail‑open tool for developers.
Workflow
- Extension grabs PR title, description, and diff
- Claude writes a short roast JSON (punchline, archetype, GIF search terms)
- Jev scores the roast against the real PR state
- User receives draft + GIF and decides whether to post
Evidence
Early versions of RoastMyPR had the same model write the joke and decide if the joke was good enough to post.
That felt clever for about a day.
Inconsistent. Same PR, two runs, two moods.
Talked into things. If the prompt got a little eager, the model started writing multi-issue checklists.
Made stuff up. Invented a file that was not in the diff.
Claude writes a short roast JSON (punchline, archetype, GIF search terms).
Jev scores that roast against the real PR state.
Does this roast the code, not the person?
Pass needs high confidence (0.80 on the yes/no scores). A hard check only blocks when confidence crashes below 0.50.
- Record date: 2026-10-09T14:40:49.840Z