The Same 11 Prompts, Three Copies, 48 Hours: A Free Model Regression Battery

Description and use

A 48‑hour regression test on a free LLM revealed output instability across 11 prompts, enabling automated validation and identification of prompts needing human review.

Problem

Ensuring consistent output from a free LLM over long runs for extraction pipelines, where output drift and serialization changes break parsers.

Outcome

The experiment produced a clear stability metric per prompt, revealing that short prompts can drift and long prompts are consistently unstable. The battery enabled automated filtering of unreliable prompts and informed decisions on where to apply human review.

Workflow

• Freeze a list of prompts and hash them for traceability.
• Run the battery: for each prompt, call the model three times per hour, record raw text, length, timestamps, and hashes.
• Store each run as JSONL, avoiding pretty‑printed output during execution.
• Compute diversity ratio per prompt to flag unstable outputs.
• Log any prompt changes with timestamps to preserve experiment integrity.
• Drop prompts that consistently score 1.0 (high diversity).
• Use simple assertions (substring, key presence, length window) for quick validation.
• Review logs after the run to decide which prompts need automation or human review.

Evidence

With three copies per prompt, a perfect repeat scores 0.33 and a fully unstable output scores 1.0.

Source

I froze a battery of 11 prompts, ranging from single-key JSON requests to multi-paragraph summaries.

Source

I used MonkeyCode's free model access for the calls and hosted the loop on its free server option.

Source