The Same 11 Prompts, Three Copies, 48 Hours: A Free Model Regression Battery

说明与用途

对免费 LLM 进行 48 小时回归测试，发现 11 个提示的输出不稳定，从而实现自动验证并识别需要人工审查的提示。

问题

Ensuring consistent output from a free LLM over long runs for extraction pipelines, where output drift and serialization changes break parsers.

结果

The experiment produced a clear stability metric per prompt, revealing that short prompts can drift and long prompts are consistently unstable. The battery enabled automated filtering of unreliable prompts and informed decisions on where to apply human review.

工作流

• Freeze a list of prompts and hash them for traceability.
• Run the battery: for each prompt, call the model three times per hour, record raw text, length, timestamps, and hashes.
• Store each run as JSONL, avoiding pretty‑printed output during execution.
• Compute diversity ratio per prompt to flag unstable outputs.
• Log any prompt changes with timestamps to preserve experiment integrity.
• Drop prompts that consistently score 1.0 (high diversity).
• Use simple assertions (substring, key presence, length window) for quick validation.
• Review logs after the run to decide which prompts need automation or human review.

证据

With three copies per prompt, a perfect repeat scores 0.33 and a fully unstable output scores 1.0.

来源

I froze a battery of 11 prompts, ranging from single-key JSON requests to multi-paragraph summaries.

来源

I used MonkeyCode's free model access for the calls and hosted the loop on its free server option.

来源