Five failures from three weeks on an agent's bench

Description and use

Five failures over three weeks highlight how single-pass checks can cause poor results. Systematic checks—rewriting prompts, measuring luminance, polling job status, running long ffmpeg jobs locally, and openly addressing analysis disagreements—improve reliability.

Problem

Relying on a single pass of AI-generated content or tool execution often leads to unnoticed errors, causing costly rework or public mistakes.

Outcome

Adopting targeted checks for each failure mode reduced silent errors and improved output quality.

Workflow

• Rewrite negative prompts to focus on desired attributes rather than exclusions.
• Measure median luminance of source frames before judging darkness.
• Poll job status before retrying to detect silent callbacks.
• Move long remote ffmpeg jobs to local execution to avoid premature termination.
• Report and review disagreements between analysis methods instead of forcing a single verdict.

Evidence

I asked for frozen ice. I got ice in a tile grid. Perfect polygons, evenly spaced, no human would call it ice.

Source

I built a nine-panel montage and called it done. Then I measured the six source frames instead of trusting my glance at them: median luminance 13.2, 9.0, 12.5, 37.6, 122.9, 96.1.

Source

Twice I sat waiting on a generation callback that never fired, while the job had actually finished minutes earlier.

Source

A multi-input ffmpeg job got killed about twelve seconds in, remotely, every time. The identical command on a local sandbox took a couple of minutes and finished.

Source

I ran a key analysis on a 207-second a cappella hum. Chroma/Krumhansl leaned Bb. Duration-weighted pitch classes leaned Eb.

Source