Four Myths About Free Model Endpoints, and the Four-Check Probe That Broke Them

Description and use

The author discovered that a free model endpoint and free server hide changes affecting output quality. A probe was built to log model, status, and environment, revealing mismatches.

Problem

A digest job that calls a free model endpoint on a free server experienced silent degradation of summaries without crashes, revealing hidden assumptions about the endpoint and environment.

Outcome

The probe identified that the free endpoint was not a single fixed model and that the free server’s environment differed from CI, causing silent failures. Logging and fingerprinting enabled detection and mitigation of these issues.

Workflow

• 1. Write a probe that sends requests to the model endpoint and logs response metadata to a SQLite database.
• 2. Include the expected model name in the request payload and capture the returned model name.
• 3. Store HTTP status, model, finish reason, and latency for each call.
• 4. Query the log to verify that only one model is returned and that finish reasons are consistent.
• 5. Capture environment fingerprints (OS, Python version, CPU flags, timezone) on the free server and CI environments.
• 6. Compare fingerprints to detect discrepancies that may affect execution.
• 7. Use the probe results to adjust assumptions and improve reliability.

Evidence

I run a small digest job. It summarizes a handful of repos every morning. It calls a free model endpoint and runs on a free server. Last month it broke twice.

Source

So log the echo. Every call, every time.

Source

A script passed on the free server. It failed in CI forty minutes later. Nothing about the code changed.

Source