Four Myths About Free Model Endpoints, and the Four-Check Probe That Broke Them

说明与用途

作者发现免费模型端点和免费服务器隐藏了影响输出质量的更改。构建了一个探针来记录模型、状态和环境，揭示了不匹配。

问题

A digest job that calls a free model endpoint on a free server experienced silent degradation of summaries without crashes, revealing hidden assumptions about the endpoint and environment.

结果

The probe identified that the free endpoint was not a single fixed model and that the free server’s environment differed from CI, causing silent failures. Logging and fingerprinting enabled detection and mitigation of these issues.

工作流

• 1. Write a probe that sends requests to the model endpoint and logs response metadata to a SQLite database.
• 2. Include the expected model name in the request payload and capture the returned model name.
• 3. Store HTTP status, model, finish reason, and latency for each call.
• 4. Query the log to verify that only one model is returned and that finish reasons are consistent.
• 5. Capture environment fingerprints (OS, Python version, CPU flags, timezone) on the free server and CI environments.
• 6. Compare fingerprints to detect discrepancies that may affect execution.
• 7. Use the probe results to adjust assumptions and improve reliability.

证据

I run a small digest job. It summarizes a handful of repos every morning. It calls a free model endpoint and runs on a free server. Last month it broke twice.

来源

So log the echo. Every call, every time.

来源

A script passed on the free server. It failed in CI forty minutes later. Nothing about the code changed.

来源