The Repo Is Not the Service: A Portability FAQ for Free AI Coding Tiers

Description and use

Developers cannot reliably compare free hosted AI coding tiers with self‑hosted clones because service‑side components differ, leading to inconsistent patch quality and timing. The parity_probe.py script records trials to quantify these differences.

Problem

Developers cannot reliably compare the performance of free hosted AI coding tiers with self‑hosted clones because the service‑side components (model endpoint, sandbox image, queue, meter) are not part of the repository, leading to inconsistent patch quality and timing.

Outcome

The probe demonstrates that the same code can yield different patches and runtimes depending on whether the service is hosted or self‑hosted, underscoring the need to measure availability claims directly.

Workflow

• Clone the repository into a throwaway directory
• Set environment variables WORK, PROBE_CMD, TRIALS, LEDGER, TIMEOUT
• Run the provided parity_probe.py script
• Collect the ledger.jsonl output for analysis

Evidence

A developer on a small platform team pulled a clean patch out of a free hosted coding tier and used it to open a demo PR. By Thursday, the same task against a self-hosted clone of the same open-source project produced a patch that imported a package nobody could install.

Source

The script below executes any command you hand it three times inside a throwaway clone, resets the working tree between trials, and appends one JSON line per trial with wall-clock time, exit code, touched files, and a hash of the produced patch.

Source