The Repo Is Not the Service: A Portability FAQ for Free AI Coding Tiers

Описание и применение

Разработчики не могут сравнивать бесплатные хостинг‑тарифы и самоустанавливаемые версии AI‑кодеров из‑за разницы в сервисных компонентах, что приводит к разной качеству патчей и времени выполнения. Скрипт parity_probe.py позволяет измерять эти различия.

Задача

Developers cannot reliably compare the performance of free hosted AI coding tiers with self‑hosted clones because the service‑side components (model endpoint, sandbox image, queue, meter) are not part of the repository, leading to inconsistent patch quality and timing.

Результат

The probe demonstrates that the same code can yield different patches and runtimes depending on whether the service is hosted or self‑hosted, underscoring the need to measure availability claims directly.

Рабочий процесс

• Clone the repository into a throwaway directory
• Set environment variables WORK, PROBE_CMD, TRIALS, LEDGER, TIMEOUT
• Run the provided parity_probe.py script
• Collect the ledger.jsonl output for analysis

Доказательства

A developer on a small platform team pulled a clean patch out of a free hosted coding tier and used it to open a demo PR. By Thursday, the same task against a self-hosted clone of the same open-source project produced a patch that imported a package nobody could install.

Источник

The script below executes any command you hand it three times inside a throwaway clone, resets the working tree between trials, and appends one JSON line per trial with wall-clock time, exit code, touched files, and a hash of the produced patch.

Источник