A Free AI Server Is a Job Candidate. Interview It Like One.

Description and use

A five‑task test harness measures response time, HTTP status, output size, and token usage of a free AI coding server, uncovering cold‑start delays, silent truncation, and single‑threaded concurrency issues.

Problem

Free AI coding servers often fail under real-world conditions such as cold starts, silent truncation, and concurrency limits, leading to unreliable performance and token exhaustion.

Outcome

The test revealed that the free server suffers from long cold starts, silent truncation of large inputs, and single‑threaded concurrency, leading to unreliable performance and potential token exhaustion for real workloads.

Workflow

• Create a directory of five markdown task files (01‑refactor.md, 02‑debug.md, 03‑migrate.md, 04‑explain.md, 05‑feature.md).
• Write a Bash script (interview.sh) that loops over the task files, POSTs each to the server endpoint, captures HTTP status, total time, and word count, and writes results to a timestamped results directory.
• Run the script once to establish baseline performance.
• Run the script again after ten idle minutes to test cold start behavior.
• Run two instances of the script concurrently to test concurrency limits.
• Collect and analyze the output files to identify 200‑status responses with empty bodies, timeouts, or 429 errors.
• Optionally, use a token counter to estimate monthly token consumption based on the tasks’ token usage.
• Decide whether the free tier meets your team’s reliability and cost requirements.

Evidence

Run the test once. Then run it again after ten idle minutes. Then run two copies at the same time. That's where free servers confess.

Source

A passing run looks boring: --- 01-refactor --- status=200 time=8.41s words=214 --- 02-debug --- status=200 time=12.02s words=98

Source