Adopting ai-engineering-from-scratch as an Internal Harness: Where It Breaks First

Description and use

Pinning the curriculum and interpreter, using catalog diffs, vendoring skills, and routing model calls through a failover gateway eliminated import errors, missing lessons, quiz bias, dataset path mismatches, and provider rate‑limit hangs, enabling reliable lesson execution in the production pipeline.

Problem

Integrating a large, continuously evolving AI curriculum into an existing production pipeline caused operational failures such as import errors, missing lessons, quiz bias, dataset path mismatches, and model‑call timeouts.

Outcome

Pinning the curriculum and interpreter, using catalog diffs, vendoring skills, and routing model calls through a failover gateway eliminated import errors, missing lessons, quiz bias, dataset path mismatches, and provider rate‑limit hangs, enabling reliable lesson execution in the production pipeline.

Workflow

• Pin curriculum to a specific commit SHA and cache it locally.
• Run audit_lessons.py to enforce invariants and build_catalog.py to generate catalog.json.
• Diff catalog.json between pins to detect removed or renamed lessons.
• Pin the Python interpreter to 3.12 in the bootstrap script.
• Vendor skills with install_skills.py to a controlled path.
• Run lesson_run.py early for smoke tests, especially for data‑fetching lessons.
• Route model calls through a failover gateway with timeout and retry settings.
• Monitor TTFT and adjust gateway configuration as needed.