Postmortem: Why Our One-Shot AST Sanitizer for LLM Code Collapsed Under Real-World Constraints

Description and use

The postmortem explains why a one-shot AST sanitizer for LLM code collapsed under real-world constraints: hidden initialization latency, computational blowup in unparse on deep ASTs, and unsafe signal‑handler operations.

Problem

A one-shot AST sanitizer designed to intercept and sanitize LLM-generated code failed under realistic workloads, causing catastrophic timeouts and CPU exhaustion.

Outcome

The sanitizer’s design caused hidden latency from initialization, computational blowup in unparse on deep ASTs, and unsafe signal handling, leading to frequent timeouts and crashes in real deployments.

Workflow

• 1. Receive JSON payload via sys.stdin.buffer
• 2. Validate schema with Pydantic v2
• 3. Parse code string with ast.parse
• 4. Transform AST with ast.NodeTransformer to replace prohibited calls
• 5. Fix missing locations with ast.fix_missing_locations
• 6. Unparse AST back to source with ast.unparse
• 7. Serialize result with orjson and write to stdout

Evidence

The timer was armed at the topmost entrypoint of the script: ... signal.alarm(TIMEOUT_SECONDS)

Source

In microbenchmarks on an idle developer machine, interpreter initialization took under 40 milliseconds. But inside CI/CD containers, heavily loaded Kubernetes worker nodes, or environments subject to cgroup CPU throttling, the initialization path accumulated massive hidden latency

Source

When ast.NodeTransformer replaces target nodes, it breaks location metadata (lineno, col_offset), requiring a full reconciliation pass via ast.fix_missing_locations() prior to code generation

Source