I Built a JIT Compiler for AI Agents: How We Turned 30s LLM Chains into 0.1ms Deterministic Python

说明与用途

AgentJIT 将 AI 代理轨迹编译为快速、确定性的 Python AST 管道，将延迟降至 <0.1 ms 并消除令牌成本。

问题

AI agents in production suffer from high latency, costly API usage, non‑determinism, and redundant inference when executing multi‑step tool chains.

结果

A 356× speedup (37.21 ms → 0.1044 ms), 100 % deterministic execution, zero token cost, and automatic speculative de‑optimization for safety.

工作流

• 1. Decorate tools with @trace_tool and the agent with @jit.
• 2. First run triggers Tracer to record tool calls, inputs, outputs, and timings, building a DAG.
• 3. DAG Flow Analyzer resolves dependencies and arithmetic expressions.
• 4. AST Code Generator synthesizes pure Python AST with runtime guards.
• 5. Compiled JIT Pipeline runs subsequent invocations in <0.1 ms; guard failures deopt back to the LLM agent.

证据

We ran a 100‑iteration benchmark in Google Colab simulating an uncompiled multi‑step LLM chain versus the compiled AgentJIT pipeline: Mean Latency 37.21 ms vs 0.1044 ms (356.4× faster).

来源

Over 90 % of recurring invocations execute the exact same sequence of tool calls, differing only by input parameters.

来源