How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

Description and use

A sandboxed evaluation infrastructure using Proxmox, systemd, a macOS relay, and Tailscale prevents eval cheating, API key leaks, and runaway spend when benchmarking autonomous AI agents.

Problem

Benchmarking autonomous AI coding agents risks eval cheating, API key leaks, and network isolation issues.

Outcome

The sandboxed setup prevents eval cheating, API key leakage, and runaway spend while maintaining network isolation and reproducibility.

Workflow

• Build verified .deb packages and experiment payloads into iso9660 CD‑ROMs (≈0.8s).
• Clone a stopped Debian 13 golden base VM and mount the ISO as read‑only CD‑ROMs.
• Start the sandbox VM with 0 NICs; use QEMU Guest Agent over virtio‑serial to signal readiness.
• Run the agent (labexp) under systemd with ProtectSystem=strict and ReadWritePaths for target .py files.
• After agent exits, run the grader (labjudge) under systemd with InaccessiblePaths to /evaluation.
• Collect evidence files and SHA‑256 manifests on the macOS controller.
• Verify evidence hashes and enforce API call, token, and budget limits via the macOS relay.

Evidence

Network‑Isolated VMs (0 NICs) + Inspected Relay Proxy: Experiment VMs have zero network interfaces. The sole external data path is structured JSON passed over QEMU Guest Agent (virtio‑serial) to a macOS relay that loads the API key from the OS Keychain into memory once and enforces strict call, token, and USD budget caps (zero API keys inside the VM).

Source

OS‑Level Separation via Linux UIDs and systemd (ProtectSystem=strict / InaccessiblePaths): While the agent (labexp) runs, ProtectSystem=strict mounts the filesystem read‑only (except for pre‑created target .py files bound via ReadWritePaths), and InaccessiblePaths blocks access to the grading suite (evaluation/).

Source

0.8s Cached ISO Generation + Disposable Full‑Clone VMs: Pre‑verified .deb packages (verified via InRelease ➔ Packages SHA‑256 chains) and experiment payloads are burned into read‑only iso9660 CD‑ROM images in 0.8 seconds, mounted onto a fresh full clone of a stopped Debian 13 base VM, and executed end‑to‑end in 60–90 seconds per experiment.

Source