Generative Simulation Benchmarking for autonomous urban air mobility routing under real-time policy constraints

Description and use

Static benchmarks for autonomous urban air mobility fail to capture dynamic policy events, leading to misleading success rates. A generative stack with wind and policy event models enables realistic evaluation.

Problem

Static benchmarks for UAM routing fail to capture dynamic policy events, leading to misleading success rates.

Outcome

Incorporating a generative policy layer revealed failure modes that static benchmarks missed, reducing the apparent success rate from 98% to 61% and highlighting the need for dynamic policy modeling.

Workflow

• 1. Define the routing problem with physical, policy, and temporal constraints.
• 2. Build a three‑layer architecture: world model, policy process, agent evaluation.
• 3. Implement a VAR(1) wind field generator for fast, reproducible scenarios.
• 4. Implement a Hawkes‑style policy event process to generate dynamic TFRs, curfews, and altitude caps.
• 5. Run agentic harness to roll out episodes, score them, and adapt scenario difficulty.
• 6. Compare success rates before and after incorporating the generative policy layer.

Evidence

Routes that scored 98% success in static evaluation dropped to 61% — not because the agent was bad, but because my benchmark itself was blind to the distribution of policy events.

Source

I modeled TFRs as a spatiotemporal point process conditioned on city context (events, weather, time of day), my benchmark started surfacing failure modes I'd never have hand-authored.

Source

For the physical layer, I used a conditional diffusion model trained on historical wind and traffic traces to produce plausible futures. But for the benchmark’s scoring loop, I found a lighter-weight approach more practical: a vector-autoregressive process with learned residual noise.

Source