Generative Simulation Benchmarking for planetary geology survey missions for extreme data sparsity scenarios

说明与用途

针对行星地质模型评估中极端数据稀疏问题，本文提出生成式仿真方法，生成无限标注合成世界，并在设计的参数网格上评估模型鲁棒性。

问题

Evaluating machine learning models for planetary geology with extremely sparse, biased, and geographically narrow data leads to over‑optimistic benchmarks that fail in new terrains.

结果

The generative benchmarking pipeline revealed that the model’s high benchmark accuracy was due to memorization of a narrow geological province. By evaluating across a designed grid of plausible terrains, the model’s true generalization performance was exposed, guiding targeted improvements.

工作流

• Build a conditional generative terrain model that can be steered along geological parameters.
• Generate synthetic scenes and construct ground truth labels by design.
• Sample a stratified grid of geological conditions and evaluate the perception model on each synthetic world.
• Report robustness profiles across the parameter space to identify failure modes.
• Iterate on model training using insights from the generative benchmark.

证据

When I ran my benchmark, my model reported a stellar 91% accuracy. I was thrilled — until I deployed it in a fresh simulation and watched it confidently misclassify nearly every rock formation.

来源

The key insight from my experimentation: because the generator knows the ground truth by construction, you get unlimited labeled data from distributions your model has never seen.

来源