Generative Simulation Benchmarking for planetary geology survey missions for extreme data sparsity scenarios

Description and use

The challenge of evaluating planetary geology models with sparse data is addressed by a generative simulation pipeline that produces unlimited labeled synthetic worlds and assesses model robustness across a designed parameter grid.

Problem

Evaluating machine learning models for planetary geology with extremely sparse, biased, and geographically narrow data leads to over‑optimistic benchmarks that fail in new terrains.

Outcome

The generative benchmarking pipeline revealed that the model’s high benchmark accuracy was due to memorization of a narrow geological province. By evaluating across a designed grid of plausible terrains, the model’s true generalization performance was exposed, guiding targeted improvements.

Workflow

• Build a conditional generative terrain model that can be steered along geological parameters.
• Generate synthetic scenes and construct ground truth labels by design.
• Sample a stratified grid of geological conditions and evaluate the perception model on each synthetic world.
• Report robustness profiles across the parameter space to identify failure modes.
• Iterate on model training using insights from the generative benchmark.

Evidence

When I ran my benchmark, my model reported a stellar 91% accuracy. I was thrilled — until I deployed it in a fresh simulation and watched it confidently misclassify nearly every rock formation.

Source

The key insight from my experimentation: because the generator knows the ground truth by construction, you get unlimited labeled data from distributions your model has never seen.

Source