Generative Simulation Benchmarking for planetary geology survey missions for extreme data sparsity scenarios

Description and use

Generative simulation enables realistic benchmarking of planetary geology models in extreme data sparsity scenarios by providing unlimited, labeled synthetic terrains.

Problem

Evaluating machine learning models for planetary geology with extremely sparse, biased, and geographically narrow data leads to over‑optimistic benchmarks that fail in new terrains.

Outcome

Using generative benchmarks provides unlimited labeled data from distributions unseen by the model, enabling a realistic assessment of generalization in extreme data sparsity scenarios.

Workflow

• Build a generative terrain model conditioned on geological parameters
• Generate synthetic scenes with known ground truth
• Sample a stratified grid of geological conditions
• Evaluate the perception model across the grid and report robustness profiles

Evidence

The benchmark had leaked information. My validation set shared the same simulated geological province as my training set, and my model had simply memorized the distribution of that one region.

Source

Because the generator knows the ground truth by construction, you get unlimited labeled data from distributions your model has never seen.

Source