Generative Simulation Benchmarking for planetary geology survey missions for extreme data sparsity scenarios

Описание и применение

Проблема оценки моделей машинного обучения для геологии планет из-за крайне ограниченных данных решена генеративным симуляционным подходом, создающим бесконечное количество синтетических миров и проверяющим модели на всесторонней устойчивости.

Задача

Evaluating machine learning models for planetary geology with extremely sparse, biased, and geographically narrow data leads to over‑optimistic benchmarks that fail in new terrains.

Результат

The generative benchmarking pipeline revealed that the model’s high benchmark accuracy was due to memorization of a narrow geological province. By evaluating across a designed grid of plausible terrains, the model’s true generalization performance was exposed, guiding targeted improvements.

Рабочий процесс

• Build a conditional generative terrain model that can be steered along geological parameters.
• Generate synthetic scenes and construct ground truth labels by design.
• Sample a stratified grid of geological conditions and evaluate the perception model on each synthetic world.
• Report robustness profiles across the parameter space to identify failure modes.
• Iterate on model training using insights from the generative benchmark.

Доказательства

When I ran my benchmark, my model reported a stellar 91% accuracy. I was thrilled — until I deployed it in a fresh simulation and watched it confidently misclassify nearly every rock formation.

Источник

The key insight from my experimentation: because the generator knows the ground truth by construction, you get unlimited labeled data from distributions your model has never seen.

Источник