What the DOE GEMS Prize metric actually rewards (and a free local scorer)

Описание и применение

Метрика DOE GEMS награждает более высокий рекалл, чем точность, и позволяет вычислить пороговое значение для предсказаний. Учитывая точное маскирование известных разломов, можно улучшить результаты соревнования.

Задача

Determine how to optimize submissions for the DOE GEMS Prize by understanding the metric and adjusting model thresholds and handling of known faults.

Результат

The local scorer and baseline reveal that maximizing recall, using a closed‑form threshold, and properly handling known faults can significantly improve competition scores.

Рабочий процесс

• Read the metric formula TP/(TP+0.2FP+0.8FN).
• Derive that recall is worth four times precision.
• Use Dinkelbach’s trick to compute a closed‑form threshold: mark pixels with fault probability >0.2×lambda for target score lambda.
• Adjust the threshold (e.g., at lambda=0.4 use 0.08 instead of 0.5).
• Handle known faults: suppress halo false positives and optionally extend fault tips beyond mapped tips.
• Implement the scorer and baseline in Python (MIT licensed).
• Run offline tests (27) and prepare GeoTIFF submission checker.
• Submit to DrivenData and monitor leaderboard for extension length effects.

Доказательства

Recall is worth four times precision.

Источник

If your model can reach a score of lambda, a pixel is worth marking when its fault probability exceeds 0.2 x lambda.

Источник

The organisers confirmed on the forum that USGS/INGENIOUS fault pixels are excluded from scoring, but the pixels next to them are scored normally against the new faults.

Источник