What the DOE GEMS Prize metric actually rewards (and a free local scorer)

Description and use

The DOE GEMS metric rewards recall over precision and provides a closed‑form threshold. Properly handling known fault masking and extending fault tips improves competition scores.

Problem

Determine how to optimize submissions for the DOE GEMS Prize by understanding the metric and adjusting model thresholds and handling of known faults.

Outcome

The local scorer and baseline reveal that maximizing recall, using a closed‑form threshold, and properly handling known faults can significantly improve competition scores.

Workflow

• Read the metric formula TP/(TP+0.2FP+0.8FN).
• Derive that recall is worth four times precision.
• Use Dinkelbach’s trick to compute a closed‑form threshold: mark pixels with fault probability >0.2×lambda for target score lambda.
• Adjust the threshold (e.g., at lambda=0.4 use 0.08 instead of 0.5).
• Handle known faults: suppress halo false positives and optionally extend fault tips beyond mapped tips.
• Implement the scorer and baseline in Python (MIT licensed).
• Run offline tests (27) and prepare GeoTIFF submission checker.
• Submit to DrivenData and monitor leaderboard for extension length effects.

Evidence

Recall is worth four times precision.

Source

If your model can reach a score of lambda, a pixel is worth marking when its fault probability exceeds 0.2 x lambda.

Source

The organisers confirmed on the forum that USGS/INGENIOUS fault pixels are excluded from scoring, but the pixels next to them are scored normally against the new faults.

Source