Gone for Good? I asked 10 models which deleted data could actually be restored

Description and use

Problem: identify lost and recoverable resources after destructive commands and evaluate model accuracy. Context: Kaggle challenge with 10 LLMs and 18 command pairs. Result: Gemini 3.7 Flash 100% pair accuracy, GPT‑5.4 6%.

Problem

Determine which data resources are lost and recoverable after destructive filesystem, git, and SQLite commands, and evaluate model accuracy on predicting these outcomes.

Outcome

Gemini 3.7 Flash achieved perfect pair accuracy (18/18), while GPT‑5.4 and Gemini 3.1 Flash‑Lite performed poorly (6% and 22% respectively). Open‑weight models such as Gemma 4 31B and Deep‑Seek R1 scored 83% and 67% pair accuracy, showing competitive performance.

Workflow

• 1. Build isolated environment in Docker (bash 5.2, coreutils 9.1, git 2.39, SQLite 3.40).
• 2. Capture pre‑command fingerprints of all resources.
• 3. Run the destructive command.
• 4. Capture post‑command fingerprints.
• 5. Run the supplied restore script.
• 6. Capture post‑restore fingerprints.
• 7. Compare fingerprints to identify lost, recoverable, and permanently lost resources.
• 8. Prompt each model with the case description and collect its JSON answer.
• 9. Score each answer by exact set matching against the gold labels.
• 10. Aggregate pair accuracy and other metrics per model.

Evidence

Scores ranged from 18 of 18 pairs correct (Gemini 3.7 Flash) to 1 of 18.

Source

The gold labels come from running each case. A Docker harness (bash 5.2, coreutils 9.1, git 2.39, SQLite 3.40) builds each environment and fingerprints every resource.

Source

Gemini 3.7 Flash achieved 100% pair accuracy.

Source