Gone for Good? I asked 10 models which deleted data could actually be restored

说明与用途

问题：确定破坏性命令后哪些资源丢失、可恢复，并评估模型准确性。背景：Kaggle 挑战，10 个 LLM 与 18 对命令。结果：Gemini 3.7 Flash 100% 对配对准确率，GPT‑5.4 6%。

问题

Determine which data resources are lost and recoverable after destructive filesystem, git, and SQLite commands, and evaluate model accuracy on predicting these outcomes.

结果

Gemini 3.7 Flash achieved perfect pair accuracy (18/18), while GPT‑5.4 and Gemini 3.1 Flash‑Lite performed poorly (6% and 22% respectively). Open‑weight models such as Gemma 4 31B and Deep‑Seek R1 scored 83% and 67% pair accuracy, showing competitive performance.

工作流

• 1. Build isolated environment in Docker (bash 5.2, coreutils 9.1, git 2.39, SQLite 3.40).
• 2. Capture pre‑command fingerprints of all resources.
• 3. Run the destructive command.
• 4. Capture post‑command fingerprints.
• 5. Run the supplied restore script.
• 6. Capture post‑restore fingerprints.
• 7. Compare fingerprints to identify lost, recoverable, and permanently lost resources.
• 8. Prompt each model with the case description and collect its JSON answer.
• 9. Score each answer by exact set matching against the gold labels.
• 10. Aggregate pair accuracy and other metrics per model.

证据

Scores ranged from 18 of 18 pairs correct (Gemini 3.7 Flash) to 1 of 18.

来源

The gold labels come from running each case. A Docker harness (bash 5.2, coreutils 9.1, git 2.39, SQLite 3.40) builds each environment and fingerprints every resource.

来源

Gemini 3.7 Flash achieved 100% pair accuracy.

来源