Gone for Good? I asked 10 models which deleted data could actually be restored

Описание и применение

Проблема: определить, какие ресурсы теряются и восстанавливаются после разрушительных команд, и оценить точность моделей. Контекст: Kaggle‑челлендж с 10 LLM и 18 парами команд. Результат: Gemini 3.7 Flash 100 % точности, GPT‑5.4 6 %.

Задача

Determine which data resources are lost and recoverable after destructive filesystem, git, and SQLite commands, and evaluate model accuracy on predicting these outcomes.

Результат

Gemini 3.7 Flash achieved perfect pair accuracy (18/18), while GPT‑5.4 and Gemini 3.1 Flash‑Lite performed poorly (6% and 22% respectively). Open‑weight models such as Gemma 4 31B and Deep‑Seek R1 scored 83% and 67% pair accuracy, showing competitive performance.

Рабочий процесс

• 1. Build isolated environment in Docker (bash 5.2, coreutils 9.1, git 2.39, SQLite 3.40).
• 2. Capture pre‑command fingerprints of all resources.
• 3. Run the destructive command.
• 4. Capture post‑command fingerprints.
• 5. Run the supplied restore script.
• 6. Capture post‑restore fingerprints.
• 7. Compare fingerprints to identify lost, recoverable, and permanently lost resources.
• 8. Prompt each model with the case description and collect its JSON answer.
• 9. Score each answer by exact set matching against the gold labels.
• 10. Aggregate pair accuracy and other metrics per model.

Доказательства

Scores ranged from 18 of 18 pairs correct (Gemini 3.7 Flash) to 1 of 18.

Источник

The gold labels come from running each case. A Docker harness (bash 5.2, coreutils 9.1, git 2.39, SQLite 3.40) builds each environment and fingerprints every resource.

Источник

Gemini 3.7 Flash achieved 100% pair accuracy.

Источник