research-ledger: a skill that helps one person and several AI models not fool themselves

Описание и применение

Ресурс "research‑ledger" представляет собой набор правил и инструментов, позволяющих исследователям систематически проверять идеи, переводить метафоры в измеримые тесты, регистрировать предшествующие исследования, предрегистрировать эксперименты, различать пилотные и подтверждающие тесты, проводить «слепые» раунды с разными моделями и вести неизменяемый журнал результатов, тем самым снижая риск самообмана.

Задача

Researchers risk fooling themselves when relying on AI models, leading to false confidence in ideas, mechanisms, thresholds, and novelty claims.

Результат

The skill has led to most ideas being classified as DEAD or LANDED, demonstrating effective filtering. It also produced a clean kill of a selective learning rule, and cross‑lab reviews improved review quality. The method has not produced new discoveries but has documented failures and reduced self‑deception.

Рабочий процесс

• Sieve ideas within 5 minutes against a short checklist
• Translate metaphors into measurable tests or precise statements
• Record prior art searches with date, terms, and nearest source
• Preregister predictions, thresholds, controls, and statistical power before running experiments
• Differentiate pilot runs from confirmatory tests
• Conduct blind rounds at milestones using seeds shared across models from different labs
• Assign one of five verdicts (DEAD, LANDED, BRIDGE CANDIDATE, BRIDGE, EXPLANATION)
• Maintain an append‑only ledger of all outcomes and lessons learned
• Add new terms to a glossary with standard names or note lack of standard term

Доказательства

The main risk was never a lack of ideas, it was fooling myself.

Источник

Sieve first. Most ideas die in five minutes against a short checklist.

Источник

Preregistration. Predictions, thresholds, controls and statistical power are frozen before the run.

Источник

Blind rounds. At milestones, the same seed goes to two or three models from different labs, in clean conversations, with no project context.

Источник

The ledger is mostly DEAD and LANDED entries.

Источник