research-ledger: a skill that helps one person and several AI models not fool themselves

Description and use

The "research‑ledger" resource is a set of rules and tools that help researchers systematically vet ideas, translate metaphors into measurable tests, record prior art, preregister experiments, distinguish pilots from confirmatory tests, conduct blind rounds with different models, and maintain an immutable ledger of outcomes, thereby reducing self‑deception.

Problem

Researchers risk fooling themselves when relying on AI models, leading to false confidence in ideas, mechanisms, thresholds, and novelty claims.

Outcome

The skill has led to most ideas being classified as DEAD or LANDED, demonstrating effective filtering. It also produced a clean kill of a selective learning rule, and cross‑lab reviews improved review quality. The method has not produced new discoveries but has documented failures and reduced self‑deception.

Workflow

• Sieve ideas within 5 minutes against a short checklist
• Translate metaphors into measurable tests or precise statements
• Record prior art searches with date, terms, and nearest source
• Preregister predictions, thresholds, controls, and statistical power before running experiments
• Differentiate pilot runs from confirmatory tests
• Conduct blind rounds at milestones using seeds shared across models from different labs
• Assign one of five verdicts (DEAD, LANDED, BRIDGE CANDIDATE, BRIDGE, EXPLANATION)
• Maintain an append‑only ledger of all outcomes and lessons learned
• Add new terms to a glossary with standard names or note lack of standard term

Evidence

The main risk was never a lack of ideas, it was fooling myself.

Source

Sieve first. Most ideas die in five minutes against a short checklist.

Source

Preregistration. Predictions, thresholds, controls and statistical power are frozen before the run.

Source

Blind rounds. At milestones, the same seed goes to two or three models from different labs, in clean conversations, with no project context.

Source

The ledger is mostly DEAD and LANDED entries.

Source