I Built an Agent Skill to Stop AI From Answering the Wrong Question Beautifully

Description and use

Judgment Loop is an agent skill that prevents AI from answering the wrong question. It acts as a routing and reasoning guardrail for consequential decisions, requiring hypothesis testing and falsification checks before taking action.

Problem

AI systems produce coherent, detailed, and persuasive answers that can be misleading. The model may optimize a proxy while the underlying reality stays unchanged, or turn an attractive plan into a commitment before testing its weakest assumption.

Outcome

The skill routes between five modes (Quick, Decision, Research, Learning, Review) to ensure answers are falsifiable and actions remain under user ownership. It prevents premature commitment to uncertain claims by requiring isolated falsification tests.

Workflow

• Identify the core question and the reality supposed to change.
• Check if the chosen metric could improve while that reality remains the same.
• Classify claims as verified facts, reasonable inferences, or untested hypotheses.
• Identify the strongest reason the current judgment could fail.
• Determine the cheapest test that separates competing explanations.
• Decide what action should close now and what belief should remain open to revision.

Evidence

The model may optimize a proxy while the underlying reality stays unchanged. It may turn an attractive plan into a commitment before testing its weakest assumption.

Source

The skill routes between five modes: Quick, Decision, Research, Learning, Review.

Source

The repository is available at: https://github.com/sirsws/judgment-loop

Source