Self-Calibration via Prompt History Analysis

Description and use

The author analyzed 1,427 prompts over six weeks to objectively assess skill progression. The data revealed a significant increase in prompt complexity and experimental design capabilities, contradicting their subjective feelings of incompetence. This highlights the gap between self-perception and actual demonstrated judgment.

Problem

Developers often suffer from a disconnect between their subjective self-assessment (feeling like they are wasting time or lacking skills) and their actual demonstrated capability, leading to unnecessary anxiety and imposter syndrome.

Outcome

The analysis revealed that while the developer felt stagnant, their prompts had evolved from simple commands to complex, context-rich strategies involving verification and experimental design. This objective evidence disproved their subjective feelings of incompetence.

Workflow

• Locate local conversation history files (e.g., JSONL files in ~/.claude/projects/ for Claude Code).
• Write a script to extract and parse prompts from a specific time period.
• Categorize prompts by vocabulary and intent (e.g., adversarial/verify, meta/governance, delegation/automation).
• Measure metrics such as average prompt length and frequency of verification keywords over time.
• Compare the quantitative data of prompt evolution against subjective self-assessment.
• Use the discrepancy to recalibrate self-perception and identify actual skill growth.

Evidence

I pulled every prompt I typed myself between June 15 and July 23 — 1,427 of them... cut them into six time windows, and measured them window by window.

Source

In under six weeks, 'make the AI push back, verify this, find the root cause' grew fivefold...

Source

If you use Claude Code or a similar tool, your conversation history is sitting in jsonl files on your machine... Spend half an hour writing a script to pull your own prompts...

Source