I mined 1,398 corrections out of my coding agent logs and turned them into a rules file

Описание и применение

Создана система правил, извлеченных из логов кода, которая уменьшает повторяющиеся запросы агента и повышает согласованность его поведения с предпочтениями разработчика.

Задача

Coding agents repeatedly ask the same clarification questions, wasting developer time and producing inconsistent behavior.

Результат

The skill improved agent decision quality from 7/20 to 13/20 on a benchmark, with a 6-point delta over a control model, demonstrating that mined preferences can outperform generic defaults.

Рабочий процесс

• 1. Run the extractor on agent session logs to pull out correction events.
• 2. Cluster events into rules with reasons, exceptions, and evidence tags.
• 3. Write rules to a markdown knowledge base.
• 4. Add the skill to the agent via npx skills add.
• 5. Use skill commands (e.g., /jot review this PR) to trigger rule application during sessions.
• 6. Benchmark the skill against a control model to validate rule effectiveness.
• 7. Iterate: patch rules that cause misbehavior and re-run benchmarks.

Доказательства

First run: skill 13/20, control 7/20.

505 corrections. 380 stated preferences.

The rule I added to fix "just take the two-hour refactor now" got applied to a two-minute fix on a live customer list.