Case Study: A Doc-Drift Ledger That Only Sends Changed Symbols to a Model

Description and use

An offline Python tool that hashes public symbols’ signatures and docstrings, compares ledgers, and lists symbols whose signatures changed without docstring updates, optionally annotating them with a model.

Problem

Detect when a public symbol’s signature changes without its docstring updating, so that documentation drift is caught cheaply and locally.

Outcome

The tool identifies exactly the symbols that have signature drift without docstring updates, without any network calls, and produces a deterministic, reviewable list that can be integrated into CI.

Workflow

• Collect signatures and docstrings from all .py files and hash them into a ledger.
• Commit the ledger JSON as a lockfile next to the code.
• Run the diff tool to compare the committed ledger with a new one.
• Filter candidates where signature hash changed but docstring hash stayed the same.
• Optionally query a language model for each candidate and cache the response.
• Use the list to review or annotate documentation drift.
• Fail CI with a non‑zero exit code if any candidates remain.

Evidence

The fix I built for a small internal library is a ledger, not a chatbot: it hashes every public symbol's signature and docstring, then asks a model about the handful of symbols whose signature moved while the prose did not.

Source

The library in question exports roughly forty functions across a dozen modules, and three of them had quietly changed their parameter names during a cleanup.

Source

The goal was one command that answers a single question: which exported symbols changed shape without their documentation following along?

Source