Ten files is not a budget

Description and use

gopher updated its file selection algorithm by imposing a total character budget (40,000), capping the directory tree, adding per-file ceilings, and refining ranking to favor root manifests and primary language files. This prevents token budget errors and ensures relevant files are included.

Problem

The MCP server gopher produced an error when digesting large GitHub repositories because the fixed limits on number of files and per-file size caused the output to exceed the token budget or miss important files.

Outcome

The updated gopher digest algorithm consistently produces outputs under 40,000 characters, prioritizes core project files, and avoids large irrelevant files, improving reliability and usefulness for Claude.

Workflow

• 1. Measure repository size and content distribution across six real repos.
• 2. Add a total character budget and allocate it first to the directory tree.
• 3. Cap the tree to 25% of the budget and note omitted entries.
• 4. Allocate remaining budget to files in priority order, applying a per-file ceiling of 40% of remaining budget.
• 5. Refine ranking: decay priority bonus with directory depth, give extra points to root manifests and primary language files.
• 6. Detect GitHub API responses with "encoding: none" and refetch via blobs endpoint for large files.
• 7. Test the updated logic against the six repositories and verify output sizes and file selection.

Evidence

A fifty-fold range between the smallest and largest output, from the same tool with the same settings.

Source

The tree needed its own cap, at a fraction of the total, and a note saying how many entries it left out.

Source

The priority bonus now decays with directory depth, so the root manifest beats a vendored one.

Source