Ten files is not a budget

Описание и применение

gopher обновил алгоритм выбора файлов, ограничив общий размер вывода до 40 000 символов, добавив лимиты для дерева каталогов и отдельных файлов, а также улучшив ранжирование по глубине и важности. Это устраняет ошибки превышения токенов и пропуск важных файлов.

Задача

The MCP server gopher produced an error when digesting large GitHub repositories because the fixed limits on number of files and per-file size caused the output to exceed the token budget or miss important files.

Результат

The updated gopher digest algorithm consistently produces outputs under 40,000 characters, prioritizes core project files, and avoids large irrelevant files, improving reliability and usefulness for Claude.

Рабочий процесс

• 1. Measure repository size and content distribution across six real repos.
• 2. Add a total character budget and allocate it first to the directory tree.
• 3. Cap the tree to 25% of the budget and note omitted entries.
• 4. Allocate remaining budget to files in priority order, applying a per-file ceiling of 40% of remaining budget.
• 5. Refine ranking: decay priority bonus with directory depth, give extra points to root manifests and primary language files.
• 6. Detect GitHub API responses with "encoding: none" and refetch via blobs endpoint for large files.
• 7. Test the updated logic against the six repositories and verify output sizes and file selection.

Доказательства

A fifty-fold range between the smallest and largest output, from the same tool with the same settings.

Источник

The tree needed its own cap, at a fraction of the total, and a note saying how many entries it left out.

Источник

The priority bonus now decays with directory depth, so the root manifest beats a vendored one.

Источник