Your API Gateway Rate Limit Can't See the Agent Looping Your Search Tool

Description and use

An agent can loop on one expensive tool, producing many valid HTTP requests that appear normal to an HTTP-layer gateway rate limiter. tool-throttle adds a per-tool token-bucket rate limiter on MCP tool calls, with per-tool quotas, glob patterns, reject or queue modes, retry_after hints, and stats() observability, letting developers budget at the tool-call semantic layer rather than the HTTP layer.

Problem

An agent loops on a single expensive tool (e.g., web_search), making many valid HTTP requests that appear normal to an HTTP-layer gateway rate limiter, so the gateway never detects the abuse and the bill exceeds budget.

Outcome

tool-throttle lets an agent developer budget per-tool call rates at the tool-call semantic layer instead of relying solely on HTTP-layer gateway limits, catching agents that loop on expensive tools while still allowing legitimate bursts through bounded queuing.

Workflow

• Install tool-throttle: pip install tool-throttle
• Create a ThrottlerConfig with default Quota and per_tool Quotas keyed by tool name or glob pattern, setting calls and per_seconds for each tool
• Set on_exceed to REJECT for fail-fast backoff or QUEUE for bounded-wait queuing
• Wrap the tool dispatcher with th.wrap(tool_name, mcp_client.call_tool)
• Call the wrapped tool; over quota raises ThrottledError with retry_after hint
• Inspect th.stats() to see allowed/rejected/queued/waited_seconds per tool
• Optionally save config as JSON via cfg.save('throttle.json') and use the CLI: tool-throttle check, tool-throttle acquire, or tool-throttle demo

Evidence

An agent gets stuck. Not stuck like a crash — stuck like a loop: it calls web_search, gets results, doesn't like them, calls web_search again with a slightly different query. 10 times. 50 times. 400 times a minute.

Source

The problem: gateway rate limits live at the HTTP layer. Agent abuse lives at the tool-call semantic layer. One HTTP request can fan out into many tool calls. One looping tool can spread its calls across many endpoints.

Source

I built the fuse that belongs at the tool-call layer.

Source

tool-throttle — a zero dependency Python package (stdlib only, Python >= 3.9) that puts a token bucket rate limiter on MCP tool calls, per tool name.

Source

Quotas live in the language of the agent — tool names and glob patterns, with per-tool bursts — not in the language of HTTP.

Source

It complements gateway rate limiting; it doesn't replace it. The gateway protects your infrastructure. tool-throttle protects your wallet from your own agent.

Source