Sync Anthropic Client Blocked My asyncio Loop: Voice p95 Hit 11.4s

Description and use

A sync Anthropic client called inside an async handler blocked the asyncio event loop, raising latency from 0.9 s to 11.4 s with eight concurrent sessions. Switching to AsyncAnthropic and moving CPU‑bound work off the loop reduced p95 to 0.9 s.

Problem

A sync Anthropic client called inside an async handler blocked the asyncio event loop, causing latency to grow from 0.9 s with one user to 11.4 s with eight concurrent voice interview sessions.

Outcome

Switching to the async Anthropic client and moving CPU‑bound work off the event loop reduced p95 latency from 11.4 s to 0.9 s across eight concurrent sessions.

Workflow

• 1. Enable asyncio debug: loop.set_debug(True); loop.slow_callback_duration=0.1 to log slow callbacks.
• 2. Add a lightweight loop‑lag monitor that sleeps 100 ms and logs when wake‑up is delayed.
• 3. Identify blocking calls: sync Anthropic client, pypdf resume parsing, sync requests.post webhook, and time.sleep in retry helper.
• 4. Replace sync Anthropic client with AsyncAnthropic and use async for over the stream.
• 5. Wrap remaining sync calls in asyncio.to_thread only if thread pool capacity is sufficient; otherwise move CPU‑bound work to ProcessPoolExecutor.
• 6. Verify latency under load with a replayed load test measuring TTFA.
• 7. Deploy changes and monitor loop lag warnings to ensure no regressions.

Evidence

Calling the sync Anthropic() client inside an async def handler blocks the asyncio event loop for the whole generation. No other coroutine runs until it finishes: no WebSocket frames, no audio, nothing.

Source

With one user, the voice AI answered in 0.9 seconds. With eight users, the last person in line waited 11.4 seconds, and it sounded broken.

Source

Fix: switch to AsyncAnthropic and async for over the stream. Wrapping the sync call in asyncio.to_thread only helps until the default thread pool fills (6 threads on a 2-vCPU box).

Source

The longest single block was 7.9 seconds: pypdf grinding through a 14-page resume while every live interview on the box sat silent.

Source