Sync Anthropic Client Blocked My asyncio Loop: Voice p95 Hit 11.4s

Описание и применение

Синхронный клиент Anthropic, вызываемый внутри async‑обработчика, блокировал цикл asyncio, увеличивая задержку с 0,9 с до 11,4 с при восемь одновременных сессий. Замена на AsyncAnthropic и перенос CPU‑тяжёлых задач в отдельный процесс снизили p95 до 0,9 с.

Задача

A sync Anthropic client called inside an async handler blocked the asyncio event loop, causing latency to grow from 0.9 s with one user to 11.4 s with eight concurrent voice interview sessions.

Результат

Switching to the async Anthropic client and moving CPU‑bound work off the event loop reduced p95 latency from 11.4 s to 0.9 s across eight concurrent sessions.

Рабочий процесс

• 1. Enable asyncio debug: loop.set_debug(True); loop.slow_callback_duration=0.1 to log slow callbacks.
• 2. Add a lightweight loop‑lag monitor that sleeps 100 ms and logs when wake‑up is delayed.
• 3. Identify blocking calls: sync Anthropic client, pypdf resume parsing, sync requests.post webhook, and time.sleep in retry helper.
• 4. Replace sync Anthropic client with AsyncAnthropic and use async for over the stream.
• 5. Wrap remaining sync calls in asyncio.to_thread only if thread pool capacity is sufficient; otherwise move CPU‑bound work to ProcessPoolExecutor.
• 6. Verify latency under load with a replayed load test measuring TTFA.
• 7. Deploy changes and monitor loop lag warnings to ensure no regressions.

Доказательства

Calling the sync Anthropic() client inside an async def handler blocks the asyncio event loop for the whole generation. No other coroutine runs until it finishes: no WebSocket frames, no audio, nothing.

Источник

With one user, the voice AI answered in 0.9 seconds. With eight users, the last person in line waited 11.4 seconds, and it sounded broken.

Источник

Fix: switch to AsyncAnthropic and async for over the stream. Wrapping the sync call in asyncio.to_thread only helps until the default thread pool fills (6 threads on a 2-vCPU box).

Источник

The longest single block was 7.9 seconds: pypdf grinding through a 14-page resume while every live interview on the box sat silent.

Источник