Sync Anthropic Client Blocked My asyncio Loop: Voice p95 Hit 11.4s

说明与用途

在 async 处理器中调用同步 Anthropic 客户端会阻塞 asyncio 事件循环，使延迟从 0.9 秒升至 11.4 秒。改用 AsyncAnthropic 并将 CPU 密集型任务移到单独进程后，p95 延迟降至 0.9 秒。

问题

A sync Anthropic client called inside an async handler blocked the asyncio event loop, causing latency to grow from 0.9 s with one user to 11.4 s with eight concurrent voice interview sessions.

结果

Switching to the async Anthropic client and moving CPU‑bound work off the event loop reduced p95 latency from 11.4 s to 0.9 s across eight concurrent sessions.

工作流

• 1. Enable asyncio debug: loop.set_debug(True); loop.slow_callback_duration=0.1 to log slow callbacks.
• 2. Add a lightweight loop‑lag monitor that sleeps 100 ms and logs when wake‑up is delayed.
• 3. Identify blocking calls: sync Anthropic client, pypdf resume parsing, sync requests.post webhook, and time.sleep in retry helper.
• 4. Replace sync Anthropic client with AsyncAnthropic and use async for over the stream.
• 5. Wrap remaining sync calls in asyncio.to_thread only if thread pool capacity is sufficient; otherwise move CPU‑bound work to ProcessPoolExecutor.
• 6. Verify latency under load with a replayed load test measuring TTFA.
• 7. Deploy changes and monitor loop lag warnings to ensure no regressions.

证据

Calling the sync Anthropic() client inside an async def handler blocks the asyncio event loop for the whole generation. No other coroutine runs until it finishes: no WebSocket frames, no audio, nothing.

来源

With one user, the voice AI answered in 0.9 seconds. With eight users, the last person in line waited 11.4 seconds, and it sounded broken.

来源

Fix: switch to AsyncAnthropic and async for over the stream. Wrapping the sync call in asyncio.to_thread only helps until the default thread pool fills (6 threads on a 2-vCPU box).

来源

The longest single block was 7.9 seconds: pypdf grinding through a 14-page resume while every live interview on the box sat silent.

来源