Our AI agent pipeline kept stalling on CAPTCHA-heavy scraping until we rebuilt it around queues and short browser sessions

Description and use

<p>I learned this one the hard way.</p> <p>We had a scraping workflow that looked solid in staging:</p> <ul> <li>Playwright handled JavaScript-heavy pages</li> <li>a few retries cleaned up transient failures</li> <li>extracted data flowed into downstream AI steps</li> </ul> <p>Then real traffic hit.</p> <p>A target site tightened anti-bot rules. CAPTCHA pages started showing up more often. A few browser workers got stuck for minutes at a time. The queue behind them kept growing. And suddenly the scraping issue wasn’t just a scraping issue anymore.</p> <p>It broke the rest of the pipeline:</p>

Problem

<p>I learned this one the hard way.</p> <p>We had a scraping workflow that looked solid in staging:</p> <ul> <li>Playwright handled JavaScript-heavy pages</li> <li>a few retries cleaned up transient failures</li> <li>extracted data flowed into downstream AI steps</li> </ul> <p>Then real traffic hit.</p> <p>A target site tightened anti-bot rules. CAPTCHA pages started showing up more often. A few browser workers got stuck for minutes at a time. The queue behind them kept growing. And suddenly the scraping issue wasn’t just a scraping issue anymore.</p> <p>It broke the rest of the pipeline:</p>

Outcome

<p>I learned this one the hard way.</p> <p>We had a scraping workflow that looked solid in staging:</p> <ul> <li>Playwright handled JavaScript-heavy pages</li> <li>a few retries cleaned up transient failures</li> <li>extracted data flowed into downstream AI steps</li> </ul> <p>Then real traffic hit.</p> <p>A target site tightened anti-bot rules. CAPTCHA pages started showing up more often. A few browser workers got stuck for minutes at a time. The queue behind them kept growing. And suddenly the scraping issue wasn’t just a scraping issue anymore.</p> <p>It broke the rest of the pipeline:</p>

Evidence

Our AI agent pipeline kept stalling on CAPTCHA-heavy scraping until we rebuilt it around queues and short browser sessions

Source