We cut the AI replies. Daily impressions fell from 447,575 to 5,686

Description and use

Automated replies on X drove impressions to 447,575, but disabling them dropped reach to 5,686 and revealed most Stripe checkouts were bot‑generated.

Problem

Relying on AI agents to reply to large threads inflated daily impressions, but stopping replies caused a sharp drop in reach and revenue.

Outcome

Stopping AI replies eliminated the primary source of reach, causing a 98.7% drop in impressions and revealing that most checkout sessions were bot‑generated. The experiment highlighted the need for accurate funnel tracking and the importance of human‑driven content for sustainable growth.

Workflow

• 1. Deploy AI agents to reply to large X threads from a personal account.
• 2. Track daily impressions and replies via X analytics.
• 3. On Oct 15, restrict agents to publishing articles only, disabling replies, likes, and follows.
• 4. Monitor Stripe checkout sessions and filter out bot‑generated tests.
• 5. Analyze reach correlation with replies and adjust strategy.
• 6. Implement safeguards: tag test requests, convert buy links to button pages, count only paid sessions, flag rapid session bursts, verify source of reach.

Evidence

Daily impressions went from 447,575 at the peak to 5,686 today

Source

Stripe logged 1,865 checkout sessions in one day, almost all of the "shoppers" were our own agents

Source

The reply was the product. Our biggest day, Oct 13, was also our biggest reply day: 125 replies

Source