Why "works on my machine" often means "I'm still in variant B" — QA rules for testing behind A/B experiments and feature flags

Description and use

The article discusses variant amnesia in QA teams using A/B experiments, presents four rules to mitigate it, and introduces TestPerch, a Chrome extension that automates enrollment management.

Problem

Variant amnesia causes false bugs, irreproducible reports, and cleanup archaeology when QA testers remain enrolled in A/B experiments without realizing it.

Outcome

Variant amnesia is reduced, leading to fewer false bugs, more reproducible reports, and less cleanup effort.

Workflow

• Use a disposable alias instead of the main account.
• Record the enrolled variant in the bug report’s reproduction steps.
• Exit the testing group as part of the test cycle’s exit criteria.
• When encountering a strange behavior, first check the current enrollment state.
• Use TestPerch to join, switch, and leave variants per identity without opening experiment pages.

Evidence

If your product ships behind A/B experiments and feature flags, your QA team has a failure mode nobody tracks: variant amnesia.

Source

Test on aliases, never on your main account.

Source

Treat enrollment as part of the repro steps.

Source

Leave tests when you're done.

Source

Check enrollment first, debug second.

Source

I built TestPerch — a free, MIT-licensed Chrome extension for Amplitude that turns the rules into clicks.

Source