Why "works on my machine" often means "I'm still in variant B" — QA rules for testing behind A/B experiments and feature flags

说明与用途

文章讨论了使用 A/B 实验的 QA 团队中的“variant amnesia”问题，提出了四条规则并介绍了 TestPerch Chrome 扩展来自动化实验参与管理。

问题

Variant amnesia causes false bugs, irreproducible reports, and cleanup archaeology when QA testers remain enrolled in A/B experiments without realizing it.

结果

Variant amnesia is reduced, leading to fewer false bugs, more reproducible reports, and less cleanup effort.

工作流

• Use a disposable alias instead of the main account.
• Record the enrolled variant in the bug report’s reproduction steps.
• Exit the testing group as part of the test cycle’s exit criteria.
• When encountering a strange behavior, first check the current enrollment state.
• Use TestPerch to join, switch, and leave variants per identity without opening experiment pages.

证据

If your product ships behind A/B experiments and feature flags, your QA team has a failure mode nobody tracks: variant amnesia.

来源

Test on aliases, never on your main account.

来源

Treat enrollment as part of the repro steps.

来源

Leave tests when you're done.

来源

Check enrollment first, debug second.

来源

I built TestPerch — a free, MIT-licensed Chrome extension for Amplitude that turns the rules into clicks.

来源