Why "works on my machine" often means "I'm still in variant B" — QA rules for testing behind A/B experiments and feature flags

Описание и применение

В статье описывается проблема «variant amnesia» в QA‑командах, работающих с A/B‑экспериментами. Предложены четыре правила и инструмент TestPerch, который упрощает управление участием в экспериментах.

Задача

Variant amnesia causes false bugs, irreproducible reports, and cleanup archaeology when QA testers remain enrolled in A/B experiments without realizing it.

Результат

Variant amnesia is reduced, leading to fewer false bugs, more reproducible reports, and less cleanup effort.

Рабочий процесс

• Use a disposable alias instead of the main account.
• Record the enrolled variant in the bug report’s reproduction steps.
• Exit the testing group as part of the test cycle’s exit criteria.
• When encountering a strange behavior, first check the current enrollment state.
• Use TestPerch to join, switch, and leave variants per identity without opening experiment pages.

Доказательства

If your product ships behind A/B experiments and feature flags, your QA team has a failure mode nobody tracks: variant amnesia.

Источник

Test on aliases, never on your main account.

Источник

Treat enrollment as part of the repro steps.

Источник

Leave tests when you're done.

Источник

Check enrollment first, debug second.

Источник

I built TestPerch — a free, MIT-licensed Chrome extension for Amplitude that turns the rules into clicks.

Источник