Skip to main content

Why A/B Testing Based on Browser LocalStorage Is Incompatible with Test Automation

Ines avatar
Written by Ines
Updated over 2 months ago

Test Automation is fundamentally incompatible with A/B testing systems and Browser LocaStorage.

This is not a technical shortcoming.

It is not a missing feature.

It is a deliberate architectural choice.

And that choice has consequences.

1. As a test automation solution, Thunders is Built as a Stateless System

A/B testing and LocalStorage share a single, non-negotiable requirement:

👉 they must keep the browser instance over time.

  • Assigning a user to variant A or B

  • Persisting that choice across sessions

  • Storing identifiers, preferences, or behavioral history

Thunders does none of this.

Every interaction is processed as a standalone moment.

No persistent identity. No session memory. No hidden continuity.

As a result:

  • A user cannot be reliably assigned to a test variant

  • Consistent exposure to “A” or “B” cannot be guaranteed

A/B testing depends on memory, Thunders is designed to operate without it.

2. A/B Testing Breaks Test automation Core Principle: Universality

A/B testing fragments reality: Two users. Two different truths. Two silent manipulations.

Optimization happens behind the curtain, driven by probabilistic experiments rather than explicit logic.

Test automation tools like Thunders takes the opposite stance: the same input must always produce the same output.

Introducing A/B testing would inject controlled randomness into a system that explicitly rejects it.

It would be like asking a compass to point north differently depending on who’s holding it.

3. Browser LocalStorage Conflicts with the Trust Model of Test Automation

Browser LocalStorage is not a neutral mechanism in automated testing.

It introduces hidden persistence into systems that are supposed to be predictable.

LocalStorage creates an implicit contract: state is retained, behavior depends on past executions, and outcomes are no longer fully explainable from the test itself.

Even when used for “simple” feature flags, LocalStorage implies:

  • Invisible state carried across runs

  • Strong coupling to a browser instance

  • Non-obvious test dependencies

  • Increased debugging and cleanup complexity

Test automation draws a hard line: tests must control all relevant state explicitly.

Anything stored client-side and implicitly reused violates that rule.

This clarity simplifies test design, execution, and diagnosis but makes test automation fundamentally incompatible with systems that rely on hidden browser persistence to function.

4. Why “Making Them Coexist” Is a Bad Idea

Yes, workarounds exist:

  • Server-side storage

  • Temporary identifiers

  • Cleverly disguised fingerprinting

That would mean:

  • Adding complexity

  • Reducing transparency

  • Increasing legal and technical risk

  • Betraying Thunders’ original intent

Thunders as much as all test automation systems, is designed to operate cleanly, predictably, and without hidden dependencies.

Conclusion

Test automation cannot coexist with A/B testing systems that rely on browser LocalStorage because they undermine determinism, isolation, and trust.

Automation requires explicit state and predictable behavior.

LocalStorage introduces hidden memory and inconsistent outcomes.

Some techniques no longer fit.

What replaces them is better: reliable tests, stable pipelines, and results you can actually believe in.

Progress in testing comes from clarity not from preserving outdated assumptions.

Did this answer your question?