Where A/B testing comes from, and who it is for
A/B testing is now part of the toolkit offered to every online store and digital product. The promise is easy to grasp. Instead of guessing which version of a page works better, a site shows both to its visitors and keeps the one that converts more. That promise, however, rests on conditions that few projects meet. An A/B test only delivers a reliable and useful result with a volume of visitors and a level of revenue that most websites do not have.
A method born at the largest websites
Amazon, Google and Microsoft developed the method in the early 2000s, for services that already drew millions of visits a day. Their pages had been refined so thoroughly that any single change could only improve them slightly. In fact, most tests fail. Ronny Kohavi, who led experimentation at Microsoft for more than a decade, reports that Microsoft wins only one test in three, against one in ten at Booking, Netflix and Google Ads. These companies have nonetheless made A/B testing standard practice, because their traffic lets them measure such thin margins and their revenue turns those margins into very large sums.
A method open to every website
The method has since spread far beyond these giants, to the point of becoming an industry of its own. Google Optimize, which was free, served as the entry point until it shut down in September 2023. Conversion rate optimization platforms such as VWO, AB Tasty and Optimizely made it their core feature, sold on subscription to marketers. Their visual editors let anyone launch a test without writing code. Product analytics tools offer it too, PostHog among them, which ties each test to recordings of visitor sessions. Open-source projects like GrowthBook also let developers host their tests themselves and run them on their own data. E-commerce platforms have now followed, with Shopify building it into its mid-tier plans in June 2026. Whatever its budget, any website can therefore run an A/B test today.
The two limits of A/B testing
A reliable result needs traffic
For websites with far less traffic than the industry giants, the first limit is volume. An A/B test compares two trends, which only emerge from a large number of visits. Below that threshold, results fluctuate even when nothing on the site has changed. A page can go from one sale to four from one week to the next. Between two versions, the measured gap therefore mixes the effect of the change with these natural swings. Only accumulating data can separate the two, so a site with modest traffic has to wait months, sometimes years, before getting a dependable answer.
Testing tools nevertheless present results the same way whatever the traffic. A test on a few hundred visits appears on the same dashboard as a test on millions, as if both volumes were equally solid. This apparent equivalence unfortunately skews the results even further. The tool can put one version ahead well before enough data has come in, while its lead still owes everything to chance. Kohavi notes that an experiment run on too few people most often produces a false winner. Anyone who relies on it risks keeping a change that performs no better than what was there before.
Gains that only matter at scale
The second limit lies in what a successful test brings in. Kohavi puts the effect of a winning test on Bing at between 0.1% and 1%. On revenue in the billions, a gap that size is worth millions and justifies a whole team working on it every day. On a site recording a few hundred sales or sign-ups a month, it means a handful of extra customers. Yet a test costs roughly the same whatever the size of the site. Someone has to pay for the tool, build the variants and wait several weeks. A result can therefore be statistically significant without changing anything for the business.
That calculation depends on ambition, though. A startup planning to multiply its traffic has every reason to measure properly from day one, since tests will become meaningful as volume arrives. A project that does not aim for that kind of growth, on the other hand, has no reason to buy a tool it will never have enough traffic to use.
What to work on before running an A/B test
In short, A/B testing serves websites that already have volume and want to make the most of it. For more modest projects, other questions come before choosing the best version of a page. The first is understanding where and why visitors give up. The second is bringing in more of them.
Understand where and why visitors drop off
Counting visitors at each step of the journey, up to payment or to the first use of a SaaS product, shows where they leave. This kind of measurement does not require large volumes. Even under French rules, among the strictest in Europe, it remains exempt from consent. Watching a few people use the site then explains what stops them, since according to Jakob Nielsen, five users are enough to uncover around 85% of usability problems. Published research fills in the rest. Baymard, for example, has found that unexpected costs at checkout and forced account creation cause a significant share of abandoned carts. Fixing those issues does not require a test. None of these sources is enough on its own, but together they give a more reliable picture than a test run on too few people.
Bring in more visitors
That leaves traffic, which matters more than conversion at this scale. A site with 5,000 visits a month and a 2% conversion rate gains ten orders a month by moving to 2.2%, after more than two years of testing. If it doubles its traffic instead, it gains a hundred. It also sees the difference from the very first month. SEO, content and communication are therefore the priorities. They all depend on a clear offer, a subject covered in our article on websites that don't convert. A/B testing becomes worthwhile once that traffic is there, because a few tenths of a point then start to carry weight.
At Nualt, metrics that fit each project
Before choosing any measurement tool, we start with the business behind the project. What the site sells, to whom and at what scale determines which metrics will inform its decisions. For a site that is just starting out, that often means tracking the steps of the purchase journey and where visits come from, with visitors' consent. For a product with enough traffic, A/B testing has its place, built on an architecture that can run it server-side. In every case, the code, the data and the accounts belong to the client. They can therefore expand their tracking at the same pace as their business.
Your questions about A/B testing
Answers to the questions we hear most often on this topic.
What is an A/B test?
An A/B test shows two versions of the same page to randomly assigned visitors, then compares their conversion rates. The winning version is kept, provided the measured gap cannot be explained by chance.
How many visitors do you need for a reliable A/B test?
It depends on the starting conversion rate and the size of the gap you want to detect. For a site converting 2% of its visitors, detecting a lift to 2.2% takes around 156,000 visitors in total. A gap half that size requires four times as many.
How long should an A/B test run?
The duration should be calculated before launch, based on available traffic, and then respected. Ronny Kohavi recommends at least one to two full weeks, since weekend visitors behave differently from weekday visitors. Stopping a test as soon as one version seems to win sharply increases false results.
Do you need cookie consent to run an A/B test in Europe?
It depends on the country. EU rules require consent for cookies that are not strictly necessary, and France's CNIL counts A/B testing among them. The UK regulator, on the other hand, allows A/B testing without consent under certain conditions.
What is the difference between an A/B test and a user test?
An A/B test measures which of two versions converts better, across a large number of visitors. A user test watches a few people use the site to understand what stops them. It therefore does not require large volumes.
Which tools can you use to run an A/B test?
Dedicated platforms such as VWO, AB Tasty and Optimizely, product analytics tools such as PostHog and open-source projects such as GrowthBook all offer A/B testing. Shopify has also included it in its mid-tier plans since June 2026.
What is a good conversion rate for an ecommerce site?
Most online stores convert between 1% and 3% of their visitors. Littledata, for example, reports a median of 1.3% across the Shopify stores of its clients, and IRP Commerce reports 1.97% in the UK. These figures come from analytics vendors and vary widely by sector.
