Skip to content
GetProfit
Log in

← All terms

A/B test

An A/B test is a comparison of two page versions shown at random to different visitors at the same time, to see which leads more of them to the target action.

How it works

The test splits visitors at random between the current page (A) and the changed one (B). Both run at the same time, under the same season, prices and ads, so the gap in conversion rate can be put down to the change. Statistical significance shows whether it exceeds chance.

You fix the number of visitors before the start. A difference half as large needs four times as many. A store without that traffic has other ways to judge a change: a careful before-and-after comparison, or session recordings and heatmaps, free in Microsoft Clarity. The test is one method of conversion rate optimisation.

Formula

Visitors per version ≈ 16 × p × (1 − p) ÷ d², where p is the current conversion rate and d the difference to detect (5% significance, 80% power)

Where you see it

Google has had no website testing tool since Google Optimize closed on 30 September 2023. Instead, Google works on Analytics integrations with AB Tasty, Optimizely and VWO.

Example

Example store, not client data.

The tableware shop converts 3.75% of its 8,000 monthly visits from ads. To detect a rise to 4.5%: 16 × 0.0375 × 0.9625 ÷ 0.0075² ≈ 10,270 visitors per version, about 20,500 in all, or 2.6 months of that traffic. To detect 4.0%, d is a third as large and the test needs nine times more: about 92,400 per version.

Not to be confused with

  • Campaign experiment — a Google Ads test that splits ad traffic or budget between two campaigns, not site visitors between two pages.

Right and wrong readings

  • Wrong: “On day three B is ahead at 95% significance, so we stop.” Right: Evan Miller’s worst case checks after every visitor and stops at a significant result or at 150 visitors. A change with no effect then shows as significant 26.1% of the time, not 5%.
  • Wrong: “B converted 4.2% against 3.8% on 1,000 visitors each, so B wins.” Right: that is 42 orders against 38; even 3.75% against 4.5% needs about 10,270 per version.

Sources