A lot of A/B tests at small companies can't finish. The idea being tested can be excellent and the test still goes nowhere, because the traffic isn't there and nobody ran the numbers before splitting it in two.
So let's run them. Say your site converts 2% of visitors, and you want to know whether a new headline lifts that to 2.2%, a 10% improvement. To tell that difference apart from random noise with the usual level of confidence, you need about 80,000 visitors on each version. That's 160,000 visitors to judge one headline.
At 20,000 visitors a month, the test runs for eight months. By then you'll have changed the offer twice and forgotten what the headline said.
Before anything goes live, I do one boring check: can this test give us an answer at all? This piece walks through that check, what to do when the answer is no, and a simple rule for picking the next move.
01Small differences need a lot of visitors
Two numbers decide how much traffic a test needs: how often people convert today, and how small a change you're hoping to catch. The rarer the conversion and the subtler the change, the more visitors it takes to be sure you're looking at a real effect and not a lucky week.
You don't need a statistics degree to estimate it. In his classic post on A/B testing mistakes, Evan Miller gives a rule of thumb that gets you close enough to make a decision:
Here n is the number of visitors per version, p is your current conversion rate, and δ is the difference you want to detect, in absolute terms. For our headline: 16 × 0.02 × 0.98 ÷ 0.002² gives about 78,400. A proper calculator, set to the usual 95% confidence and 80% power, says 80,700, which is close enough to ruin the afternoon either way.
Here's the full picture, in visitors per version:
| Current conversion rate | To detect +10% | To detect +20% | To detect +50% |
|---|---|---|---|
| 1% | 163,000 | 42,700 | 7,750 |
| 2% | 80,700 | 21,100 | 3,800 |
| 5% | 31,200 | 8,200 | 1,500 |
| 20% | 6,500 | 1,700 | 300 |
The right-hand column is where things get interesting. If you only care about changes big enough to lift conversion by half, a site with modest traffic can get an answer in a few weeks. The left-hand column is where most "let's test the button color" ideas live, and it's a long way from home.
One more trap while we're here. Checking the results every morning and stopping the day it looks like a winner feels efficient, but it quietly breaks the math. In Evan Miller's own example, peeking ten times turns a result reported at 1% significance into something closer to 5%. Pick the sample size before you start, then leave it alone.
02When the traffic isn't there, change the question
A test that can't finish still leaves you with a decision to make. Here are the moves I reach for, roughly in the order I try them.
Test a bigger change. Swapping a word in a headline asks for a 10% effect. Rewriting the page around a different promise, changing the offer or the price can plausibly move things by half, and at a 2% conversion rate that takes 3,800 visitors per version. If the change is too small to show up in a test, it's often too small to matter much.
Measure a step that happens more often. A purchase might happen 2% of the time, but a click on "Get a quote" might happen 20% of the time. At that rate, catching a 10% lift needs about 6,500 visitors per version. The catch is that people click on plenty of things they never buy. Check a few weeks later that the extra clicks turned into extra leads, or you may have just made a more tempting button for the same number of buyers.
Watch five people use it. A test can tell you which version won, and nothing at all about why. Jakob Nielsen's well-known finding is that five users uncover about 85% of usability problems. Five calls, one afternoon, and you usually know what to fix before you've spent a single visitor.
Ship it and watch, when it's easy to undo. Put the change live, note the date in your analytics, and compare the weeks before and after. It's less rigorous than a test: a holiday week or a mention in someone's newsletter can move the numbers on their own. But for a reversible change, "live and watched" beats "waiting for significance" until next summer.
Just decide. Some choices cost almost nothing to get wrong. If you'd switch back in five minutes, the test is overhead. Pick the version your best judgment likes and spend the traffic on a question that deserves it.
03Do the two-minute check before every test
This is the rule I use. It fits on a sticky note, which is roughly where it belongs.
- Find your row. Write down the monthly visitors on the page you want to test, its current conversion rate, and the smallest lift you'd act on. Look up the visitors per version in the table, or run them through a sample size calculator.
- Work out the duration. Visitors per version × 2 ÷ your weekly traffic = weeks to an answer. With 5,000 visitors a week, a 2% page and a 20% lift to catch, that's 21,100 × 2 ÷ 5,000, so a bit over eight weeks.
- Decide what to do with that number. If you get an answer within a month or two, run the test and don't peek. If you don't, look for a step that happens more often and test that instead. If neither works, pick from the list above based on two questions: how much would a wrong call cost, and how fast could you undo it?
The whole check takes two minutes with your analytics open. Do it for the test you're planning right now. It might save you eight months of watching two lines wobble around each other.
And yes, you can still test the button color. Just budget about 160,000 visitors for it.
