How long to run an A/B test
Until each version has seen enough visits to tell a real difference from luck. How long that takes depends on three things you can estimate today. This guide explains them, gives example numbers, and has a free estimate you can run on your own traffic.
How long should an A/B test run? Three things decide it
- Your traffic. Only visitors who see the page you are testing count. A test on a page with 300 visits a month runs ten times slower than one on a page with 3,000.
- Your conversion rate. The share of visitors who complete the goal today. A goal many visitors complete, such as a button click, gives an answer much faster than a rare one, such as a purchase.
- The size of the change. A small difference takes many more visitors to detect than a big one. A new headline that lifts enquiries by 50 % shows up quickly; a button colour that lifts them by 2 % may never show up on a small site.
The number of versions matters too: every extra version takes its own share of the visitors, and the statistics need a little more evidence per version when there are more of them.
Example numbers
How many visits each version needs before a winner can be called, for a test with two versions (the original and one variant). Visits, not unique visitors: someone who comes back on another day counts again. Most tests finish within this range; about half finish by the first number.
| Conversion rate | To show a 20 % lift | To show a 30 % lift | To show a 50 % lift |
|---|---|---|---|
| 1.0 % | about 51,000 to 76,000 | about 24,000 to 36,000 | about 10,000 to 15,000 |
| 2.0 % | about 25,000 to 38,000 | about 12,000 to 18,000 | about 5,000 to 7,500 |
| 5.0 % | about 9,700 to 15,000 | about 4,500 to 6,800 | about 1,900 to 2,900 |
| 10.0 % | about 4,600 to 6,800 | about 2,200 to 3,200 | about 1,000 to 1,500 |
Lift is relative: a 20 % lift on a 5 % conversion rate means 6 %. To turn visits into time, multiply by the number of versions and divide by the page's daily visits. For example, a page with 10,000 visits a month and a 5 % conversion rate needs about 4 to 6 weeks to show a 30 % lift.
Estimate your own test
Enter your page's traffic and conversion rate to see how long a test needs to find a winner, if the change works.
From your analytics: visits or sessions, not unique users. If the test targets only some visitors, count just theirs.
The share of visitors who complete the goal today. Not sure? Keep 3.
The original plus each variant. Two is a classic A/B test.
Enter the page's monthly visits to see an estimate.
Rules that keep the answer honest
- Decide what you will measure before you start. Pick one goal. Choosing the goal after you see the numbers finds patterns that are not there.
- Run whole weeks. Visitors behave differently on weekdays and weekends. Stopping on a Tuesday after starting on a Friday mixes the two unevenly.
- Don't stop on a lucky day. Early results swing a lot. With a classic significance test, checking every day and stopping at the first good-looking number calls far more false winners than the test promises.
- Use a test that allows checking early, or don't check. Sequential tests are built to be checked as often as you like. Twinpage uses one, so checking every day keeps false winners low. The verdict can still move while data comes in, so wait for the minimums below.
- Accept "no clear difference". If each version has had enough visits and there is still no winner, any real difference is probably smaller than you can detect. Keep the version you prefer and test something bolder.
What if the estimate is months?
On a small site this is common. Test a bigger change, pick a goal more visitors complete, test fewer variants, or test on your busiest page. The guide A/B testing with low traffic goes through each of these.
Frequently asked questions
What is the minimum time for an A/B test? One full week, even on a busy page, so every day of the week is included. On most small sites the visitor count, not the calendar, is what limits you.
How many visits does Twinpage need before it calls a winner? At least 1,000 visits and 25 conversions per version, and a difference large enough to be unlikely by chance. Before that it says the result is leading at most, and tells you not to decide yet.
Can I stop a test early if one version is clearly ahead? Only if your tool uses a test designed for it. With a classic significance test, stopping early is the most common way to call a false winner.
Testing with Twinpage
Twinpage is A/B testing for marketing sites: one script tag, a visual editor, one main goal per test and a verdict you can trust, with prices you can read before you start.