A/B testing in marketing: how to run tests that make you money
A/B testing explained for business owners: what to test first, how many visitors you need, how to test Google and Meta ads, real examples and common mistakes.
Most owners think A/B testing means changing a button from green to orange and waiting for a miracle. That is why most of them try it once and give up. A/B testing is not a design trick. It is how you stop guessing with your ad budget.
The idea is simple: show two versions of the same ad, page or email to similar people at the same time, and keep the one that brings in more customers. The hard part is doing it in a way that gives you an answer you can trust, with the traffic a normal business actually has.
Even the companies best at this lose most of their tests. Microsoft found that about one third of its experiments improve the metric they target, one third make no difference and one third make things worse. At Google and Bing, only about 10% to 20% of experiments produce a positive result (Kohavi and Thomke, Harvard Business Review, Sept–Oct 2017). The same article tells the other side: one small change to how Bing displayed ad headlines, an idea that sat on a list for more than six months, lifted revenue by 12%, worth more than $100 million a year in the US alone.
So the point of testing is not that every idea wins. It is that you find out which ones do, before you put more money behind them.
- Change one thingTwo versions, one difference, shown at the same time.
- Measure money, not clicksBooked calls, sales or cost per customer.
- Test big ideas firstThe offer and the message beat button colors.
- Decide the sample size firstAnd don't stop the test the moment one version pulls ahead.
- Low traffic? Test bigger changesSmall tweaks need far more visitors than most businesses get.
What is A/B testing?
A/B testing (also called split testing) compares two versions of something to see which one performs better. Version A is what you have now, often called the control. Version B is the same thing with one change. People are split at random, so each person sees only one version, and you compare the results.
Because both versions run at the same time with similar people, the difference in results comes from the change you made, not from the season, a holiday or a competitor’s sale. That is what makes it better than “we changed the page last month and leads went up”.
A/B testing vs multivariate testing
| A/B test | Multivariate test | |
|---|---|---|
| What changes | One thing (or one whole version vs another) | Several things at once, in every combination |
| Traffic needed | Less | Much more, because traffic is split across many combinations |
| What you learn | Which version wins | Which combination of parts wins |
| Best for | Almost every small and mid-size business | Sites with very high traffic |
If you are asking which one to use, the answer is almost always an A/B test.
What to test first
Most testing time is wasted on small things. The changes that move revenue are the ones closest to the buying decision. Test in roughly this order:
| What to test | Example | Where |
|---|---|---|
| The offer | “Free quote” vs “inspection this week” vs “first visit at a set price” | Ads and landing page |
| The main message | Price-led headline vs speed-led vs trust-led | Ad headline, page headline |
| The next step | Long form vs short form vs “book a time” calendar | Landing page |
| The ad creative | Photo of your team vs product photo vs short video | Meta and Google ads |
| The audience | Broad targeting vs a lookalike vs interest groups | Meta ads |
| Proof | Reviews near the form vs further down the page | Landing page |
| Small details | Button text, colors, images | Only once the big things are settled |
If you are not sure what your offer should be, start with our guide to building an offer stack. A better offer usually beats a better button.
How to run an A/B test, step by step
- Pick one number that means money. Booked calls, sales, cost per customer or revenue per visitor. Clicks and likes are easy to win and often go the wrong way: a version can get more clicks and fewer customers.
- Write down what you expect and why. For example: “A clear price on the page will bring in fewer but better leads, because price shoppers drop out early.” This stops you from rewriting the story after the result comes in.
- Change one thing. If you change the headline, the photo and the form together and B wins, you won’t know why. Testing one complete new page against the old one is fine too, as long as you treat it as one idea.
- Work out how many visitors you need before you start (next section). Then set the end date.
- Split traffic at random and run both versions at the same time. Use the testing tool in your ad platform or page builder, not “A this week, B next week”.
- Run for full weeks. People behave differently on Monday than on Saturday. Two full weeks is a sensible floor for most service businesses.
- Don’t stop early. Checking every day and stopping the moment one version looks like a winner is the most common way tests go wrong (more below).
- Decide, then write it down. Keep the winner, note what you learned, and use it to choose the next test. A simple spreadsheet with the date, the idea, the result and the decision is enough.
How many visitors do you need?
This is the part most guides skip, and the reason most small business tests give the wrong answer. The smaller the difference you are trying to detect, the more visitors you need. The numbers grow fast.
What this means in practice:
- If you have low traffic, test bold changes. A new offer or a new page can produce a big difference you can actually detect. A new button colour cannot.
- Use a sample size calculator before you start. Enter your current conversion rate and the smallest lift worth knowing about. Evan Miller’s free sample size calculator is a common choice.
- Count conversions, not just visitors. A test with 40 leads per version will rarely give a clear answer, whatever the dashboard says.
Why you shouldn’t stop a test early
Most testing tools show a live “winner” and a confidence score. It is tempting to stop the moment it turns green. Don’t. Every time you check and stand ready to stop, you give random noise another chance to look like a winner.
Evan Miller showed how big the effect can be: in his worst-case example, checking after every visitor turned a test meant to be wrong 5% of the time into one that was wrong 26.1% of the time (How Not To Run an A/B Test). His fix is simple: decide the sample size in advance and only read the result at the end.
How to A/B test your ads
Both big ad platforms have testing built in. Use them, rather than turning ads on and off by hand.
Google Ads has custom experiments. You copy a campaign, make one change in the copy, and Google splits the budget evenly between the original and the experiment. You pick the success metric, and you can tell Google to only apply the winner if the result is statistically significant. A few rules from Google’s help page: the campaign has to be active and not already in another experiment, only one experiment can run at a time, and changing either version during the test makes the result harder to read. Good things to test are bidding strategies, landing pages and ad copy.
Meta (Facebook and Instagram) has A/B tests in the Experiments section of Ads Manager. You can test two to five versions that differ in creative, audience, placement or optimization, and Meta splits people at random so each person only sees one version. The report shows how likely you would be to get the same result if you ran the test again (Jon Loomer’s walkthrough). If that number is low, wait or rerun before you act. For which ads to test in the first place, see our guide to ad creative.
On both platforms, judge the test on cost per lead that turns into a customer, not just cost per lead. That only works if your tracking passes booked jobs or sales back to the platform. Not sure which platform to start with? Read Google Ads vs Facebook Ads.
How to A/B test landing pages and emails
Landing pages are where testing usually pays off most, because every ad you run sends people there. Most page builders and CRMs include a split test feature: you create a second version of the page and the tool sends half the visitors to each. Start with the headline and offer, then the form or booking step. Our guide to landing page optimization lists the 10 fixes most worth testing. Our small business website checklist covers the basics to fix before you start testing, and our conversion rate optimization work covers the rest.
Emails are the easiest place to practise, because most email tools can send version A to part of your list, version B to another part, and the winner to everyone else. Subject lines are the usual test, but judge them on replies, bookings or sales, not opens.
A/B testing examples for service businesses
These are example tests, not results. They show the kind of question worth asking.
| Business | Version A | Version B | What to measure |
|---|---|---|---|
| Med spa | “20% off your first treatment” | “Free skin consultation with a nurse” | Consultations booked and shown up |
| Roofer | Ten-field quote form | Three fields, then a call | Inspections booked |
| Dental practice | Stock smile photo in the ad | Photo of the real team | Cost per new patient booked |
| Law firm | “Contact us” button | “Free 15-minute case review” | Consultations that become cases |
| HVAC | Generic service page | Page that matches the ad (“AC repair today”) | Calls per visitor |
When A/B testing isn’t the answer
A/B testing needs traffic. If your page gets a few hundred visitors a month, a test could take months and still not give a clear answer. In that case:
- Fix the obvious leaks first. Slow replies, missed calls and confusing booking steps lose more customers than any headline. See why leads don’t turn into customers and how speed to lead changes the numbers.
- Talk to customers. Ask the last ten who bought why they chose you and what almost stopped them. Those answers give you better test ideas than any tool.
- Make bold changes and watch closely. Without a proper test, a before-and-after comparison is weaker evidence, so compare equal periods and watch for outside causes such as seasons.
Common A/B testing mistakes
- Testing tiny details first, like button colors, while the offer and message stay untested.
- Judging on clicks or opens instead of booked calls, sales or cost per customer.
- Changing several things at once and not knowing which one made the difference.
- Stopping the test early because one version looked ahead after a few days.
- Running A one week and B the next, so the season or the news decides the winner.
- Editing a version while the test runs, which resets what you are measuring.
- Not writing results down, so the same ideas get tested again a year later.
- Calling a test with too little data. A dozen leads per version is not an answer.
A/B testing works best as part of a wider plan for how to get and keep customers: know what a customer is worth, fix the leaks, then test the ideas most likely to bring more of them in.
Questions about A/B testing
What is A/B testing in marketing?
A/B testing in marketing means showing two versions of an ad, landing page or email to similar people at the same time, and keeping the version that brings in more leads, sales or customers. Only one thing changes between the two versions, so you know what caused the difference.
How long should an A/B test run?
Until it reaches the sample size you worked out before starting, and for at least one or two full weeks so every day of the week is included. Don’t stop early because one version looks ahead.
How much traffic do you need for an A/B test?
It depends on your current conversion rate and how big a difference you want to detect. In the example above, a page converting at 5% needs about 1,500 visitors per version to spot a 50% lift, but about 8,200 per version to spot a 20% lift. Low-traffic businesses should test bigger changes.
What should I A/B test first?
Start with the things closest to the buying decision: the offer, the main message and the next step (form, call or booking). Leave small details such as button colors until those are settled.
Can you A/B test Google Ads and Facebook Ads?
Yes. Google Ads has custom experiments that split budget evenly between the original campaign and a changed version. Meta has A/B tests in Ads Manager’s Experiments section that split people at random across two to five versions.
What is an example of A/B testing?
Example: a plumbing company sends half its ad visitors to a landing page with the headline “Same-day drain repair” and half to “Blocked drain? We’re there in 2 hours”. After enough visitors, it compares how many booked a job from each page and keeps the winner. Everything else on the two pages stays the same, so the headline is the only reason for the difference.
What should I test first on a landing page?
Start with the things that change the most decisions: the headline and offer, the main call to action, the length of the form and whether proof (reviews, results) is near the top. Small design details like button colour rarely make a difference you can measure with normal traffic.