A/B testing
A/B testing is an experiment that shows two versions to separate audiences at the same time and compares which one performs better, using numbers.
In an A/B test you prepare two versions of one thing: A and B. For example, two ad images for the same product, two email subject lines or two button labels. You split the audience at random, so half see A and half see B. At the end you see which brought more of the result you want, such as clicks or orders.
The value of a test is that it puts a measurement where a guess would be. A team may argue over which sentence works better, and a test settles it with a number. Two rules matter: only one thing should differ between the versions, and the test must reach enough people to show that the difference is not down to chance. Otherwise the result misleads.
Why It Matters
- You base decisions on data rather than opinion. The version that works better may not be the one you like.
- You can see the effect of small changes. A single headline or image can shift the result noticeably.
- You spot a weak idea on a small audience before the budget goes to a large one.
How It Works
- Pick a single questionAsk something clear, such as "Does the product name in the subject line raise the open rate?"
- Change one thingOnly the tested element should differ between A and B. If image and headline both change, you cannot tell which one caused the result.
- Split the audience at randomPut people into two groups at random, in similar sizes. Run both versions in the same period so timing does not distort the result.
- Wait long enoughDo not decide on a few hours or a few people. Wait until the time and size you set in advance are reached.
- Apply the winner and write it downIf the difference is clear, use the winning version. No difference is also a result. Note what you tested and what you found.
Example
Liva Butik prepares two images for a Linen shirt ad: in A the shirt is on a model, in B it is on a plain background. Burak Demir splits the same budget evenly across the two ads and runs both in the same week. Only the image changes; headline and price stay the same. At the end of the week, B has a slightly higher click rate, but both ads received very few clicks. Burak does not treat the result as settled and extends the test by another week. The gap holds in week two. Only then does he make B the main image and note the finding.
| A/B testing | Multivariate testing | Before-and-after comparison | |
|---|---|---|---|
| What changes | One element | Several elements together | Usually several things |
| Audience | Same time, split in two groups | Same time, split in many groups | The same audience in different periods |
| How much data it needs | Moderate | A lot | Little, but less reliable |
| When it fits | Testing a clear question | When traffic is large | Getting a quick idea |
Not to Be Confused With
A/B testing is not a random trial or a pre-launch check. In an A/B test, two versions go to a real audience at the same time and the results are compared. When you read the result, look at it together with the [conversion rate](/glossary/donusum-orani).
Frequently Asked Questions
How many people do I need for an A/B test?
There is no fixed number. The smaller the difference you want to detect, the more people you need. A practical rule for stores is to keep running until each version has reached at least a few hundred people or a few dozen orders. A result from very few people is mostly chance and will not repeat.
Can I test more than one thing at once?
You can, but then it is no longer an A/B test. It is a multivariate test and needs a far larger audience. For a small store the smart route is to go in order: test the headline first, then the image, then the offer.
What should I do if the result is unclear?
That is a finding too: what you tested does not move the result much. You can extend the test or try a bigger change. Note the outcome anyway. To tie the decision to real customer behaviour, read measures such as conversion rate and ROAS alongside it.