What a split test is
A split test, also called an A/B test, compares two versions of something by showing each to a different group of visitors at the same time. Version A is the current page, version B has one change, such as a different headline, button label, photo or shipping message. You then see which version leads to more of the outcome you care about, usually completed orders or add-to-cart clicks.
The point of running both at once is fairness. If you simply change a page and compare this week with last week, other things also changed: the weather, a promotion, which ads ran, the day of the week. A split test controls for those, because both versions face the same conditions.
The problem of chance
Visitor behavior has a lot of randomness. Flip a fair coin 20 times and you might see 12 heads, even though the coin is fair. In the same way, two identical pages can show different results purely by luck.
Here is a hypothetical example. Suppose each version of a page gets 1,000 visitors. Version A produces 20 orders (2.0 percent) and version B produces 25 orders (2.5 percent). B looks 25 percent better, but the difference is five orders. Five orders can appear by chance alone, and with numbers this small you cannot tell luck from a real improvement. Declaring B the winner would be a false conclusion that is easy to reach and hard to notice.
This is why traffic size matters. The smaller the true difference between versions, and the lower your baseline conversion rate, the more visitors you need before the result means anything. As a rough guide, detecting a modest improvement in a low conversion rate often takes thousands of visitors per version, and sometimes far more. Free online sample size calculators can give you an estimate for your own numbers; treat their output as a rough guide.
What this means for a small store
If your store gets a few hundred visits a month, classic split testing on small tweaks will usually be inconclusive. That does not mean you cannot make good decisions, but it changes the approach.
Test bigger changes. A larger difference is easier to detect than a slight one. Changing the entire offer layout, adding clear shipping information near the price, or replacing weak photos is more likely to produce a visible gap than altering a button color.
Test where the traffic is. Your busiest pages, such as the home page or a top product, collect data fastest. Testing a rarely visited page will take very long.
Use a more sensitive measure with care. Clicks on “add to cart” occur more often than purchases, so you reach a signal sooner. But the signal is only useful if more add-to-cart clicks also lead to more orders, so confirm that afterward.
Combine testing with qualitative feedback. When numbers are thin, other evidence counts: customer emails, screen recordings, a few conversations, or asking friends to try to buy something and talk through their thinking. Several people getting stuck at the same spot is meaningful even without a statistical test.
Consider sequential changes with caution. A before-and-after comparison is weaker than a split test, but when traffic is too low for a proper test, it can still be informative if you compare longer periods, record other events that happened, and accept that the answer is uncertain.
Running a simple test
- Pick one question and one change. For example, “Does showing delivery time under the price increase add-to-cart clicks on this product page?” Changing several things together means you will not know which mattered.
- Choose your main measure first. Decide before you start whether you will judge by orders, add-to-cart rate or something else. Choosing afterward invites cherry-picking the number that happens to look good.
- Estimate the traffic needed. If the page cannot reach it in a reasonable time, say four to eight weeks, choose a different page or a larger change.
- Split visitors randomly. Most testing tools handle this. Avoid manual splits like “version A on weekdays, B on weekends,” which bring in day-of-week effects.
- Run both versions at the same time, and keep other things steady. Avoid launching a sale or large ad campaign in the middle of a test.
- Let it run for full weeks. Behavior often differs between weekdays and weekends, so full-week cycles give a fairer picture.
- Decide the stopping point in advance and stick to it.
Mistakes that produce false winners
Stopping when it looks good. Results bounce around early on. If you check daily and stop the moment B pulls ahead, you are likely to stop on a lucky streak. This habit, sometimes called peeking, inflates false positives. Set the duration or visitor count beforehand.
Running too many tests at once. Testing five variations, or many different metrics, increases the chance that one looks like a winner by luck alone.
Ignoring seasonality and unusual events. A holiday, a press mention or a stock-out can distort a test. Note such events and, if serious, restart.
Judging only the percentage. “30 percent lift” sounds big, but if it comes from a handful of orders, it is fragile. Always look at the actual counts behind a percentage.
Declaring “no difference” too confidently. A test that finds no clear winner on low traffic does not prove the two versions are equal. It only means the test could not tell. Treat it as inconclusive.
Forgetting the downstream effect. A change can raise orders while increasing returns or lowering profit margin. A pop-up that boosts email sign-ups but irritates shoppers is another example. Check the metrics around the one you are testing.
Tools
Many website platforms and analytics services include basic testing features, and there are dedicated testing tools as well. For a small store, a simple setup is enough. Beware of tools that load heavy scripts and slow the page; slower pages can hurt the results you are trying to improve, so keep an eye on page speed. Also remember that testing uses visitor data, so make sure your privacy policy and consent practices cover it.
What to test first
Some reasonable candidates, roughly in order of promise:
- Clear delivery and returns information near the price on product pages.
- The main product photo or the first line of the description.
- The wording of the call-to-action on the home page.
- Whether showing total costs earlier in the cart reduces drop-off, as explored in unexpected fees.
- Trust elements near checkout.
Pick the one that addresses a known problem, not simply the easiest to change. The best test ideas come from evidence: a point in the funnel where many visitors leave, or the same confusion appearing again and again in customer emails.
Keep a record
Write down each test: what you changed, why, the dates, the visitor counts and the result. Over time this becomes a reliable guide to what works with your customers, and it prevents rerunning old experiments. It also keeps you honest about the many tests that turn out inconclusive, which is normal.
Split testing is a useful tool, not a requirement. For a small store, careful observation, steady improvement of the obvious problems and testing only where there is enough traffic will usually serve better than chasing tiny differences that the numbers cannot support.