Conversion rate is the default metric in most ecommerce A/B testing software, but it can mislead you. A variant can raise conversion rate while lowering average order value, pushing up returns, or attracting one-time bargain hunters. For most stores, the better primary metric is revenue per visitor, or gross profit per visitor if you can calculate it, with conversion rate, order value, and refund rate tracked as supporting numbers. Pick these before the test starts, and you’ll avoid most of the false wins that make teams lose faith in testing.
Let’s go through the metrics worth knowing, when to use each, and the traps that come with them.
Why conversion rate alone isn’t enough
Conversion rate answers one question: what share of visitors placed an order? It ignores how much they spent and whether the order stuck.
Imagine a test that adds a prominent discount banner. Conversion rate goes up. But the average basket shrinks because shoppers buy just enough to qualify, margins fall because of the discount, and some of the new buyers never return. On a conversion rate dashboard, it’s a winner. On your profit and loss statement, it isn’t.
That’s why ecommerce testing needs metrics that reflect money and quality, not just activity.
The core metrics
Revenue per visitor
Revenue per visitor (RPV) is total revenue divided by visitors in each group. It combines conversion rate and average order value into one number, so it catches the trade-offs between them. For most store tests, it’s the best default primary metric.
The catch is variance. Revenue varies a lot between orders. One customer buys a $20 item, another spends $400. That spread makes RPV noisier than conversion rate, so tests need more traffic to reach a confident result. Plan for longer runtimes, and consider capping extreme orders in your analysis if a few very large purchases could swing the outcome.
Gross profit per visitor
If your tests touch pricing, discounts, or shipping offers, revenue isn’t enough. A discount can raise revenue and still lower profit. Gross profit per visitor subtracts product cost, and ideally discount cost and shipping subsidy, from revenue. It’s harder to set up, because your testing tool needs cost data, but it’s the most honest metric for offer tests.
Conversion rate
Still worth tracking. It’s less noisy than revenue metrics and helps explain why RPV moved. If RPV rose because conversion rose, the change helped more people decide. If RPV rose because order value rose, it changed what they bought.
Average order value
Useful as a supporting metric for bundle, upsell, and threshold tests. Rarely a good primary metric alone, because you can raise AOV by losing small orders.
Add-to-cart rate
A helpful early signal, especially for product page tests on lower-traffic stores, because add-to-cart events happen more often than purchases. Just don’t treat it as the final answer. Plenty of carts never become orders.
Guardrail metrics
Guardrails are metrics you don’t expect to improve, but that must not get worse. They protect you from wins that cause damage elsewhere.
Good ecommerce guardrails include refund or return rate, cancellation rate, support contacts per order, and page speed. A product page variant that raises orders by overselling the product might raise returns a few weeks later. A variant with a heavy new widget might slow the page. Neither shows up in conversion rate on day one.
Decide on two or three guardrails before launch, and agree on how much movement is acceptable.
Long-term metrics
Some changes affect behavior beyond the first order. Welcome discounts, subscription offers, and loyalty prompts all fall into this group. For tests like these, look at 30 or 60 day repeat purchase rate per group once enough time has passed. You can’t wait months on every test, but for offer and pricing tests it’s worth the patience.
Watch for novelty and timing effects
Sometimes a metric moves because something is new, not because it’s better. Returning customers notice a redesigned product page and click around more for a week, then settle back into old habits. If you stop the test during that first week, you’ll credit the change with a lift that fades.
The fix is mostly patience. Run tests for full weeks, and when the change is visually large, look at how the effect behaves over time. If the gap between groups shrinks steadily, the early lift was probably curiosity. Splitting results by new versus returning visitors also helps here, since first-time visitors have nothing to compare against.
Timing matters in another way too. Sale periods, holidays, and big email sends change who’s shopping and why. A test that runs entirely during a promotion tells you how promotional shoppers behave, which may not match a normal week. Where you can, avoid launching tests right before major campaigns, or make sure the test covers both promotional and regular days.
Trust the data before you trust the metric
Even the right metric is useless if the data behind it is broken. Two checks matter most.
First, sample ratio mismatch. If you planned a 50/50 split and ended up meaningfully off, something went wrong with assignment or tracking. Microsoft’s experimentation team, writing about diagnosing sample ratio mismatch, found about 6% of their A/B tests had an SRM, and they don’t analyze a test’s effects until it passes this check. Mature website experimentation software runs this check automatically.
Second, order tracking. Check that the orders your testing tool records roughly match your store platform’s numbers, including orders through third-party checkouts. If one variant’s orders are undercounted, you’ll get a false result.
And don’t stop early. Evan Miller’s well-known post How Not To Run an A/B Test explains how repeatedly checking for significance and stopping when it appears inflates false positives. Revenue metrics, being noisy, are especially prone to early swings that later disappear.
Choosing metrics by test type
A simple mapping covers most cases:
| Test type | Primary metric | Supporting | Guardrails | |—|—|—|—| | Product page content | Revenue per visitor | Add-to-cart, conversion rate | Return rate, page speed | | Cart and shipping offers | Gross profit per visitor | AOV, conversion rate | Cancellations | | Price tests | Gross profit per visitor | Conversion rate, AOV | Repeat purchase, support contacts | | Landing pages for ads | Revenue per visitor | Bounce rate, conversion rate | Return rate |
Use it as a starting point, then adjust to your business.
What your software needs to support this
Not all split testing software makes these metrics easy. When evaluating ecommerce A/B testing software, check that it tracks orders and revenue natively, reports revenue per visitor by default, supports guardrail metrics, and can pull in cost data if you want profit metrics. A ranked overview of ecommerce testing tools can help you compare how different products handle revenue reporting.
Revenue tracking in practice
Getting revenue data right is often the hardest part, especially with third-party checkouts. CustomFit.ai is one platform that focuses on this for D2C brands. It’s a no-code A/B testing, personalization and CRO platform for Shopify and other ecommerce stores, it sends data to and from GA4, and it can track revenue through checkout providers such as GoKwik, Shopflo, Shiprocket and Razorpay Magic Checkout, which matters for stores where many orders never touch the default checkout.
Key takeaways
Make revenue per visitor your default primary metric, and switch to gross profit per visitor for pricing and offer tests. Track conversion rate, order value, and add-to-cart as supporting numbers. Set guardrails like returns and page speed before launch. Before you read any result, confirm the traffic split came out close to plan (no sample ratio mismatch) and that orders are being recorded correctly. Then wait for the planned end date, even if day four looks amazing. Get the metric right and testing stops starting arguments. It starts settling them.
