Your A/B testing programme just found a winner. A new discount banner lifted conversion by a healthy margin, the test ran long enough to be statistically significant, and the result is sitting in a dashboard somewhere with a green checkmark next to it. Mission accomplished, right?
For most eCommerce teams, the default answer is the same: roll it out to 100% of visitors.
That decision gets made without a second thought, because it doesn't feel like a decision at all. It feels like the natural conclusion of a successful test. You've gone and validated your hypothesis. And when you've got a winner, there's a paper trail to an uplift.
It isn't. It's a choice, and for most brands it's the wrong one.
A/B testing tells you which experience wins across your traffic as a whole. It doesn't tell you which specific visitor should see that winner, or when. That's a different question, that's answered in an entirely different way. It's a decision that requires you needing to know something about the person you're serving an experience to in the first place.
Let's be clear — A/B testing definitely has a place
A/B testing tools do exactly what they're built to do, and they do it well. They replace opinion with evidence. A test that runs to statistical significance tells you, with confidence, that variant B outperforms variant A on your website or mobile app.
Made With Intent customers continue to use tools like Convert, VWO, and AB Tasty alongside our tool, because testing answers a question nothing else can: does this specific change move the number we care about?
Think about what a test actually proves. It proves that, across hundreds or thousands of visitors, variant B beat variant A. It says nothing about any single visitor in front of you right now. Statistical significance is for large volumes of people not a prediction about the individual standing at your checkout, which is a limitation CRO metrics share more broadly.
The difference between A/B testing tools and Made With Intent
Should 100% of people get your A/B test winner?
Once a test determines a winner, most teams roll that experience out broadly, often to every visitor who fits the original test's targeting. Many testing tools do offer segment targeting: device type, geography, new versus returning.
Some brands go further and layer in basic rules, like showing a winning banner only after 30 seconds on site, or only to visitors on their third page view. These rules feel like personalisation. They're really just a slightly more granular version of the same default: a fixed condition, checked once, applied broadly.
But look closely at how it gets set up. Teams choose segments once, at test design time, and then leave them alone. Even a well-built intent-based segmentation programme needs continuous application, not just at launch, or it drifts back into the same static-rule problem.
A rule that says "show this to returning visitors" doesn't change three minutes later when a returning visitor starts behaving like someone about to abandon their basket, or like someone who was going to buy anyway with or without an incentive.
The real problem is narrower and more difficult to solve. The targeting decision is frozen at the moment the test launches, while the person in front of it keeps changing their mind. (Their intent)
A discount that lifts conversion in aggregate can still be the wrong call for two specific people in your test group. A first-time visitor who's price-sensitive and hesitating needs it. A loyal customer who's already three products deep into checkout, on their fourth order this year, does not.
It's the same tension at the heart of whether discounting is a race to the bottom or a genuine growth lever. There's no problem with offering discounts, but it can be a blunt instrument or entirely inappropriate depending on where your customer is at in their buying journey.
How reading intent allows you to respond immediately to changing demands
A/B testing tools make their decision once, when the page loads or the session starts. Whatever segment a visitor falls into at that moment determines what they see for the rest of that test.
But purchase intent isn't fixed at page load. A visitor can arrive browsing casually, spend four minutes comparing two products, add one to their basket, then hesitate at the price for 30 seconds before starting to type a discount code into Google in another tab. That's not a static segment. That's a person, moving in real time, and most testing infrastructure was never built to react to this situation as it unfolds.
Some testing and CRO tools react to single behavioural triggers: an exit-intent popup fired on mouse movement toward the browser bar, a scroll-based banner, a basic engagement score. We think those are great, and definitely worth using. However, they're single-trigger rules, checked once a condition is met, and not a continuous read on a person as they use your website and apps.
What real-time behavioural data adds
Made With Intent's platform tracks over 900+ behavioural signals through each visitor's session, including scroll speed, click sequences, time spent, and product comparison patterns, updating its prediction of purchase intent continuously rather than once. When intent shifts, the response can shift with it.
None of this means every session tells a dramatic story. Plenty of visitors browse, leave, and never come close to converting in that session, and no amount of real-time signal turns a casual browser into a buyer. The claim isn't that every visitor is secretly ready to buy. It's that the visitors who are ready look different from the ones who aren't, and a rule set once at test launch can't tell the difference between them three minutes into the session.
What real-time intent-based deployment looks like
Picture the same winning discount from earlier. Instead of firing it to every visitor who matches a fixed segment, an intent-based system asks a narrower question: Is this specific visitor showing signals of hesitation or exit risk right now, or are they already moving smoothly toward checkout?
Made With Intent's Intent Scoring analyses those signals continuously and its Timing Engine decides the moment to act, rather than firing on a page-load rule or a fixed timer. A hesitant first-time visitor comparing prices across tabs might see the discount. A returning customer already at the payment page, showing no hesitation signals at all, doesn't need it, and doesn't get it. This is what discounting with intent looks like in practice: the same offer, reserved for the visitors who actually need it to convert.
The winning experience from your test doesn't change. Who receives it, and when, does.
Take a product page test that proved a scarcity message ("only 3 left") increased add-to-basket rate. Fired at everyone, it also lands on a loyal customer who's bought the same product line four times before and knows exactly what they want. To that visitor, it reads as pressure, not information. Deployed by intent, the same message is reserved for the visitors actually showing comparison and hesitation behaviour, the ones it was designed to nudge in the first place.
How an actual company used Made With Intent alongside A/B testing
Appliances Direct, a UK home appliances retailer, ran into a version of this problem. Their discounting was broad and largely untargeted; offers went out early in the visitor journey, without real-time context, regardless of whether a given shopper actually needed the incentive to convert.
Using Made With Intent, they began reserving discounts for visitors showing genuine exit and abandonment signals, rather than firing them to everyone who matched a static segment. They left everyone else to convert at full price, as they were always going to.
In the end, they made a 42% saving in margin previously given away to visitors who would have converted regardless, without a drop in conversions from the visitors who needed the discount to purchase.
But don't take our word for it. Take a look at our Appliances Direct case study and make up your mind yourself.
That figure lines up with a broader pattern. Made With Intent's own Intent Gap research found that 83% of shoppers have used a discount code despite being ready to pay full price.
Why new test winners are rare
That scarcity of new winners isn't unique to Appliances Direct. Research from Microsoft's own experimentation team, drawn from thousands of controlled experiments at Bing and elsewhere, found that only 10 to 20% of test ideas move the metric they were designed to move, and that a genuine breakthrough, the kind of result that reshapes a programme rather than nudging it, shows up in roughly one in 500 experiments. The findings come from Kohavi et al.'s Seven Rules of Thumb for Web Site Experimenters (2014). Most of what a mature testing programme finds after the early wins are used up is small, single-digit-percentage gains, not new step changes.
Testing tells you what works. Intent tells you for whom, right now.
None of this is an argument against testing. It's an argument for finishing the job testing starts.
Your A/B testing programme answers one question well: does this experience outperform the alternative? What it was never built to answer is a second, quieter question that gets decided by default instead of by design: who should see this winning experience, and at what moment in their session?
Testing finds your best experience. Making sure it reaches the right person, at the right moment, without giving away margin to people who didn't need it, is a different problem, and it needs a different kind of data to solve.
If you want to see what your existing winning experiences would look like deployed by real-time intent instead of a blanket rule, book a demo and we'll walk through it against your own traffic.
Become an Intent Insider
Get subscriber-only insights we don't publish anywhere else and event invites before anyone else.
By submitting this form you agree to our (more than fair) terms.
