An A/B testing case study becomes useful when it helps a team frame a better decision. The reported lift needs its metric, audience and comparison. A result for pricing-link clicks answers a different question from a result for completed purchases.
The eight cases below include positive results, a losing variant and an earlier test with no discernible effect. They span B2B software, financial services and consumer purchases. Each result belongs to its original setting. None supplies a forecast for another website.
Most are original named-customer reports published by the testing vendor. That makes them traceable accounts, with a commercial reason to emphasise successful work. We have not independently audited their raw data. Where a baseline, sample or statistical result is absent, the interpretation stays limited.
Use the B2B website performance scorecard to identify the decision on your own site before looking for a comparable experiment.
What does an A/B test percentage actually describe?
Check whether the number is a relative change or a percentage-point change. In a hypothetical example, conversion rising from 4% to 5% is a 25% relative increase and a one percentage-point increase. Calling it a 25 percentage-point increase would substantially misstate the result.
Next, identify the event. Clicking a button, starting an application, completing a purchase and generating revenue are separate outcomes. A case that reports one cannot silently provide evidence for the others.
Also check what the business measured after the immediate action. An application might be rejected. A meeting might be unsuitable. A purchase might be returned. A website experiment can improve an intermediate event while leaving the final commercial effect uncertain.
B2B software: Bandwidth’s product-page proposition
Bandwidth’s vendor case reports a 12% relative increase in visit-to-lead conversion after a product-page redesign. Approximately 3,500 visitors participated over 60 days. The baseline and statistical uncertainty are undisclosed, and the report does not show subsequent lead qualification.
The redesign combined several elements. Its result therefore concerns the complete page proposition. Using it as proof that one logo strip or headline will increase leads would overstate what was tested.
For a B2B product page, investigate whether buyers can explain what the product does and recognise a relevant use case. If the current page fails that task, a coherent replacement may be a reasonable candidate for research. Decide whether the business needs to evaluate the whole page or isolate one element before choosing the experiment structure.
B2B software: Veeam’s pricing link
Veeam’s historical CTA case changed “Request a quote” to “Request pricing” and reports click-through rising from 0.54% to 1.40%. Downstream lead outcomes are absent. The publisher reports a 161.66% relative lift, which cannot be reproduced exactly from the rounded rates. The underlying conversion counts are not disclosed.
A useful application is to investigate the information buyers expect when clicking a CTA. Ask them what they anticipate on the destination page and compare that answer with the actual experience.
Changing a label to promise pricing creates an obligation to satisfy that expectation. If the destination merely asks for contact details without explaining what the buyer receives, the business still needs to examine the resulting enquiries. Choose a success measure that extends far enough to evaluate that trade-off.
Consumer finance: Garanti’s application banner
Garanti’s vendor report describes a 24% relative application increase for a banner variant, with 119,000 visitors over about 18 days. It omits absolute conversion rates and uncertainty. The change combined direct application with messaging.
Applications are the reported outcome. The case does not establish growth in approved accounts or revenue. That distinction is especially relevant when a business has a screening stage after submission.
For a B2B team, the comparable question could concern bringing an initial enquiry action closer to relevant product information. Before copying the pattern, establish whether visitors understand the request and whether the information collected supports a useful response. A prominent form can expose an unclear offer more widely without resolving its underlying ambiguity.
Insurance: Admiral’s clicks and policy conversions
Admiral’s Optimizely case reports lower CTA clicks alongside higher insurance conversion rates during successive proposition tests. It supplies no exact lift, sample or uncertainty. Several revisions contributed to the selected experience.
This makes the relationship between intermediate activity and outcomes worth investigating. A team judging only CTA clicks could prefer a different version from a team judging completed business actions.
Translate that into a B2B reporting requirement. Put form completion, accepted enquiries and opportunities beside page interactions. When those measures move in different directions, inspect the counts and time windows before choosing an explanation. The purpose is to understand the commercial trade-off, including the workload created by low-quality enquiries.
Consumer finance: Tinkoff’s losing information variant
Tinkoff’s first reported experiment tested additional product information with more than 60,000 visitors over 13 days. A separate-page version performed below control; an inline version reported a 15.5% relative increase. The loss magnitude and statistical uncertainty are absent.
Extra information can create different experiences depending on where it appears. The case gives a reason to examine that decision without assuming the original explanation transfers to every form.
On a B2B website, ask buyers to find an answer while completing a relevant task. Observe whether they can return to the task and whether their previous work remains available. That research could support an inline explanation, a nearby link or a more substantial content change. Select the option based on the observed need.
Paid education: WikiJob’s earlier inconclusive placement
WikiJob’s founder-interview case reports 34% more purchases for a testimonial placement. An earlier lower-page placement had no discernible effect. Both accounts omit sample, duration, baseline and statistical uncertainty.
The earlier result deserves attention. “No discernible effect” leaves room for several possibilities, including a small effect the test could not detect. It does not establish that testimonials in that location can never matter.
For a B2B page, define the question a piece of evidence is meant to answer. It might concern the suitability of the service, experience with a relevant challenge or confidence in ongoing support. Ask a representative buyer whether the evidence resolves that concern. A familiar design pattern cannot replace a relevant, substantiated claim.
Retail: Lotte Duty Free’s product-list actions
Lotte Duty Free’s case reports a 45.70% relative add-to-cart increase across 17,440 mobile users over about two weeks. The product-list variant added cart and restock actions. Baseline rates and uncertainty are undisclosed.
The recorded outcome concerns adding products. A claim about completed sales would require additional evidence. The distinction prevents an attractive interaction result from becoming an unsupported revenue claim.
For B2B websites, consider which page transitions help a buyer and which introduce avoidable work. A transition to a detailed service explanation may be useful even if it adds a click. Observe whether the destination supplies information needed for the decision before treating fewer steps as the objective.
Telecom ecommerce: Vodafone’s page performance
Vodafone’s original Google collaboration case, published in March 2021, reports 8% more sales in a performance A/B test with visually identical pages. Total sample, duration and statistical uncertainty are absent, and commercial baselines are redacted.
The result supports investigating performance as a commercial factor. It does not establish a fixed sales gain for each second removed from any page. Starting performance, audience and task all affect the relevance of the comparison.
On a B2B site, identify whether loading behaviour obstructs an important task on devices the audience uses. Inspect the experience after performance changes as well as the measured timings. A faster page that drops required content would introduce a different problem into the review.
Which evidence should influence a B2B experiment?
Choose comparability before headline size. A modest result on a similar buying task may be more informative than a large result from an unrelated purchase process. Even a close match provides a hypothesis to investigate locally.
Review the source using these questions:
- Was the comparison concurrent and controlled, or did it compare periods before and after a release?
- Who entered the experiment, and how were they assigned?
- What changed between versions?
- What was the primary outcome and its denominator?
- Are the baseline, conversion counts and uncertainty available?
- Did the business assess later quality, revenue or costs?
- Is the reported outcome relevant to the decision currently being made?
Missing details should narrow the claim. They do not require rejecting every useful observation. A case can suggest a worthwhile buyer question while providing insufficient evidence to predict the effect of a change.
Avoid averaging lifts across these examples. The metrics, populations and reporting methods differ. Such an average would provide a number without a defensible business meaning.
A brief for adapting an external experiment
Complete this brief before adding a case-inspired idea to the build queue:
Local observation: State what happened on the business’s own site. Include the audience, page and evidence date.
External reference: Link the source and identify the relevant result. Record what the source leaves undisclosed.
Proposed explanation: Describe why the local obstacle might occur. Include another explanation worth investigating.
Variant: Specify the complete change and the experience that will remain constant.
Primary outcome: Name the event, eligible population, counting unit and observation period.
Commercial checks: Identify later outcomes or costs that could change the decision.
Method: Record assignment, required sample, statistical approach and the stopping rule before launch.
Decision: Define what would justify adopting, rejecting or investigating the variant further.
If the sample requirement is impractical, use the low-traffic CRO workflow to choose another research method. A demo-form diagnosis can establish delivery failures and usability obstacles before a controlled experiment becomes necessary.
Frequently asked questions
Can we copy a winning A/B test from another industry?
Use it to identify a question or possible change. Check whether the audience, task, offer and success measure are comparable. Validate the experience locally and describe any remaining uncertainty.
Does a statistically significant result guarantee a useful improvement?
No. Assess the size of the effect, its uncertainty and its commercial consequences. A result can affect an intermediate metric while leaving the buying outcome unresolved. The quality of assignment and measurement also needs verification.
Where can we find more original experiment reports?
The VWO case library, Optimizely customer stories and Google web.dev case studies provide starting points. Check each study’s design and disclosure before quoting its outcome.
What should an agency report after an experiment?
Expect the tested versions, audience, counts, method, uncertainty and decision, with commercial limitations stated beside the result. Our CRO work and measurement service connect that evidence to the next website decision.
