8 ecommerce CRO case studies: what moves conversion when shoppers can ask questions

Most conversion rate optimization case studies tell you the lift and leave out what you'd need to trust it. We read seven of the pages that rank for this search, which list about 77 cases between them, and found one case that reported statistical significance. None gave a sample size or a test length, and none said whether the shoppers counted had opted in.

This list has eight ecommerce cases, all about one kind of friction: shoppers who had a question before they'd buy. We checked each one against its original source (a research paper, a retailer's own announcement, or the published case study), and we labeled each by the kind of evidence behind it, so you can tell a randomized test from a launch announcement. One case, Balsam Hill, is a Gladly customer, and it's marked that way.

How to read a CRO case study

Several kinds of evidence show up in this list, and they carry different weight. Before you copy a result, check which kind you're looking at.

Controlled tests

In a controlled test (an A/B test or a randomized experiment), shoppers are split at random between the current experience and the new one, so the change is the only thing that differs between the groups, and any gap in results can be credited to it. Good write-ups also report a p-value. A p-value under 0.05 means a gap this large would show up less than 5% of the time by chance if the change did nothing at all.

Watch for the difference between a relative lift and percentage points, too. A store that converts at 2% and moves to 2.2% can report "+10% conversion" or "+0.2 points," and both are true.

Before/after comparisons

A before/after comparison measures a period before launch against a period after it. It's often the only option for a single brand, and it's useful for direction. The weakness is that everything else changes at the same time: season, promotions, traffic mix, and the product catalog. Treat the size of a before/after lift as approximate.

Engaged-user and attributed results

An engaged-user result compares shoppers who used a feature with shoppers who didn't, and the catch is that shoppers decide for themselves whether to engage. A shopper who opens a chat or types a question into a search assistant is often already close to buying, so the comparison measures the feature and the shopper together. As Julian Skelly of Publicis Sapient told Retail Systems, "Early adopters of any new interface tend to be more motivated shoppers, which naturally flatters the numbers."

An engaged-user number can still point at something real. It just can't tell you how much of the gap the feature caused.

Case studies: unanswered product questions

The first three cases look at what happens when a shopper's question gets answered before they buy, or goes unanswered.

1. An AI pre-sale chatbot on a cross-border marketplace

Evidence: controlled test

A cross-border online retail platform ran seven randomized field experiments between September 2023 and June 2024, each adding generative AI to a different part of the business. The results are in a working paper (not yet peer reviewed) by Lu Fang, Zhe Yuan, Kaifu Zhang, Dante Donati, and Miklos Sarvary. In the pre-sale chatbot test, which ran in September and October 2023 with 44,614 shoppers, the control group got an automatic reply saying service was unavailable and the treatment group got answers from a GenAI chatbot. Two of the authors consulted for the platform during the project and a third worked there, which the paper discloses.

Conversion rose 1.31 percentage points, a 21.7% relative lift, and sales per shopper rose 16.3%. Both results were significant at p<0.01. Cart value moved down 4.5%, but that change wasn't statistically significant.

The test compares answering a question with not answering it, so a store that already answers pre-sale questions shouldn't expect a 21.7% lift from switching tools. The authors tested that too. Against human agents, the chatbot alone showed no significant difference in sales or conversion, and the chatbot with escalation to a person raised sales 11.5% (p<0.1), a result they treat as a conservative lower bound.

2. Live chat on Taobao

Evidence: observational study

Haoyan Sun, Jianqing Chen, and Ming Fan studied panel data from sellers on Taobao, China's leading consumer-to-consumer marketplace, for a 2021 paper in Production and Operations Management. They found that live chat had a positive effect on traffic-to-sales conversion. The effect was stronger when a product page's information was less complete, and when the product's perceived value was higher (a higher rating or a lower price).

This is an observational study, so the authors used a statistical model to separate live chat's effect from other seller and product differences, which is weaker than randomizing who gets chat. The paper's abstract doesn't give effect sizes, so we haven't either.

3. Balsam Hill (Gladly customer)

Evidence: before/after, plus results among shoppers who used chat

Balsam Hill, which sells artificial Christmas trees and holiday decor, extended Gladly into the pre-sale journey so shoppers could get product help in the same AI conversation they'd use for service. Nearly one in five shoppers who viewed a product in chat added one to their cart, and chat drove six figures in add-to-cart value in the first 30 days. Resolution rose from 80% before launch to 91%.

The resolution change is a before/after comparison. The add-to-cart figures describe shoppers who chose to use chat, so they belong with the engaged-user results, and an add-to-cart isn't a completed order. The case does show shoppers using chat to shop as well as to get help once it could answer product questions.

Case studies: sizing and fit

"Will this fit me?" is a question a size chart is supposed to answer before the shopper has to ask it.

4. The Iconic and Fit Analytics' Fit Finder

Evidence: controlled test (vendor-reported)

Australian fashion retailer The Iconic ran an A/B test for three months with 250,000 shoppers, comparing Fit Analytics' Fit Finder size recommendation tool against a traditional static size chart. The Fit Finder group showed a 1% increase in conversion rate, a 0.4% reduction in return rate, and 6% more revenue after returns.

The case study comes from the vendor, and it doesn't say whether the 1% is relative or in percentage points or report any significance figures. Still, the metric is worth copying: by measuring revenue after returns, the test caught a benefit that a conversion-only test would have missed.

Case studies: finding the right product

These three cases cover shoppers who knew what they needed but not which product it was.

5. Frasers' Ask Frasers

Evidence: launch claim (method not given)

Frasers launched Ask Frasers on its premium fashion platform in April 2026. Shoppers can use the assistant to "ask questions, refine preferences, compare products and narrow down options." Frasers said conversion rose up to 25% compared with traditional search experiences since the assistant launched.

Frasers didn't say whether that compares shoppers who chose the assistant with shoppers who searched the usual way, or the whole site before and after, and it gave no time window. In the same Retail Systems analysis, James Heimers of RAPP said, "In most cases this is a relative uplift on a low baseline, so the absolute gain is more modest than it first appears."

6. GenAI search query refinement

Evidence: controlled test

The same cross-border platform from case 1 tested AI-refined search queries in May and June 2024, in three nine-day experiments covering Arabic, Japanese, and Polish searches, with about 1.85 million shoppers in total. The control translated queries without interpreting their meaning, while the treatment used generative AI to work out what the shopper was asking for.

Conversion rose 1.2% and sales rose 2.9%, both significant at p<0.05. That's a small lift on something the site already did, and it took close to two million shoppers to measure it with confidence.

7. Kingfisher

Evidence: engaged-user

Kingfisher has built more than 50 AI services across its retail brands, including assistants such as Hello B&Q. Mohsen Ghasempour, Kingfisher's chief AI officer, told Retail Systems, "We see significantly higher conversion rates, in some cases more than double, from customers who engage with our AI agents." He said the assistants matter most "where products can be more technical."

No baseline was given, and the comparison is between shoppers who engage and shoppers who don't. His point about technical products matches case 2, where chat helped most when the product page left questions open.

Case studies: answering before shoppers ask

Some questions can be answered before anyone types them, on the product page itself.

8. AI-generated product descriptions

Evidence: controlled test

In December 2023, the case 1 platform ran five one-week experiments in English, Spanish, French, Portuguese, and Korean, with about 4.77 million shoppers in total. The control saw the existing human-written descriptions (or none, for products that didn't have one), and the treatment saw AI-generated descriptions added on top.

Conversion rose 1.3% (p<0.01) and sales rose 2.1% (p<0.05). Nearly half the products had little or no text description. In the English-language test, the only one with description-length data, the sales gain was concentrated there: products with 50 words of description or less saw sales rise 6.5% (p<0.05), while sales for products with longer descriptions didn't change significantly.

What the strongest results have in common

The biggest controlled lift in this list came where shoppers previously got no answer at all. In case 1, going from "service unavailable" to a working chatbot raised conversion 21.7%. Cases 4, 6, and 8 improved a size chart, a search box, and product pages that already existed, and conversion moved about 1% in each. Even there, case 8's gain was concentrated in the pages that were missing information. Both kinds of result are worth having, but they shouldn't be compared as if they measured the same thing.

The gains came from more shoppers buying. The authors of the arXiv study write that "the gains operate primarily through higher conversion rates rather than larger cart values." If you're testing conversational help, conversion rate is the primary metric to watch, and average order value may not move.

The two uncontrolled claims (up to 25% at Frasers and "more than double" at Kingfisher) are larger than every controlled result in the list, including case 1. That's what you'd expect when a comparison can include the shoppers' own intent along with the feature's effect, as Kingfisher's does and Frasers' may.

The test that didn't work

The same study ran one test that didn't work. In a 12-day experiment in January 2024 covering about 1.2 million products, the platform replaced existing seller-written Google ad titles with AI-generated ones. Sales moved down 4.5% and conversion moved down 2.3%, and neither change was statistically significant. The authors say the model wasn't fine-tuned for ecommerce advertising and may have left out keywords that matter commercially, and the AI-titled ads got fewer views and clicks. The team and the experimental setup were the same as in case 1, so the gap in results comes down to the job. In case 1 the AI answered questions nobody else was answering, and here it rewrote seller ad titles that already carried the keywords shoppers search for.

How to test conversational help on your own site

You can run a version of the strongest tests on this list on your own store, with whatever assistant or chat tool you're evaluating.

Use a holdout group

Keep a random share of visitors from seeing the assistant or chat option, and show it to everyone else. The holdout is your control. Without one, you're back to a before/after comparison or an engaged-user comparison.

Measure all exposed traffic, not just shoppers who engage

Compare conversion for everyone who could have used the assistant with conversion for everyone in the holdout, whether or not they ever opened it. That's what turns an engaged-user number into a controlled one. The lift will look smaller than "shoppers who chat convert at X%," and it'll be the number you can defend in a budget meeting.

Work out your sample size before launch

Small lifts need big samples, and case 6 needed about 1.85 million shoppers to measure a 1.2% conversion lift with confidence. Before you launch, work out how many visitors and how many weeks it'll take to detect the lift you care about, starting from your current conversion rate. Our ecommerce conversion rate optimization framework walks through the sample size math, and our ecommerce conversion rate benchmarks can help you sanity-check your starting point.