A New Approach to A/B Testing Can Pay Off
A/B testing has become the standard for determining which of two versions of a marketing communication resonates with consumers. Websites, display ads, social media, email and direct marketing all use A/B testing to determine which version of a communication—A or B—should be directed at consumers. However, researchers are now finding that A/B testing may in many cases hinder company profits when marketers spend too much time in the testing phase of a rollout plan.
In the paper “Test & Roll: Profit Maximizing A/B Tests” published in Marketing Science, Associate Professor of Marketing Elea McDonnell Feit and her co-author Ron Berman, assistant professor of marketing at the Wharton School of Business, suggest that the traditional technique used for analyzing an A/B test is not always well suited for these tests because of the large sample size needed to achieve statistical significance.
“We recast A/B tests as a decision problem where marketers explicit goal is to maximize the total profit earned both during the test and after a winner is chosen,” said Feit. “During the test some customers get the worse option, so we don’t want test sample size to be too big, but if the test sample is too small, marketers risk rolling out the wrong option.” The researchers call this decision problem setup “Test & Roll.”
Instead of the traditional sample size for A/B testing formulas, the researchers propose marketers use a smaller sample size while also having some idea of the range of outcomes expected for the test based on consumer behavior in previous experiments.
It’s better not to focus too much on statistical significance in experiments, especially when there is a limited population of total customers, according to Feit.
In online advertising testing, for example, where the effects are often small but profitable, the recommended sample size for the traditional A/B testing formula may be larger than the population of consumers itself. In these cases it’s best for marketers to roll out the ad that performed best even with a smaller sample size rather than spending time and money to achieve statistical significance.
To make their recommendations more accessible to marketers, Feit and Berman launched a Test & Roll website where practitioners can compute profit-maximizing sample sizes for their own experiments.
Feit’s paper titled Test & Roll: Profit Maximizing A/B Tests was published in the journal Marketing Science in May 2019. An earlier version of this article appeared on Drexel’s News Blog
