Chapter 10

Iterate, Iterate, Iterate: The Art of Asking Many Small Questions Rather than One Big One

First published: 02 January 2012

Summary

This chapter explains the iterative approach in A/B testing methodology. It is possible to create large-scale tests to assess a number of different variables simultaneously, and these big, compound experiments are known as multivariate tests. It is pointed out that multivariate tests are a powerful way to test a number of variables simultaneously, and can reveal interaction effects between them. However, they require more traffic and can be slower to achieve statistical significance. It has been found that the companies that have had the greatest success with A/B testing favor a nimbler, more iterative approach that tests a handful of different variants of a single variable at a time and incorporates the winner as they go on to the next test. When working on a complex change like a site redesign, it is recommended to move testing further up the process so that it becomes something that happens during the design and rollout of the new site, rather than after.

The full text of this article hosted at iucr.org is unavailable due to technical difficulties.