David Housman

Multivariate Testing Research


Let’s say a product manager has an idea, but they’re not sure if it will work or not. One alternative to the magic 8-ball is a multivariate experiment. In a multivariate experiment, someone takes that idea, puts it on the site under controlled conditions. We then measure what happens after people see it.

The “Test” is a statistical test that compares two versions of a website. In the end, someone like me gets to say that A is better than B. This is not the whole story. A statistical test is only one of many possible approaches to analyzing a multivariate study.

Multivariate testing platforms must become sounding boards.

Product Managers and designers work with assumptions and decisions, and play them out over numerous iterations. Usability testing and multivariate research is a mechanism to test their output. They need more than a simple thumbs up or thumbs down.

Usability Testing usually involves quantitative measurements, just like Multivariate Testing. The real opportunities came from watching users use a new version of the site, asking questions, and interpreting what happened. Even good ideas have room to improve. Since 2004, every usability test I have ever run or observed has uncovered some opportunity for improvement.

The name “Multivariate Research” reflects that Multivariate Experiments are a form of Usability Testing. In Multivariate Research, SiteCatalyst becomes our eyes, Qualtrics becomes our mouth and ears, and I am told that aromalytics technology is just around the corner. Usability studies of 8 users scale up nicely into the tens of thousands. Any way you slice it, User Researchers and Web Analysts need to start getting chummy: Big Data is coming to User Experience Research.

Leave a Reply