David Housman

Recruiting: Multivariate Research vs. Usability Testing


The first lesson of User Research is that you get out what you put in. Before every usability test, the User Experience Team makes decisions about who they want to test their idea out on, which usually means finding people that are representative of people that might actually visit your website. Multivariate Research shares this step, but execution is very different. Usually you want to exclude people that are not representative, such as:

  • People that have a lot of experience with the current system
  • The Elderly
  • Nerds, Techies, Geeks, Software Engineers and pretty much anyone that works in tech
  • Noobs and other abnormally un-technical people, such as my brother Mike
  • Ugly people. This one isn’t really backed up by Nielsen, but research on attractive people is totally preferable to the alternative. Don’t you agree?

Recruiting for Traditional Usability Testing

Once we’ve decided who we want to observe, I’d draft an ad, which linked to a short survey. I’d post the ad somewhere online, and picked candidates from the survey data to start calling. Each call has two objectives: establish that the candidate qualified for the study, and they were not “bogus” or making stuff up to get included in the study and get the participation incentive.

Recruiting for Multivariate Research

In order to unlock the benefits of Multivariate Research recruiting, the mindset of investigators needs to change from that of experimenter to one of observer. The approach is very similar to segmentation: start by launching a popover intercept on entry to ask the visitor to self-identify as a member of a group of interest.

Qualtrics is probably not going to pick up on all of the subtleties that a telephone conversation between a recruiter and a participant. But there are tradeoffs. Multivariate research can conduct a study on real people that visit your site, while they visit your site, doing the things that they normally do. It provides opportunities to deviate from the constraints of traditional Usability Testing recruitment.


Multivariate research scales much better than user research. Dozens of sessions can be run simultaneously. After the initial setup, “recruiting” is largely unmanned. Thousands of sessions can be run simultaneously. Aside from duration, there is little to no additional cost of running an additional 100 users. The degree of rigor in recruiting for Multivariate Research can be relaxed due to the context: people don’t visit websites so that they can lie to get included in a research study.

Strength in Numbers

Usability Testing does not require a great number of sessions to encounter usability issues, but scalability and sample size are still limitations. It is one of the strengths of Multivariate Research. In the course of such a small sample (usually 5-8 sessions, depending on who you talk to), idiosyncrasies of individual sessions are almost impossible to tease out from larger themes.


Usability severity ratings have two parts: impact and frequency. When we identify a usability problem, it is sometimes necessary to project the frequency where the issue will happen in the larger population. A problem that occurs in 1 of 8 sessions may mean that 25% of the population will encounter that issue, or it might mean that 99% of visitors will encounter it. Or it may mean that the only person that will encounter that problem is that user, on that day, on that machine. Big data has some advantages.

Leave a Reply