How To Run Live User Testing, Part 1: Setup

by Brenden Mulligan (@mulligan) via TechCrunch

Editor’s Note: Brenden Mulligan is a co-founder and designer at Cluster, and previously created Onesheet, MorningPics, ArtistData, and others. Find him on Twitter @mulligan.

At Cluster, we’re big fans of iteration and experimentation. Since we launched publicly in February 2013, we have rapidly iterated the product on both iOS and Android. In the first eight weeks of being live in the Apple App Store, we launched 10 updates. On Android, there was a week that we pushed out five releases in five days while we did some heavy A/B testing.

While rapid iteration is wonderful, at times we also slow down and make more deliberate decisions about larger changes. When this happens, we tend to make rapid prototypes and then test them in front of different groups. Most of these tests are fairly informal, but occasionally (admittedly not often enough) we run full-blown user testing where we recruit and bring in potential users to walk through the app and give us feedback.

We are working on a big update, so we recently ran multiple sessions for different prototypes. When talking about it with fellow entrepreneurs, they asked us for details. Here is our ever-evolving playbook.

Because this is fairly lengthy, this will be the first of three parts:

1. Setup
2. Running the Tests
3. Debriefing

Before getting started, I want to give a huge amount of thanks to Michael Margolis and theGoogle Ventures design team, who taught us most of these techniques.

Part 1: Setup

User tests are extremely valuable and require a lot of work to get set up. It takes a significant amount of one person’s time over the course of a week to set up the tests and run them well. So make time to do it correctly.

The setup process is divided into the following sections:

1. Decide on a specific thing to test
2. Decide when and where to do the user study
3. Decide what type of users to study
4. Recruit users with Craigslist
5. Trim the candidates list
6. Prioritize and schedule
7. Get the right equipment

Decide on a specific thing to test

We were recently lucky enough to do a sprint with the exceptionally talented design team from Google Ventures (entire post on that coming soon). Over the course of five days, we identified core opportunities with Cluster, brainstormed improvements, built several simple prototypes, and tested the prototypes with potential users. The result of this process was a clear idea of what new concepts were working and which weren’t.

With the list of successful ideas, the team then rapidly built a single functional prototype based off our current app. With this working prototype completed, it was time to show it to users and see if all the insights gathered from the design sprint worked in the context of our actual app.

Decide when and where to do the user study

I was scheduled to be spending a week in Nashville, which gave us a great opportunity to run our test outside of the Bay Area. This is incredibly valuable because the Bay Area tends to be filled with tech-savvy, early adopters. Nashville has its fair share of them, but technology isn’t as core to the community there, so we felt like this would give us an opportunity to meet more “real” users.

It’s also important to pick a quiet, private, and neutral place. As tempting as it might be to use your company’s conference room, I’d recommend not bringing users into the corporate office. Use a friend’s office or co-working space. All advice we’ve been given is the more neutral the location, the better.

The privacy and quietness is important because you’ll be recording the session, so you don’t want to do it in a coffee shop where there’s a lot of distraction and background noise.

When I was in Nashville, I rented a conference room from the Entrepreneur Center. It worked out perfectly.

READ MORE