The $5 Guerrilla User Test

The internet is all atwitter about ReadWriteWeb’s article on how a quirk in Google had them highly ranked for the search term “Facebook login”.

It was like we had unearthed a long-lost city, the Atlantis of the Internet. But instead of treasures and gold we’d found a steady deluge of confused and frustrated users who had tried everything they knew to do and just wanted to log in to Facebook, damnit. But how had this happened? It certainly wasn’t that thousands and thousands of people had just started searching for “facebook login” yesterday. This stream of people has been there all along and something is broken.

There is a persistent meme, which this article is only helping reinforce, that user experience professionals are needed because the average user is far less intelligent than the average designer so we need to hire some people who can “think stupid” just like the user. Such thinking doesn’t benefit anyone, developers lose respect for the user and start creating condescending, dumbed down UIs. Users continue to find the new software hard to use because it doesn’t address their core issues.

Users aren’t so much unintelligent as they are distracted and indifferent

Your average user may be perfectly competent and zip through your app like a charm when they’re in a controlled setting, focusing exclusively on your application and incentivized to succeed. But such a scenario is almost never likely to happen in the real world. What’s more realistic is that they’re devoting, at best, 10% of their attention towards your app while they have the TV blaring in the background, an IM conversation they’re also involved in, thoughts about whether that meeting with the boss tomorrow means a promotion or getting fired. Your application is at best, 5th on their priority list and they’re largely moving on autopilot as they navigate through it. Once you understand this basic reality, user behavior becomes a lot easier to understand.

A couple of years ago, a grizzled UX professional taught me one invaluable fact.

Drunk people are a pretty accurate mimic of distracted, indifferent people

This insight has lead to a wonderful technique I’ve been refining over the years that I call “The $5 Guerrilla User Test”.

The $5 Guerrilla User Test

Here’s the 5 second version:

1. Bring a laptop to a bar
2. Offer to buy someone a beer in exchange for participating in a user study
3. Watch your application crash & burn as people do all sorts of ridiculous ass shit they would never do in a lab but constantly do in real life
4. Go back, apply the lessons you have learnt, repeat until you have an app that is 100% drunk person proof

This is the slightly longer version for those who are interested:

• Like conventional user studies this is best done with a group of two, one to run through the script, the other to take notes.
• Approach in a friendly manner, explain who you are and who you work for and ask them if you can have a moment of their time.
• If they don’t seem receptive from the get go, thank them for their time and move on to a different target
• Explain to them that you want some insight on a piece of software you’re currently building and tell them that you’re willing to buy them a pint of beer as compensation for their participation
• If they accept (and 90% of them will), ask them their preference of beer and then ask your partner to go off and order it
• While you’re waiting for your partner, inform them of your data collection policies, the procedure and the standard stuff about how they can quit at any time. It doesn’t much matter what exactly you say to them, the key is to make it boring. This step is key. When you first approached them, you were something novel for the night so they’re interested and motivated to perform. 2 minutes of dull chatter is going to lose their attention and they’re back to being utterly indifferent about your problems again.
• Once your partner gets back, run it just like any other user study.
• At the end, hand them their beer, thank them for their time and drop off some business cards for some easy word of mouth marketing. If you offer any sort of premium features, give them a year’s access to it as a gift as well. It’s a nice surprise and converts surprisingly well.

That’s it! It’s cheap, fast, can be done by anyone and gives you insights you never would have gotten hiring a professional usability consulting firm. Go out and do it!

Here’s a couple more tips I’ve picked up over the years:

• The first time you do it, you’re probably going to be suffering from approach anxiety. Start off approaching a group of the same sex as you so that the encounter isn’t sexualized. Next, move on to a mixed sex group and then finish the night with an opposite gender interaction so you get a nice demographic spread.
• Focus on people who don’t normally get talked to at bars, middle aged people, homely girls, the guy sitting in the corner.
• People alone, reading a book, have a 50% chance of agreeing to participate. People alone, reading a newspaper, have a 99% chance of agreeing to participate.
• If you ever get the feeling that you’re being messed with, politely end the experiment, give them their beer and move on to the next round of testing.
• Groups of all guys tend to be the only ones who ever mess with you. I avoid asking them as a rule.
• You want someone who’s pretty drunk but not completely trashed. A good way to calibrate is to keep on asking progressively drunker people until your results become garbage, then back off one notch from there. After a few rounds, you learn to spot the signs of an ideal participant.
• Conveniently for you, weekdays end up being more effective than weekends so you can do this after work.
• I generally try to keep the entire session down to around an hour which works out to be roughly 4 * 15 minute user studies. Much more than that is tiring and too much data to analyze the next day.
• While beer is good before & after the user study, try and keep beer away during the user study. Spilled beer on a laptop can be an expensive mistake.
• Bars are wonderful at segmenting by demographic. Match the bar you’re going to with the user population you want to target. Different bars will produce slightly different results but the variation is not huge.
• If you’re planning on regularly using one bar to do your tests, tip the bartender well from the start, like, 30 – 40%. That extra $1 you spend isn’t going to break the bank and having a bartender on your side brings all sorts of benefits.
• edit: One extra tip from a friend, mobile app development benefits even more from this technique since mobile use tends to involve even more distraction. Users are using your app while walking down the street, driving, holding a conversation etc.

Anyway, that’s it. I’d love to hear from people how their experience with the $5 Guerrilla User Test goes.

Related Posts

1. Oct 21st (day 9): Test driven management
2. The User Experience of Comics is abysmally poor
3. Not impressed with the iPhone user interaction
4. The fallacy of the facebook redesign