There are two established New York tech companies who do a great job of getting close and staying close to their customers: Meetup and The Ladders.
Andres Glusman (VP of Strategy, Meetup) and Jeff Gothelf (Dir of UX, The Ladders) established regular, lightweight usability sessions at their respective companies (that’s them above giving Lean Ignite talks — see bottom for links). Every week they bring in 2 or 3 actual or potential customers to learn about user behavior and mindsets, test out new concepts, and expose current flaws in design or strategy.
I asked Andres and Jeff for their biggest challenge in running a lean usability lab. Both answers boiled down to interpreting, and thus acting upon, learnings. To Andres, the key is balancing different people’s reactions and keeping things in tune with bigger goals. As he put it, you neither want to “freak out and change everything immediately, nor dig in your heals and refuse to change anything. The trick is finding the right rhythm for integrating learnings into the dev process.”
Jeff wrote, “Tempering reactions is the hardest thing. Sometimes it’ll take 12 participants saying the same thing to push an idea through and other times one user can skew an entire session.”
I have felt this first hand, both as a team leader trying to keep a cool head and as a product designer over-reacting to a single user’s pain. It takes discipline and an open mind to handle customer data well.
Here are five tips that might help:
Jeff notes, “Nielsen always said that between 5-8 users are needed to see 80% of the big problems in your experience. Without a dedicated user researcher we don’t have the luxury of doing 5-8 tests per week as it is time consuming and cost-prohibitive. We’ve found that with 3-4 users in every week we can both knock it out in a half day and get enough insight as to what the ‘boulders in the road’ are to give us direction on where to go next.”
Even a couple users every other week is better than what most companies do, which is none at all, or the infrequent binge. The critical thing is getting a continual, lightweight process in place. There are plenty of excuses to put this off, but it all comes down to what you consider important. Just do it.
Examine your biggest risks and test accordingly. Decide what type of customer you want to learn from, and know your core questions or user tasks before they begin. Always make room to understand the customer’s mindset, context and behavior right at the start of a usability / customer development session. As Andres recommends, “with earlier-stage things you want to be more broadly exploratory, while later on you can focus more exclusively on more tactical stuff and straight-up usability.”
You do not need statistical significance if you are seeing a recurring pattern and your gut tells you there is indeed a problem. However, to have a pattern you do need more than one person!
Sometimes a flaw is exposed that you, as a product creator, are dying to fix. We all take pride in our work, and seeing a design flaw can literally, viscerally hurt. But you have to ask yourself where the problem falls in the bigger picture of what the company needs.
Don’t get self-indulgent to mitigate your own pride/pain if there are bigger fish to fry for the company. Especially in the earliest stages of a startup, don’t get lost in endlessly optimizing usability and features if you haven’t solved the bigger problem of finding product-market fit.
User feedback can clarify product debates in a very healthy way. It gives you the “why” behind the metrics you are seeing. It shifts the discussion out of the realm of opinion and ego and into reality. But what do you do if someone really does dig in their heels and becomes a total recalcitrant, refusing to participate in the process but blocking the team?
Andres says, “No matter how good a storyteller you are, you cannot convey user pain as effectively as just watching a user struggle.” If someone cannot observe live, then use video. If a decision is important enough to warrant it, Jeff recommends editing together a video with multiple clips of users laboring to solve a problem area.
Evidence is powerful stuff.
More resources:
- Andres’ great presentation on running a lean usability program (full of great advice)
- Ignite talks from Andres and Jeff
- Steve Krug’s Rocket Surgery Made Easy
- My 12 tips for customer development interviews