I’ve been hearing an excuse lately for avoiding experiments and “getting out of the building”:

It boils down to this: “if the results don’t have clarity and repeatability then why test in the first place?” Or put another way, “if you can’t perfectly design the experiment and isolate a single variable, and if you can’t have absolute confidence in your results, then what is the point?”

Here’s a truth with startups and new products: understanding test results and root-causes is often really hard. Yet it rarely makes sense to spend the time and money to get statistical significance or perfect clarity. We need to exercise judgement and intuition to interpret results, but that does not invalidate the usefulness of getting outside of our own heads. Sticking one’s head in the sand is not a valid approach.

When I discussed this challenge with my project-teammate Jon Berger, he said, “We test to uncover clues, not facts.”

I thought that was an excellent phrase that honestly acknowledges the purpose and limits of lightweight testing. You are getting facts within your test, but only clues for the world beyond. You might be getting metrics, but still have to understand the “why” behind them. After all, we’re talking about human beings here.

Avoiding any experiments is as foolish as running over-designed, over-resourced tests in an attempt for perfect clarity.*

You won’t hear me argue with the premise that you want to structure your experiments intelligently, but I don’t consider the muddiness of results data to be a reason to avoid the process. You want and need both vision and validation. You want both intuition and data-driven iteration.

* When it comes to tests, the descriptors I think you want to shoot for are nimble, lightweight, creative, prioritized, and iterative. I didn’t say “frequent” because in the “think, make, check” or “build, measure, learn” cycle, there are times where it is ok not to be testing (when you are implementing the learnings of the previous tests and thus teeing up new ones).

I recently gave a talk to LUXr New York about MVPs, or really about running experiments. Instead of using the term “MVP”, I find myself using the word experiment for a few different reasons: less jargon; a clearer connotation of lightweight, learning, and not necessarily tied to digital product, and a clearer signal that this is about bringing the scientific method to startups as a balancing force to vision and judgement, and not just a 1.0 release.

Here are the slides from that talk (I don’t have the video of the talk, sorry):

There are strategic UX leaders, and then there are tactical UX implementers.

To be a strategic leader, one needs to broaden thinking beyond design and usability, and start thinking holistically about critical business goals and risks.

As the broader UX profession moves from being artifact-based to results-based, this is going to be critical. However, I see the online UX community spending most of its time talking about usability, psychology, content and visual design, with surprisingly rare mentions of business.

UX doesn’t need to justify its existence to the business team. UX *is* a critical part of the business team, and somehow everyone needs to get on board with this, including UXers.

In a product business, everything revolves around and is built upon the core product. The product team is responsible for answering core questions: are we making the right solution? for the right customer? for the right problem? and will this fuel a successful business?

If you help your organization solve for *all* of these, you transcend being a tactical UI designer and become a strategic UX leader.

Jeff Gothelf and his team at The Ladders try to resolve this issue by thinking about new designs as hypotheses to test, and by focusing their product teams on KPIs. If you don’t have that structure, and want a framework to help you think through how business and UX come together, I recommend Dave McClure’s startup metrics for pirates (if you haven’t seen Dave’s decks, you’ll recognize a lot of UX thinking… just expect the slide styling to make your eyes bleed):

• Activation
• Acquisition
• Retention
• Revenue
• Referral

These are all UX challenges!

I also recommend looking at Eric Ries’ The Lean Startup, with particular attention paid to the section on “innovation accounting”.

Many senior UX folks already embed this in their work. But it still amazes me how rarely I see these topics come up in the community. It amazes me how often I talk to UX practitioners who are obsessing about how and why something gets accomplished by the user, but fail to loop back to the strategic and financial implications for the business.

UX practitioners should not become “short order cooks” to business folks (a phrase I’ve lifted from Jeff), any more than agile development teams should abdicate responsibility for deciding what to build and why.

Think as much about empowering (and de-risking) the business as empowering users, and find the magical balance between the two that leads to awesomeness. As a UX person wielding invaluable skills and processes, you have a lot to bring to the table if you properly take a seat.

On Quora, Jared Spool asked the question “What’s the difference between Agile Development & Lean UX?” There are a number of interesting responses. Here is mine:

Agile dev and UX already focuses on rapid cycle times, constant collaboration, staying user-focused and continuous delivery of valuable software.

Unfortunately too often, the definition of valuable gets muddied (or worse, is abdicated altogether if design and/or tech becomes “order takers” to the business).

Lean’s insight is to turn things into results-oriented experiments, and by things, I mean every important assumption and risk. And when you are creating something new, whether startup or enterprise, you inevitably have a lot of assumptions. The team ideally levels up from thinking about silo-ed roles to all being product owners, and ideally all thinking like business owners.

Lean also helps teams escape thinking purely in terms of product. Once you set your goal as quickly validating or invalidating critical hypotheses, rather than building features, it frees you up to do all sorts of creative tests that might be digital or analog, online or offline.

Lean doesn’t let you get lost in process at the expense of important outcomes, and it forces the business leg of the tripod to join the agile party.

A lot of customer development language revolves around ensuring that you are solving a real problem, and the right problem.

But what if you aren’t solving a problem?

Lean startup principles still apply to games and entertainment apps. You have the same things to validate: user experience and an understanding of your value, who your customers are (and when certain types adopt), user acquisition, market size, and business model. You can still treat your efforts as experiments.

With an entertainment experience like a game, your initial focus should be validating fun.

You won’t get that from interviews. You validate fun by testing your game.

You *can* learn a lot from interviews. A great game designer studies fun. She plays tons of games, reads the theory, investigates who plays what games, and tries to understand why certain games are winners or losers. But even great, thoughtful designers come up with lousy games.

Where teams go wrong is not realizing that digital games can indeed be tested early.

Paper testing is a critical tool. This is a natural fit for puzzle and board games, but sometimes less-obvious games can be paper tested with a little creativity. It is worth a shot, and usually faster and cheaper than writing software.

Great teams don’t just paper test internally. They get out of their own heads and watch how other people interact with the motivation loops, with the story, and with other players.

Some games are really hard to paper test. Some things do require non-trivial MVPs. World of Warcraft, for example, would be very challenging. But you don’t need all of WoW to know whether you have something good. You wouldn’t need multiple game areas, high visual quality, the full economy, full character powers, a leveling system, or countless other capabilities. All of those features could be added once the root experience was proven to be fun.

And games, just like any other application, should use a mixture of qualitative and quantitative testing to understand user experience and measure progress as changes are implemented. For qualitative, I am referring more to guided observation of actual game play rather than interviews.

The truth about MVPs is that most people think they need a bigger MVP than they actually do. Another classic mistake is that people get caught up thinking about digital product, and they fail to think creatively about analog tests or tests that don’t revolve around *features*.

For your game, think about the smallest unit of fun.

Final thoughts:
With software, there is going to be one core experience or value that captivates your early adopters. You don’t always know what it will be going in, but you want to find it. More features is rarely the answer. It doesn’t matter what you are creating — don’t build a mansion until you have confidence in a great foundation.

In September, Fred Wilson was generous with his time and joined our NYC Lean Startup Meetup to discuss lean startup ideas and other related topics. The event was video-taped and you can browse through all the clips at the below link.

The video editor broke up the interview into smaller chunks, which hopefully you find easier to browse. In most cases, we’ve cropped out everything but Fred’s answer. It was a very interesting session and I hope you enjoy!

Fred Wilson interviewed on Lean Startup

I spent Saturday mentoring the latest Lean Startup Machine in NYC, and as always it was a great experience. I love how the weekend isn’t about hacking up a cool idea, but the much harder task of figuring out whether something is worth doing. The participants push themselves and their ideas, and there’s a strong vibe around learning and improving. Here are a few thoughts that emerged:

Someone asked me, “what is the hardest thing about doing lean startup?” For many, the hardest thing is ruthlessly questioning their exciting ideas, but that is table stakes and I wanted to go deeper. I think the hardest thing is switching to a mindset of running experiments, especially small, fast experiments. It takes practice and it is something that most people get better at over time. You commonly see teams where their first MVP took them months, the next one weeks, and the next one days (Exhibit A: Yipit).

One of the best things about mentoring LSM is seeing how small experiments can actually be (as in one-morning-long small) that still deliver actionable insights to drive decisions.

As always, I saw a bunch of teams that were getting reasonably positive responses to their tests, but were questioning if they should continue or pivot. In some cases, it was worth running more tests. For one in particular, the big red flag was that the potential customers were saying nice things about the new product idea, but they were clearly not feeling much pain over their current methods for solving the problem.

If current solutions are good enough, then expect a long battle ahead, unless you can dramatically undercut the current market price. It is damn hard to change behavior. However, I will note that fast-moving trends (such as the sudden prevalence of smartphones) can shake up behavior, so this is all very context dependent.

Dropbox is an interesting company to look at on this topic. Most of their early customers had tried various solutions and either given up or hacked out frustrating semi-solutions. Even though most industry observers might have said that the cloud storage market was already too crowded, the Dropbox team could look at people’s true behavior and see that the problem they were interested in had clearly not been solved.

Trevor asked me to walk participants through my 12 tips for customer development, but the one thing I forgot to stress was not to let the customer define the solution. Perhaps this is obvious, but the entrepreneur needs to have the vision to look deep into a problem and come up with the right solution. You don’t ask people what they want, but rather you study their behavior for what they do and what they need. As much as you can, get your interviewee talking about specific situations, not abstract feelings and concepts.

Josh Seiden gave some good, practical advice for what he called “dump and sort”, and I saw multiple teams using this to very good effect. Josh recommended that you write down your custdev observations on sticky notes or index cards — one observation for each — and then try to sort them into basic buckets or themes. It is important that you write down observations, not conclusions about what to do. If one observation needs to be in two buckets, then just copy the card.

Breaking everything down into discrete units and visually laying out everything in front of you can be a powerful exercise.

I would only add that you should try to keep in mind where the observations come from and see if this exercise can help you come up with relevant customer types. I don’t care if you do formal personas or not (ed. I personally don’t have a lot of patience for over-wrought persona write-ups), but you do want to explicitly try to understand and prioritize the different kinds of customers you are seeing.

• Lane Halley’s “I’m out of the building, now what?”
• Josh Seiden’s “Intro to Lean UX Methods”

On Saturday, I’ll be mentoring teams at Lean Startup Machine, so this morning I took advantage of the train ride into town to consider what I would love to see from each group.  The below “fill in the blank” sentences (an embedded slideshare) are intended to force someone to concisely break out their core assumptions.

It might even be an interesting exercise for a more developed startup too.  If everyone on your team had to fill this out, would they be on the same page?

If you can’t see the slides, here is the format:
1. Target Customer/User and Need Assumptions:
I believe _________________ have a need to _________________________________________.

2. Solution Assumption
This need can be solved with ________________________________________________________.

3. Business Model Assumptions
My paying customer is _____________________.
They will pay me via _______________________.

4. Competition Assumptions
My primary competition will be ______________ and ______________.
We will beat them in the market due to __________________ and __________________.

5. Customer Assumptions Part 1
My early adopters will be ____________________.
I will acquire them through _____________________________________________.

6. Customer Assumptions Part 2
I will acquire the majority of my users/customers through _______________________ and ________________________.

7. Product Assumptions
My biggest product risk is ___________________.
We will solve this through ________________________________________________.

There are two established New York tech companies who do a great job of getting close and staying close to their customers: Meetup and The Ladders.

Andres Glusman (VP of Strategy, Meetup) and Jeff Gothelf (Dir of UX, The Ladders) established regular, lightweight usability sessions at their respective companies (that’s them above giving Lean Ignite talks — see bottom for links). Every week they bring in 2 or 3 actual or potential customers to learn about user behavior and mindsets, test out new concepts, and expose current flaws in design or strategy.

I asked Andres and Jeff for their biggest challenge in running a lean usability lab. Both answers boiled down to interpreting, and thus acting upon, learnings. To Andres, the key is balancing different people’s reactions and keeping things in tune with bigger goals. As he put it, you neither want to “freak out and change everything immediately, nor dig in your heals and refuse to change anything. The trick is finding the right rhythm for integrating learnings into the dev process.”

Jeff wrote, “Tempering reactions is the hardest thing. Sometimes it’ll take 12 participants saying the same thing to push an idea through and other times one user can skew an entire session.”

I have felt this first hand, both as a team leader trying to keep a cool head and as a product designer over-reacting to a single user’s pain. It takes discipline and an open mind to handle customer data well.

Here are five tips that might help:

Jeff notes, “Nielsen always said that between 5-8 users are needed to see 80% of the big problems in your experience. Without a dedicated user researcher we don’t have the luxury of doing 5-8 tests per week as it is time consuming and cost-prohibitive. We’ve found that with 3-4 users in every week we can both knock it out in a half day and get enough insight as to what the ‘boulders in the road’ are to give us direction on where to go next.”

Even a couple users every other week is better than what most companies do, which is none at all, or the infrequent binge. The critical thing is getting a continual, lightweight process in place. There are plenty of excuses to put this off, but it all comes down to what you consider important. Just do it.

Examine your biggest risks and test accordingly. Decide what type of customer you want to learn from, and know your core questions or user tasks before they begin. Always make room to understand the customer’s mindset, context and behavior right at the start of a usability / customer development session. As Andres recommends, “with earlier-stage things you want to be more broadly exploratory, while later on you can focus more exclusively on more tactical stuff and straight-up usability.”

You do not need statistical significance if you are seeing a recurring pattern and your gut tells you there is indeed a problem. However, to have a pattern you do need more than one person!

Sometimes a flaw is exposed that you, as a product creator, are dying to fix. We all take pride in our work, and seeing a design flaw can literally, viscerally hurt. But you have to ask yourself where the problem falls in the bigger picture of what the company needs.

Don’t get self-indulgent to mitigate your own pride/pain if there are bigger fish to fry for the company. Especially in the earliest stages of a startup, don’t get lost in endlessly optimizing usability and features if you haven’t solved the bigger problem of finding product-market fit.

User feedback can clarify product debates in a very healthy way. It gives you the “why” behind the metrics you are seeing. It shifts the discussion out of the realm of opinion and ego and into reality. But what do you do if someone really does dig in their heels and becomes a total recalcitrant, refusing to participate in the process but blocking the team?

Andres says, “No matter how good a storyteller you are, you cannot convey user pain as effectively as just watching a user struggle.” If someone cannot observe live, then use video. If a decision is important enough to warrant it, Jeff recommends editing together a video with multiple clips of users laboring to solve a problem area.

Evidence is powerful stuff.

More resources:

• Andres’ great presentation on running a lean usability program (full of great advice)
• Ignite talks from Andres and Jeff
• Steve Krug’s Rocket Surgery Made Easy
• My 12 tips for customer development interviews

This post originally appeared on the Lean Startup Machine blog, with a few small changes. I wrote it with LSM’s bootcamp in mind, where in the course of a weekend multiple teams try to validate/invalidate as much as they possibly can around their startup idea (it is really something to watch). However, I like to think that these mentoring tips have pertinence outside of that weekend structure as well.

Make sure that the team can explain the fundamental idea behind their product:

• What is the problem they are solving?
• Who is the customer?
• How they are solving the problem or meeting the need?

Then you want them to priorize their biggest 2 or 3 risks and the assumptions for each. The key is making sure the team is looking across all aspects of the business:

• What is the business model? (Who pays for the product, how, and do they have money?)
• What is the market size? Are their expectations of market size realistic?
• Who is their competition and why are they better?
• What are their guesses for customer acquisition?
• Are their any technical risks to creating the solution?

For the identified “biggest risks”, drill into the specific tactics they are planning and ask yourself:

• Will those tactics deliver useful data to validate or invalidate assumptions?
• Can the tactics be streamlined in any way?
• Can you come up with any other test-tactics that would benefit their process?

Think about how you can facilitate their tactics — this is especially important when the team needs to get to an unusual type of customer. Can you make an introduction and get them on the phone with someone? Are you friends with someone who could? External people often are happy to help.

Don’t shy away from tough questions. Force the team to stress-test their assumptions and stretch their thinking.

Never put things forth as an answer or fait accompli. Do not hesitate to point out risks, competitors or precedents you have seen before, but make the team come to a conclusion themselves after reviewing their learning.

Rove around but be careful how you interrupt teams. You can often quickly tell if a team is struggling or busily productive with just a couple questions. If the former, let them execute. If they seem to be struggling, or haven’t gotten out of the building enough, then be more forceful.

The teams have a lot to do in a short period of time, so manage your interaction so that it remains high-impact and efficient. Don’t be afraid to politely but firmly cut someone off who wants to spend a lot of time explaining their great solution and all the features.

You want to keep the team moving. Sometimes it makes sense to speak to team members individually or in smaller groups so that they can divide and conquer. Prioritize and break up your own mentoring as needed — you do not need to be comprehensive in one sitting.

If you are feeling stuck, or want help coming up with more efficient tactics, don’t hesitate to pull in another mentor. No one has all the answers, and sometimes it can be really tricky to decide what advice to give. However, if you pull in another mentor, brief them first yourself and don’t make the team do the entire download process again.

Common instances when you might want a second opinion:

• When a team struggles with a pivot/leap
• When a team is internally conflicted on priorities
• When your gut tells you that their tactics will be low-return, but you don’t have better ones coming to mind

Remember that you are a mentor, not a team lead. Participants go through Lean Startup Machine as much for the process as anything. Reading about “lean startup” is fine and good, but this stuff only sinks in when you actually start to put the ideas into real practice. They might need to be pushed to get out of their comfort zone, but teach them to fish — don’t give them the fish.