A few years ago, I wrote about the Truth Curve, and refined those thoughts later in Talking to Humans. Essentially it states that the believability of information you receive from market tests increases as the fidelity of your product test increases. You should not wait until you have a live, instrumented product in the market, but nor should you take early signals too literally.
I used to think about this order:
Custdev Conversations > Landing Pages > Paper Tests > Prototypes > Concierges > MVP
Fast forward to 2015. When it comes to validating an idea, now all I really want to do is:
Custdev Conversations > Concierge > MVP.
You can invalidate an idea with customer development, but you cannot validate it. But you still do customer development to gain deep insights.
You then put people through some sort of experience and watch what they do. Then you interview them to dig into the motivations behind those actions. Concierge is a good placeholder name for this because it reminds you to think about a delightful customer experience, but not one that needs to scale to many people.
Landing page tests are overused and, in my opinion, uncreative. I’ve seen quite a few in action, and I don’t think you get much useful, believable information. I would much rather see a team figure out a more creative experiment.
Would I ever put up a landing page? Yes, but not really as a test for value or market demand.
It is very useful to generate a waiting list of people to tap for better experiments or your MVP.
Testing advertising conversions is also useful. Again, not so much for testing the value of your business, but for exploring customer acquisition costs and effectiveness.
Another thing on that truth curve was paper tests. They also have their uses, but not in validation. Paper tests help inform design decisions (not unlike card sorting exercises).
So don’t be lazy about “lean.” Constantly ask yourself whether the way you are testing something is as creative as it can be, and as believable as it can be, while still allowing you to move fast and scrappy.