(updated version from Testing with Humans)
I wanted to share one of the slides from my talk at QCon last week. I call it the truth curve.
On the X-axis is your product sophistication. On the Y-axis is how much you can believe your learnings versus having to strongly filter your results through your judgement and vision. Different types of experiments have different levels of “believability.”
When you are working on a new product, you want to get out of the building early and start learning. If you aren’t “out of the building” at all, you are in fantasy land and most things should be viewed with questioning and skepticism.
On the other end of the spectrum, real truth comes in the form of a product live in the market, and seeing the actual metrics. You still need qualitative research to understand why things are happening, but the quantitative metrics are your reality.
However, you don’t want to wait until you have your live minimally viable product (as in an actual product) to learn.
You can start with qualitative customer development. You can run lightweight experiments such as paper tests, landing pages, pre-selling, concierge (be the software) experiences, or wizard of oz (be the software, but behind a UI veneer) tests.
The more “experiential” the test (i.e. you are observing behavior as someone goes through an actual experience), the more you can believe the results. Conversely, the earlier you are in the curve, the more you need to filter your learnings before drawing conclusions.
No matter where you are on the curve, you will always need to exercise judgement and rely on vision.