Alex Danco publishes an excellent newsletter Two Truths and a Take and today’s thought provoking essay is all about the limitations of having hypotheses: they can close our minds off to creativity and discovery. Danco writes, “when we analyze the results of an experiment, our mental focus on a specific hypothesis can prevent us from exploring other aspects of the data, effectively blinding us to new ideas.”
This is a common dynamic, and why I tell teams to start with vision before thinking about experiments. You can’t A/B test your way to something great. Vision work also can’t just be at the beginning of the process. You have to constantly return to it, armed with new data and insights.
Product work is filled with these dual modalities:
- When brainstorming, it’s best to diverge then converge.
- In doing customer discovery, there’s a natural on-off cycle: learn; then when the patterns are clear, stop, absorb, and act; then start learning again.
- Same goes for learning versus shipping — if you spend too much time in either, at the detriment of the other, you either accomplish nothing or accomplish the wrong things.
- While hammering on risks and hypotheses, you also need to find time to stop and let your mind play, imagine, and get creative with what you are learning.
If you spend too much time in the analytical, or conversely too much time in the imagined, you fall short. Danco talks about how, in science, the (oddly named) “day science” and “night science” are complementary. The imagination comes up with bigger ideas, some of which sound brilliant in the moment but are really mirages. Experimental rigor then helps you winnow out the good ideas from the foolish ones.
It’s a little harder in product because we’re dealing with humans, not laws of nature, but the concept still applies.
Danco then shifts into a discussion about data and causation versus correlation. He writes, “Patternicity helps us in generating new night science ideas; it is the seed of many discoveries. On the flipside, patternicity makes us vulnerable to being fooled by randomness… In a sense then, correlations are the domain of night science, while causation is solidified by day science. Day science is the adult in the room, rigorously testing hypotheses. But despite its power, the day science mode is not amenable to generating the ideas in the first place.”
Once again, it’s all about having balance in the Force. Too much reliance on data OR intuition leaves you vulnerable. My worry about the craft of product is that, justifiably excited about the power of data science and machine learning, we’ve put it a bit too much on a pedestal. The best approach is to use data but to also be deeply skeptical of it. Data sets have bias, both in terms of how the data is collected as well as the gaps that we fail to see or understand.
An example: I was working on a business marketplace and the data science team discovered that increasing interpersonal connections on the network increased the retention rate. It was deemed an important leading indicator and the business invested a lot of energy in maximizing connections.
What the data didn’t show, but what even a tiny amount of qualitative research would have uncovered, is that the real goal of the participants wasn’t networking solely for itself. Participants wanted to close more and better transactions (that was the true goal), and the old school way of accomplishing this was shaking a lot of hands. We had an opportunity to use technology to rethink those old ways — not to do away with them necessarily but to augment, complement, and improve upon them.
The data had revealed a clear pattern. It didn’t lie — relationship building was and is important — but it wasn’t the whole truth, and it didn’t reveal a deeper understanding of our customer’s true goals.
There are times when AND, rather than OR, is dangerous: not making a hard decision; trying to do too many things at the same time. But even so, duality is an important concept in our quest to make great products. Qual and quant. Testing and intuition. Data and imagination. Diverging and converging.