A young product leader I’m mentoring asked me a common question: “How do you set metrics for a new initiative when you don’t have a baseline, nor any idea what good looks like?”
First off, it’s ok to begin with guesses. You just want to make sure that everyone knows that it’s a guess. Product managers can fall prey to pressure to have answers. This is especially hard for folks just making that shift from individual contributor to product leader. You don’t need to have all the answers, and projecting false confidence can actually hurt you. Your honesty about what is not known will build trust as long as it is matched with competence and forward progress.
So how do you make guesses?
For one thing, ground yourself in what you’re really trying to do for your customers. Stay really practical here and focus on the essence of the issue. Don’t let a need to “measure something” distract you from those ultimate outcomes. If less engagement actually helps bring about a successful customer, then be comfortable with less engagement (i.e. that’s not the thing to optimize).
Eventually you will get to a small set of predictive metrics but you simply won’t know them at the start. Still, I bet you know what you want to happen for your business. Maybe it’s a certain level of conversion and retention? Maybe it’s lift to some other part of your business? It’s ok to start there and work backwards to your inputs. For example, maybe you set a retention goal first, and then let the correlated engagement behaviors emerge from the metrics.
Side note: as a general rule, I think retention goals are often more useful than growth or non-retention engagement targets. Retention makes growth a lot easier, as long as you haven’t aimed at too small a customer segment. Lack of retention makes growth brutally painful/expensive. Yeah, been there. Not everyone agrees with this, btw. Brian Balfour for example, stresses engagement metrics (see post) but while I see where he is coming from, I think equally teams can get lost in engagement for the sake of engagement and lose sight of bigger goals.
Alternatively, sometimes you need to start with a qualitative metric that you make semi-quantitative. A classic example of this is Sean Ellis’ “very disappointed” question:
How would you feel if you could no longer use our product? A) Very disappointed; B) Somewhat disappointed; C) Not disappointed.
Sean found across many startups that if more than 40% of customers responded with “very disappointed” then you were in good shape. The benefit of that question is a baseline does exist (from Sean’s research), but it’s ok creating your own if something new fits your context better.
Lastly, zoom out and think about the big decisions that are lurking in front of you, or biggest risks to the initiative. Ask yourself what measurements could help make those decisions. Then make sure you are instrumenting for that. It’s always good to go back to basics!
Even if it feels like bullshit at the beginning, you should try to create initial hypotheses about the behavior you want to see (and why), and then set targets ahead of time. Your targets will probably be wrong, but it’s still helpful to set them. This creates rigor when you look at the actual results, even if you eventually conclude that you picked the wrong things to measure or targets to hit. Again, just make sure everyone understands that the numbers are guesses made for the purposes of rigor, not false confidence.
This is a fun stage of product development. Lean into the uncertainty!
Have another question you want me to answer? Hit me up on Twitter.
—
Top image from @gallarotti on Unsplash