My former colleagues Jeff Gothelf and Josh Seiden have been writing a lot about OKRs and outcomes. I very publicly admit to my colleagues that I have a love/hate relationship with metrics, and a slight allergic reaction to phrases like “data driven” and “you can’t manage what you can’t measure”.
Impact goals are important, but there are times when dogmatically requiring a data-focused metric for every unit of work can really hold you back.
My previous company was a fintech marketplace that saw strong initial growth but struggled with churn. The company focused on connecting two sides of an inefficient market — in this case M&A and capital investments for small-to-medium sized businesses. Unfortunately, at the time we set this story, both sides of the market were unhappy. They were frustrated both with the product as well as the behavior of the other side.
Buyers would complain about not seeing enough deal flow, but then they would also complain about the quality of the deal flow. Bankers would also complain about quality, claiming they were getting too many non-serious “tire kickers.”
We had two problems feeding this dilemma:
- the matching engine was not smart enough to filter out the noise. In particular, the industry taxonomy was too vague to enable targeted matching.
- the pricing/revenue model incentivized the wrong behavior, fueling the unhappiness on both sides of the market.
I’m going to start with the second. The company made money by charging an annual subscription to the buy-side. The pricing model essentially bundled the number of deals you got to see into the subscription. This was good for monthly recurring revenue, but didn’t stop annual churn. It was bad for incentivizing buyers to be more careful about every deal they matched with.
Product designers know to be careful listening to stated, conscious desires and not looking for deeper, subconscious needs. This was very much in effect here. The buyer’s instinct was to see as many deals as possible (kick every tire, don’t miss a potential deal), but what they really needed was something to pre-qualify deals and weed out the noise. So, if you wanted people to be more careful on a deal-by-deal basis, what would you do? One path was to dump the subscription model and charge per deal.
But we couldn’t charge per deal because the matching engine, particularly the industry taxonomy, wasn’t good enough. Imagine: you are paying per deal, and you’re getting what you think of as crappy deals. You’d be pissed off.
So we needed to fix pricing but couldn’t because of the product. Therein lies the OKR dilemma.
We needed to change the industry taxonomy first, which was a big project, but we also suspected that it wasn’t going to change, on its own, any of our most important metrics.
The way a lot of people approach feature prioritization is to weight effort and impact (and ideally confidence/risk). This boils down to statements like: “We’ll do X and it will change our metrics by Y.” In the story, we could fix our taxonomy but once completed, that work would not change behavior. Alone, it would not move any key metrics, let alone our #1 company-level OKR of fixing churn.
So should we have done the work? Obviously my answer is yes. The taxonomy work was laddering up to a really important goal, even if it couldn’t be measured by itself with hard numbers.
I believe the pressure to tie every unit of work to a direct impact metric causes people to work on micro-optimizations, where you get immediate small lift, rather than projects of foundational import, where you might have delayed gratification but a bigger return.
That doesn’t mean that the taxonomy work couldn’t have a KR, it just meant that we needed to think beyond “do X, see number change by Y”.
Our key result could have been “unlock a pricing change”. Alternatively, I really like qualitative metrics for situations like this. For example: “70% of the customers who use the new taxonomy to tighten their matching filters tell us that they are getting better quality results.”
In these situations, you sometimes have to pause a direct connection to company-level OKRs because you need to take a few leaps forward before you’re ready.
TLDR: Don’t let an obsession with hard numbers make you avoid foundational moves. Think bigger than that.