How your team’s qualitative metrics act as the canary in the coal mine

There are three qualitative metrics I find interesting to use with a product team to help spot, and deal with, changes that could be opportunities or problems.

1. Retrospectives: scoring the “sprint”

My former colleagues from Edgecase used to have every product team “score” how things felt since their last retro on a 1 to 10 scale. They didn’t over-think it or try to scientifically build it up from sub-scores on productivity versus code quality versus work enjoyment. That would be too complicated. No, the score was simply a gut call from each person. Simple to ask and simple to gather. The average number itself didn’t matter as much as how it was changing. Was it getting worse? better? why? As a manager, it helps you jump on issues quickly.

2. Story point velocity

For agile teams, setting story points based on complexity has some obvious benefits. Combined with velocity, it helps you set rough expectations (not promises) for pace. It helps you spot whether your eyes are bigger than your stomach. Velocity, however, is most interesting to me as a relative metric. Like the retro scoring, the actual number doesn’t matter. How it is changing matters, because then you have to go figure out why.

The why is critically important. Example: when someone new joins a team, you usually see a drop in velocity. A little drop is to be expected and isn’t a problem. A big drop should make you question your onboarding approach. A steady drop over several months might be indicative of a build up of tech debt, but it could have many other causes as well. Use relative change as a clue for investigation and improvement.

3. OKR confidence

My favorite book this year is a short, practical book on OKRs by Christina Wodtke, called Radical Focus. One of her approaches is to have the team score their confidence in hitting a quarterly OKR on a weekly basis. The scoring is 1 to 10 (10 = 100% confidence that the team will hit the goal).

Wodtke recommends setting OKRs as stretch goals. You should start with confidence at a “5” (i.e. 50% confidence that the team can hit the goal). Every week, you track the pulse of the team — are we more or less confident this week that we will hit the goal? Again, the relative changes are really interesting and important to dig into as a manager.

When it comes to management tools, simple is usually best. What is great about all three of these tools is they are very easy to gather. It’s ok that they are “fuzzy,” because their real purpose is to be your canary in the coal mine and help you with perpetual improvement.

(Photo credit: Canary, by quimby on Flickr)