Be Suspicious of Metrics

by Giff on June 25, 2014

People love metrics. I love metrics. I love getting arguments out of opinion and into data. However, I’m also deeply suspicious of metrics, especially since I focus on new products and thus live in a world of constant change and evolution.

Some points of discussion:

New Employees

I have new employees join and ask “what targets do I need to hit to be successful?”

There are two sides to this. First, it is important to set outcome goals so that everyone is on the same page regarding expectations. Second, if you are doing anything entrepreneurial, chances are high that the performance metrics you set on Day N will be wrong at N+60, or even sooner.

I try to make a pact with the employee: if at any point, you or I realize that you’re working on the wrong thing, let’s agree to blow up the metrics and rethink. I also have a goal of reviewing the metrics, not just the results, every three months.

The same thing should go for any team. Set goals. Set checkpoints when you evaluate them. Be willing to blow them up at any point in time.

Choose wisely

You want to make sure you are paying attention to the right metrics. Obvious, right? But it’s easy to fall off the rails here. We recently launched a new business line for a client, and now have revenue coming through the door. Before the project was funded, we created a financial model that forecasted customer growth and how that might translate to revenue and profit. Once the product was live, some people started to fixate on hitting the “new acquired customer” target for each month from the original model.

But while new customers was an interesting metric, it was not the true goal. Our true goal wasn’t to have a certain number of new customers, but rather to hit certain revenue targets. We could hit our revenue goals in a myriad of ways — acquiring new customers, raising prices, reducing churn, and more. Fixating on a metric that fed into our goal, rather than our true goal, could handcuff our thinking. A young business needs flexiblity not handcuffs.

Locking Metrics in Stone

One of my current management headaches is tied to the M&A (merger and acquisition) activity that preceded my taking the helm. The company wanted to pay for performance, and so it created earnout structures and locked this into the acquisition legal agreements.

Unfortunately, this only works if you can reasonably predict how operations will unfold.

Our company, even though it is a services business and not completely re-inventing the wheel, is trying to push the edges of how consulting companies operate. We are testing out new models. Rather than improving performance, the earnout structures have turned out to really constrain us in all sorts of ways because M&A legal agreements are much harder to rip up and rethink compared to a set of employee goals.


Metrics are only as good as how you intepret them. Some are clear cut, but in the early days of a product, you often don’t know what “good” vs “bad” performance really is.

The Internet is awash with information, but it is not awash with comparables. Take conversion rates, for example. In one project, we are closely looking at our new customer conversion rates. We know if we are making progress, which is powerful, but how do we know if something is good? What do we have to compare it to, other than our previous numbers, our experiences on other projects, and perhaps some whispered shared metrics from peers at other companies.

The intepretation challenge doesn’t stop just with lack of comparables. People can also slice up data in ways to confirm or deny their own opinions. Once upon a time, I did M&A and IPO investment banking for tech companies. We always laughed that every single bank was “#1 in tech”. Everyone just sliced the numbers in a way that made them come up as number one. Every claim was true — everyone was number one at SOMETHING.

Final Thoughts

I like that the startup world is trying to rely on metrics more than pure gut alone. I just think that some people are a bit too caught up with the idea that we need to measure everything, and let metrics rule all.

There needs to be room for judgement and flexibility.

Does Occulus Mean That VR Is Finally Here?

by Giff on March 29, 2014

Occulus Rift and better immersion rigs have allowed the promise of virtual worlds to rear its head again. Raph wrote a great post the other night, and I feel compelled to weigh in as well.

Once upon a time, I was a near-expert in the space, obsessing about it from 2004 to 2008. I was also admittedly slightly bitter about it for a few years after the fall and Electric Sheep’s (our startup) drawn-out collapse. Because the third wave (there had been two runs at it before) of virtual worlds did fall, and while the games industry stayed strong, only Second Life (SL) remained standing, holding on to life as a creative/social MMO.

When I was bit by the bug, I was bit big, as you can tell from this video I wrote and shot in mid-2007:

I remember back to the Metaverse Roadmap session in 2007, which Raph alludes to in his post. My worry then was that VR would not become a reality until our immersion tools became near-invisible.

What I realized later is that it will require more than that:

  • great immersion capabilities (we seem to be getting closer)
  • natural-feeling “input/control” mechanisms (is fluid movement enough? or do we need facial expressions as well, which is what Neal Stephenson hypothesized?)
  • scalable environments (SL could never handle very many avatars together, or high levels of physics, without terrible lag)
  • more extensible environments, allowing for sophisticated 3rd party software development on top of a metaverse platform, necessary to create rich experiences (what we finally have on the Web with rich browser-based applications)

Social acceptance is also a tricky issue. During my years focused on this, I acquired a deep knowledge of why people fell in love with virtual worlds, and how they acted inside of virtual worlds. What I didn’t do was deeply study why people weren’t falling in love with virtual worlds. While I knew that virtual worlds would have an adoption-curve chasm crisis, I misjudged how long it would take to solve the technical and experiential problems. How about the social ones? Heck, in Ready Player One, Ernest Cline hypothesizes it taking a global energy crisis to drive people into the virtual world OASIS, and indeed OASIS is positioned as a crutch that distracts humanity from its real problems.

When we realized, at the end of 2007, that SL wasn’t going to be the metaverse platform we dreamed of, we turned to browser-based technologies, but that didn’t work either. I walked away thinking that it would take 15 to 20 years for the pieces to finally be in place. It has only been 7. While Occulus is exciting, I wonder how much progress has been made on those other issues.

When we all met in early 2007 for the Metaverse Roadmap, I remember Randy Farmer being the grizzled, cynical veteran in the room. He had created Habitat in the mid-80s, the first stab at a virtual world. At the time, we were the bright and shiny newcomers. We said, “this time it is really going to happen!” It didn’t. And so I became the grizzled, cynical veteran. But that doesn’t mean that it won’t happen.


Lean startup is like a sine wave

by Giff on March 27, 2014

There are a lot of people trying to do “lean startup” or “lean UX” and fretting about whether they are doing pure lean (there are even more who talk about lean but don’t actually do anything close, but that’s a different story).

There is no such thing as pure lean. The right balance takes into account what you need to accomplish and learn, but also your context and constraints.

What you shouldn’t feel, in my opinion, is a need to be learning every second, or a need to test everything. You’ll end up being way too inefficient.

And when you do focus on the important things, I suspect that you’ll find that lean feels a bit like a sine wave.

You create an experiment and learn. Then your learning curve starts to flatten. You need to absorb and interpret your data. You need to make decisions and put your decisions into action. And then at that point you are ready to start a learning cycle again. Out of the building, then inside the building, then outside again. Learn, build, measure, learn, rinse repeat.

What Pixar Got Right

March 27, 2014

Pixar is an inspiring place. In addition to prioritizing storytelling, they got a lot right when it came to process and human interactions.  Fast Company has a nice piece interviewing Pixar alumni, and I pulled out my favorite bits: “There’s still a back and forth between creative and the audience, and you can’t be like [...]

Read the full article →

What Lurks Beneath the Surface

March 23, 2014

I’ll believe it with my own two eyes Humans have a tendency to focus on what they can see to the exclusive of other things. Many years ago, my wife and I lived in a shoebox in Manhattan. We couldn’t afford to buy a place in the city, but ironically we could afford to rent [...]

Read the full article →

You Are Spending 3x-5x More Than You Should

March 12, 2014

In carpentry, the rule is “measure twice, cut once” because once you cut, you can’t go back. If you get it wrong, you end up wasting a lot time, money, and materials. Somehow, that concept has not made it into the vast majority of software projects. Even though “lean startup” has become a popularized term, [...]

Read the full article →

What Is A Product?

February 23, 2014

The title is a question we have been kicking around at Neo for the last few weeks. I often refer to us as a product agency, as opposed to digital agencies that do marketing websites and campaign work. But what is a product? Here’s an initial stab. A product is a repeatable capability that delivers [...]

Read the full article →

Creating, Improving… Managing?

February 15, 2014

I bet most of you have listened to Ira Glass on being creative. If not, ignore this post and go watch that video. His comments resonate with me deeply. Frustration with not being able to always produce work to the level of your taste. Demanding more. Overcoming that frustration through perseverance and relentless creating, making, [...]

Read the full article →

The Misogyny Has Got to Stop

February 3, 2014

This weekend I was startled by a tweet from Margaret Wallace, a game industry executive whom I respect in New York. Welcome to my world: She Was Harassed By A Games Reporter. Now She's Speaking Out. #onereasonwhy — Margaret Wallace ♕ (@MargaretWallace) February 1, 2014 Margaret is referring to a series of unacceptable, appalling [...]

Read the full article →

Myth-Making and the “Authentic Founder”

December 19, 2013

One of my pet peeves about startup-land is myth-making. I refer to the stories that get made up about a startup’s founding and trajectory. Entrepreneurs spin tales for two reasons: to get press, and, particularly, to get funding. Investors have a name for their intuitive filtering — they call it pattern recognition. But at the [...]

Read the full article →