Metrics aren't "Magic"

Bobby Johnson

There's a legend from Facebook about how data saved them. The way it's usually told is that the "data showed them" that a user would stay forever if they got seven friends in their first ten days, so armed with this new lightning bolt from the data gods they quickly grew to a billion users.

If you've been collecting data and wondering why you haven't had this sort of magic lightning bolt handed to you from the gods, don't feel bad, it's not really how it happened at Facebook either.

The data was enormously important to our growth, but it was a much more complicated story that also involved a lot of human ingenuity. It's a great story of how to use data effectively - the real story is much more useful than the myth of the magic metric.

It wasn't the metric that was "magic", it was that someone found a correlation between early friend acquisition and long-term retention. This was mostly good intuition from a person who understood the product and the users. Then we looked at data to check this intuition, and to know we were on the right track.

We also looked at the data for a bunch of other things that sounded like good intuition but turned out to not be correlated. Note that there was never any evidence of causation, and certainly not any "magic", but there was a plausible case for causation, and it was the best we had so we went for it.

The "magic metric" with the specific numbers had nothing to do with data - it was a management device. It was a simple straightforward goal to set the team at. It was an enormously important part of our eventual success, but it had everything to do with getting people rallied behind a goal, and very little to do with data.

The real value of data came after we chose the goal. Once we had a goal we tracked it every day, cut it by tons of different dimensions, made lots of hypotheses, and carefully measured the results of our changes. An important factor for success here was how quickly we could turn around experiments. We put a lot of effort into tooling so we could go from idea to test in a day or two, rather than a week or two.

Previous article Blog Summary Next article