“There’s a legend from Facebook about how data saved them,” Interana CTO Bobby Johnson wrote on this blog last June, “The way it’s usually told is that the ‘data showed [Facebook]’ that a user would stay forever if they got seven friends in their first ten days, so armed with this new lightning bolt from the data gods they quickly grew to a billion users.” Said Bobby:
If you’ve been collecting data and wondering why you haven’t had this sort of magic lightning bolt handed to you from the gods, don’t feel bad, it’s not really how it happened at Facebook either.
Growth is a discipline where wisdom is largely passed down through oral tradition and blog posts. Sometimes that means we wind up playing a game of Telephone with the truth.
In the case of Facebook, what was truly a multi-year process of intuitive judgment, exemplary people management—and yes, data analysis—became reduced to an “Aha! moment.” The tale of “7 friends in 10 days” became an overly-simplistic rallying cry for the idea that startups should be data-driven, rather than relying on intuition, but this interpretation glosses over some of the crucial facts of the story.
It’s better to use data than to not, of course. But at Interana, we prefer to talk about being data-informed rather than data-driven. The difference isn’t just semantic.It's time to stop being “data-driven.” Click To Tweet
The difference between driven and informed
Every massively successful tech company from Google to Uber has found their competitive advantage in data.
From the tracking of fundamental metrics like revenue and burn rate to KPIs and success metrics that vary between teams, organizations today need data to guide decision-making. That’s table stakes. The thing that separates great companies from the rest, however, lies in how much of the thinking they do themselves and how much of it is left to the data.
This is the fundamental difference between being data-driven and being data-informed:
- Data-driven: You let the data guide your decision-making process
- Data-informed: You let data act as a check on your intuition
When data guides your decision making, you’re always collecting more of it to have the most accurate models and find trends. But here’s the catch—data is never going to tell you the full story, and it’s never going to tell you exactly what you need to do next to keep growing. Too many companies think data should be their only guide and wind up over-optimizing small things rather than developing a bigger product vision.
Being data-informed means using both intuition and data to come up with testable hypotheses about your product. The qualitative complements the quantitative, and vice versa. Data is part of a larger context, one which acknowledges the possibility of bias in both its collection and analysis.
People assume that the process at Facebook was totally data-driven—that “7 friends in 10 days” was a capital-T truth, which after being discovered, set the agenda for the entire company. It’s thought that it was part of the logic of Facebook itself, a physical law underlying its growth, and once it was found, Facebook’s fate was all but sealed.
The reality is much messier. The work at Facebook was data-informed—not data-driven—in that the key insights came from the intuition of people who knew the product and its users.
Why you need to be data-informed
The early growth team at Facebook got started by closely observing those users who stuck with Facebook and kept using it over the long-term. While looking for events correlated with that outcome, they stumbled across a connection between early friend acquisition and retention.
It wasn’t causal, but there was a plausible case for a link—Facebook was all about connecting to your friends, so it stood to reason that users who connected to more friends would see the value in Facebook (and stick around). It was the best metric they had, so they went for it. Management got everyone rallied around the idea of getting users to “7 friends in 10 days” and started tracking the team’s progress. They “cut it by different dimensions and made lots of hypotheses,” investing time into tooling and measurement to both make changes quickly and see the effects of those changes.
There was no lightning bolt, just an observed correlation that provided a signpost for the entire company to follow. As long as they monitored their progress with data, they could know they were growing.
But the “7 friends in 10 days” metric isn’t what told Facebook what kinds of experiments to run on their way to a billion users and beyond. They knew what to do because of their great intuition for the product and its users. “Knowing true product value allows you to design the experiments necessary so that you can really isolate cause and effect,” as Chamath Palihapitiya has said of Facebook’s early growth and experimentation.Being data-informed means using both intuition and data to come up with testable hypotheses.Click To Tweet
How data-informed experimentation works
At one point, when you searched for listings on Airbnb, you got a page like this:
The design-minded folks at Airbnb redesigned the search results page as the site grew in popularity:
It’s not hard to argue that the new design looks nicer—it makes greater use of listing photos, expands the map for easier searching, and minimizes the influence of aspects like the copy attached to each listing.
When they analyzed the results of the rollout, they found that the new design was actually performing worse than the old one.
As Airbnb notes in their post on experimentation, it’s valuable to have a single global metric that you assess experiments against—it keeps you from looking for results that validate your biases by “cherrypicking” the right evidence. This case, however, demanded deeper context and more information—the results were that strange.
When the Airbnb team looked into the test further, they found that the new design actually was performing better—as long as you weren’t looking at the site on Internet Explorer. On IE, some crucial elements of the website were broken by the new design. Once fixed, the data started to reflect improved performance on Airbnb’s core metric.
Who needs to be data-informed?
Being data-informed was critical to getting the redesign out to users in a way that improved their experience with Airbnb. Data could not generate the idea for the redesign in the first place, nor could it (on its own) make clear why it initially seemed to perform poorly. Human guidance was needed for that. Data was there merely to help.
This doesn’t mean you can relegate all data-related tasks to a single product manager or data scientist. On the contrary, it makes it ever more important that every member of your team be fluent with data.
You need to be data-informed when you’re first interviewing customers and trying to get a better feel for the problem your company solves. Scoring your potential customers along dimensions of enthusiasm, intent, and use case isn’t about turning customer development into a soulless process—it’s about coming away with easily digestible insights.
You need to be data-informed when doing customer support. Without metrics that incentivize the members of your support team to focus on the customer, they’ll focus on getting good ratings, answering quickly, or finding the easiest tickets.
You need to be data-informed when doing design and UX. Data will never be able to tell you what your product should look and feel like, but it can help you assess whether a particular design is helping your users reach their goals or not.
When your organization is data-informed, everyone in it can derive value from data.When your organization is data-informed, everyone in it can derive value from data.Click To Tweet