How Data Access Informs Big Product Changes

Aditya VempatyMarch 29, 2017

LinkedIn organizes your network by degrees. Your 1st-degree connections are the people you’ve invited, or who have invited you to connect. Your 2nd-degree connections are connected to them, and connected to them are your 3rd-degree connections.

At one point, you couldn’t see the full profile for any of your 3rd-degree connections unless you paid for a LinkedIn membership. The surprisingly complex ruleset behind this feature was annoying users and developers alike. Then the executive team decided to look for another solution. The proposed visibility model was a monthly use limit for unpaid users, with a cut-off based on usage. LinkedIn’s decision scientists simulated the effects of this change, using historical behavior to predict the impact on revenue and engagement.

According to Daniel Tunkelang, a former LinkedIn executive, the effect of the decision to open up 3rd-degree profiles to unpaid users (with a cap on usage) was hugely beneficial. A lot of people got value from it. A lot of people started to see the value that a premium membership to LinkedIn might bring. And yet this was precisely the sort of experiment that might never have happened had the data to analyze its consequences not been available.

The #1 fear people have about data

People presume that a business is at its most creative when it’s just starting out. There’s no data to influence what you’re doing, just your gut intuition.

With success comes revenue, users, and data. But many entrepreneurs are wary of becoming too reliant on data out of the fear that the numbers will begin to drive everything.

It’s one of the classic cliches about being “data-driven.” You start to make the wrong decisions, pursuing only those opportunities that can be seen on a spreadsheet. You allow no decisions to be made without the data to back them up. Pretty soon, having missed the forest for the trees like Xerox in the 80’s, you end up relegated to the ash heap of history.

Having access to data about your business does not, however, have to mean spending all your time and energy on small optimizations—nor does it mean not building a roadmap or having a strategic vision. Used correctly, data can help you build better roadmaps and a clearer strategic vision.

It’s not what you do, it’s how you measure

At LinkedIn, the key to measuring the true impact of their massive UX change was combining historical, adjusted data with real-time data from experimentation.

If that sounds complicated, know that even the simplest-seeming experiments can have highly complex measurement implications.

Wistia CEO Chris Savage had been experimenting with a type of advertising his company hadn’t tried before: billboards. The data said it was a flop. But when he went out and talked to people, he found they (qualitatively) loved the billboards featuring Wistia’s office pooch, Lenny.

Measurement is really hard—and unfortunately, when it comes to data, measurement is everything. The way you analyze and take stock of your data is what matters.

The VC Brad Feld elegantly explains this point in the context of his experience on the board of a certain company:

“A long time ago, I realized that every successful business was a continuous process of small experiments that operated in the context of a long-term vision. When an experiment worked, you did more of it. When it didn’t, you ended it and moved on.”

Sometimes the data isn’t going to help you deliver value to your customers, increase revenue or retention—but when that happens, you account for it and incorporate it into your vision.

When something does work out, but the result isn’t exactly in line with your long-term vision for the company, it’s an opportunity to reflect on just what your long-term vision is. Perhaps the data is telling you that there’s something wrong with your vision—perhaps they can bring, as Feld says, “strong suggestions about better approaches.”

Building a data-informed culture

The most important way that data access comes to inform large-scale product changes is through its impact on culture.

At Imgur, Interana doesn’t just give product teams the tools they need to understand how first-time users experience Imgur. It makes them think about it on a regular basis and gives them the ability to have informed conversations about it with other people in the company. Even their business team can use Interana to access “user patterns and journeys,” and “public relations and media-related traffic data.”

Before you begin to worry about whether you’re hewing too close to your data with your decision-making, check out whether your entire company even has access to data. Chances are it doesn’t.

At Teambition, the story is similar. Data, far from shutting down the conversation, allows great ideas to come up from all over the organization. Rather than rely on the HiPPO (highest paid person’s opinion), data allows them to “solve disputes and clear confusion” in an objective way that doesn’t neglect anyone’s opinion.

The data boogeyman

There are lots of surface-level analyses out there that would have you think that using data is cheap or over-optimizing, or that it’s inevitably going to lead you up a “local maxima” rather than guide you to a better product.

Many of those analyses, however, are based on false premises—on organizations and companies where data is siloed away or made the responsibility of a single team or group. In those companies, data isn’t used to its full potential as a cultural tool, and data gets the flack.

If you’re interested in seeing what a truly accessible analytics solution looks like, try the new live, public demo of Interana today.