I used to think dealing with "data" was utterly cumbersome and boring. It was something you did because you had to, not because you wanted to. After a decade or so of working with "data", I now say without hesitation: nothing could be further from the truth. Data isn't boring! It's the most under-appreciated goldmine you’re sitting on. What makes us so averse to data isn’t the data itself, it’s the processes and software that often seem specifically designed to kill our collective spirits. Like a bad teacher draining life from an amazing subject. Software is the teacher, and existing software is almost always the bottleneck to insight and inspiration. Software done well, however, can be the teacher that makes the subject come alive. Data done well is learning. It’s understanding and insight. It allows you to make the best most informed decisions each and every day, and occasionally, hit the spark of inspiration that drives a brilliant idea.
Products and Business are "Black Boxes"
I’ve seen how dozens of businesses manage their data, and the story is almost always the same: it’s mostly a black box. They expose a few "key metrics” from that box. They may have a handful of employees with ability to peer into the box, with oft-inscrutable SQL queries. Shining a flashlight towards one particular nook or cranny. Perhaps even a dedicated research or data science team that can iteratively (and painfully) ask small handfuls of questions at a time, usually below their higher-degree pay grade, and with the uncommon patience to wait, and wait.
Those making day to day business or product decisions are starved. They're relegated to obtuse tools. Trusting outmoded or narrow “key” metrics. Going through layers of indirection and overwhelmed data teams. Bust most commonly, they just opt out. When it’s your product, your business, you starve for any insight to do your job better. To understand. Yet too often we're forced to succumb to what's available, what works.
Almost all scientific discovery stems from data. Observations and insights, learnings from the "shoulders of giants". Imagine bringing similar insights from your now observable data to your fingertips. Truly understanding the behavior of your users or customers. Following threads to their logical conclusions, formulating hypotheses and ideas and then testing them -- the hurdles today are daunting:
- Large-scale systems are slow - Existing software is built for other purposes. Queries take minutes, hours, even days.
- Data is hard to work with - If you need programmer-like levels of reasoning to ask a simple behavioral question, you're operating at the wrong layer of abstraction.
- Common questions are hard to express - Databases were meant to store shipping status, not ask questions to understand customer behavior.
Imagine if every time you wanted to run a Google “query" you had to leap these hurdles. You never would! (Maybe once or twice for that critical term paper).
Unleashing the Power
Analytics didn't get this way on purpose. With each generation's problems, new solutions arose, and tools adapted to fit those problems. Transactional databases. Processing at scale. Fast pre-aggregated metrics for critical monitoring. Businesses collect and sit on so much data that no one approach from yesterday’s solutions let’s us come close to effectively analyzing it. To have true clear visibility into it. For analytics, data users are left feeling disconnected from the vast and rich data they're managing. The systems built to manage that data were simply not built for analyzing it.
We all know humans see patterns. People closest to the product, the ones who own it, are the ones with the context - and importantly - the drive to seek understanding. Most people yearn for insights from the data they care about. Their business, their department, their product. Once you tear down hurdles, their rumbling hunger for it becomes self-evident. When you make data easy, when you make it interactive, when you make it visual and intuitive, people want to make data an integral part of their daily lives. And not because they have to...