"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…"
"It's a people problem." "We just need more data literacy." "The tools are fine; it's the implementation that's wrong."
If you've ever voiced frustration about your company's data infrastructure, chances are you've heard these responses. After speaking with hundreds of data professionals across industries, we've noticed a pattern: when confronted with the limitations of their data infrastructure, people often fall back on these excuses. But there's something deeper happening here. The people telling you these things might be unintentionally misleading themselves too.
We've accepted a troubling premise: our struggles are not because we're using approaches that don't match our needs, but because people lack the right skills or understanding.
"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…" - Dan Ariely
That quote was from over ten years ago, and "big data" is out of fashion as a term, but the reality of what he was joking about is as true as ever.
Organizations assume others have figured it out, so the problem must be internal. The truth is that almost no one is happy with their data. Companies face the same challenges again and again and again regardless of their team's expertise.
The pattern:
Self-blame prevents organizations from questioning their fundamental approaches.
When we frame data problems as human problems, we respond with more training, new roles, or organizational changes. Meanwhile, nothing changes and the data challenges persist:
This affects companies of all sizes across industries.
Consider an alternative view: What if the prevalent approaches to data infrastructure simply don't align with how most businesses actually operate and evolve?
The reality is that current approaches reflect either outdated assumptions or borrowed patterns. Many data architectures were conceived when storage was expensive, data volumes were smaller, use cases were more predictable, and business changes happened more slowly. Others are patterns adopted from very large tech companies who had more narrow use cases with large data volumes and less fundamental change in their day-to-day business operations.
Most organizations aren't Google or Facebook. They're dynamic businesses facing constant change, with diverse data needs that span departments and functions.
The first step toward improvement might be questioning some fundamental assumptions about data infrastructure. Perhaps it's worth considering approaches that:
The next time someone suggests "it's a people problem," consider whether the underlying fundamental assumptions might be creating unnecessary friction. Your teams are likely doing their best within the constraints of the tools and approaches available to them.
What we need isn't just more training, new company organizations or more complex tools, but approaches that better reflect how data actually flows through modern organizations.
