And we're all just pretending it's going to work out fine
A data scientist told me something that perfectly captures our industry's delusion: "I spend most of my time wrangling data. I can't trust my data engineers because they don't know about data leakage. I want to see the data myself in its raw state."
This person should be finding insights that transform businesses. Instead, they're a glorified data janitor. And we've all agreed this is normal.
Meanwhile, we're pouring billions into AI initiatives built on this same broken foundation.
Every company is racing to implement AI, but nobody wants to admit their data infrastructure can't support it. So we've created this elaborate fiction where everyone pretends their duct-taped systems will suddenly handle the most demanding workloads we've ever thrown at them.
Consider the countless ML platform partnerships with data warehouses that never materialized. The ML vendors needed direct compute access to be effective, but the warehouse sales teams were only compensated on storage and query credits. These partnerships made perfect technical sense but died because one side needed to sell capability while the other needed to sell consumption.
Multiply that dysfunction across every vendor, every integration, every pipeline in your stack. That's the foundation we're building AI on.
Data scientists hate their jobs. Not the science part - the part where they spend 80% of their time fixing data broken by engineers who don't understand the domain.
I heard about a company demoing a data tool. They did a fancy join operation and everyone was impressed. Then a data scientist said, "You just created data leakage. We cannot combine those pieces of data." Everyone else was confused.
The people who understand data science can't trust the people building data infrastructure. The people building infrastructure don't understand the constraints that matter for data science. But we're pretending this works for AI initiatives that depend on both groups.
We've normalized complete dysfunction. Data requests take weeks? Standard. Migrations are two-year death marches? Expected. Getting access to your own data requires tribal knowledge, three approval processes, and a six-demon bag? Tuesday.
The insidious part: this acceptance isn't just cultural, it's economic. Companies budget for the dysfunction. They hire armies of engineers to maintain systems everyone knows are broken. They pay vendors millions to manage complexity that shouldn't exist.
We've created an economy around accepting that data infrastructure is inherently broken.
The current vendor landscape depends on this dysfunction. Integration platforms wouldn't exist if data moved cleanly. Data cleaning tools wouldn't be billion-dollar markets if pipelines preserved quality. Monitoring tools wouldn't be essential if systems were predictably reliable.
Every vendor has an incentive to solve just enough of the problem to be valuable, but not so much that they eliminate their market. Snowflake wants everything in their walled garden. Even when partnerships would help customers, business models fight against each other.
AI doesn't just depend on data infrastructure - it amplifies every flaw. Bad data quality becomes exponentially worse when AI learns from it. Pipeline brittleness becomes catastrophic when AI needs real-time feeds. Coordination problems become impossible when AI needs data from multiple sources.
Instead of fixing the foundation, we're doubling down on automation. AI agents to manage broken pipelines. ML systems to clean dirty data. Automated monitoring to alert us faster when things break.
We're using AI to paper over the same problems that broke data infrastructure in the first place.
Companies make billion-dollar decisions based on data limitations they think are laws of physics. They abandon transformative opportunities assuming the data work would take two years. They accept competitive disadvantages believing certain integrations are impossible.
These aren't laws of physics. They're just the current state of tooling we've agreed to accept. This creates a feedback loop where business strategy gets constrained by infrastructure limitations, which justifies not investing in better infrastructure.
Companies will launch AI initiatives on these broken foundations. Some will work despite the infrastructure. Many will fail spectacularly. A few will succeed and everyone will copy their approach without understanding that success came from working around problems, not solving them.
The companies that fix their data infrastructure first will have enormous advantages. Not because they have better AI models, but because they can actually use their data.
But most won't do this. They'll keep accepting dysfunction, paying vendors, hiring armies of engineers to maintain systems that shouldn't need maintaining. Because admitting the foundation is broken means admitting billions in investments were based on flawed assumptions.
The most provocative thing about our industry isn't that bad companies sometimes succeed. It's that we've all agreed to pretend fundamentally broken systems are the best we can do.
We've collectively decided that data infrastructure is inherently complex, unreliable, and expensive. And now we're building the future of business on this shared delusion.
The question isn't whether this will collapse under its own contradictions. The question is how much value gets destroyed while we all pretend otherwise.