The medallion architecture emerged as the data industry's answer to data lake chaos. Organizations had dumped vast amounts of raw data into cheap storage, creating impenetrable data swamps. The medallion's promise was elegant: three progressive layers—Bronze for raw data, Silver for cleaned data, Gold for
I talk to data professionals and they're frequently frustrated. For example spending three months migrating everything to Parquet files in their data lake. Clean, columnar, compressed. Beautiful. But now their real-time service team needs that same data, and now it's painfully slow because, well, scanning columns
You know what you're supposed to do. We've heard the same refrains for a decade or more. Conference keynotes. Blog posts. LinkedIn thought leadership. Build a data culture. Invest in data literacy. Improve data quality at the source. Get executive buy-in. Implement strong governance. Focus on
There's a version of a stat that gets thrown around a lot. Data teams spend 80% of their time on data preparation or cleaning. Eighty percent. We've just... accepted this? Like it's some law of nature? As if the universe decreed that for every
You know that feeling when you've been doing something the same way for so long that you can't imagine any other approach? That's where Josh Pendergrass was when his company first started using Matterbeam. "At first I was like, that seems great. I
And we're all just pretending it's going to work out fine A data scientist told me something that perfectly captures our industry's delusion: "I spend most of my time wrangling data. I can't trust my data engineers because they don'
Data problems in companies are not due to people's lack of skills, but from the wrong fundamental approach that doesn't align with how businesses actually operate and evolve.
The developments of generative AI in the past few years have been amazing. There are promises of AI agents revolutionizing everything from customer service to software development. Now they're coming for your data infrastructure. The pitch is seductive: AI agents will automatically handle data integration, quality, and governance.
The lakehouse idea springs from a common pain point: warehouses excel at handling structured data and delivering strong analytics performance, but they falter when faced with unstructured data and scalability challenges. On the other hand, lakes shine with unstructured data and flexibility but struggle with governance, consistency, and transactional integrity.
