A data pipeline is a set of choices. It's almost always a bet that you know how this data will be used, how it needs to be modeled, how it needs to be queried. I've watched a data engineers spend weeks building ETL pipelines. Then the
“Reverse ETL.” An entire category of tooling to acknowledge that data flowing in one direction is right, and natural. That moving it the other way requires a special designation. These patterns are so deeply embedded in our data architectures that we can no longer see them. The pipeline is the
For decades, the language around data has barely changed. Every few years a new architecture or philosophy rises. We hear about data lakes, warehouses, meshes, fabrics, and observability platforms. Each is a promise to finally tame the chaos of data management. Billions have been invested across multiple generations of tooling
I wrote a post about thinking past medallion architectures. That one went a little deeper about the architectural characteristics that make thinking in “medallions” unnecessary. You don’t need to internalize all that. I’m guessing you sense that data just doesn’t work, even with the fancy medallion architecture.
Let’s talk about something nobody wants to admit. Your marketing team has their own copy of customer data. Sales has a different version. Product is maintaining yet another extract. Finance built their own dashboard using data they pulled last month. Each team has created their own shadow copy of
Picture this: You’re in an executive meeting. The company just acquired another business, and the CEO wants to change how you calculate monthly active users to include the new customer base. Simple request, right? “That’ll be six months,” comes the response from the data team. Six months?! To
I was talking to a VP of Engineering recently who told me something that perfectly captures the current state of data infrastructure: “We just kind of accept that this is the problem. And a lot of people don’t even realize that you can do it a different way.” He
A ritual playing out in boardrooms everywhere: the new data strategy presentation. This time it’s different. This time we’ll capture the value. The slides are beautiful. The architecture diagrams are comprehensive. Everyone nods. Nobody believes it. We’ve been here before. At least three times, in fact. The
December 4 at 11 a.m. PT / 2 p.m. ET Six-month data migration timelines. Projects on hold. Costs climbing. What if you could test your migration in parallel with production? Roll back mistakes without losing work? Compress months into weeks? Join us December 4 to see how Matterbeam
