Between a Rock and a Cloud Bill

You’re Doing ETL Wrong – Here’s a Better Way

If you’re still managing your ETL/ELT pipelines like it’s 2015, it’s time to rethink your approach. The recent pricing change by Fivetran is just another glaring example of why the current trend in data integration is broken—and how you’ve been overpaying for years.

Here’s the kicker: you’re already paying Snowflake (or another warehouse) for compute to run your transformations. Now, Fivetran and dbt want you to pay them for scheduling and executing the same transformations. That’s right: triple charges for a single transformation pipeline. How did we get here, and more importantly, how do we fix it?

The Pain Points of the Current Model

1. Double Charging for Transformations

Fivetran doesn’t focus on advanced data transformations itself, so it partners with dbt to make it easy to write SQL transformations and execute them downstream (e.g., in Snowflake). But starting January 1, 2025, Fivetran is charging separately for Fivetran-hosted dbt Core transformations and Quickstart Data Models. So now you’re:

• Paying for Fivetran-hosted transformations.
• Paying dbt Cloud for scheduling transformations (if you use it).
• Paying Snowflake for compute to actually run those transformations.

This is the ELT equivalent of death by a thousand cuts.

2. Punishing Small and Medium Businesses (SMBs)

Small businesses with modest data volumes are disproportionately affected by this pricing change:

• Free Tier Limitation: The free tier allows up to 5,000 model runs per month, but even simple dbt projects often have hundreds of models.
• Cost Burden: Exceeding 5,000 runs means paying $0.01 per run, which can quickly outpace the cost of Fivetran’s core data replication service.
• Forced Optimization: SMBs will need to:
  • Optimize workloads, requiring engineering time and resources.
  • Self-host dbt Core for cost control, which adds infrastructure and operational complexity.
  • Migrate to alternatives like dbt Cloud or Orchestra, requiring engineering resources.

For many SMBs, avoiding these costs means staying locked into Fivetran’s pricing model for the sake of convenience.

3. Broken Data Practices

Today’s data teams are stuck with antiquated practices born in the on-prem era:

• On-Prem Legacy: Back when systems were hosted on-premises, data was moved to faster systems using batch ETL tools. ETL tools charged for the “T,” but unused compute on target systems made ELT the new standard.
• Cloud Era (Not Serverless): ELT persisted in the cloud era, because teams still were paying for and had spare compute resources they might as well have been using.
• Serverless Opportunities: With serverless compute, users only pay for what they use. This resulted in a rise of new tooling in the data movement and transformation space so there’s no longer a reason to push transformations out to expensive target systems or pay extra.

A New Way: PTL (Persist, Transform, Load)

Let’s face it: ETL/ELT pipelines as we know them are outdated. At Matterbeam, we’re pioneering a PTL (Persist, Transform, Load) approach that redefines how data is managed and transformed in modern systems. Here’s why it works:

1. Persist Data as Immutable Logs

All data and pipelines in Matterbeam are stored as immutable logs at no extra charge. This means:

• You never lose track of your data.
• Every transformation is fully observable and traceable.
• You unlock built-in governance and collaboration capabilities.

2. Transform Data Without Overpaying

With Matterbeam, transformations are:

• Performed in a serverless environment where you only pay for what you use.
• Integrated into the persistence layer, avoiding the need for external orchestration tools.

3. Load Data with Full Flexibility

Data is loaded into your target system after it’s been transformed, ensuring:

• Minimal processing in your warehouse (and lower compute costs).
• Streamlined pipelines that reduce complexity.

Why PTL is Better Than ELT

Observability

With immutable logs, every step of your pipeline is transparent. Know exactly what happened, when, and why.

Governance

Centralized persistence ensures compliance and simplifies audits.

Collaboration

Teams can work together seamlessly with code and low-code options, making data accessible to technical and non-technical users alike.

Flexibility

Use whatever tool is cheaper, faster, or better for your needs. PTL eliminates the vendor lock-in that ELT pipelines often create.

Stop Doing Data Wrong

The pricing changes by Fivetran and dbt highlight a fundamental problem with the way data pipelines are designed today. You’re paying multiple vendors for fragmented, overly complex workflows that don’t deliver real-time insights. It’s time to break free from the legacy practices and embrace a new, serverless-first future.

With Matterbeam, you can:

• Avoid hidden costs and double charges.
• Simplify your data architecture and introduce true transparency.
• Build smarter, reusable pipelines with PTL.

Ready to ditch the old way and embrace the new? Join us and transform your approach to data forever. Get started with Matterbeam today.