.svg){kind=icon}
.svg){kind=icon}
TL;DR
- A leading American construction management SaaS company was running over 5,000 dbt models daily across a fragmented data stack — paying for it in infrastructure costs, operational drag, and an architecture that worked against the business.
- LatentView migrated their entire dbt transformation ecosystem from Snowflake into Databricks: profiling the pipeline, automating the SQL conversion with MigrateMate, and replacing Airflow with native Databricks orchestration.
- The result is one unified platform with lower compute, storage, I/O, and egress costs, no redundant data copies, a simpler stack, and transformed data sitting exactly where the company’s analytics and machine learning work needs it.
When a company’s transformation layer runs on one platform and everything around it runs on another, the business pays for the gap, every single day.
That was the situation for a leading American construction management SaaS company. Over 5,000 dbt models ran daily across a fragmented data stack, and the architecture itself had become the problem. LatentView stepped in to fix it at the root: a full migration of the dbt transformation ecosystem from Snowflake into Databricks.
The Challenges
The issue wasn’t any single tool. It was an architecture forcing those tools to work against each other.
Rising infrastructure costs. Running more than 5,000 dbt models daily on Snowflake drove high compute, storage, and I/O costs and the trajectory was the real concern, scaling in a way that wasn’t sustainable.
Platform fragmentation. Data moved constantly between Snowflake and Databricks. Every crossing added cost on multiple fronts: network egress charges, duplicated storage, and higher latency. The transformation layer was effectively isolated from the systems feeding it and consuming its output.
Operational complexity. Airflow-based orchestration added pipeline dependencies, multiple failure points, and ongoing troubleshooting effort. More moving parts meant more that could break and more time spent identifying what did.
LatentView’s Approach
LatentView consolidated the stack by migrating the entire dbt transformation ecosystem from Snowflake to Databricks, in three stages.
Automated discovery. The work started with profiling the Master Data Management pipeline to assess model complexity, dependencies, and migration effort a clear picture of the scope before anything changed.
Accelerated conversion. Rather than rewrite thousands of models by hand, LatentView used MigrateMate to convert Snowflake-specific dbt SQL into Databricks-optimized code at scale.
Pipeline modernization. dbt workflows moved directly into Databricks using Databricks Asset Bundles, removing the external Airflow dependency and the layer of failure points that came with it.
The Business Impact
Cost reduction. Consolidating workloads into Databricks reduced compute, storage, I/O, and egress costs.
Data redundancy elimination. The migration ended the recurring data copies between Snowflake and Databricks cutting egress overhead, improving data lineage, and removing duplicated storage.
Operational simplification. Removing Airflow from the stack simplified deployments and accelerated troubleshooting.
ML and data science readiness. With transformed data sitting directly in Databricks, teams can use it immediately for advanced analytics and machine learning no further data preparation delays.
The Outcome
A data platform that no longer works against the business. One ecosystem, lower costs, and the infrastructure finally in place to support what comes next.
FAQ
01. Why migrate transformations to Databricks instead of keeping them on Snowflake?
The company’s data already moved constantly between Snowflake and Databricks. Keeping transformations on Snowflake meant ongoing cross-platform movement that drove up egress costs, duplicated storage, and increased latency. Consolidating onto Databricks closed that gap.
02. How many dbt models were involved?
More than 5,000, running daily.
03. How was the Snowflake-specific SQL converted?
LatentView used MigrateMate to convert Snowflake-specific dbt SQL into Databricks-optimized code at scale, rather than rewriting models manually.
04. What were the main cost drivers before the migration?
Two factors compounded: running 5,000+ dbt models daily on Snowflake, and the constant data movement between Snowflake and Databricks that added egress charges and duplicated storage on top of compute, storage, and I/O costs.
05. How did this affect machine learning and data science work?
With all transformed data consolidated in Databricks, teams can use it directly for advanced analytics and machine learning without further data preparation delays.
