BigQuery to Databricks Migration: Strategies, Phased Roadmap and Validation Guide

 & LatentView Analytics

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

SHARE

Table of Contents

This guide helps data leaders plan a move from Google BigQuery to Databricks, pick the right migration strategy, and retire BigQuery workloads wave by wave without breaking the reports, pipelines and models teams depend on.

Migrating from Google BigQuery to Databricks comes down to three workstreams: moving the data, translating SQL and pipeline code, and validating the results. BigQuery is a serverless data warehouse, while Databricks is a lakehouse built on Delta Lake and governed by Unity Catalog.

Choosing the right ingestion path for each table and resolving SQL dialect differences early are the decisions that shape the rest of the plan.

Key Takeaways

  • BigQuery to Databricks migration shifts storage, transformation and analytics from BigQuery onto Delta Lake or Apache Iceberg tables governed by Unity Catalog
  • Many organizations are exploring ways to bring BI, data engineering and AI under one governance model
  • The main strategy choices are BI-first or ETL-first, lift and shift or modernize, and wave-based or big-bang cutover
  • Lakehouse Federation and open table formats let BigQuery and Databricks run side by side until each wave passes validation
  • Profiling, permission mapping and automated reconciliation decide the scope, safety and finish line of every wave

BigQuery to Databricks Migration as a Data and AI Platform Evolution

Moving from BigQuery to Databricks can be a step in a wider platform evolution, where BI, data engineering and AI move onto open tables under one governance and operating model.

Many organizations run BI, ETL and AI workloads on BigQuery today and want a single governance and operating model for all three.

On Databricks, the main changes look like this:

DriverWhat Changes on Databricks
One governance modelUnity Catalog governs tables, files, models and other AI assets with shared permissions, lineage that reaches dashboards and audit logs
Open formatsData sits in Delta Lake or Iceberg tables in your own cloud storage, readable by other engines
AI on the same dataFeature engineering, model training, MLflow tracking and model serving run next to the warehouse data
Query performancePhoton is the default engine on SQL warehouses, and serverless SQL warehouses are available for BI
Engineering practicesGit integration and CI/CD move teams from writing queries to building data products

A phased path works well: one domain moves first, BigQuery and Databricks share data for a period, and BigQuery retires once every consumer has switched.

A successful migration takes more than copying tables. Sponsors can judge it by business outcomes, such as faster dashboards or new AI use cases, as well as by tables moved.

BigQuery vs Databricks: Key Differences That Shape Your Migration

BigQuery is a serverless warehouse billed by bytes scanned or slots, whereas Databricks runs open tables on your own cloud storage, so storage, SQL and security all change shape.

A handful of structural differences shape the engineering work in a BigQuery migration. Seeing them side by side helps teams size each workstream early:

AreaBigQueryDatabricks
StorageManaged storage inside BigQueryDelta Lake or Iceberg tables in your own cloud storage, including Google Cloud Storage
Object hierarchyProject, dataset, tableCatalog, schema, table in Unity Catalog
Compute pricingOn-demand per TiB scanned, or slot capacity through editionsDatabricks Units (DBUs), billed per second of use
SQL dialectGoogleSQL, with STRUCT and ARRAY types and procedural scriptingDatabricks SQL and Spark SQL, plus Python and Scala
Data layoutPartitioned and clustered tablesLiquid clustering on Delta tables
Access controlIAM roles, row-level access policies and policy tagsUnity Catalog grants, row filters, column masks and governed tags
SchedulingScheduled queries, Dataform and Managed Airflow (Cloud Composer)Lakeflow Jobs and Lakeflow pipelines
Machine learningBigQuery MLMLflow, model training and Model Serving

The pricing row deserves early attention. BigQuery on-demand bills for bytes scanned, so cutting the data a query reads lowers its cost, while Databricks bills for compute time, so the same query is tuned through clustering and warehouse sizing.

Databricks also runs on Google Cloud. Teams that want to stay there can keep data in Cloud Storage buckets. For a wider comparison of the two platforms, see our Databricks vs BigQuery guide.

BigQuery to Databricks Migration Strategies: Choosing the Right Approach

A BigQuery migration strategy answers three questions: which workloads move first, how much logic gets redesigned on the way, and how the switch from BigQuery happens.

Each question has two common answers, and the choice can differ by domain.

BI-First Migration Strategy

A BI-first strategy moves the dashboards decision-makers use every day before anything else. Teams rebuild the most-used reports on Databricks SQL and AI/BI dashboards, while the data still sits in BigQuery and is read through Lakehouse Federation.

It suits estates where:

  • A small set of dashboards drives most query volume and executive attention
  • Analysts want natural-language analytics with Genie Agents early
  • Report performance is a visible pain point today

Business users meet the new platform in the first wave, which gives later waves visible proof to build on.

ETL-First Migration Strategy

An ETL-first strategy moves ingestion and transformation first. Scheduled queries, Dataform workflows and Airflow DAGs land on Databricks, and curated tables are written to Cloud Storage in open formats.

This path fits teams whose BigQuery spend is dominated by heavy scheduled transformations, or who want an AI-ready data foundation before anything else. The cost is lower visibility, because report users notice little until dashboards follow. Share runtime and cost results after each wave so sponsors can see the gains.

If the choice is unclear, check where the slots go. When a few dashboards generate most of the query volume, start with BI. When scheduled transformations use most of the capacity, start with ETL.

Lift and Shift or Modernize

The second decision is how much to change the logic itself:

ApproachWhat HappensBest Used WhenWatch Out For
Lift and shiftSQL and pipelines move with minimal changesSpeed matters and parity must be proven firstOptimization waits until parity is proven
ModernizePipelines are redesigned as incremental Lakeflow pipelines with built-in quality checksLogic is costly, fragile or due for change anywayLonger timelines and harder side-by-side comparison

The two approaches can run in sequence. Move the logic as is to get a baseline that matches BigQuery, then redesign the highest-value pipelines once the numbers are trusted.

Wave-Based or Big-Bang Cutover

A big-bang cutover switches the whole estate at once, so one failure puts every workload at risk. Wave-based migration switches one self-contained group at a time, which limits the damage of any single problem and shows progress after each wave.

Rate each workload on business value and migration complexity, and fill the first waves with the high-value, low-effort ones. A single go-live is defensible only for a small estate with low-risk workloads, where a short disruption costs less than a long overlap. In every other case, waves are safer.

How to Assess a BigQuery Environment Before Migration

A BigQuery assessment profiles datasets, query patterns and slot consumption, so migration waves cover the workloads people actually use and rarely queried tables are retired.

Databricks reports that in most BigQuery estates a significant share of tables are rarely or never queried. Finding them first cuts the migration scope before any engineering starts.

Lakebridge is an open-source migration toolkit from Databricks Labs, and its BigQuery profiler automates this discovery. The profiler is still labeled Experimental. It reads BigQuery’s information schema views for query history, table storage, streaming activity and reservations, then writes the results to a DuckDB file. Reservation data applies only to customers on BigQuery Editions.

Use the profile to build an inventory for each business domain:

  • Datasets and tables with their size, freshness and query frequency
  • Scheduled queries, Dataform workflows and Airflow DAGs that write to them
  • Dashboards and BI tools that read from them
  • BigQuery ML models, remote functions and JavaScript user-defined functions
  • External tables, streaming inserts and Pub/Sub feeds
  • Owners from engineering and the business

Slot and cost data add a second lens, showing which workloads drive cost. Rarely used tables can be retired or archived to Cloud Storage rather than migrated.

Finish the assessment with a wave plan. Each wave should be a self-contained business domain, with its sources, pipelines and dashboards grouped together so it can be migrated, validated and switched over as one unit.

How to Migrate BigQuery Data to Delta Lake

BigQuery data can move through bulk exports to Cloud Storage, the Spark BigQuery connector or Lakehouse Federation, and table size and freshness decide which route fits each table.

RouteHow It WorksBest ForCost Notes
Bulk export to Cloud StorageBigQuery extract operations write tables as Parquet files, which Databricks loads into Delta tablesLarge historical tablesExtracts on the shared slot pool are free up to 50 TiB a day; you pay for Cloud Storage
Spark BigQuery connectorDatabricks reads tables through the BigQuery Storage Read APISmaller tables and incremental loads, though Databricks now favors federation where it fitsThe Storage Read API charges $1.10 per TiB read, after a free 300 TiB of reads each month for every billing account
Redirected ingestionUpstream feeds write straight to Delta tables on DatabricksTables that change continuouslyRemoves repeated reads from BigQuery during the transition
Lakehouse FederationUnity Catalog queries BigQuery in place through a foreign catalogSmall or low-priority tables waiting for a later waveReads are billed by BigQuery, through the Storage Read API or as query compute when views are involved

Pricing figures come from Google’s BigQuery pricing page. Extracts must write to a bucket in the same location as the dataset, and cross-region reads add network charges, so keep the Databricks workspace in that region where possible.

An export also gives you a fixed reference. Row counts and checksums can be compared against files that no longer change, which is hard to do against a live table.

Continuously updated tables need a different plan. Redirect the upstream feed to write into Delta, backfill history from an export, and let the two meet at a clear cutoff timestamp so no records are lost or duplicated.

Organize landed data in medallion layers: bronze for raw copies, silver for cleaned data and gold for business-ready tables.

How to Convert BigQuery SQL and Pipelines to Databricks

Routine BigQuery SQL converts with rule-based tools and AI-assisted rewriting, while scripting, user-defined functions and orchestration need engineers for the remaining complex cases.

These common patterns map directly, which makes them good targets for automation:

BigQuery PatternDatabricks Equivalent
project.dataset.table referencescatalog.schema.table in Unity Catalog
SAFE_CASTtry_cast
UNNEST over arraysexplode
DATE_TRUNC(date_col, MONTH)date_trunc(‘MONTH’, date_col)
Partitioning and clustering optionsLiquid clustering on Delta tables
JavaScript user-defined functionsSQL or Python user-defined functions

Start with rule-based translators, such as the open-source SQLGlot library, for routine statements. Use AI-assisted conversion for the dialect quirks the rules miss, and keep engineers for scripting blocks, remote functions and logic tied to BigQuery-specific behavior.

Orchestration moves alongside the SQL:

  • Scheduled queries: Run as Lakeflow Jobs, with task dependencies, retries and alerts
  • Managed Airflow (Cloud Composer) DAGs: Become Lakeflow Jobs workflows, or keep Airflow and call Databricks from it during the transition
  • Dataform projects: Move to Lakeflow pipelines, built on Spark Declarative Pipelines, or to dbt, which runs on Databricks
  • BigQuery ML models: Are retrained in Databricks and tracked with MLflow in Unity Catalog

Keep converted code in Git from day one. Reviewing changes through pull requests gives every wave an audit trail and makes rollbacks simple.

How to Migrate BigQuery Security and Governance to Unity Catalog

BigQuery projects, datasets and tables map directly onto Unity Catalog catalogs, schemas and tables, so permissions and fine-grained policies can be recreated before any data moves.

Set up access rules before the first table lands. When permissions are already in place at cutover, the only thing users notice is the new platform, not a change in what they may read.

The main BigQuery controls map like this:

BigQuery ControlUnity Catalog Equivalent
IAM roles on projects and datasetsGrants on catalogs and schemas, assigned to groups from your identity provider
Row-level access policiesRow filters that restrict rows per user or group
Policy tags and data maskingColumn masks, or governed tags with attribute-based access policies
Cloud audit logsUnity Catalog system tables for access history (the audit log table is in Public Preview), plus table and column lineage

Begin by copying the current rules as they are. Users keep the access they have today, and auditors can compare the two platforms rule by rule.

Governance can evolve once the estate is stable. Governed tags and attribute-based policies let one rule protect every table carrying a sensitive tag, instead of granting access table by table. Some of these policy features, including DENY policies and policies on views, are still in Beta.

Service accounts need the same care. Replace BigQuery service accounts with Databricks service principals for production workflows, scoped to the catalogs each pipeline needs. Our Unity Catalog migration guide covers best practices for the move.

Running BigQuery and Databricks Side by Side During Migration

During a wave-based migration, BigQuery and Databricks share data through Lakehouse Federation and open table formats, so dashboards and pipelines keep working while workloads move.

While both platforms run, the bridge between them depends on the strategy:

  • For BI-first waves: A Lakehouse Federation foreign catalog in Unity Catalog lets rebuilt dashboards query BigQuery tables where they sit until their data moves
  • For ETL-first waves: Databricks writes curated tables to Cloud Storage and BigQuery reads the Delta tables through BigLake, so existing reports run from one copy of the data (Source)

Both bridges come with limits. BigQuery can only read external tables. They are read-only on its side and need a manual schema refresh after schema changes. Federated reads from Databricks also add charges on the BigQuery side, so the overlap should stay as short as each wave allows.

Open formats make the period after migration more flexible too. Because Delta and Iceberg tables in Cloud Storage can be read by more than one engine, teams that keep some BigQuery use cases can do so without copying data back. Databricks has also announced a preview of catalog-level federation, so both platforms can work from one set of tables with no duplicate copies.

Do not let the bridge become the destination. Put a date on moving the last dashboards and applications that BigQuery still serves, and make it the closing milestone of the plan.

How to Validate a BigQuery to Databricks Migration

Validation compares row counts, schemas and values across BigQuery and Databricks, and an agreed parity threshold decides when each legacy pipeline can be retired.

Treat validation as a workstream of its own, with named owners and time in every wave. Reconciling each table, schema and report is easy to underestimate when it is left to the end. Checks run at three levels:

  1. Row counts: Every migrated table, per partition where tables are large
  2. Schema: Column names, data types and nullability on both platforms
  3. Values: Aggregate totals for key business measures, plus row-level hashes

Lakebridge reconcile supports BigQuery as a source and automates schema, row and column-level comparisons against Databricks tables.

Expect some false alarms. A handful of built-in functions behave slightly differently in GoogleSQL and Databricks SQL, so a row hash can differ even when every value matches. Trace each difference to its cause first, and only then decide whether anything was lost.

Decide before the parallel run what counts as done. A match rate on row counts, key totals and hashes, set per wave and agreed with the business owners, gives the team a date to retire the old pipeline. Without one, the overlap tends to drift on and both platforms keep running.

Include dashboards in the checks as well. A table can reconcile perfectly while a report still shows a different number, because of a changed filter, time zone or rounding rule in the BI layer, so business owners should sign off on the reports they use.

BigQuery to Databricks Migration Roadmap: Phase-by-Phase Plan

The roadmap runs from assessment through governance setup, wave-by-wave migration and parallel validation to decommissioning, with each wave closed before the next one starts.

Phase 1: Assess the Estate and Choose a Strategy

Profile query history and slot consumption, retire unused tables and pick the entry point for each domain. The output is a wave plan, ranked on business value and migration complexity, with owners and parity criteria agreed for each wave.

Phase 2: Build the Unity Catalog Foundation

Set up the platform every wave will share:

  • Catalogs and schemas that mirror BigQuery projects and datasets
  • Grants, row filters and column masks recreated from BigQuery policies
  • Service principals, Lakehouse Federation connections and network access to Google Cloud
  • Git repositories and CI/CD release pipelines

Phase 3: Migrate Data and Logic Wave by Wave

Export or stream each wave’s tables into Delta, convert its SQL and user-defined functions, and rebuild its orchestration in Lakeflow Jobs. Dashboards in the wave are repointed to Databricks SQL warehouses once their data is in place.

Pick the data route table by table, using the options in the data migration table: bulk export for large history, redirected feeds for tables that change continuously, and federation for low-priority tables waiting for a later wave. Once exported Parquet files land in Cloud Storage, Auto Loader can load them into Delta tables incrementally as they arrive.

Keep the first Lakeflow Jobs schedules identical to the BigQuery ones, so a timing change cannot hide a logic difference. Commit every converted query and pipeline to Git, so each wave has a reviewable history if something has to be rolled back.

Phase 4: Run in Parallel and Reconcile

Both platforms process the same inputs until the wave meets its parity threshold. Mismatches are traced to their cause, fixed and rechecked, and business owners sign off before any switch.

Run the three levels of checks from the validation section on every table in the wave, and add the reports built on those tables. Lakebridge reconcile can automate the table comparisons, while report checks sit with the business owners who use them.

  • Log each mismatch with its cause, so repeated causes point to one rule or one transform
  • Compare reports side by side for the same period, including time zone and rounding rules
  • Record the match rate and the sign-off for the wave, so the retirement decision has evidence behind it

A wave moves to Phase 5 only once that sign-off is on record.

Phase 5: Decommission and Modernize

Pause the wave’s BigQuery pipelines, keep its datasets read-only for an agreed fallback period, then delete them. With BigQuery retired for that domain, refactor its highest-value pipelines into incremental Lakeflow pipelines and share runtime and cost results with sponsors before the next wave begins.

Before anything is deleted, check that nothing still reads from BigQuery for this domain, including dashboards and applications served from BigLake external tables. Rarely used tables that were not migrated can be archived to Cloud Storage rather than deleted outright.

Close the wave with a cost comparison. Set the BigQuery billing and slot usage exported before the wave against Databricks spend for the same workloads, and share the result with sponsors. It also gives the next wave a realistic budget.

BigQuery to Databricks Migration Cost: What to Budget For

Migration cost depends on how data leaves BigQuery, how long both platforms run together and how quickly duplicate processing is retired once each wave passes validation.

Budget lines should cover the transition as well as the target platform. Four areas deserve their own line:

Data extraction: Bulk extracts carry no processing charge. Repeated reads through the Storage Read API cost $1.10 per TiB once an account passes its free 300 TiB a month. Choose the route per table, based on size and how often it changes.

The dual-run window: Every workload running on both platforms is paid for twice. Moving in waves limits the overlap to the workloads in flight, and a parity threshold keeps each overlap short.

Engineering and validation effort: Automate conversion with rule-based and AI-assisted tools, and validation with reconciliation tooling, to keep SQL conversion, testing and reconciliation effort down.

Steady-state compute: Choose serverless SQL warehouses where available, and build incremental pipelines so tables are not reprocessed in full on every run.

Compare costs with real data. Export BigQuery billing and slot usage for each wave before it moves, then track Databricks spend for the same workloads afterward. Our Databricks cost optimization guide lists the spend controls to put in place from the first wave.

Preparing Data Teams for Databricks After BigQuery

A BigQuery migration changes how analysts, engineers and data scientists work together, so team enablement needs the same planning as data and code to make the new platform stick.

On Databricks, analysts, engineers and data scientists can work in the same workspace and share notebooks, which reduces the handoffs between them.

Several changes help that shift land well:

  • SQL stays central: Databricks SQL and AI/BI dashboards let analysts keep working in SQL
  • AI assistance: Genie Agents let business users ask questions about their data in plain language
  • Engineering practices: Git integration, CI/CD and declarative pipelines move teams from writing one-off queries to maintaining governed data products
  • Clear ownership: Each domain gets named data product owners, so questions about definitions and access have a clear home after cutover
  • Structured learning: Databricks Academy training and certification paths spread skills across the whole team, so knowledge does not sit with a few early adopters

In a BI-first move, the dashboard users feel the change first. Name a champion in each business team, train users before their wave and keep a feedback channel open after cutover.

Common BigQuery to Databricks Migration Challenges and How to Avoid Them

Several pitfalls recur in BigQuery migrations, from unscoped estates and hand-converted SQL to late governance and a parallel run with no exit condition, and each is avoidable with early planning.

ChallengeHow to Avoid It
Migrating everything that existsProfile usage first and archive what nobody queries
Hand-converting SQLUse rule-based and AI-assisted conversion, with engineers on the complex tail
Moving data before permissionsRecreate access rules in Unity Catalog before each wave
Big-bang cutoverMove in waves, with federation keeping both platforms connected
Time zone and type mismatchesTest TIMESTAMP, DATETIME and BIGNUMERIC handling early
Open-ended dual runningAgree parity thresholds and decommission dates up front

Data types need care. BigQuery TIMESTAMP values are absolute points in time, DATETIME values carry no time zone, and BIGNUMERIC holds about 76.8 digits while Databricks DECIMAL stops at 38.

For planning advice that applies across source platforms, read our Databricks migration guide.

How LatentView Helps With BigQuery to Databricks Migration

LatentView pairs a dedicated Center of Excellence with migration accelerators for the move to Databricks.

We are a Databricks Elite C&SI partner. Our Databricks Center of Excellence brings together 400 data engineers, including 150+ certified and 200+ trained Databricks specialists, and we serve 50+ Fortune 500 clients.

MigrateMate, our platform-agnostic migration solution, automates assessment, conversion, migration and validation, and typically reduces migration costs by 30% to 40%. UCMate, our Unity Catalog migration solution, simplifies and accelerates the move to Unity Catalog. Scope your first BigQuery wave with our data engineering team: Talk to a Data Engineering Expert.

Frequently Asked Questions About BigQuery to Databricks Migration

1. What Is the Best Strategy for BigQuery to Databricks Migration?

Databricks recommends migrating in waves, starting BI-first or ETL-first based on usage data, with lift and shift first and modernization after parity is proven.

2. Can Databricks Query BigQuery Without Moving the Data?

Yes. Lakehouse Federation creates a foreign catalog in Unity Catalog, so Databricks can query BigQuery tables in place during the migration.

3. What Is the Cheapest Way to Export Data From BigQuery?

Bulk extracts to Cloud Storage are usually cheapest for large tables, since extract jobs carry no processing charge and extracts on the shared slot pool are free up to 50 TiB a day.

4. Does BigQuery SQL Run on Databricks Without Changes?

Not entirely. Rule-based and AI-assisted tools convert routine GoogleSQL, while scripting, JavaScript functions and logic tied to BigQuery behavior need engineers.

5. How Do BigQuery Permissions Move to Databricks?

Projects, datasets and tables map to Unity Catalog catalogs, schemas and tables. Row-level access policies become row filters, and policy tags become column masks.

6. How Long Does a BigQuery to Databricks Migration Take?

Timelines depend on estate size, SQL complexity and the number of waves. Profile the estate first, then size each wave from its tables, queries and dashboards.

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

CATEGORY

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

Take to the Next Step

"*" indicates required fields

consent*

Related Blogs

This guide helps data leaders move pipelines from Azure Data Factory to Databricks one use case…

This guide helps data leaders plan an Azure Synapse to Databricks migration that consolidates several Synapse…

This playbook helps data leaders move Hadoop workloads to Databricks with a clear plan and current…

Scroll to Top