This playbook helps data leaders move Hadoop workloads to Databricks with a clear plan and current tooling.
Hadoop to Databricks migration moves data and metadata from Hadoop clusters to the Databricks Data Intelligence Platform, along with the Spark and Hive jobs that use them, usually in waves.
Key Takeaways
- Hadoop to Databricks migration replaces always-on clusters with elastic cloud compute on open Delta Lake tables, governed centrally through Unity Catalog
- Databricks frames the move as five workstreams: platform administration, data migration, data processing, security and governance, and the SQL and BI layer
- Spark code moves with targeted changes, mostly around SparkContext handling and job submission, while MapReduce and Pig jobs need rewrites
- New Databricks accounts are Unity Catalog only, so Hive metastore migration should land on Unity Catalog from day one
- Dual ingestion and parallel validation let you move workloads in waves while Hadoop keeps serving the business
What Is Hadoop to Databricks Migration and Why Move Now?
Hadoop to Databricks migration moves your HDFS data and Hive metadata onto Databricks in the cloud, where Spark-based processing keeps running with far less platform upkeep for your team.
Hadoop is an ecosystem of open-source projects for distributed storage and processing, while Databricks is a managed cloud platform built on an optimized version of Apache Spark. Databricks was founded by Spark’s original creators, which is why Spark jobs and much of your HiveQL carry over with targeted changes.
Databricks points to four costs of running Hadoop on premises: wasted hardware capacity, a heavy DevOps burden, rising electricity costs and constant software version upgrades. It also argues that moving Hadoop to cloud-managed services such as Amazon EMR, Azure HDInsight or Google Cloud Dataproc keeps many of the same reliability and resource-management problems, since teams still stitch services together and tune clusters.
Two recent changes make the timing matter:
- Unity Catalog by default: Databricks accounts created after December 18, 2025 can’t use legacy features such as DBFS mounts and the workspace Hive metastore (Source), so new migrations land on Unity Catalog from the start
- Hadoop upgrade pressure: Apache Hadoop 3.5 requires Java 17 on servers, so staying current on Hadoop has become a planned project of its own
Hadoop to Databricks Migration Plan: Five Workstreams Step by Step
The migration plan follows Databricks’ five workstreams, preceded by discovery and followed by validation, so every part of your Hadoop estate has a clear destination and an owner.
Databricks organizes Hadoop migration around five workstreams. In practice, discovery comes first and validation comes last, with the five workstreams running in parallel tracks during each migration wave.
Before You Start: Discovery and a Pilot Use Case
Discovery decides how smoothly everything else goes. Inventory HDFS paths and file formats, Hive databases, Spark, Hive, Pig and MapReduce jobs, Oozie workflows, shell scripts, ingestion tools, Ranger or Sentry policies and every downstream consumer.
Give each workload a disposition before any code moves:
| Disposition | What It Means on Databricks |
|---|---|
| Rehost | Spark DataFrame jobs that need only configuration changes |
| Refactor | HiveQL with Hive-only functions, or Spark code built on RDDs |
| Replace | Oozie, Sqoop and similar tools swapped for managed Databricks services |
| Retire | Tables and jobs with no active consumers, archived or dropped |
Then run one self-contained use case end to end as a pilot. It proves networking and security, and it gives your engineers a pattern to copy for every wave after it.
Workstream 1: Platform Administration
Administration replaces the cluster management your Hadoop team does today. A Databricks deployment is a workspace you manage through a web UI or programmatically through REST APIs and the CLI. Many enterprises run separate workspaces per environment, such as development and production, or per business unit.
Set up the platform in this order:
- Identity: Connect single sign-on with multi-factor authentication through your identity provider, and provision account-level users and groups
- Networking: Deploy into your own VPC or VNet, keep cluster nodes off public IPs, and restrict access with IP access lists
- Encryption: Use customer-managed keys where your security policy requires them
- Governance foundation: Create a Unity Catalog metastore for the region, with external locations and volumes in your cloud storage
- Compute policies: Define which instance types and sizes each team can launch, with tags for chargeback
Compute is where Hadoop habits change most. YARN shares one always-on cluster across many jobs through queues, while Databricks recommends running each production job on its own autoscaling job compute that terminates when the job finishes. That gives stronger isolation and avoids jobs competing for the same resources.
| Hadoop Habit | Databricks Practice |
|---|---|
| Shared always-on cluster | Job compute or serverless compute per production job |
| YARN capacity queues | Compute policies, tags and budget policies |
| Fixed cluster size | Autoscaling between a minimum and maximum number of workers |
| Idle capacity overnight | Auto-termination on interactive compute |
| Slow node provisioning | Pools of idle instances that cut start-up and autoscaling time |
| Custom node setup | Init scripts or custom container images |
Workstream 2: Data Migration
Start with dual ingestion. Fork your existing feeds so new data lands in both HDFS and cloud storage, which gives you a live backup in the cloud and lets teams build on current data immediately.
Historical data follows, prioritized by the use cases you migrate first. Databricks describes a push approach, where your team sends data out of Hadoop, and a pull approach, where cloud-side jobs read from HDFS. Push is more common, since data owners and security teams keep control of what leaves and when.
The right transfer method depends on volume:
| Data Volume | Transfer Options |
|---|---|
| Up to hundreds of terabytes | A VPN or dedicated private connection to the cloud, DistCp, or HDFS-to-cloud replication tools |
| Petabytes and above | Physical transfer services from your cloud provider, such as AWS Snowball or Azure Data Box |
| Continuous IoT or event data | Streaming into a message bus that Databricks reads |
Once files land, Auto Loader ingests them incrementally. Convert to Delta Lake as you go: Parquet tables convert in place with CONVERT TO DELTA, while ORC data is read into a DataFrame and written as Delta.
For the Hive metastore, Databricks now recommends Hive metastore federation, which registers your existing metastore as a foreign catalog in Unity Catalog, combined with the UCX toolkit for upgrading tables. Partition tables on low-cardinality columns, and remember that Delta tables don’t support Hive-style bucketing.
Workstream 3: Data Processing
This is where most engineering effort goes. Each processing framework follows its own path:
- HiveQL: Runs on Spark SQL with mostly syntax-level changes, covered in the HiveQL section below
- Spark on YARN: Moves with changes to SparkContext handling and job submission, covered in the Spark section below
- MapReduce and Pig: Rewritten as Spark DataFrame code or Spark SQL, since Databricks has no equivalent runtime
- Sqoop: Replaced with Spark’s JDBC source or Lakeflow Connect, with your own watermark logic for incremental loads, since Spark doesn’t track the last value the way Sqoop jobs do
- Oozie: Rebuilt as Lakeflow Jobs, or orchestrated from Apache Airflow or Azure Data Factory through Databricks’ native integrations
- Zeppelin and Jupyter notebooks: Moved into Databricks notebooks, where Spark-API code usually runs with few changes
Bring source control in early. Databricks notebooks link to GitHub, GitLab, Bitbucket and Azure DevOps through Git folders, and the CLI lets you sync notebooks and packaged code into your existing CI/CD process.
Workstream 4: Security and Governance
Hadoop teams are used to role-based and attribute-based access through Apache Ranger. The older Databricks approach used views with table ACLs or partner integrations, and Unity Catalog now covers this natively.
On Databricks, access starts with grants on catalogs and tables for account-level groups. Row filters and column masks protect sensitive data at a finer level, and attribute-based access control applies policies through tags. Unity Catalog also records lineage and audit logs, which most Hadoop clusters never had out of the box.
Ranger policies don’t import directly, so treat this as a redesign. It’s a good moment to drop years of exceptions.
Workstream 5: SQL and BI Layer
Impala and Hive LLAP users move to Databricks SQL warehouses. Databricks SQL provides a SQL editor, query history, dashboards and scheduled queries with alerts, and its warehouses scale out to more clusters as concurrent queries rise.
BI tools such as Tableau and Power BI connect through JDBC and ODBC drivers. Repoint each dashboard to Databricks during its migration wave, and keep the old connection available until the report owner signs off.
Validate in Parallel Before You Retire Hadoop
Run Hadoop and Databricks in parallel for each wave. Compare row counts, then aggregates by key dimensions, then the business reports themselves, watching especially for timestamp handling and decimal precision.
Cut over one wave at a time with clear rollback triggers. When downstream tools point to Databricks and SLAs hold for an agreed period, retire the matching Hadoop jobs, and finally the hardware and licenses.
Spark on YARN vs Spark on Databricks: What Changes for Your Code
Spark code runs on Databricks with targeted changes, because Databricks shares one SparkContext across users and manages executors and job submission differently from YARN.
Most Spark DataFrame code runs as it is. The differences show up in how applications start, how resources are allocated and how jobs are submitted.
| Area | Spark on YARN | Spark on Databricks |
|---|---|---|
| SparkContext | Each application creates its own | Databricks creates one shared context, so code calls SparkContext.getOrCreate or uses the SparkSession |
| Stopping the context | Common at the end of a job | Avoid it, since it can affect other users on shared compute |
| Application name | Set in code and shown in YARN | Setting it has no effect |
| Executors | Sized per application | One executor per worker by default, adjustable through executor cores and memory settings |
| Resource sharing | YARN queues and schedulers | Fair scheduling with preemption on shared compute, and isolated compute per production job |
| RDD and SparkContext APIs | Available everywhere | Need dedicated access mode compute, since Unity Catalog standard access mode doesn’t support them |
Job submission changes too. Databricks supports spark-submit tasks on new job compute, though the Spark Submit task doesn’t support autoscaling or Databricks utilities. Databricks recommends the JAR task for packaged jobs, and converting a spark-submit job takes four changes:
- Retrieve the context with SparkContext.getOrCreate
- Pass job parameters as task parameters in JSON
- Move files passed with –files to cloud storage and reference them there
- Set the number of workers and Spark configuration in the compute settings
HiveQL vs Spark SQL on Databricks: Syntax Differences to Plan For
HiveQL runs on Spark SQL with mostly syntax-level changes, and the main edits involve table creation, file formats, bucketing and custom functions written for Hive.
Spark SQL and HiveQL both follow ANSI SQL, and most Hive functions for dates, strings, collections and math run on Databricks. Functions that exist only in Hive need a Spark SQL equivalent.
| Topic | Hive Practice | Databricks Practice |
|---|---|---|
| Table creation | STORED AS PARQUET or ORC | USING DELTA, or USING PARQUET where you keep plain Parquet |
| External tables | EXTERNAL keyword with LOCATION | LOCATION alone makes the table external |
| ACID transactions | Hive ACID tables | Delta Lake transactions, with time travel and schema enforcement |
| Bucketing | CLUSTERED BY for high-cardinality columns | Supported for Parquet tables only, so use liquid clustering with Delta |
| Partitions on plain tables | Metastore tracks partitions | Run MSCK REPAIR TABLE for tables you don’t convert to Delta |
| Custom Hive UDFs | Java UDFs registered in Hive | Java UDFs reuse with minor changes, attached as a library and tested on your target access mode |
UDF performance deserves a check. Row-at-a-time Python UDFs run slower than Java or Scala functions because data must move between the JVM and Python, so use pandas UDFs, which are vectorized with Apache Arrow, or keep heavy logic in Java or Scala.
Hadoop to Databricks Technology Mapping: Component by Component
Each Hadoop component maps to a Databricks capability or a cloud service, and the mapping shows where code moves almost unchanged and where your team needs real redesign.
| Hadoop Component | Databricks Equivalent | Migration Note |
|---|---|---|
| HDFS | Cloud object storage with Delta Lake tables | Move with DistCp or replication tools, then convert to Delta |
| Hive metastore | Unity Catalog | Federate first, then upgrade tables with UCX |
| HiveQL | Spark SQL and Databricks SQL | Most syntax carries over, with STORED AS replaced by USING |
| Impala and Hive LLAP | Databricks SQL warehouses | Serverless warehouses suit interactive and BI queries |
| Spark on YARN | Spark on Databricks job or serverless compute | Adjust SparkContext handling and job submission |
| MapReduce and Pig | Spark DataFrames and Spark SQL | Rewrite the logic |
| Oozie | Lakeflow Jobs | Airflow and Azure Data Factory integrations are also available |
| Sqoop and Flume | Lakeflow Connect, Auto Loader and Spark JDBC | Add watermark logic for incremental loads |
| Kafka consumers | Structured Streaming and Lakeflow Spark Declarative Pipelines | Kafka can stay or move to a managed cloud message service |
| HBase | Cloud-native NoSQL stores, or Lakebase for Postgres-based operational workloads | Test latency against your SLAs |
| Ranger and Sentry | Unity Catalog grants, row filters and column masks | Redesign policies, since there’s no direct import |
| Zeppelin and Jupyter | Databricks notebooks and IDE integrations | Notebook code usually needs few changes |
Databricks Tools That Support Hadoop to Databricks Migration
Databricks provides tools for each part of the move, from Hive metastore federation and UCX for metadata to Auto Loader for data and Lakeflow Jobs for orchestration.
| Tool | Role in a Hadoop Migration |
|---|---|
| Hive metastore federation | Mirrors your Hive metastore as a foreign catalog in Unity Catalog during the transition |
| UCX | Databricks Labs toolkit that assesses your estate and upgrades tables and permissions to Unity Catalog |
| Auto Loader | Ingests new files incrementally as they land in cloud storage |
| Lakeflow Connect | Managed connectors for databases and SaaS sources |
| Lakeflow Spark Declarative Pipelines | Declarative batch and streaming pipelines, formerly Delta Live Tables |
| Lakeflow Jobs | Orchestration for notebooks, JARs, Python scripts and pipelines |
| Lakebridge | Assessment and conversion for data warehouse and ETL code, with SQL converters focused on warehouse dialects |
UCX, like other Databricks Labs projects, is published on GitHub and isn’t directly supported by Databricks. Lakebridge’s built-in converters don’t list Hive or Impala, so check coverage before you plan Hive code conversion around it.
Hadoop to Databricks Migration Challenges and How to Solve Them
The hardest parts of moving off Hadoop are hidden job dependencies and Spark code that depends on cluster behavior, and both have practical fixes, as does governance redesign.
| Challenge | How to Solve It |
|---|---|
| Hidden dependencies in scripts and cron jobs | Inventory beyond Oozie and move dependent jobs in the same wave |
| SparkContext and RDD code | Use SparkContext.getOrCreate, move to DataFrames, or run on dedicated access mode compute |
| spark-submit habits | Convert packaged jobs to JAR tasks and move settings into compute configuration |
| Data type and timestamp differences | Compare precision and time zone handling during parallel runs |
| YARN chargeback and queue rules | Rebuild with compute policies, tags and budget policies |
| Ranger and Sentry policies | Redesign as Unity Catalog grants, row filters and column masks |
| Sqoop incremental loads | Add watermark logic or move to Lakeflow Connect |
| Lift-and-shift cost surprises | Refactor high-cost jobs and right-size compute after each wave |
Hadoop to Databricks Migration Cost and Timeline Factors
Cost and timeline depend mainly on data volume and the share of jobs that need refactoring, along with how many consumers repoint at cutover, so scope your estate before you commit to dates.
Timelines vary widely between enterprises, so we’ve avoided quoting a generic range. These factors drive both cost and duration:
- Data volume: Petabyte estates may need physical transfer services in addition to network copies
- Workload mix: Clean Spark DataFrame code moves faster than MapReduce or RDD-heavy jobs
- Hive metastore complexity: Large numbers of views and custom UDFs add metadata and testing effort
- Governance scope: Large Ranger policy sets take time to redesign in Unity Catalog
- Parallel running: Running Hadoop and Databricks together adds cost, so set exit criteria for each wave
On the Databricks side, you pay for compute in Databricks Units based on usage, plus your cloud provider’s infrastructure for classic compute. Build that model into your business case next to today’s Hadoop costs, including hardware refresh and the staff time spent on upgrades.
Hadoop to Databricks Migration Patterns by Workload Type
Different Hadoop workloads follow different migration paths: Hive data warehouses move to Delta and Databricks SQL, Spark ETL moves to Lakeflow, and machine learning moves to MLflow.
Hive Data Warehouses and Impala Reporting
These usually move first, since business users see the payoff quickly. A typical sequence:
- Federate the Hive metastore so tables appear in Unity Catalog straight away
- Convert Parquet tables in place with CONVERT TO DELTA, and rewrite ORC tables as Delta
- Upgrade the tables to Unity Catalog, preferably as managed tables
- Repoint Tableau or Power BI dashboards to a Databricks SQL warehouse
- Have report owners compare numbers on both platforms before sign-off
Most of the effort here is SQL syntax review and report validation, which is why business owners need time in the plan.
Spark Batch ETL Pipelines
Spark DataFrame pipelines are often the quickest rehosting candidates, because the transformation logic itself rarely changes. The work sits around the code: SparkContext handling, storage paths that point to HDFS, credentials set in code, and spark-submit commands that become Databricks job tasks.
Rehost first, then improve. Once a pipeline runs reliably on job compute with its schedule rebuilt in Lakeflow Jobs, look at the pipelines with heavy data quality logic. Those are often worth rebuilding as Lakeflow Spark Declarative Pipelines, which add expectations for data quality and built-in monitoring of each update.
Streaming Pipelines on Kafka
- Move Kafka consumers to Spark Structured Streaming, writing to Delta tables analysts can query as data arrives
- Keep Kafka where it runs today until the streaming jobs prove stable
- Consider Lakeflow Spark Declarative Pipelines for streaming tables that need managed checkpoints and monitoring
- Decide on a managed cloud message service once the Databricks side is settled
Machine Learning and Data Science
| On Hadoop | On Databricks |
|---|---|
| Models trained on whatever fits in YARN containers | GPU and memory-optimized compute chosen per training job |
| Experiments tracked by hand or in spreadsheets | MLflow tracking, with automatic logging from notebooks |
| Models copied between environments | Models registered and governed in Unity Catalog |
| Feature data rebuilt for each project | Feature tables in Delta, shared across teams |
Migrate the feature pipelines first, then move model training once the data sits in Delta.
Legacy MapReduce and Pig Jobs
These need full rewrites, which makes them the best retirement candidates. Rewrite only the jobs with active consumers.
Databricks Best Practices for Running Former Hadoop Workloads
Former Hadoop workloads run best on Databricks when you isolate production jobs, store everything in Delta through Unity Catalog, and write Spark code for a managed, shared runtime.
Compute Practices
- Isolate production jobs: Run each scheduled job on its own autoscaling job compute so failures and spikes stay contained
- Use current runtimes: Newer Databricks Runtime versions carry performance improvements and fixes
- Cut start-up time with pools: Pools keep idle instances ready, and you pay only the cloud cost for those idle instances
- Tag everything: Tags on compute flow into billing data for team and project chargeback
Storage and Table Practices
Store data in Unity Catalog volumes and external locations, and avoid writing tables to DBFS root. Convert to Delta wherever you can, since it adds ACID transactions and time travel to the Parquet files underneath.
Partition only on low-cardinality columns, and use liquid clustering for large tables you previously bucketed in Hive. Unity Catalog managed tables also get predictive optimization, which handles compaction and clustering for you.
Code Practices
| Practice | Why It Helps |
|---|---|
| Use the SparkSession Databricks creates | Avoids errors from creating or stopping a SparkContext on shared compute |
| Prefer JAR tasks over Spark Submit tasks | Keeps autoscaling and Databricks utilities available |
| Keep heavy UDF logic in Java or Scala | Avoids serialization overhead between the JVM and Python |
| Use pandas UDFs for Python logic | Vectorized execution through Apache Arrow |
| Keep secrets out of code | Databricks secrets replace hard-coded storage credentials |
Operations Practices
Monitor jobs through compute metrics and the Spark UI, and deliver cluster logs to cloud storage for troubleshooting. Review billing system tables monthly so oversized compute gets caught early, and connect every notebook and job to source control before go-live.
If you’re moving other platforms to Databricks as well, our Databricks migration guide covers the strategy decisions that apply across source systems.
How LatentView Helps With Hadoop to Databricks Migration
We bring Databricks depth and Hadoop migration experience to the same program, as a Databricks Gold Partner with a Center of Excellence of 400 data engineers and more than 30 joint Databricks customers. Our Databricks partnership page shows how we work together, and this case study describes a Hadoop estate we moved to the cloud with 10x lower TCO.
Transform your Hadoop estate into a governed Databricks lakehouse with an end-to-end analytics partner that brings 20+ years of enterprise analytics experience and 50+ Fortune 500 clients. MigrateMate automates migration from assessment through validation, while BrickShift moves dashboards from legacy BI tools onto Databricks AI/BI. Start with one workload: talk to our data migration team about a Hadoop to Databricks assessment.
Frequently Asked Questions About Hadoop to Databricks Migration
1. How Long Does a Hadoop to Databricks Migration Take?
Duration depends on data volume and the share of jobs that need refactoring, along with the number of downstream consumers. A phased program delivers the first wave sooner, while full decommissioning follows once every wave passes validation.
2. Do Hive Queries Run on Databricks Without Changes?
Most HiveQL runs on Spark SQL with small changes, such as replacing STORED AS with USING DELTA. Queries that rely on Hive-only functions or custom UDFs need review and testing before cutover.
3. What Happens to the Hive Metastore During Migration?
Hive metastore federation registers the existing metastore as a foreign catalog in Unity Catalog. Tables then upgrade to Unity Catalog in stages, often with the UCX toolkit from Databricks Labs.
4. Do Spark Jobs Need Changes to Run on Databricks?
Spark DataFrame jobs usually need only small changes, such as using the shared SparkContext. Jobs submitted with spark-submit are best converted to JAR tasks, and RDD-based code needs dedicated access mode compute.
5. How Do Ranger Policies Move to Databricks?
Ranger and Sentry policies don’t import directly. Teams redesign access as Unity Catalog grants plus row filters and column masks, which also adds built-in lineage and audit logs.