Amazon EMR to Databricks Migration Guide: Strategy, Six Phases and Best Practices

 & LatentView Analytics

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

SHARE

Table of Contents

This guide helps data leaders plan a move from Amazon EMR to Databricks, from choosing a migration strategy through each phase of the work, with the field details that decide how smoothly Spark, Hive and pipeline workloads land.

EMR to Databricks migration moves Spark workloads, metastores, pipelines and BI consumers from Amazon EMR clusters onto the Databricks lakehouse, while most data stays in place on Amazon S3.

Key Takeaways

  • EMR to Databricks migration shifts Spark processing, orchestration and governance from EMR clusters onto Databricks compute, Lakeflow Jobs and Unity Catalog
  • Strategy depends on workload types, dependencies and goals, and Databricks recommends a phased migration over a single cutover
  • The work runs in six phases: discovery, architecture mapping, data, code, pipelines and downstream tools
  • Databricks clusters work differently from YARN, so cluster sizing, intermediate storage and workload submission all change during the move
  • Validation runs at every phase, and Delta Lake tuning, governance and team enablement keep the estate healthy after cutover

EMR to Databricks Migration Strategy

An EMR migration strategy is shaped by the current architecture, workload mix, business priorities and team skills, and it decides which workloads lift and shift and which get redesigned.

Databricks calls EMR moves relatively easy next to legacy on-premises migrations, because both platforms run Spark on S3 data. The difficulty sits around Spark: YARN-tuned clusters, pipelines spread across AWS services, and access rules split between IAM, Lake Formation and cluster settings.

The route depends on how much the estate relies on other AWS services, the mix of batch, streaming and interactive work, how critical each use case is, the main goal (cost, a contract end date or governance) and team skills. Pipeline coupling and shared versus dedicated clusters matter too.

Those inputs send each workload down one of three paths:

PathWhat It Means for EMR WorkloadsTypical Candidates
Lift and shiftSpark code moves with small changes, and tables register in Unity CatalogHealthy PySpark and Spark SQL applications
RefactorPipelines move to Delta Lake, Lakeflow pipelines and Databricks featuresCostly, slow or fragile pipelines
Re-architectOlder engines and designs are rebuiltMapReduce, Sqoop, DStreams, HBase and Hive-heavy estates

Rank pipelines by business need first. Any pipeline that can be retired is one nobody has to convert, which often decides between lift and shift and re-architecture.

A bulk move switches everything at once. A phased migration moves related pipelines in stages, shows progress early and limits risk, which is why Databricks recommends it in its EMR migration guide (Source).

The Six Phases of an EMR to Databricks Migration

EMR migrations run through six phases, from discovery and architecture mapping to data, code, pipeline and downstream tool migration, with change management running alongside all of them.

PhaseFocusMain Output
1. Discovery and assessmentWhat runs on EMR today, and whyWorkload inventory, priorities and migration waves
2. Architecture and feature mappingHow EMR concepts translate to DatabricksTarget workspace design and compute standards
3. Data migrationMoving data and metadata into the lakehouseDelta tables in Unity Catalog, organized in layers
4. Code migrationAdapting Spark, Hive, MapReduce and Sqoop codeConverted and tested applications
5. Data pipeline migrationOrchestration, sources, sinks and query tuningProduction pipelines running on Databricks
6. Downstream tools integrationBI tools and applicationsDashboards and consumers repointed to Databricks

Change management spans all six: train a few champions first, then widen enablement wave by wave. In a phased migration, phases 3 to 6 repeat per wave, so early lessons shorten later waves.

Phase 1: EMR Migration Discovery and Assessment

Discovery captures business goals, SLAs, clusters, workloads, data, users and security on EMR, so every later phase is planned around facts and the true dependencies between pipelines.

Begin with the business view: which use cases EMR serves, the SLAs attached and the outcomes to protect. Then inventory clusters (counts, sizes, instance types, spot mix, Graviton use, EMR versions), frameworks, data sources and volumes, orchestration, consumers and security.

Manual inventories miss things on large estates. Databricks’ guide recommends its EMR profiler, available through account teams, which reads cluster, step and application logs through EMR APIs such as DescribeCluster and ListSteps. Its output covers monthly instance hours, usage by account and region, transient versus long-running clusters, and EMR versions in use.

Several details are easy to overlook:

  • Intermediate data: Redundant copies written to S3 or HDFS between steps
  • Interactive users: Who works in notebooks, Hue or Zeppelin, and with which permissions
  • Security accounts: Users, service accounts, admin roles and the IAM policies behind them
  • Operations: CI/CD scripts, schedulers and alerts in CloudWatch, SNS, SQS, Datadog or PagerDuty
  • Libraries: Third-party tools installed through bootstrap actions or custom images

Close the phase with switch-over criteria per pipeline. For critical ones, run EMR and Databricks side by side and compare outputs before EMR is turned off.

Phase 2: Mapping EMR Architecture and Features to Databricks

Databricks clusters work differently from YARN-based EMR clusters, so compute types, scaling, storage and S3 access all need a deliberate mapping before any workload moves.

Job Compute and All-Purpose Compute

Databricks separates interactive from automated work. All-purpose compute serves notebooks and shared analysis. Job compute runs scheduled workloads, starts per run and costs less per Databricks Unit. Serverless compute needs no cluster configuration at all.

How the Databricks Cluster Manager Differs From YARN

YARN allows several drivers per cluster and several executors per EC2 instance. Databricks runs a single driver per cluster and one executor per instance on classic compute.

Jobs on a shared cluster therefore share one driver, one Spark UI and one log, and each executor gets a larger memory heap. Fewer JVMs mean less overhead, and executors tolerate skewed data a little better before spilling to disk.

Autoscaling, Pools and Fleet Instances

EMR scaling often depends on CloudWatch metrics and specialist tuning. Databricks autoscaling needs only a minimum and maximum worker count, and with no YARN or HDFS to decommission, nodes come and go faster.

  • Pools: Hold idle, ready instances so clusters start and scale faster, with no Databricks Units charged while instances wait
  • Fleet instance types: Resolve to the best-priced instance of the same size, using AWS Spot placement data to pick stronger availability zones

Storage and S3 Access

Many EMR jobs park intermediate results on cluster HDFS. Databricks compute has no HDFS, so chain those steps or write intermediate data to S3, and use storage-optimized instances with the disk cache for data read repeatedly.

Access also changes. EMR reaches S3 through IAM instance roles, while Databricks recommends Unity Catalog external locations and storage credentials that grant access to specific paths per group. S3 has been strongly consistent since December 2020, so listing delays no longer threaten pipelines (AWS).

Phase 3: Migrating EMR Data and Metastores to Delta Lake

Data migration moves EMR metadata into Unity Catalog, lands HDFS data in S3, converts files to Delta Lake and validates the results before any pipeline is switched over.

Catalog and Data Model

Databricks treats external Hive metastores and the Glue Data Catalog as a legacy governance model, so adopt Unity Catalog during the migration. In the interim, Hive metastore federation lets Unity Catalog govern tables registered in Glue or a Hive metastore with read-only access, so both platforms read the same metadata.

Use the move to settle the data model. Bronze holds source-shaped data converted to Delta, Silver joins sources into an enterprise view, and Gold serves consumers through dimensional models, plus sandboxes for analysts. Redesign where the old layout no longer fits, because a one-to-one copy carries old problems forward.

Moving HDFS and EMRFS Data

Some HDFS data is redundant and can be dropped. For the rest, fork ingestion so new data lands in both HDFS and S3, then copy history in the order of the use cases that move first. AWS CLI, S3DistCp or Spark jobs can do the copying, and every path reference in scripts and workflows must be updated.

Converting File Formats

Source FormatRecommended Path
ParquetCONVERT TO DELTA in place, which builds a transaction log and collects statistics without rewriting files
Iceberg on ParquetCONVERT TO DELTA using the Iceberg manifest, or Iceberg reads (UniForm) when Iceberg clients still need access
HudiLoad as a DataFrame and write to Delta; the guide cautions that newer Hudi versions conflict with Databricks Runtime, so confirm the supported version first
ORC, CSV and JSONIngest into Delta through Auto Loader or Lakeflow pipelines

Validating Migrated Data

Compare representative samples from both platforms: row-level values, aggregates by day or category, schemas, key relationships, data quality and business rules. Repeat on fresh samples until confidence is high, and record results in a validation report. Open-source tools such as DataComPy speed up the comparisons.

Phase 4: Converting EMR Spark, Hive and MapReduce Code

Code migration adapts Spark submission, sessions and dependencies for Databricks, upgrades older APIs, and rewrites MapReduce, Sqoop and Hive scripts into Spark and Spark SQL.

Spark Applications

Spark code needs fewer changes than most teams expect. The usual edits are:

  • Spark versions: EMR 5.x releases run Spark 2.x, so the move crosses major versions and needs regression testing
  • Submission: spark-submit flags for master, deploy mode and executor sizing disappear, and JARs run as JAR tasks
  • Sessions: Databricks creates the SparkContext, so code calls getOrCreate() and never stop(), which would disrupt other users on shared compute
  • Configuration: YARN settings and hard-coded storage paths are removed
  • APIs: RDD code moves to DataFrames, since standard and serverless compute do not support RDD APIs

Python projects ship as wheel tasks. Java and Scala dependencies are uploaded as libraries or resolved through Maven on classic compute.

Streaming, MapReduce and Sqoop

DStreams code moves to Structured Streaming, which uses one DataFrame API for batch and streaming. Flink applications can be rewritten there too. MapReduce logic maps onto Spark transformations such as map(), flatMap() and reduceByKey(), with custom input and output formats replaced by Spark’s built-in readers and writers.

Sqoop, retired by Apache in 2021, is often why MapReduce lingers. Spark’s JDBC source replaces it, with credentials in Databricks secrets. One difference needs planning: Sqoop remembered the last value loaded, so Spark code must store its own timestamp or sequence ID and filter on it.

Hive and Impala

Most HiveQL runs on Spark SQL unchanged, so DDL gets the attention:

  • STORED AS becomes USING DELTA
  • EXTERNAL is unnecessary, since a LOCATION makes a table external
  • Bucketing isn’t supported on Delta tables, so liquid clustering replaces it
  • Impala functions need mapping, such as int_months_between to months_between

Hive UDFs run on dedicated compute, or are rewritten as Unity Catalog functions for standard and serverless compute.

Secrets

Inventory secrets in AWS Secrets Manager and Parameter Store, move them into secret scopes or Unity Catalog secrets, update the code that reads them, rotate them and retire the AWS copies once nothing depends on them.

Phase 5: Migrating EMR Data Pipelines and Orchestration

Pipeline migration moves orchestration to Lakeflow Jobs or keeps an external scheduler, reconnects sources and sinks, tunes queries for Databricks and validates each pipeline end to end.

Orchestration Options

EMR pipelines often run under Airflow or Amazon MWAA, Livy, Oozie, AWS Data Pipeline or custom scripts. Two routes exist.

Lakeflow Jobs: Recommended for tighter integration, simpler operations and lineage. Oozie workflows and Airflow DAGs are rebuilt as jobs, and Lakeflow pipelines add declarative processing with data quality checks and slowly changing dimension support.

An external orchestrator: Airflow keeps scheduling through the Databricks provider, using operators such as DatabricksRunNowOperator, DatabricksSubmitRunOperator and DatabricksSqlOperator. Livy submissions are rewritten against the Databricks Jobs REST API to start runs, check status and fetch output.

Sources and Sinks

Source or SinkHow It Connects on Databricks
Amazon S3Unity Catalog external locations, with no credentials in code
Amazon KinesisStructured Streaming source, tuned through fetch timing options such as maxFetchDuration
Amazon RedshiftLakehouse Federation for queries in place, which Databricks recommends, or the legacy Redshift connector that stages data in S3
Kafka, RDS and other databasesNative connectors, JDBC or Lakeflow Connect

Query Tuning and Validation

Most queries run unchanged because the Spark APIs match. Databricks Runtime then adds optimizations: adaptive skew handling, range join hints with a tuned bin size, low shuffle merge for upserts, and predictive I/O with deletion vectors on Photon.

Validate each pipeline for data consistency, performance, functionality and integrations. Add monitoring and alerts, compare cost with the EMR baseline, run user acceptance testing and keep a rollback plan until sign-off.

Two field lessons apply. Databricks clusters are often smaller than their EMR counterparts, so size from usage metrics. And queries that used HDFS for temporary tables or checkpoints need restructuring, or S3 as the temporary location.

Phase 6: Integrating BI and Downstream Tools With Databricks

The final phase repoints dashboards, reports and applications to Databricks SQL, after every upstream table and pipeline they depend on has moved and been validated.

Consumers move last. A dashboard switches only after every table and view behind it has migrated with its pipeline, so plan reporting cutovers around the data waves.

Databricks SQL serves BI workloads with Photon and high concurrency, and works with Power BI, Tableau, Looker, Qlik and ThoughtSpot. The effort depends on the schema:

  • Unchanged schema: Pointing the tool at a Databricks SQL warehouse is usually enough
  • Changed schema: Adjust views in the lakehouse to match what the report expects, then publish them as a new data source

Pilot a few dashboards, then go in waves. Check storage permissions early, since Cloud Fetch lets BI tools download large results in parallel through signed URLs.

For estates with many Power BI, Tableau or ThoughtSpot dashboards, our BrickShift guide explains how dashboards move onto Databricks AI/BI.

EMR to Databricks Migration Best Practices

After workloads land, best practices for platform setup, Delta Lake performance, governance and team enablement keep the new estate fast, secure and cheaper to run than the EMR clusters it replaced.

Set Up the Platform for Scale

  • Use cluster policies to standardize instance types, autoscaling limits and tags
  • Choose instance families for Databricks Runtime, since the best EMR instance types are not always right for it
  • Manage code and workflows through Git and CI/CD with Declarative Automation Bundles

Tune Delta Lake for Performance

PracticeGuidance
PartitioningSkip partitioning below 1 TB and use liquid clustering up to 100 TB; at 100 TB or more, partitioning might help, so test it against liquid clustering and keep at least 1 GB per partition
Data skippingExternal tables collect statistics on the first 32 columns by default, so place filter and join columns early; predictive optimization covers other frequently filtered columns on managed tables
File sizesLet Databricks tune file sizes automatically on managed tables, and set delta.targetFileSize mainly for external tables
StatisticsKeep them fresh with ANALYZE TABLE or predictive optimization

The query profile in Databricks SQL exposes slow operations such as poor file pruning, full table scans and exploding joins.

Govern From Day One

Manage identities at the account level with single sign-on, grant access to groups, and run production workloads as service principals. Unity Catalog’s privilege inheritance carries grants from catalogs and schemas down to tables, keeping rules short and auditable. Our Unity Catalog migration guide covers the setup in detail.

Keep growing champions in each team and use Databricks Academy training. For cost controls worth setting from the first wave, see our Databricks cost optimization guide.

How LatentView Helps With EMR to Databricks Migration

As a Databricks Elite C&SI partner, we run a Databricks Center of Excellence with 400 data engineers, including 150+ certified specialists. For a global leader in digital media, we moved analytics from a legacy Hadoop ecosystem to Databricks, delivering faster pipelines, lower costs and a scalable foundation, as this case study shows.

Transform your EMR clusters into well-governed Databricks workloads with a partner that serves 50+ Fortune 500 clients. MigrateMate, our platform-agnostic migration solution, automates assessment, conversion, migration and validation, and typically reduces migration costs by 30% to 40%. UCMate, our Unity Catalog migration solution, simplifies and accelerates the move to Unity Catalog. Scope your first EMR wave with our data engineering team: Talk to a Data Engineering Expert.

Frequently Asked Questions About EMR to Databricks Migration

1. What Is the Best Strategy for EMR to Databricks Migration?

Databricks recommends a phased migration, with lift and shift for healthy Spark code, refactoring for costly pipelines and redesign for MapReduce, Sqoop and DStreams.

2. What Are the Phases of an EMR to Databricks Migration?

The six phases are discovery and assessment, architecture mapping, data migration, code migration, pipeline migration and downstream tools integration.

3. Do EMR Spark Applications Run on Databricks Without Changes?

Mostly. Spark code needs small changes to sessions, submission and configuration, while RDD, MapReduce, Sqoop and DStreams code needs more rework.

4. What Happens to HDFS Data When Moving From EMR to Databricks?

Databricks compute has no HDFS layer. Needed HDFS data is copied to S3, and pipelines that stored intermediate results on HDFS are restructured.

5. Can Airflow Still Orchestrate Pipelines After Moving to Databricks?

Yes. Airflow can trigger Databricks through the Databricks provider and its operators, though Lakeflow Jobs offer tighter integration and lineage.

6. How Long Does an EMR to Databricks Migration Take?

Duration depends on the number of pipelines, engines and dependencies. Discovery sizes the work, so plan each wave from the workload inventory.

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

CATEGORY

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

Take to the Next Step

"*" indicates required fields

consent*

Related Blogs

This guide helps data and AI leaders move from Azure Machine Learning to Databricks, from choosing…

This guide helps data leaders plan a move from Palantir Foundry to Databricks, decide which workloads…

This guide helps data leaders move pipelines from Azure Data Factory to Databricks one use case…

Scroll to Top