Migrating From Azure Data Factory to Databricks Lakeflow Jobs: A Practical Enterprise Guide

 & LatentView Analytics

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

SHARE

Table of Contents

This guide helps data leaders move pipelines from Azure Data Factory to Databricks one use case at a time, so each workload switches over cleanly while the rest of the estate keeps running.

Azure Data Factory to Databricks migration moves ingestion, transformation and orchestration into Lakeflow Jobs, formerly Databricks Workflows, with column-level lineage in Unity Catalog.

Key Takeaways

  • Azure Data Factory to Databricks migration replaces ADF’s visual, activity-based orchestration with code-first workflows that run on the same platform as the data
  • ADF pipelines fall into a few clear patterns, and each pattern needs a different amount of rework
  • Every use case follows the same cycle: rebuild, validate side by side, switch the schedule, then retire the ADF pipeline
  • ADF and Databricks can share the work during the transition, and some enterprises keep ADF for narrow ingestion tasks
  • Microsoft hasn’t announced a retirement date for Azure Data Factory, so the timing is a business decision

What Changes When You Migrate From Azure Data Factory to Databricks

The biggest change is where control sits: ADF directs work from outside Databricks today, while Lakeflow Jobs schedules, retries and tracks work inside the platform that processes the data.

In most Azure estates, ADF acts as the conductor. It copies data into storage, then calls Databricks notebooks one activity at a time, and each team checks two tools to understand a single failed run.

After migration, that control moves inside Databricks:

  • Scheduling and retries: Lakeflow Jobs runs tasks in order, retries failed steps and lets teams repair a failed run without starting again
  • Lineage and access: Unity Catalog records which workflows read and write each table, and one permission model covers data and pipelines
  • Definitions: Workflows live as code in Git, in place of large exported pipeline files and ARM templates
  • Triggers: Workflows can start on a schedule, when new files arrive, when source tables update or continuously

Databricks also flags the cost of the older pattern. With ADF Notebook activities, each notebook can wait for its own cluster to start, and the integration needs workspace admin rights or the right to create clusters (Source). For a side-by-side view of the two services, see our Azure Data Factory vs Databricks comparison.

When to Migrate From Azure Data Factory to Databricks vs. Keep ADF

Migrate when Databricks already does most of the processing and ADF mainly schedules it, and keep ADF where it coordinates many non-Databricks services or reaches on-premises sources.

Migrate to Databricks when:

  • ADF mainly triggers Databricks notebooks
  • Failures take time to trace because logs sit in two tools
  • Lineage from source to report has gaps between ADF and Databricks
  • Pipeline changes are slow to review and promote between environments
  • Teams want features ADF calls can’t reach, such as serverless compute or Databricks SQL

Keep ADF, fully or partly, when:

  • Pipelines coordinate many Azure services where Databricks plays a small role
  • Copy activities rely on ADF’s self-hosted integration runtime to reach on-premises systems
  • Mapping Data Flows hold critical logic that the team can’t rebuild yet

The decision doesn’t depend on a deadline. Microsoft hasn’t announced a retirement date for Azure Data Factory, though it does offer tooling to upgrade ADF pipelines into Data Factory in Microsoft Fabric (Source). So the real question is where orchestration should live, and it usually follows where the data is processed.

What You’re Actually Migrating From Azure Data Factory to Databricks

An ADF migration covers far more than pipelines: triggers, connections, integration runtimes, data flows, parameters and alerts all need a new home, and each lands in a different part of Databricks.

Teams often scope an ADF exit by counting pipelines. The real inventory is wider, because each pipeline depends on objects defined elsewhere in the factory:

ADF AssetWhat It Holds TodayWhere It Lands on Databricks
Pipelines and activitiesThe sequence of steps, retries and dependenciesLakeflow Jobs workflows and their tasks
TriggersSchedules and storage events that start runsScheduled, file arrival or table update triggers
Linked services and datasetsConnection details and pointers to source dataUnity Catalog connections, external locations and tables
Integration runtimesThe compute that reaches cloud and on-premises sourcesServerless or dedicated compute, plus private networking for Lakeflow Connect
Mapping Data FlowsVisual transformation logicLakeflow pipelines, SQL or PySpark
Global parameters and variablesShared settings and values passed between stepsWorkflow parameters, bundle variables and task values
Alerts and monitoring rulesWho hears about failures and howFailure notifications to email, Slack or webhooks
ARM templates and Git definitionsHow pipelines move between environmentsDeclarative Automation Bundles in Git

Pipelines themselves also differ in how much work they need. Sorting them early shows where the effort really sits:

Pipeline TypeWhat It Looks LikeRework Needed
Orchestration onlyADF calls Databricks notebooks in order and does little elseLow, since notebooks reading widget parameters usually work unchanged as notebook tasks
Ingestion-heavyCopy activities pull from databases, files and SaaS appsMedium, using Auto Loader for files and Lakeflow Connect for sources such as Salesforce, Workday, ServiceNow and SQL Server
Mapping Data FlowsBusiness logic built in ADF’s visual designerHigh, since the logic must be rebuilt as code and tested against the original
Cross-service or on-premisesCalls to Azure Functions or Logic Apps, or copies through a self-hosted integration runtimeHigh, often kept in ADF until private networking is ready

Databricks’ Lakebridge analyzer lists ADF and SSIS among its supported sources, which speeds up this inventory across large estates.

How to Structure an Azure Data Factory to Databricks Migration

Structure the migration around a few parallel workstreams and waves organized by business domain, so platform work, rebuilds and switch-overs run at a steady rhythm.

Five workstreams usually run side by side:

  • Platform foundations: Unity Catalog, service principals, secrets, networking and the release process, built once and shared by every wave
  • Ingestion: Replacing copy activities with Auto Loader, Lakeflow Connect or a narrowed ADF ingestion layer
  • Orchestration and transformation: Rebuilding pipelines as Lakeflow Jobs workflows and data flows as code
  • Validation and switch-over: Running old and new side by side, comparing results and switching schedules
  • Enablement: Training engineers on Lakeflow Jobs, bundles and Git-based releases before their pipelines move

Waves follow business domains, such as finance or supply chain. A domain-based wave moves related pipelines together, keeps their dependencies on one platform and gives a business owner a clear moment to sign off.

Clear roles keep waves moving. A platform team owns foundations and standards, pipeline owners rebuild and validate their use cases, and business owners confirm the numbers before each switch. One shared register records every pipeline, its type, its current scheduler and its switch-over date, which becomes the single view of progress for sponsors.

Step-by-Step Azure Data Factory to Databricks Migration Strategy

The strategy builds shared foundations once, then moves each use case through rebuild, side-by-side validation and a clean schedule switch, so only one scheduler ever owns a pipeline.

Step 1: Inventory and Classify Every Pipeline

Export the full list of pipelines from the factory and capture, for each one:

  • Its triggers, schedules and run frequency
  • The activities it runs and the sources and targets it touches
  • Linked services, integration runtimes and secrets it depends on
  • Downstream reports or systems that rely on its output
  • A named owner in the business and in engineering

Then tag each pipeline with one of the four types above. Pipelines that haven’t run in months are candidates for retirement, which shrinks the scope before rebuilding starts.

Step 2: Prioritize Use Cases and Choose a Pilot

Rank use cases by business value and rework effort, then pick a pilot that matters to a business team without putting a critical feed at risk. An orchestration-only pipeline feeding a visible report is usually ideal, because it proves the full cycle quickly.

Step 3: Build Shared Foundations Once

Set up everything later waves will reuse before the first rebuild:

  • Access: Unity Catalog permissions for the tables, volumes and connections pipelines use
  • Identity: Service principals to run production workflows
  • Secrets: Key Vault-backed secret scopes for credentials already held in Azure Key Vault
  • Release process: Declarative Automation Bundles, formerly Databricks Asset Bundles, stored in Git and promoted through Azure DevOps or GitHub Actions
  • Environments: Separate development, test and production workspaces or catalogs, with the same workflow definitions deployed to each

Getting these right once removes most of the friction from every later use case.

Step 4: Rebuild the Use Case in Lakeflow Jobs

Recreate the pipeline in development, with tasks, dependencies, retries and parameters defined in the workflow. Notebook activities become notebook tasks, and the mapping table in the next section covers triggers, loops, branches and parameters.

Ingestion needs its own decisions here. Files usually move to Auto Loader, supported databases and SaaS apps to Lakeflow Connect, and on-premises feeds either to Lakeflow Connect over private networking or to a narrowed ADF copy activity for now. Logic buried in deeply nested ADF expressions is usually clearer as tested Python or SQL than as a long chain of tasks.

Step 5: Validate Side by Side

Run the new workflow and the ADF pipeline on the same inputs in a test environment, then compare:

  1. Row counts and totals for every table the pipeline writes
  2. Run times and failure behavior, including retries
  3. The downstream reports and extracts that use the output

Keep both running until results match across a full business cycle, such as a week of daily loads or one month-end close.

Step 6: Switch the Schedule Without Double Runs

Pick a quiet window, then pause the ADF trigger and enable the Databricks schedule in the same change. Watch the first few production runs closely and tell downstream teams the switch has happened, so any question goes to the right owner.

Step 7: Retire the ADF Pipeline and Repeat

Keep the paused ADF pipeline as a fallback for an agreed period. Once that ends, delete it along with any triggers and linked services nothing else uses, update the migration register and start the next use case.

Mapping Azure Data Factory Activities and Triggers to Lakeflow Jobs

Most ADF control flow has a direct Lakeflow Jobs equivalent, while deeply nested loops and long expressions are usually easier to maintain as tested code.

ADF ConceptLakeflow Jobs Equivalent
Schedule triggerScheduled trigger
Storage event triggerFile arrival or table update trigger
ForEach activityFor each task
If Condition activityIf/else task
Execute Pipeline activityA task that runs another workflow
Pipeline parametersWorkflow parameters
Set Variable and activity outputsTask values
Activity dependency conditionsRun if conditions
Linked services and Key Vault secretsUnity Catalog connections and Key Vault-backed secret scopes

The table covers most pipelines, but not every one translates task for task. Practitioners often find that ADF pipelines with loops inside loops and long dynamic expressions turn into hard-to-read task graphs. Moving that logic into a notebook or Python module, with its own tests, is usually simpler to maintain.

Agree a parameter contract early, with consistent names for dates, environments and source systems. Each later use case then moves faster, since notebooks and workflows already speak the same language.

Coexistence Patterns for ADF and Databricks During Migration

Most migrations run both tools for months, so the transition needs clear rules on which tool owns each pipeline and how the two hand work to each other.

Three coexistence patterns cover most estates:

  • ADF calls Databricks workflows: ADF keeps scheduling but triggers a whole Databricks workflow through the Databricks Job activity, which needs no cluster defined in the ADF linked service (Source). Databricks calls this the new best practice for teams still using Notebook activities
  • ADF lands data, Databricks does the rest: ADF keeps narrow copy activities, often for on-premises sources, while Lakeflow Jobs owns everything downstream
  • Databricks owns the whole pipeline: The use case runs fully in Databricks, and ADF no longer touches it

Write down which pattern each pipeline follows and who owns it. Without that record, teams lose track of which tool runs what, and duplicate loads or missed runs follow.

Security, CI/CD and Governance After Migrating From ADF to Databricks

Once pipelines run on Databricks, security teams review one permission model, auditors trace lineage down to columns and every pipeline change goes through Git.

The foundations from Step 3 change how governance works day to day:

  • Audits get simpler: Unity Catalog captures lineage automatically, down to the column level, and shows which workflows read and write each table
  • Access reviews shrink: Permissions for tables, volumes and connections sit in Unity Catalog, in place of separate rules across ADF, storage and Databricks
  • Changes are traceable: Every workflow change is a reviewed commit, and ADF’s JSON definitions and ARM templates retire with the last pipeline
  • Identities stay stable: Workflows running as service principals keep working when people change roles, and audit logs show one consistent identity

Network controls carry over too. Virtual network deployment and Azure Private Link keep traffic private, matching what ADF used.

Post-Migration Monitoring and Cost Optimization on Databricks

Track reliability, run times and total cost per use case after each switch, so the business case is proven with real numbers before the next domain moves.

Reliability: Lakeflow Jobs shows each run with its tasks, durations and errors, and failed tasks can be repaired without rerunning the whole workflow. Notifications can go to email, Slack or webhooks when a run fails or takes too long.

Cost: ADF bills for orchestration, data movement and data flow compute, while Databricks bills for the compute that does the work. Compare total cost per use case, including the weeks both tools run in parallel. Serverless compute and compute shared across tasks cut cluster start-up time and avoid paying for always-on all-purpose clusters. Our Databricks cost optimization guide covers the controls worth setting early.

Clean-up: Delete retired ADF pipelines, triggers and unused linked services once the fallback period ends. Leftover items invite accidental reruns and keep credentials alive longer than needed.

Best Practices for Azure Data Factory to Databricks Migration

Strong ADF migrations start with simple pipelines, keep one scheduler per pipeline and build shared foundations once, so every later use case moves faster.

  • Start with orchestration-only pipelines: They move with the least change and give teams early proof
  • Keep one scheduler per pipeline: Switch the Databricks schedule on and the ADF trigger off in the same change
  • Build foundations before the first rebuild: Permissions, service principals and bundles set up once save rework later
  • Agree a parameter contract early: Consistent names for dates, environments and sources speed up every later use case
  • Move domain by domain: Keeping related pipelines together keeps dependencies inside one platform
  • Keep ADF only where it adds value: Narrow ingestion tasks can stay while the rest moves
  • Measure each use case before the next: Reliability and cost numbers from one switch make the case for the next domain

For broader planning across other source systems, read our Databricks migration guide.

Key Pitfalls to Avoid in Azure Data Factory to Databricks Migration

Most pitfalls in ADF migrations come from copying pipelines activity for activity, or from underestimating on-premises connectivity and the switch between schedulers.

PitfallWhat to Do Instead
Copying every activity one to oneMove nested loops and long expressions into tested code
Leaving the ADF trigger active after go-liveSwitch both schedules in the same change to avoid duplicate loads
Treating on-premises access as an afterthoughtPlan private networking early, or keep ADF for those feeds
Running workflows under personal accountsUse service principals from the first production workflow
Deleting ADF pipelines straight awayPause first and delete after the agreed fallback period
Skipping the ownership recordLog which tool runs each pipeline and who owns it

Most of these pitfalls surface at the switch-over stage, when two schedulers briefly overlap. A short checklist reviewed before every cutover, covering triggers, owners and fallback dates, catches them before they reach production.

How LatentView Helps With Azure Data Factory to Databricks Migration

Our Databricks Center of Excellence has 400 data engineers, 150+ of them Databricks certified, and we are a Databricks Elite C&SI partner. For a global automotive supplier, we modernized the entire data platform on Databricks in under two months, rebuilding pipelines with Spark, governing access with Unity Catalog and automating deployments with CI/CD, as this case study shows.

Turn your ADF estate into governed Databricks pipelines with a partner that brings 20+ years of experience serving Fortune 500 clients. When legacy databases or warehouses move alongside ADF, our MigrateMate solution handles assessment, conversion and validation, and typically cuts migration costs by 30% to 40%. Plan your first ADF use case with our Databricks partnership team and see which pipelines should move first.

Frequently Asked Questions About Azure Data Factory to Databricks Migration

1. Can ADF Pipelines Move to Databricks One Use Case at a Time?

Yes. Each use case can be rebuilt in Lakeflow Jobs, validated in development, then switched over while other pipelines keep running in ADF.

2. Is Azure Data Factory Being Retired?

Microsoft hasn’t announced a retirement date for Azure Data Factory. Only individual features are retiring, such as Memory Optimized Data Flows on March 31, 2027.

3. How Does Azure Data Factory Connect to Databricks?

Azure Data Factory connects through a Databricks linked service. The Databricks Job activity triggers a full Databricks workflow and passes parameters to it.

4. Which ADF Pipelines Should Move to Databricks First?

Orchestration-only pipelines that simply call Databricks notebooks move first. These pipelines need the least rework and prove the cutover process early.

5. Do Databricks Notebooks Need Changes When Leaving ADF?

Often very few. Notebooks that read parameters through widgets usually work as notebook tasks in Lakeflow Jobs with little or no change.

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

CATEGORY

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

Take to the Next Step

"*" indicates required fields

consent*

Related Blogs

This guide helps data leaders plan a move from Palantir Foundry to Databricks, decide which workloads…

This guide helps data leaders plan a move from Google BigQuery to Databricks, pick the right…

This guide helps data leaders plan an Azure Synapse to Databricks migration that consolidates several Synapse…

Scroll to Top