This guide helps data leaders move pipelines from Azure Data Factory to Databricks one use case at a time, so each workload switches over cleanly while the rest of the estate keeps running.
Azure Data Factory to Databricks migration moves ingestion, transformation and orchestration into Lakeflow Jobs, formerly Databricks Workflows, with column-level lineage in Unity Catalog.
Key Takeaways
- Azure Data Factory to Databricks migration replaces ADF’s visual, activity-based orchestration with code-first workflows that run on the same platform as the data
- ADF pipelines fall into a few clear patterns, and each pattern needs a different amount of rework
- Every use case follows the same cycle: rebuild, validate side by side, switch the schedule, then retire the ADF pipeline
- ADF and Databricks can share the work during the transition, and some enterprises keep ADF for narrow ingestion tasks
- Microsoft hasn’t announced a retirement date for Azure Data Factory, so the timing is a business decision
What Changes When You Migrate From Azure Data Factory to Databricks
The biggest change is where control sits: ADF directs work from outside Databricks today, while Lakeflow Jobs schedules, retries and tracks work inside the platform that processes the data.
In most Azure estates, ADF acts as the conductor. It copies data into storage, then calls Databricks notebooks one activity at a time, and each team checks two tools to understand a single failed run.
After migration, that control moves inside Databricks:
- Scheduling and retries: Lakeflow Jobs runs tasks in order, retries failed steps and lets teams repair a failed run without starting again
- Lineage and access: Unity Catalog records which workflows read and write each table, and one permission model covers data and pipelines
- Definitions: Workflows live as code in Git, in place of large exported pipeline files and ARM templates
- Triggers: Workflows can start on a schedule, when new files arrive, when source tables update or continuously
Databricks also flags the cost of the older pattern. With ADF Notebook activities, each notebook can wait for its own cluster to start, and the integration needs workspace admin rights or the right to create clusters (Source). For a side-by-side view of the two services, see our Azure Data Factory vs Databricks comparison.
When to Migrate From Azure Data Factory to Databricks vs. Keep ADF
Migrate when Databricks already does most of the processing and ADF mainly schedules it, and keep ADF where it coordinates many non-Databricks services or reaches on-premises sources.
Migrate to Databricks when:
- ADF mainly triggers Databricks notebooks
- Failures take time to trace because logs sit in two tools
- Lineage from source to report has gaps between ADF and Databricks
- Pipeline changes are slow to review and promote between environments
- Teams want features ADF calls can’t reach, such as serverless compute or Databricks SQL
Keep ADF, fully or partly, when:
- Pipelines coordinate many Azure services where Databricks plays a small role
- Copy activities rely on ADF’s self-hosted integration runtime to reach on-premises systems
- Mapping Data Flows hold critical logic that the team can’t rebuild yet
The decision doesn’t depend on a deadline. Microsoft hasn’t announced a retirement date for Azure Data Factory, though it does offer tooling to upgrade ADF pipelines into Data Factory in Microsoft Fabric (Source). So the real question is where orchestration should live, and it usually follows where the data is processed.
What You’re Actually Migrating From Azure Data Factory to Databricks
An ADF migration covers far more than pipelines: triggers, connections, integration runtimes, data flows, parameters and alerts all need a new home, and each lands in a different part of Databricks.
Teams often scope an ADF exit by counting pipelines. The real inventory is wider, because each pipeline depends on objects defined elsewhere in the factory:
| ADF Asset | What It Holds Today | Where It Lands on Databricks |
|---|---|---|
| Pipelines and activities | The sequence of steps, retries and dependencies | Lakeflow Jobs workflows and their tasks |
| Triggers | Schedules and storage events that start runs | Scheduled, file arrival or table update triggers |
| Linked services and datasets | Connection details and pointers to source data | Unity Catalog connections, external locations and tables |
| Integration runtimes | The compute that reaches cloud and on-premises sources | Serverless or dedicated compute, plus private networking for Lakeflow Connect |
| Mapping Data Flows | Visual transformation logic | Lakeflow pipelines, SQL or PySpark |
| Global parameters and variables | Shared settings and values passed between steps | Workflow parameters, bundle variables and task values |
| Alerts and monitoring rules | Who hears about failures and how | Failure notifications to email, Slack or webhooks |
| ARM templates and Git definitions | How pipelines move between environments | Declarative Automation Bundles in Git |
Pipelines themselves also differ in how much work they need. Sorting them early shows where the effort really sits:
| Pipeline Type | What It Looks Like | Rework Needed |
|---|---|---|
| Orchestration only | ADF calls Databricks notebooks in order and does little else | Low, since notebooks reading widget parameters usually work unchanged as notebook tasks |
| Ingestion-heavy | Copy activities pull from databases, files and SaaS apps | Medium, using Auto Loader for files and Lakeflow Connect for sources such as Salesforce, Workday, ServiceNow and SQL Server |
| Mapping Data Flows | Business logic built in ADF’s visual designer | High, since the logic must be rebuilt as code and tested against the original |
| Cross-service or on-premises | Calls to Azure Functions or Logic Apps, or copies through a self-hosted integration runtime | High, often kept in ADF until private networking is ready |
Databricks’ Lakebridge analyzer lists ADF and SSIS among its supported sources, which speeds up this inventory across large estates.
How to Structure an Azure Data Factory to Databricks Migration
Structure the migration around a few parallel workstreams and waves organized by business domain, so platform work, rebuilds and switch-overs run at a steady rhythm.
Five workstreams usually run side by side:
- Platform foundations: Unity Catalog, service principals, secrets, networking and the release process, built once and shared by every wave
- Ingestion: Replacing copy activities with Auto Loader, Lakeflow Connect or a narrowed ADF ingestion layer
- Orchestration and transformation: Rebuilding pipelines as Lakeflow Jobs workflows and data flows as code
- Validation and switch-over: Running old and new side by side, comparing results and switching schedules
- Enablement: Training engineers on Lakeflow Jobs, bundles and Git-based releases before their pipelines move
Waves follow business domains, such as finance or supply chain. A domain-based wave moves related pipelines together, keeps their dependencies on one platform and gives a business owner a clear moment to sign off.
Clear roles keep waves moving. A platform team owns foundations and standards, pipeline owners rebuild and validate their use cases, and business owners confirm the numbers before each switch. One shared register records every pipeline, its type, its current scheduler and its switch-over date, which becomes the single view of progress for sponsors.
Step-by-Step Azure Data Factory to Databricks Migration Strategy
The strategy builds shared foundations once, then moves each use case through rebuild, side-by-side validation and a clean schedule switch, so only one scheduler ever owns a pipeline.
Step 1: Inventory and Classify Every Pipeline
Export the full list of pipelines from the factory and capture, for each one:
- Its triggers, schedules and run frequency
- The activities it runs and the sources and targets it touches
- Linked services, integration runtimes and secrets it depends on
- Downstream reports or systems that rely on its output
- A named owner in the business and in engineering
Then tag each pipeline with one of the four types above. Pipelines that haven’t run in months are candidates for retirement, which shrinks the scope before rebuilding starts.
Step 2: Prioritize Use Cases and Choose a Pilot
Rank use cases by business value and rework effort, then pick a pilot that matters to a business team without putting a critical feed at risk. An orchestration-only pipeline feeding a visible report is usually ideal, because it proves the full cycle quickly.
Step 3: Build Shared Foundations Once
Set up everything later waves will reuse before the first rebuild:
- Access: Unity Catalog permissions for the tables, volumes and connections pipelines use
- Identity: Service principals to run production workflows
- Secrets: Key Vault-backed secret scopes for credentials already held in Azure Key Vault
- Release process: Declarative Automation Bundles, formerly Databricks Asset Bundles, stored in Git and promoted through Azure DevOps or GitHub Actions
- Environments: Separate development, test and production workspaces or catalogs, with the same workflow definitions deployed to each
Getting these right once removes most of the friction from every later use case.
Step 4: Rebuild the Use Case in Lakeflow Jobs
Recreate the pipeline in development, with tasks, dependencies, retries and parameters defined in the workflow. Notebook activities become notebook tasks, and the mapping table in the next section covers triggers, loops, branches and parameters.
Ingestion needs its own decisions here. Files usually move to Auto Loader, supported databases and SaaS apps to Lakeflow Connect, and on-premises feeds either to Lakeflow Connect over private networking or to a narrowed ADF copy activity for now. Logic buried in deeply nested ADF expressions is usually clearer as tested Python or SQL than as a long chain of tasks.
Step 5: Validate Side by Side
Run the new workflow and the ADF pipeline on the same inputs in a test environment, then compare:
- Row counts and totals for every table the pipeline writes
- Run times and failure behavior, including retries
- The downstream reports and extracts that use the output
Keep both running until results match across a full business cycle, such as a week of daily loads or one month-end close.
Step 6: Switch the Schedule Without Double Runs
Pick a quiet window, then pause the ADF trigger and enable the Databricks schedule in the same change. Watch the first few production runs closely and tell downstream teams the switch has happened, so any question goes to the right owner.
Step 7: Retire the ADF Pipeline and Repeat
Keep the paused ADF pipeline as a fallback for an agreed period. Once that ends, delete it along with any triggers and linked services nothing else uses, update the migration register and start the next use case.
Mapping Azure Data Factory Activities and Triggers to Lakeflow Jobs
Most ADF control flow has a direct Lakeflow Jobs equivalent, while deeply nested loops and long expressions are usually easier to maintain as tested code.
| ADF Concept | Lakeflow Jobs Equivalent |
|---|---|
| Schedule trigger | Scheduled trigger |
| Storage event trigger | File arrival or table update trigger |
| ForEach activity | For each task |
| If Condition activity | If/else task |
| Execute Pipeline activity | A task that runs another workflow |
| Pipeline parameters | Workflow parameters |
| Set Variable and activity outputs | Task values |
| Activity dependency conditions | Run if conditions |
| Linked services and Key Vault secrets | Unity Catalog connections and Key Vault-backed secret scopes |
The table covers most pipelines, but not every one translates task for task. Practitioners often find that ADF pipelines with loops inside loops and long dynamic expressions turn into hard-to-read task graphs. Moving that logic into a notebook or Python module, with its own tests, is usually simpler to maintain.
Agree a parameter contract early, with consistent names for dates, environments and source systems. Each later use case then moves faster, since notebooks and workflows already speak the same language.
Coexistence Patterns for ADF and Databricks During Migration
Most migrations run both tools for months, so the transition needs clear rules on which tool owns each pipeline and how the two hand work to each other.
Three coexistence patterns cover most estates:
- ADF calls Databricks workflows: ADF keeps scheduling but triggers a whole Databricks workflow through the Databricks Job activity, which needs no cluster defined in the ADF linked service (Source). Databricks calls this the new best practice for teams still using Notebook activities
- ADF lands data, Databricks does the rest: ADF keeps narrow copy activities, often for on-premises sources, while Lakeflow Jobs owns everything downstream
- Databricks owns the whole pipeline: The use case runs fully in Databricks, and ADF no longer touches it
Write down which pattern each pipeline follows and who owns it. Without that record, teams lose track of which tool runs what, and duplicate loads or missed runs follow.
Security, CI/CD and Governance After Migrating From ADF to Databricks
Once pipelines run on Databricks, security teams review one permission model, auditors trace lineage down to columns and every pipeline change goes through Git.
The foundations from Step 3 change how governance works day to day:
- Audits get simpler: Unity Catalog captures lineage automatically, down to the column level, and shows which workflows read and write each table
- Access reviews shrink: Permissions for tables, volumes and connections sit in Unity Catalog, in place of separate rules across ADF, storage and Databricks
- Changes are traceable: Every workflow change is a reviewed commit, and ADF’s JSON definitions and ARM templates retire with the last pipeline
- Identities stay stable: Workflows running as service principals keep working when people change roles, and audit logs show one consistent identity
Network controls carry over too. Virtual network deployment and Azure Private Link keep traffic private, matching what ADF used.
Post-Migration Monitoring and Cost Optimization on Databricks
Track reliability, run times and total cost per use case after each switch, so the business case is proven with real numbers before the next domain moves.
Reliability: Lakeflow Jobs shows each run with its tasks, durations and errors, and failed tasks can be repaired without rerunning the whole workflow. Notifications can go to email, Slack or webhooks when a run fails or takes too long.
Cost: ADF bills for orchestration, data movement and data flow compute, while Databricks bills for the compute that does the work. Compare total cost per use case, including the weeks both tools run in parallel. Serverless compute and compute shared across tasks cut cluster start-up time and avoid paying for always-on all-purpose clusters. Our Databricks cost optimization guide covers the controls worth setting early.
Clean-up: Delete retired ADF pipelines, triggers and unused linked services once the fallback period ends. Leftover items invite accidental reruns and keep credentials alive longer than needed.
Best Practices for Azure Data Factory to Databricks Migration
Strong ADF migrations start with simple pipelines, keep one scheduler per pipeline and build shared foundations once, so every later use case moves faster.
- Start with orchestration-only pipelines: They move with the least change and give teams early proof
- Keep one scheduler per pipeline: Switch the Databricks schedule on and the ADF trigger off in the same change
- Build foundations before the first rebuild: Permissions, service principals and bundles set up once save rework later
- Agree a parameter contract early: Consistent names for dates, environments and sources speed up every later use case
- Move domain by domain: Keeping related pipelines together keeps dependencies inside one platform
- Keep ADF only where it adds value: Narrow ingestion tasks can stay while the rest moves
- Measure each use case before the next: Reliability and cost numbers from one switch make the case for the next domain
For broader planning across other source systems, read our Databricks migration guide.
Key Pitfalls to Avoid in Azure Data Factory to Databricks Migration
Most pitfalls in ADF migrations come from copying pipelines activity for activity, or from underestimating on-premises connectivity and the switch between schedulers.
| Pitfall | What to Do Instead |
|---|---|
| Copying every activity one to one | Move nested loops and long expressions into tested code |
| Leaving the ADF trigger active after go-live | Switch both schedules in the same change to avoid duplicate loads |
| Treating on-premises access as an afterthought | Plan private networking early, or keep ADF for those feeds |
| Running workflows under personal accounts | Use service principals from the first production workflow |
| Deleting ADF pipelines straight away | Pause first and delete after the agreed fallback period |
| Skipping the ownership record | Log which tool runs each pipeline and who owns it |
Most of these pitfalls surface at the switch-over stage, when two schedulers briefly overlap. A short checklist reviewed before every cutover, covering triggers, owners and fallback dates, catches them before they reach production.
How LatentView Helps With Azure Data Factory to Databricks Migration
Our Databricks Center of Excellence has 400 data engineers, 150+ of them Databricks certified, and we are a Databricks Elite C&SI partner. For a global automotive supplier, we modernized the entire data platform on Databricks in under two months, rebuilding pipelines with Spark, governing access with Unity Catalog and automating deployments with CI/CD, as this case study shows.
Turn your ADF estate into governed Databricks pipelines with a partner that brings 20+ years of experience serving Fortune 500 clients. When legacy databases or warehouses move alongside ADF, our MigrateMate solution handles assessment, conversion and validation, and typically cuts migration costs by 30% to 40%. Plan your first ADF use case with our Databricks partnership team and see which pipelines should move first.
Frequently Asked Questions About Azure Data Factory to Databricks Migration
1. Can ADF Pipelines Move to Databricks One Use Case at a Time?
Yes. Each use case can be rebuilt in Lakeflow Jobs, validated in development, then switched over while other pipelines keep running in ADF.
2. Is Azure Data Factory Being Retired?
Microsoft hasn’t announced a retirement date for Azure Data Factory. Only individual features are retiring, such as Memory Optimized Data Flows on March 31, 2027.
3. How Does Azure Data Factory Connect to Databricks?
Azure Data Factory connects through a Databricks linked service. The Databricks Job activity triggers a full Databricks workflow and passes parameters to it.
4. Which ADF Pipelines Should Move to Databricks First?
Orchestration-only pipelines that simply call Databricks notebooks move first. These pipelines need the least rework and prove the cutover process early.
5. Do Databricks Notebooks Need Changes When Leaving ADF?
Often very few. Notebooks that read parameters through widgets usually work as notebook tasks in Lakeflow Jobs with little or no change.