Palantir Foundry to Databricks Migration: A Practical Guide to Full and Hybrid Moves

 & LatentView Analytics

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

SHARE

Table of Contents

This guide helps data leaders plan a move from Palantir Foundry to Databricks, decide which workloads to move and which to keep, and switch over without disrupting the teams that rely on Foundry today.

Palantir Foundry to Databricks migration moves datasets, transformation logic and pipelines into Databricks, while many enterprises keep Foundry’s Ontology and operational apps connected to it.

Key Takeaways

  • Palantir Foundry to Databricks migration shifts data engineering, storage and compute from Foundry onto Databricks, governed by Unity Catalog.
  • Enterprises choose between reading Foundry data in place, a hybrid split and a full exit, and each choice changes the scope.
  • Datasets and code transforms move with tooling help, while the Ontology, Workshop apps and AIP workloads need the most thought.
  • Palantir and Databricks announced a strategic product partnership on March 13, 2025, so both platforms can stay connected during and after the move.
  • Databricks reads Foundry data in place only for Foundry Iceberg tables, and access is read-only.
  • Running both platforms in parallel and comparing outputs before each switch reduces the risk of every cutover.

Palantir Foundry vs Databricks: What Changes When Workloads Move

Foundry bundles data integration, transformation, apps and security in one platform, while Databricks separates them, and those differences shape every migration decision.

Foundry combines data integration, transformation, an operational data model called the Ontology and app building in one platform.

On Databricks, Delta Lake and Apache Iceberg tables hold the data, Unity Catalog governs data and models, and compute is billed by usage.

Here is how the core Foundry building blocks translate:

Foundry ConceptHow It Works in FoundryDatabricks Counterpart
DatasetA versioned collection of files, updated through SNAPSHOT, APPEND, UPDATE or DELETE transactionsA Delta table in Unity Catalog, with version history and time travel
BranchIsolated copies of pipelines and data for safe changesGit branches for code, plus separate development catalogs for data
ProjectFoundry organizes data and resources in Projects, which carry role-based accessCatalogs and schemas with Unity Catalog grants
MarkingA mandatory control that travels with data into every derived datasetGoverned tags with attribute-based access control policies, plus row filters and column masks
Build and scheduleRebuilds datasets in dependency orderLakeflow Jobs workflows with task dependencies
Compute-secondsFoundry’s unit for measuring compute workDatabricks Units (DBUs), a normalized unit of processing power

Markings need early attention. In Foundry, mandatory controls such as markings travel with data through derivation. Unity Catalog uses governed tags and attribute-based policies instead.

Some of those Databricks controls are still in Beta, including policies on views and DENY policies.

The move doesn’t have to end the relationship with Palantir. On March 13, 2025, Palantir and Databricks announced a strategic product partnership, and the release names bp and several US government agencies among customers already using both.

For a wider comparison, see our Databricks vs Palantir guide.

Full Migration or Hybrid: Choosing How Far to Move Off Palantir Foundry

Enterprises can read Foundry data from Databricks, split the work between the two or exit Foundry completely, and the right choice depends on how much value sits in the Ontology.

ApproachWhat Moves to DatabricksWhat Stays in FoundryFits When
Read in placeAnalytics and AI that query Foundry Iceberg tablesAll pipelines, the Ontology and appsTeams want Databricks analytics on Foundry data without moving anything yet
Hybrid splitData storage, transformation and pipeline computeThe Ontology, Workshop apps and operational workflowsFoundry apps deliver real value, but processing costs and engineering limits are rising
Full exitData, pipelines, models, semantic logic and appsNothing, once the switch-over endsThe organization wants one platform and can rebuild operational apps elsewhere

Reading in place has narrow limits. Databricks catalog federation reads only Foundry Iceberg tables, access is read-only and views are not supported.

Foundry Iceberg tables are a separate resource type from standard Foundry datasets.

Setup needs Databricks Runtime 18 or above, pro or serverless SQL warehouses and an OAuth application in Foundry. The Databricks page shows no release-stage label, so confirm the feature’s status before planning around it.

In a hybrid split, Databricks takes over the heavy processing and Foundry reads the finished tables through its Databricks connector. Palantir lists virtual tables, bulk import and compute pushdown for Databricks as generally available.

A full exit is a bigger decision. On Databricks, the Ontology’s jobs are split across separate tools such as metric views, Databricks Apps and Agent Bricks, so the Ontology, its actions and the Workshop apps need redesign rather than conversion.

What You’re Actually Migrating From Palantir Foundry to Databricks

A Foundry migration covers datasets, code and no-code pipelines, schedules, permissions, models and the Ontology layer, with each asset landing in its own part of Databricks.

Foundry AssetWhat It HoldsWhere It Lands on Databricks
DatasetsVersioned tables and files stored inside FoundryDelta or Iceberg tables in Unity Catalog
Python and SQL transformsCode in Foundry Code Repositories that builds datasetsNotebooks, SQL and Lakeflow pipelines
Pipeline Builder flowsVisual, no-code transformation logicLakeflow pipelines or SQL, rebuilt from the flow’s logic
Schedules and build dependenciesWhen datasets rebuild and in what orderLakeflow Jobs workflows with task dependencies
Data Connection syncsIngestion from source systemsLakeflow Connect, Auto Loader or partner tools
Permissions and markingsWho can see which data, including mandatory markingsUnity Catalog grants, row filters, column masks and tag-based policies
ModelsMachine learning models built or hosted in FoundryMLflow models registered in Unity Catalog
Ontology, Workshop, Contour, Quiver and AIP logicBusiness objects, actions, operational apps, analysis tools and AI workflowsKept in Foundry, or redesigned with Databricks apps, AI and BI tools

The data layer has the most tooling support. Palantir’s Read Table API returns tables as Arrow or CSV, and Foundry datasets store tabular data in open formats such as Parquet.

If you push data to Databricks through Data Connection table exports, Palantir sets limits. The underlying files must be Parquet, array, map and struct columns fail to export, and only the master branch is exported.

The Ontology layer has the least tooling support, because it captures business meaning and actions designed for Foundry.

Two asset groups are easy to miss. Pipeline Builder flows live outside Code Repositories, and markings and permissions sit on Foundry resources, so they must be rebuilt in Unity Catalog.

How to Structure a Palantir Foundry to Databricks Migration

Organize the migration into a handful of workstreams and waves organized by business domain, with the Ontology decision made early so every wave knows what it hands back to Foundry.

Five workstreams usually run side by side:

WorkstreamScopeDone When
DataMoving datasets into Delta or Iceberg tables and keeping them in sync during the transitionEvery in-scope dataset has a governed Unity Catalog table
LogicConverting code transforms and rebuilding Pipeline Builder flowsConverted logic reproduces Foundry outputs on the same inputs
GovernanceRecreating permissions and markings in Unity Catalog before any sensitive data landsSecurity owners have signed off the mapping
ApplicationsDeciding if each Ontology object and Workshop app stays in Foundry, reads from Databricks or gets rebuiltEach app has a recorded decision and an owner
ValidationRunning both platforms in parallel and confirming outputs match before each switchOutputs match across a full business cycle

Plan waves by business domain, such as supply chain or maintenance, so the datasets, transforms and schedules that feed one set of apps move together and lineage stays intact.

Start with a domain that has heavy processing and few Ontology dependencies.

One decision board, with data, security and business owners, should rule on the Ontology question for each domain. Leaving it to individual teams creates an estate where some apps read Databricks, some read Foundry and nobody is sure which numbers are current.

Step-by-Step Palantir Foundry to Databricks Migration Process

The process maps the Foundry estate and its lineage, sets up governance first, then moves data, logic and schedules domain by domain, with both platforms running side by side until results match.

Step 1: Map the Foundry Estate and Its Lineage

Foundry’s Data Lineage application shows how datasets connect. Use it to list, for each domain:

  • Datasets, their sizes and how often they rebuild
  • The transforms and Pipeline Builder flows that produce them
  • Source syncs feeding the domain
  • Ontology objects, Workshop apps, Contour or Quiver analyses and models that consume the outputs
  • Business and engineering owners
  • Compute-seconds used, taken from Foundry’s resource management application, as a cost baseline

Flag datasets nobody has used in months. Retiring them shrinks the work before it starts.

Step 2: Decide What Stays in Foundry

Agree the target for each domain: read in place, hybrid or full exit. For hybrid domains, list exactly which outputs Foundry will read back from Databricks, so the data team builds to that contract.

Step 3: Set Up Unity Catalog and Security Before Data Moves

Foundry’s permissions and markings often protect sensitive data, so recreate them before the first dataset lands:

  • Catalog design: Catalogs and schemas that mirror domains and environments
  • Access rules: Unity Catalog grants, row filters and column masks mapped from Foundry permissions and markings
  • Identity: Groups synced from your identity provider and service principals for production workflows
  • Release process: Declarative Automation Bundles, formerly Databricks Asset Bundles, stored in Git

Have security teams sign off the mapping. Markings are mandatory controls in Foundry, and any gap here becomes a data exposure risk.

Step 4: Move Datasets Into Delta Tables

Most migrations use a mix of three routes:

  1. Bulk history through Palantir’s APIs. Read tables through the Read Table API as Arrow or CSV, then load them into Delta tables.
  2. Ongoing feeds through virtual table outputs. Foundry pipelines can write to Databricks tables during the transition. The output must be a managed Iceberg or external Delta table, and Spark pipelines need external access enabled.
  3. Read-only access to Iceberg tables. Databricks reads Foundry Iceberg tables in place through catalog federation.

Land data in layers, from raw to cleaned to business-ready, so each step can be tested on its own.

Step 5: Convert Transforms and Rebuild Pipeline Builder Flows

Python transforms can keep their core Spark logic. The Foundry-specific parts, such as transform decorators and input and output references, are replaced with standard reads and writes against Unity Catalog tables. SQL transforms may need small dialect changes.

AI-assisted conversion can speed up large code bases, but every converted transform needs review and testing against Foundry’s output.

Pipeline Builder can export batch pipelines as Java code. Streaming and faster pipelines can’t be exported, there is no built-in export to PySpark, and the exported output may not match the original exactly.

User-defined functions and LLM calls need manual work, so treat the export as a reference and rebuild the logic as Lakeflow pipelines or SQL.

Step 6: Recreate Schedules and Dependencies in Lakeflow Jobs

Use the lineage map from Step 1 to rebuild the order in which datasets refresh. Each Foundry schedule becomes a Lakeflow Jobs workflow with task dependencies, retries and failure notifications.

Step 7: Run Both Platforms in Parallel and Compare

Keep Foundry and Databricks running on the same inputs and compare:

  1. Row counts and schemas for every migrated table
  2. Totals and distributions for key business measures
  3. The outputs that Ontology objects and Workshop apps rely on

Run the comparison across a full business cycle before any switch.

Step 8: Repoint Consumers and Retire Foundry Pipelines

For hybrid domains, point Foundry’s Ontology and apps at the Databricks tables through the Databricks connector. For full-exit domains, switch users to the rebuilt apps. Then pause the Foundry pipelines for an agreed fallback period before retiring them.

Keeping Palantir Foundry and Databricks Connected During and After Migration

Connectors run in both directions, so Foundry can read Databricks tables and write to some of them, and Databricks can read Foundry’s Iceberg tables, which makes a gradual, low-risk migration possible.

DirectionHow It WorksBest Used For
Foundry reads Databricks tablesUnity Catalog tables appear in Foundry as virtual tables, without copying them into Foundry datasetsHybrid domains where Foundry apps use data built on Databricks
Foundry writes to Databricks tablesExternal Delta and managed Iceberg tables accept writes, while managed Delta tables and views are read-onlyMoving ongoing pipeline outputs into Unity Catalog
Foundry pushes processing to DatabricksPython transforms and Pipeline Builder pipelines that use Databricks virtual tables can run their compute on Databricks. Streaming builds and faster pipelines aren’t supportedCutting Foundry processing costs before a full rewrite
Foundry copies data from DatabricksBulk import copies Databricks tables into Foundry datasetsWorkflows that still need native Foundry datasets
Databricks reads Foundry Iceberg tablesCatalog federation reads them in place, read-onlyAnalytics on Foundry data without building copy pipelines
Foundry uses Databricks modelsA Unity Catalog model deployed on Databricks Model Serving can be called from Foundry through a model adapterAI built on Databricks serving Foundry workflows

These connections change the shape of a migration. Data can move domain by domain while Foundry apps keep working, and nothing has to switch at once.

They also make a lasting hybrid architecture practical for enterprises that want Databricks for data and AI and Foundry for operational applications.

One setup rule matters early. Virtual tables need a Foundry worker source, and connections that use an agent worker aren’t supported. Without external access in Unity Catalog, Foundry connects over JDBC instead of reading storage directly.

Migrating the Palantir Ontology, Workshop Apps and AIP Workloads

The Ontology and the apps built on it are the hardest part of any Foundry exit, since they encode business meaning and actions, so each one needs a deliberate keep, rebuild or retire decision.

The Ontology turns datasets into business objects, such as aircraft, orders or suppliers, with links between them and actions people can take. Workshop apps, AIP agents and logic sit on top of it.

Recreating this layer is a product decision as much as a technical one. Three options cover most cases:

OptionWhat It MeansDatabricks Building Blocks
Keep it in FoundryThe Ontology reads Databricks tables through the connector, and users see no changeUnity Catalog tables exposed to Foundry as virtual tables
Rebuild on DatabricksBusiness definitions, operational apps and AI workflows are redesigned on DatabricksUnity Catalog metric views for definitions, Databricks Apps for apps, Agent Bricks for AI workflows and Lakebase for data that apps write back
Retire itApps that mainly show reports are replaced by analytics toolsAI/BI dashboards and Genie Agents

Some of these features carry release labels. Metric view relationships are Beta, and Lakebase’s change history to Delta is in Public Preview.

Models are simpler. Models trained in Foundry can be retrained or re-registered with MLflow in Unity Catalog, then served from Databricks or connected back to Foundry through Model Serving.

Involve the business users of each app early. They know which actions and views matter, and that knowledge rarely sits in documentation.

Security and Governance When Moving From Foundry to Unity Catalog

Foundry’s permissions, markings and lineage need a full equivalent in Unity Catalog before sensitive data moves, so governance is designed first and verified at every switch.

Foundry combines role-based permissions with markings, which restrict access to sensitive data wherever it flows. On Databricks, Unity Catalog handles the same needs with different building blocks:

  • Access control: Grants on catalogs, schemas and tables
  • Fine-grained rules: Row filters and column masks, plus attribute-based access control policies built on governed tags
  • Lineage: Automatic lineage for queries run on Databricks, down to the column level
  • Auditing: Audit logs in the system.access.audit table, which Databricks labels Public Preview

The mapping rarely matches one to one. Markings that follow data through every derived dataset need to be designed as consistent tags and policies across catalogs, and tested with real user accounts before switch-over.

Palantir notes that discretionary row and column controls don’t extend to downstream outputs or exports, so check which of your rules are mandatory.

For hybrid estates, decide which platform is the source of truth for each access rule. Two rule sets that drift apart create the very exposure governance is meant to prevent.

Our guide on moving to Unity Catalog covers the governance setup in more depth.

Best Practices for Palantir Foundry to Databricks Migration

Strong Foundry migrations decide the Ontology question early, set up governance before data moves and keep both platforms connected until each domain proves itself.

PracticeWhy It Matters
Decide what stays in Foundry before rebuildingClear boundaries stop teams rebuilding apps that should have stayed
Use Foundry’s lineage to plan wavesMoving whole lineage chains together keeps dependencies on one platform
Recreate markings before sensitive data landsA gap in access rules becomes a data exposure risk
Start hybrid where apps carry valueMoving processing first cuts cost while users keep familiar apps
Check the release stage of each featureFederation shows no stage label, and some Unity Catalog policies are Beta
Review every AI-converted transformAutomated conversion speeds the work but can change logic quietly
Validate against Foundry outputsBusiness trust depends on numbers matching before each switch
Involve app users earlyThey know which actions and views matter most

Our Databricks migration guide covers strategy choices that apply to other source platforms too.

Common Palantir Foundry to Databricks Migration Challenges

Most challenges in Foundry migrations come from underestimating the Ontology, the security model or the no-code pipelines, all of which hold logic that isn’t visible in code.

  1. Treating the Ontology as data: Objects and actions carry business meaning that a table export doesn’t capture. Decide early to keep or redesign them.
  2. Leaving markings behind: Foundry markings don’t exist in Unity Catalog until you design equivalents. Recreate access rules before the data lands.
  3. Forgetting Pipeline Builder logic: Pipeline Builder logic sits outside Code Repositories, so it’s easy to miss. Include it in the inventory from the start.
  4. Trusting converted code without tests: Automated rewrites can change joins or filters in small ways. Compare every output with Foundry’s.
  5. Assuming read in place covers every dataset: Catalog federation reads Foundry Iceberg tables only, so standard datasets need another route.
  6. Switching everything at once: A single cutover removes the safety net. Move domain by domain, with both platforms connected.

How LatentView Helps With Palantir Foundry to Databricks Migration

LatentView is a Databricks Elite C&SI partner, and our Databricks Center of Excellence unites 400 data engineers, including 150+ certified and 200+ trained Databricks specialists. For a leading US technology company, we moved legacy ETL processes and data from AWS-hosted, Postgres-backed Python pipelines to Databricks in under six months, with Unity Catalog governance, as this case study shows. MigrateMate, our platform-agnostic migration solution, automates assessment, conversion and validation, and typically reduces migration costs by 30% to 40%. UCMate, our Unity Catalog migration solution, simplifies and accelerates the move to Unity Catalog.

Transform your Foundry data layer into a governed Databricks lakehouse with a partner that serves 50+ Fortune 500 clients. Map your first Foundry domain with our data engineering team: Talk to a Data Engineering Expert.

Frequently Asked Questions About Palantir Foundry to Databricks Migration

1. Can Palantir Foundry and Databricks Run Side by Side?

Yes. Foundry’s Databricks connector reads Unity Catalog tables as virtual tables, and Databricks can read Foundry’s Iceberg tables through catalog federation, so both platforms can share data.

2. How Is Data Exported From Palantir Foundry?

Palantir’s Read Table API returns datasets as Arrow or CSV tables, which then load into Delta tables in Databricks. Foundry pipelines can also write to Databricks tables through virtual table outputs.

3. Does the Palantir Ontology Move to Databricks?

Not directly. The Ontology can stay in Foundry reading Databricks tables, or be redesigned using Unity Catalog metric views and Databricks Apps.

4. Which Foundry Workloads Should Move First?

Heavy data processing with few app dependencies moves first. These workloads cut Foundry compute costs quickly with the least disruption to users.

5. Is Databricks a Full Replacement for Palantir Foundry?

For data engineering, analytics and AI, yes. Foundry’s Ontology-driven operational apps need redesign on Databricks or can stay in Foundry.

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

CATEGORY

LatentView Analytics has been helping enterprises make data-driven decisions for nearly 20 years. The company brings deep expertise in data engineering, business analytics, GenAI, and predictive modeling to 50+ Fortune 500 clients across tech, retail, financial services, and CPG. A publicly traded company serving the US, India, Canada, Europe, and Singapore, LatentView is recognized in Forrester's Customer Analytics Service Providers Landscape.

Take to the Next Step

"*" indicates required fields

consent*

Related Blogs

This guide helps data and AI leaders move from Azure Machine Learning to Databricks, from choosing…

This guide helps data leaders move pipelines from Azure Data Factory to Databricks one use case…

This guide helps data leaders plan a move from Google BigQuery to Databricks, pick the right…

Scroll to Top