This guide helps data leaders plan a move from Palantir Foundry to Databricks, decide which workloads to move and which to keep, and switch over without disrupting the teams that rely on Foundry today.
Palantir Foundry to Databricks migration moves datasets, transformation logic and pipelines into Databricks, while many enterprises keep Foundry’s Ontology and operational apps connected to it.
Key Takeaways
- Palantir Foundry to Databricks migration shifts data engineering, storage and compute from Foundry onto Databricks, governed by Unity Catalog.
- Enterprises choose between reading Foundry data in place, a hybrid split and a full exit, and each choice changes the scope.
- Datasets and code transforms move with tooling help, while the Ontology, Workshop apps and AIP workloads need the most thought.
- Palantir and Databricks announced a strategic product partnership on March 13, 2025, so both platforms can stay connected during and after the move.
- Databricks reads Foundry data in place only for Foundry Iceberg tables, and access is read-only.
- Running both platforms in parallel and comparing outputs before each switch reduces the risk of every cutover.
Palantir Foundry vs Databricks: What Changes When Workloads Move
Foundry bundles data integration, transformation, apps and security in one platform, while Databricks separates them, and those differences shape every migration decision.
Foundry combines data integration, transformation, an operational data model called the Ontology and app building in one platform.
On Databricks, Delta Lake and Apache Iceberg tables hold the data, Unity Catalog governs data and models, and compute is billed by usage.
Here is how the core Foundry building blocks translate:
| Foundry Concept | How It Works in Foundry | Databricks Counterpart |
|---|---|---|
| Dataset | A versioned collection of files, updated through SNAPSHOT, APPEND, UPDATE or DELETE transactions | A Delta table in Unity Catalog, with version history and time travel |
| Branch | Isolated copies of pipelines and data for safe changes | Git branches for code, plus separate development catalogs for data |
| Project | Foundry organizes data and resources in Projects, which carry role-based access | Catalogs and schemas with Unity Catalog grants |
| Marking | A mandatory control that travels with data into every derived dataset | Governed tags with attribute-based access control policies, plus row filters and column masks |
| Build and schedule | Rebuilds datasets in dependency order | Lakeflow Jobs workflows with task dependencies |
| Compute-seconds | Foundry’s unit for measuring compute work | Databricks Units (DBUs), a normalized unit of processing power |
Markings need early attention. In Foundry, mandatory controls such as markings travel with data through derivation. Unity Catalog uses governed tags and attribute-based policies instead.
Some of those Databricks controls are still in Beta, including policies on views and DENY policies.
The move doesn’t have to end the relationship with Palantir. On March 13, 2025, Palantir and Databricks announced a strategic product partnership, and the release names bp and several US government agencies among customers already using both.
For a wider comparison, see our Databricks vs Palantir guide.
Full Migration or Hybrid: Choosing How Far to Move Off Palantir Foundry
Enterprises can read Foundry data from Databricks, split the work between the two or exit Foundry completely, and the right choice depends on how much value sits in the Ontology.
| Approach | What Moves to Databricks | What Stays in Foundry | Fits When |
|---|---|---|---|
| Read in place | Analytics and AI that query Foundry Iceberg tables | All pipelines, the Ontology and apps | Teams want Databricks analytics on Foundry data without moving anything yet |
| Hybrid split | Data storage, transformation and pipeline compute | The Ontology, Workshop apps and operational workflows | Foundry apps deliver real value, but processing costs and engineering limits are rising |
| Full exit | Data, pipelines, models, semantic logic and apps | Nothing, once the switch-over ends | The organization wants one platform and can rebuild operational apps elsewhere |
Reading in place has narrow limits. Databricks catalog federation reads only Foundry Iceberg tables, access is read-only and views are not supported.
Foundry Iceberg tables are a separate resource type from standard Foundry datasets.
Setup needs Databricks Runtime 18 or above, pro or serverless SQL warehouses and an OAuth application in Foundry. The Databricks page shows no release-stage label, so confirm the feature’s status before planning around it.
In a hybrid split, Databricks takes over the heavy processing and Foundry reads the finished tables through its Databricks connector. Palantir lists virtual tables, bulk import and compute pushdown for Databricks as generally available.
A full exit is a bigger decision. On Databricks, the Ontology’s jobs are split across separate tools such as metric views, Databricks Apps and Agent Bricks, so the Ontology, its actions and the Workshop apps need redesign rather than conversion.
What You’re Actually Migrating From Palantir Foundry to Databricks
A Foundry migration covers datasets, code and no-code pipelines, schedules, permissions, models and the Ontology layer, with each asset landing in its own part of Databricks.
| Foundry Asset | What It Holds | Where It Lands on Databricks |
|---|---|---|
| Datasets | Versioned tables and files stored inside Foundry | Delta or Iceberg tables in Unity Catalog |
| Python and SQL transforms | Code in Foundry Code Repositories that builds datasets | Notebooks, SQL and Lakeflow pipelines |
| Pipeline Builder flows | Visual, no-code transformation logic | Lakeflow pipelines or SQL, rebuilt from the flow’s logic |
| Schedules and build dependencies | When datasets rebuild and in what order | Lakeflow Jobs workflows with task dependencies |
| Data Connection syncs | Ingestion from source systems | Lakeflow Connect, Auto Loader or partner tools |
| Permissions and markings | Who can see which data, including mandatory markings | Unity Catalog grants, row filters, column masks and tag-based policies |
| Models | Machine learning models built or hosted in Foundry | MLflow models registered in Unity Catalog |
| Ontology, Workshop, Contour, Quiver and AIP logic | Business objects, actions, operational apps, analysis tools and AI workflows | Kept in Foundry, or redesigned with Databricks apps, AI and BI tools |
The data layer has the most tooling support. Palantir’s Read Table API returns tables as Arrow or CSV, and Foundry datasets store tabular data in open formats such as Parquet.
If you push data to Databricks through Data Connection table exports, Palantir sets limits. The underlying files must be Parquet, array, map and struct columns fail to export, and only the master branch is exported.
The Ontology layer has the least tooling support, because it captures business meaning and actions designed for Foundry.
Two asset groups are easy to miss. Pipeline Builder flows live outside Code Repositories, and markings and permissions sit on Foundry resources, so they must be rebuilt in Unity Catalog.
How to Structure a Palantir Foundry to Databricks Migration
Organize the migration into a handful of workstreams and waves organized by business domain, with the Ontology decision made early so every wave knows what it hands back to Foundry.
Five workstreams usually run side by side:
| Workstream | Scope | Done When |
|---|---|---|
| Data | Moving datasets into Delta or Iceberg tables and keeping them in sync during the transition | Every in-scope dataset has a governed Unity Catalog table |
| Logic | Converting code transforms and rebuilding Pipeline Builder flows | Converted logic reproduces Foundry outputs on the same inputs |
| Governance | Recreating permissions and markings in Unity Catalog before any sensitive data lands | Security owners have signed off the mapping |
| Applications | Deciding if each Ontology object and Workshop app stays in Foundry, reads from Databricks or gets rebuilt | Each app has a recorded decision and an owner |
| Validation | Running both platforms in parallel and confirming outputs match before each switch | Outputs match across a full business cycle |
Plan waves by business domain, such as supply chain or maintenance, so the datasets, transforms and schedules that feed one set of apps move together and lineage stays intact.
Start with a domain that has heavy processing and few Ontology dependencies.
One decision board, with data, security and business owners, should rule on the Ontology question for each domain. Leaving it to individual teams creates an estate where some apps read Databricks, some read Foundry and nobody is sure which numbers are current.
Step-by-Step Palantir Foundry to Databricks Migration Process
The process maps the Foundry estate and its lineage, sets up governance first, then moves data, logic and schedules domain by domain, with both platforms running side by side until results match.
Step 1: Map the Foundry Estate and Its Lineage
Foundry’s Data Lineage application shows how datasets connect. Use it to list, for each domain:
- Datasets, their sizes and how often they rebuild
- The transforms and Pipeline Builder flows that produce them
- Source syncs feeding the domain
- Ontology objects, Workshop apps, Contour or Quiver analyses and models that consume the outputs
- Business and engineering owners
- Compute-seconds used, taken from Foundry’s resource management application, as a cost baseline
Flag datasets nobody has used in months. Retiring them shrinks the work before it starts.
Step 2: Decide What Stays in Foundry
Agree the target for each domain: read in place, hybrid or full exit. For hybrid domains, list exactly which outputs Foundry will read back from Databricks, so the data team builds to that contract.
Step 3: Set Up Unity Catalog and Security Before Data Moves
Foundry’s permissions and markings often protect sensitive data, so recreate them before the first dataset lands:
- Catalog design: Catalogs and schemas that mirror domains and environments
- Access rules: Unity Catalog grants, row filters and column masks mapped from Foundry permissions and markings
- Identity: Groups synced from your identity provider and service principals for production workflows
- Release process: Declarative Automation Bundles, formerly Databricks Asset Bundles, stored in Git
Have security teams sign off the mapping. Markings are mandatory controls in Foundry, and any gap here becomes a data exposure risk.
Step 4: Move Datasets Into Delta Tables
Most migrations use a mix of three routes:
- Bulk history through Palantir’s APIs. Read tables through the Read Table API as Arrow or CSV, then load them into Delta tables.
- Ongoing feeds through virtual table outputs. Foundry pipelines can write to Databricks tables during the transition. The output must be a managed Iceberg or external Delta table, and Spark pipelines need external access enabled.
- Read-only access to Iceberg tables. Databricks reads Foundry Iceberg tables in place through catalog federation.
Land data in layers, from raw to cleaned to business-ready, so each step can be tested on its own.
Step 5: Convert Transforms and Rebuild Pipeline Builder Flows
Python transforms can keep their core Spark logic. The Foundry-specific parts, such as transform decorators and input and output references, are replaced with standard reads and writes against Unity Catalog tables. SQL transforms may need small dialect changes.
AI-assisted conversion can speed up large code bases, but every converted transform needs review and testing against Foundry’s output.
Pipeline Builder can export batch pipelines as Java code. Streaming and faster pipelines can’t be exported, there is no built-in export to PySpark, and the exported output may not match the original exactly.
User-defined functions and LLM calls need manual work, so treat the export as a reference and rebuild the logic as Lakeflow pipelines or SQL.
Step 6: Recreate Schedules and Dependencies in Lakeflow Jobs
Use the lineage map from Step 1 to rebuild the order in which datasets refresh. Each Foundry schedule becomes a Lakeflow Jobs workflow with task dependencies, retries and failure notifications.
Step 7: Run Both Platforms in Parallel and Compare
Keep Foundry and Databricks running on the same inputs and compare:
- Row counts and schemas for every migrated table
- Totals and distributions for key business measures
- The outputs that Ontology objects and Workshop apps rely on
Run the comparison across a full business cycle before any switch.
Step 8: Repoint Consumers and Retire Foundry Pipelines
For hybrid domains, point Foundry’s Ontology and apps at the Databricks tables through the Databricks connector. For full-exit domains, switch users to the rebuilt apps. Then pause the Foundry pipelines for an agreed fallback period before retiring them.
Keeping Palantir Foundry and Databricks Connected During and After Migration
Connectors run in both directions, so Foundry can read Databricks tables and write to some of them, and Databricks can read Foundry’s Iceberg tables, which makes a gradual, low-risk migration possible.
| Direction | How It Works | Best Used For |
|---|---|---|
| Foundry reads Databricks tables | Unity Catalog tables appear in Foundry as virtual tables, without copying them into Foundry datasets | Hybrid domains where Foundry apps use data built on Databricks |
| Foundry writes to Databricks tables | External Delta and managed Iceberg tables accept writes, while managed Delta tables and views are read-only | Moving ongoing pipeline outputs into Unity Catalog |
| Foundry pushes processing to Databricks | Python transforms and Pipeline Builder pipelines that use Databricks virtual tables can run their compute on Databricks. Streaming builds and faster pipelines aren’t supported | Cutting Foundry processing costs before a full rewrite |
| Foundry copies data from Databricks | Bulk import copies Databricks tables into Foundry datasets | Workflows that still need native Foundry datasets |
| Databricks reads Foundry Iceberg tables | Catalog federation reads them in place, read-only | Analytics on Foundry data without building copy pipelines |
| Foundry uses Databricks models | A Unity Catalog model deployed on Databricks Model Serving can be called from Foundry through a model adapter | AI built on Databricks serving Foundry workflows |
These connections change the shape of a migration. Data can move domain by domain while Foundry apps keep working, and nothing has to switch at once.
They also make a lasting hybrid architecture practical for enterprises that want Databricks for data and AI and Foundry for operational applications.
One setup rule matters early. Virtual tables need a Foundry worker source, and connections that use an agent worker aren’t supported. Without external access in Unity Catalog, Foundry connects over JDBC instead of reading storage directly.
Migrating the Palantir Ontology, Workshop Apps and AIP Workloads
The Ontology and the apps built on it are the hardest part of any Foundry exit, since they encode business meaning and actions, so each one needs a deliberate keep, rebuild or retire decision.
The Ontology turns datasets into business objects, such as aircraft, orders or suppliers, with links between them and actions people can take. Workshop apps, AIP agents and logic sit on top of it.
Recreating this layer is a product decision as much as a technical one. Three options cover most cases:
| Option | What It Means | Databricks Building Blocks |
|---|---|---|
| Keep it in Foundry | The Ontology reads Databricks tables through the connector, and users see no change | Unity Catalog tables exposed to Foundry as virtual tables |
| Rebuild on Databricks | Business definitions, operational apps and AI workflows are redesigned on Databricks | Unity Catalog metric views for definitions, Databricks Apps for apps, Agent Bricks for AI workflows and Lakebase for data that apps write back |
| Retire it | Apps that mainly show reports are replaced by analytics tools | AI/BI dashboards and Genie Agents |
Some of these features carry release labels. Metric view relationships are Beta, and Lakebase’s change history to Delta is in Public Preview.
Models are simpler. Models trained in Foundry can be retrained or re-registered with MLflow in Unity Catalog, then served from Databricks or connected back to Foundry through Model Serving.
Involve the business users of each app early. They know which actions and views matter, and that knowledge rarely sits in documentation.
Security and Governance When Moving From Foundry to Unity Catalog
Foundry’s permissions, markings and lineage need a full equivalent in Unity Catalog before sensitive data moves, so governance is designed first and verified at every switch.
Foundry combines role-based permissions with markings, which restrict access to sensitive data wherever it flows. On Databricks, Unity Catalog handles the same needs with different building blocks:
- Access control: Grants on catalogs, schemas and tables
- Fine-grained rules: Row filters and column masks, plus attribute-based access control policies built on governed tags
- Lineage: Automatic lineage for queries run on Databricks, down to the column level
- Auditing: Audit logs in the system.access.audit table, which Databricks labels Public Preview
The mapping rarely matches one to one. Markings that follow data through every derived dataset need to be designed as consistent tags and policies across catalogs, and tested with real user accounts before switch-over.
Palantir notes that discretionary row and column controls don’t extend to downstream outputs or exports, so check which of your rules are mandatory.
For hybrid estates, decide which platform is the source of truth for each access rule. Two rule sets that drift apart create the very exposure governance is meant to prevent.
Our guide on moving to Unity Catalog covers the governance setup in more depth.
Best Practices for Palantir Foundry to Databricks Migration
Strong Foundry migrations decide the Ontology question early, set up governance before data moves and keep both platforms connected until each domain proves itself.
| Practice | Why It Matters |
|---|---|
| Decide what stays in Foundry before rebuilding | Clear boundaries stop teams rebuilding apps that should have stayed |
| Use Foundry’s lineage to plan waves | Moving whole lineage chains together keeps dependencies on one platform |
| Recreate markings before sensitive data lands | A gap in access rules becomes a data exposure risk |
| Start hybrid where apps carry value | Moving processing first cuts cost while users keep familiar apps |
| Check the release stage of each feature | Federation shows no stage label, and some Unity Catalog policies are Beta |
| Review every AI-converted transform | Automated conversion speeds the work but can change logic quietly |
| Validate against Foundry outputs | Business trust depends on numbers matching before each switch |
| Involve app users early | They know which actions and views matter most |
Our Databricks migration guide covers strategy choices that apply to other source platforms too.
Common Palantir Foundry to Databricks Migration Challenges
Most challenges in Foundry migrations come from underestimating the Ontology, the security model or the no-code pipelines, all of which hold logic that isn’t visible in code.
- Treating the Ontology as data: Objects and actions carry business meaning that a table export doesn’t capture. Decide early to keep or redesign them.
- Leaving markings behind: Foundry markings don’t exist in Unity Catalog until you design equivalents. Recreate access rules before the data lands.
- Forgetting Pipeline Builder logic: Pipeline Builder logic sits outside Code Repositories, so it’s easy to miss. Include it in the inventory from the start.
- Trusting converted code without tests: Automated rewrites can change joins or filters in small ways. Compare every output with Foundry’s.
- Assuming read in place covers every dataset: Catalog federation reads Foundry Iceberg tables only, so standard datasets need another route.
- Switching everything at once: A single cutover removes the safety net. Move domain by domain, with both platforms connected.
How LatentView Helps With Palantir Foundry to Databricks Migration
LatentView is a Databricks Elite C&SI partner, and our Databricks Center of Excellence unites 400 data engineers, including 150+ certified and 200+ trained Databricks specialists. For a leading US technology company, we moved legacy ETL processes and data from AWS-hosted, Postgres-backed Python pipelines to Databricks in under six months, with Unity Catalog governance, as this case study shows. MigrateMate, our platform-agnostic migration solution, automates assessment, conversion and validation, and typically reduces migration costs by 30% to 40%. UCMate, our Unity Catalog migration solution, simplifies and accelerates the move to Unity Catalog.
Transform your Foundry data layer into a governed Databricks lakehouse with a partner that serves 50+ Fortune 500 clients. Map your first Foundry domain with our data engineering team: Talk to a Data Engineering Expert.
Frequently Asked Questions About Palantir Foundry to Databricks Migration
1. Can Palantir Foundry and Databricks Run Side by Side?
Yes. Foundry’s Databricks connector reads Unity Catalog tables as virtual tables, and Databricks can read Foundry’s Iceberg tables through catalog federation, so both platforms can share data.
2. How Is Data Exported From Palantir Foundry?
Palantir’s Read Table API returns datasets as Arrow or CSV tables, which then load into Delta tables in Databricks. Foundry pipelines can also write to Databricks tables through virtual table outputs.
3. Does the Palantir Ontology Move to Databricks?
Not directly. The Ontology can stay in Foundry reading Databricks tables, or be redesigned using Unity Catalog metric views and Databricks Apps.
4. Which Foundry Workloads Should Move First?
Heavy data processing with few app dependencies moves first. These workloads cut Foundry compute costs quickly with the least disruption to users.
5. Is Databricks a Full Replacement for Palantir Foundry?
For data engineering, analytics and AI, yes. Foundry’s Ontology-driven operational apps need redesign on Databricks or can stay in Foundry.