In many investment platforms, the data that advanced personalisation depends on (holdings, transactions, tax-wrapper allowances, transfer status) is held in an investment management platform (IMP) operated as a service by a third-party provider. Marketing and customer engagement systems do not read that data directly. They receive it through a chain of copies: a change request to the IMP provider, change data capture (CDC) triggers configured at the provider’s end, a landing in an operational data store (ODS), an event published to a stream, and further copies inside each engagement platform.
That chain was designed for operational integration, not for personalisation. Each new attribute requires a provider change, each copy adds latency and reconciliation effort, and history from before a CDC trigger went live is typically absent. As customer communication moves from segment-level messaging towards individual, context-aware engagement, these constraints, rather than the capability of the engagement tools, tend to become the binding limit.
Zero copy data sharing changes the integration contract. Instead of the provider pushing changes into a firm-specific pipeline, the provider publishes governed tables that the firm reads in place. Databricks’ documentation on Delta Sharing describes the model plainly: the data stays in the provider’s storage and the recipient reads it live. This paper sets out the current state, four personalisation use cases it constrains, and two implementation patterns on Databricks: one where the IMP can publish a share, and one where it cannot. In both patterns the event stream remains, but only for triggers that are genuinely real time.
Current state: the IMP-to-engagement copy chain
In a typical configuration, customer and investment data reaches engagement systems in five steps:
- Change request. The firm specifies the tables and fields it needs and submits a change request to the IMP provider.
- CDC trigger. The provider configures change data capture on its database for the agreed scope. The trigger emits a record whenever a row in scope is inserted, updated or deleted.
- ODS landing. The changes land in the firm’s operational data store, where they are merged into current-state tables.
- Event publication. The ODS, or an integration layer beside it, publishes business events (for example, “contribution received”) to an event stream.
- Downstream copies. The customer data platform and each engagement tool consume the events, or batch extracts from the ODS, and maintain their own copies.
The result is that a single customer attribute frequently exists in four or more places: the IMP, the ODS, the event stream’s retained log, the customer data platform and one or more engagement tools. Each copy is owned by a different team, refreshed on a different schedule and governed under a different access model.
The change request is the less visible constraint. A CDC trigger is defined at table and column level inside the provider’s database, so the feed contains only what was specified when the trigger was built. Extending it requires analysis, build, testing and release by the provider. Because the provider operates a regulated, multi-client platform, such changes are typically batched into scheduled release cycles, and lead times are commonly measured in weeks to months.
This design is appropriate for its original purpose. Operational integration, where a defined set of transactions must be delivered reliably and in order, is well served by CDC and an event stream. The difficulty arises when the same feed is asked to support personalisation, which needs a broader, faster-changing and more historical view of the customer than any fixed feed specification anticipates.
Constraints on advanced personalisation
Four use cases illustrate where the copy chain, rather than the engagement tooling, limits what can be delivered.
Tax-year allowance prompts
A prompt to customers who have used less than half of their annual tax-advantaged allowance requires the year-to-date contribution position for each wrapper. If the CDC feed was specified for transactions but not for allowance positions, the firm has two options: request the attribute from the provider, or derive it downstream by re-aggregating transactions. The second option duplicates IMP business logic (carry-forward rules, transfers in, withdrawals and corrections) and risks diverging from the provider’s own figure. A prompt built on a divergent figure can encourage a customer to contribute beyond the allowance, which is a conduct risk rather than a marketing error.
Transfer-in progress communication
A transfer from another provider passes through several states held in the IMP: request received, awaiting the ceding provider, cash or assets received, re-registration complete, and invested. Personalised progress communication requires each state and its timestamp. Where the feed carries only a subset (for example, initiated and completed), intermediate communications cannot be triggered, and adding states is a further change request. Where a stale state is used, the firm can send an “awaiting transfer” message after the assets have already arrived.
Exposure-aware market communication
During a market drawdown, a firm may wish to vary its communication by each customer’s actual exposure: equity allocation, concentration in a single fund, or proximity to drawing an income in retirement. This requires holdings-level data valued at current prices. Holdings values change with every price update and corporate action, so publishing them as events is impractical, and the ODS therefore typically holds a daily snapshot. Personalisation then operates on the previous day’s exposure at the point it matters most.
Propensity and next-best-action models
Models that estimate, for example, the likelihood of a customer consolidating pensions or stopping regular contributions typically require several years of transaction history. CDC delivers changes from the date the trigger went live; earlier history requires a separate backfill extract, which is itself a project. Model features computed downstream are refreshed at ODS cadence, and the resulting scores have no governed route back to servicing channels that read from the IMP.
The common cause across the four cases is that the firm consumes a feed defined by a past change request, not the underlying data. A secondary effect is governance: a customer’s right-to-erasure request, or a change in marketing consent, must be applied to every copy in the chain, and each copy is a point at which it can be missed.
Pattern 1: the IMP publishes a Delta Share
Where the IMP provider runs its data platform on Databricks, or on Delta tables it can expose through the open Delta Sharing protocol, the provider can publish the firm’s data as a share. The mechanism has five elements.
- The share. A share is a securable object in Unity Catalog that lists exactly which data assets are exposed. A multi-client provider would typically include only the partitions, or filtered views, containing the firm’s own customers: for example accounts, wrappers and allowance positions, holdings, transactions and transfer status.
- The recipient. The provider registers the firm as a recipient. Two modes exist. In Databricks-to-Databricks sharing, the recipient is identified by its Unity Catalog metastore. In open sharing, the provider issues a credential, and the firm can read the share from any client that implements the protocol, so it is not required to run Databricks itself.
- The read. The firm mounts the share as a read-only catalog. Queries run on the firm’s compute against the provider’s storage; no file is produced and no pipeline is built.
- History and change data feed. If the provider shares a table with its history, the firm can read it as a Spark Structured Streaming source. With change data feed enabled on the source table, the firm receives row-level inserts, updates and deletes. This reproduces what the provider-built CDC trigger delivers today, but as a property of the shared table rather than a bespoke integration, typically at latencies of minutes rather than seconds depending on the streaming trigger interval.
- Scope changes and revocation. Adding a table or partition to the share is an administrative change by the provider, not a CDC build. Revoking a recipient or removing an asset takes effect immediately, with no residual copy on the firm’s side to reconcile.
Applied to the four use cases, the allowance prompt reads the provider’s own computed allowance position instead of re-deriving it; transfer communication reads the full status table; exposure-aware messaging queries holdings at the time of send; and propensity models read history through time travel, within the provider’s retention window. Downstream engagement platforms that support zero-copy access to a lakehouse query the firm’s catalog directly, and the ODS bulk copy is no longer required for these use cases.
The pattern has limits. It depends entirely on the provider’s willingness and ability to publish shares, and on commercial terms that cover them. History is bounded by the provider’s table retention policy, so multi-year model training may still require a one-off historical extract. Where provider and firm sit in different clouds or regions, data egress costs arise and are usually priced into the contract. Finally, the share becomes a data product with a schema: a column renamed by the provider breaks the firm’s queries as surely as it would break a CDC feed, so the data sharing agreement needs explicit schema versioning and change notice.
Pattern 2: the IMP cannot publish a share
Where the provider does not operate Delta tables and cannot expose a share, zero copy cannot begin at the IMP. Three options remain, and they differ in where the first copy sits.
Federated query against the IMP database. Lakehouse Federation lets Databricks push queries down to an external relational database over JDBC, governed through Unity Catalog. Databricks itself recommends federation for ad hoc reporting and proof-of-concept work, and managed ingestion for volume. Against a provider-operated, multi-client production database, direct query access is also unlikely to be granted on performance, security and operational resilience grounds. Federation is therefore rarely viable as the primary pattern.
Provider export in an open table format. The provider writes the agreed tables as Delta or Parquet files to object storage on a schedule. The firm registers them as external tables in Unity Catalog or ingests them incrementally. This requires no proprietary platform at the provider and leaves one provider-maintained copy, but it is still a scheduled export with the corresponding latency.
Land once in the lakehouse, then share downstream. The firm retains the provider’s CDC feed but redirects it into the lakehouse rather than into an ODS and multiple platform copies. Changes land in raw Delta tables, are merged into governed current-state customer and investment tables in Unity Catalog, and those tables are published with change data feed enabled. Every downstream consumer (analytics, models, the customer data platform and engagement tools) then reads from that single governed copy, either in place or through Delta Sharing.
The third option is the most common in practice. It does not remove the change request: a new attribute still has to be requested from the provider. It does mean that each attribute arrives once and is immediately available to every consumer, rather than being re-plumbed into each platform in turn. The copy count falls from four or more to two (the IMP and the governed lakehouse tables), and erasure and consent changes are applied in one place on the firm’s side.
The distinction between the two patterns is therefore narrow and specific. Pattern 1 removes the first copy as well as those after it. Pattern 2 removes only the copies after the first. Downstream, the two converge on the same architecture.
Role of the event stream after zero copy
Zero copy data sharing is a read model: consumers query or stream from shared tables when they need data. It does not push an action into an engagement platform at the moment an event occurs in the IMP. Triggers that must fire within seconds, such as a trade confirmation, a failed direct debit or a security alert, continue to require the event route.
In practice, use cases sort into three latency tiers:
| Latency requirement | Example | Recommended route |
|---|---|---|
| Seconds | Failed payment alert, trade confirmation | Event stream (existing CDC route) |
| Minutes | Transfer stage reached, contribution received | Change data feed streamed from the share or governed lakehouse table |
| Hours or at send time | Allowance prompts, exposure-aware campaigns, model scoring | Query in place at segment build or send time |
Where an engagement platform requires a resident, ingested profile for edge or in-session decisioning, an ingested copy of the attributes that decisioning uses remains the appropriate design. Zero copy reduces the scope of that copy to what genuinely needs sub-second access; it does not eliminate it. In most investment platforms, the seconds tier is a small fraction of the total use-case set, which is why redesigning the remainder around shared tables reduces both the change-request burden and the number of copies substantially.
Conclusion
The copy chain from the IMP was built to move transactions reliably, and it continues to do that well. Its limitation for personalisation is structural: a fixed feed, specified in advance and re-copied into every platform, cannot keep pace with use cases that need new attributes, current positions and long history. Zero copy data sharing on Databricks addresses that limitation by making governed tables, not feed specifications, the integration contract. Where the IMP can publish a Delta Share, that contract sits with the provider; where it cannot, it sits in a single governed lakehouse copy on the firm’s side. In either case the event stream narrows to the triggers that genuinely need it.
As a first step, the firm can put three questions to its IMP provider: whether it can publish a Delta Share of the firm’s data; whether tables can be shared with history and change data feed enabled; and what retention period applies. In parallel, it can list its twenty most-used personalisation attributes, record every system that holds a copy of each, and assign each use case to one of the three latency tiers above. The answers determine whether Pattern 1 or Pattern 2 applies and how much of the current chain can be retired.
Next step
The patterns in this paper apply one capability, data sharing and access, within a wider customer data infrastructure. The customer data infrastructure guide sets out the ten core capabilities that layer must provide and how they inform architecture selection, and is the natural reference for placing zero copy alongside ingestion, storage and identity decisions.
Frequently Asked Questions
Zero Copy Data Sharing — FAQ
What is zero copy data sharing?
Zero copy data sharing is a method of giving another party access to data without creating a physical copy of it in the recipient's system. The provider publishes governed tables, and the recipient queries or streams them from the provider's storage. Access can be scoped, audited and revoked without any residual copy to delete.
Does Delta Sharing require both parties to use Databricks?
No. Databricks-to-Databricks sharing requires both parties to use Unity Catalog, but open sharing issues the recipient a credential that works with any client implementing the open Delta Sharing protocol, including Spark, pandas and several business intelligence tools. The provider must, however, hold the shared data in Delta or another supported table format.
Is zero copy data sharing the same as data virtualisation?
Not exactly. Data virtualisation, or query federation, sends queries to a source database at run time and uses that database's compute. Zero copy data sharing typically exposes tables in object storage that the recipient reads with its own compute, which avoids loading the provider's operational database.
Who pays for compute and storage in zero copy data sharing?
In most implementations the provider pays for storing the shared data and the recipient pays for the compute used to query it. Where the two parties are in different clouds or regions, data egress charges also apply, and these are usually allocated in the commercial agreement.
How is a right-to-erasure request handled when data is shared rather than copied?
When the provider deletes or updates a record in the shared table, the change is visible to the recipient on its next read, because there is no separate copy. Recipients that stream changes into their own tables must process delete records from the change data feed, so erasure still needs to be designed into any downstream copies.
Is zero copy data sharing suitable for regulated financial data?
It can be, provided access controls, row and column filtering, audit logging and revocation are specified in the data sharing agreement and verified in operation. Because no copy is transferred, the number of places where personal data is held falls, which typically simplifies data protection and operational resilience assessments.
