Regulated Financial Data Platform¶

Scope¶

Covers the design of a data lake or lakehouse operated by a regulated financial institution — the constraints that make a bank's analytics platform different from a generic one, and the design decisions those constraints force.

Topics: encryption and key custody (provider-managed, CMK, BYOK, HYOK/external key store) and what each actually buys under a control review; data residency and the boundaries that leak; retention schedules and legal hold, and their genuine conflict with lake-native compaction, file rewrite, and snapshot expiry; lineage adequate to answer "where did this number come from"; PII and GLBA-scoped data segmentation and why engine-enforced masking is not a boundary over open table formats; tokenisation to keep cardholder data out of PCI scope; access review and least privilege including the service-principal problem; DR and the period-shaped RTO; third-party and concentration risk; evidence generation as a first-class design requirement; platform choice under thin staffing, which is the most consequential and least written-down decision on this page; and designing ahead of growth so the platform is examination-ready before it needs to be.

This file is about the platform. Getting the data out of the core banking system in the first place — the constraint that governs everything here — is patterns/core-banking-data-integration.md. Generic pipeline mechanics are patterns/data-pipeline.md, and the preservation architecture this file depends on is general/legal-hold.md.

On regulatory specifics. This file is deliberately architectural. It does not state supervisory thresholds, filing requirements, or examination procedures — those live in compliance/ffiec.md and compliance/bank-regulatory-reporting.md, and obligations vary by charter, size, and regulator. Where this file describes "what gets asked," it is describing the shape of the design conversation, not asserting a rule. Confirm every specific obligation with the institution's compliance function.

Overview¶

A regulated financial institution's data platform is judged on two axes that generic platforms are not. The first is defensibility: for any number the platform produces, the institution must be able to show where it came from, that it ties to the ledger, who could have changed it, and that the controls around it operated as described. The second is disposability: the institution must be able to show that it keeps what it is obliged to keep, for as long as it is obliged to keep it, and — harder — that it does not keep what it should have destroyed, except where a hold says otherwise.

Both axes push in the same direction and it is not the direction lake architecture naturally goes. Lakehouse table formats are built for throughput and cost: rewrite small files into big ones, expire old snapshots, reclaim the space, keep the query fast. Every one of those operations moves rows between files and destroys prior physical state. That is exactly what a retention schedule, a legal hold, and a reproducibility obligation are designed to prevent. The tension is real, it is under-discussed, and resolving it after the platform is live is far more expensive than designing for it — a theme that recurs throughout this file.

There is a third axis that decides more outcomes than either: who operates it. An institution in the low billions of assets typically has a small infrastructure team, no dedicated platform-engineering bench, and no on-call rotation that could carry a distributed query engine. Self-managed open-source lake stacks in that setting fail on operations rather than architecture, and they fail slowly enough that the failure is attributed to something else. This is treated as a first-class design input below rather than as a footnote, because it is.

Checklist¶

Platform choice under thin staffing — decide this honestly first¶

[Critical] Has the institution's actual platform-operations capacity been established before the platform is selected — how many engineers can maintain it, whether there is an on-call rotation that covers it, who patches it, and what happens when the one person who understands it is on leave or leaves the organisation? Choose the platform the institution can operate in its worst month, not its best. This single question predicts platform outcomes in small and mid-size institutions better than any architectural comparison, and it is usually answered aspirationally or not at all.
[Critical] If a self-managed open-source stack (Spark or Trino on Kubernetes, a self-hosted catalog, self-operated Iceberg or Delta maintenance) is under consideration, can the institution name the specific people who will run compaction and snapshot expiry on a schedule, keep the catalog highly available, patch the engine and its dependencies, upgrade across breaking releases, and respond at 2 a.m. when the metastore is down at period end? If those names do not exist, the honest answer is that this option is not available regardless of its architectural merits. The licence saving is real and it is not the binding constraint.
[Critical] Are managed platforms and the core vendor's own analytics offering treated as legitimate first-class options rather than as a fallback for teams that cannot build? For an institution without a platform bench, a managed lakehouse or warehouse is frequently the correct architecture, not a compromise — the operational burden it removes (upgrades, patching, catalog availability, table maintenance, access-control plumbing, audit logging) is precisely the burden the institution cannot carry, and the governance features arrive built rather than assembled.
[Critical] Is the degradation curve of an under-staffed self-managed stack understood and priced in? The characteristic failure is not a go-live failure: the stack works at launch and decays over twelve to twenty-four months as small files accumulate and nobody owns compaction, as the engine falls behind on security patches, and as institutional knowledge concentrates in one person who then leaves. The end state is an unpatched, unowned, slow dependency holding customer financial data — a materially worse position to explain than a boring managed service, and a much harder one to exit from than to have avoided.
[Recommended] Has the core vendor's analytics or reporting product been evaluated on its actual trade-offs rather than dismissed? It is genuinely the fastest path to value: extraction is solved by the vendor, the data model already matches the core, and no integration has to be built. The costs are equally concrete — it deepens concentration with a vendor that already holds the system of record, it typically cannot blend non-core sources (digital banking, card, origination, payments) which is where most of the interesting questions live, and the analytics asset generally does not survive a core conversion, which negates the strongest strategic reason to build a platform at all. See patterns/core-banking-data-integration.md.
[Recommended] If a managed-service partner will operate the platform, is it clear that the institution still owns the data model, the reconciliation logic, and the evidence — and that the partner introduces its own third-party risk obligation, its own key-person risk, and its own exit problem? Outsourcing operations is legitimate and common; outsourcing understanding is how an institution ends up unable to explain its own numbers.
[Recommended] Is the managed-platform cost model understood before commitment — consumption-based pricing that scales with usage in ways small institutions consistently underestimate, the separation of platform charges from underlying cloud infrastructure charges, and the effect of a few analysts writing unbounded queries against a large table? The licence premium over self-managed is the visible cost; the variable component is the one that produces the surprise. See general/cost.md and general/finops.md.
[Optional] Where a managed platform is chosen, is format-level portability preserved by keeping table data in an open format (Iceberg, Delta, Parquet) in storage the institution controls, so that the platform's compute and catalog are replaceable even though switching is never free? This is the one technical mitigation that meaningfully reduces platform lock-in, and it is much cheaper to establish at the start than to retrofit.

Encryption and key custody¶

[Critical] Is the key custody model an explicit, documented decision — provider-managed keys, customer-managed keys (CMK) in the provider's key service, bring-your-own-key (BYOK) with imported material, or hold-your-own-key (HYOK) via an external key store — rather than a default inherited from whatever the platform provisioned? Each answers a genuinely different question, and the institution should be able to state which question it is answering and why.
[Critical] Is it clear what each model actually buys, in these terms? Provider-managed buys the statement "data at rest is encrypted" and nothing further — the institution has no custody, no revocation, and no key-level audit. CMK buys control over key policy, rotation, and disablement, per-key access logging, and a credible crypto-revocation path — but the material lives in the provider's key service and the provider performs the cryptographic operations. BYOK additionally buys the ability to attest to how the material was generated and to let it expire so the provider can no longer decrypt — but during the key's life the operational position is CMK's. HYOK / external key store is the only model where the material never leaves the institution's HSM and the provider must call out for every operation, which is the only one that changes the answer to "can the provider decrypt without us."
[Critical] Are the costs of HYOK accepted before it is chosen — a hard availability dependency (an HSM or network outage makes data unavailable, not merely unreadable), latency on the cryptographic path, materially narrower service coverage (many managed services do not support external key stores), and a real key-management operational capability with runbooks, rotation ceremonies, and staff? A stronger custody model that the institution cannot operate reliably is not a stronger control. CMK with a tested runbook, evidenced rotation, and clean separation of duties is a better answer than HYOK without one — and that is the trade-off most often got backwards.
[Critical] Is the authority to destroy or disable a key separated from the authority to access the data it protects, and is that separation enforced in the key policy rather than by convention? Key deletion is the fastest path to irreversible data loss on a cloud platform, and the control question is not whether encryption exists but who could render the institution's records unreadable and whether anyone would notice.
[Critical] Is there a tested recovery position for key compromise, accidental disablement, and scheduled deletion — including what becomes unrecoverable, over what window a scheduled deletion can be cancelled, and how backups encrypted under a prior key are handled? Key management failures are recoverable only if the recovery was designed; there is no vendor escalation that undoes a destroyed key.
[Recommended] Is the envelope-encryption hierarchy designed with the deletion unit in mind — per-object or per-partition data keys wrapped by a managed key — so that crypto-shredding (destroying a data key to render a scoped set of records unreadable) is available as an alternative to physically rewriting files? This is directly relevant to the retention and erasure section below, and it only works if the data-key granularity aligns with the erasure scope, which must be decided up front. Confirm separately whether crypto-shredding is accepted as destruction in the institution's own control framework — acceptance varies.
[Recommended] Does the custody decision extend to the platform's metadata and derived surfaces, not just the data files — the catalog, lineage store, query history and query text, result caches, notebook outputs, BI extracts, and logs? Encrypting the Parquet files under a customer-managed key while the query-history store holding literal customer values sits under provider-managed keys is a common and easily-found inconsistency.
[Recommended] Is key rotation scheduled, automated where the service supports it, and evidenced — with the evidence produced by the platform rather than reconstructed by a person before a review? See the evidence section below.
[Optional] Where the institution's requirements reference validated cryptographic modules, has the specific service's validation status been confirmed rather than assumed from the provider's overall posture? Validation is per-module and per-mode, and coverage differs across services within the same provider.

Data residency and boundary¶

[Critical] Is the residency requirement stated precisely — which data classes, which jurisdictions, and whether the constraint is on storage location, processing location, or personnel access — before regions are chosen? "Keep the data in-country" and "no foreign national may access the data" are different requirements with different, and differently expensive, solutions.
[Critical] Are the paths that leak out of the chosen region enumerated rather than assumed absent — managed-service control planes, provider support access and telemetry, default backup and DR replication targets, CDN and edge caching, model and inference endpoints, and every SaaS tool in the analytics chain (BI, catalog, observability, alerting, ticketing)? Region-pinning the storage account is the easy 20% of residency; the leaks are the rest, and they are found by enumeration, not by inspection of the primary data path.
[Critical] Is metadata residency addressed separately from data residency? Catalog entries, column names, profiling statistics (minimum and maximum values on a name or address column are customer data), sample rows, and above all query history — which routinely contains customer values as literals in predicates — may be stored in a different region or under a different service than the data itself.
[Recommended] Is provider support access understood and controlled — whether provider personnel can access the environment, under what approval, whether sessions are logged and reviewable, and whether the institution can require approval per incident? This is generally a service-level capability that must be enabled, not a default.
[Optional] Where a cross-border or cross-region DR target is required for resilience but conflicts with residency, is the conflict resolved explicitly and documented, rather than left as an unreconciled contradiction between the resilience design and the privacy design?

Retention, legal hold, and lake-native file rewrite¶

This is the section most likely to be missing from a lakehouse design in a regulated institution, and the one whose omission is most expensive to correct later. Read general/legal-hold.md alongside it — the deletion-gate architecture described there is what the items below plug into.

Lineage and reproducibility¶

[Critical] Can the platform answer "where did this number come from" end to end — from a reported figure, back through the transformations and their code versions, to the specific source extract file and the control totals it was reconciled against? Table-to-table lineage alone does not answer it. The complete answer needs source provenance at the bottom and attribution for a specific figure at the top, and most tooling supplies only the middle layer.
[Critical] Is source provenance carried as data on the rows, not only in an external lineage catalog — the source file identifier, the extract cycle or business date, and the transform version stamped on every fact row at load? A lineage graph describes the pipeline as it is configured now; provenance columns describe how this particular row actually got here, and only the latter survives a pipeline refactor.
[Critical] Has reproducibility been tested at distance — can a figure reported several quarters ago be re-derived today, from retained raw extracts and a pinned snapshot and the code version of the time, and does it match? Lineage that has never been exercised against a real historical question is decorative. This test also validates the snapshot-pinning and landing-zone retention decisions above, which is why it is worth running early.
[Recommended] Are the query, the snapshot identifier, and the code version pinned alongside every published output, so that a reported number carries the means of its own reproduction rather than depending on someone remembering how it was produced?
[Recommended] Is lineage captured automatically by the platform or an open standard rather than maintained by hand in documentation? Hand-maintained lineage is accurate on the day it is written and misleading thereafter, and the divergence is invisible until someone relies on it.
[Optional] Does lineage extend downstream past the platform boundary into BI reports, spreadsheets, and regulatory submissions, so that an impact assessment for a source change can identify what breaks? Coverage here is usually partial; the useful question is whether the gap is known rather than whether it is closed.

PII, GLBA-scoped data, and segmentation¶

[Critical] Is customer identity separated from transactional fact in the physical model — identifying attributes in a restricted zone, facts keyed by a surrogate — so that the default analytical surface is de-identified and access to re-identification is a deliberate, separately-granted step? Retrofitting this separation after analysts have built on a wide joined table is one of the more painful remediations in this domain.
[Critical] Is it understood that over open table formats, engine-enforced column masks and row filters are not a security boundary if any path exists to read the files directly? Catalog-enforced policies hold only when every route to the bytes goes through the catalog. If a second engine, a notebook with storage credentials, a replication job, or a backup process can reach the underlying objects, then storage-level access is the actual boundary and the masks are a usability feature. Decide which one you are relying on and close the other paths, or physically separate the sensitive columns.
[Critical] Is data classified at ingestion rather than discovered later, with the classification driving placement, masking, retention, and access automatically? Classification applied as an after-the-fact scanning exercise finds most of the data most of the time, which is a different guarantee from knowing. See general/data-classification.md and compliance/glba.md.
[Recommended] Are the secondary surfaces where sensitive values escape covered — query history and saved queries, error messages and stack traces, catalog profiling statistics, notebook outputs and checkpoints, BI extracts and scheduled report attachments, sample datasets in lower environments, and support bundles? These are where sensitive data is most often found outside its intended zone, and none of them are addressed by table-level access control.
[Recommended] Are non-production environments either fully synthetic, masked at the point of copy, or governed identically to production? Copying production data into a development environment is the most common way a well-designed segmentation model is defeated, and it is usually done for a defensible reason by someone who did not know the rule.
[Optional] Is re-identification risk from combinations considered rather than only from direct identifiers — the classic case being a de-identified dataset where date, amount, branch, and product jointly identify an individual? De-identification is a property of the dataset in context, not of a column list.

Tokenisation and PCI scope¶

[Critical] Is the primary account number tokenised before it enters the analytics platform, so that the lake holds only the token plus whatever truncated elements (BIN, last four) are genuinely needed — keeping the platform out of the cardholder data environment entirely? The cheapest and most durable PCI control for an analytics platform is not being in scope, and that outcome is decided by where the tokenisation boundary sits, which is an early architectural choice rather than a later control. See compliance/pci-dss.md.
[Critical] Is the detokenisation service scoped narrowly and logged comprehensively — a separate service, a small explicit set of principals, per-call logging, rate limits, and alerting on volume anomalies? Tokenisation moves the risk rather than removing it; the detokenisation path is where the residual risk concentrates and it should be small enough to review by hand.
[Recommended] Is the tokenisation approach chosen deliberately — vault-based (a mapping store, strongest separation, an availability dependency and a high-value target) versus format-preserving encryption (no vault, keys become the sensitive material, format compatibility with legacy consumers)? The choice affects availability coupling and key custody, both of which appear elsewhere on this page.
[Recommended] Is token determinism an explicit decision? Deterministic tokens are necessary to join card activity across systems and are, for the same reason, a re-identification vector across the whole estate. If tokens are deterministic, treat them as pseudonymous identifiers subject to the segmentation rules above rather than as inert values.
[Optional] Is the same tokenisation discipline applied to other high-sensitivity identifiers that are not card data — government identification numbers, full account numbers — where the analytical need is joining rather than reading the value?

Access review and least privilege¶

[Critical] Are all grants made to groups synchronised from the identity provider, never to individual users directly? Direct user grants are the mechanism by which entitlements become unreviewable, and the platform's access model should make the correct pattern the easy one. See general/identity.md.
[Critical] Are service principals, machine identities, and integration credentials in scope of access review with a named human owner each? They are the hard part of least privilege in a data platform: they accumulate broad permissions for convenience, they have no joiner-mover-leaver event to trigger review, they frequently outlive the project that created them, and they are usually the most privileged identities in the environment.
[Critical] Does the access review produce a durable artifact — who reviewed, what was in scope, what was revoked, when — generated by the platform rather than assembled manually into a spreadsheet before a deadline? A review whose output cannot be produced on demand months later did not happen in any useful sense.
[Recommended] Is the review conducted over roles and their meanings rather than over raw grants? Asking a business owner to approve several hundred individual permissions produces approval without comprehension. Roles that map to job functions, with a plain-language description of what each can see, produce a review that can actually be performed.
[Recommended] Is elevated access time-bound and broken-glass by default — just-in-time elevation with an expiry, an approval, a reason, session logging, and an alert — rather than standing administrative access held "in case"? Standing privilege on the data platform is the single largest insider-risk surface it presents.
[Optional] Is there a periodic reconciliation between granted entitlements and actual usage, surfacing permissions nobody has exercised in months as revocation candidates? This is how least privilege stays true rather than being true once at design time.

Resilience, DR, and the period-shaped RTO¶

[Critical] Is the platform's criticality tier stated honestly? An analytics platform is usually not tier-one — the institution can serve customers and post transactions without it — and pretending otherwise buys expensive resilience for the wrong component. Say so plainly, and spend the resilience budget where the criticality actually is.
[Critical] Is the recovery objective recognised as period-shaped rather than flat? Mid-period, an extended outage is an inconvenience. At period end, with reporting deadlines in view, the same outage is a genuine problem. A single flat RTO is either over-engineered for most of the calendar or wrong at exactly the moment it matters. State the requirement as a calendar-dependent one and design the recovery capability against the tight window.
[Critical] Is it recognised that the real recovery position is the immutable landing zone, not the tables — because the tables are derived and rebuildable, while the raw extracts are not reproducible if lost? That inverts the usual durability spend: the landing zone deserves tier-one durability, replication, and immutability treatment even though the platform as a whole does not.
[Recommended] Has the rebuild been exercised, not just the restore — reconstructing the analytics estate from retained raw extracts and pipeline code, and reconciling the result to the general ledger? A backup restore proves the storage layer; a rebuild proves that provenance, code, and reconciliation together still work, and it is also the exit-readiness test from patterns/core-banking-data-integration.md.
[Recommended] Are upstream dependencies included in the recovery plan — the core extract delivery path, the network path to the vendor, key management availability (especially under HYOK, where an HSM outage is a data-availability outage), and the identity provider? Recovering the platform while the extract cannot be delivered restores an empty platform.
[Optional] Is there a degraded-mode plan for producing period-end obligations if the platform is unavailable at the worst moment, even if that plan is manual and unpleasant? Knowing it exists is materially better than discovering it under time pressure.

Third-party and concentration risk¶

[Critical] Is the full vendor chain for the platform inventoried — core provider, cloud provider, data platform, managed-service partner, tokenisation provider, BI vendor — with the criticality, the data each holds, and the exit position for each recorded? The platform's third-party risk profile is the union of these, and it is commonly assessed per-contract rather than as a whole.
[Critical] Is concentration assessed in both directions? Internally: how many critical functions one vendor supplies (core plus digital plus card plus analytics from a single company is a different risk from the same functions across four). Externally: how much of the sector depends on the same vendor — which the institution cannot influence but is expected to have considered.
[Critical] Is there a written and costed exit plan for the data platform specifically — where the data would go, in what format, how long a migration would take, and what the institution would lose? An exit plan that has never been costed is an aspiration, and it is exactly the artifact that gets requested.
[Recommended] Is fourth-party risk considered — the subprocessors and infrastructure providers the institution's vendors depend on — at least to the extent of knowing whether several apparently independent vendors share a single underlying dependency?
[Recommended] Are contractual rights established up front rather than sought under pressure — audit and assessment rights, access to control reports, breach notification timelines, data return and deletion on termination, and subcontracting notification? These are cheap at contract signature and effectively unobtainable afterwards.
[Optional] Is the open-format-in-own-storage mitigation applied deliberately as the technical component of the exit plan, recognising that it reduces switching cost without eliminating it — compute, catalog, orchestration, and access-control configuration all still have to be rebuilt?

Evidence generation as a design requirement¶

[Critical] Is evidence production designed alongside the data pipelines rather than after them — every material control emitting a dated artifact on a schedule, into durable storage, alongside the query or script that produced it and the period it covers? If producing evidence requires a person to run something ad hoc, it will be inconsistent, stale, and unavailable at the moment it is needed, and the control will be judged on the evidence rather than on the control.
[Critical] Are the platform's own operational records treated as data and kept in the platform — reconciliation results and breaks, load failures and their resolution, quarantined records, access reviews, key rotations, backup and rebuild tests, hold placements and suppressed deletions? This turns "show me that this control operated over the last year" from a research project into a query, and it is nearly free if designed in from the start.
[Recommended] Is evidence retained on its own schedule, independent of the data it describes, and is that schedule at least as long as the review cycle that will ask for it? Evidence that ages out before the period it covers is examined is worse than no evidence, because its absence looks like a control failure.
[Recommended] Is there a single place to look — a control register mapping each control to the artifact that evidences it, where it lives, how often it is produced, and who owns it? The common failure is not missing evidence but evidence scattered across systems such that assembling it takes weeks.
[Optional] Are exceptions and their resolutions captured with the same rigour as successes? A clean record with no exceptions is less credible than a record showing exceptions detected, escalated, and resolved, and the exception trail is what demonstrates the control is actually operating.

Designing ahead of growth¶

[Critical] Is the platform designed to be examination-ready before the institution needs it to be, rather than at the point the requirement arrives? The asymmetry is the whole argument: reporting can be built later, but provenance that was never captured cannot be backfilled, extract vintages that were never retained cannot be recovered, and snapshots that were already expired cannot be reconstructed. Retrofitting lineage and retention onto a live estate with years of accumulated data is substantially more expensive than designing for it, and parts of it are simply not recoverable at any price.
[Critical] Is the capture decision separated from the reporting decision, and made much earlier? Capturing provenance columns, retaining raw extracts immutably, and pinning period-end snapshots are cheap going forward and impossible going backward. Defer the reporting build if resources demand it; never defer the capture.
[Recommended] Is the platform's granularity chosen for the questions it may be asked later, not only for current reporting? Aggregating at ingestion to save storage is the decision most frequently regretted, because the detail is gone and the storage saving was small relative to the cost of not having it.
[Recommended] Are the institution's growth trajectory and the supervisory expectations that scale with it an explicit input to the platform roadmap, obtained from the compliance function rather than inferred by the architecture team? The specific thresholds and obligations are covered in compliance/ffiec.md and compliance/bank-regulatory-reporting.md; the architectural point here is only that the platform should reach maturity before the requirement does.
[Optional] Is there a periodic re-assessment of whether the platform's operating model still matches the institution's size and staffing? The right answer at $2B in assets and four infrastructure engineers is not necessarily the right answer at $10B with a platform team, and the transition is easier to plan than to discover.

Why This Matters¶

Platform choice under thin staffing decides more outcomes than architecture does, and it is the least written-down thing in this domain. The published guidance on lakehouse design is written by and for organisations with platform-engineering teams, and it quietly assumes an operating capability that most institutions in the low billions of assets do not have. When a self-managed open-source stack is selected in that setting, the failure is not visible at go-live — it works. It degrades over the following year or two: small files accumulate because nobody owns compaction, snapshot expiry is disabled after it once deleted something inconvenient, the engine falls behind on patches because the upgrade needs a maintenance window nobody can staff, and the knowledge concentrates in one engineer who then leaves. The resulting position — an unpatched, unowned, slow platform holding customer financial data with no one able to explain how it works — is far harder to defend and far more expensive to escape than the licence premium that was avoided. Managed platforms and the core vendor's own analytics products are legitimate architectures, not compromises, and for a thinly-staffed institution a managed platform is frequently the correct choice on the merits. The test that predicts the outcome is simple and uncomfortable: name the two people who will run compaction and patch the catalog at 2 a.m. at period end. If they do not exist, the option is not available.

Retention and legal hold conflict with lake-native maintenance at the file layer, and almost every design misses it. Compaction and snapshot expiry are configured as cost and performance settings, which is why they are not recognised as what they also are: pipelines that move rows between files and permanently destroy prior physical state. Four consequences follow and each has bitten real deployments. An object-lock legal hold does not stop compaction — it stops reclamation, so the maintenance job starts failing and storage grows without bound. A fixed-duration compliance lock on a live table prefix turns nightly compaction into a storage amplifier, accumulating a fresh undeletable copy of the touched data every cycle for the duration of the lock. Holding specific files is defeated by the next compaction, which rewrites the held rows into new unheld files and leaves the lock protecting data nobody queries. And a logical delete is not an erasure: Iceberg delete files and Delta deletion vectors leave the row physically present until a rewrite, and prior snapshots reference it regardless, so the erasure obligation is discharged only after compaction and expiry complete — the two operations a hold suppresses. Legal hold and right-to-erasure therefore collide through the file layer, not merely as competing policies. The resolution that works is structural rather than clever: put the preservation obligation on an immutable raw landing zone that is never rewritten, treat the tables as rebuildable derivatives, pin the snapshots that reproducibility depends on, and make the reclamation step hold-aware and fail-safe like any other deletion path.

Snapshot expiry is a records-destruction event whenever reproducibility is the obligation, and it is almost never configured that way. The default retention windows in lake table formats are measured in days because they are designed to bound storage cost. If the institution's position is that it can reproduce a reported figure by querying the table as it stood at period close, then that position expires along with the snapshot — typically about a week after the period closes, and silently. Expressing retention over snapshots rather than only over rows, and pinning period-end states with explicit long retention, is a small configuration change that has to be made before the first period end rather than after the first request to reproduce something.

Key custody is judged on operability, not on which acronym was selected. There is a strong pull toward the strongest-sounding model, and HYOK genuinely does buy something the others do not: the provider cannot decrypt without the institution, and the institution can revoke unilaterally. It also buys a hard availability dependency where an HSM or network fault makes data unavailable rather than merely unreadable, latency on every cryptographic operation, narrower service coverage, and a real key-management function with ceremonies, runbooks, and staff. An institution that adopts HYOK without that function has a control it cannot evidence, cannot rotate reliably, and cannot recover from — which reviews poorly against CMK with clean separation of duties, evidenced rotation, and a tested recovery runbook. The questions that actually get asked are about custody being documented and consistent with the risk assessment, separation between the authority to destroy keys and the authority to read data, evidence that rotation occurs, and a tested position for key loss. Choose the strongest model the institution can operate and evidence, not the strongest model available.

Masking over an open table format is a usability feature unless every path to the bytes goes through the catalog. Column masks and row filters enforced by a query engine are genuinely useful and genuinely not a boundary if a second engine, a notebook holding storage credentials, a replication job, or a backup process can read the underlying files directly. This is a specific consequence of the lakehouse's central virtue — many engines over one open storage layer — and it means the segmentation design has to decide explicitly whether it is relying on catalog enforcement (in which case all direct paths must be closed and kept closed) or on storage-level separation (in which case sensitive columns are physically elsewhere). Designs that assume the mask is the boundary while leaving direct storage access open have a gap that is straightforward to find and awkward to explain.

Evidence is the deliverable, and evidence that requires a person to produce it will not be there. The control is judged on what can be shown about its operation, not on its design, and the gap between "we do reconcile every load" and "here is the reconciliation result for every load in the last four quarters, with the breaks and how they were resolved" is the entire distance between a control that passes and one that does not. Building evidence emission alongside the pipelines is close to free — the platform is already a system for storing and querying records of what happened — and assembling it retroactively is a multi-week exercise performed under time pressure by people who have other work. The same logic applies to the platform's own operational history: reconciliation results, quarantined loads, access reviews, key rotations, hold placements, and suppressed deletions belong in the platform as data, because that is what turns a year-long evidence request into a query.

Capture is cheap forward and impossible backward, which is the whole argument for designing ahead of growth. As an institution grows, the granularity of what it must be able to produce grows with it, and the platform is expected to have been operating properly for the periods in question — not merely to be capable now. Reporting can be built later; a dashboard is a few weeks of work at any point. Provenance columns that were never written, extract files that were never retained, start events for retention that were never captured, and snapshots that were already expired cannot be recovered at any price. The practical rule is to separate the capture decision from the reporting decision and make the capture decision far earlier and far more generously than current requirements justify, because the asymmetry is severe and entirely one-directional.

Common Decisions (ADR Triggers)¶

Operating model — self-managed open-source lake stack (lowest licence cost, maximum control, requires a genuine platform bench and an on-call rotation) vs managed lakehouse or warehouse (highest licence cost, lowest operational headcount, governance built in, platform lock-in partially mitigated by open formats) vs the core vendor's analytics product (fastest to value, extraction solved, deepens vendor concentration, cannot blend non-core sources, does not survive a core conversion) vs a managed-service partner operating the stack (fills the staffing gap, adds a third-party risk obligation and key-person risk)
Key custody model — provider-managed (no custody, no revocation) vs CMK (policy, rotation, disablement, key-level audit; provider still holds the material) vs BYOK (adds attestation of generation and expiry of provider capability) vs HYOK/external key store (only model where the provider cannot decrypt unilaterally; availability coupling, latency, narrower service coverage, real operational cost)
Where the preservation obligation sits — on an immutable raw landing zone with the tables as rebuildable derivatives (clean separation, maintenance runs freely, requires disciplined landing and rebuild capability) vs on the live tables themselves (no separate tier to operate, forces every maintenance operation to become hold-aware and constrains compaction and expiry permanently)
Object-lock mode and placement — indefinite legal-hold-mode locks on scoped, pinned copies (releasable, matches an open-ended hold) vs fixed-duration compliance-mode retention on the landing zone (matches a retention schedule, must never be pointed at a rewriting table prefix) — the two are not interchangeable and the failure modes differ
Snapshot retention policy — short default expiry treating snapshots as a cost knob (cheapest storage, destroys historical reproducibility within days) vs long retention on pinned period-end snapshots with normal expiry elsewhere (preserves the reproducibility obligation at bounded cost, requires deliberate tagging discipline)
Erasure mechanism — physical rewrite plus snapshot expiry (unambiguous destruction, expensive at scale, blocked by any overlapping hold) vs crypto-shredding by destroying a scoped data key (no rewrite, requires the key hierarchy to match the erasure boundary by design, acceptance as destruction must be confirmed)
PII segmentation enforcement point — catalog-enforced masks and row filters over a shared table (single copy, convenient, only a boundary if every direct path to storage is closed) vs physical separation of identifying attributes into a restricted zone (a real boundary, more modelling and joins, harder to retrofit)
Tokenisation approach and boundary — tokenise upstream of the platform so the lake is never in PCI scope (strongest and cheapest scope outcome, requires the boundary to be set early) vs tokenise within the platform (simpler ingestion, brings the platform into scope); and vault-based (strong separation, availability dependency, high-value target) vs format-preserving encryption (no vault, keys become the sensitive material)
Residency posture — strict single-region with every leak path closed including metadata, support access, and SaaS tooling (defensible, restricts service selection and DR options) vs primary-region storage with documented and accepted exceptions (broader service choice, requires the exceptions to be enumerated and owned rather than unexamined)
Recovery objective shape — a single flat RTO/RPO for the platform (simple, over-engineered most of the year or wrong at period end) vs a calendar-dependent objective with the tight window at period end (matches the real requirement, more complex to state and to test)
Durability spend allocation — treat the platform uniformly (simple, over-spends on rebuildable tables and under-spends on the landing zone) vs tier-one durability and immutability on the landing zone with the tables treated as rebuildable (matches where the irreplaceable data actually is, depends on the rebuild being tested rather than assumed)
Evidence production — emitted automatically by the platform on a schedule into durable storage with its generating query (near-free if designed in, requires design in) vs assembled on request (no build cost, unreliable and expensive exactly when it is needed)

Reference Architectures¶

Table format mechanics — the operations that rewrite files

Apache Iceberg table specification — snapshots, manifests, data files, and the positional and equality delete files that make a logical delete distinct from a physical one
Iceberg maintenance — snapshot expiry, orphan-file removal, and compaction; the operations that must become hold-aware
Iceberg Spark procedures — rewrite_data_files, expire_snapshots, remove_orphan_files, and the tag and branch procedures used to pin period-end snapshots
Delta Lake utility commands — VACUUM, retention settings, and the file-retention semantics that govern when deleted data physically disappears

Immutability and preservation primitives

Amazon S3 Object Lock — governance and compliance retention modes and the distinct indefinite legal-hold flag; the difference between these two is the difference between the two failure modes described above
Azure immutable blob storage — time-based retention policies and legal holds at container and blob scope
Google Cloud Storage bucket lock — retention policies, retention policy locks, and object holds

Key management and custody

AWS KMS external key store — the HYOK model on AWS, including the availability and latency consequences of keeping key material outside the provider
Azure Key Vault Managed HSM — single-tenant HSM-backed key management and its security domain model
Google Cloud External Key Manager — externally-held key material with Cloud KMS calling out for cryptographic operations
NIST SP 800-57 Part 1 Rev. 5 — key management lifecycle, key hierarchies, and cryptoperiods; the reference for designing an envelope hierarchy that supports crypto-shredding
FIPS 140-3 — cryptographic module validation, relevant when the institution's requirements reference validated modules

Governance, lineage, and access control

AWS Lake Formation — catalog-enforced fine-grained access control over a lake, and a concrete example of the catalog-versus-storage boundary question
OpenLineage — open standard for automatic lineage collection across pipeline tools
DataHub — open-source metadata and lineage platform

Compliance references

PCI Security Standards Council — the standards that define cardholder data environment scope and therefore what tokenisation upstream of the platform actually achieves