Core Banking Data Integration¶
Scope¶
Covers getting data out of a financial institution's core banking system (the system of record for accounts, balances, and postings) and into an analytics platform — the extraction problem that governs every downstream data-lake, warehouse, and reporting design decision and is almost always underestimated.
Topics: the core platform vendor landscape (Fiserv, FIS, Jack Henry, Temenos, Finastra) and what each realistically offers for bulk data access; the vendor-supplied relational reporting replica and why establishing whether one exists is the first question to ask; why extraction is hard (nightly batch windows and end-of-day cycles, fixed-width and COBOL-derived record formats, EBCDIC and packed-decimal encoding, copybook-driven schemas, metered or absent APIs, per-extract licensing, and contract terms constraining what may leave the vendor's environment); the landing-zone and operational-data-store patterns that work; full-extract versus delta files; whether log-based change data capture is even available; reconciliation against the general ledger on every load; the end-of-day boundary as the definition of "as-of"; the memo-post versus posted distinction; the adjacent source systems (digital banking, card processing, loan origination, payments, and the customer information file) that the core does not contain; and what portability and exit actually look like when the vendor holds the authoritative record.
This file is about acquisition. The platform that consumes the result — encryption, key custody, retention, lineage, evidence — is patterns/regulated-financial-data-platform.md. Generic pipeline mechanics are in patterns/data-pipeline.md.
A note on sourcing. Core banking platform documentation is almost entirely behind customer login. Public, citable technical detail on record layouts, extract catalogs, and API surfaces is thin to nonexistent for most of these platforms. This file therefore describes mechanisms and the questions to ask, and deliberately avoids asserting platform-specific specifics that cannot be verified without a customer relationship. Where something is commonly reported in practice but not publicly documented, it is labelled as such. Verify every platform-specific assumption against your own contract and your vendor's customer documentation before designing to it.
Overview¶
A core banking system is the ledger: it holds accounts, balances, the posting engine, and the general ledger, and it is the authoritative record for the institution's financial position. Everything an analytics platform says about the institution is, ultimately, a copy of something the core knows.
The architectural consequence is that the core's extraction interface — not the analytics platform's capabilities — sets the ceiling on what the data platform can do. If the core produces one flat file per night after end-of-day posting completes, then the platform is a daily-batch platform, and no amount of streaming infrastructure downstream changes that. Teams routinely design a lakehouse around real-time ambitions, then discover in week six that the only sanctioned interface is an SFTP drop of fixed-width files at 4 a.m. whose schema is described by a COBOL copybook and whose delivery time varies by two hours depending on how long month-end posting ran.
Two delivery models dominate and they produce completely different extraction stories:
- In-house (licensed) — the institution runs the core on its own hardware (frequently IBM Power/IBM i or a Windows/SQL Server estate, depending on the platform). Database-level access is physically possible; whether it is contractually and supportably possible is a separate question the vendor answers.
- Outsourced (service bureau / hosted / vendor-operated) — the vendor runs the core in its own environment and the institution receives files and reports. There is no database to attach to. Batch files and whatever APIs the vendor sells are the entire interface. This is the common case for community and mid-size institutions, and it is the case most analytics designs fail to plan for.
Establish which model applies before anything else in the design. It changes the answer to nearly every subsequent question.
Cutting across both models is a third possibility that is routinely discovered too late: many core vendors ship, or will provision on request, a relational reporting replica alongside the core — a reporting database, ODS, or vendor data warehouse, in practice very often SQL Server. Where one exists it changes the entire integration design, because a relational source opens paths a fixed-width nightly file does not: query-based change detection against the replica even when the core itself permits no log access, the managed replication and mirroring services the major cloud data platforms offer for relational sources (which remove most of the pipeline that would otherwise have to be built), and a schema that is at least documented rather than reconstructed from copybook archaeology.
"Does a relational replica exist?" is the highest-value first question in a core integration, and it is routinely asked far too late — usually after a copybook-parsing pipeline has already been built. Ask it in the first vendor conversation, before the extraction design is drafted. The answer is not automatically "use it": a replica has its own traps, covered below.
Checklist¶
The vendor-supplied relational reporting replica — ask this first¶
- [Critical] Has the vendor been asked directly whether a relational reporting database, ODS, or data warehouse is available for this core — shipped as part of the platform, sold as an add-on, or provisionable on request? This question is cheap, it is frequently answered "yes" for platforms where the institution assumed flat files were the only option, and a "yes" invalidates most of a file-based extraction design. Ask before drafting the design, not after.
- [Critical] Is the replica's refresh mechanism and actual lag established from the vendor in writing, rather than inferred from the word "replica"? The range in practice spans continuous transactional replication or log shipping (minutes), scheduled incremental refresh (hours), and — very commonly — a nightly restore or full rebuild after the end-of-day cycle, which is a batch snapshot wearing a relational interface. A nightly-restore replica offers a queryable schema and managed-replication tooling, but it offers no latency improvement whatsoever over a flat file, and any "near-real-time" promise built on it is false. Get the cadence, the typical completion time, and the behaviour on month-end.
- [Critical] Is it clear whose replica it is — the vendor's product running in the vendor's environment, the vendor's product running on the institution's infrastructure, or a replica the institution built itself from delivered extracts? Ownership determines who can change the schema, who can add indexes, whether the institution may connect third-party tooling, whether the connection can leave the vendor's network, and what happens to it at contract exit. A vendor-hosted replica is a vendor dependency in the same way the core is; an institution-hosted one is an estate the institution must operate and patch.
- [Critical] Is reading directly from the replica permitted and supported, and at what cost — direct SQL connectivity, read-only credentials, connection and concurrency limits, licensing for the underlying database engine, and whether attaching third-party ETL, CDC, or mirroring tooling voids support? Whether a nightly reporting database may be queried by an external replication agent is a contract and support question first and a technical one second, and it is the question most likely to invalidate an otherwise sound design late.
- [Critical] Is it established whether the replica is a faithful record of the core or a reporting-shaped view of it? This is the most consequential and most frequently missed question. Reporting databases are commonly denormalised, pre-aggregated, filtered to "active" records, subject to their own retention (purging closed accounts or aged transactions on a schedule the core does not apply), and populated by transformation logic the vendor owns and does not fully document. A replica that is convenient for a dashboard may be inadequate for regulatory, audit, or restatement use precisely because it is not the record — it is a derivative whose derivation the institution cannot inspect. Convenience and evidentiary adequacy are different tests and must both be applied.
- [Critical] Does the replica still reconcile to the general ledger, and has that been proven rather than assumed? The tie-out requirement does not weaken because the source became relational — if anything a transformed replica makes it more necessary, because the transformation is an additional place for divergence. Where the replica does not carry GL or trial-balance detail, the reconciliation source must come from the core's own reporting, and the pipeline still gates publication on the tie. A replica that cannot be reconciled is usable for exploration and unusable for anything reported.
- [Recommended] If the replica is the chosen source, is query-based change detection designed around a trustworthy change marker — a vendor-maintained last-modified timestamp, a row version, or a monotonic sequence — with an explicit answer for hard deletes, which query-based detection cannot see? The usual robust construction is incremental capture on the change marker plus a periodic full-key reconciliation snapshot to detect disappearances and drift. Where the underlying engine's native change tracking or CDC features are available and the vendor permits enabling them, prefer those over a hand-rolled watermark.
- [Recommended] Have the managed replication and mirroring services of the target data platform been evaluated against the replica before building a custom pipeline — the cloud providers' database migration and streaming-replication services, and the data platforms' native mirroring of relational sources? Where the source is relational, permitted, and reachable, these remove most of the ingestion code, and the resulting operational burden is far lower than a hand-built extractor. This is the single largest practical benefit of a replica existing, and it is worth confirming eligibility (engine, version, edition, network path, permissions) early because the constraints are specific.
- [Recommended] Is the replica's schema drift at core release upgrades planned for, with a change-notification path from the vendor and a validation step in the pipeline? A documented schema is a large improvement over a copybook, but it is still the vendor's schema, it changes on the vendor's release cycle, and the institution generally has no veto. Column additions are benign; type changes, renames, and semantic changes to a status-code domain are not, and they should fail the load rather than pass through.
- [Recommended] Is the query load the platform places on the replica bounded and scheduled, and is it clear whether the replica shares infrastructure with anything the core depends on? Extraction queries against a reporting database that shares an instance, storage, or a maintenance window with production processing can affect the core's own cycles. Read from a secondary or a restored copy where the topology allows it, and agree a window with the vendor where it does not.
- [Optional] Where both a relational replica and flat-file extracts are available, has a deliberate split been made rather than defaulting to one — replica for the breadth of master and reference data and for exploration, flat files retained immutably as the evidence and rebuild source? The raw-file immutability requirement below does not go away because a replica exists; the replica is a convenience layer over the record, not a substitute for holding the record.
Establishing the extraction surface¶
- [Critical] Is the core delivered in-house (licensed, institution-operated) or outsourced (vendor-hosted / service bureau), and has that been confirmed with the vendor rather than assumed from how the institution talks about it? The two models produce entirely different extraction options: in-house may permit database-level access and log-based CDC (subject to contract and support terms), outsourced almost never does. Every downstream latency, freshness, and CDC decision depends on this answer, and it is the cheapest question to get right early.
- [Critical] Has the actual catalog of available extracts been obtained from the vendor in writing — file names, record layouts, delivery mechanism, delivery schedule, and which are included in the current contract versus billable? Do not design against an assumed extract. Vendors typically maintain a customer-portal catalog; the useful artifact is the list of what this institution's contract entitles it to, which is frequently narrower than the platform's full capability.
- [Critical] For every extract, is the schema artifact identified and version-controlled alongside the data — a COBOL copybook, a fixed-width position map, a data dictionary, or a documented relational schema? On legacy-derived extracts the copybook is often the only schema description that exists, it is supplied as a file rather than a queryable catalog, and it changes at core release upgrades. An extract whose copybook is not stored next to it is unreadable the moment the layout changes.
- [Critical] Is the character encoding and numeric representation of each field established — ASCII versus EBCDIC, and per-field display/zoned-decimal versus packed-decimal (COBOL
COMP-3) versus binary (COMP)? A mixed record cannot be transcoded wholesale: running an EBCDIC-to-ASCII conversion across a record that contains packed-decimal or binary fields silently corrupts those fields while leaving the text fields looking correct. Parse per-copybook, field by field; never transcode the whole record and then split it. - [Critical] Are the awkward COBOL constructs in the layout identified —
REDEFINES(the same bytes meaning different things depending on a discriminator field),OCCURS DEPENDING ON(variable-length repeating groups whose count comes from another field), signed overpunch in zoned decimal (the sign encoded in the high nibble of the trailing byte), and implied decimal points (PIC S9(11)V99carries no decimal character in the data)? Each of these breaks naive fixed-width parsers in a way that produces plausible-looking wrong numbers rather than an error. - [Critical] Is the extract full-snapshot or delta, and if delta, what defines the delta — changed records since the last cycle, or only transactions posted in the cycle? A "delta" that carries only postings does not tell you about non-posting changes (address updates, status flags, rate changes, closures), so a downstream table built only from deltas silently drifts from the core. A common resilient design is nightly delta for transactions plus a periodic full snapshot of master files for reconciliation and drift correction.
- [Recommended] Is there an API surface in addition to batch, and has its practical shape been established — read scope, rate limits, pagination, whether it is priced per call, and crucially whether it is designed for transactional lookups or bulk extraction? Core vendor APIs are generally built for single-record servicing calls from a channel application. Using them to move a portfolio nightly is usually rate-limited, expensive, or contractually disallowed, and paginating a million accounts through a servicing API is not a bulk-extract strategy.
- [Recommended] Has log-based change data capture been evaluated realistically rather than assumed? On an outsourced core it is normally unavailable. On an in-house core it may be technically possible (database transaction logs; on IBM i, journals) but is frequently constrained by vendor support terms, may void support, and breaks at release upgrades when the vendor changes the physical schema. Treat CDC availability as a contractual question first and a technical one second.
- [Optional] If the core is a newer API-first platform (for example Fiserv's Finxact, FIS's Modern Banking Platform, Temenos Banking Cloud, or Finastra's FusionFabric.cloud-exposed products), does it offer event streaming or a documented bulk-export path that changes the batch assumption? These platforms genuinely differ from the legacy cores, but they are a minority of installed base — confirm what the institution actually runs rather than what the vendor's marketing site describes.
The batch window and the as-of boundary¶
- [Critical] Is the end-of-day cycle understood — when posting starts, how long it typically runs, and when the extract becomes available? The extract cannot be produced until posting completes. Month-end, quarter-end, and year-end cycles run materially longer, and those are exactly the cycles whose output is most time-critical downstream.
- [Critical] Is the ingestion pipeline file-arrival-triggered rather than clock-triggered? A pipeline scheduled at a fixed time will either process a stale file or process nothing on the nights the core runs long, and the failure is silent — the job succeeds and publishes yesterday's data. Trigger on arrival, validate the trailer/manifest, and alert on non-arrival by a deadline rather than assuming arrival at a time.
- [Critical] Is the core's business date distinguished from the calendar date, and are the several dates on a transaction modelled separately — effective date, posting date, and processing/cycle date? Weekend and holiday cycles post multiple business dates in one run; back-dated and correction entries post to a prior business date. A table keyed on an undifferentiated "date" column cannot answer either "what happened that day" or "what did we know as of that day," and the two questions have different answers.
- [Critical] Is the platform bitemporal where it matters — carrying both the business/effective date and the extract cycle in which the fact arrived — so that a restated prior period can be reproduced both as-originally-reported and as-currently-known? Regulatory and management reporting both ask "reproduce the number we published" and "give me the corrected number," and a single-date model can only answer one of them.
- [Recommended] Is the memo-post versus posted distinction explicit in the model? Memo posts (pending card authorizations, holds, real-time debits) affect available balance but have not hit the ledger; posted entries have. They are not reliable predictors of postings — authorizations expire, settle for different amounts (tip adjustments, fuel pre-authorizations), or never settle. A nightly extract taken after posting generally carries ledger state and may not carry intraday memo state at all, so "balance" computed from postings will not tie to a report built on available balance. Name which balance each field is.
- [Recommended] Is intraday state explicitly declared as in or out of scope, with the consequence written down? Most core extraction paths give an end-of-day picture only. If the business asks for intraday, the honest answer is usually that the data does not exist in the extract and must come from a different source (the digital banking platform, the card processor's authorization feed) with different identifiers and different completeness.
- [Optional] Are non-daily cycles catalogued — monthly statement cycles, quarterly interest accrual and capitalization, annual tax reporting extracts — and modelled as their own arrival contracts rather than treated as unusual daily files? Their layouts, volumes, and reconciliation rules differ.
Landing zone, staging, and reconciliation¶
- [Critical] Is the vendor extract landed byte-for-byte and immutably in a raw zone before any parsing, with its manifest, trailer record, control totals, and the copybook/layout version that describes it? The raw file is the institution's evidence and its rebuild source; every derived table is reproducible from it and nothing else is. This single decision is what makes the rest of the platform recoverable, auditable, and portable. See
general/legal-hold.mdandpatterns/regulated-financial-data-platform.mdfor the immutability and retention mechanics. - [Critical] Does every load reconcile to control totals before publishing — record counts and hash/amount totals from the file trailer, and balance totals tied to the core's trial balance or general ledger by GL account, product, and branch? A load that does not tie must be quarantined and alerted, never published. This is the single control that most distinguishes a banking data pipeline from a generic one, and generic data-quality tooling does not supply it because the ground truth (the GL) lives in another system.
- [Critical] Is there a staging / operational data store layer between the raw landing zone and the analytics model, where the extract is parsed into typed columns, control totals are verified, and referential integrity across master and transaction files is checked — before anything is exposed to consumers? Publishing directly from parse to consumption means a layout drift or a partial file becomes a business-facing incident instead of a quarantined load.
- [Critical] Is the load idempotent and replayable from the raw zone, so that a re-delivered or corrected extract can be reprocessed without duplicating rows or requiring manual cleanup? Cores do re-issue files. Design for reprocessing on day one rather than discovering the need during a month-end correction.
- [Recommended] Is layout drift detected rather than tolerated — the parse validated against the declared copybook version, with a hard failure (not a best-effort parse) when field positions, lengths, or the record length change? Fixed-width parsing degrades silently: a one-byte insertion upstream shifts every subsequent field and produces a file that parses cleanly into wrong values. Assert the total record length and validate a checksum column on every load.
- [Recommended] Is the as-of snapshot for master files (accounts, customers, loans) modelled as a slowly-changing dimension against the extract cycle, so that a question asked about a prior period resolves against the master data as it stood then rather than as it stands now? Overwriting master records in place destroys the ability to reproduce any historical report.
- [Recommended] Are rejected and quarantined records retained with the reason, the source file, and the byte offset, rather than dropped? On a fixed-width feed the rejects are the early-warning signal for layout drift, and they are also records the institution holds and must account for.
- [Optional] Is a parallel-run reconciliation planned for the first several cycles — the new platform's outputs compared line-by-line against the incumbent report for the same period — before the platform is trusted for anything consequential? The differences found in a parallel run are almost always in date semantics and balance definitions, which is exactly what the checklist items above exist to surface.
Adjacent source systems¶
- [Critical] Is the customer information file / party master (CIF) treated as its own integration with its own identity problem, rather than as an attribute of the account extract? The account-to-party relationship is many-to-many with role semantics (primary owner, joint owner, authorized signer, beneficiary, guarantor, power of attorney) and the relationship table — not the account table — is the hard part of the model. Duplicate party records for the same human being are normal in a core that has absorbed acquisitions, and household/relationship rollups sit on top of that.
- [Critical] Are the identifier mismatches between systems mapped explicitly — the digital banking platform's user id is not the core's party id, the card processor's cardholder id is not either, and the loan origination system's applicant id exists only until booking? Every cross-system analysis depends on a maintained crosswalk, and the crosswalk is a first-class asset that needs an owner, a quality measure, and a match-rate SLA, not a
LEFT JOINwritten once. - [Recommended] Is the digital banking / online and mobile platform identified as a separate ingestion with its own vendor, its own extract mechanism, and frequently the richest behavioural data in the institution (sessions, enrollments, devices, alerts, secure messages)? It is often a different vendor from the core even when both are sold by the same company, and its data is not in the core extract.
- [Recommended] Is the card processor feed scoped separately for authorizations, settlements, interchange, disputes and chargebacks — and is the platform designed so that the primary account number (PAN) is tokenized before it lands, keeping the analytics estate out of PCI scope? See
compliance/pci-dss.mdand the tokenisation checklist inpatterns/regulated-financial-data-platform.md. - [Recommended] Is the loan origination system ingested in addition to the core? The core generally holds only booked loans. Applications that were declined, withdrawn, or expired exist only in the origination system — and those are precisely the records that fair-lending and pipeline analysis need. A platform that ingests only the core is structurally blind to them.
- [Recommended] Are payment rails (ACH, wire, instant payments, remote deposit) ingested with their operational metadata — returns and return reason codes, notifications of change, addenda records, and screening/exception queues — rather than only as the resulting core postings? The posting says money moved; the rail data says how, why it failed, and what had to be worked by hand.
- [Optional] Are servicing and ancillary systems inventoried early — collections, trust, wealth, treasury management, safe deposit, item processing and imaging, and any acquired-institution system still running — so that the platform roadmap is honest about coverage rather than discovering a whole business line in year two?
Contract, licensing, and exit¶
- [Critical] Has the core contract been read for what the institution is actually permitted to extract, how often, and where the data may go? Terms that constrain data leaving the vendor's environment, require vendor approval for third-party integrations or connectivity, or restrict use of the institution's own data are commonly reported in the industry and are not visible from any technical investigation. This is a contract review, not an architecture review, and it should happen before the platform design is fixed.
- [Critical] Are the costs of access established — per-extract, per-record, per-API-call, per-report, one-time setup for a new extract, and the change fee to modify an existing one? Charging for access to the institution's own data is commonly reported practice on legacy core contracts. It changes the economics of "just extract everything nightly" and it is frequently the reason a design that is technically sound is commercially unworkable.
- [Critical] Is the raw extract retained by the institution, under the institution's control, in the institution's own storage — not only in the vendor's environment or the vendor's analytics product? This is the concrete form of exit readiness, and it is the difference between the institution owning its history and renting it.
- [Recommended] Are de-conversion terms understood — what the vendor is obliged to produce on exit, in what format, on what timeline, at what cost, and how much history (versus current balances and open records) is included? Substantial de-conversion fees and limited historical conversion are commonly reported. Design on the assumption that on exit the institution receives current state plus whatever history it independently retained.
- [Recommended] Is the field-level mapping documentation kept outside the ETL code — a maintained dictionary from core field to business meaning, with the copybook vintage it applies to? When the only description of what
ACCT-STAT-CD = '7'means lives in aCASEstatement written by someone who has left, the institution has lost the ability to migrate, audit, or explain its own data. - [Recommended] Is concentration with the core vendor assessed across all functions the vendor supplies — core, digital, card, payments, and analytics — rather than per-product? See the concentration-risk checklist in
patterns/regulated-financial-data-platform.md. - [Optional] Is there a written rebuild procedure proving the analytics estate can be reconstructed from retained raw extracts alone, and has it been exercised on a non-trivial subset? Exit readiness that has never been tested is a claim, not a capability.
Why This Matters¶
The extraction interface, not the analytics platform, is the binding constraint — and it is discovered late. The pattern is consistent: a platform is scoped around business questions, a modern lakehouse is selected, and then extraction turns out to be a nightly fixed-width file whose schema is a copybook, delivered at a variable time, containing a subset of the fields the design assumed, at a per-extract fee. Every latency promise, every real-time dashboard, and a good share of the data model has to be rebuilt around what the core actually emits. Establishing the extraction surface in writing — delivery model, extract catalog, layouts, schedule, and cost — before the platform architecture is fixed is the single highest-leverage sequencing decision on this kind of programme, and it costs a few conversations with the vendor.
The relational replica question is worth asking before anything else because a "yes" makes most of the file-based design unnecessary — and because a naive "yes" is its own trap. Teams routinely build a copybook parser, a fixed-width validation layer, and a bespoke incremental-load framework, and then learn in month four that the vendor would have provisioned a SQL Server reporting database that the target platform's managed mirroring service could have consumed directly. The wasted effort is large and entirely avoidable by asking one question in the first vendor conversation. The symmetric error is treating any relational source as a solved problem. Two properties have to be verified independently and usually are not. The first is lag: a very common implementation is a nightly restore after the end-of-day cycle, which is a batch snapshot with a SQL interface — it gives a documented schema and access to managed replication tooling, and it gives exactly zero latency improvement over a flat file, so any near-real-time commitment made on the strength of "we have a replica" is unfounded. The second is fidelity: reporting databases are frequently denormalised, pre-aggregated, filtered to active records, and subject to their own purge schedules, populated by vendor-owned transformation logic the institution cannot inspect. That makes the replica a derivative rather than the record. It is often perfectly good for analysis and simultaneously inadequate for a regulatory or audit-grade answer, and the two tests are different. The practical resolution is usually not either/or: use the replica for breadth and convenience, keep landing the raw extracts immutably as the evidence and rebuild source, and reconcile both to the general ledger. The tie-out obligation does not relax because the source became relational — a transformation step is one more place for the numbers to diverge.
Silent corruption is the characteristic failure mode of legacy-format extraction, and it produces wrong numbers rather than errors. Packed-decimal fields read as text, a record wholesale-transcoded from EBCDIC, a signed overpunch parsed as a letter, an implied decimal point applied in the wrong place, or a one-byte upstream insertion shifting every field after it — none of these throw. They produce a file that parses, loads, and populates dashboards with values that are plausibly shaped and materially wrong. In an institution where the outputs feed management reporting and regulatory submission, "plausibly shaped and wrong" is a considerably worse outcome than a failed job. This is why per-field copybook-driven parsing, hard record-length assertions, and control-total reconciliation on every load are not optional hygiene — they are the only mechanisms that convert a silent corruption into a loud failure.
Reconciliation to the general ledger is the control that generic data engineering does not supply. Standard data-quality tooling validates a dataset against itself: schemas, nulls, ranges, distributions. None of that detects a load that is internally consistent but does not tie to the ledger, which is the failure that matters. The ground truth lives in another system, and the discipline — sum the detail, compare to the trial balance by GL account and product, quarantine on mismatch — has to be built deliberately. An analytics platform in a financial institution that cannot demonstrate that its balances tie to the core is not usable for anything consequential, and retrofitting the tie-out after consumers have built on untied data is a credibility problem as much as an engineering one.
Date semantics are where the model quietly goes wrong. Effective date, posting date, processing date, and the core's business date are four different things; weekend and holiday cycles post multiple business dates in one run; corrections and back-dated entries land in a cycle after the period they belong to. A model that collapses these into one date column produces a platform that can answer neither "what happened on that day" nor "what did we believe on that day," and the divergence from the incumbent reporting shows up as an unexplainable variance that erodes trust in the platform. The bitemporal shape — business date plus arrival cycle — is more work up front and is the only structure that survives contact with restatements.
The data platform is the one copy of institutional history the vendor does not own — if it is built that way. When a core conversion happens, the outgoing vendor produces a de-conversion extract on contractual terms, and it is commonly reported that historical detail is only partially converted; the practical fallback is retaining the old system or its files in read-only form. An institution that has been landing raw extracts immutably into its own storage for years already holds its own history, in its own account, independent of any vendor's cooperation. That reframes the platform's business case: the reporting and analytics are the visible return, but the durable strategic asset is vendor-independent ownership of the institution's own record. It is also the argument that decides between building an institution-owned lake and adopting the core vendor's analytics product — the latter is faster to value and does not survive the conversion.
The adjacent systems are where the interesting questions live, and the identifier crosswalk is the real project. The core answers what the balances are. Why a customer left, which channel they used, which applications were declined, which payments failed and had to be worked by hand — none of that is in the core. Each adjacent system arrives with its own identifiers, and joining them is not a technical detail but a sustained data-management commitment with a match rate that degrades if nobody owns it. Platforms that treat the crosswalk as a one-time join produce cross-system analyses whose coverage silently decays, and the decay is invisible because the queries keep returning rows.
Common Decisions (ADR Triggers)¶
- Extraction interface — vendor batch files (universally available, end-of-day granularity, layout-fragile) vs a vendor-supplied relational reporting replica (documented schema, managed-replication eligible, fidelity and lag must be proven) vs vendor API (record-level, rate-limited and often priced per call, rarely a bulk path) vs direct database access or log-based CDC (lowest latency, generally only possible on in-house installs, and a contract/support question before a technical one)
- Relational replica as primary source vs as convenience layer — make the replica the system of ingestion (least pipeline code, fastest delivery, inherits the vendor's transformation and purge semantics, and may not be evidentiary) vs keep raw extracts as the authoritative landing and use the replica for exploration and breadth (two ingestion paths to operate, preserves an inspectable record and rebuild source) — the choice turns on whether the platform's outputs will ever need to be defended
- Replica ingestion mechanism — a managed replication or mirroring service consuming the relational source directly (minimal code, low operational burden, eligibility constrained by engine/version/edition/network and by what the vendor permits) vs native database change tracking or log-based CDC (finer-grained, requires enabling features on a vendor-owned database) vs a hand-built watermark extractor on a last-modified column (works anywhere, blind to hard deletes, needs a periodic full-key reconciliation to stay correct)
- Delta versus full snapshot — nightly delta only (smallest transfer, drifts from the core on non-posting changes) vs periodic full snapshot only (always correct, largest transfer and coarsest history) vs the hybrid of nightly delta plus scheduled full reconciliation snapshot (the resilient default, at the cost of two ingestion paths)
- Raw-file immutability and retention scope — retain every extract byte-for-byte indefinitely (maximum rebuild and exit optionality, growing storage and retention/hold obligations) vs retain only a rolling window (cheaper, and permanently forecloses rebuild and historical restatement beyond the window)
- Parsing approach for legacy layouts — copybook-driven parsing with a maintained library (correct on
REDEFINES,OCCURS DEPENDING ON, packed decimal and overpunch; requires the copybook to be supplied and versioned) vs hand-written fixed-width parsing (fast to start, systematically wrong on the constructs above) vs commercial mainframe-data tooling (handles the edge cases and adds licence cost and a vendor) - Where reconciliation is enforced — a hard gate in the pipeline that quarantines any load failing GL tie-out (protects consumers, blocks publication on vendor-side anomalies) vs a post-publication reconciliation report (data always flows, and consumers may act on untied numbers before anyone reads the report)
- Temporal model — single effective-date model (simpler, cannot reproduce as-reported figures after restatement) vs bitemporal business-date plus arrival-cycle (reproduces both as-reported and as-corrected, materially more modelling and query complexity)
- Landing target and ODS placement — land and stage in the cloud data platform directly (fewer moving parts, requires the extract to leave the vendor/on-prem boundary early) vs an on-premises ODS that parses and reconciles before anything moves (satisfies data-movement constraints and adds an estate to operate) — see
patterns/hybrid-cloud.md - Core vendor analytics product versus institution-owned platform — vendor analytics (fastest to value, extraction solved, data model matches the core, deepens vendor concentration and does not survive a conversion) vs institution-owned platform (owns its own history, blends non-core sources, requires solving extraction and operating a platform)
- Adjacent-source sequencing — core-only first release (fastest credible reconciliation story, structurally cannot answer channel, origination, or payments questions) vs core plus one adjacent system in the first release (forces the identifier crosswalk to be designed rather than deferred, longer to first value)
- Identifier crosswalk ownership — deterministic keying on a vendor-supplied cross-reference where one exists (accurate, coverage limited to what the vendor maintains) vs probabilistic matching with a stewardship process (higher coverage, requires a match-rate SLA, review queue, and a named owner)
- CDC pursuit on an in-house core — pursue log-based CDC (near-real-time, may conflict with vendor support terms, breaks at release upgrades, needs re-validation every upgrade) vs accept batch and design the business expectations around end-of-day (predictable and supportable, permanently forecloses intraday use cases from the core)
Reference Architectures¶
Core banking vendors — public sites and developer portals. Detailed platform documentation for all of these is behind customer login; the developer portals below expose primarily payments, digital, and integration APIs rather than bulk core-data extraction.
- Fiserv: fiserv.com — account processing platforms including Premier, Signature, DNA, Precision, Cleartouch and Portico; developer portal at developer.fiserv.com
- Finxact (Fiserv): finxact.com — cloud-native, API-first core; the contrast case to the legacy batch-extract model
- FIS: fisglobal.com — core platforms including Systematics, Horizon, IBS, Profile and Miser, plus the newer Modern Banking Platform; API marketplace at Code Connect
- Jack Henry: jackhenry.com — SilverLake System, CIF 20/20 and Core Director for banks, Symitar Episys for credit unions; developer portal at jackhenry.dev
- Temenos: temenos.com and product index — Transact (formerly T24) and Temenos Banking Cloud; developer portal at developer.temenos.com
- Finastra: finastra.com — Fusion core and lending products; open developer platform at FusionFabric.cloud
Legacy record formats and platform mechanics
- IBM COBOL for z/OS documentation —
USAGE,PICTURE,REDEFINES,OCCURS DEPENDING ON, and the packed/zoned decimal representations that fixed-width core extracts are built from - IBM i journal management — the journaling mechanism on IBM i, the log-based change source on Power-based cores where the vendor permits access
- Cobrix — open-source COBOL copybook parser and Spark data source for mainframe-format files; a working reference for copybook-driven parsing including
REDEFINESandOCCURS DEPENDING ON - Precisely — commercial mainframe and IBM i data-integration tooling, the usual alternative to building copybook parsing in-house
Relational reporting replicas — change detection and managed replication
Where the vendor supplies a relational replica (very often SQL Server), these are the mechanisms to evaluate before building a custom extractor. Confirm in every case that the vendor permits enabling the feature or attaching the tool to a database it owns.
- SQL Server change tracking — lightweight built-in change detection recording which rows changed; the natural basis for query-based incremental capture where log-based CDC is not available or not permitted
- SQL Server change data capture — log-based capture recording the changed values; check edition, version, and vendor support terms before assuming it can be enabled on a vendor-owned reporting database
- SQL Server transactional replication and readable secondary replicas — the topologies that let extraction read without loading the instance the core depends on
- Debezium SQL Server connector — streaming CDC from a SQL Server replica into an event log
- AWS Database Migration Service — managed full-load plus ongoing replication from a relational source into cloud targets
- Google Cloud Datastream — managed serverless change-data streaming from relational sources
- Microsoft Fabric mirroring — near-continuous replication of a relational source into a lake-format target with no pipeline to build; check source eligibility before designing to it
Change data capture and streaming
- Debezium Db2 connector — log-based CDC reference; confirm which Db2 variant a connector targets (Db2 for LUW, z/OS, and IBM i are different platforms with different change mechanisms) before assuming it fits a given core
- Confluent Hub — connector catalog for evaluating what exists against a given source before committing to a custom build
Payments and messaging context
- Nacha — ACH rules, return reason codes, and addenda formats needed to model payment-rail data rather than only the resulting core postings
See Also¶
patterns/regulated-financial-data-platform.md— the platform that consumes these extracts: encryption and key custody, retention and legal hold, lineage, segmentation, and evidence generationpatterns/data-pipeline.md— generic batch and streaming pipeline mechanics, orchestration, dead-letter handling, and idempotencygeneral/legal-hold.md— preservation architecture for the immutable raw landing zone and the deletion gate every retention process must consultgeneral/data.md— general data architecture and storage selectiongeneral/database-migration.md— migration mechanics applicable to a core conversion or ODS rehostgeneral/data-migration-tools.md— tooling options for bulk movement of legacy-format datapatterns/hybrid-cloud.md— connectivity and placement when the core or the ODS stays on-premises and the platform is in cloudpatterns/migration-coexistence.md— running old and new reporting in parallel during a platform or core transitioncompliance/glba.md— safeguards obligations over the customer information these extracts carrycompliance/pci-dss.md— cardholder-data scope when card processor feeds are ingestedcompliance/sox.md— controls over financial reporting where the platform feeds reported figurescompliance/ffiec.md— examination context for bank data and reporting programmesfailures/data.md— data-layer failure modesproviders/sqlserver/database.md— the engine a vendor-supplied relational reporting replica most often runs onproviders/confluent/kafka.md— event-log target when CDC from a replica is streamed rather than batchedproviders/databricks/data-platform.md— one managed lakehouse target for the landed extractsproviders/snowflake/data-platform.md— one managed warehouse target for the landed extracts