Data Governance¶

Scope¶

Cross-provider data governance: cataloging and discovery, lineage capture, data quality measurement, classification and sensitivity labelling, policy-based access control, retention and right-to-erasure mechanics on open table formats, and the evidence an auditor or examiner can actually be shown. Provider instances covered are Microsoft Purview (Data Map and Unified Catalog), Databricks Unity Catalog, AWS Glue Data Catalog with Lake Formation (and the DataZone/SageMaker Catalog layer above them), and Google Knowledge Catalog (formerly Dataplex Universal Catalog), plus the open standards — OpenLineage, Apache Atlas, DataHub, OpenMetadata, Egeria, and the Iceberg REST Catalog — that increasingly sit between them. This file covers governance as a control system. For the analytics platform decisions it governs, see general/data-analytics.md; for sensitivity taxonomy design, see general/data-classification.md; for organisational governance frameworks, see general/governance.md.

Product naming in this space churned heavily in 2025 and 2026 — Purview split into classic and Unified Catalog, Dataplex became Knowledge Catalog, Google Data Catalog entered phased shutdown, and DataZone was rebranded into SageMaker Catalog without being retired. Status claims below were verified against first-party documentation on 2026-07-26 and should be re-verified before being relied on.

Checklist¶

Catalog and Discovery¶

[Critical] Is there a single authoritative catalog per data estate, and is it the one that actually enforces something? A catalog that only describes data becomes shelfware within two quarters; a catalog that also grants access, applies policy, or gates publication stays current because it has to. Unity Catalog and Lake Formation enforce; Purview Unified Catalog and Knowledge Catalog primarily describe and delegate enforcement to the underlying engines.
[Critical] Is the catalog's coverage of the actual source estate verified rather than assumed? Every catalog product markets breadth and every one has gaps that matter. Check the specific sources in use against the vendor's supported-source matrix — including whether each source yields metadata only, metadata plus classification, or metadata plus lineage, because these differ per source within the same product.
[Critical] Is the namespace model chosen before the first assets are registered, given how expensive it is to change? Unity Catalog uses a strict three-level catalog.schema.object namespace across tables, views, volumes, functions, models, and services, with the catalog as the natural isolation boundary. Purview Unified Catalog organises around governance domains, data products, glossary terms, and critical data elements. Knowledge Catalog uses entries, entry groups, entry types, aspects, and aspect types. These are not interchangeable shapes and migrating between them is a re-modelling exercise, not an export.
[Recommended] For Microsoft estates, is the split between Purview Data Map and Purview Unified Catalog understood, and is the classic experience's status stated accurately? Data Map is the scanning and metadata foundation and is not deprecated — it underpins Unified Catalog. The classic Purview Data Catalog, Data Health Insights, and Purview Workflow are documented as "no longer taking on new customers… now in customer support mode." No retirement date has been announced; the Microsoft lifecycle entry for Purview reads "In Support." Plan a migration to the Unified Catalog experience, but do not plan it against a date that does not exist.
[Recommended] For Google estates, is the current product name and the Data Catalog shutdown both accounted for? The product is now Knowledge Catalog (renamed from Dataplex Universal Catalog; API, client library, CLI, and IAM names are unchanged). Google Data Catalog is deprecated and began a phased shutdown on 1 June 2026 — but the deprecation is selective: "methods that are related to policy tags and policy tag taxonomies are not deprecated," and BigQuery column-level security still depends on them. Migration mapping is tag templates → aspect types and tags → aspects.
[Recommended] For AWS estates, is the division of labour between Glue Data Catalog and Lake Formation clear? Glue Data Catalog is "a centralized repository that stores metadata… acts as an index to the location, schema, and runtime metrics of your data sources," populated by crawlers. Lake Formation provides "its own permissions model that augments the IAM permissions model" for fine-grained access to that data and metadata. Amazon DataZone still exists as a standalone service and is actively maintained; Amazon SageMaker Catalog is "built on Amazon DataZone" — an additive rebrand of the experience layer, not a retirement.
[Optional] Is an open metadata standard adopted where the estate spans more than one provider catalog? OpenLineage (LF AI & Data, graduated) is the interchange format with the widest emitter support — Airflow, Spark, Flink, Hive, dbt, Great Expectations, Presto, and Trino among them — and Knowledge Catalog can import OpenLineage events directly. Apache Atlas remains active (2.5.0, April 2026) and is the API surface Purview Data Map exposes (Atlas 2.2). DataHub, OpenMetadata, and Egeria are the credible open-source catalog platforms.

Lineage¶

[Critical] Is lineage capture automatic at the engine level rather than dependent on developers registering it? Hand-maintained lineage is wrong within weeks. Unity Catalog "captures lineage automatically for queries run on Databricks, down to the column level, and aggregates it across all workspaces attached to the metastore." Purview derives lineage from pipeline execution in ADF, Synapse, Fabric, Power BI, Databricks, and Azure ML. Knowledge Catalog lineage is GA with automatic capture from BigQuery, Dataflow, Cloud Data Fusion, managed Airflow and Spark, and Vertex AI.
[Critical] Are the documented lineage gaps enumerated, since a lineage graph with silent holes is more dangerous than none? Purview produces no lineage for Amazon Redshift, Amazon RDS, MongoDB, SAP BW, SAP HANA, Qlik Sense, Salesforce, Tableau, Dataverse, and HDFS, and only "limited" lineage for Blob Storage, ADLS Gen2, Azure Files, and Amazon S3 — captured solely through their participation in an ADF or Synapse pipeline. Unity Catalog does not capture column lineage when a source or target is referenced as a path (delta.\s3://...``) or when a UDF obscures the mapping.
[Critical] Is lineage retention long enough to satisfy the impact-analysis and audit window it is being relied on for? Unity Catalog retains lineage indefinitely in Catalog Explorer but the lineage system tables hold a rolling one-year window — a query written against system tables silently loses history the UI still shows. Confirm the equivalent retention for whichever catalog is in use before citing lineage as an audit control.
[Recommended] Is lineage actually used in change management — as the impact-analysis step before a schema change ships — rather than only produced? The value of lineage is realised at the moment someone is about to break a downstream report, not in the diagram.
[Optional] Where lineage must span provider boundaries, is OpenLineage used as the interchange rather than reconciling two vendor graphs by hand? Note that Microsoft Purview is not an OpenLineage consumer — its interoperability story is the Atlas 2.2 API.

Classification and Sensitivity Labelling¶

[Critical] Is classification automated at scan time and treated as the input to policy rather than as documentation? Purview Data Map ships more than 200 built-in system classifications; Lake Formation and Knowledge Catalog rely on tag-based schemes the organisation defines. Whichever is used, the test is whether a classification result changes what a policy does — if not, it is metadata theatre.
[Critical] Is the distinction between catalog classifications and enterprise sensitivity labels understood where both exist? In the Microsoft stack these are different constructs with different scope: Data Map system classifications (200+, catalog-scoped) are not the same thing as Microsoft 365 sensitive information types (no total count is published), and applying a sensitivity label in the Data Map is metadata only — the label is not applied to the file, and labelling in Data Map is still preview and requires an M365 licence in the same Entra tenant.
[Critical] Is label or tag inheritance verified in the direction it will actually be relied on? Inheritance is rarely symmetric. In Fabric and Power BI, downstream inheritance flows Power BI → Power BI, Fabric → Fabric, and Fabric → Power BI, but not Power BI item → Fabric item; autogenerated items take their label from their immediate parent and "don't inherit the label from items further upstream." In Lake Formation, LF-Tags inherit from database to table and table to column and can be overridden at each level. Assuming inheritance that does not exist is how sensitive derivatives end up unlabelled.
[Recommended] Are tag-based access controls preferred over per-resource grants at any meaningful scale? AWS documents the arithmetic directly: named-resource grants scale as principals × resources, while LF tag-based access control scales as principals + resources. Databricks implements the same idea as attribute-based access control driven by governed tags, with row filter and column mask policies attached at catalog, schema, or table level and evaluated dynamically. Note AWS's own warning that "IAM tags are not the same as LF-Tags. These tags are not interchangeable."
[Recommended] Are the tag scale limits checked against the intended taxonomy? Unity Catalog allows 50 tags per securable, 1,000 column tags per table, and 256 characters per key and value; governed tags are capped at 1,000 per account with 500 allowed values each. ABAC policies are limited to 10,000 per metastore and 50 per table.
[Optional] Is mandatory labelling verified as actually enforced on the item types it is claimed to cover? In Fabric, mandatory labelling is "currently fully supported for Power BI items only," and for lakehouses, warehouses, and pipelines "mandatory labeling logic isn't enforced" — the user can save the item without a label. A control that is configured but not enforced is worse than an acknowledged gap.

Data Quality¶

[Critical] Are quality checks positioned as gates that block promotion rather than alerts that fire after bad data has already landed in a dashboard? Post-load alerting requires manual remediation and is routinely ignored; a failing expectation that quarantines the batch does not need anyone's attention to be effective.
[Critical] Are quality rules expressed against named dimensions so that results are comparable across domains? Purview Data Quality uses six — completeness, consistency, conformity, accuracy, freshness, and uniqueness — and caps rules at 200 per asset. Google's auto data quality (GA) and data profiling (GA in standard mode) provide the equivalent on BigQuery and Iceberg REST Catalog tables. Great Expectations and Soda are the platform-neutral options.
[Recommended] Is data quality scored at a level a business owner can be accountable for — data product, governance domain, or critical data element — rather than only per table? Purview Unified Catalog scores at rule, asset, data product, and governance domain level for exactly this reason.
[Recommended] Are freshness and volume checks in place alongside correctness checks? The most common silent failure in an analytics platform is not wrong data but absent data — a pipeline that stopped and a dashboard that kept rendering yesterday's numbers.
[Optional] Are data contracts defined between producers and consumers where domain teams publish independently, with schema and SLA validated in CI rather than discovered at runtime?

Access Control and Policy Enforcement¶

[Critical] Is it established which engines actually enforce each policy, rather than assuming a policy defined in the catalog binds everywhere? This is the single most common governance failure across all four providers. Lake Formation's fine-grained filtering support is not uniform: Athena SQL, Redshift Spectrum, EMR-on-EC2 Spark, and EMR Serverless Spark get column plus row and cell filtering; EMR Hive gets column filtering but not row or cell; Athena Spark, EMR Serverless Hive, and EMR on EKS get none; and Glue ETL requires Glue 5.0 or later. In Fabric, T-SQL row- and column-level security applies only through the SQL endpoint and is invisible to Spark. Enumerate the engines, then verify each one.
[Critical] Is it understood that fine-grained filtering typically governs reads only? AWS states it plainly: "filters apply only to read operations." A row filter is not a write control, and a governance design that relies on one to prevent writes has no control at all.
[Critical] Are the administrative roles that bypass data policy identified and treated as privileged access? In Fabric, workspace Admin, Member, and Contributor roles carry Write to OneLake and thereby override OneLake security Read permissions, and semantic-model RLS is not enforced for anyone with Write on the model. Equivalent bypasses exist in every platform. The governance model is only as strong as the population that can step around it.
[Recommended] Are grants made to groups synchronised from the identity provider rather than to individual users? Direct user grants produce permission sprawl that becomes unauditable within a year and cannot be reasoned about during an access review. Verify how the platform resolves nested groups — some do not: Fabric documents that across a producer/consumer boundary, identities "must be mapped exactly 1:1… nested or effective group membership is not resolved."
[Recommended] Is cross-account or cross-tenant sharing done through the platform's native mechanism rather than by copying data? Lake Formation shares cross-account through AWS RAM, with five CROSS_ACCOUNT_VERSION levels where v3 brought tag-based access control onto RAM, v4 is required for hybrid access mode and federated catalogs, and v5 removes per-resource-type association limits and cannot be downgraded. Delta Sharing, Fabric external data sharing, and BigQuery Analytics Hub are the equivalents elsewhere.
[Recommended] Are policy propagation delays known and excluded from urgent-revocation runbooks? Fabric OneLake security role definitions propagate in about five minutes, group membership changes take about an hour, and engine caches can add another. No catalog-layer revocation should be the sole control for an urgent removal — disable the account at the identity provider.
[Optional] Where an open catalog API is exposed for interoperability, are its policy carve-outs known? Databricks documents that tables carrying row filters or column masks cannot be accessed through the Unity Catalog Iceberg REST endpoint — the interop path and the fine-grained policy path are mutually exclusive for those tables. Fabric's OneLake Iceberg and /delta catalog endpoints are read-only.

Retention, Deletion, and Right to Erasure¶

[Critical] Is it understood that a DELETE on a modern lakehouse table does not remove the data from storage? Delta Lake deletion vectors mark rows deleted in metadata and apply the vector at read time; Databricks states directly that "modified data might still exist in a table's old data files after a purge operation." Apache Iceberg has the same shape, writing separate delete files — position deletes (encoded as deletion vectors in V3+) and equality deletes (V2+). A GDPR or CCPA erasure obligation is not discharged by a DELETE statement.
[Critical] Is the physical purge procedure documented and exercised rather than assumed? The Delta two-step is REORG TABLE … APPLY (PURGE), which "rewrites all data files containing records with modifications recorded using deletion vectors," followed by VACUUM with the retention threshold set to the purge completion timestamp. For Iceberg, expire_snapshots is the operation that actually removes files — and "data files are not deleted until they are no longer referenced by a snapshot that may be used for time travel or rollback," with snapshots referenced by branches or tags retained regardless.
[Critical] Is time travel retention reconciled against the erasure obligation, since the two are in direct conflict? Every day of time-travel history is a day during which deleted personal data remains recoverable. Pick the retention window deliberately, document the reasoning, and make sure the erasure runbook shortens it for the affected tables rather than working around it.
[Recommended] Is erasure propagated from the earliest layer rather than applied table by table? Databricks documents the pattern: delete from the bronze layer first via a scheduled job reading a deletion-requests table, then let materialized views cascade and configure streaming tables with skipChangeCommits. Deleting only from gold leaves the personal data in bronze and silver.
[Recommended] Is deletion preferred over masking where erasure is the obligation? Databricks' guidance is explicit that "complete deletion is preferable to obfuscation" because masking cannot eliminate re-identification risk. Masking is an access control, not an erasure mechanism.
[Optional] Are downstream copies — extracts, BI caches, semantic model imports, mirrored replicas, shortcut caches — inventoried as part of the erasure scope? Every copy is a place the record survives, and copies created for convenience are the ones nobody remembers.

Operating Model and Evidence¶

Why This Matters¶

Governance failures compound rather than accumulate. Without a catalog, analysts cannot find data, so they make their own extracts, and every extract is an ungoverned copy that inherits no policy and appears in no lineage graph. Without lineage, a schema change breaks downstream reports in ways that take days to trace, so teams stop making changes, and the platform ossifies. Without quality gates, decision-makers lose confidence and revert to spreadsheets — at which point the organisation is paying for a data platform that it has routed around. Each failure is individually recoverable; together they produce a platform nobody trusts, everybody works around, and finance keeps funding.

The specific technical trap that catches the most organisations is assuming that a policy defined in the catalog is enforced by every engine that can reach the data. It usually is not, and the gaps are documented but not prominent. Lake Formation's row and cell filtering does not apply to EMR Hive, Athena Spark, or EMR on EKS at all. Fabric's T-SQL row-level security applies through the SQL endpoint and is invisible to Spark reading the same Delta files. Warehouse SQL security in Fabric does not project into OneLake, so a shortcut reader sees what the warehouse denies. Unity Catalog tables carrying row filters or column masks cannot be read through its own Iceberg REST endpoint. In each case the control is real and the coverage is partial, and the only reliable method is to enumerate every engine with a path to the storage and verify each one individually. Anything else produces a control matrix that is confidently wrong.

Right to erasure on a lakehouse is the second trap, and it is a legal one rather than an operational one. Deletion vectors and Iceberg delete files were designed to make deletes cheap, and they achieve that by not removing data — a DELETE marks rows as absent from query results while the underlying files remain intact and time-travel-readable. An organisation that runs DELETE FROM customers WHERE id = ? in response to an erasure request, records it as complete, and retains 30 days of time travel has satisfied nothing and has created a record showing it believed it had. The correct procedure is documented by both vendors and involves a rewrite step followed by an expiry step, and it needs to be exercised before it is needed rather than reconstructed under time pressure.

The organisational failure is more common than the technical one, and it has a predictable shape. A platform gets built first because that is what can be funded and demonstrated, and governance is scheduled for "once the platform is finished." But governance decisions are not deferred by that choice — they are made anyway, implicitly, by whoever builds each pipeline. The engineer who wires up the first finance dataset decides, by default, that their table is the source of truth. The analyst who publishes the first revenue report decides, by default, that they own it. Nobody records either decision, and both become load-bearing. When the governance programme finally arrives, its job is no longer to make decisions but to renegotiate dozens of undocumented ones with teams that are now dependent on them — which is why these programmes routinely take longer than the platform build they were meant to follow. The cheap version of governance is a small number of explicit decisions made early: who owns each domain, who may certify a dataset, and what certification obliges. The expensive version is the same decisions made two years later against incumbents.

The related failure is conflating ownership with stewardship. Accountability for data being fit for purpose and the daily work of curating it are different jobs requiring different authority and different amounts of time. Assign both to a senior owner and the curation never happens, because they have no capacity. Assign both to an analyst and the escalations never land, because they cannot compel a source-system team to fix anything. The pattern that works is an owner with authority and an escalation path, plus a steward with capacity and a defined queue — and the queue matters as much as the roles, because unadjudicated quality exceptions with no SLA are indistinguishable from no quality process at all. Certification is where these come together: it is only meaningful if the certifier is authorised rather than self-appointed, if the owner has accepted a standing maintenance obligation, and if consumers learn when a certified dataset changes. A platform where anyone can mark their own dataset certified has a badge, not a control.

Finally, product churn in this space is itself a governance risk. Between 2024 and 2026, Purview split into classic and Unified Catalog experiences with the classic one closed to new customers on no published timeline, Google renamed Dataplex twice and put Data Catalog into a phased shutdown while carving policy tags out of the deprecation, and AWS rebranded DataZone into SageMaker Catalog without retiring DataZone. An architecture document written against 2024 product names will contain claims that are now wrong in ways that are hard to spot, because the products still exist under different names with different boundaries. Name the version, date the claim, and re-verify before a decision depends on it.

Common Decisions (ADR Triggers)¶

Provider-native vs open-source vs commercial catalog -- provider-native (Purview, Unity Catalog, Glue plus Lake Formation, Knowledge Catalog) for the deepest enforcement integration and lowest operational overhead within one cloud vs open-source (DataHub, OpenMetadata, Egeria, Atlas) for cross-platform coverage without licensing cost, accepting the self-hosting burden and generally weaker enforcement vs commercial (Collibra, Alation, Atlan) for business glossary, stewardship workflow, and compliance automation at significant licence cost
Enforcing catalog vs describing catalog -- an enforcing catalog stays current because access depends on it, but couples governance to one platform's engines vs a describing catalog spans more platforms but decays unless a process forces updates. Most mature estates end up with an enforcing catalog per platform and a describing catalog above them, and must then decide which one is authoritative for audit
Named-resource grants vs tag-based access control -- named grants for small, stable, high-sensitivity resource sets where explicitness matters vs tag-based (LF-Tags, Unity Catalog governed tags with ABAC, sensitivity labels driving policy) once the principal × resource product becomes unmanageable; the crossover arrives earlier than teams expect
Where fine-grained policy lives -- in the catalog layer (consistent across engines that honour it, but engine coverage is uneven) vs in the query engine (reliably enforced for that engine, invisible to every other one) vs in the storage layer through separate physical datasets (crude, engine-independent, and the only option when engine coverage genuinely cannot be relied on)
Purview classic vs Unified Catalog migration timing -- migrate now to align with where investment is going, accepting a re-modelling exercise into governance domains and data products vs defer, which remains defensible because no retirement date is published and the classic experience is in customer-support mode rather than end-of-life
Purview billing model -- pay-as-you-go on governed assets plus processing units, where cost scales with what is actually governed rather than what is scanned vs classic Elastic Data Map capacity units plus scanning charges; note that Data Map and scan charges do not apply once a customer is on the Unified Catalog billing model
Catalog interoperability layer -- Iceberg REST Catalog as the emerging cross-engine standard, implemented by AWS Glue, S3 Tables, Databricks Unity Catalog, Google's Lakehouse runtime catalog, and Apache Polaris vs vendor-specific APIs, which are richer but bind the estate to one control plane. Note that fine-grained policy and REST-catalog access are mutually exclusive on Databricks
Lineage capture strategy -- rely on engine-native automatic capture, which is accurate and free but has documented per-source gaps vs emit OpenLineage from orchestration to cover sources the platform misses vs accept the gaps explicitly and document which parts of the estate have no lineage, which is a legitimate choice and far better than an incomplete graph presented as complete
Erasure mechanism -- logical delete with deletion vectors or Iceberg delete files, which is fast and cheap but leaves data recoverable vs scheduled physical purge (REORG … APPLY (PURGE) plus VACUUM, or expire_snapshots) on a cadence that satisfies the obligation vs shortened time-travel retention estate-wide, which discharges the obligation most simply and costs the recovery capability that time travel provides
Data mesh vs centralised governance -- federated domain ownership with central interoperability standards once the central team is a measurable bottleneck and domains can genuinely staff ownership vs a central data team while the estate is small enough for one group to hold, since premature federation adds organisational overhead without proportional benefit. Federating to domains that cannot staff ownership produces drift with no accountability, which is worse than an acknowledged bottleneck
Governance sequencing: alongside the platform vs after it -- define ownership, certification authority, and quality accountability during the platform build, accepting that it slows the first delivery vs defer governance to a later programme, accepting that the decisions get made implicitly by pipeline authors in the meantime and must later be renegotiated with dependent teams. Deferring the tooling is usually fine; deferring the decisions is what makes the programme expensive
Ownership and stewardship: combined vs separated -- one accountable owner who also curates, viable only for a small domain with a hands-on owner vs separated owner and steward, where the owner holds authority and the escalation path and the steward holds capacity and the exception queue. Combining them at scale fails in one of two predictable ways: no capacity to curate, or no authority to escalate
Certification authority: self-service vs designated reviewers -- allow teams to endorse their own datasets, which scales and asserts little vs restrict certification to admin-designated reviewers, optionally per domain, which asserts something real and creates a review bottleneck that must be staffed. A tiered scheme — self-service promotion plus restricted certification plus a separate authoritative-source designation — is the usual resolution
Decertification notification -- rely on catalog badges, which consumers see only when they browse and which make decertification silent vs actively notify subscribed consumers on certification change, which requires knowing who the consumers are and is where lineage earns its cost
Audit evidence retention -- platform-native audit retention where the obligation fits inside it, accepting the licence dependency on the record-generating identity vs SIEM export where the obligation is longer or where correlation with non-data-platform events is required

Reference Architectures¶

Microsoft Purview overview -- the solution set and how governance, protection, and compliance relate
Microsoft Purview data governance overview -- Data Map and Unified Catalog as the two governance solutions
Purview legacy governance solutions -- the classic experience's customer-support-mode status
Microsoft Purview lifecycle -- the authoritative support-status record
Purview data governance billing -- governed assets and Data Governance Processing Units
Purview Data Map supported sources -- the per-source matrix including which sources yield lineage
Purview Data Map classifications -- system classifications and custom classification rules
Sensitivity labels in the Purview Data Map -- label application scope and the metadata-only caveat
Purview Unified Catalog data quality -- the six quality dimensions, rule types, and scoring levels
Purview Unified Catalog governance domains -- the domain and data product ownership model
Purview Data Map Atlas 2.2 API -- the Apache Atlas-compatible interoperability surface
Manage audit log retention policies -- default retention, licence dependencies, and the 10-year add-on
What is Unity Catalog? -- the three-level namespace and governed object types
Unity Catalog data lineage -- automatic column-level capture, requirements, and the documented gaps
Attribute-based access control in Unity Catalog -- governed tags driving row filters and column masks
Row filters and column masks -- fine-grained policy definition and evaluation
Tags on Databricks objects -- tag scope, limits, and governed tag policies
Unity Catalog Iceberg REST endpoint -- external engine access and the row-filter/column-mask exclusion
Unity Catalog open source -- the Apache 2.0 project, its API, and Iceberg REST Catalog support
AWS Glue Data Catalog and crawlers -- metadata repository population and structure
What is AWS Lake Formation? -- the permissions model that augments IAM
Lake Formation tag-based access control -- LF-Tags, inheritance, the scaling arithmetic, and the IAM-tag warning
Lake Formation data filtering -- column, row, and cell filters and the read-only constraint
Lake Formation integration with other services -- the per-engine enforcement matrix
Lake Formation cross-account access -- AWS RAM sharing and cross-account version levels
What is Amazon DataZone? -- the still-current standalone business data catalog
Amazon SageMaker Unified Studio -- the experience layer built on DataZone
Knowledge Catalog introduction -- the current Google catalog product (formerly Dataplex Universal Catalog)
Knowledge Catalog overview -- entries, entry groups, aspects, and aspect types
Google Data Catalog deprecations -- the June 2026 phased shutdown and the policy-tag carve-out
Transition to Knowledge Catalog -- tag templates to aspect types migration mapping
Dataplex data lineage -- automatic sources, GA status, and OpenLineage import
Auto data quality overview -- rule-based quality scans over BigQuery and Iceberg tables
BigQuery column-level security -- policy tags and the Data Catalog dependency
BigQuery row-level security -- row access policies and their documented limitations
BigQuery dynamic data masking -- masking built on column-level access control
OpenLineage documentation -- the lineage interchange specification and facet model
OpenLineage integrations -- which tools emit OpenLineage events
Apache Atlas -- open metadata and governance framework, and the API Purview Data Map exposes
DataHub -- open-source metadata platform for discovery, observability, and governance
OpenMetadata -- open-source metadata management with catalog, lineage, and quality
Egeria -- open metadata standards and the cross-repository metadata highway
Apache Iceberg REST Catalog specification -- the emerging cross-engine catalog interoperability standard
Apache Polaris -- open-source Iceberg REST catalog implementation
Apache Iceberg table specification -- position and equality deletes, snapshots, and metadata layout
Apache Iceberg maintenance -- expire_snapshots and physical file removal
Delta Lake deletion vectors -- how deletes are recorded and why data persists
REORG TABLE -- the purge step that physically rewrites affected files
GDPR and CCPA compliance with Delta Lake -- the bronze-first deletion pattern and the deletion-over-obfuscation guidance
Endorsement overview (Fabric and Power BI) -- Promoted, Certified, and Master data tiers, who may grant each, and per-domain certifier delegation
Great Expectations -- platform-neutral data quality validation and profiling
Soda -- SQL-based data quality monitoring and alerting