AWS Lake Formation¶

Scope¶

AWS Lake Formation is the fine-grained authorization layer over the AWS Glue Data Catalog and the S3 data it describes. It grants database-, table-, column-, row-, and cell-level permissions to IAM principals, and vends temporary scoped credentials to integrated analytics services (Athena, Redshift Spectrum, EMR, Glue ETL, QuickSight) so those services can read S3 data the caller has no direct S3 permission to read.

This file covers the permission model and its resource types, LF-Tag-based access control, what "registering a location" actually does and how credential vending works, the interaction with IAM and S3 bucket policies (including the enforcement gap that makes Lake Formation ineffective if principals retain direct S3 access), the legacy IAMAllowedPrincipals setting and hybrid access mode, row- and cell-level data filters and which engines enforce them, cross-account sharing and resource links, and the operational limits and cost model.

Lake Formation governance was previously documented only as a bullet inside providers/aws/s3.md's Data Lake Foundation reference architecture. For the catalog itself see providers/aws/glue.md; for the main query surface see providers/aws/athena.md; for the table formats underneath see general/open-table-formats.md.

Lake Formation's feature surface (cross-account versions, hybrid access mode, catalog federation, the SageMaker Lakehouse framing) has changed repeatedly. Verify version-dependent details against current AWS documentation before committing to a design.

Checklist¶

Foundations: Catalog, Registered Locations, Credential Vending¶

[Critical] Is it understood that Lake Formation governs the Glue Data Catalog, not S3 directly? Lake Formation has no data-plane presence on S3 buckets. It authorizes catalog operations and vends temporary credentials to integrated analytics services, which are then responsible for applying the restrictions. A principal reading s3://bucket/prefix/ with the S3 API is not subject to Lake Formation at all.
[Critical] Have the S3 locations holding lake data been registered with Lake Formation, and is the registration role's scope understood? Registering a location associates an IAM role (either a customer role or the Lake Formation service-linked role) that holds the actual S3 read/write permissions. Lake Formation then assumes that role and vends short-lived credentials to the querying service. Unregistered locations fall back to the caller's own IAM permissions -- Lake Formation grants against them do nothing.
[Critical] Have direct S3 permissions been removed from the analyst and application principals whose access is meant to be governed? This is the single most consequential Lake Formation deployment step and the one most often skipped. If a data scientist's role still carries s3:GetObject on the lake bucket, column masking and row filters are advisory: the governed path enforces them and the direct path does not. Lake Formation is only as strong as the weakest path to the bytes.
[Critical] Do the querying principals hold the IAM permissions Lake Formation itself requires -- notably lakeformation:GetDataAccess, which must be granted with Resource: "*" because it is not resource-scoped? Lake Formation permissions and IAM permissions are both required for the governed path: IAM authorizes the API call, Lake Formation authorizes the data. Neither substitutes for the other.
[Critical] Are data lake administrators treated as a privileged, small, audited group? A data lake administrator can grant any permission on any catalog resource, including to themselves. The default quota is on the order of a few dozen administrators per account, which is far more than any organization should use. Treat this like an account-level break-glass role.
[Recommended] Is the S3 bucket policy still doing its job as a second control -- restricting access to the registration role and the specific principals that legitimately need direct access (ingestion writers, backup tooling)? Bucket policies and Lake Formation are complementary: the bucket policy bounds who can reach the bytes at all; Lake Formation shapes what governed queries return.
[Recommended] Are Requester Pays buckets and cross-region reads accounted for before registering a location? Registered locations introduce a role whose data-transfer and request charges accrue against the registering account rather than the caller.
[Optional] Is CloudTrail capturing Lake Formation API calls (GetDataAccess, GrantPermissions, RevokePermissions) and routing them to the security data lake? GetDataAccess events are the audit record of who actually read governed data through which service.

The Permission Model¶

[Critical] Is the permission vocabulary understood, and is SUPER avoided outside of administrative roles? Lake Formation permissions are DESCRIBE, SELECT, INSERT, DELETE, ALTER, DROP, CREATE_DATABASE, CREATE_TABLE, DATA_LOCATION_ACCESS, and SUPER. SUPER is every permission on the resource and is the Lake Formation equivalent of a wildcard IAM policy -- convenient during a proof of concept and indefensible in production.
[Critical] Is DATA_LOCATION_ACCESS understood as a catalog permission, not a data permission? It authorizes a principal to create catalog resources that point at a registered S3 location. It does not grant the ability to read or write the underlying objects. Teams routinely grant it expecting data access and then debug the wrong layer.
[Critical] Is every grant issued to a group or role rather than to individual users? Lake Formation grants are per principal per resource, and a per-user grant sprawl is effectively impossible to audit or revoke at scale. Grant to IAM roles backed by IdP groups; the grant graph should be a small number of role-to-resource edges.
[Recommended] Is the grantable flag (WITH GRANT OPTION) used sparingly and only for delegated administration of a bounded domain? Grant option turns a data consumer into a data grantor and makes the effective permission set much harder to reason about. Note also that column-filtered SELECT cannot be granted with grant option.
[Recommended] Are DESCRIBE and SELECT grants distinguished deliberately? DESCRIBE reveals metadata (a table exists, its columns) without data access, which is the correct grant for a discovery experience where analysts browse a catalog and request access to what they find.
[Recommended] Are INSERT/DELETE/ALTER/DROP reserved for pipeline roles rather than analyst roles, so that the write path and the read path have genuinely different blast radii? A single role that both writes silver tables and serves ad-hoc BI queries defeats the layering in patterns/lakehouse-medallion.md.
[Optional] Is there a periodic review of effective permissions per principal, using the Lake Formation permissions APIs, rather than relying on the console view of a single resource at a time? Grants accumulate and nothing expires them.

`IAMAllowedPrincipals` and Hybrid Access Mode¶

[Critical] Has the legacy IAMAllowedPrincipals grant been removed from the databases and tables that are meant to be governed, and has the account default been changed? By default, new databases and tables can be created with a grant to the virtual IAMAllowedPrincipals group, which means "fall back to IAM-only access control." While that grant is present, Lake Formation enforces nothing on that resource -- the fine-grained permissions are recorded and ignored. This is by far the most common reason a Lake Formation deployment appears configured but is not enforcing.
[Critical] Is there a verification step -- a test principal with a Lake Formation deny and no direct S3 access, running a real query -- rather than an assumption that configuration equals enforcement? Both the IAMAllowedPrincipals fallback and residual direct S3 permissions fail open, silently. Enforcement must be demonstrated, not configured.
[Recommended] Where a lake must be migrated incrementally, is hybrid access mode used instead of a flag-day cutover? Hybrid access mode lets a single catalog resource be governed by Lake Formation for the principals you have onboarded while existing IAM-based access continues to work for everyone else -- opt-in per principal, per resource. It converts a risky big-bang migration into a per-team rollout.
[Recommended] Is hybrid access mode treated as a migration state with an end date, not a destination? A resource left permanently in hybrid mode has an ungoverned population whose size nobody tracks. Record which principals remain on the IAM path and burn the list down.
[Optional] Are new databases created with the "use only IAM access control" default turned off at the account level, so that newly created resources are governed from creation rather than needing remediation later?

Tag-Based Access Control (LF-TBAC)¶

[Critical] For any lake beyond a handful of tables, is LF-Tag-based access control used in preference to named-resource grants? Named-resource grants scale as (principals x resources) and require a grant action every time a table is created. LF-Tags are key/value pairs attached to catalogs, databases, tables, and columns; policies grant on tag expressions, so a newly created table inherits access the moment it is tagged. This is the difference between governance that keeps up with the lake and governance that lags it permanently.
[Critical] Is the LF-Tag ontology designed up front and kept small? A workable starting set is a sensitivity tag (classification = public | internal | confidential | restricted), a domain tag (domain = finance | sales | ops), and a layer tag (layer = bronze | silver | gold). Tag proliferation reproduces the named-resource problem with extra indirection.
[Critical] Is tag inheritance understood? Tags assigned at the database level are inherited by tables, and table-level tags by columns, with the more specific assignment overriding. This is what makes the model economical -- and it is also how an over-broad database-level tag silently grants access to every table beneath it.
[Recommended] Is tag assignment automated in the table-creation path (Glue job, IaC module, or a post-crawler Lambda) rather than performed manually? An untagged table under an LF-TBAC model is invisible to everyone, which surfaces as "the pipeline ran but nobody can see the data."
[Recommended] Are the documented quotas checked against the intended design before building on them? The defaults are on the order of a thousand LF-Tags per account, a thousand values per tag, and a few dozen tag expressions per principal per resource type. They are adjustable, but a design that needs a quota increase on day one is usually over-modelled.
[Recommended] Is a classification source of truth defined so that LF-Tags are derived from it rather than being the record itself? See general/data-classification.md. Tags applied ad hoc by whoever created the table drift from the organization's actual classification scheme within a quarter.
[Optional] Are column-level LF-Tags used for PII marking so that a single policy ("nobody outside the privacy group sees classification=restricted columns") covers every table without enumerating columns per table?

Row, Column, and Cell-Level Filtering¶

[Critical] Is it understood that Lake Formation delegates enforcement to the querying engine? Lake Formation computes the authorized column set and row predicate and passes it, with the vended credentials, to the integrated analytics service -- which is responsible for applying it. This is why capability differs by engine, why a non-integrated engine cannot be governed by Lake Formation at all, and why "Lake Formation enforces row-level security" is only true for the engines that implement it.
[Critical] Have data filters been validated on each engine that will use them, rather than tested once in Athena and assumed everywhere? Data filters combine a column include/exclude specification with a row-filter expression, and support varies: some engines support column filtering but not row filtering, and some Glue ETL features are unavailable against cell-filtered tables.
[Critical] Are the documented Glue ETL restrictions accounted for before designing pipelines against cell-filtered tables? Reading a table subject to cell-level filters through Glue ETL disables several capabilities that pipelines commonly depend on -- job bookmarks, bounded execution, pushdown predicates, server-side catalog partition predicates, and catalog schema updates from the job. Pipelines should generally read the unfiltered table under a pipeline role and produce a governed downstream table, rather than reading through the filter.
[Critical] Are partition key columns excluded from column-filtering designs? Partition keys cannot be hidden by a column filter -- the engine needs them to prune. If a partition key is itself sensitive (a customer identifier used as a partition column), the fix is a layout change, not a filter.
[Recommended] Are row filters expressed against columns that are stable and indexed by the layout, rather than against a derived expression that forces a full scan? A row filter is applied as a predicate on every governed query; an expensive one taxes the entire user population.
[Recommended] Where the requirement is masking rather than exclusion, has the alternative been considered -- producing a masked or tokenized column in the silver layer -- instead of relying on column exclusion? Excluding a column makes it invisible; many requirements actually want a consistent surrogate that still supports joins and counts.
[Recommended] Are open-table-format maintenance operations checked against Lake Formation coverage? Lake Formation's permission model does not extend uniformly to table-maintenance DML on Iceberg, Hudi, and Delta tables (OPTIMIZE, VACUUM, MERGE), and Athena imposes additional restrictions on DDL against Lake-Formation-registered Iceberg tables. Maintenance jobs generally need their own privileged role outside the governed path.
[Optional] Are filters version-controlled as IaC (CloudFormation, CDK, or Terraform) rather than created in the console? A row-filter expression is business logic; it belongs under review like any other.

[Critical] Is the account's Lake Formation cross-account version known, and is a version upgrade treated as a deliberate change? The cross-account sharing implementation has been revised several times, with later versions adding tag-based cross-account grants, direct-to-principal sharing, support for federated and hybrid-mode resources, and larger share scale. Capabilities differ materially by version and downgrades are not supported. Check the current version before designing a sharing model on a capability the account does not have.
[Critical] Is the sharing target chosen deliberately -- an entire account, a specific IAM principal in another account, or an AWS Organizations organization/OU? Sharing to an account requires the consumer's administrator to re-grant onward to their own principals, which is the right model for federated ownership. Sharing directly to a principal skips that step but generally cannot be re-shared by the recipient, which is the right model for a narrow, controlled hand-off.
[Critical] Does the producing account's grantor role carry the cross-account manager permissions AWS requires, and has the consumer side accepted the AWS RAM invitation where one is generated? Shares outside an AWS Organization arrive as RAM invitations that must be accepted; shares within an organization can be automatic depending on version and configuration. A share that appears to have been granted but never accepted is a common dead end.
[Critical] Have resource links been created in the consumer account? Shared databases and tables do not appear directly in the consumer's Athena or Redshift Spectrum experience -- a resource link is a local catalog object pointing at the shared resource, and permissions must be granted on both the link and the target. This is the most common "the share worked but I can't query it" failure.
[Recommended] Is it known which permissions cannot cross an account boundary? Notably, DROP and SUPER on a database are not grantable cross-account. Designs that assume the consumer can fully administer a shared database will fail late.
[Recommended] For multi-account lakes, is there a single producer account owning the catalog and registered locations, with consumer accounts holding only resource links? See providers/aws/multi-account.md. A catalog per consumer account is the standard way to lose track of where the authoritative table definition lives.
[Optional] Is Lake Formation sharing compared against the alternatives before being adopted -- S3 Access Points with bucket policies for coarse sharing, or a data-sharing product for external parties? Lake Formation cross-account sharing is the right answer when the consumer needs governed SQL over the same physical data; it is heavier than necessary for a periodic extract.

Catalog Substrate, Operations, and Cost¶

[Critical] Is the Glue Data Catalog treated as the single metadata substrate, with Lake Formation as the authorization layer over it, rather than as two separate systems? Every table Lake Formation governs is a Glue Data Catalog table; every crawler, ETL job, and Athena query resolves through the same catalog. Catalog design decisions (databases per layer, naming, partition strategy) therefore determine the shape of the permission model. See providers/aws/glue.md.
[Recommended] If the account uses multiple or federated catalogs (Redshift-managed catalogs, S3 Tables catalogs, catalogs federated to external metastores), has Lake Formation support for those catalog types been confirmed for the intended operations? Support for non-default catalog types has been added incrementally and is version-dependent.
[Recommended] Is the cost model understood -- Lake Formation permissions management itself is not separately charged, and the bill comes from the underlying services (Athena bytes scanned, Glue DPU-hours, S3 requests and storage)? Where the Lake Formation Storage API is used, it is charged on data scanned with a small per-request minimum. Budget the query engine, not the governance layer.
[Recommended] Are the registered-location quotas checked for large lakes -- the default limits on the number of registered S3 paths, the maximum path length, and the number of path segments are all real and all adjustable? A lake registered at thousands of individual prefixes rather than at a small number of bucket roots will hit them.
[Optional] If governed tables appear in an existing design or an older proposal, has their current status been checked before building on them? Governed tables were an early Lake Formation-specific table type with automatic compaction and ACID semantics; AWS's current guidance for those capabilities is Apache Iceberg with Glue table optimizers. Confirm the present state of the feature with AWS rather than assuming either that it is supported or that it is gone.
[Optional] Has AWS's current lakehouse framing (SageMaker Lakehouse and the unified studio experience over the Glue Data Catalog) been evaluated, given it is the direction AWS is documenting new analytics work against? This affects which console and which permission surface new users will encounter, more than it affects the underlying model.

Why This Matters¶

Lake Formation fails open, and it fails open quietly. Two independent mechanisms produce a deployment that looks governed and is not: the legacy IAMAllowedPrincipals grant, which tells Lake Formation to defer to IAM on that resource, and residual direct S3 permissions on the consuming roles, which bypass the catalog entirely. In both cases the console shows the column masks and row filters exactly as configured, audit evidence can be produced showing they exist, and queries return unfiltered data. Any Lake Formation implementation whose acceptance test is "the permissions are configured" rather than "a test principal is demonstrably denied" has not been tested.

The second structural point is that Lake Formation does not enforce anything itself. It computes an authorization decision and hands it, with temporary credentials, to an integrated analytics service that applies it. This has three consequences that shape architecture. Engine capability differs, so a filter that works in Athena may behave differently or be unsupported elsewhere. Non-integrated engines cannot be governed at all -- a self-managed Spark cluster or a third-party tool reading S3 directly is outside the model, no matter what the catalog says. And a query path that avoids the integrated services avoids the governance, which is why removing direct S3 access is not a hardening nicety but the precondition for the whole design.

Named-resource grants are where Lake Formation deployments go to die operationally. They scale as principals times resources and require a grant action for every new table. Six months into a lake, the grant graph has thousands of edges, nobody can answer "what can this role see," and new tables ship ungoverned because the grant step was forgotten. LF-Tag-based access control exists precisely to break this: tags on resources, policies on tag expressions, inheritance down the hierarchy, so a correctly tagged new table is governed the moment it is created. The cost of adopting LF-TBAC is designing a small tag ontology up front and automating tag assignment in the table-creation path. The cost of not adopting it compounds monthly.

The pipeline-versus-consumer distinction is worth designing explicitly. Cell-level filters restrict what Glue ETL can do -- job bookmarks, pushdown predicates, and bounded execution are among the capabilities that become unavailable against a filtered table -- which means pipelines that read through the filter lose exactly the features that make them incremental and cheap. The correct pattern is that pipeline roles read unfiltered under their own grants and produce a governed table, while human and BI consumers read the filtered view. Discovering this after building a medallion pipeline on filtered reads is an expensive rework.

Cross-account sharing carries the most incidental complexity in the service. The share must be granted, the invitation accepted, a resource link created in the consumer account, and permissions granted on both the link and the target -- and some permissions simply do not cross the boundary. Each of these is individually documented and collectively the reason a share that "was definitely granted" is not queryable. Multi-account lake designs should budget real time for this, and should settle on one producer account owning the catalog rather than distributing catalog ownership.

Common Decisions (ADR Triggers)¶

Lake Formation vs IAM-and-bucket-policies alone -- Lake Formation for column-, row-, or cell-level requirements, multi-team lakes, and cross-account sharing of governed SQL vs plain IAM/S3 policies when access is table-or-coarser, the consumer set is small, and the operational simplicity is worth more than the granularity. Lake Formation adds a real operational surface; it should be adopted for a requirement, not for completeness.
LF-Tag-based access control vs named-resource grants -- LF-TBAC for any lake expected to grow, because governance keeps pace with table creation vs named-resource grants for a small, static set of tables where the indirection of tags is not worth the design effort. Mixed models are workable but should be a deliberate exception, not a drift.
Migration approach -- hybrid access mode for incremental, per-team onboarding with a documented end state vs a flag-day cutover where the lake is small enough to migrate and verify in one change window. Hybrid mode is safer; its risk is becoming permanent.
LF-Tag ontology -- a minimal orthogonal set (sensitivity, domain, layer) vs a richer taxonomy modelling more attributes. Richer ontologies rarely pay for themselves and reintroduce the enumeration problem tags were meant to solve.
Where filtering happens -- Lake Formation data filters (centralized, engine-enforced, applies to all governed consumers, engine-capability-dependent) vs materialized masked/filtered tables in the silver or gold layer (portable across every engine including non-integrated ones, costs storage and a pipeline, decouples from Lake Formation) vs both, with filters for exploratory access and materialized views for production consumption.
Cross-account sharing model -- share to the consumer account and let its administrator re-grant (federated ownership, scales to many consumers, requires a competent consumer-side admin) vs share directly to named principals (tighter control, no onward re-sharing, does not scale) vs replicate data into the consumer account (highest cost and drift risk, but removes all cross-account coupling).
Catalog topology -- a single central Glue Data Catalog in a dedicated data-lake account with resource links outward vs per-domain catalogs in domain accounts shared laterally. The first is simpler to govern; the second aligns with a data-mesh operating model and costs more coordination.
Table format under governance -- Iceberg with Glue table optimizers as the current default for ACID lake tables vs alternatives, noting that maintenance DML on open table formats is not uniformly covered by the Lake Formation permission model and generally needs a privileged maintenance role. See general/open-table-formats.md.

Reference Architectures¶

Single-account governed lake¶

S3 bucket with bronze/, silver/, gold/ prefixes registered with Lake Formation using the service-linked role -> Glue Data Catalog holds one database per layer -> LF-Tags applied by the table-creation pipeline (layer, domain, classification) -> LF-TBAC policies grant the data-engineering role read on all layers, analysts read on layer=silver|gold excluding classification=restricted columns, and BI service principals read on layer=gold only -> Athena is the query surface, with per-workgroup cost controls -> analyst roles carry no direct s3:GetObject on the lake bucket, and a scheduled test asserts that a control principal is denied. See providers/aws/athena.md, providers/aws/glue.md.

Multi-account lake with a central producer¶

A dedicated data-lake account owns the S3 buckets, the Glue Data Catalog, the registered locations, and all Lake Formation grants -> consumer accounts (analytics, data science, a business unit) receive LF-Tag-based cross-account shares -> each consumer account creates resource links to the shared databases and re-grants to its own IAM roles -> queries execute in the consumer account against data that never leaves the producer's buckets -> CloudTrail from all accounts lands in a security account for GetDataAccess auditing. Confirm the account's cross-account version supports tag-based cross-account grants before designing to this shape. See providers/aws/multi-account.md.

Row-level multi-tenant isolation¶

A shared fact table partitioned by event date, with a tenant_id column -> one Lake Formation data filter per tenant role with a row-filter expression on tenant_id and a column include list that omits internal-only columns -> tenant-facing roles receive SELECT on the table only through the filter -> the ingestion pipeline role holds unfiltered SELECT/INSERT and is explicitly excluded from the filtered grant, so it retains pushdown predicates and bookmarks. Validate filter behaviour on each engine the tenants can reach.

Incremental migration of an existing ungoverned lake¶

Start by registering locations and enabling hybrid access mode on the existing catalog databases -> onboard one team at a time: grant them Lake Formation permissions, remove their direct S3 permissions, verify with a denial test, then record them as migrated -> keep a burn-down of principals still on the IAM path -> once empty, remove IAMAllowedPrincipals from the resources and turn off the "use only IAM access control" account defaults so new resources are governed from creation.

Reference Links¶

AWS Lake Formation Developer Guide -- service overview, concepts, and the governed-access model
Lake Formation Permissions Reference -- the full permission matrix per resource type, including which permissions can be granted cross-account
Managing Access to Underlying Data -- registered locations, the data access role, credential vending, and the required IAM permissions
Lake Formation Tag-Based Access Control -- LF-Tags, tag inheritance, and tag expression policies
Data Filtering and Cell-Level Security -- column specifications, row filter expressions, and per-engine support
Hybrid Access Mode -- running Lake Formation and IAM access control side by side during migration
Cross-Account Data Sharing -- cross-account versions, sharing targets, and prerequisites
Cross-Account Sharing with AWS RAM -- RAM invitations, organization-scoped shares, and share scale
Lake Formation Notes and Restrictions -- documented limitations, including engine-specific and table-format-specific restrictions
Lake Formation Quotas -- default limits on LF-Tags, registered paths, administrators, and path structure
AWS Lake Formation Pricing -- what is and is not separately charged
AWS Glue Data Catalog -- the metadata substrate Lake Formation authorizes over