AWS Glue¶

Scope¶

AWS Glue is two loosely related things sharing a name: the Glue Data Catalog, which is the metadata substrate that Athena, Redshift Spectrum, EMR, and Lake Formation all resolve against, and Glue ETL, a serverless Spark (and Ray, and Python shell) runtime. This file covers both, plus crawlers, job bookmarks, the Schema Registry, Data Quality, and the Iceberg table optimizers.

Specifically: catalog structure and its pricing model, crawler behaviour and its documented failure modes, Glue versions and their support lifecycle, worker types and DPU sizing, the standard-vs-Flex execution classes, job bookmarks and the traps that cause silent reprocessing or skipped files, interactive sessions, the Schema Registry, DQDL-based Data Quality, and Iceberg table optimization.

Glue is referenced in roughly a dozen files across the library — as an ETL option in patterns/data-pipeline.md's sized cost tables, as a catalog in general/data-analytics.md's governance ADR, as a schema registry in general/messaging-patterns.md and patterns/event-driven.md, and as a data-quality option in several compliance files — but had no page of its own. This is that page; it consolidates rather than repeating those references. For the query surface see providers/aws/athena.md; for access control over the catalog see providers/aws/lake-formation.md; for the table formats see general/open-table-formats.md.

Rates below are published us-east-1 figures taken from AWS's pricing feed and documentation at the time of writing, and are included because DPU economics drive the design. Verify current rates and regional availability before using them in an estimate.

Checklist¶

Glue Data Catalog¶

[Critical] Is the Data Catalog treated as the shared metadata substrate for the whole AWS analytics estate rather than as a Glue-specific feature? Athena, Redshift Spectrum, EMR, Glue ETL, and Lake Formation all resolve tables through it. Catalog design decisions — database-per-layer, naming, partition strategy — therefore constrain every one of those consumers, and the catalog is where access control is applied.
[Critical] Is the catalog cost model understood, and specifically what counts as an object? The first million objects stored and the first million requests per month are free; beyond that, storage is billed per object-month (published at $1.00 per 100,000 objects/month in us-east-1) and requests per million (published at $1.00 per million). AWS defines a metadata object as "a table, table version, partition, partition indexes, statistics, database, or catalog" — so table versions and partition indexes count as objects too. A heavily-partitioned lake with frequent schema updates accrues objects much faster than a naive count of tables and partitions suggests.
[Critical] Is partition count treated as a cost and performance decision rather than a side effect? Every partition is a catalog object, so over-partitioning is paid for three times: in catalog storage, in slower query planning, and in the small-file problem it creates downstream. See general/open-table-formats.md. For very high partition counts, partition projection in Athena avoids catalog partitions entirely — see providers/aws/athena.md.
[Recommended] Are databases organized to match the access-control and lifecycle boundaries that will be needed later — typically one database per medallion layer, or per domain? Lake Formation grants and LF-Tags attach naturally to databases, so a flat single-database catalog forces per-table grants forever. See patterns/lakehouse-medallion.md.
[Recommended] Are partition indexes created on tables with many partitions where catalog partitions are in use? Without an index, a partition filter requires evaluating all partitions; with one, it is served from the index. Note that the indexes themselves count as catalog objects.
[Recommended] Is catalog encryption configured where metadata itself is sensitive, recognizing that column names and partition values can leak business information even when the data is protected?
[Optional] If the account uses multiple catalogs, federated catalogs, or Redshift-managed catalogs, has the specific combination been confirmed to work with the intended engines and with Lake Formation? Multi-catalog support has been added incrementally and capability varies by catalog type.

Crawlers¶

[Critical] Has the choice been made deliberately between crawlers and pipeline-managed table definitions? A crawler is discovery tooling: it is genuinely useful for unfamiliar or externally-produced data. For data you produce yourself, having the writing pipeline register partitions explicitly is cheaper, faster, deterministic, and immune to a crawler changing a column's type. Running a crawler on a schedule against your own output is the most common unnecessary Glue cost.
[Critical] Is crawler cost understood — the same DPU-hour rate as ETL (published at $0.44/DPU-hour in us-east-1), with a documented minimum billing duration per run? A crawler pointed at a lake with millions of objects re-lists them on every run. Crawlers are frequently scheduled hourly out of caution and cost more than the pipelines they support.
[Critical] Is the crawler's schema-change and deleted-object behaviour configured explicitly rather than left on defaults? The configuration options control whether a schema change updates the table, is only logged, and whether objects that have disappeared cause the table or partition to be deprecated, deleted, or ignored. The dangerous default combination is one where an upstream format change silently rewrites the table schema that every downstream query depends on.
[Critical] Is the S3 prefix layout designed so the crawler produces the tables you intend? A crawler groups objects into one table when their schemas are sufficiently similar and the layout looks like a partitioned table; when they are not, it creates a separate table per variant. The classic failure is a badly-organized prefix producing dozens or hundreds of near-duplicate tables. Table-grouping and schema-combining configuration exists to control this, and it needs to be set before the first run, not after the mess exists.
[Recommended] Is type inference reviewed rather than trusted? Crawlers infer types from sampled data, so a column that is integral in the sample and alphanumeric later will be typed wrongly, and a re-crawl can flip a type under a live table. Where the schema is known, declare it.
[Recommended] Are crawlers scoped with include and exclude patterns so they scan only what changes, rather than the whole bucket? Exclusions for temporary directories, _SUCCESS markers, staging prefixes, and Spark scratch output prevent both cost and spurious tables.
[Optional] For tables in an open table format, has the crawler been dropped entirely? Iceberg tracks its own partitions and schema in table metadata, so crawling is redundant — one of the more immediate practical benefits of adopting Iceberg.

Glue Versions and the Runtime¶

[Critical] Is the Glue version pinned explicitly on every job rather than relying on the default? Jobs created without a version default to the current release (Glue 5.1 at the time of writing), which means a job definition created today and one created next year can land on different Spark and Python versions. Pinning turns a runtime upgrade into a deliberate, testable change.
[Critical] Have jobs on end-of-life Glue versions been migrated? Glue 0.9, 1.0, and 2.0 reached end of life on 1 April 2026. Anything still declaring those versions is running on an unsupported runtime and should be treated as remediation work, not backlog.
[Critical] Is the job timeout default change across Glue versions accounted for? The default timeout is 2,880 minutes for Glue 4.0 and earlier but 480 minutes for Glue 5.0 and later. A long-running job migrated from Glue 4.0 to 5.x without an explicit timeout can start failing at eight hours for no apparent reason. The hard ceiling is 7 days (10,080 minutes) for all versions.
[Recommended] Is the Java version change noted when migrating? Glue 5.0 and 5.1 run Java 17; Glue 4.0 and 3.0 run Java 8. Custom JARs and any dependency with bytecode or reflection constraints need revalidation.
[Optional] Is the job type matched to the work — glueetl for Spark batch, gluestreaming for Spark streaming, pythonshell for small single-node Python (0.0625 or 1 DPU, defaulting to 0.0625), glueray for distributed Python via Ray? A small metadata or API-calling task on a Python shell job at a sixteenth of a DPU costs almost nothing; the same task on a Spark job pays for a cluster.

Worker Types and DPU Sizing¶

[Critical] Is the worker type chosen from the actual memory profile of the job rather than defaulting to G.1X everywhere? Each worker provides one executor. Published sizing: G.025X 0.25 DPU / 2 vCPU / 4 GB / 84 GB disk (low-volume streaming only); G.1X 1 DPU / 4 vCPU / 16 GB / 94 GB; G.2X 2 DPU / 8 vCPU / 32 GB / 138 GB; G.4X 4 DPU / 16 vCPU / 64 GB / 256 GB; G.8X 8 DPU / 32 vCPU / 128 GB / 512 GB; G.12X 12 DPU / 48 vCPU / 192 GB / 768 GB; G.16X 16 DPU / 64 vCPU / 256 GB / 1,024 GB.
[Recommended] For memory-bound work — wide shuffles, large broadcast joins, heavy Python UDFs — have the R-series memory-optimized workers been evaluated? They double memory per DPU relative to the G series: R.1X 1 DPU / 4 vCPU / 32 GB, R.2X 2 DPU / 8 vCPU / 64 GB, R.4X 4 DPU / 16 vCPU / 128 GB, R.8X 8 DPU / 32 vCPU / 256 GB. Reaching for more G-series workers to solve an out-of-memory error buys CPU you do not need at the same time as the memory you do.
[Critical] Has regional and version availability been checked for the larger worker types? G.4X and G.8X require Glue 3.0 or later and are documented as available only in a specific subset of regions; G.12X and G.16X and all R-series types require Glue 4.0 or later. A job definition that works in one region can fail to deploy in another.
[Recommended] Is WorkerType plus NumberOfWorkers used rather than the legacy MaxCapacity/AllocatedCapacity fields? From Glue 2.0 onward, maximum capacity cannot be specified for Spark jobs and the worker-based fields are the supported mechanism.
[Optional] For Ray jobs, is the different accounting understood — Z.2X maps to 2 M-DPU (8 vCPU, 64 GB, 128 GB disk) and provides up to 8 Ray workers under the autoscaler, rather than one executor per worker?

Cost Control on ETL¶

[Critical] Is the DPU-hour model understood as the whole cost story — published at $0.44 per DPU-hour for Spark ETL in us-east-1, billed per second? Cost is DPUs multiplied by wall-clock time, so an oversized cluster that finishes faster is not necessarily cheaper, and an undersized one that spills to disk repeatedly is usually much more expensive.
[Critical] Has the Flex execution class been applied to every job that is genuinely time-insensitive? Flex is published at $0.29 per DPU-hour against $0.44 standard — roughly a third off — in exchange for variable start and completion times and no dedicated capacity. AWS documents standard as "ideal for time-sensitive workloads that require fast job startup and dedicated resources" and Flex as "appropriate for time-insensitive jobs whose start and completion times may vary." Overnight backfills, non-urgent conversions, and reprocessing jobs are the obvious candidates. Note the constraint: Flex requires Glue 3.0 or later and job command type glueetl — Spark batch only.
[Recommended] Are jobs profiled before being scaled up? The Spark UI and job metrics usually reveal skew, an accidental full scan, a missing pushdown predicate, or an oversized shuffle. Adding workers to a skewed job pays more for the same wall clock.
[Recommended] Is MaxConcurrentRuns set intentionally, given it defaults to 1? A job triggered more often than it completes will error rather than queue, unless job run queuing is enabled — in which case runs wait for capacity instead of failing immediately. The VPC caveat matters: queuing activates for IP exhaustion only if the shortage is detected at launch, not if it occurs while executors are provisioning.
[Recommended] Are interactive sessions given a short idle timeout? Sessions bill at the same DPU-hour rate as jobs, and an abandoned notebook session is a cluster nobody is using. This is the Glue equivalent of a forgotten interactive cluster.
[Optional] Have Glue's costs been compared against the alternatives for the specific workload — EMR on EC2 with Spot for large sustained batch, EMR Serverless, Athena CTAS for pure SQL conversion, or plain Lambda for small event-driven transforms? Glue's advantage is no cluster management; it is not automatically the cheapest way to run Spark. See patterns/data-pipeline.md.

Job Bookmarks¶

[Critical] Is it understood that bookmarks for S3 sources key off the object's last modified time, and what that implies? AWS documents plainly that "if your input source data has been modified since your last job run, the files are reprocessed." A pipeline that rewrites files in place — a compaction job, a corrected export, an aws s3 sync that touches everything — will cause the whole set to be reprocessed. This is the single most surprising bookmark behaviour and the usual cause of "the incremental job suddenly processed everything."
[Critical] Is bookmark support confirmed for the specific source and format in use? Bookmarks are implemented for JDBC sources, the Relationalize transform, and some S3 sources. For S3, Glue 1.0 and later support JSON, CSV, Avro, XML, Parquet, and ORC. A source outside that set silently has no bookmark, so the job reprocesses everything every run while appearing to be incremental.
[Critical] For JDBC sources, do the bookmark keys actually satisfy the requirement? Glue defaults to the primary key provided it is sequentially increasing or decreasing with no gaps. A UUID primary key, a key with gaps from deletes, or a non-monotonic key will not track correctly. Note also that case-sensitive column names are not supported as bookmark keys.
[Critical] Is it known that resetting or rewinding a bookmark does not clean the target? Glue tracks sources only, not sinks. Reprocessing after a reset therefore duplicates output unless the job writes to a different target or the transformation is an idempotent MERGE. This is the mechanism behind most duplicate-data incidents in Glue pipelines. See the idempotency guidance in patterns/lakehouse-medallion.md.
[Critical] Is transformation_ctx treated as part of the persisted contract? Bookmark state elements are keyed by transformation_ctx, so changing a source's S3 path without changing its transformation_ctx makes the job apply the old bookmark state to the new path — AWS's documented consequence is missing or skipped files, because Glue assumes they were already processed. Changing an input path is therefore a two-part change.
[Recommended] Are the three bookmark modes used deliberately — Enable to track and advance, Disable (the default) to always process everything, and Pause to process an increment or a specified range without advancing the state? Pause is the right tool for a controlled backfill; teams often use Disable and then clean up duplicates instead.
[Recommended] Is bookmark reset a documented, rehearsed runbook step (aws glue reset-job-bookmark, the ResetJobBookmark API, or the console) with the target-cleanup implications spelled out? Doing this for the first time during an incident is how duplicates get created.
[Optional] For very large partitions, is the S3 file-lister option used? A bookmark lists all files under each input partition to do its filtering, so a partition with a very large number of files can drive the Spark driver out of memory.
[Optional] Is it noted that deleting a job deletes its bookmark, so a job recreated under the same name starts from nothing?

Schema Registry, Data Quality, and Table Optimizers¶

[Recommended] Is the Glue Schema Registry used to enforce producer/consumer contracts on streaming data, rather than relying on convention? It integrates with MSK, Kinesis Data Streams, Managed Service for Apache Flink, and Lambda, and enforces configurable compatibility so a breaking producer change is rejected at registration instead of discovered downstream. This is the concrete AWS instance of the data-contract guidance in general/messaging-patterns.md and patterns/event-driven.md.
[Recommended] Is the compatibility mode chosen from the actual evolution requirement rather than left permissive? A registry configured to allow anything provides discovery without protection, which is the failure mode it exists to prevent.
[Recommended] Is Glue Data Quality evaluated for rule-based validation expressed in DQDL, applied either inside ETL jobs or against catalog tables? Note the billing shape before rolling it out broadly: recommendation and evaluation tasks are documented with a minimum of 2 DPUs and a 1-minute minimum billing duration, and anomaly detection and retraining bill per statistic. Quality checks on hundreds of tables are a real recurring cost, not a free feature. See the quality-gate placement guidance in patterns/lakehouse-medallion.md.
[Critical] For Iceberg tables in the catalog, are the table optimizers enabled — compaction, snapshot retention, and orphan-file deletion? These are the managed equivalent of the maintenance jobs that every mutable Iceberg table needs and that teams routinely forget. Published at $0.44 per DPU-hour, billed per second with a 1-minute minimum. Unmaintained tables do not merely slow down: Glue enforces a metadata-size limit and rejects requests to tables that exceed it. See general/open-table-formats.md.
[Recommended] Is column-statistics generation enabled on tables where the query engine's optimizer benefits, with its separate DPU-hour charge budgeted? Missing statistics are a common cause of poor join ordering.
[Optional] If DataBrew appears in a design, is it evaluated on its own merits? It remains an actively priced service (published at $1.00 per 30-minute interactive session and $0.48 per node-hour for jobs, billed per minute) and is aimed at visual, analyst-driven preparation rather than engineered pipelines. It is a different tool for a different user, not a substitute for Glue ETL.

Security and Operations¶

[Critical] Are secrets kept out of job arguments? AWS documents explicitly that "job arguments may be logged" and directs users to retrieve secrets from a Glue connection, Secrets Manager, or another secret manager instead. A password passed as --db_password is a password in CloudWatch. See providers/aws/secrets-manager.md.
[Critical] Do Glue jobs run under least-privilege IAM roles scoped per job or per pipeline, rather than one shared role with broad S3 and catalog access? The Glue job role is the identity that reads and writes the lake; a single shared role makes the medallion layer boundaries unenforceable.
[Recommended] Are jobs that access private resources configured with Glue connections placing elastic network interfaces in the right subnets, with enough IP capacity for peak concurrency? IP exhaustion in an undersized subnet is a common and confusing Glue failure, and job run queuing only mitigates it when the shortage is visible at launch.
[Recommended] Are job definitions managed as code (CloudFormation, CDK, Terraform) or via the built-in source-control integration with GitHub or CodeCommit, rather than authored in the console? Visual-editor jobs that exist only in the console are not reviewable, not reproducible across accounts, and not recoverable.
[Recommended] For streaming jobs, is the maintenance window configured deliberately? AWS periodically restarts streaming jobs and will do so within three hours of the configured window; streaming jobs are also restarted after seven days. A streaming job with no checkpointing discipline will lose state on a restart it did not expect.
[Recommended] Are job failures alarmed through CloudWatch Events/EventBridge to the on-call rotation, and is MaxRetries set with an understanding that retrying a non-idempotent job duplicates data?
[Optional] Is the job-authoring mode recorded and consistent per team — script, visual editor, or notebook? Mixed modes on one pipeline make review and migration harder than the convenience is worth.

Why This Matters¶

The Data Catalog is the highest-leverage and least-visible component in an AWS analytics estate. It is not a Glue feature that Athena happens to read; it is the shared namespace that Athena, Redshift Spectrum, EMR, Glue ETL, and Lake Formation all depend on, which means catalog structure determines what access control can express, what query planning costs, and how many places break when a schema changes. Teams that treat it as an artifact the crawler happens to produce end up with a flat namespace of hundreds of inferred tables, per-table Lake Formation grants forever, and no reliable answer to which table is authoritative.

Crawlers deserve particular scepticism because they are the default path and frequently the wrong one. They are genuinely valuable for exploring data someone else produced. Pointed at your own pipeline's output on an hourly schedule, a crawler re-lists your lake, charges DPU-hours for the privilege, infers types from samples, and retains the ability to change a live table's schema under running queries. The alternative — the writing pipeline registering its own partitions — is cheaper, faster, deterministic, and cannot surprise anyone. For Iceberg tables the crawler is redundant outright, because the format tracks its own schema and partitions.

Job bookmarks are where Glue's abstraction leaks most damagingly, because every failure mode is silent and each looks like something else. Bookmarks on S3 key off last-modified time, so any process that rewrites files in place causes a full reprocess — and a job that reprocesses everything looks like a job that is working, just slowly and expensively. A source format outside the supported set has no bookmark at all, so the job is not incremental despite being configured as such. A JDBC primary key that is a UUID or has gaps does not track correctly. Resetting a bookmark does not clean the target, so recovery duplicates data. And changing a source path without changing its transformation_ctx makes Glue apply the old state to the new location and skip files it has never seen. Each of these is documented; none is discoverable from the job's behaviour without knowing to look.

The cost model rewards two specific decisions and punishes their absence. The first is Flex: at a published $0.29 against $0.44 per DPU-hour, moving genuinely time-insensitive work to the flexible execution class is roughly a third off for a configuration flag, and most estates have a substantial share of overnight backfills and reprocessing that qualify. The second is worker type: reaching for more G.1X workers to fix an out-of-memory error buys CPU alongside the memory, while an R-series worker doubles memory per DPU directly. Beyond those, cost is DPUs times wall clock, which means the highest-return activity is usually profiling one long-running daily job rather than tuning many small ones.

Finally, the version lifecycle is a real and current obligation rather than housekeeping. Glue 0.9, 1.0, and 2.0 reached end of life on 1 April 2026, so any job still declaring them is on an unsupported runtime today. The migration has sharp edges that are easy to miss: Glue 5.x moves to Java 17, and the default job timeout drops from 2,880 minutes to 480, which turns a previously-fine eight-hour job into a job that fails at eight hours with no code change. Pinning the version explicitly on every job is what makes that a planned migration instead of an incident.

Common Decisions (ADR Triggers)¶

Crawlers vs pipeline-managed table definitions — crawlers for discovery of unfamiliar or third-party data vs explicit partition registration from the writing pipeline for data you produce (cheaper, deterministic, no type-inference risk) vs neither, for Iceberg tables that track their own metadata.
Glue ETL vs EMR vs EMR Serverless vs Athena CTAS vs Lambda — Glue for serverless Spark with no cluster management and catalog integration vs EMR on EC2 with Spot for large sustained batch at lower unit cost and more control vs EMR Serverless for Spark without cluster management but different economics vs Athena CTAS for pure SQL conversion vs Lambda for small event-driven transforms. Glue's advantage is operational, not price.
Standard vs Flex execution class — standard for anything with a delivery SLA or a downstream dependency waiting on it vs Flex for time-insensitive backfills, conversions, and reprocessing at roughly a third less per DPU-hour. This should be a per-job decision recorded in the job definition, not a global default.
Worker type and count — G-series for balanced CPU and memory, R-series for memory-bound shuffles and UDF-heavy work, G.025X for low-volume streaming, larger G types for very large workloads subject to version and regional availability. Size from a profiled run, not from a guess.
Job bookmarks vs an explicit watermark — bookmarks for standard S3 and JDBC incremental reads where the source's behaviour matches their assumptions vs an explicit high-watermark table or a table-format change feed (Iceberg incremental reads, Delta Change Data Feed) where files are rewritten in place, keys are non-monotonic, or the semantics need to be visible and testable.
Catalog partitions vs partition projection — catalog partitions with partition indexes when partitions are enumerable and other engines need them vs Athena partition projection for high-cardinality regularly-shaped time partitions, which removes both the catalog cost and the crawler. See providers/aws/athena.md.
Self-managed Iceberg maintenance vs Glue table optimizers — optimizers for lower operational risk at a metered DPU cost, with AWS running compaction, snapshot retention, and orphan-file deletion vs self-scheduled Spark maintenance for control and lower unit cost, accepting that unscheduled maintenance is the common failure. See general/open-table-formats.md.
Glue Data Quality vs an external framework — Glue Data Quality for DQDL rules integrated with the catalog and ETL jobs, priced per DPU-hour with a 2-DPU floor per task vs Great Expectations, Soda, or dbt tests for portability, richer expectation libraries, and no per-scan DPU charge. Cost per checked table is the deciding factor at scale.
Glue Schema Registry vs Confluent Schema Registry vs Apicurio — Glue's registry for AWS-native streaming with MSK, Kinesis, and Managed Flink and no separate infrastructure vs Confluent's for Kafka-centric estates already on Confluent tooling vs Apicurio for self-hosted and cross-platform needs.
Job authoring: code vs visual editor vs notebook — code (script jobs in source control, deployed by IaC) for anything production vs the visual editor for prototyping and analyst hand-offs vs notebooks and interactive sessions for exploration. Console-authored jobs are not reproducible and should not run production pipelines.

Reference Architectures¶

Glue as the metadata and ETL layer for an S3 lake¶

S3 with bronze/, silver/, gold/ prefixes -> three Glue databases in the Data Catalog, one per layer -> Glue Spark ETL jobs (pinned Glue version, G.2X workers sized from a profiled run, Flex on the overnight backfills) perform the hops, registering partitions explicitly rather than via crawlers -> Glue Data Quality rules gate bronze-to-silver -> Athena and Redshift Spectrum read -> Lake Formation grants per database with bronze restricted to the pipeline role -> job definitions and catalog resources deployed by CDK or Terraform. See providers/aws/lake-formation.md, providers/aws/athena.md, patterns/lakehouse-medallion.md.

Iceberg lake with managed maintenance¶

Iceberg tables in S3 registered in the Glue Data Catalog, which also exposes an Iceberg REST catalog endpoint -> Glue ETL or EMR Spark as the writer of record -> Glue table optimizers run compaction, snapshot retention, and orphan-file deletion per table, replacing hand-scheduled maintenance jobs -> no crawlers, because Iceberg tracks its own schema and partitions -> Athena serves interactive SQL and occasional row-level corrections. Watch the catalog metadata-size limit on high-commit tables. See general/open-table-formats.md.

Discovery of third-party data¶

An external partner delivers files to a dedicated S3 prefix -> a crawler with tight include/exclude patterns and Log-only schema-change behaviour catalogues the incoming structure without being permitted to rewrite the live table schema -> a Glue ETL job converts to partitioned Parquet or Iceberg in a curated database with a declared schema -> quality rules reject records that do not match the contract, routing them to a quarantine table. The crawler is confined to the landing zone, which is the one place its behaviour is appropriate.

Streaming ingestion with enforced contracts¶

Producers register Avro or JSON schemas in the Glue Schema Registry with a compatibility mode that rejects breaking changes -> records flow through MSK or Kinesis Data Streams -> a Glue streaming job (gluestreaming, with a configured maintenance window and checkpointing that survives the periodic restart) appends to bronze Iceberg tables -> a scheduled batch job merges into silver. The registry is what stops a producer deploy from breaking the pipeline. See general/messaging-patterns.md, patterns/event-driven.md.

Small event-driven transforms¶

S3 event notification -> Lambda for lightweight per-object work, or a Glue Python shell job at 0.0625 DPU for small single-node Python that needs the catalog or a JDBC driver -> results written to a curated prefix. Reserving Spark jobs for work that actually needs Spark avoids paying cluster prices for scripting. See providers/aws/lambda-serverless.md.

Reference Links¶

What is AWS Glue -- service overview and component map
AWS Glue components -- Data Catalog, crawlers, jobs, and how they relate
Data Catalog and crawlers -- catalog structure, databases, tables, partitions, and schema management
Defining crawlers -- crawler targets, scheduling, include/exclude patterns, and table creation behaviour
Crawler configuration options -- schema-change and deleted-object behaviour, table grouping, and schema combining
AWS Glue versions -- Spark, Python, and Java versions per Glue version, the current default, and per-version changes
Glue version support policy -- end-of-life dates and the supported-version policy
Defining job properties for Spark jobs -- worker types, number of workers, execution class, and timeouts
Authoring Glue Spark jobs -- job types, scripts, and the visual editor
Glue Jobs API reference -- authoritative worker-type sizing, ExecutionClass, timeout defaults, and job-run queuing
Job parameters -- special parameters including bookmark options, and the warning that arguments may be logged
Tracking processed data using job bookmarks -- supported sources and formats, last-modified-time semantics, JDBC bookmark keys, reset and rewind, and best practices
Glue interactive sessions -- notebook sessions, configuration, and idle timeout
Glue Schema Registry -- supported formats, compatibility modes, and integrations
AWS Glue Data Quality -- DQDL rule types, recommendation and evaluation, and anomaly detection
Glue table optimizers -- Iceberg compaction, snapshot retention, and orphan-file deletion
Glue Iceberg REST catalog -- the Glue-hosted Iceberg REST endpoint and supported operations
AWS Glue Pricing -- DPU-hour rates for ETL, Flex, crawlers, interactive sessions, table optimization, and Data Quality; Data Catalog free tier and object definition