AWS EC2 Image Builder¶

Scope¶

AWS-native golden image pipeline service for building, testing, and distributing AMIs and container images. Covers image and container recipes, components (AWSTOE), infrastructure and distribution configurations, image pipelines, lifecycle policies, cross-account/cross-region distribution, scheduling and parent-image-change triggers, Inspector integration, and IAM/network design for build instances. The cloud-agnostic alternative is providers/hashicorp/packer.md.

Checklist¶

Why This Matters¶

Image Builder is the AWS-native execution of the [Critical] golden-image-pipeline checklist item that appears in general/cloud-workload-hardening.md and providers/aws/ec2-asg.md. Without it (or an equivalent like Packer), AMIs are either built by hand, copied from old snapshots, or pulled fresh from public AMIs at launch — all of which produce drift, miss patches, lack hardening, and leave no audit trail of what is in production.

Image Builder's specific value over alternatives is integration: the build instance can use SSM Session Manager and IAM instance profiles instead of SSH keys; Inspector findings flow directly into the pipeline as a gate; cross-region distribution re-encrypts with per-region KMS keys without separate Lambda glue; lifecycle policies remove snapshots automatically (a cost trap that catches teams running Packer without a separate cleanup job); and AWS-managed components for common hardening (amazon-cloudwatch-agent-linux, update-linux, stig-build-linux-high) eliminate boilerplate that every Packer shop ends up writing.

The most common failure modes:

Re-baking hardening in every application image. Recipes that combine OS hardening, patching, agents, and application install in a single recipe rebuild slowly and make it impossible to know when hardening last changed. The fix is layered recipes: a hardened-base recipe builds the parent image, application recipes reference the parent and only add the application layer.
Pipelines that never run. A pipeline created once with no schedule and no parent-image trigger produces an AMI that is fresh on day one and stale on day 60. The schedule + EXPRESSION_MATCH_AND_DEPENDENCY_UPDATES_AVAILABLE combination is non-default and frequently missed.
No lifecycle policy. Each build creates a new AMI with EBS snapshots. At one build per week per region per account, snapshot cost climbs into the thousands per month within a year. Lifecycle policies (introduced in late 2023) are the AWS-native cleanup; without them, teams write their own cleanup Lambda or simply pay for the storage.
Build instance attached to a production role. Image Builder uses an EC2 instance profile during build; if that profile has broad production permissions, a malicious or buggy build component can act on production resources. Build instance profiles should have only what AWSTOE needs.
Inspector findings ignored. Image Builder integrates with Inspector for vulnerability scans, but the integration is opt-in and the build-stage gate (fail on critical findings) is opt-in. Without both, scans become reports nobody reads.

Common Decisions (ADR Triggers)¶

Image Builder vs Packer vs HCP Packer — Image Builder is the AWS-native choice and the right default for AWS-only shops because of Inspector integration, lifecycle policies, cross-region KMS handling, and zero licensing cost; Packer (open source / BSL) is the right choice when the same image must be produced for multiple clouds or for on-prem hypervisors; HCP Packer adds a registry layer with channel promotion and ancestry tracking that Image Builder approximates with image lifecycle and tags but does not match for cross-cloud lineage. See providers/hashicorp/packer.md for the alternative.
AWS-managed components vs custom components — AWS-managed components for update-linux, update-windows, amazon-cloudwatch-agent-*, and STIG/CIS hardening are maintained by AWS with documented end-of-support windows and remove a maintenance burden; custom components are required for organization-specific hardening exceptions, internal agent installation, application-layer setup, and any test logic. Use AWS-managed components wherever they exist and custom components for the gaps.
Single central pipeline account vs per-account pipelines — central pipeline account with cross-account distribution gives one source of truth, one set of IAM controls, one Inspector configuration, and one lifecycle policy to manage; per-account pipelines duplicate everything and let images drift; the only reason to run per-account is when accounts are owned by independent teams that need to customize images without coordination, in which case a shared base layer with per-account application recipes is the better compromise.
Marketplace hardened base image vs build-from-scratch — marketplace CIS/STIG images (CIS Hardened Images, Nemu, AWS STIG AMIs) are the fastest path to compliance and include monthly vendor patching, but they carry per-hour marketplace cost across every running instance and limit which hardening exceptions you can document; build-from-scratch via the AWS-managed STIG/CIS components is free but requires the team to maintain the hardening pipeline and keep up with benchmark version changes; most mature organizations build from scratch using the AWS-managed components and reserve marketplace images for narrow regulatory cases.
AMI pipeline vs container pipeline for the same workload — when a workload is being modernized from EC2 to ECS/EKS, both pipelines may be needed during the transition; design the application-layer custom components to work in both contexts (avoid hardcoded paths assuming a full OS) so the container recipe and image recipe share the same component library and the hardening story stays parallel.
Build cadence — monthly is the floor for security; weekly is appropriate for environments under active CVE pressure; daily builds are usually overkill except for base images consumed by many downstream pipelines, where the parent-image-change trigger plus a weekly schedule is a better pattern than a daily cron.
Image lifecycle retention window — retain three to six versions of any AMI currently referenced by a launch template or ASG (rollback capacity); deprecate older versions immediately; delete six to twelve months after deprecation; never delete an AMI still referenced by a running launch template version (the lifecycle policy can be configured to skip referenced AMIs).
Distribution scope — distribute to every region where the workload runs, plus disaster-recovery regions; re-encrypt with per-region KMS CMKs at distribution time rather than relying on the source-region key; for cross-account, use targetAccountIds to copy the AMI fully into spoke accounts that need to manage it independently, and launchPermissions for accounts that only need to launch.

Reference Architectures¶

Layered Pipeline (Hardened Base + Application)¶

+----------------------+           +-----------------------+
| AWS-managed base AMI |           | AWS-managed components|
| (Amazon Linux 2023,  |           | - update-linux        |
|  Windows Server 2025)|           | - stig-build-linux-*  |
+----------+-----------+           | - cloudwatch-agent    |
           |                       +-----------+-----------+
           |                                   |
           v                                   v
  +-----------------------------------------+
  | Hardened-Base Image Recipe              |
  | + custom components: EDR agent, certs   |
  +--------------------+--------------------+
                       |
                       v
            +------------------------+
            | Hardened-Base Pipeline |   schedule: weekly
            +-----------+------------+   trigger: parent change
                        |
                        v
            +------------------------+
            | AMI: hardened-base v.N |   tags: profile=stig,
            +-----------+------------+         scan_id=...
                        |
        +---------------+---------------+
        |               |               |
        v               v               v
+--------------+ +--------------+ +--------------+
| App-A Recipe | | App-B Recipe | | App-C Recipe |
| parent: HB-N | | parent: HB-N | | parent: HB-N |
| + app layer  | | + app layer  | | + app layer  |
+------+-------+ +------+-------+ +------+-------+
       |                |                |
       v                v                v
[App-A AMI]      [App-B AMI]      [App-C AMI]
       |                |                |
       v                v                v
[Dev/Stg/Prod]   [Dev/Stg/Prod]   [Dev/Stg/Prod]

When the hardened-base pipeline produces a new AMI (parent image change), application pipelines configured with EXPRESSION_MATCH_AND_DEPENDENCY_UPDATES_AVAILABLE automatically rebuild on their next scheduled run. Hardening changes propagate without per-application intervention.

Central Pipeline Account with Cross-Account Distribution¶

                 +--------------------+
                 | Pipeline Account   |
                 |  (build runs here) |
                 |                    |
                 |  Image Builder     |
                 |  Pipeline + Recipe |
                 |  Inspector         |
                 |  Lifecycle Policy  |
                 |  KMS CMK (us-east-1)|
                 +----------+---------+
                            |
            distribution config:
            - copy to us-west-2, eu-west-1
            - re-encrypt with destination-region KMS CMKs
            - share with: org/o-xxxx, accts: [spoke-A, spoke-B]
                            |
        +-------------------+-------------------+
        |                   |                   |
        v                   v                   v
+---------------+   +---------------+   +---------------+
| Spoke Acct A  |   | Spoke Acct B  |   | Spoke Acct C  |
| (us-east-1,   |   | (us-west-2)   |   | (eu-west-1)   |
|  us-west-2)   |   |               |   |               |
| AMI: shared   |   | AMI: shared   |   | AMI: shared   |
| (launch only) |   | (launch only) |   | (launch only) |
| ASG launches  |   | ASG launches  |   | ASG launches  |
| from AMI      |   | from AMI      |   | from AMI      |
+---------------+   +---------------+   +---------------+

Spoke accounts receive launchPermissions only — they cannot deregister or modify the AMI. Use targetAccountIds (full copy) instead when a spoke account needs independent lifecycle management. Per-region KMS CMKs avoid cross-region KMS API calls at instance launch.

Pipeline with Inspector Gate¶

[Image Pipeline Execution]
        |
        v
+------------------+
| Build Stage      |   build components run on build instance
| - update-linux   |
| - stig-hardening |
| - install agents |
+--------+---------+
        |
        v
+------------------+
| Validate Stage   |   build components' validate phase
| - config checks  |
+--------+---------+
        |
        v
[EBS snapshot of build instance]
        |
        v
+------------------+
| Test Stage       |   test components on a fresh instance
| - Inspector scan |   from the snapshot
| - Goss/InSpec    |
| - smoke tests    |
+--------+---------+
        |
   Inspector findings -> Security Hub
        |
   Critical CVE present?
        |
   +----+----+
   | Yes     | No
   v         v
[FAIL]   [Distribution Stage]
              |
              v
         [Cross-region copy + cross-account share]
              |
              v
         [AMI tagged + lifecycle policy applies]

Test stage runs on a new instance launched from the snapshot, not the build instance — this catches anything baked into the running build instance that is not actually present in the snapshot (services started by hand, files in /tmp).

Reference Architectures and Documentation¶

What is EC2 Image Builder -- service overview and resource model
Image Builder distribution settings -- cross-region, cross-account, and ECR distribution patterns
Cross-account AMI distribution -- full reference for launchPermissions vs targetAccountIds
Image lifecycle policies -- retain, deprecate, disable, delete actions and policy scopes
Components and AWSTOE -- component model, build vs test phases, AWS-managed vs custom
Cron expressions for pipeline scheduling -- six-field cron syntax and rate expressions
Image Builder integration with Amazon Inspector -- vulnerability scan as a build-stage gate
AWS-managed components reference -- catalog of update, hardening, and agent components