Skip to content

ArgoCD GitOps

Scope

ArgoCD continuous delivery platform for Kubernetes. Covers ArgoCD architecture (API server, repo server, application controller, Redis, Dex), Application and ApplicationSet custom resources, sync strategies (automatic vs manual, self-heal, prune), App of Apps pattern, multi-cluster management, Kustomize and Helm integration, RBAC and SSO configuration, secrets management (with Sealed Secrets, External Secrets Operator, SOPS), notification system, resource hooks and sync waves, health assessments, diffing customization, ApplicationSet generators (Git, cluster, matrix, merge, pull request), image updater, and disaster recovery for ArgoCD itself.

Checklist

  • [Critical] Design ArgoCD Application resources with appropriate sync policies: define Applications mapping Git repository paths to Kubernetes cluster/namespace targets; choose between automatic sync (ArgoCD applies changes when Git state diverges) and manual sync (requires explicit approval); enable selfHeal: true to revert manual cluster changes that diverge from Git; enable prune: true to delete resources removed from Git; set syncOptions including CreateNamespace=true, PruneLast=true, and ServerSideApply=true for large resources
  • [Critical] Implement RBAC with SSO integration: configure ArgoCD RBAC policies in argocd-rbac-cm ConfigMap using Casbin syntax; map SSO groups to ArgoCD roles (admin, read-only, project-scoped); integrate with OIDC providers (Okta, Azure AD, Dex with LDAP/SAML) for authentication; define AppProjects to scope team access -- each project restricts source repositories, destination clusters/namespaces, and allowed resource kinds; never share the default admin account across teams
  • [Critical] Secure Git repository and cluster credentials: store Git repository credentials (SSH keys, HTTPS tokens) as Kubernetes Secrets with ArgoCD labels; use credential templates for wildcard repository patterns; register external clusters using kubeconfig or service account tokens stored as Secrets; rotate credentials on schedule; consider using Git credential managers or cloud IAM (GKE Workload Identity, EKS IRSA) to avoid long-lived tokens; restrict repo-server access to only required repositories
  • [Critical] Design repository structure for GitOps: separate application source code repositories from GitOps configuration repositories; structure the config repo with environment directories (base/, overlays/dev/, overlays/staging/, overlays/prod/) using Kustomize, or separate Helm value files per environment; avoid storing secrets in Git -- use Sealed Secrets, External Secrets Operator, or SOPS with age/PGP keys for encrypted secrets; ensure every change is traceable through Git history
  • [Recommended] Implement the App of Apps pattern for cluster bootstrapping: create a root Application that manages child Application resources; the root Application points to a directory of Application manifests, enabling declarative management of all applications on a cluster; use this pattern for cluster bootstrapping -- deploying infrastructure components (cert-manager, ingress, monitoring) and application workloads from a single root; version child Applications by pinning targetRevision
  • [Recommended] Configure ApplicationSets for multi-cluster and multi-tenant deployment: use ApplicationSet generators to template Applications across clusters, environments, or teams; Git generator creates Applications from directory structure in the config repo; cluster generator creates Applications for each registered cluster; matrix generator combines two generators (e.g., clusters x applications); pull request generator creates preview environments per PR; use merge generator for complex parameter combination
  • [Recommended] Design sync waves and resource hooks for ordered deployment: use argocd.argoproj.io/sync-wave annotations to control resource application order (lower numbers first -- namespaces, then config maps, then deployments, then ingresses); use resource hooks (PreSync, Sync, PostSync, SyncFail) for database migrations, smoke tests, and notifications; configure argocd.argoproj.io/hook-delete-policy to clean up hook resources after completion
  • [Recommended] Configure notifications for deployment visibility: install argocd-notifications controller; configure notification templates and triggers for sync success, failure, health degradation, and sync operation running; integrate with Slack, Microsoft Teams, email, PagerDuty, OpsGenie, or webhook endpoints; use trigger conditions to avoid notification fatigue (e.g., notify only on sync failure or health degradation, not on every successful sync)
  • [Recommended] Implement health assessment customization: ArgoCD includes built-in health checks for standard Kubernetes resources (Deployments, StatefulSets, Services); add custom health checks in argocd-cm ConfigMap for CRDs (Kafka topics, database clusters, Istio VirtualServices) using Lua scripts; configure resource tracking method (label, annotation, or annotation+label) based on whether other tools modify the same resources
  • [Recommended] Plan ArgoCD high availability and disaster recovery: deploy ArgoCD in HA mode with multiple replicas of API server, repo server, and application controller (with sharding for large-scale deployments); back up ArgoCD configuration (Applications, AppProjects, repositories, clusters) using argocd admin export or by managing ArgoCD itself with GitOps; store the ArgoCD namespace manifests in a separate Git repo; test restoration by redeploying ArgoCD from backup
  • [Optional] Evaluate ArgoCD Image Updater for automatic image promotion: ArgoCD Image Updater watches container registries for new image tags and updates Kustomize or Helm values in Git; configure update strategies (semver, latest, digest); useful for dev/staging environments with automatic promotion; avoid in production where image updates should go through PR review; configure allowlists and registries to prevent unwanted image updates
  • [Optional] Consider ArgoCD Autopilot for opinionated bootstrapping: ArgoCD Autopilot provides an opinionated repository structure and bootstrap process; creates a GitOps repo with ArgoCD self-management and project structure; suitable for greenfield deployments wanting a prescriptive approach; may be too opinionated for organizations with existing repository conventions
  • [Optional] Implement progressive delivery with Argo Rollouts integration: use Argo Rollouts alongside ArgoCD for canary and blue-green deployment strategies with automated analysis; define Rollout resources (replacing Deployments) with canary steps, traffic management (Istio, Nginx, ALB), and AnalysisTemplates for automated promotion/rollback based on metrics; ArgoCD manages the Rollout resource while Argo Rollouts handles the progressive delivery logic

Why This Matters

ArgoCD is the most widely adopted GitOps continuous delivery tool for Kubernetes, with the Git repository serving as the single source of truth for cluster state. This model provides an auditable deployment history (every change is a Git commit), declarative desired state management (ArgoCD reconciles cluster state to match Git), and the ability to detect and optionally revert configuration drift (unauthorized manual changes to the cluster).

The sync policy decision (automatic vs manual, self-heal, prune) is the most consequential configuration choice. Automatic sync with self-heal and prune enabled means any Git commit immediately changes cluster state and any manual cluster modification is immediately reverted. This is powerful for maintaining consistency but dangerous if misconfigured -- a malformed commit to the main branch can immediately break production. Organizations typically use automatic sync for dev/staging and manual sync (or automatic sync without prune) for production, with PR-based approvals gating changes to the production branch.

Multi-cluster management is a primary use case for ArgoCD. A single ArgoCD instance can manage hundreds of clusters, deploying consistent configurations across environments. ApplicationSets are the scaling mechanism -- rather than manually creating Application resources for each cluster/app combination, generators automatically create and manage Applications based on cluster registration, directory structure, or external data sources. The cluster generator combined with the Git generator (matrix pattern) is the standard approach for deploying a set of platform services across all clusters.

Secrets management in a GitOps workflow is a persistent challenge because secrets cannot be stored in plain text in Git. The three dominant approaches are: Sealed Secrets (encrypts secrets with a cluster-specific key, stores encrypted SealedSecret resources in Git), External Secrets Operator (references secrets from external stores like Vault, AWS Secrets Manager, Azure Key Vault), and SOPS (encrypts secret values in-place in YAML files using age, PGP, or cloud KMS keys). External Secrets Operator has become the most popular choice for production environments because it avoids storing any form of the secret in Git and supports rotation.

Common Decisions (ADR Triggers)

  • ArgoCD vs FluxCD -- ArgoCD provides a rich web UI for visualization and management, Application CRDs for explicit app-to-cluster mapping, RBAC with SSO integration, and multi-cluster management from a single instance. FluxCD provides a more lightweight, controller-based approach with no UI dependency, native Kustomize and Helm controller integration, and multi-tenancy through Flux instances per tenant. Use ArgoCD for organizations wanting centralized visibility, UI-based management, and RBAC across teams. Use FluxCD for platform engineering teams wanting lightweight, decentralized GitOps with less operational overhead.
  • Automatic sync vs manual sync for production -- Automatic sync applies Git changes immediately, reducing deployment latency to seconds. Manual sync requires explicit approval, providing a human gate before production changes. Use automatic sync for dev/staging environments and for production environments with robust PR review processes and automated testing. Use manual sync for production in regulated environments requiring explicit deployment approval or for teams building confidence in GitOps.
  • App of Apps vs ApplicationSets -- App of Apps uses a root Application managing child Application manifests in Git, providing full Git-based control over the Application definitions. ApplicationSets use generators to dynamically create Applications based on templates and data sources. Use App of Apps for smaller deployments with stable application sets and when full control over each Application manifest is desired. Use ApplicationSets for dynamic environments (auto-provisioned clusters, PR previews) and large-scale multi-cluster deployments.
  • Sealed Secrets vs External Secrets Operator vs SOPS -- Sealed Secrets encrypts with a per-cluster key and stores encrypted resources in Git (simple but cluster-specific). External Secrets Operator pulls secrets from external stores at runtime (most flexible, supports rotation). SOPS encrypts values in-place in YAML using various key management systems (developer-friendly but requires key distribution). Use External Secrets Operator for production environments with existing secrets managers. Use Sealed Secrets for simpler setups without external secrets infrastructure. Use SOPS for small teams wanting encrypted-at-rest Git secrets.
  • Single ArgoCD instance vs per-cluster ArgoCD -- A single instance managing all clusters provides centralized visibility and management. Per-cluster instances provide isolation and eliminate cross-cluster blast radius. Use a single instance for organizations with fewer than 50 clusters wanting centralized control. Use per-cluster instances for large-scale deployments, strict isolation requirements, or when clusters span trust boundaries.
  • Kustomize vs Helm for GitOps manifests -- Kustomize provides patch-based customization of plain YAML manifests (no templating language). Helm provides templated charts with values files. ArgoCD supports both natively. Use Kustomize for teams preferring plain YAML with overlay-based customization. Use Helm when upstream charts are the source (cert-manager, ingress-nginx) or when complex parameterization is needed. Many organizations use both -- Helm for third-party charts and Kustomize for internal applications.

See Also

  • general/ci-cd.md -- general CI/CD pipeline patterns and GitOps comparison
  • general/deployment.md -- deployment strategies (rolling, blue-green, canary) and rollback procedures
  • providers/fluxcd/gitops.md -- FluxCD as an alternative GitOps platform
  • providers/kubernetes/operations.md -- Kubernetes operations and cluster management
  • providers/kubernetes/security.md -- Kubernetes RBAC and security for ArgoCD-managed clusters
  • providers/hashicorp/vault.md -- Vault integration with External Secrets Operator for ArgoCD
  • providers/github/actions.md -- GitHub Actions as the CI component paired with ArgoCD for CD
  • providers/gitlab/ci-cd.md -- GitLab CI as the CI component paired with ArgoCD for CD