Skip to content

Jenkins CI/CD

Scope

Jenkins CI/CD automation server. Covers pipeline types (declarative and scripted), Jenkinsfile configuration, controller/agent architecture, executor management, plugin ecosystem (Blue Ocean, Pipeline, Credentials, Docker, Kubernetes), agent provisioning (static, cloud-based, Kubernetes pods), shared libraries, credentials management, security hardening (RBAC, CSRF, script approval, agent-to-controller access control), Jenkins X for cloud-native CI/CD, Jenkins Configuration as Code (JCasC), pipeline best practices, and migration strategies from Jenkins to modern platforms.

Checklist

  • [Critical] Design controller/agent architecture for isolation and scalability: run the Jenkins controller (formerly master) as a coordination node only -- never execute builds on the controller; provision agents (formerly slaves) with appropriate labels for workload routing; size the controller for job scheduling overhead (8+ GB heap for 500+ jobs); use separate agents for different build environments (JDK versions, OS, security contexts); configure agent connection methods (SSH, JNLP/WebSocket inbound, or Kubernetes pod agents)
  • [Critical] Implement security hardening: enable matrix-based or role-based access control (Role Strategy plugin) with least-privilege permissions; enable CSRF protection (crumb issuer); configure script approval for pipeline sandbox escapes (do not blanket-approve all scripts); disable agent-to-controller access control bypass (Jenkins security advisory critical item); enforce HTTPS with a reverse proxy (nginx, HAProxy); integrate with LDAP/Active Directory or OIDC for authentication; disable Jenkins CLI over remoting
  • [Critical] Use declarative pipelines as the default pipeline type: prefer declarative syntax (pipeline { } block) over scripted pipelines for readability, built-in validation, and restart-from-stage support; use stages, steps, post, when, environment, and options blocks; use the script { } block within declarative pipelines only when imperative logic is unavoidable; store Jenkinsfiles in source control alongside application code (pipeline-as-code)
  • [Critical] Manage credentials securely: use the Jenkins Credentials plugin to store secrets (username/password, SSH keys, secret text, secret files, certificates); scope credentials to the minimum required level (folder, job, or global); use withCredentials() in pipelines to bind secrets to environment variables or files; never print credentials in console output; integrate with external secrets managers (HashiCorp Vault plugin, AWS Secrets Manager plugin) for production secrets; audit credential usage
  • [Recommended] Implement shared libraries for pipeline standardization: create a shared library repository with vars/ (global variables/functions) and src/ (Groovy classes); configure libraries at the folder or global level with version pinning (@Library('my-lib@v2.1.0')); use shared libraries to enforce organizational standards (build conventions, deployment patterns, notification logic); test shared library code with unit tests (JenkinsPipelineUnit or Spock)
  • [Recommended] Configure Kubernetes-based dynamic agents: use the Kubernetes plugin to provision pod-based agents on demand; define pod templates with multiple containers (build, docker, deploy) using podTemplate or declarative kubernetes agent; configure resource requests and limits for predictable scheduling; use persistent volume claims for workspace caching; set idleMinutes: 0 for immediate pod termination after build; use node affinity for workload-specific node pools
  • [Recommended] Implement Jenkins Configuration as Code (JCasC): define Jenkins system configuration, security settings, credentials, cloud configurations, and tool installations in YAML files (jenkins.yaml); store JCasC files in version control; apply configuration at startup or via the JCasC reload endpoint; eliminates manual UI configuration that drifts between environments; essential for disaster recovery (rebuild Jenkins from code)
  • [Recommended] Design plugin management strategy: maintain a fixed list of plugins with pinned versions (plugins.txt); use the Jenkins Plugin Installation Manager tool for reproducible plugin installs in Docker images; audit plugins quarterly for security vulnerabilities (Jenkins security advisories); remove unused plugins to reduce attack surface; test plugin updates in a staging Jenkins instance before applying to production; be cautious with plugins that modify the Groovy sandbox
  • [Recommended] Configure pipeline performance and resilience: set timeout on pipeline and stage level to prevent hung builds; use retry for flaky external dependencies; implement milestone steps to prevent older builds from deploying over newer ones; configure pipeline durability (PERFORMANCE_OPTIMIZED for high-throughput, MAX_SURVIVABILITY for critical pipelines); use parallel stages for concurrent test execution; archive only necessary artifacts
  • [Recommended] Implement backup and disaster recovery for the controller: back up $JENKINS_HOME (configuration, job definitions, credentials, build history) regularly; at minimum back up config.xml, credentials.xml, jobs/*/config.xml, and the secrets/ directory; use thin backup plugin or filesystem snapshots; with JCasC and pipeline-as-code, the controller can be rebuilt from code -- focus backup on build history and credentials; test restoration procedures quarterly
  • [Optional] Evaluate Jenkins X for Kubernetes-native CI/CD: Jenkins X provides opinionated GitOps-based CI/CD for Kubernetes using Tekton pipelines (not traditional Jenkins); automatic preview environments per pull request, promotion between environments via GitOps, and ChatOps integration; suitable for greenfield Kubernetes projects wanting an opinionated workflow; significantly different from traditional Jenkins -- evaluate as a separate platform decision
  • [Optional] Plan migration from Jenkins to modern CI/CD platforms: assess migration drivers (operational overhead, security patching burden, plugin compatibility issues, talent availability); inventory all pipelines, shared libraries, and plugin dependencies; use GitHub Actions Importer or GitLab CI migration tools for automated conversion of Jenkinsfiles; migrate incrementally -- run both systems in parallel during transition; preserve build history exports for compliance
  • [Optional] Consider Blue Ocean for improved pipeline visualization: Blue Ocean provides a modern UI for pipeline creation and visualization; note that Blue Ocean is in maintenance mode (no new features); useful for existing Jenkins installations wanting improved pipeline visualization; not a factor for new Jenkins deployments

Why This Matters

Jenkins remains the most widely deployed CI/CD automation server, powering an estimated 44% of CI/CD pipelines worldwide. Its plugin ecosystem (1800+ plugins) provides unmatched integration breadth, and its flexibility allows virtually any CI/CD workflow to be implemented. However, this flexibility comes with significant operational cost. Jenkins controllers require regular patching (monthly security advisories are common), plugin compatibility management, and infrastructure scaling that modern managed CI/CD platforms handle automatically.

The controller/agent architecture is the most critical design decision. Running builds on the Jenkins controller is the number one operational anti-pattern -- it creates a single point of failure where a misbehaving build can crash the controller, taking down the entire CI/CD system. The controller should only orchestrate pipelines while agents execute builds. Kubernetes-based dynamic agents have become the standard for modern Jenkins deployments, providing on-demand agent provisioning with clean environments per build and no persistent state to manage.

Security hardening is frequently neglected on Jenkins installations. Common findings include: disabled CSRF protection, overly permissive script approvals (allowing arbitrary Groovy code execution), credentials stored in pipeline scripts rather than the credentials plugin, and the controller accessible without authentication from the internal network. Jenkins security advisories frequently disclose critical vulnerabilities in both core and plugins. Organizations must have a process for monitoring and applying security updates -- the Jenkins LTS release train (every 12 weeks with biweekly security fixes) provides a more stable patching cadence than the weekly release.

Plugin management is an ongoing operational challenge. Plugins frequently have incompatible dependencies, and upgrading one plugin can break others. The "plugin dependency hell" problem is a top reason organizations consider migrating away from Jenkins. Maintaining a tested, pinned plugin list and using a staging Jenkins instance for plugin testing before production upgrades is essential for stability.

Common Decisions (ADR Triggers)

  • Jenkins vs GitHub Actions vs GitLab CI -- Jenkins provides maximum customization, on-premises deployment, and the largest plugin ecosystem. GitHub Actions and GitLab CI provide managed infrastructure, simpler configuration, and tighter SCM integration. Use Jenkins for organizations with existing Jenkins investment, complex build requirements, or on-premises/air-gapped requirements. Use managed platforms for teams wanting lower operational overhead and simpler pipeline development.
  • Declarative vs scripted pipelines -- Declarative pipelines provide structured syntax, built-in validation, Blue Ocean compatibility, and restart-from-stage. Scripted pipelines provide full Groovy programming capability for complex logic. Use declarative pipelines as the default for all new pipelines. Use scripted pipelines only for pipelines requiring complex control flow that cannot be expressed in declarative syntax (rare with script {} blocks available).
  • Static agents vs Kubernetes dynamic agents vs cloud (EC2/Azure) agents -- Static agents are always available with zero provisioning latency but waste resources when idle. Kubernetes agents provision clean pod-based environments per build with autoscaling but add pod scheduling latency (15-60 seconds). Cloud agents (EC2 plugin, Azure VM Agents) provision full VMs with longer startup (2-5 minutes) but support any OS. Use Kubernetes agents for container-based builds. Use cloud agents for VM-based builds (Windows, specialized OS). Use static agents for low-latency requirements.
  • Shared libraries vs inline pipeline code -- Shared libraries centralize build logic, enforce standards, and reduce duplication across pipelines. Inline pipeline code is simpler and self-contained. Use shared libraries for organizations with 20+ pipelines sharing common patterns. Use inline code for small teams with few, distinct pipelines.
  • JCasC vs manual configuration -- JCasC enables version-controlled, reproducible Jenkins configuration. Manual configuration through the UI is quicker for initial setup but drifts over time. Use JCasC for all production Jenkins instances (essential for disaster recovery). Use manual configuration only for proof-of-concept or ephemeral instances.
  • Jenkins LTS vs weekly release -- LTS releases every 12 weeks with backported security fixes and stable plugin compatibility. Weekly releases provide the latest features but more frequent breaking changes. Use LTS for all production Jenkins instances. Use weekly releases only for non-production experimentation.

See Also

  • general/ci-cd.md -- general CI/CD pipeline patterns, platform comparison, branching strategies
  • general/deployment.md -- deployment strategies (rolling, blue-green, canary) and rollback procedures
  • providers/github/actions.md -- GitHub Actions as a migration target from Jenkins
  • providers/gitlab/ci-cd.md -- GitLab CI/CD as a migration target from Jenkins
  • providers/kubernetes/operations.md -- Kubernetes operations for Jenkins agent pod management
  • providers/hashicorp/vault.md -- Vault integration for Jenkins credentials management