Jenkins — from first pipeline to production

The landing page at / is the job list. It is dense and old-fashioned, and once you can read it you can diagnose most things without clicking.

The columns

The S column colours are worth knowing exactly:

• Blue — success. (Jenkins uses blue, not green. There is a plugin to change it, and arguments about it going back a decade.)
• Red — failure.
• Yellow — unstable. The build ran but something like a test reporter marked it degraded.
• Grey — never built, or aborted.
• Flashing/animated — currently running.

Unstable versus failed matters. Failed means the command returned non-zero. Unstable usually means the build succeeded but tests failed and a publisher downgraded it. Chapter 8 covers making that distinction deliberately.

The left sidebar

• New Item — create a job. The most-used link on the page.
• People — users known to Jenkins, including ones seen only in commit metadata.
• Build History — every build across every job, on a timeline.
• Manage Jenkins — all administration. Covered below.
• My Views — personal dashboards.

The build queue and executor status

Bottom-left, and the first place to look when "Jenkins is stuck":

• Build Queue — jobs waiting. Hover the ⓘ for why an item is queued: no available executor, waiting for a lock, or "there are no nodes with the label X" — which is the answer to most mysteries.
• Build Executor Status — every executor across all agents, and what it is running. An idle grid plus a full queue means a label mismatch, not a capacity problem.

Views: the first thing to do once you have 20 jobs

The flat list stops working quickly. Create a view: the + tab next to "All".

• List View — pick jobs explicitly or by a regex on the name. ^payments-.* gathers a team's jobs.
• My View — everything you have permission to see, automatically.

List View options worth setting: Add column for things like Last Stable, and Recurse in subfolders if you use folders — without it a view silently ignores every job inside a folder.

Folders

Install Folders (usually already present) and New Item → Folder. Folders give you namespacing, and — more importantly — a scope for credentials and permissions. Credentials defined in a folder are visible only to jobs inside it, which is how you stop one team's pipeline reading another team's deploy key.

Takeaway: learn the S/W columns and the queue tooltip. "Nothing is building" is almost always a label mismatch, and the queue tooltip says so in plain English.

Click a job and you land on the job page. This is where you will spend most of your time.

The job page

Left sidebar:

• Status — where you are. Recent builds and the trend graph.
• Changes — commits included in each build, aggregated. Excellent for "when did this break and who touched it".
• Workspace — browse the files on the agent as the last build left them. Only present for jobs with a workspace, and only until it is wiped.
• Build Now / Build with Parameters — trigger.
• Configure — edit the job. For a Pipeline job this is where the Jenkinsfile path or inline script lives.
• Delete — with a confirmation.
• Rename, Move (into a folder)
• Pipeline Syntax — the snippet generator. See its own note; it is the most under-used page in Jenkins.

Main area:

• Stage View — a grid of stages across recent builds with per-stage timings. Red cell tells you exactly which stage failed, and the timing column tells you what got slow.
• Build History — down the left, oldest at the bottom. Each entry links to a run.
• Permalinks — "Last build", "Last stable build", "Last successful build". These are stable URLs you can reference from scripts or dashboards.

The build page

Click a build number:

• Console Output — the log. The one you want 90% of the time.
• Edit Build Information — rename the build or add a description. Underused: a build description like "release candidate 2.1" makes history readable months later.
• Changes — commits in this specific run.
• Workspace — as above.
• Restart from Stage — with Declarative Pipeline, re-run from a chosen stage using the same commit. Saves enormous time on a long pipeline whose deploy step failed.
• Replay — re-run this build with an edited pipeline script, without committing. The best debugging feature in Jenkins; see Chapter 4.
• Pipeline Steps — every step executed, in a tree, each linking to its own log fragment. Where you look when the console is 40,000 lines.

Requires — On the machine you run this from: jq

Jenkins has navigation affordances that almost nobody uses because they are not advertised.

The search box

Top right of every page. It is not full-text search — it matches job names, view names, users and build numbers, and it understands paths.

Type a job name and press Enter and you land on the job. More usefully it accepts commands:

api-build          →  the job
api-build 142      →  build #142 of that job
api-build lastBuild

URL-driven navigation

Faster than clicking, once the shapes are in your fingers:

/job/<name>/                        job page
/job/<folder>/job/<name>/           job inside a folder — note the repeated /job/
/job/<name>/configure               straight to configuration
/job/<name>/<n>/console             console of build n
/job/<name>/build?delay=0sec        trigger (POST; needs a crumb)
/view/<view-name>/                  a view
/user/<username>/                   a user's page

The repeated /job/ for folders trips everyone up. A job web inside folder team-a is /job/team-a/job/web/, not /job/team-a/web/.

Keyboard

Classic Jenkins has few shortcuts, but recent versions added a command palette. Press ? on any page to see what the version in front of you supports. On modern releases, Ctrl/Cmd + K opens a search palette.

The API is navigation too

Every page has a machine-readable twin — append :

A shared Jenkins with hundreds of jobs is unusable until you make it yours. These take five minutes and pay back daily.

Your user page

Click your name (top right) → Configure, or /user/<you>/configure:

• Full Name — what appears against builds you trigger.
• Email — where notifications go. Set it, or culprits()-style notification silently skips you.
• API Token — for scripts and the CLI. Add one per purpose (laptop, deploy-script) so you can revoke individually.
• SSH Public Keys — for the SSH-based CLI, if enabled.
• My Views — which view you land on.
• Notification preferences — added by whichever notification plugins are installed.

Personal views

Dashboard → My Views → +

A view visible only to you, listing only what you care about. This is the single best quality-of-life change on a large instance because it changes nothing for anyone else.

Two useful kinds:

• List View with a regular expression on the job name — ^(payments|billing)-.* catches your team's jobs and keeps catching new ones automatically.
• My View — everything you have permission to see, no configuration.

Options worth setting on a List View:

• Recurse in subfolders — without it the view silently ignores everything inside folders, which on a folder-organised Jenkins means it shows nothing.
• Add column → Last Duration, Last Failure, Build Button. The build button in the list saves a navigation every time.
• Filter build queue / Filter build executors — narrows the sidebar to this view's jobs. On a busy instance this makes the queue readable.

Shared views

The + tab next to All on the main dashboard creates a view for everyone. Useful for a team dashboard on a wall display. Requires View/Create.

For a screen in the room, combine a shared view with the plugin, which renders a full-screen grid of job status readable from across an office.

There are three ways to look at a pipeline run. One of them is on its way out, and you will still meet it on existing installations.

Stage View — always there

The grid on the job page, provided by Pipeline: Stage View. A column per stage, a row per build, with per-stage timings. No installation decision to make; it is part of the standard set.

Good for: spotting which stage broke and what got slower. Poor at parallel branches, which it flattens.

Pipeline Graph View — the one to use now

Install Pipeline Graph View. It draws the run as a real graph, with parallel branches as actual branches, and gives per-stage log panes.

This is the plugin the Jenkins project now points people at. It is actively maintained and it covers the part of Blue Ocean that people actually valued — reading a complex pipeline at a glance.

Blue Ocean — legacy

Blue Ocean was an alternative UI, and for years the standard recommendation for pipeline visualisation. That has changed:

Jenkins' documentation states Blue Ocean will be deprecated in July 2026 and will not receive further security fixes or functionality updates, and directs users to Pipeline Graph View and Stage View instead.

Read that second clause carefully. "No further security fixes" on a plugin that renders build output into your Jenkins origin is not a neutral status.

What to do:

• New installations — do not install it. Use Pipeline Graph View.
• Existing installations — you will find it at /blue. Do not build workflows, bookmarks or documentation around it, and plan its removal.
• If you keep it — understand you are running an unmaintained plugin in a privileged application, and weigh that accordingly.

It never covered administration anyway: no Manage Jenkins, no credentials, no agent management. You always bounced back to classic Jenkins, which is part of why replacing it costs so little.

Choosing

Requires — Plugin: Workspace Cleanup · Plugin: JUnit · Plugin: Docker Pipeline · Plugin: Kubernetes

Declarative Pipeline is a strict structure. Everything has a place, and putting something in the wrong place is a compile error rather than a mystery at runtime. That strictness is the point.

The complete skeleton

pipeline {
    agent { label 'linux' }

    options {
        timestamps()
        timeout(time: 1, unit: 'HOURS')
        buildDiscarder(logRotator(numToKeepStr: '30'))
        disableConcurrentBuilds()
        skipDefaultCheckout()
    }

    environment {
        REGISTRY = 'registry.example.com'
        IMAGE    = "${REGISTRY}/api:${env.BUILD_NUMBER}"
    }

    parameters {
        choice(name: 'ENVIRONMENT', choices: ['dev', 'staging', 'prod'])
    }

    triggers {
        cron('H 2 * * *')
    }

    tools {
        jdk 'jdk21'
        maven 'maven-3.9'
    }

    stages {
        stage('Checkout') {
            steps { checkout scm }
        }

        stage('Build') {
            steps { sh 'mvn -B package' }
        }

        stage('Test') {
            steps { sh 'mvn -B test' }
            post {
                always { junit 'target/surefire-reports/*.xml' }
            }
        }
    }

    post {
        always   { cleanWs() }
        success  { echo 'green' }
        failure  { echo 'red' }
        unstable { echo 'tests failed' }
        changed  { echo 'result differs from last build' }
    }
}

Order matters

The directives must appear roughly in the order above — agent first, stages last before post. Jenkins tells you if you get it wrong, but knowing the order saves the round trip.

The blocks, one line each

agent in its forms

agent any                          // any available executor
agent none                         // none at top level; each stage declares its own
agent { label 'linux && docker' }  // label expression
agent { node { label 'linux'; customWorkspace '/mnt/fast/ws' } }

agent {
    docker {
        image 'node:20-alpine'
        args  '-v /tmp/cache:/cache'
        label 'docker'
    }
}

agent {
    dockerfile {
        filename 'Dockerfile.ci'
        dir 'build'
        additionalBuildArgs '--build-arg VERSION=1.2'
    }
}

agent {
    kubernetes {
        yamlFile 'build-pod.yaml'
    }
}

agent none at the top with per-stage agents is the pattern for a pipeline whose stages need different machines. Be aware that with agent none there is no shared workspace between stages — you must stash/unstash to move files. Chapter 6.

options worth knowing

options {
    timeout(time: 30, unit: 'MINUTES')             // whole pipeline
    retry(2)                                        // retry the whole pipeline
    timestamps()                                    // prefix every log line
    disableConcurrentBuilds()                       // one at a time
    disableConcurrentBuilds(abortPrevious: true)    // newest wins
    skipDefaultCheckout()                           // do not auto-checkout
    skipStagesAfterUnstable()
    buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '5'))
    parallelsAlwaysFailFast()
    quietPeriod(15)
    preserveStashes(buildCount: 5)
}

disableConcurrentBuilds(abortPrevious: true) is the one to reach for on pull-request pipelines: when someone pushes three times in a minute, only the last build matters and the other two are wasted agent time.

environment, and the two forms

environment {
    APP   = 'api'
    IMAGE = "registry.example.com/${APP}:${env.BUILD_NUMBER}"
    NPM_TOKEN = credentials('npm-publish-token')
}

credentials() in an environment block makes secrets available as variables. For a username/password credential it creates three: NAME, NAME_USR, NAME_PSW. Chapter 7 covers this and its pitfalls.

Stage-level environment blocks override the global one for that stage only.

Takeaway: learn the skeleton by heart. Declarative's rigidity means a misplaced block fails fast with a clear message, which is much better than the alternative.

Stages carve a pipeline into readable phases. Parallel and matrix are how you stop it taking an hour.

Sequential stages

stages {
    stage('Build')  { steps { sh 'make build' } }
    stage('Test')   { steps { sh 'make test' } }
    stage('Deploy') { steps { sh 'make deploy' } }
}

Parallel

stage('Tests') {
    parallel {
        stage('Unit') {
            steps { sh 'make test-unit' }
        }
        stage('Integration') {
            agent { label 'docker' }
            steps { sh 'make test-integration' }
        }
        stage('Lint') {
            steps { sh 'make lint' }
        }
    }
}

Each branch can declare its own agent, environment and post. Branches run on separate executors — if you have one executor, "parallel" is a queue.

Fail-fast:

stage('Tests') {
    failFast true
    parallel { /* … */ }
}

Or globally with parallelsAlwaysFailFast() in options. Without it, a failing branch lets the others run to completion — which you sometimes want (to see all the failures) and sometimes do not (to free the agents).

Matrix

Matrix generates the parallel branches for you across combinations.

Requires — Plugin: Docker Pipeline

when decides whether a stage runs. It is evaluated when the pipeline reaches that stage.

The conditions

when { branch 'main' }
when { branch pattern: 'release/.*', comparator: 'REGEXP' }
when { buildingTag() }
when { tag 'v*' }
when { changeRequest() }                       // any PR
when { changeRequest target: 'main' }          // PRs targeting main
when { environment name: 'DEPLOY', value: 'true' }
when { equals expected: 2, actual: currentBuild.number }
when { expression { params.DEPLOY == true } }  // arbitrary Groovy returning a boolean
when { changeset '**/*.js' }                   // only if matching files changed
when { changelog '.*\\[deploy\\].*' }          // commit message matches
when { triggeredBy 'TimerTrigger' }
when { not { branch 'main' } }
when { allOf { branch 'main'; environment name: 'ENV', value: 'prod' } }
when { anyOf { branch 'main'; branch 'develop' } }

The combinators

when {
    allOf {
        branch 'main'
        not { changeRequest() }
        anyOf {
            changeset '**/*.go'
            changeset '**/go.mod'
        }
    }
}

beforeAgent — the one that saves money

Requires — Plugin: Workspace Cleanup · Plugin: JUnit · Plugin: Slack Notification · Plugin: Email Extension

post runs after a stage or the whole pipeline, keyed on result. It is where cleanup and notification belong.

The conditions

post {
    always       { echo 'runs no matter what' }
    success      { echo 'only if SUCCESS' }
    failure      { echo 'only if FAILURE' }
    unstable     { echo 'only if UNSTABLE' }
    aborted      { echo 'manually stopped or timed out' }
    unsuccessful { echo 'failure OR unstable OR aborted' }
    changed      { echo 'result differs from the previous build' }
    fixed        { echo 'previous failed, this one passed' }
    regression   { echo 'previous passed, this one did not' }
    cleanup      { echo 'always, and last of all' }
}

Stage-level post

stage('Test') {
    steps { sh 'make test' }
    post {
        always {
            junit 'reports/**/*.xml'
            archiveArtifacts artifacts: 'reports/**', allowEmptyArchive: true
        }
        failure {
            archiveArtifacts artifacts: 'logs/**', allowEmptyArchive: true
        }
    }
}

Publishing test results in post { always { … } } matters: if the test command exits non-zero the block stops, but still runs — so you still get the report telling you test failed.

Requires — Plugin: Active Choices · On the machine you run this from: jq

Parameters turn a job into something you can drive: which branch, which environment, whether to skip tests.

Declaring them

pipeline {
    agent any

    parameters {
        string(
            name: 'BRANCH',
            defaultValue: 'main',
            description: 'Branch to build'
        )
        choice(
            name: 'ENVIRONMENT',
            choices: ['dev', 'staging', 'production'],
            description: 'Where to deploy'
        )
        booleanParam(
            name: 'RUN_TESTS',
            defaultValue: true,
            description: 'Uncheck to skip the test stage'
        )
        text(
            name: 'RELEASE_NOTES',
            defaultValue: '',
            description: 'Notes for the changelog'
        )
        password(
            name: 'OVERRIDE_TOKEN',
            defaultValue: '',
            description: 'Only for break-glass runs'
        )
    }

    stages {
        stage('Show') {
            steps {
                echo "branch=${params.BRANCH} env=${params.ENVIRONMENT}"
            }
        }
        stage('Test') {
            when { expression { params.RUN_TESTS } }
            steps { sh 'make test' }
        }
    }
}

Read them as params.NAME. They are also injected as environment variables, but params. is explicit and does not collide.

Requires — Locally: Java + jenkins-cli.jar

Two features turn pipeline development from painful into quick. Both are on the build page and both are easy to miss.

Replay

Open any completed build → Replay in the left sidebar.

You get an editable copy of the Jenkinsfile that ran. Change it, click Run, and a new build starts with your edited script — without committing anything.

The normal loop is edit → commit → push → wait for webhook → wait for build → find the typo. Replay collapses that to edit → run.

Things to know:

• The replayed build uses the same commit. Only the script differs.
• Shared library code is editable too — Replay shows a box per library file. This is the fastest way to develop a shared library.
• The replayed script is stored with that build, so you can see what ran.
• Copy your change back into the real Jenkinsfile and commit it. Replayed scripts are not persisted to your repo; forgetting this is a classic way to lose twenty minutes of work.

Restart from Stage

Open a completed build → Restart from Stage → choose a stage. Re-runs from there, same commit, same pipeline.

If a fifteen-minute build failed in the two-minute deploy stage because a token expired, restart at deploy.

Caveats:

• Declarative only.
• The workspace may have been cleaned or the agent may be gone. Restarting a stage that depends on earlier build output fails unless you stash. This is the main reason stash/unstash earns its place even in single-agent pipelines.
• Keep stashes around for restarts:

options {
    preserveStashes(buildCount: 5)
}

Replaying from the command line

Requires — On the machine you run this from: jq

Nobody memorises Jenkins step parameters. There are hundreds of steps, each contributed by a plugin, each with its own options. The Pipeline Syntax page generates the code for you.

Getting there

From any Pipeline job: left sidebar → Pipeline Syntax. Or directly at /job/<name>/pipeline-syntax/.

Snippet Generator

Pick a step from the dropdown, fill in the form, click Generate Pipeline Script, and it emits the exact Groovy.

This is the correct way to write anything non-obvious. Example — a git checkout with credentials, generated:

git branch: 'main',
    credentialsId: 'github-deploy-key',
    url: 'git@github.com:acme/api.git'

And a full checkout with options you would never guess:

checkout([
  $class: 'GitSCM',
  branches: [[name: '*/main']],
  extensions: [
    [$class: 'CloneOption', depth: 1, shallow: true, noTags: true],
    [$class: 'CleanBeforeCheckout']
  ],
  userRemoteConfigs: [[
    credentialsId: 'github-deploy-key',
    url: 'git@github.com:acme/api.git'
  ]]
])

Shallow clone with depth: 1 on a large repository is often the single biggest build-time win available. You would not find that syntax by guessing.

The other tabs

• Declarative Directive Generator — the same idea for directives rather than steps: agent, options, when, triggers. Very useful for conditions, whose syntax is fiddly.

Every pipeline starts by getting the code. On a small repo any approach works; on a large one, checkout is often the slowest stage and the options below are where the time goes.

The three ways

// 1. Implicit — with "Pipeline script from SCM", Jenkins checks out to find
//    the Jenkinsfile. `checkout scm` repeats that same configuration.
checkout scm

// 2. The git step — simple, limited
git branch: 'main',
    credentialsId: 'github-deploy-key',
    url: 'git@github.com:acme/api.git'

// 3. The full checkout step — everything
checkout([
  $class: 'GitSCM',
  branches: [[name: '*/main']],
  userRemoteConfigs: [[
    credentialsId: 'github-deploy-key',
    url: 'git@github.com:acme/api.git'
  ]],
  extensions: []
])

Use checkout scm in a Multibranch pipeline — it checks out the right branch or PR automatically. Use the explicit form when you need a second repository or non-default options.

The extensions that matter

extensions: [
  // Shallow clone — usually the single biggest win
  [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30],

  // Only fetch the branch you need
  [$class: 'CloneOption', honorRefspec: true],

  // Start from a clean tree — slower, but removes a class of ghost failures
  [$class: 'CleanBeforeCheckout'],
  [$class: 'CleanCheckout'],

  // Submodules
  [$class: 'SubmoduleOption',
   recursiveSubmodules: true,
   parentCredentials: true,
   shallow: true,
   depth: 1],

  // Check out into a subdirectory — essential when using two repos
  [$class: 'RelativeTargetDirectory', relativeTargetDir: 'app'],

  // Sparse checkout — only some paths, huge win in a monorepo
  [$class: 'SparseCheckoutPaths',
   sparseCheckoutPaths: [[path: 'services/api/'], [path: 'shared/']]],

  // Keep a local reference clone to avoid re-downloading history
  [$class: 'CloneOption', reference: '/var/cache/git/api.git']
]

shallow: true, depth: 1 on a repository with ten years of history can turn a four-minute checkout into fifteen seconds. The catch: shallow clones break anything that needs history — git describe --tags, changelog generation, git diff origin/main...HEAD. If you need history, use depth: 50 rather than full.

parentCredentials: true on submodules is the fix for "submodule checkout works locally but fails in Jenkins" — without it Jenkins does not reuse your credentials for the submodule URLs.

reference: points at a bare clone kept on the agent. New workspaces borrow objects from it instead of re-downloading. On a large monorepo with many jobs this is transformative:

# on each agent, once, kept fresh by a nightly job
sudo git clone --mirror git@github.com:acme/monorepo.git /var/cache/git/monorepo.git
sudo git -C /var/cache/git/monorepo.git remote update --prune

Two repositories in one pipeline

stage('Checkout') {
    steps {
        dir('app') {
            checkout scm
        }
        dir('infra') {
            checkout([
              $class: 'GitSCM',
              branches: [[name: '*/main']],
              userRemoteConfigs: [[
                credentialsId: 'github-deploy-key',
                url: 'git@github.com:acme/infra.git'
              ]]
            ])
        }
    }
}

dir() is cleaner than RelativeTargetDirectory and works for both.

What checkout gives you

After a Git checkout these are populated:

GIT_COMMIT         full SHA
GIT_PREVIOUS_COMMIT
GIT_PREVIOUS_SUCCESSFUL_COMMIT
GIT_BRANCH         origin/main
GIT_LOCAL_BRANCH   main
GIT_URL
GIT_AUTHOR_NAME / GIT_COMMITTER_NAME

GIT_PREVIOUS_SUCCESSFUL_COMMIT is the useful one — the diff since the last green build is exactly the set of changes that could have broken things.

Capturing the return value gives you the same data as a map, which is more reliable than the environment variables in some configurations:

script {
    def scmVars = checkout scm
    env.GIT_COMMIT = scmVars.GIT_COMMIT
    echo "building ${scmVars.GIT_COMMIT.take(7)} on ${scmVars.GIT_BRANCH}"
}

Takeaway: shallow clone and reference repositories are where checkout time goes. Use dir() for multiple repos, and remember shallow clones break git describe and three-dot diffs.

Requires — Plugin: Generic Webhook Trigger · Plugin: GitLab · Locally: ngrok (lab only)

Polling asks "has anything changed?" every few minutes. A webhook is the repository telling Jenkins the moment something does. Webhooks are faster, cheaper, and the only sane choice at scale.

GitHub → Jenkins

On the Jenkins side:

Install GitHub plugin (part of the suggested set). Then Manage Jenkins → System → GitHub → Add GitHub Server, with a credential holding a personal access token if you want Jenkins to manage hooks itself.

Confirm Jenkins URL is correct — GitHub calls back to it.

On the GitHub side:

Repository → Settings → Webhooks → Add webhook:

• Payload URL: https://ci.example.com/github-webhook/ — the trailing slash matters.
• Content type: application/json
• Secret: a shared secret (strongly recommended)
• Events: Just the push event, plus Pull requests if you build PRs.

For a Multibranch job the endpoint is the same; Jenkins routes by repository URL.

On the job side:

• Freestyle / single Pipeline: tick GitHub hook trigger for GITScm polling.
• Multibranch: nothing to tick — the folder listens automatically.

GitLab

Install GitLab plugin. Endpoint:

https://ci.example.com/project/<job-name>

In the pipeline:

triggers {
    gitlab(
        triggerOnPush: true,
        triggerOnMergeRequest: true,
        branchFilterType: 'All'
    )
}

Requires — Plugin: Pipeline: Multibranch (+ a branch source such as GitHub)

A Multibranch Pipeline scans a repository, finds every branch containing a Jenkinsfile, and creates a job for each. Branches appear when created and disappear when deleted.

This is how you should run CI for a real repository. It is not an advanced feature; it is the default.

Creating one

Dashboard → New Item → name → Multibranch Pipeline → OK.

Branch Sources → Add source → Git (or GitHub / Bitbucket for richer integration):

• Project Repository: git@github.com:acme/api.git
• Credentials: your deploy key or token
• Behaviours — the important part:
  • Discover branches — all, or only those that are not PRs
  • Discover pull requests from origin — merge or head strategy
  • Discover pull requests from forks — with a trust policy
  • Filter by name (with wildcards) — e.g. include main release/* PR-*
  • Clean before checkout, Advanced clone behaviours (shallow, reference)

Build Configuration → Mode: by Jenkinsfile, Script Path: Jenkinsfile.

Scan Multibranch Pipeline Triggers → tick Periodically if not otherwise run, interval 1 day. This is a safety net; webhooks do the real work.

Orphaned Item Strategy → keep 10 old branches for 7 days, so a deleted branch's history does not vanish instantly.

What you get

The job becomes a folder. Inside it, one job per branch and per PR:

/job/api/                    the multibranch folder
/job/api/job/main/           the main branch
/job/api/job/PR-431/         a pull request
/job/api/job/release%2F2.1/  a branch with a slash — URL-encoded

Requires — Plugin: Pipeline: Multibranch (+ a branch source such as GitHub) · Plugin: SSH Build Agents · On the agent: gh (GitHub CLI)

Tag-driven releases are the cleanest release trigger: you tag a commit, CI builds and publishes that exact thing.

Discovering tags in Multibranch

In the Multibranch job: Branch Sources → Behaviours → Add → Discover tags.

Tags then appear as jobs alongside branches. Combine with a filter so you do not build every historical tag:

Add → Filter by name (with wildcards) → Include: main release/* v*

Reacting to a tag in the pipeline

stage('Release') {
    when {
        beforeAgent true
        buildingTag()
    }
    steps {
        sh 'make release'
    }
}

stage('Release semver only') {
    when { tag pattern: 'v\\d+\\.\\d+\\.\\d+', comparator: 'REGEXP' }
    steps { sh "make publish VERSION=${env.TAG_NAME}" }
}

env.TAG_NAME holds the tag. buildingTag() is true for any tag build.

Deriving a version from git

environment {
    VERSION = sh(
        script: 'git describe --tags --always --dirty',
        returnStdout: true
    ).trim()
}

This requires tags and history, so it is incompatible with shallow: true, noTags: true. If you use git describe, fetch tags:

Jenkins uses cron syntax with one important extension.

Where

• Pipeline: a triggers block.
• Freestyle: Configure → Build Triggers → Build periodically.

pipeline {
    agent any
    triggers {
        cron('H 2 * * *')          // nightly, around 02:00
    }
    stages { /* … */ }
}

The fields

MINUTE HOUR DAY_OF_MONTH MONTH DAY_OF_WEEK
0-59   0-23  1-31        1-12  0-7 (0 and 7 are Sunday)

H — the part that matters

H means "hash" — Jenkins picks a value deterministically from the job name, and that job always gets the same value.

cron('0 2 * * *')    // every job with this line fires at exactly 02:00
cron('H 2 * * *')    // each job fires at its own fixed minute between 02:00 and 02:59

The first version is how you build a thundering herd: fifty jobs all starting on the stroke of the hour, queueing behind each other, all finishing late. Use H for the minute field, always. There is essentially no case where you want fifty jobs to start in the same second.

H works in ranges too:

Requires — Plugin: Copy Artifact

Real pipelines chain: build, then test, then deploy — often as separate jobs owned by different teams.

The build step

stage('Trigger deploy') {
    steps {
        build job: 'deploy-api',
              parameters: [
                  string(name: 'VERSION', value: "${env.BUILD_NUMBER}"),
                  string(name: 'ENVIRONMENT', value: 'staging'),
                  booleanParam(name: 'SMOKE_TEST', value: true)
              ],
              wait: true,
              propagate: true
    }
}

• wait: true — block until the downstream job finishes. Default.
• propagate: true — if downstream fails, fail this build too. Default.
• wait: false — fire and forget. Useful for notifications; the return value is then useless.

Using the downstream result

script {
    def down = build job: 'integration-tests',
                     parameters: [string(name: 'SHA', value: env.GIT_COMMIT)],
                     propagate: false

    echo "downstream finished: ${down.result} (#${down.number})"

    if (down.result == 'UNSTABLE') {
        unstable('Integration tests were flaky')
    } else if (down.result == 'FAILURE') {
        error('Integration tests failed')
    }
}

A monorepo where every push runs every pipeline wastes enormous amounts of time. The goal is to build only the affected parts.

The reliable path filter

when { changeset } is unreliable on first and retried builds (Chapter 4). Compute the diff explicitly:

def changedIn(String path) {
    return sh(
        returnStatus: true,
        script: "git diff --quiet origin/${env.CHANGE_TARGET ?: 'main'}...HEAD -- '${path}'"
    ) == 1
}

stage('API') {
    when { beforeAgent true; expression { changedIn('services/api/**') } }
    steps { sh 'make -C services/api build test' }
}

stage('Web') {
    when { beforeAgent true; expression { changedIn('services/web/**') } }
    steps { sh 'make -C services/web build test' }
}

The three-dot form origin/main...HEAD diffs against the merge base, which is what you want — it ignores changes that landed on main since you branched.

This needs history, so combine with depth: 50 rather than depth: 1, and make sure origin/main is actually fetched:

sh 'git fetch --no-tags --depth=50 origin +refs/heads/main:refs/remotes/origin/main'

Building a dynamic matrix

Compute the affected services, then generate parallel branches:

The controller holds every credential, every job configuration, and the master encryption key. A build running on it can read all of that.

What a build on the controller can do

If a job has a shell step and runs on the built-in node:

cat /var/lib/jenkins/secrets/master.key
cat /var/lib/jenkins/credentials.xml
ls /var/lib/jenkins/jobs/*/config.xml

That is every secret in your organisation's deployment path, readable by anyone who can edit any job — or by anyone who can open a pull request, if you build PRs.

It is not a subtle privilege escalation. It is the whole store, in plain reach.

The fix

Manage Jenkins → Nodes → Built-In Node → Configure → Number of executors: 0

Then confirm:

// Manage Jenkins → Script Console
println Jenkins.instance.numExecutors      // expect 0

With zero executors, a pipeline with agent any and no other node simply queues forever — which is the correct, loud failure. Add an agent before you do this, or do both in one sitting.

The other reasons

Beyond security:

• Stability. A build that exhausts memory or fills the disk takes Jenkins down with it. On an agent it takes down one build.
• Scaling. The controller is a single JVM. Builds compete with the UI, and the UI loses.
• Cleanliness. Build tools, compilers and language runtimes accumulate on the controller and become an upgrade problem.

The label discipline that follows

Once builds are on agents, agent any becomes a liability — it means "whichever machine happens to be free", which is how a build that needs Docker lands on a machine without it.

Give agents labels describing capabilities, and ask for capabilities:

agent { label 'linux && docker' }
agent { label 'windows' }
agent { label 'gpu' }

Label expressions support &&, ||, !, and parentheses:

agent { label 'linux && (docker || podman)' }
agent { label 'linux && !legacy' }

Naming agents after capabilities rather than machines means you can replace hardware without touching pipelines.

Takeaway: set controller executors to zero, and label agents by capability rather than identity. This is the highest-value security change in Jenkins.

Requires — Plugin: SSH Build Agents · On the agent: docker

An SSH agent is a machine Jenkins logs into and runs a Java process on. It is the workhorse for permanent build machines.

Prepare the machine

# on the agent
sudo adduser --disabled-password --gecos "" jenkins
sudo mkdir -p /home/jenkins/agent
sudo chown -R jenkins:jenkins /home/jenkins

# Java — must be compatible with the controller
sudo apt-get install -y openjdk-21-jre-headless

# whatever the builds need
sudo apt-get install -y git make docker.io
sudo usermod -aG docker jenkins

Key-based access

Generate a dedicated key on the controller (or anywhere), and add the public half to the agent:

ssh-keygen -t ed25519 -C "jenkins-agent" -f ~/.ssh/jenkins_agent -N ""
ssh-copy-id -i ~/.ssh/jenkins_agent.pub jenkins@agent-host

Verify by hand before involving Jenkins — this eliminates half the possible failures:

ssh -i ~/.ssh/jenkins_agent jenkins@agent-host 'java -version && echo OK'

Add the credential

Manage Jenkins → Credentials → System → Global credentials → Add Credentials

• Kind: SSH Username with private key
• ID: (set this explicitly — a generated UUID is unusable in code)

Requires — Plugin: Docker Pipeline · On the agent: docker

Not every agent can be reached from the controller. A machine behind NAT, a laptop, a container in another network — these connect to Jenkins instead.

Inbound (JNLP) agents

Manage Jenkins → Nodes → New Node → Permanent Agent → Launch method: Launch agent by connecting it to the controller.

Save, then open the node page. Jenkins shows the command to run on the agent:

curl -sO https://ci.example.com/jnlpJars/agent.jar

java -jar agent.jar \
  -url https://ci.example.com/ \
  -secret 8f3c9a…  \
  -name "build-agent-4" \
  -workDir "/home/jenkins/agent"

Modern Jenkins prefers WebSocket, which tunnels over the same HTTPS port and works through proxies:

java -jar agent.jar \
  -url https://ci.example.com/ \
  -secret 8f3c9a… \
  -name "build-agent-4" \
  -workDir "/home/jenkins/agent" \
  -webSocket

With -webSocket you do not need port 50000 open at all — worth doing for that reason alone. It does require the reverse proxy to forward Upgrade/Connection headers (Chapter 1).

Enable the protocol at Manage Jenkins → Security → Agents — set the TCP port to Disable if everything uses WebSocket.

Running it as a service

Requires — Plugin: Kubernetes · On the agent: kubectl + cluster access

The Kubernetes plugin creates a pod for each build and deletes it afterwards. Every build gets a clean, isolated environment, and capacity scales with the cluster.

Setup

Install Kubernetes plugin. Then Manage Jenkins → Clouds → New cloud → Kubernetes:

• Kubernetes URL — blank if Jenkins runs in the cluster (it uses the service account).
• Kubernetes Namespace — e.g. jenkins-agents.
• Jenkins URL — reachable from inside the cluster, e.g. http://jenkins.jenkins.svc.cluster.local:8080. Getting this wrong is the usual reason pods start and never connect.
• Test Connection — do this before anything else.

A pod template in the pipeline

Defining the pod in the repository is far better than in the UI:

pipeline {
    agent {
        kubernetes {
            yaml '''
apiVersion: v1
kind: Pod
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  containers:
    - name: node
      image: node:20-alpine
      command: ["sleep"]
      args: ["infinity"]
      resources:
        requests: { cpu: "500m", memory: "1Gi" }
        limits:   { cpu: "2",    memory: "4Gi" }
    - name: kaniko
      image: gcr.io/kaniko-project/executor:v1.23.2-debug
      command: ["sleep"]
      args: ["infinity"]
      volumeMounts:
        - name: docker-config
          mountPath: /kaniko/.docker
  volumes:
    - name: docker-config
      secret:
        secretName: registry-credentials
'''
        }
    }

    stages {
        stage('Build') {
            steps {
                container('node') {
                    sh 'npm ci && npm run build'
                }
            }
        }
        stage('Image') {
            steps {
                container('kaniko') {
                    sh '''
                      /kaniko/executor \
                        --context $(pwd) \
                        --dockerfile Dockerfile \
                        --destination registry.example.com/api:$BUILD_NUMBER
                    '''
                }
            }
        }
    }
}

Requires — Plugin: Artifact Manager on S3 · On the agent: aws CLI + credentials

When stages run on different agents, they do not share a workspace. stash and unstash move files between them.

The basic pattern

pipeline {
    agent none

    stages {
        stage('Build') {
            agent { label 'linux' }
            steps {
                sh 'make build'
                stash name: 'binaries', includes: 'dist/**'
            }
        }

        stage('Test on Windows') {
            agent { label 'windows' }
            steps {
                unstash 'binaries'
                bat 'run-tests.bat'
            }
        }

        stage('Deploy') {
            agent { label 'deploy' }
            steps {
                unstash 'binaries'
                sh 'make deploy'
            }
        }
    }
}

Options

stash name: 'src',
      includes: 'src/**,Makefile',
      excludes: '**/*.o,**/node_modules/**',
      useDefaultExcludes: true,
      allowEmpty: false

allowEmpty: false (the default) fails the build if the pattern matched nothing — usually what you want, because an empty stash produces a confusing failure two stages later instead of here.

What stash is not

Stash is not an artifact store. Stashes:

Requires — Plugin: Workspace Cleanup · On the agent: docker · On the machine you run this from: jq

Agents fail in boring, predictable ways. Most outages are disk, and most of the rest are drift.

Disk

Workspaces accumulate. Docker images accumulate faster.

Clean workspaces in the pipeline:

post {
    cleanup {
        cleanWs()
    }
}

Or selectively:

cleanWs(
    deleteDirs: true,
    patterns: [
        [pattern: 'dist/**',         type: 'INCLUDE'],
        [pattern: '.gradle/caches/**', type: 'EXCLUDE']   // keep the cache
    ]
)

Set a disk threshold so Jenkins stops scheduling onto a full agent rather than failing builds on it. Manage Jenkins → Nodes → (agent) → Configure → Node Properties → Disk space monitoring thresholds, or globally under Manage Jenkins → System → Node Monitoring.

Prune Docker regularly. This is usually the real disk consumer:

# a cron job on each Docker-capable agent
docker system prune -af --filter "until=168h"
docker volume prune -f

A cleanup job is the pragmatic answer for workspaces:

Jenkins has a built-in secret store. Everything else in this chapter depends on using it properly.

Where

Manage Jenkins → Credentials. You will see a tree:

Stores scoped to Jenkins
  └── System
        └── Global credentials (unrestricted)
Stores scoped to <folder>
  └── Folder
        └── <folder> credentials

Add Credentials on any store.

The kinds

Set the ID yourself

The ID field is optional and defaults to a random UUID. Always set it, to something descriptive and stable:

github-deploy-key
registry-prod-credentials
npm-publish-token
kubeconfig-staging

Because your pipelines reference credentials by ID:

withCredentials([usernamePassword(credentialsId: 'registry-prod-credentials', …)])

A UUID in a Jenkinsfile is unreadable, and it makes the pipeline impossible to run against a second Jenkins.

Scope

• Global — usable by any job and by the controller itself.
• System — usable only by Jenkins itself (agent connections), not by jobs. Use this for agent SSH keys so no build can read them.

Folder scope is the one that matters. A credential added to a folder store is visible only to jobs inside that folder. That is how you stop team A's pipeline reading team B's production deploy key.

The pattern for a shared Jenkins:

/team-payments/          ← folder, holds payments' credentials
   /api-build
   /api-deploy
/team-search/            ← folder, holds search's credentials
   /indexer-build

Give each team Credentials/View and Credentials/Update on their own folder only.

How it is stored

Credentials live encrypted in $JENKINS_HOME/credentials.xml, encrypted with the key material in $JENKINS_HOME/secrets/. Two consequences:

1. secrets/ plus credentials.xml is everything. Anyone with both has all your secrets. This is why controller filesystem access is equivalent to total compromise.
2. Your backup contains your secrets. Encrypt it, restrict it, and treat it like the secret store it is.

Listing what you have

// Manage Jenkins → Script Console — ids only, never values
import com.cloudbees.plugins.credentials.CredentialsProvider
import com.cloudbees.plugins.credentials.common.StandardCredentials

CredentialsProvider.lookupCredentials(
    StandardCredentials.class, Jenkins.instance, null, null
).each {
    println "${it.id}\t${it.class.simpleName}\t${it.description}"
}

Periodically auditing this list is worth doing. Old credentials accumulate, and an unused token nobody remembers is still a live token.

External secret managers

For anything beyond a small team, keep secrets in a real manager and let Jenkins fetch them:

• HashiCorp Vault plugin — short-lived credentials, per-build.
• AWS Secrets Manager / Azure Key Vault / Google Secret Manager plugins — the credential store becomes a view onto them.
• Kubernetes Credentials Provider — Kubernetes Secrets appear as Jenkins credentials.

The advantage is not just storage. It is rotation and audit: you can rotate centrally, and you can see who read what and when. Jenkins' own store gives you neither.

Takeaway: always set a readable credential ID, scope credentials to folders rather than globally, and remember that credentials.xml plus secrets/ is your entire secret store in two files.

Requires — Plugin: SSH Build Agents · On the agent: docker · On the agent: kubectl + cluster access

There are two mechanisms, and a set of ways to leak a secret with either.

withCredentials — explicit and scoped

stage('Publish') {
    steps {
        withCredentials([
            usernamePassword(
                credentialsId: 'registry-prod-credentials',
                usernameVariable: 'REG_USER',
                passwordVariable: 'REG_PASS'
            ),
            string(
                credentialsId: 'npm-publish-token',
                variable: 'NPM_TOKEN'
            ),
            sshUserPrivateKey(
                credentialsId: 'github-deploy-key',
                keyFileVariable: 'SSH_KEY',
                usernameVariable: 'SSH_USER'
            ),
            file(
                credentialsId: 'kubeconfig-staging',
                variable: 'KUBECONFIG'
            )
        ]) {
            sh '''
                set -euo pipefail
                echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
                npm publish
                kubectl --kubeconfig "$KUBECONFIG" rollout status deploy/api
            '''
        }
    }
}

The secret exists only inside the block. That scoping is the point.

The environment shorthand

Manage Jenkins → Security has two halves. This is the first: the Security Realm, which decides who is allowed in.

The options

Jenkins' own user database — accounts stored in JENKINS_HOME. Fine for a small team or a personal instance.

• Uncheck Allow users to sign up unless you truly want open registration. This is on by default in some setups and is how a public Jenkins acquires uninvited administrators.

LDAP / Active Directory — the standard for a company. Configure the server, bind DN, and the user/group search bases. Use Test LDAP settings before saving; a wrong bind DN locks everyone out including you.

SAML / OIDC — SSO through Okta, Entra, Google, Keycloak. Preferred where available: central offboarding, MFA, and no passwords in Jenkins at all.

GitHub / Google OAuth — practical for open-source or small teams. Restrict by organisation membership, or anyone with a GitHub account can log in.

Unix user/group database — delegates to PAM on the controller.

API tokens

Users authenticate to the API with a token, not their password:

Click your name (top right) → Configure → API Token → Add new Token → name it → Generate. Copy it now; it is not shown again.

curl -u "alice:11a2b3c4d5e6f7..." https://ci.example.com/api/json

Tokens are per-user and revocable individually, which is why they beat passwords for scripts. Rotate them like any other credential.

Locked out?

It happens — usually a bad LDAP change. Recovery is on the filesystem:

sudo systemctl stop jenkins
sudo nano /var/lib/jenkins/config.xml

Set:

Requires — Plugin: Role-based Authorization Strategy

The second half of Manage Jenkins → Security: the Authorization strategy.

The strategies

Anyone can do anything — no authorisation. Only ever acceptable on a disconnected laptop.

Legacy mode — admin role for administrators, read for everyone else.

Logged-in users can do anything — the wizard's default. Every authenticated user is effectively an administrator, including script console access. Fine for one person; wrong the moment there are two.

Matrix-based security — a grid of permissions by user or group. Explicit and auditable.

Project-based Matrix Authorization — the matrix, plus per-job overrides.

Role-based Authorization Strategy (plugin) — named roles with pattern-matched scope. The right answer for anything with teams.

The permissions worth understanding

Requires — Plugin: Pipeline: Shared Groovy Libraries

Pipeline code runs in a sandbox that blocks dangerous calls. Understanding it saves confusion and prevents you from disabling the only thing standing between a developer and your controller.

What it does

Any Jenkinsfile written by a user runs sandboxed. Calls outside an allowlist are rejected:

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException:
Scripts not permitted to use staticMethod java.lang.System getenv

The block is on method calls, not syntax. Ordinary pipeline work is unaffected.

Approving something

Manage Jenkins → In-process Script Approval

Pending signatures appear here with Approve / Deny. Approving adds that method to the allowlist for everyone, permanently.

Read what you are approving. Some signatures are harmless:

method java.lang.String toLowerCase

Others hand over the machine:

staticMethod jenkins.model.Jenkins getInstance
method java.lang.Runtime exec java.lang.String
new java.io.File java.lang.String

Approving Runtime.exec means any pipeline, from any user, can run any command as the Jenkins user. That is the entire security model gone, granted from a page that looks like a to-do list.

Working within the sandbox instead

Nearly every rejection has a supported alternative:

A checklist, roughly in order of how much it reduces risk.

The high-value items

1. Zero executors on the controller. Covered in Chapter 6. Builds on the controller can read every secret.

2. Keep Jenkins and plugins patched. Plugin CVEs are the main attack surface and are exploited quickly after disclosure. Subscribe to the Jenkins security advisories mailing list. The dashboard banner also flags vulnerable plugins.

3. Do not expose it to the internet. Put it behind a VPN or an identity-aware proxy. If it must be public, at minimum: SSO with MFA, no anonymous read, and a WAF in front.

4. Restrict Overall/Administer to a named, audited list. Review it quarterly.

5. Scope credentials to folders. Chapter opener. A flat global store means every job can use every secret.

6. Turn off open sign-up and remove accounts on offboarding — automatic with SSO, manual otherwise.

The configuration items

7. CSRF protection — on by default; verify at Manage Jenkins → Security.

8. Agent protocols — disable the legacy TCP port if all agents use WebSocket: Security → Agents → TCP port for inbound agents: Disable.

9. Markup formatter — leave at Plain text. Setting it to raw HTML lets anyone who can edit a description inject script into pages other users load.

10. CLI over remoting — disabled by default on modern versions. Confirm it stays that way; it has a poor security history.

11. Build authorisation. Install Authorize Project so builds run as a specific user rather than as SYSTEM. Without it, a build has more authority than the person who triggered it. Manage Jenkins → Security → Access Control for Builds.

12. Set the Jenkins URL — beyond usability, a wrong URL can send agents and callbacks somewhere unintended.

The operational items

13. Back up JENKINS_HOME and test a restore. Chapter 10.

14. Log and monitor. The Audit Trail plugin records who did what. Ship it somewhere you actually read.

Requires — Plugin: Configuration as Code · Plugin: Audit Trail · Plugin: Job Configuration History · On the machine you run this from: jq

When something goes wrong you need to know who changed what, and when.

Audit Trail plugin

Install Audit Trail. Manage Jenkins → System → Audit Trail:

• Log location — a file, syslog, or the Jenkins log.
• URL patterns to log — the default catches configuration changes and build triggers.
• Log build causes — on.

.*/(?:configSubmit|doDelete|postBuildResult|enable|disable|cancelQueue|stop|toggleLogKeep|doWipeOutWorkspace|createItem|createView|toggleOffline|cancelQuietDown|quietDown|restart|exit|safeExit)

Ship the file to your log system. An audit log nobody reads is a compliance artefact, not a control.

Job Configuration History

The Job Configuration History plugin keeps a versioned diff of every job and global configuration change.

Job → Job Config History, or Manage Jenkins → Job Config History for global changes. You get a side-by-side diff and a restore button.

This is the single most useful plugin when someone says "it worked yesterday" about a Freestyle job. It also answers "who changed the global SMTP settings" — which is otherwise unknowable.

Who ran this build

Every build records its cause:

script {
    def causes = currentBuild.getBuildCauses()
    echo "Triggered by: ${causes*.shortDescription.join(', ')}"

    def user = currentBuild.getBuildCauses('hudson.model.Cause$UserIdCause')
    if (user) {
        echo "Started by ${user[0].userId}"
    }
}

Requires — Plugin: Artifact Manager on S3 · On the agent: aws CLI + credentials

archiveArtifacts attaches files to a build so you can download them later.

The step

stage('Package') {
    steps {
        sh 'make package'
        archiveArtifacts(
            artifacts: 'dist/*.tar.gz,dist/*.sha256',
            fingerprint: true,
            onlyIfSuccessful: true,
            allowEmptyArchive: false
        )
    }
}

• fingerprint: true — records an MD5 of each file so Jenkins can track it across jobs. See the fingerprinting note below.
• onlyIfSuccessful: true — do not keep artifacts from failed builds. Usually right; occasionally you want failure artifacts, in which case archive them from post { failure } instead.
• allowEmptyArchive: false — fail if the pattern matched nothing. Keep this false so a broken build path fails loudly rather than silently producing nothing.

Where they appear

On the build page, under Build Artifacts. And at a predictable URL:

/job/api-build/142/artifact/dist/app.tar.gz
/job/api-build/lastSuccessfulBuild/artifact/dist/app.tar.gz

That second form is the useful one — a stable URL to the latest good build, which other systems can pull from without an API client:

curl -fSL -u "$USER:$TOKEN" \
  -o app.tar.gz \
  https://ci.example.com/job/api-build/lastSuccessfulBuild/artifact/dist/app.tar.gz

Add /*zip*/archive.zip to any artifact path to download a directory as a zip.

What not to archive

Artifacts live on the controller's disk, forever, per build. This is the most common cause of a Jenkins controller running out of space.

Do not archive:

• node_modules/, .venv/, vendor/ — dependency trees. Enormous and reproducible.
• Whole workspaces. Someone always tries artifacts: '**'.
• Container images. They belong in a registry.
• Anything you have already pushed elsewhere. If it went to S3 or Artifactory, archiving it again just doubles the storage.

Do archive:

• The built artefact itself, when small.
• Test reports and coverage HTML.
• Logs that help diagnose a failure.

Retention

options {
    buildDiscarder(logRotator(
        numToKeepStr: '50',          // keep 50 builds' logs and metadata
        artifactNumToKeepStr: '5'    // keep artifacts for only the last 5
    ))
}

Splitting these two is the trick: build history is cheap, artifacts are not. Keeping fifty builds of history and five sets of artifacts gives you a useful trend graph without the disk cost.

Offloading to S3

On a busy controller, install Artifact Manager on S3. Once configured, archiveArtifacts and stash transparently write to S3 instead of controller disk. Pipelines do not change; the disk problem goes away.

Alternatively, do it explicitly and skip Jenkins storage entirely:

sh '''
  set -euo pipefail
  aws s3 cp dist/app.tar.gz \
    "s3://artifacts/${JOB_NAME}/${BUILD_NUMBER}/app.tar.gz"
'''

Fingerprinting

With fingerprint: true, Jenkins records a hash for each file. When another job archives or uses a file with the same hash, Jenkins links them.

The payoff is on the artifact's page: "this file was produced by build #142 of api-build and used by build #87 of deploy-prod". Answering "which build is running in production?" becomes a click rather than an investigation.

It costs almost nothing. Turn it on.

Takeaway: archive the artefact and reports, never dependency trees, split numToKeepStr from artifactNumToKeepStr, and turn on fingerprinting to trace a binary across jobs.

Requires — Plugin: JUnit · Plugin: Coverage

A build that says "failed" is much less useful than one that says "3 tests failed, here they are".

JUnit XML

Almost every test framework can emit JUnit XML, which is the format Jenkins understands.

stage('Test') {
    steps {
        sh 'make test'
    }
    post {
        always {
            junit(
                testResults: '**/target/surefire-reports/*.xml',
                allowEmptyResults: false,
                keepLongStdio: true,
                skipPublishingChecks: false
            )
        }
    }
}

post { always } is essential. If tests fail, make test exits non-zero and the steps block stops — but post still runs, so you still get the report. Putting junit in steps after the test command means you never see results when tests fail, which is exactly when you need them.

allowEmptyResults: false makes a missing report a failure. Without it, a test command that crashed before writing any XML produces a green build with zero tests — the worst possible outcome.

Producing the XML

# pytest
pytest --junitxml=reports/junit.xml

# jest
jest --reporters=default --reporters=jest-junit
# JEST_JUNIT_OUTPUT_DIR=reports

# go
go test ./... -v 2>&1 | go-junit-report > reports/junit.xml

# maven / gradle — automatic
mvn test          # target/surefire-reports/*.xml
gradle test       # build/test-results/test/*.xml

# dotnet
dotnet test --logger "junit;LogFilePath=reports/junit.xml"

# rspec
bundle exec rspec --format RspecJunitFormatter --out reports/junit.xml

Requires — Plugin: Warnings Next Generation · Plugin: OWASP Dependency-Check · On the agent: trivy · On the agent: gitleaks · +1 more, named inline

The Warnings Next Generation plugin parses compiler and linter output into trends, annotated source, and quality gates. It replaced a dozen older plugins and understands well over a hundred formats.

Basic use

stage('Lint') {
    steps {
        sh 'make lint > lint.log 2>&1 || true'
    }
    post {
        always {
            recordIssues(
                tools: [
                    checkStyle(pattern: 'reports/checkstyle.xml'),
                    spotBugs(pattern: 'reports/spotbugs.xml'),
                    esLint(pattern: 'reports/eslint.xml'),
                    pyLint(pattern: 'reports/pylint.log'),
                    gcc(pattern: 'build/compile.log')
                ],
                aggregatingResults: true
            )
        }
    }
}

Note || true on the lint command — you want the plugin to decide the build result from the parsed issues, not the linter's exit code.

Quality gates

recordIssues(
    tools: [esLint(pattern: 'reports/eslint.xml')],
    qualityGates: [
        [threshold: 1,  type: 'TOTAL_ERROR',  unstable: false],  // any error fails
        [threshold: 10, type: 'TOTAL_HIGH',   unstable: true],
        [threshold: 1,  type: 'NEW',          unstable: true]    // no new issues
    ]
)

Requires — Plugin: HTML Publisher · Plugin: Allure

Coverage reports, benchmark output, Lighthouse, Allure — many tools produce a directory of HTML. Jenkins can serve it from the build page.

The step

publishHTML(target: [
    reportName:  'Coverage',
    reportDir:   'htmlcov',
    reportFiles: 'index.html',
    keepAll:     true,
    alwaysLinkToLastBuild: true,
    allowMissing: false
])

Requires the HTML Publisher plugin. A link appears in the job and build sidebars.

• keepAll: true — keep the report for every build, not just the latest. Costs disk; worth it for coverage where you want to compare.
• allowMissing: false — fail if the directory is not there, rather than publishing an empty link.

The CSP problem

Out of the box, published HTML often renders unstyled — no CSS, no JavaScript, no images. This is Jenkins' Content-Security-Policy on served files, and it is deliberate: a build can write arbitrary HTML, and serving it with scripts enabled from the Jenkins origin would be stored XSS against your own users.

The usual advice is to relax it from the script console:

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP",
                   "sandbox; default-src 'self'; style-src 'self' 'unsafe-inline';")

Understand the trade-off before doing this. Loosening it means any build output can execute script in the Jenkins origin, in the browser of anyone who views the report. On a shared Jenkins that is a genuine cross-user attack path.

Safer options:

• Keep the sandbox directive ( first) — it blocks the dangerous parts while allowing styling.

Requires — Plugin: Workspace Cleanup · Plugin: JUnit · Plugin: Coverage · On the agent: trivy · +1 more, named inline

A quality gate is a rule that fails a build. The design problem is not technical — it is making a rule that survives contact with a deadline.

Why gates get disabled

Every failed gate is a person blocked. If the gate is:

• Slow — it gets skipped "just this once".
• Flaky — it gets retried until green, which trains people to ignore it.
• Unachievable — inherited debt means it can never pass, so it gets commented out.
• Unexplained — a red X with no actionable message gets escalated, then waived.

A gate that is bypassed provides negative value: it costs time and provides no assurance.

The rules that work

1. Gate on the delta, not the total.

// no
qualityGates: [[threshold: 80.0, metric: 'LINE', baseline: 'PROJECT']]

// yes
qualityGates: [[threshold: 80.0, metric: 'LINE', baseline: 'MODIFIED_LINES']]

Applies to coverage, lint, complexity, and security findings alike. You cannot fix the past in this pull request; you can be responsible for what you touched.

2. Fail fast and cheap.

Requires — Plugin: Job Cacher · On the agent: docker · On the machine you run this from: jq

Slow pipelines get worked around. Measuring first is the whole trick.

Find out where time goes

Stage View gives per-stage timings across builds. The stage that grew is usually obvious.

For per-step detail, the Pipeline Steps page shows each step's duration. Or use the API:

curl -s -u "$USER:$TOKEN" \
  "https://ci.example.com/job/api-build/lastBuild/wfapi/describe" \
  | jq -r '.stages[] | "\(.durationMillis/1000)s\t\(.name)"' \
  | sort -rn

Do this before optimising anything. The stage people assume is slow usually is not.

The usual wins, roughly in order

1. Shallow clone. Covered in Chapter 5. Often minutes.

extensions: [[$class: 'CloneOption', shallow: true, depth: 1, noTags: true]]

2. Cache dependencies. Re-downloading node_modules on every build is the most common single waste.

// Simple: a directory on the agent
environment {
    npm_config_cache = '/var/cache/jenkins/npm'
    GRADLE_USER_HOME = '/var/cache/jenkins/gradle'
    MAVEN_OPTS       = '-Dmaven.repo.local=/var/cache/jenkins/m2'
}

Or the plugin for caches that follow the job across agents:

input stops the pipeline and waits for someone to click. It is how manual approval gates work.

The simple form

stage('Deploy to production') {
    steps {
        input message: 'Deploy to production?', ok: 'Deploy'
        sh 'make deploy-prod'
    }
}

The build pauses. In Stage View the stage shows a prompt; hovering gives Proceed/Abort.

Restricting who can approve

input(
    message: 'Deploy to production?',
    ok: 'Deploy',
    submitter: 'release-managers,alice',
    submitterParameter: 'APPROVER'
)

submitter takes users or groups. submitterParameter captures who approved — exactly what an audit trail needs:

script {
    def approver = input(
        message: 'Deploy?',
        submitter: 'release-managers',
        submitterParameter: 'APPROVER'
    )
    currentBuild.description = "deployed by ${approver}"
}

Collecting input

script {
    def choice = input(
        message: 'Which version?',
        parameters: [
            choice(name: 'VERSION', choices: ['1.2.0', '1.1.9']),
            booleanParam(name: 'SKIP_SMOKE', defaultValue: false)
        ]
    )
    echo "version=${choice.VERSION} skip=${choice.SKIP_SMOKE}"
}

With one parameter input returns the value directly; with several it returns a map. This asymmetry catches people out.

The executor problem

A stage waiting on input holds its executor. A pipeline paused for two days occupies an agent slot for two days.

Put input outside the agent:

pipeline {
    agent none
    stages {
        stage('Build') {
            agent { label 'linux' }
            steps { sh 'make build' }
        }

        stage('Approval') {
            // no agent — holds nothing
            steps {
                timeout(time: 24, unit: 'HOURS') {
                    input message: 'Ship it?', submitter: 'release-managers'
                }
            }
        }

        stage('Deploy') {
            agent { label 'linux' }
            steps { sh 'make deploy' }
        }
    }
}

Always put a timeout on an input. Without one a forgotten build waits forever and shows up as a mystery months later.

The input directive is evaluated before the agent is allocated, which solves the executor problem automatically:

stage('Approval') {
    options { timeout(time: 24, unit: 'HOURS') }
    input {
        message 'Ship it?'
        submitter 'release-managers'
        parameters {
            choice(name: 'TARGET', choices: ['blue', 'green'])
        }
    }
    steps { echo "deploying to ${TARGET}" }
}

Aborting

Clicking Abort throws an exception and the build ends ABORTED. To treat that as non-fatal:

script {
    try {
        timeout(time: 1, unit: 'HOURS') {
            input message: 'Deploy?'
        }
    } catch (err) {
        echo 'Not approved in time — skipping deploy'
        currentBuild.result = 'SUCCESS'
        return
    }
}

Takeaway: always wrap input in a timeout, and keep it off an agent so a pending approval does not hold an executor for days.

Declarative is deliberately limited. When you need real logic — loops, try/catch, variables — you open a script block and write Scripted Groovy inside it.

The escape hatch

stage('Build') {
    steps {
        script {
            def targets = ['api', 'worker', 'cron']
            for (t in targets) {
                sh "make build TARGET=${t}"
            }
        }
    }
}

When it is legitimate

• Loops over a list computed at runtime.
• try / catch / finally around something you must recover from.
• Building a map or list to pass to a step.
• readJSON / readYaml then branching on the contents.

When it is a smell

An 80-line script block is a pipeline that should be a shared library function (Chapter 9) or, more often, a shell script in your repository:

steps {
    sh './ci/build.sh'
}

Logic in a shell script can be run locally, tested, reviewed and debugged without Jenkins. Logic in a script block only runs inside Jenkins. Move logic into the repository wherever you can.

The serialisation trap

Pipeline code is checkpointed so a build survives a controller restart. Every local variable must be serialisable, and the rules surprise people.

This fails:

Requires — Plugin: Docker Pipeline

When twenty repositories have nearly the same Jenkinsfile, a change means twenty pull requests. A shared library puts the common part in one place.

The problem it solves

// Repeated in every one of your services
stage('Docker') {
    steps {
        script {
            docker.withRegistry('https://registry.example.com', 'registry-creds') {
                def img = docker.build("api:${env.BUILD_NUMBER}")
                img.push()
                img.push('latest')
            }
        }
    }
}

Becomes:

stage('Docker') {
    steps {
        buildAndPush(name: 'api')
    }
}

When it is worth it

• More than about five repositories sharing a pattern.
• Standards you want enforced centrally — every build scans for secrets, every deploy records who approved it.
• Complex logic that deserves review and tests.

When it is not

• Fewer than five repos. Copy-paste is genuinely fine at that scale, and much easier to debug.
• Logic that could be a shell script. A ci/build.sh in each repo can be run locally, tested, and debugged without Jenkins. A library function can only run inside Jenkins.
• Logic that could be a container image. If the goal is "everyone uses the same tools", a shared base image achieves it without any Groovy.

The failure mode of shared libraries is a 3,000-line Groovy codebase that only two people understand, that nobody can test, and that every build in the company depends on. That is a worse position than duplicated Jenkinsfiles.

Requires — Plugin: Pipeline: Shared Groovy Libraries

The directory layout

A shared library is a git repository with a fixed structure:

(root)
├── vars/                       # global variables — the simple, callable things
│   ├── buildAndPush.groovy
│   ├── buildAndPush.txt        # markdown help, shown in Global Variable Reference
│   ├── notifySlack.groovy
│   └── standardPipeline.groovy
├── src/                        # ordinary Groovy classes
│   └── com/example/ci/
│       ├── Docker.groovy
│       └── Version.groovy
├── resources/                  # non-Groovy files, loadable with libraryResource
│   └── com/example/ci/
│       ├── pod-template.yaml
│       └── build.sh
└── test/                       # your tests (Chapter note below)

• vars/ — each file becomes a step named after the file. vars/notifySlack.groovy gives you notifySlack(...). The filename must start lowercase.
• src/ — classes with packages, imported normally. For anything with real structure.
• resources/ — templates, scripts, YAML. Read with libraryResource.

Configuring a global library

Manage Jenkins → System → Global Pipeline Libraries → Add

Requires — Plugin: Workspace Cleanup · Plugin: JUnit · Plugin: Docker Pipeline · Plugin: Pipeline: Shared Groovy Libraries

A file in vars/ becomes a pipeline step. The shape is a call method.

The simplest form

// vars/sayHello.groovy
def call(String name = 'world') {
    echo "Hello, ${name}"
}

sayHello()
sayHello('Kalyan')

Named parameters — the form you usually want

// vars/buildAndPush.groovy
def call(Map config = [:]) {
    // Defaults, overridden by whatever the caller passed
    def cfg = [
        name:       env.JOB_NAME.split('/')[0],
        dockerfile: 'Dockerfile',
        context:    '.',
        registry:   'https://registry.example.com',
        credId:     'registry-creds',
        tags:       [env.BUILD_NUMBER],
        push:       true
    ] << config

    if (!cfg.name) {
        error 'buildAndPush: name is required'
    }

    def image
    stage("Build image ${cfg.name}") {
        image = docker.build(
            "${cfg.name}:${cfg.tags[0]}",
            "-f ${cfg.dockerfile} ${cfg.context}"
        )
    }

    if (cfg.push) {
        stage("Push ${cfg.name}") {
            docker.withRegistry(cfg.registry, cfg.credId) {
                cfg.tags.each { t -> image.push(t) }
            }
        }
    }

    return image
}

Requires — Plugin: Kubernetes · Plugin: Pipeline: Shared Groovy Libraries · On the agent: docker

vars/ is for steps. src/ is for real code with structure.

A class

// src/com/example/ci/Version.groovy
package com.example.ci

class Version implements Serializable {
    private static final long serialVersionUID = 1L

    int major, minor, patch

    Version(String s) {
        def parts = s.replaceAll(/^v/, '').tokenize('.')
        major = parts[0] as int
        minor = parts[1] as int
        patch = parts[2] as int
    }

    Version bumpMinor() {
        return new Version("${major}.${minor + 1}.0")
    }

    String toString() {
        return "${major}.${minor}.${patch}"
    }
}

// in a Jenkinsfile or vars/ file
import com.example.ci.Version

def v = new Version('v1.4.2').bumpMinor()
echo "next: ${v}"        // 1.5.0

implements Serializable is not optional. Pipeline state is checkpointed, and any object held across a step boundary must serialise. Omitting it produces NotSerializableException at some unpredictable later point.

Reaching pipeline steps from a class

Classes have no access to sh, echo and friends. Pass the pipeline in:

Requires — Plugin: Pipeline: Shared Groovy Libraries · Test dependency: JenkinsPipelineUnit + Spock · On the machine you run this from: jq

A library every build depends on needs tests. Groovy makes this harder than it should be, but it is achievable.

Level 1: the linter

Cheapest useful check — validate every Jenkinsfile in CI:

CRUMB=$(curl -s --user "$USER:$TOKEN" \
  'https://ci.example.com/crumbIssuer/api/json' | jq -r .crumb)

curl -X POST -u "$USER:$TOKEN" -H "Jenkins-Crumb: $CRUMB" \
  -F "jenkinsfile=<Jenkinsfile" \
  https://ci.example.com/pipeline-model-converter/validate

Catches syntax errors before a build burns an agent.

Level 2: unit-test the pure parts

Anything without pipeline steps is ordinary Groovy and tests normally with Spock or JUnit:

// test/com/example/ci/VersionSpec.groovy
package com.example.ci

import spock.lang.Specification

class VersionSpec extends Specification {

    def "parses a v-prefixed semver"() {
        when:
        def v = new Version('v1.4.2')

        then:
        v.major == 1
        v.minor == 4
        v.patch == 2
    }

    def "bumping minor resets patch"() {
        expect:
        new Version('1.4.2').bumpMinor().toString() == '1.5.0'
    }
}

This is a strong argument for keeping logic in src/ classes with no pipeline dependency — those parts are trivially testable.

Requires — Plugin: Workspace Cleanup · Plugin: JUnit · On the agent: aws CLI + credentials

Scripted Pipeline is the original syntax: Groovy with pipeline steps, no imposed structure.

The shape

node('linux') {
    stage('Checkout') {
        checkout scm
    }

    stage('Build') {
        sh 'make build'
    }

    stage('Test') {
        try {
            sh 'make test'
        } finally {
            junit 'reports/*.xml'
        }
    }
}

No pipeline {}, no steps {}. node() allocates an executor and a workspace; everything else is ordinary Groovy.

What it gives you

Real control flow at the top level:

node {
    def services = readJSON(file: 'services.json')

    for (svc in services) {
        if (svc.enabled) {
            stage("Deploy ${svc.name}") {
                sh "make deploy SERVICE=${svc.name}"
            }
        }
    }
}

Stages generated at runtime:

The official Debian/Ubuntu package. This is the setup you want on a VM you control.

Prerequisites

Jenkins is a Java application, and the supported JVM range changes with the release line. The current LTS (the 2.555.1 line, April 2026) requires Java 21 or Java 25; Java 17 is no longer sufficient for it. Always check the Java support policy for the line you are installing — Jenkins refuses to start on an unsupported JVM.

sudo apt-get update
sudo apt-get install -y fontconfig openjdk-21-jre
java -version

fontconfig is not optional despite looking like it — some plugins render images and fail obscurely without it.

Add the repository and install

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key

echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/" \
  | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null

sudo apt-get update
sudo apt-get install -y jenkins

Note debian-stable in both URLs. That is the LTS line — releases every 12 weeks with backported fixes. The other line, debian, is weekly. Use LTS unless you have a specific reason not to.

Start it and check

sudo systemctl enable jenkins
sudo systemctl start jenkins
sudo systemctl status jenkins

If it does not come up, the log is the first place to look — not the UI, which is not running yet:

sudo journalctl -u jenkins -n 100 --no-pager

Common causes: wrong Java version, port 8080 already taken, or /var/lib/jenkins not writable.

Unlock it

Jenkins generates a one-time password on first boot so nobody can grab an unconfigured instance off the internet.

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Open http://<host>:8080, paste it, and you get the setup wizard.

The setup wizard, decided in advance

1. Install suggested plugins. Yes, on a first install. It gives you Git, Pipeline, credentials, and the basics. You can prune later.
2. Create the first admin user. Do it now. Do not skip and keep using the initial admin password — that account has an unchangeable name and a password sitting in a file on disk.
3. Jenkins URL. Set it correctly, including scheme and port. Email links, webhook callbacks and the agent JNLP URL all derive from it. Getting this wrong causes confusing breakage weeks later.

Where things live

JENKINS_HOME is the whole product. Back that directory up and you can rebuild Jenkins anywhere. Lose it and you have lost everything. Chapter 10 covers doing that properly.

Changing the port

Port 8080 collides with almost everything. On modern packages:

sudo systemctl edit jenkins

[Service]
Environment="JENKINS_PORT=8081"

sudo systemctl restart jenkins

Takeaway: install from the LTS apt repository, set the Jenkins URL correctly during the wizard, and know that /var/lib/jenkins is the only thing that matters for backup.

Requires — Plugin: Configuration as Code · On the agent: docker · On the machine you run this from: jq · Locally: helm

Jenkins ships on two lines, and picking the wrong one is a slow-burning mistake.

Weekly versus LTS

LTS is chosen from a recent weekly release and then receives backported fixes for its cycle. That means fewer surprises and a real patch stream.

The apt repository URL differs by one path segment, which is easy to get wrong:

https://pkg.jenkins.io/debian-stable   ← LTS
https://pkg.jenkins.io/debian          ← weekly

Check what you are on:

apt-cache policy jenkins
# or, from a running instance
curl -s https://ci.example.com/api/json | jq -r .
# the version is also in the footer of every page and at /manage

Pinning a version

Unattended upgrades that bump Jenkins without you knowing is a bad surprise. Hold it:

sudo apt-mark hold jenkins
apt-mark showhold

# when you are ready
sudo apt-mark unhold jenkins
sudo apt-get install --only-upgrade jenkins

Jenkins speaks plain HTTP on 8080. In front of it you want nginx or Caddy terminating TLS. Doing this wrong produces a Jenkins that half-works in ways that are hard to diagnose.

Why it goes wrong

Jenkins builds absolute URLs from what it thinks its address is. Behind a proxy it sees http://localhost:8080 while users see https://ci.example.com. Mismatch causes:

• The "It appears that your reverse proxy set up is broken" banner.
• Broken redirects after login.
• Webhooks that fire but link to the wrong place.
• WebSocket agents failing to connect.

nginx

server {
    listen 443 ssl http2;
    server_name ci.example.com;

    ssl_certificate     /etc/letsencrypt/live/ci.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ci.example.com/privkey.pem;

    # Build logs stream; buffering them makes the console look frozen.
    proxy_buffering off;
    # Large artifact uploads and plugin installs need headroom.
    client_max_body_size 100m;

    location / {
        proxy_pass          http://127.0.0.1:8080;
        proxy_http_version  1.1;

        proxy_set_header    Host              $host;
        proxy_set_header    X-Real-IP         $remote_addr;
        proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;

        # Required for inbound agents over WebSocket
        proxy_set_header    Upgrade    $http_upgrade;
        proxy_set_header    Connection $connection_upgrade;

        proxy_read_timeout  90s;
    }
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    server_name ci.example.com;
    return 301 https://$host$request_uri;
}

Requires — Plugin: Configuration as Code · Plugin: Role-based Authorization Strategy

Jenkins core does very little. Git support, pipelines, credentials, agents, JUnit reporting — all plugins. There are over 1,800 of them.

Where they live

Dashboard → Manage Jenkins → Plugins

Four tabs:

• Updates — upgrades for what you have.
• Available plugins — the catalogue.
• Installed — what you have, with the option to disable or uninstall.
• Advanced settings — proxy configuration and uploading a .hpi file by hand (for air-gapped installs).

Installing

Search, tick, and choose Install without restart or Download now and install after restart. Most plugins install live; some need the restart. If in doubt, restart during a quiet window — a plugin half-loaded is a strange thing to debug.

To restart cleanly, letting running builds finish first:

http://<jenkins>/safeRestart

Versus the blunt instrument, which kills running builds:

http://<jenkins>/restart

The plugins you will actually use

Dashboard → Manage Jenkins is where all administration lives. It is a wall of links. Here is what each group is actually for, so you can stop hunting.

System Configuration

• System — global settings: Jenkins URL, admin email, global environment variables, SMTP, and the configuration blocks that installed plugins add. This page grows enormous. Use your browser's find.
• Tools — JDK, Maven, Gradle, Git, NodeJS installations. Declare them here; reference them by name in pipelines.
• Plugins — install, update, remove. Covered in Chapter 1.
• Nodes — your agents. Add, configure, take offline. Chapter 6.
• Clouds — dynamic agents: Docker, Kubernetes, EC2. Appears once you install a cloud plugin.
• Appearance — themes and the system message banner.

Security

• Security — the big one: security realm (who can log in) and authorization (what they can do). Also CSRF, agent protocols, and Markup Formatter.
• Credentials — the credential store. Chapter 7.
• Credential Providers — which credential backends are enabled.
• Users — accounts, when using Jenkins' own database.

Status Information

• System Information — JVM properties, environment variables, plugin versions, OS. The first thing to attach to a bug report.
• System Log — the application log, live, in the browser. You can add custom loggers scoped to a package, which is the single best debugging trick in Jenkins. See the troubleshooting note in Chapter 10.
• Load Statistics — graphs of queue length and executor utilisation over time. Answers "do we need more agents?" with data.
• About Jenkins — version and licences.

Troubleshooting

• Manage Old Data — configuration left behind by removed plugins. Worth clearing after a plugin purge.

Tools and Actions

• Reload Configuration from Disk — re-reads JENKINS_HOME without a restart. Use after editing XML by hand.

A Freestyle project is the original Jenkins job: a web form where you tick boxes and paste shell commands. No code, no repository, all configuration in the UI.

Modern advice is "use Pipeline". That is correct. You still need to understand Freestyle because you will inherit hundreds of them.

Creating one

Dashboard → New Item → enter a name → Freestyle project → OK.

Names become URLs. Avoid spaces — api build becomes /job/api%20build/, which every script you write later will have to encode. Use api-build.

The configuration page, section by section

General

• Description — supports HTML if the markup formatter allows it.
• Discard old builds — set this. Days to keep, and max builds to keep.
• This project is parameterised — adds a form when triggering.
• Restrict where this project can be run — a label expression. The Freestyle equivalent of agent { label '…' }.

Source Code Management

• None / Git / Subversion. For Git: repository URL, credentials, and Branches to build (*/main).

Build Triggers

• Build periodically — cron. Runs whether or not anything changed.
• Poll SCM — cron, but only builds if the repo changed. Wasteful at scale; prefer webhooks (Chapter 5).
• Build after other projects are built — simple chaining.
• GitHub hook trigger for GITScm polling — the webhook option.

Build Environment

• Delete workspace before build starts — slower but eliminates a whole class of "works on rebuild" mysteries.
• Use secret text(s) or file(s) — binds credentials into environment variables. Chapter 7.
• Add timestamps to the Console Output — turn this on everywhere. Costs nothing, and a log without timestamps is much harder to reason about.

Manage Jenkins → Script Console, or /script. A box that runs Groovy inside the Jenkins JVM, as the Jenkins user, with no sandbox.

Why it exists

Some things have no UI. Bulk operations, inspecting internal state, repairing a broken configuration. The script console is the escape hatch.

Genuinely useful snippets

Inventory every job and its type:

Jenkins.instance.getAllItems(Job.class).each {
  println "${it.fullName}\t${it.class.simpleName}"
}

Find jobs that have not built in 90 days:

def cutoff = System.currentTimeMillis() - (90L * 24 * 60 * 60 * 1000)
Jenkins.instance.getAllItems(Job.class).findAll {
  !it.lastBuild || it.lastBuild.timestamp.timeInMillis < cutoff
}.each { println it.fullName }

Cancel everything in the queue:

Jenkins.instance.queue.clear()

Disconnect a wedged agent:

def node = Jenkins.instance.getNode('build-agent-3')
node.toComputer().disconnect(null)
node.toComputer().connect(true)

Reclaim disk by wiping old workspaces:

Requires — Locally: Java + jenkins-cli.jar

Everything the UI does has a command-line equivalent. For scripting, bulk changes, and disaster recovery, the CLI is faster than clicking.

Getting it

The jar is served by your own Jenkins, so it always matches the version:

curl -sO https://ci.example.com/jnlpJars/jenkins-cli.jar

Authenticating

Use an API token, not a password (click your name → Configure → API Token):

export JENKINS_URL=https://ci.example.com
export JENKINS_USER_ID=alice
export JENKINS_API_TOKEN=11a2b3c4d5e6f7...

java -jar jenkins-cli.jar -s "$JENKINS_URL" -auth "$JENKINS_USER_ID:$JENKINS_API_TOKEN" who-am-i

who-am-i prints your authorities — the quickest way to check what permissions a token actually carries.

The commands worth knowing

Requires — Plugin: Configuration as Code · Plugin: Role-based Authorization Strategy · Plugin: Pipeline: Multibranch (+ a branch source such as GitHub) · On the agent: docker

JCasC describes your controller's configuration in a YAML file. Instead of clicking through Manage Jenkins, you commit a file.

Why it matters

• Reproducible. A new controller comes up configured. Disaster recovery becomes "apply the YAML".
• Reviewable. Configuration changes go through pull requests, which Jenkins otherwise has no mechanism for (Chapter 7).
• Diffable. "Who changed the SMTP server" is answerable.

Setup

Install Configuration as Code. Point Jenkins at the file with an environment variable or a system property:

# /etc/default/jenkins, or the systemd unit
CASC_JENKINS_CONFIG=/var/lib/jenkins/casc/jenkins.yaml

Or a directory, or a URL. On restart Jenkins applies it.

Manage Jenkins → Configuration as Code gives you Reload existing configuration, View configuration, and — most usefully — Download current configuration, which exports your existing setup as YAML. Start there rather than writing it from scratch.

A realistic file

jenkins:
  systemMessage: "PRODUCTION CI — configuration is managed by JCasC. Changes via PR only."
  numExecutors: 0                      # no builds on the controller
  mode: EXCLUSIVE                      # only run jobs that ask for this node by label
  quietPeriod: 5
  scmCheckoutRetryCount: 3

  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: "admin"
          password: "${ADMIN_PASSWORD}"    # from the environment

  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: "admin"
            permissions: ["Overall/Administer"]
            entries:
              - group: "platform-team"
          - name: "developer"
            permissions: ["Overall/Read", "Job/Build", "Job/Cancel"]
            entries:
              - group: "developers"

  remotingSecurity:
    enabled: true

  nodes:
    - permanent:
        name: "build-1"
        remoteFS: "/home/jenkins/agent"
        numExecutors: 4
        labelString: "linux docker build"
        launcher:
          ssh:
            host: "build-1.internal"
            credentialsId: "agent-ssh-key"
            sshHostKeyVerificationStrategy:
              knownHostsFileKeyVerificationStrategy

credentials:
  system:
    domainCredentials:
      - credentials:
          - usernamePassword:
              scope: GLOBAL
              id: "registry-creds"
              username: "ci"
              password: "${REGISTRY_PASSWORD}"
              description: "Container registry"

unclassified:
  location:
    url: "https://ci.example.com/"
    adminAddress: "ci-admin@example.com"

  globalLibraries:
    libraries:
      - name: "ci-lib"
        defaultVersion: "v3"
        implicit: false
        retriever:
          modernSCM:
            scm:
              git:
                remote: "git@github.com:acme/ci-lib.git"
                credentialsId: "github-deploy-key"

tool:
  git:
    installations:
      - name: "Default"
        home: "git"
  jdk:
    installations:
      - name: "jdk21"
        home: "/usr/lib/jvm/java-21-openjdk-amd64"

Secrets in JCasC

Never commit secrets. ${VAR} reads from the environment:

export ADMIN_PASSWORD='…'
export REGISTRY_PASSWORD='…'

Or use the secrets directory — every file becomes a variable named after it:

mkdir -p /run/secrets/jenkins
echo -n 'hunter2' > /run/secrets/jenkins/REGISTRY_PASSWORD
export SECRETS=/run/secrets/jenkins

Better still, use an external secret manager and reference it, so the controller never holds the plaintext at all.

Jobs as code too

JCasC configures the controller, not jobs. For jobs, either:

• Multibranch/organisation folders — Jenkins discovers repositories and their Jenkinsfiles. Usually enough, and the least machinery.
• Job DSL plugin — generate jobs from a Groovy script, seeded by JCasC.

jobs:
  - script: >
      multibranchPipelineJob('api') {
        branchSources {
          github {
            id('acme-api')
            repoOwner('acme')
            repository('api')
            credentialsId('github-token')
          }
        }
      }

Rolling it out on an existing controller

Do not try to write the whole file at once. Export what you have, commit that as a baseline, then move sections under management one at a time — validating with View configuration after each. A JCasC file that is 80% correct and applied on restart can undo settings you did not realise it owned.

Takeaway: export your current config first, keep secrets in environment variables, and adopt JCasC section by section rather than all at once.

Requires — On the agent: aws CLI + credentials · On the controller: gpg

JENKINS_HOME is the entire product. Everything else is replaceable.

What actually needs backing up

$JENKINS_HOME/
├── config.xml              ← global configuration
├── credentials.xml         ← encrypted secrets
├── secrets/                ← the keys that decrypt them  ** critical **
├── jobs/                   ← job configs and build history
│   └── <job>/
│       ├── config.xml
│       └── builds/         ← the bulky part
├── users/
├── nodes/
├── plugins/                ← .jpi files; re-installable
└── workspace/              ← DO NOT back up; regenerable

The controller key is the exception, and it matters. Jenkins' own guidance is blunt: treat $JENKINS_HOME/secrets/master.key "like you treat your SSH private key and NEVER include it in a regular backup". It decrypts everything else in secrets/, so a backup containing both is a single object that unlocks every credential you own.

Two backup sets, stored apart:

Set A — the regular backup            Set B — the controller key
JENKINS_HOME minus workspaces          $JENKINS_HOME/secrets/master.key
  encrypted, offsite, nightly            separate, highly restricted, rarely touched
        |                                        |
        +----------- restore needs BOTH ---------+

A full restore applies A, then puts B back. Rehearse it that way — a restore drill that quietly uses the key already sitting in the same archive proves nothing.

Requires — On the agent: docker

Jenkins ships LTS every 12 weeks; plugins update constantly. Upgrades are routine and occasionally destructive.

Before

1. Back up, and confirm the backup is good. Not the scheduled one — a fresh one, now.
2. Read the changelog. jenkins.io/changelog-stable for core; each plugin's page for plugins. Look for "breaking".
3. Check Java compatibility. Newer Jenkins raises the minimum Java version periodically, and it is the most common upgrade blocker.
4. Prepare for shutdown — Manage Jenkins → Prepare for Shutdown stops new builds and lets running ones finish.

The order

Plugins before and after core. The Jenkins LTS upgrade guide is explicit: "you must ensure that all plugins have been updated both before and after upgrading. If plugins are not updated both before and after the upgrade, compatibility issues may arise."

The runbook:

1. Read the upgrade guide for EVERY LTS line you are skipping, not just the target
2. Fresh backup, and confirm it restores
3. Update plugins to their current versions        ← before
4. Manage Jenkins → Prepare for Shutdown
5. Upgrade core
6. Start, and check the log for plugin load failures
7. Update plugins again, now aligned to the new core   ← after
8. Run a smoke pipeline on each agent type

Steps 3 and 7 are both required. Updating only afterwards leaves you starting the new core against plugins built for the old one, which is the state that produces a controller that will not come up.

Requires — Plugin: Prometheus metrics · On the machine you run this from: jq

The built-in views

• Manage Jenkins → Load Statistics — queue length and executor use over time. Answers "do we need more agents" with data rather than opinion.
• Manage Jenkins → Nodes — disk, swap, clock skew, response time per agent.
• Manage Jenkins → System Information — JVM memory, uptime, versions.

Prometheus

Install Prometheus metrics. It exposes /prometheus/ for scraping.

scrape_configs:
  - job_name: jenkins
    metrics_path: /prometheus/
    static_configs:
      - targets: ['ci.example.com']

The metrics worth alerting on:

Cheap alerts without Prometheus

Requires — Plugin: Workspace Cleanup · Plugin: Job Configuration History

The problems that actually happen, and what they mean.

Nothing is building; the queue is growing

Hover the ⓘ next to the queued item. Jenkins states the reason:

• "there are no nodes with the label 'docker'" — no agent has that label. Check Manage Jenkins → Nodes and the agent's label string. Usually an agent went offline or a label was renamed.
• "Waiting for next available executor" — genuine capacity. Add agents or executors.
• "Build #N is already in progress" — disableConcurrentBuilds() doing its job.
• "…is offline" — the named agent is down.

An agent will not connect

Node page → Log shows the launch transcript. See Chapter 6's table. In short: keys, host key verification, Java version, and the remote root's permissions cover almost all of it.

A build hangs forever

options { timeout(time: 30, unit: 'MINUTES') }

Add this first so it becomes a failure rather than a mystery. Then find the cause:

• A command waiting on stdin. Non-interactive flags: apt-get -y, git -c core.askpass=true, ssh -o BatchMode=yes.
• A process that does not exit. sh 'nohup ./server &' keeps the step alive; use sh 'nohup ./server >/dev/null 2>&1 &' with a proper disown, or JENKINS_NODE_COOKIE=dontKillMe if it should survive.
• An input with no timeout (Chapter 4).

Thread dump for a truly stuck controller: https://ci.example.com/threadDump.

Works locally, fails in Jenkins

In order of likelihood:

1. Environment. Jenkins shells are non-interactive and non-login, so ~/.bashrc is not read. , , are not initialised. Add and compare.

Requires — On the agent: docker · On the machine you run this from: jq

CI produces an artefact. CD gets it into an environment. Jenkins does both, and the main design decision is not to rebuild.

Build once, promote

pipeline {
    agent { label 'linux && docker' }

    environment {
        IMAGE = "registry.example.com/api"
        TAG   = "${env.GIT_COMMIT.take(12)}"
    }

    stages {
        stage('Build') {
            steps {
                sh 'docker build -t $IMAGE:$TAG .'
            }
        }

        stage('Test') {
            steps { sh 'make test' }
        }

        stage('Push') {
            steps {
                withCredentials([usernamePassword(
                    credentialsId: 'registry-creds',
                    usernameVariable: 'U', passwordVariable: 'P'
                )]) {
                    sh '''
                        set -euo pipefail
                        echo "$P" | docker login registry.example.com -u "$U" --password-stdin
                        docker push $IMAGE:$TAG
                    '''
                }
            }
        }

        stage('Deploy dev') {
            steps { sh 'make deploy ENV=dev IMAGE=$IMAGE:$TAG' }
        }

        stage('Deploy staging') {
            when { branch 'main' }
            steps { sh 'make deploy ENV=staging IMAGE=$IMAGE:$TAG' }
        }

        stage('Approve production') {
            when { branch 'main' }
            options { timeout(time: 24, unit: 'HOURS') }
            input {
                message 'Deploy to production?'
                submitter 'release-managers'
                submitterParameter 'APPROVER'
            }
            steps {
                script { currentBuild.description = "prod by ${APPROVER}" }
            }
        }

        stage('Deploy production') {
            when { branch 'main' }
            steps { sh 'make deploy ENV=prod IMAGE=$IMAGE:$TAG' }
        }
    }
}

Requires — Plugin: Kubernetes · Plugin: Configuration as Code

Jenkins scales well up to a point, then does not. Knowing the ceiling helps you plan.

Rough capacity of one controller

A well-tuned controller handles roughly:

• Hundreds to a couple of thousand jobs.
• 50–100 concurrent builds, given enough agents.
• A few hundred agents, though connection overhead grows.

Symptoms of hitting the ceiling: slow UI, long queue-scheduling latency, GC pauses, restarts taking many minutes.

Vertical first — it is usually enough

1. More heap — see the monitoring note.
2. Fewer retained builds — thousands of build directories per job make everything slow. Discarders.
3. Faster disk — JENKINS_HOME on SSD/NVMe. Jenkins is heavy on small file operations, and this often matters more than CPU.
4. Artifacts off the controller — S3.
5. Zero executors on the controller — again.

Most "we need to scale Jenkins" situations are solved by the first two.

Horizontal — multiple controllers

There is no clustering. A Jenkins controller is a single process with a single JENKINS_HOME. Scaling out means more controllers, split by team or product.

Trade-offs:

• Isolation — one team's outage does not stop everyone. Genuinely valuable.
• Blast radius — a compromised controller holds fewer credentials.
• Cost — more instances to patch, back up and upgrade. This is the real price.
• Fragmentation — no single view.

JCasC makes this bearable: a common configuration repository, one controller per team, each reproducible from YAML.

Ephemeral agents

Before splitting controllers, make agents elastic. Kubernetes (Chapter 6) or EC2/cloud plugins create agents on demand and destroy them after.

Requires — Plugin: Configuration as Code · Plugin: Audit Trail

Everything in this book, condensed into things you can verify.

Security

[ ] Controller executors = 0
[ ] Not reachable from the internet without SSO/VPN
[ ] Overall/Administer granted to a short, named, reviewed list
[ ] Open sign-up disabled
[ ] Credentials scoped to folders, not all global
[ ] Agent SSH keys use System scope so builds cannot read them
[ ] Host key verification is not "Non verifying"
[ ] Fork PRs do not run on agents holding credentials
[ ] Script approvals reviewed — no Runtime.exec, no Jenkins.getInstance
[ ] Write access to global shared libraries restricted and review-gated
[ ] Plugins updated; no outstanding security advisories
[ ] Audit Trail installed and shipped somewhere it is read

Reliability

[ ] JENKINS_HOME backed up nightly, encrypted, offsite
[ ] Backup restored end-to-end within the last quarter
[ ] Every job has a buildDiscarder
[ ] Artifacts split from build history in retention settings
[ ] Controller disk monitored with alerting
[ ] Agent offline alerting
[ ] Queue-depth alerting
[ ] JVM heap sized and monitored
[ ] Every pipeline has a timeout

Configuration