Skip to Content
Software EngineeringBooksJenkins2 up and RunningChapters 2. The Foundations

Chapters 2. The Foundations

Syntax: Scripted Pipelines Versus Declarative Pipelines

  • Scripted Syntax:

    • The original pipeline-as-code model in Jenkins.
    • Imperative style: logic and flow are defined explicitly in the script.
    • Strongly tied to the Groovy language, particularly for error handling and exceptions.

  • Declarative Syntax:

    • A newer model introduced in Jenkins 2.
    • Declarative style: pipelines are written in predefined sections that specify desired outcomes rather than logic.
    • Simpler, more structured, and easier to use for those familiar with Jenkins web forms.

  • Code Example:

    • Scripted Pipeline:

      node('worker_node1') {
    stage('Source') {
        git 'git@diyvb2:/home/git/repositories/workshop.git'
    }
    stage('Compile') {
        sh "gradle clean compileJava test"
    }
}

    • Declarative Pipeline:

      pipeline {
    agent {label 'worker_node1'}
    stages {
        stage('Source') {
            steps {
                git 'git@diyvb2:/home/git/repositories/workshop.git'
            }
        }
        stage('Compile') {
            steps {
                sh "gradle clean compileJava test"
            }
        }
    }
}

  • Advantages of Scripted Pipelines:

    • Fewer sections and less predefined structure.
    • More procedural and flexible for complex workflows.
    • Familiar for users experienced with traditional pipeline-as-code models.
    • Compatible with backward versions and offers more customization.

  • Disadvantages of Scripted Pipelines:

    • Requires more programming knowledge.
    • Limited syntax checking and debugging tools.
    • Further from the traditional Jenkins model, potentially more complex.

  • Advantages of Declarative Pipelines:

    • Structured, easier to read, and closer to traditional Jenkins web forms.
    • Can be created using the Blue Ocean interface.
    • Improved syntax checking and error identification.
    • Better consistency across pipelines, mapping to Jenkins concepts (e.g., notifications).

  • Disadvantages of Declarative Pipelines:

    • Limited support for complex workflows and iterative logic.
    • Still evolving, may not support all use cases.
    • Rigid structure, harder to customize compared to scripted pipelines.

  • Choosing Between Scripted and Declarative Pipelines:

    • Declarative Pipelines: Easier for new users or those looking for a more structured, maintainable approach. Best for simpler workflows.
    • Scripted Pipelines: More flexible and powerful, ideal for experienced users who need to model complex workflows or prefer programmatic control.

Systems: Masters, Nodes, Agents, and Executors

  • Every Jenkins pipeline must have one or more systems to execute code.
  • In Jenkins 2, the terms used to describe systems are more generalized compared to traditional Jenkins (where “masters” and “slaves” were used).

  • Master:

    • The primary controlling system for a Jenkins instance, responsible for managing configuration, jobs, and overall operations.
    • Not intended for running heavy jobs due to security risks and to avoid blocking the system’s main functions.

  • Node:

    • A generic term in Jenkins 2 for any system (master or agent) that can execute Jenkins jobs.
    • Nodes can include physical machines, virtual machines, or containers like Docker.

  • Agent:

    • Equivalent to what was called a “slave” in older Jenkins versions.
    • A non-master system managed by the master, used to run jobs on allocated systems.
    • Only a lightweight Jenkins client is installed on agents to run jobs securely and efficiently.

  • Executors:

    • Slots that allow jobs to run on a node/agent.
    • A node can have multiple executors, enabling concurrent job processing. Jobs wait for an available executor if none are free.

Directives vs. Steps (Scripted vs. Declarative Pipelines)

  • Scripted Pipeline:

    • Uses the node step to allocate an executor on a system with an agent.

    • Example:

      node('worker') {
  stage('Source') {
    git 'git@diyvb2:/home/git/repositories/workshop.git'
  }
}

  • Declarative Pipeline:

    • Uses the agent directive to allocate a node for executing code.

    • Example:

      pipeline {
  agent { label: 'worker' }
  stages {
    stage('Source') {
      steps {
        git 'git@diyvb2:/home/git/repositories/workshop.git'
      }
    }
  }
}

  • Node Labels:

    • Labels are identifiers used to specify which nodes should handle certain jobs.
    • Labels can:
      • Identify specific nodes (e.g., unique labels).
      • Group similar nodes (e.g., same label for a group of nodes).
      • Mark a node’s characteristic (e.g., “Windows” or “West Coast”).
    • Labels can be used in pipelines to define where to execute the code.

Creating Nodes

  • Nodes can be set up in the Manage Nodes section of Jenkins.
    • Define number of executors and other parameters.
    • Additional settings like environment variables and tool locations can be configured if needed.
    • Nodes can have multiple labels to group or categorize them for specific job requirements.

Structure: Working with the Jenkins DSL

  • Jenkins DSL Overview:

    • DSL (Domain-Specific Language) in Jenkins is used to define pipelines, written using the Groovy language.
    • The DSL enables users to create pipelines in a structured, programmatic way, leveraging Groovy’s simplicity for creating domain-specific commands.

  • Basic Structure of a Scripted Pipeline:

    • Example:

      node ('worker1') {
  stage('Source') {
    git 'https://github.com/brentlaster/gradle-greetings.git'
  }
}

    • node: Defines the node (system) on which the job will run.

      • Refers to either a master or agent system.
      • Can specify a node label or leave it empty to run on the master or any available node.
      • Multiple labels can be assigned using logical operators (e.g., node("linux && east")).

    • stage: Organizes the pipeline into logical sections (e.g., Source, Build, Test).

      • Each stage must have a name for display purposes but doesn’t affect functionality.
      • Stages help break down complex pipelines into understandable parts.

    • steps: The actual commands or operations within a pipeline.

      • Steps are the lowest-level commands in the Jenkins DSL, such as Git commands or shell commands.

      • Example:

        git 'https://github.com/brentlaster/gradle-greetings.git'

      • Supports both shorthand and full syntax formats.

  • Understanding Step Syntax:

    • Steps can use named parameters (key-value pairs) or shorthand versions if only one required parameter is provided.

    • Example:

      • Full syntax:

        git branch: 'test', url: 'https://github.com/brentlaster/gradle-greetings.git'

      • Shorthand version:

        git 'https://github.com/brentlaster/gradle-greetings.git'

    • Other examples:

      • Full syntax:

        bat([script: 'echo hi'])

      • Shorthand version:

        bat 'echo hi'

  • Nodes and Mappings:

    • Multiple nodes can be used in parallel to run jobs on different systems.

    • Example:

      parallel (
  win: { node ('win64') { ... }},
  linux: { node ('ubuntu') { ... }}
)

  • Groovy and Jenkins DSL:

    • While Jenkins DSL is built using Groovy, it is advised to limit complex Groovy code in pipelines to maintain readability and accessibility for users unfamiliar with Groovy.
    • Declarative Pipelines restrict Groovy usage, offering a more structured approach with built-in capabilities similar to traditional Jenkins features.

  • Leveraging Other Languages:

    • For complex functions or iterative workflows, these can be abstracted into shared libraries (covered in Chapter 6) to keep the pipeline code clean and modular.

Supporting Environment: Developing a Pipeline Script

Developing a Pipeline Script in Jenkins

  • Pipeline Project Setup:

    • In Jenkins 2, you can create a new project by selecting the Pipeline project type.
    • You can either create a pipeline script directly within the job or store it externally as a Jenkinsfile with your source code.

  • Pipeline Script in a Job:

    • Pipeline code can be entered in the built-in Jenkins editor under the Pipeline tab.
    • Alternatively, a Jenkinsfile can be edited outside Jenkins, and adjustments might be needed for external routines or resources.

  • Editor Features:

    • Syntax Checking: The Jenkins editor attempts to check for valid Groovy syntax, flagging errors with a red “X” icon.
    • Extended Error Information: Hover over the “X” for a more detailed description of the error.
    • Autocomplete: The editor automatically inserts closing brackets for convenience but may lead to extra brackets if you’re not careful.

  • Snippet Generator:

    • A useful tool in Jenkins 2 to help generate correct DSL syntax for steps.
    • The Snippet Generator:
      • Allows searching for pipeline-compatible steps.
      • Provides web forms to enter values for parameters.
      • Generates Groovy DSL code that can be copied and pasted into your pipeline script.
    • The content is updated based on the plugins installed in your Jenkins instance.

  • Working Example with Snippet Generator:

    • Suppose you want to retrieve code from a Git repository:

      • Open the Pipeline Syntax link at the bottom of the Pipeline tab.

      • Select the git step from the drop-down menu and configure parameters.

      • Generate the following DSL code for retrieving the Git code:

        stage('Source') {
  git 'https://github.com/brentlaster/gradle-greetings.git'
}

  • Advanced Parameter Options:

    • Override default step parameters using the Snippet Generator.

    • Example:

      git branch: 'test', url: 'https://github.com/brentlaster/gradle-greetings.git'

  • Polling and Changelog Options:

    • poll: false: Prevents Jenkins from automatically detecting changes in the source management repository after the initial run.
    • changelog: false: Stops Jenkins from computing the changes that initiated a new run, reducing SCM load.

Additional Tools and Features

  • Blue Ocean Interface: Offers a visual editor for declarative pipelines, making it easier to create and manage pipelines (discussed in Chapter 9).
  • Extended Error Handling: Syntax issues may arise when dependencies or imports are not yet resolved, but this is rare.

Running a Pipeline in Jenkins

  • Stages in Pipelines:

    • Pipelines in Jenkins are made up of stages representing different phases such as compilation, testing, or deployment.
    • In earlier Jenkins versions, these stages were managed using separate Freestyle jobs chained together with plugins like the Build Pipeline plugin for visualizing job flow.

  • Pipeline Projects in Jenkins 2:

    • Jenkins 2 introduced the Pipeline project type, allowing users to script entire pipelines with multiple stages using DSL.

    • Stages are defined using the stage{} block, representing key steps in the pipeline (e.g., “Source” and “Build”).

    • Example:

      node ('worker1') {
  stage('Source') {
    git 'https://github.com/brentlaster/gradle-greetings.git'
  }
  stage('Build') {
    sh 'echo gradle build will go here'
  }
}

  • Pipeline Execution and Stage View:

    • Once a pipeline is saved, click Build Now to run it.
    • The Stage View in Jenkins 2 displays each stage as a tile in a matrix format:
      • Rows represent pipeline runs.
      • Columns represent stages in the pipeline.
      • The color of each tile indicates the stage status:
        • Green: Success.
        • Blue Stripes: In progress.
        • Rose: Failure downstream.
        • Rose Stripes: Failure in that stage.

  • Viewing Logs:

    • Console logs for each stage can be accessed by clicking the Logs button on the stage tile.
    • You can also view the full console output by selecting the Console Output link in the build history.

  • Stage View with Errors:

    • If a pipeline fails, the corresponding stage tile turns striped rose for the failed stage, while other successful stages are shown in solid rose.
    • Error details are displayed in a pop-up when hovering over the failed tile, providing information on the failure reason.

  • Replay Feature:

    • Jenkins provides a Replay feature for testing changes without modifying the original pipeline code.
    • Replay allows you to edit and rerun a pipeline after a failed or completed run, without saving the changes to the original script.
    • Replay can also be triggered from the command line using the replay-pipeline command.

  • CLI Replay Example:

    java -jar ~/jenkins-cli.jar -s http://<jenkins-url> replay-pipeline "<Name>" < Jenkinsfile

  • Replay Limitations:

    • If using a direct SCM step like git in your pipeline, Replay always pulls the latest code, even for past builds. Using checkout scm ensures the code version matches the time of the original run.

Pipeline Testing Framework

  • Jenkins Pipeline Unit:
    • A framework designed for unit testing pipeline code by providing a mock execution environment.
    • It allows testing of pipeline configuration, logic, and shared libraries with the ability to mock Jenkins commands and configurations.
    • Integrated with Gradle or Maven projects and operates similarly to JUnit tests.
    • The framework is currently challenging to use but is expected to improve as it evolves within the Jenkins community.
Last updated on
Chapter 1. Introducing Jenkins 2Operating System Concepts