CI/CD Workflows: A Guide to Jenkins Setup

Listen to this article:

0:00

Note: This article was adapted from content originally written on October 19th, 2017, titled “Setting up a Private CI/CD Solution in Azure.” It has been simplified and split into four parts for easier reading.

Part 4: Jenkins Configuration and Complete Workflow

Key Takeaways

This article concludes the series on setting up a private CI/CD solution in Azure, focusing on Jenkins configuration and workflow.
Readers learn to configure Jenkins with Blue Ocean, integrate with GitLab, and set up ephemeral build agents for dynamic builds.
The complete development workflow includes feature branch creation, merge requests, automated CI pipelines, and deployment processes.
Best practices emphasize complete privacy, high availability, scalability, modern workflows, and full control over the CI/CD infrastructure.
The article encourages enhancements such as SonarQube for code quality, security tools, artifact management, and advanced container orchestration with Kubernetes.

In this final part, we’ll configure Jenkins with Blue Ocean for modern CI/CD pipelines, set up ephemeral build agents, and walk through the complete development workflow from code commit to deployment.

Jenkins Configuration

Deploy Jenkins Master

First, create the Jenkins stack with Blue Ocean and essential plugins:

# Create Jenkins directories on manager nodes
for node in 10.0.0.4 10.0.0.5 10.0.0.6; do
  ssh spacely-eng-admin@$node << 'EOF'
    sudo mkdir -p /srv/jenkins
    sudo chown 1000:1000 /srv/jenkins
EOF
done
# Create Jenkins stack
cat << 'EOF' > jenkins-stack.yml
version: '3.7'
services:
  jenkins:
    image: jenkinsci/blueocean:latest
    user: root
    environment:
      JAVA_OPTS: '-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false'
      JENKINS_OPTS: '--httpPort=18080'
    ports:
      - target: 18080
        published: 18080
        protocol: tcp
        mode: host
      - target: 50000
        published: 50000
        protocol: tcp
        mode: ingress
    volumes:
      - /srv/jenkins:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - spacely-engineering-network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.labels.role == manager
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
networks:
  spacely-engineering-network:
    external: true
EOF
# Deploy Jenkins
sudo docker stack deploy -c jenkins-stack.yml jenkins

# Create Jenkins directories on manager nodes

for node in 10.0.0.4 10.0.0.5 10.0.0.6; do

ssh spacely-eng-admin@$node << 'EOF'

sudo mkdir -p /srv/jenkins

sudo chown 1000:1000 /srv/jenkins

EOF

done

# Create Jenkins stack

cat << 'EOF' > jenkins-stack.yml

version: '3.7'

services:

jenkins:

image: jenkinsci/blueocean:latest

user: root

environment:

JAVA_OPTS: '-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false'

JENKINS_OPTS: '--httpPort=18080'

ports:

- target: 18080

published: 18080

protocol: tcp

mode: host

- target: 50000

published: 50000

protocol: tcp

mode: ingress

volumes:

- /srv/jenkins:/var/jenkins_home

- /var/run/docker.sock:/var/run/docker.sock

networks:

- spacely-engineering-network

deploy:

mode: replicated

replicas: 1

placement:

constraints:

- node.labels.role == manager

restart_policy:

condition: on-failure

delay: 5s

max_attempts: 3

networks:

spacely-engineering-network:

external: true

EOF

# Deploy Jenkins

sudo docker stack deploy -c jenkins-stack.yml jenkins

Initial Jenkins Setup

Get the initial admin password:

sudo docker exec $(sudo docker ps -qf "name=jenkins_jenkins") cat /var/jenkins_home/secrets/initialAdminPassword

1

sudo docker exec $(sudo docker ps -qf "name=jenkins_jenkins") cat /var/jenkins_home/secrets/initialAdminPassword
Access Jenkins at http://jenkins.example.com
Enter the initial admin password
Install suggested plugins plus:
- GitLab Plugin
- Docker Pipeline
- Blue Ocean
- Pipeline Stage View
- Yet Another Docker Plugin (YADP)
Create an admin user account
Configure Jenkins URL: http://jenkins.example.com

Configure GitLab Integration

Set up the connection between Jenkins and GitLab:

In GitLab:
- Navigate to Admin Area → Applications
- Create new application:
  - Name: Jenkins
  - Redirect URI: http://jenkins.example.com/securityRealm/finishLogin
  - Scopes: api, read_user, read_repository
- Save the Application ID and Secret
In Jenkins:
- Navigate to Manage Jenkins → Configure Global Security
- Under Security Realm, select “GitLab”
- Configure:
  - GitLab Web URI: http://gitlab.example.com
  - Application ID: [from GitLab]
  - Application Secret: [from GitLab]
Create GitLab API Token:
- In GitLab, go to User Settings → Access Tokens
- Create token with API scope
- In Jenkins, go to Credentials → System → Global credentials
- Add credentials → GitLab API token

Configure Ephemeral Build Agents

Set up the Yet Another Docker Plugin (YADP) for dynamic build agents:

Navigate to Manage Jenkins → Configure System
Under the Cloud section, add new cloud → Docker
Configure Docker cloud for each worker node:

# Worker 1 Configuration
Name: docker-worker-1
Docker Host URI: tcp://10.0.0.7:2376
Server Credentials: None (if Docker is configured without TLS)
Test Connection: Should show "Version x.x.x"

# Worker 2 Configuration  
Name: docker-worker-2
Docker Host URI: tcp://10.0.1.4:2376
Server Credentials: None
Test Connection: Should show "Version x.x.x"

# Worker 1 Configuration

Name: docker-worker-1

Docker Host URI: tcp://10.0.0.7:2376

Server Credentials: None (if Docker is configured without TLS)

Test Connection: Should show "Version x.x.x"

# Worker 2 Configuration

Name: docker-worker-2

Docker Host URI: tcp://10.0.1.4:2376

Server Credentials: None

Test Connection: Should show "Version x.x.x"

For each Docker cloud, add a Docker Agent Template:

Labels: docker-agent
Enabled: Yes
Name: jenkins-agent
Docker Image: jenkins/jnlp-slave:alpine
Remote File System Root: /home/jenkins
Usage: Only build jobs with label expressions matching this node
Idle Timeout: 10
Max Instances: 5

# Container Settings
Volumes: /var/run/docker.sock:/var/run/docker.sock
Extra Hosts: docker-registry.example.com:10.0.250.12
Network: spacely-engineering-network
Run container privileged: Yes

Labels: docker-agent

Enabled: Yes

Name: jenkins-agent

Docker Image: jenkins/jnlp-slave:alpine

Remote File System Root: /home/jenkins

Usage: Only build jobs with label expressions matching this node

Idle Timeout: 10

Max Instances: 5

# Container Settings

Volumes: /var/run/docker.sock:/var/run/docker.sock

Extra Hosts: docker-registry.example.com:10.0.250.12

Network: spacely-engineering-network

Run container privileged: Yes

Pipeline Configuration

Create Sample Pipeline Project

Let’s create a sample Spring Boot project to demonstrate the complete workflow:

In GitLab, create a new project:
- Name: spring-boot-demo
- Visibility: Private
- Initialize with README
Add project files:

# Clone the project
git clone git@gitlab.example.com:2222/your-group/spring-boot-demo.git
cd spring-boot-demo
# Create Dockerfile
cat << 'EOF' > Dockerfile
FROM maven:3.6-jdk-11-slim AS build
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline
COPY src ./src
RUN mvn clean package
FROM openjdk:11-jre-slim
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8080
CMD ["java", "-jar", "app.jar"]
EOF
# Create docker-compose.yml
cat << 'EOF' > docker-compose.yml
version: '3.7'
services:
  app:
    build: .
    image: docker-registry.example.com:5000/spring-boot-demo:${BUILD_TAG:-latest}
    ports:
      - "8080:8080"
    networks:
      - app-network
networks:
  app-network:
    driver: bridge
EOF
# Create Jenkinsfile
cat << 'EOF' > Jenkinsfile
pipeline {
    agent {
        label 'docker-agent'
    }
    
    environment {
        REGISTRY = 'docker-registry.example.com:5000'
        IMAGE_NAME = 'spring-boot-demo'
        DOCKER_CREDENTIALS = credentials('docker-registry-creds')
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Build') {
            steps {
                script {
                    sh 'docker-compose build'
                }
            }
        }
        
        stage('Test') {
            when {
                changeRequest()
            }
            steps {
                script {
                    sh '''
                        docker-compose up -d
                        sleep 10
                        curl -f http://localhost:8080/health || exit 1
                        docker-compose down
                    '''
                }
            }
        }
        
        stage('Push to Registry') {
            when {
                branch 'develop'
            }
            steps {
                script {
                    sh '''
                        docker login -u ${DOCKER_CREDENTIALS_USR} \
                          -p ${DOCKER_CREDENTIALS_PSW} ${REGISTRY}
                        docker push ${REGISTRY}/${IMAGE_NAME}:latest
                    '''
                }
            }
        }
        
        stage('Deploy to Staging') {
            when {
                branch 'develop'
            }
            steps {
                script {
                    sh '''
                        docker service update --image ${REGISTRY}/${IMAGE_NAME}:latest \
                          spring-boot-demo_app || \
                        docker service create --name spring-boot-demo_app \
                          --network spacely-engineering-network \
                          --publish 8080:8080 \
                          ${REGISTRY}/${IMAGE_NAME}:latest
                    '''
                }
            }
        }
        
        stage('Deploy to Production') {
            when {
                branch 'master'
            }
            steps {
                input 'Deploy to production?'
                script {
                    sh '''
                        docker service update --image ${REGISTRY}/${IMAGE_NAME}:latest \
                          spring-boot-demo-prod_app || \
                        docker service create --name spring-boot-demo-prod_app \
                          --network spacely-engineering-network \
                          --publish 8081:8080 \
                          ${REGISTRY}/${IMAGE_NAME}:latest
                    '''
                }
            }
        }
    }
    
    post {
        always {
            cleanWs()
            sh 'docker system prune -f'
        }
        success {
            echo 'Pipeline completed successfully!'
        }
        failure {
            echo 'Pipeline failed!'
        }
    }
}
EOF
# Commit and push
git add .
git commit -m "Add CI/CD pipeline configuration"
git push origin master

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

# Clone the project

git clone git@gitlab.example.com:2222/your-group/spring-boot-demo.git

cd spring-boot-demo

# Create Dockerfile

cat << 'EOF' > Dockerfile

FROM maven:3.6-jdk-11-slim AS build

WORKDIR /app

COPY pom.xml .

RUN mvn dependency:go-offline

COPY src ./src

RUN mvn clean package

FROM openjdk:11-jre-slim

WORKDIR /app

COPY --from=build /app/target/*.jar app.jar

EXPOSE 8080

CMD ["java", "-jar", "app.jar"]

EOF

# Create docker-compose.yml

cat << 'EOF' > docker-compose.yml

version: '3.7'

services:

app:

build: .

image: docker-registry.example.com:5000/spring-boot-demo:${BUILD_TAG:-latest}

ports:

- "8080:8080"

networks:

- app-network

networks:

app-network:

driver: bridge

EOF

# Create Jenkinsfile

cat << 'EOF' > Jenkinsfile

pipeline {

agent {

label 'docker-agent'

}

environment {

REGISTRY = 'docker-registry.example.com:5000'

IMAGE_NAME = 'spring-boot-demo'

DOCKER_CREDENTIALS = credentials('docker-registry-creds')

}

stages {

stage('Checkout') {

steps {

checkout scm

}

stage('Build') {

steps {

script {

sh 'docker-compose build'

}

stage('Test') {

when {

changeRequest()

}

steps {

script {

sh '''

docker-compose up -d

sleep 10

curl -f http://localhost:8080/health || exit 1

docker-compose down

'''

}

stage('Push to Registry') {

when {

branch 'develop'

}

steps {

script {

sh '''

docker login -u ${DOCKER_CREDENTIALS_USR} \

-p ${DOCKER_CREDENTIALS_PSW} ${REGISTRY}

docker push ${REGISTRY}/${IMAGE_NAME}:latest

'''

}

stage('Deploy to Staging') {

when {

branch 'develop'

}

steps {

script {

sh '''

docker service update --image ${REGISTRY}/${IMAGE_NAME}:latest \

spring-boot-demo_app || \

docker service create --name spring-boot-demo_app \

--network spacely-engineering-network \

--publish 8080:8080 \

${REGISTRY}/${IMAGE_NAME}:latest

'''

}

stage('Deploy to Production') {

when {

branch 'master'

}

steps {

input 'Deploy to production?'

script {

sh '''

docker service update --image ${REGISTRY}/${IMAGE_NAME}:latest \

spring-boot-demo-prod_app || \

docker service create --name spring-boot-demo-prod_app \

--network spacely-engineering-network \

--publish 8081:8080 \

${REGISTRY}/${IMAGE_NAME}:latest

'''

}

post {

always {

cleanWs()

sh 'docker system prune -f'

}

success {

echo 'Pipeline completed successfully!'

}

failure {

echo 'Pipeline failed!'

}

EOF

# Commit and push

git add .

git commit -m "Add CI/CD pipeline configuration"

git push origin master

Create Jenkins Pipeline Job

In Jenkins, click “New Item”
Name: spring-boot-demo
Type: Multibranch Pipeline
Configure:
- Branch Sources → Add source → Git
- Project Repository: http://gitlab.example.com/your-group/spring-boot-demo.git
- Credentials: [GitLab credentials]
- Build Configuration → Script Path: Jenkinsfile
Save and scan repository

Complete Development Workflow

Now let’s walk through the complete development workflow that brings everything together:

Step 1: Developer Creates Feature Branch

# Fork the project in GitLab UI first

# Clone your fork
git clone git@gitlab.example.com:2222/your-username/spring-boot-demo.git
cd spring-boot-demo

# Create feature branch
git checkout -b feature/add-new-endpoint

# Make changes to the code
echo "// New feature code" >> src/main/java/com/example/NewFeature.java

# Commit and push
git add .
git commit -m "Add new feature endpoint"
git push origin feature/add-new-endpoint

# Fork the project in GitLab UI first

# Clone your fork

git clone git@gitlab.example.com:2222/your-username/spring-boot-demo.git

cd spring-boot-demo

# Create feature branch

git checkout -b feature/add-new-endpoint

# Make changes to the code

echo "// New feature code" >> src/main/java/com/example/NewFeature.java

# Commit and push

git add .

git commit -m "Add new feature endpoint"

git push origin feature/add-new-endpoint

Step 2: Create Merge Request

Navigate to your fork in GitLab
Click “Create Merge Request”
Configure:
- Source branch: feature/add-new-endpoint
- Target branch: develop (upstream)
- Title: “Add new feature endpoint”
- Description: Describe the changes
- Assign reviewers
Submit merge request

Step 3: Automated CI Pipeline Triggers

When the merge request is created, Jenkins automatically:

Detects the merge request via GitLab webhook
Spins up an ephemeral build agent
Checks out the code
Builds the Docker image
Runs tests
Reports status back to GitLab

Pipeline Status in GitLab

GitLab will show the pipeline status directly in the merge request:

✅ Pipeline passed – Ready to merge
❌ Pipeline failed – Review logs and fix issues
🔄 Pipeline running – Wait for completion

Step 4: Code Review and Merge

Once the pipeline passes and code review is complete:

The reviewer approves the merge request
Click the “Merge” button in GitLab
Code is merged into the develop branch
Jenkins automatically:
- Builds the updated develop branch
- Pushes image to Docker Registry
- Deploys to staging environment

Step 5: Promote to Production

# When ready for production release
git checkout master
git merge develop
git push origin master

# Jenkins will:
# 1. Build master branch
# 2. Request manual approval
# 3. Deploy to production after approval

# When ready for production release

git checkout master

git merge develop

git push origin master

# Jenkins will:

# 1. Build master branch

# 2. Request manual approval

# 3. Deploy to production after approval

Advanced Pipeline Features

Load Balancing Build Servers

The pipeline intelligently distributes builds across available workers:

// Advanced Jenkinsfile with server health checks
def selectBuildServer() {
    def servers = [
        [name: 'worker-1', url: 'http://10.0.0.7:8080/health'],
        [name: 'worker-2', url: 'http://10.0.1.4:8080/health']
    ]
    
    def availableServers = []
    servers.each { server ->
        try {
            def response = httpRequest url: server.url, 
                                      timeout: 5, 
                                      validResponseCodes: '200'
            availableServers.add(server.name)
        } catch (Exception e) {
            echo "Server ${server.name} is unavailable"
        }
    }
    
    if (availableServers.isEmpty()) {
        error "No build servers available!"
    }
    
    return availableServers[new Random().nextInt(availableServers.size())]
}

pipeline {
    agent {
        label selectBuildServer()
    }
    // ... rest of pipeline
}

// Advanced Jenkinsfile with server health checks

def selectBuildServer() {

def servers = [

[name: 'worker-1', url: 'http://10.0.0.7:8080/health'],

[name: 'worker-2', url: 'http://10.0.1.4:8080/health']

]

def availableServers = []

servers.each { server ->

try {

def response = httpRequest url: server.url,

timeout: 5,

validResponseCodes: '200'

availableServers.add(server.name)

} catch (Exception e) {

echo "Server ${server.name} is unavailable"

}

if (availableServers.isEmpty()) {

error "No build servers available!"

}

return availableServers[new Random().nextInt(availableServers.size())]

}

pipeline {

agent {

label selectBuildServer()

}

// ... rest of pipeline

}

Blue Ocean Interface

Access the modern Blue Ocean interface for better pipeline visualization:

Navigate to http://jenkins.example.com/blue
Features include:
- Visual pipeline editor
- Real-time log streaming
- Branch and pull request visualization
- Pipeline analytics and trends

Monitoring and Alerting

Set up notifications for pipeline events:

// Add to Jenkinsfile
post {
    failure {
        emailext (
            subject: "Pipeline Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER}",
            body: "The pipeline has failed. Check logs at ${env.BUILD_URL}",
            to: 'team@example.com'
        )
    }
    success {
        slackSend (
            channel: '#ci-cd',
            color: 'good',
            message: "Pipeline succeeded: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
        )
    }
}

// Add to Jenkinsfile

post {

failure {

emailext (

subject: "Pipeline Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER}",

body: "The pipeline has failed. Check logs at ${env.BUILD_URL}",

to: 'team@example.com'

)

}

success {

slackSend (

channel: '#ci-cd',

color: 'good',

message: "Pipeline succeeded: ${env.JOB_NAME} #${env.BUILD_NUMBER}"

)

}

Troubleshooting Common Issues

Issue	Solution
Jenkins can’t connect to GitLab	Verify network connectivity and DNS resolution. Check GitLab API token permissions.
Docker commands fail in the pipeline	Ensure Docker socket is mounted and the agent has proper permissions.
Registry push fails	Verify registry certificates are installed on all nodes. Check credentials.
Build agents not starting	Check the Docker daemon on worker nodes. Verify YADP configuration.
Services unreachable	Verify load balancer health probes. Check network security groups.

Best Practices

Security:
- Regularly update all components
- Use secrets management for credentials
- Implement RBAC in Jenkins and GitLab
- Enable audit logging
Performance:
- Use Docker layer caching
- Implement parallel stages in pipelines
- Clean up old images and containers regularly
- Monitor resource usage
Reliability:
- Implement health checks for all services
- Set up automated backups
- Use version tags for production deployments
- Maintain staging environment parity with production
Development Workflow:
- Enforce branch protection rules
- Require code reviews for merges
- Automate as much testing as possible
- Keep pipelines fast (< 10 minutes ideally)

Conclusion

Congratulations! You’ve successfully built a comprehensive, private CI/CD solution in Azure. This infrastructure provides your team with:

Complete Privacy: All components run within your private network, accessible only via VPN
High Availability: Clustered services ensure continuous operation
Scalability: Easy to add more build capacity by adding worker nodes
Modern Workflow: Git-based development with automated testing and deployment
Full Control: Complete ownership of your CI/CD infrastructure

This solution serves as a solid foundation that can be adapted to meet your specific needs. Consider these potential enhancements:

Add SonarQube for code quality analysis
Integrate security scanning tools
Implement artifact management with Nexus or Artifactory
Add Kubernetes for more advanced container orchestration
Integrate monitoring with Prometheus and Grafana

The beauty of this setup is that once you see its benefits firsthand, you’ll never look back. The investment in setting up a proper CI/CD pipeline pays dividends in improved code quality, faster delivery times, and happier development teams.

Additional Resources

This is Part 4 of a 4-part series on setting up a private CI/CD solution in Azure.

Thank you for following along with this series. You are welcome to adapt this solution to meet your needs, and don’t hesitate to provide feedback or share your improvements!

Part 4: Jenkins Configuration and Complete Workflow

Key Takeaways

Jenkins Configuration

Deploy Jenkins Master

Initial Jenkins Setup

Configure GitLab Integration

Configure Ephemeral Build Agents

Pipeline Configuration

Create Sample Pipeline Project

Create Jenkins Pipeline Job

Complete Development Workflow

Step 1: Developer Creates Feature Branch

Step 2: Create Merge Request

Step 3: Automated CI Pipeline Triggers

Step 4: Code Review and Merge

Step 5: Promote to Production

Advanced Pipeline Features

Load Balancing Build Servers

Blue Ocean Interface

Monitoring and Alerting

Troubleshooting Common Issues

Best Practices

Conclusion

Additional Resources

Start typing and press enter to search