Jenkins Modernization
When to Use
- Migrating legacy Freestyle jobs to Declarative Pipeline
- Adding Kubernetes agents to eliminate static build slaves
- Extracting shared logic into Jenkins Shared Libraries
- Evaluating migration from Jenkins to GitHub Actions
- Auditing credential management (no plaintext secrets in Jenkinsfile)
Declarative Pipeline Structure
// Jenkinsfile
pipeline {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
spec:
containers:
- name: python
image: python:3.11-slim
command: ['cat']
tty: true
resources:
requests:
cpu: 500m
memory: 1Gi
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
command: ['cat']
tty: true
"""
}
}
options {
timeout(time: 30, unit: 'MINUTES')
buildDiscarder(logRotator(numToKeepStr: '20'))
disableConcurrentBuilds() // prevent parallel builds of same branch
timestamps()
}
environment {
IMAGE_REGISTRY = "ghcr.io/my-org"
IMAGE_NAME = "airflow"
}
stages {
stage('Lint') {
steps {
container('python') {
sh '''
pip install ruff sqlfluff --quiet
ruff check .
sqlfluff lint dbt/models --dialect trino
'''
}
}
}
stage('Test') {
parallel {
stage('Unit Tests') {
steps {
container('python') {
sh 'pytest tests/unit/ -v --junitxml=test-unit.xml'
}
}
}
stage('DAG Validation') {
steps {
container('python') {
sh '''
pip install apache-airflow==2.8.0 --quiet
python -c "
from airflow.models import DagBag
bag = DagBag(dag_folder='dags/', include_examples=False)
assert not bag.import_errors
"
'''
}
}
}
}
}
stage('Build Image') {
when {
branch 'main'
}
steps {
container('kaniko') {
withCredentials([usernamePassword(
credentialsId: 'ghcr-credentials',
usernameVariable: 'REGISTRY_USER',
passwordVariable: 'REGISTRY_PASS'
)]) {
sh '''
mkdir -p /kaniko/.docker
echo '{"auths":{"ghcr.io":{"auth":"'"$(echo -n ${REGISTRY_USER}:${REGISTRY_PASS} | base64)"'"}}}' \
> /kaniko/.docker/config.json
/kaniko/executor \
--context . \
--dockerfile Dockerfile \
--destination ${IMAGE_REGISTRY}/${IMAGE_NAME}:${GIT_COMMIT:0:7} \
--cache=true \
--cache-repo ${IMAGE_REGISTRY}/${IMAGE_NAME}-cache
'''
}
}
}
}
stage('Deploy Staging') {
when { branch 'main' }
steps {
container('python') {
withCredentials([file(credentialsId: 'k8s-staging-kubeconfig', variable: 'KUBECONFIG')]) {
sh '''
kubectl set image deployment/airflow-scheduler \
scheduler=${IMAGE_REGISTRY}/${IMAGE_NAME}:${GIT_COMMIT:0:7} \
-n airflow-staging
kubectl rollout status deployment/airflow-scheduler -n airflow-staging
'''
}
}
}
}
}
post {
always {
junit 'test-*.xml'
cleanWs()
}
failure {
slackSend(
channel: '#data-eng-alerts',
color: 'danger',
message: "Pipeline FAILED: ${env.JOB_NAME} #${env.BUILD_NUMBER}\n${env.BUILD_URL}"
)
}
success {
archiveArtifacts artifacts: 'dbt/target/manifest.json', fingerprint: true
}
}
}
Shared Library Structure
jenkins-shared-library/
vars/
dbtRun.groovy # callable as dbtRun() in Jenkinsfile
dockerBuild.groovy
slackNotify.groovy
src/
org/myorg/
DbtHelper.groovy # class-based helpers
SlackClient.groovy
resources/
pod-templates/
python-pod.yaml
spark-pod.yaml
// vars/dbtRun.groovy
def call(Map config = [:]) {
def target = config.get('target', 'ci')
def select = config.get('select', '')
def baseDir = config.get('baseDir', 'dbt')
container('python') {
withCredentials([
string(credentialsId: 'trino-host', variable: 'DBT_TRINO_HOST'),
usernamePassword(credentialsId: 'trino-creds',
usernameVariable: 'DBT_TRINO_USER',
passwordVariable: 'DBT_TRINO_PASSWORD')
]) {
sh """
cd ${baseDir}
dbt deps
dbt run --target ${target} ${select ? '--select ' + select : ''}
dbt test --target ${target} ${select ? '--select ' + select : ''}
"""
}
}
}
// Jenkinsfile — using shared library
@Library('jenkins-shared-library@main') _
pipeline {
agent any
stages {
stage('dbt') {
steps {
dbtRun(target: 'staging', select: 'state:modified+')
}
}
}
}
Multibranch Pipeline (Jenkinsfile Discovery)
// Jenkins Job DSL or via UI:
// New Item → Multibranch Pipeline
// Branch Sources → GitHub
// Build Configuration → Jenkinsfile path: Jenkinsfile
// Per-branch behavior in Jenkinsfile:
pipeline {
agent any
stages {
stage('Test') {
steps { sh 'pytest tests/' }
}
stage('Deploy') {
when {
anyOf {
branch 'main'
branch 'release/*'
}
}
steps {
sh './scripts/deploy.sh ${BRANCH_NAME}'
}
}
}
}
Credential Management Best Practices
// ✅ Always use withCredentials — never echo secrets or store in env
withCredentials([
string(credentialsId: 'airflow-fernet-key', variable: 'FERNET_KEY'),
usernamePassword(credentialsId: 'aws-creds',
usernameVariable: 'AWS_ACCESS_KEY_ID',
passwordVariable: 'AWS_SECRET_ACCESS_KEY')
]) {
sh 'python deploy.py' // FERNET_KEY, AWS_* are available only in this block
}
// ✅ File credentials for kubeconfig, service accounts
withCredentials([file(credentialsId: 'gcp-service-account', variable: 'GOOGLE_APPLICATION_CREDENTIALS')]) {
sh 'gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS'
}
// ❌ Never do this:
// environment {
// AWS_SECRET = "hardcoded-secret" // visible in build logs, Jenkinsfile
// }
Jenkins Configuration as Code (JCasC)
# jenkins.yaml — managed by JCasC plugin
jenkins:
systemMessage: "Data Platform Jenkins"
numExecutors: 0 # all builds run on agents, not master
clouds:
- kubernetes:
name: "kubernetes"
serverUrl: "https://kubernetes.default.svc"
namespace: "jenkins"
jenkinsUrl: "http://jenkins.jenkins.svc.cluster.local:8080"
podRetention: "never"
templates:
- name: "python-agent"
label: "python"
containers:
- name: "python"
image: "python:3.11-slim"
command: "cat"
ttyEnabled: true
credentials:
system:
domainCredentials:
- credentials:
- usernamePassword:
id: "ghcr-credentials"
username: "${GHCR_USER}"
password: "${GHCR_TOKEN}" # injected from Kubernetes secret
scope: GLOBAL
Migration Path to GitHub Actions
| Jenkins Concept | GitHub Actions Equivalent |
|---|
| Jenkinsfile | .github/workflows/*.yml |
| Shared Library | Reusable workflows / composite actions |
| Kubernetes agent pod | runs-on: [self-hosted, k8s] |
withCredentials | secrets: GITHUB_TOKEN + OIDC |
| Multibranch Pipeline | Workflow triggers with branches: filter |
| Blue Ocean | GitHub Actions UI |
post { failure {} } | if: failure() job step |
disableConcurrentBuilds() | concurrency: cancel-in-progress: true |
Migration decision criteria:
- Migrate if: team already uses GitHub, pipeline is < 200 lines, no complex shared library
- Stay on Jenkins if: on-prem with air-gap, heavy shared library investment, complex plugin ecosystem
Anti-Patterns
- Freestyle jobs with shell scripts — no code review, no history, not reproducible; migrate to Declarative Pipeline in Jenkinsfile.
- Secrets in Jenkinsfile environment block — stored in plaintext in VCS; use
withCredentials with Jenkins Credential Store.
- Long-running builds on Jenkins master — master handles scheduling, not execution; all builds should use pod agents.
- No
disableConcurrentBuilds() — two developers push simultaneously, both builds deploy to the same environment; add concurrent build prevention.
- Not archiving dbt manifest.json — slim CI requires the manifest; archive it as a build artifact and download in subsequent runs.
References
- Jenkins Declarative Pipeline:
jenkins.io/doc/book/pipeline/syntax/
- Jenkins Shared Libraries:
jenkins.io/doc/book/pipeline/shared-libraries/
- JCasC plugin:
plugins.jenkins.io/configuration-as-code/
- Kaniko (rootless Docker builds):
github.com/GoogleContainerTools/kaniko
- Related skills:
[[dataops-cicd-pipeline-review]], [[infra-docker-best-practices]], [[dataops-github-actions-optimizer]]