Build Docker containers on Kubernetes with Jenkins and Kaniko

This writeup documents the current best way to build Docker containers within transient Jenkins agents inside a Kubernetes cluster. This setup has unique features and unique caveats you need to consider, and intend to save you, dear reader, the time I invested trying different solutions.

Setup

These are the components used in this setup. You can replace the K8S provider and the solution and considerations will hold. You can also change the destination registry for your images, and this will require a change in the authentication method.

• GKE cluster

• GCR for storing images and cache layers

• Kaniko official docker image to build Dockerfiles

• Kubernetes and GCP Service Account to provide credentials to the workers following the Principle of Least Privilege.

• Jenkins official docker images for Master and for the Worker

Why not docker build?

The docker command requires a working docker daemon, which requires setting up several components, customizing the Jenkins docker images, and more work. Using Kaniko allows us to use the official images and to avoid a lot of work. The resulting images are very similar to the ones build by docker and also totally compatible.

In a nutshell

The steps for this setup are:

1. Create the Kubernetes cluster

2. Create a ServiceAccount on GCP with StorageAdmin privileges to be able to read and push images to the registry. (This may not be needed depending on your setup)

3. Create a ServiceAccount in Kubernetes

4. Join both ServiceAccounts

5. Use helm to install the Jenkins Master. Do not use the Jenkins Controller as it is broken at the time of this writing. helm install Jenkins-ci jenkinsci/Jenkins.

6. Add this code at the top of your Jenkinsfile:

pipeline {
  agent {
    kubernetes {
      //cloud 'kubernetes'
      defaultContainer 'kaniko'
      yaml """
kind: Pod
spec:
  serviceAccountName: jenkins-sa
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug-539ddefcae3fd6b411a95982a830d987f4214251
    imagePullPolicy: Always
    command:
    - sleep
    args:
    - 9999999
"""
    }
  }

Add this line to your Jenkinsfile to build and upload your image:

sh '/kaniko/executor -f `pwd`/Dockerfile -c `pwd` --cache=true --destination=<DESTINATION-REPO><IMAGE>:$CI_COMMIT_TAG'

After executing this job, your container will be building and uploaded to your GCP registry. For ECR or other registries, you need to set up a different authentication mechanism.

Would you like to receive our newsletter with more TeraTips? Leave us your comments.

Carlos Barroso

Senior MLOps Engineer

Teracloud