Local Jenkins Development Environment on Minikube on OSX

Developing Jenkinsfile pipelines is hard. I think my world record for the number of attempts to get a working Jenkinsfile is around 20. When you have to continually push and run your pipeline on a managed Jenkins instance, the feedback cycle is long. And the primary bottleneck to developer productivity is the length of the feedback cycle.

So I wanted to deploy a local version of Jenkins to aid development. This will run on Minikube on your laptop but will work just like a managed version. The only downside is that it can take a while for Minikube to get setup and to download all the containers. I would recommend only using this setup when you have a task to significantly change a Jenkinsfile (e.g. a new one or a refactoring). For simple changes it’s probably easiest to use your hosted version.

Prerequisites

Install minikube. The easiest way to do this on OSX is to enable Kubernetes through the Preferences->Kubernetes section. An error of Kubernetes failed to start is usually just a timeout. kubectl get nodes will probably work. Right click on the Docker menu then press Kubernetes->Disable local cluster. Then go to the preferences and repeat the enabling again.
Install Helm

Helm Installation

Make sure the current kubeconfig context is pointing to your local server (not a production server!)
helm init

Jenkins configuration

Create a file called values.yaml with the content below.

Master:
  ImageTag: "lts"
  ServiceType: NodePort
  InstallPlugins:
    - kubernetes:1.14.0
    - workflow-job:2.31
    - workflow-aggregator:2.6
    - credentials-binding:1.17
    - git:3.9.1
    - filesystem_scm:2.1
Agent:
  Enabled: true
  volumes:
    - type: HostPath
      hostPath: /Users
      mountPath: /Users
    - type: HostPath
      hostPath: /var/run/docker.sock
      mountPath: /var/run/docker.sock
rbac:
  install: true
Persistence:
  volumes:
    - name: source-code
      hostPath:
        path: /Users
  mounts:
    - mountPath: /Users
      name: source-code

Important Settings

There are a few important settings in the values file. First is the list of installed plugins. The important one is the filesystem_scm plugin, which allows you to read from disk rather than a repository. Feel free to add more plugins as you need them.

The next is the volumes. You need to make sure that you mount your source code directory into the container. Note that if you are using docker for mac, then these directories are shared through the docker app. Go to Preferences->File Sharing. These are the only directories that can be shared. /Users is shared by default, so if you have your code in your home directory, you should be good to go.

Note you also need the same path mounted in the Persistence section. An annoying name, but this is mounted into the Jenkins master. The master also needs the files so that it can read the Jenkinsfile if you have one.

Jenkins Installation

helm install --name jenkins --values values.yaml stable/jenkins

The subsequent pull of the jenkins containers and starting of jenkins may take a while. Watch the progress with kubectl get pods. You can speed this up by giving Docker more CPU/RAM.

If the pod fails to start, start debugging why. Check disk usage. Check RAM usage. If you get an obscure error saying The node was low on resource: imagefs., it is likely that you will need to increase your Docker CPU to 4, RAM to 6GB and Swap to 2GB. Your laptop may become a mobile heating device. Worst case, reset docker to factory settings and try again.

Accessing Jenkins

Once everything is up and running then you should be able to browse to the nodeport on the IP address that the machine is running from. For people using docker for mac, then you can get this address with:

open "http://localhost:$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services jenkins)"

The username is admin and the password is described in the helm status jenkins output:

printf $(kubectl get secret --namespace default jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

Add Jobs and Credentials

Add a custom pipeline job which uses the Filesystem SCM plugin, and set path to the directory that holds your code (/Users/...). Also add any credentials that you might need.

Running a Jenkinsfile Declarative Pipeline

There is one key option that must be set in the Kubernetes Pipeline options. This is inheritFrom 'default'. For example:

pipeline {
  agent {
    kubernetes {
      defaultContainer 'jnlp'
      inheritFrom 'default'
      yaml """
---
apiVersion: v1
kind: Pod

This is necessary so that the custom container also has access to the host volume mount (/Users) that the jnlp container is using. View the inheritance documenation here.

Credits

This was strongly inspired thanks to the work from @garunvagidov here.