Kubernetes: Building Docker images within a cluster

tl;dr: For building Docker images in a container without using Docker it would be useful to consider kaniko. Furthermore it would be great to know how to run kaniko locally and also within a Kubernetes cluster.

Furthermore you have to know about Docker in Docker and its alternative kaniko which enables you to build Docker images without using Docker. And last but not least you have to learn how to set up building images within a Kubernetes cluster.

For more details on Kubernetes in general see Recommended reading: Kubernetes in Action.

Starting with a real-world use case

In our ecosystem at the native web we have a lot of private Docker images that have to be stored somewhere. Hence we wanted to implement some kind of private Docker Hub. Regarding this, there are two features of the public Docker Hub that we were especially interested in.

First, we wanted to create a kind of queue that can asynchronously build Docker images within Kubernetes. The second feature we wanted to implement was to push the built images to a private Docker Registry.

The most common way to get these features implemented is to use the Docker CLI directly:

$ docker build ...
$ docker push ...

But inside the Kubernetes cluster we have containers that are mainly based on very rudimentary and small Linux images, which do not contain Docker by default. If we now want to use Docker (for example docker build ...) within a container we need something like Docker in Docker.

What is wrong with Docker in Docker?

To build container images with Docker we need a Docker Daemon within a container up and running, called Docker in Docker. A Docker Daemon is a virtualized environment, and a container within Kubernetes is also by nature virtualized. This means if we want to get a Docker Daemon up and running within a container, we need to use nested virtualization. For that we have to start the container in a privileged mode, to get access to the host system. However, this brings up some security issues, such as dealing with different filesystems (the host filesystem and the container filesystem) or using the build cache from the host system, and that is why we didn't want to use Docker in Docker.

Introducing kaniko

Alongside Docker in Docker there is another solution called kaniko. It is a tool written in Go that builds container images from Dockerfiles – without using Docker. Afterwards it pushes the built images to a specified Docker Registry. The recommended way to set up kaniko is to use the readymade executor image which can be started as a Docker container or as a container within Kubernetes.

Please keep in mind that kaniko is under ongoing development and maybe not all commands from the Dockerfile are supported right now, e.g. the --chown flag for the COPY command.

Running kaniko

If you now want to get kaniko up and running, you have to know the arguments that need to be provided to the kaniko container. At first you have to mount your Dockerfile and its dependencies into the kaniko container. Locally (with Docker) you can do this with the -v <path-on-host>:<path-inside-container> parameter. On Kubernetes you may want to use Volumes.

After mounting the Dockerfile and its dependencies into the kaniko container you also have to specify the --context argument, which is the path to the mounted directory (inside the container). The next argument is --dockerfile, which is the path to the Dockerfile (including the file name). Another important argument is --destination, which represents the full url to the Docker Registry (including the image's name and tag).

Running locally

There are several ways to get kaniko up and running. To get a quick start without the need to set up a Kubernetes cluster you can start kaniko on your local machine using Docker. Use the following command to start kaniko locally:

$ docker run \
  -v $(pwd):/workspace \
  gcr.io/kaniko-project/executor:latest \
  --dockerfile=<path-to-dockerfile> \
  --context=/workspace \
  --destination=<repo-url-with-image-name>:<tag>

If your destination Docker Registry has authentication enabled then kaniko has to login first. This can be achieved by mounting the local Docker config.json file to the kaniko container, as it already contains the credentials for the destination Docker Registry. For that, use the following command:

$ docker run \
  -v $(pwd):/workspace \
  -v ~/.docker/config.json:/kaniko/.docker/config.json \
  gcr.io/kaniko-project/executor:latest \
  --dockerfile=<path-to-dockerfile> \
  --context=/workspace \
  --destination=<repo-url-with-image-name>:<tag>

Running on Kubernetes

For our real-world use case we wanted to run kaniko within a Kubernetes cluster. Furthermore we wanted something such as a queue for building images. If a build or push to the Docker Registry fails then it would be helpful if the process would automatically run again. For this Kubernetes knows the concept of a Job. With the configuration backoffLimit you can configure how often the process should be retried automatically.

The easiest way to get a Dockerfile and its dependencies into the kaniko container is to use a PersistentVolumeClaim (in our example called: kaniko-workspace). This will be mounted as a directory to the container and requires that the data already exists in the kaniko-workspace. Let's assume that another container has already stored the Dockerfile and its dependencies in the /my-build directory inside the kaniko-workspace.

Please keep in mind that there is an issue with the PersistentVolumeClaim on AWS. If you create a PersistentVolumeClaim on AWS then it will be created on one node in your AWS cluster and it is only available on this node. This means if you now start a kaniko Job, and if it starts on another node, then the kaniko Job can't start, because the PersistentVolumeClaim is not available. Hopefully the Amazon Elastic File System will be available soon in Kubernetes, so that this issue gets fixed.

A Job resource for building Docker images typically looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: build-image
spec:
  template:
    spec:
      containers:
      - name: build-image
        image: gcr.io/kaniko-project/executor:latest
        args:
          - "--context=/workspace/my-build"
          - "--dockerfile=/workspace/my-build/Dockerfile"
          - "--destination=<repo-url-with-image-name>:<tag>"
        volumeMounts:
        - name: workspace
          mountPath: /workspace
      volumes:
      - name: workspace
        persistentVolumeClaim:
          claimName: kaniko-workspace
      restartPolicy: Never
  backoffLimit: 3

If your destination Docker Registry requires a login, you have to pass a config.json file with credentials to the kaniko container. The easiest solution here is also to mount a PersistentVolumeClaim to the container, which already contains the config.json file. A special feature here is that the PersistentVolumeClaim will not be mounted as a directory rather as an file to the path /kaniko/.docker/config.json inside the kaniko container:

apiVersion: batch/v1
kind: Job
metadata:
  name: build-image
spec:
  template:
    spec:
      containers:
      - name: build-image
        image: gcr.io/kaniko-project/executor:latest
        args:
          - "--context=/workspace/my-build"
          - "--dockerfile=/workspace/my-build/Dockerfile"
          - "--destination=<repo-url-with-image-name>:<tag>"
        volumeMounts:
        - name: config-json
          mountPath: /kaniko/.docker/config.json
          subPath: config.json
        - name: workspace
          mountPath: /workspace
      volumes:
        - name: config-json
          persistentVolumeClaim:
            claimName: kaniko-credentials
        - name: workspace
          persistentVolumeClaim:
            claimName: kaniko-workspace
      restartPolicy: Never
  backoffLimit: 3

If you want to check the status of the running build Job then you can use kubectl for it. To get the status filtered to stdout use the following command:

$ kubectl get job build-image -o go-template='{{(index .status.conditions 0).type}}'

Summary

In this article you have learned why Docker in Docker may not be the best solution for building Docker images within Kubernetes. You got insights to kaniko, an alternative to Docker in Docker, which allows you to build Docker images without using Docker. Furthermore you have learned how to write Job resources that can build Docker images within Kubernetes. And last but not least you have learned how to get the status of a running Job.

Twitter Facebook LinkedIn

Jan-Hendrik Grundhöfer

Core development

Since everybody is deeply rooted somewhere, we want you to live wherever you want. Although we have various offices, any place with a cozy desk, a cellular network and web access is fine. On earth and beyond.