Docker image building on GitLab CI

Published on 9/26/20 at 7:14 PM. Updated on 4/25/21 at 1:28 PM.

I decided to reconsider ma actual pipeline and to user my own GitLab Runner. It has been harder than expected.

Docker image building on GitLab CI

Photo by Chris Liverani on Unsplash


Recently, Docker Hub announced its new image retention policy to delete images which have been unused for more than 6 months, and GitLab announced a free build minutes reduction to 400 minutes.
Well, only free accounts are affected and these constraints aren't blocking for my own projects, but it motivated me to install my own runner and get rid of Docker Compose in my CI.

For those who are not familiar with the mentioned technologies:

  • Continuous Integration (or CI) is a practice where you continously test an application to detect errors as soon as possible.
  • GitLab CI is a CI tool. Generally, when code is pushed to the Git repository, a pipeline is triggered.
  • a runner is responsible of the execution of a pipeline, GitLab offer theirs (the « shared runners ») but you can use your own runner.
  • Docker is a container technology, many CI tools execute jobs (the tasks of a pipeline) in container to have an isolated environment.
  • Docker Compose is a tool which eases the use of multiple containers at once.

And Docker in Docker (« DinD » in short) means executing Docker in a Docker container. It can be interesting as I want to build a Docker image during my pipeline which will be executing containers itself.


The starting point

To these days I used shared runners on GitLab CI with the Docker executor, using DinD with an image I created which contains Docker Compose too: aymdev/dind-compose.

The goal was to start a Docker Compose stack for some jobs (e.g. functional tests, needing a database next to the app), and because the app (this website) is deployed in a Docker Swarm cluster, it is relevant to directly test the deployed image.

But my image was just a try I don't maintain, has not much interest, and starting the whole application stack is a bit overkill (4 containers started for only 2 used in tests). And my docker-compose.yml files were accumulating, which was bad for maintainability.


First try, first fail

During the GitLab Runner installation comes the executor choice. As I have control of the host, I thought that the shell executor would be a good candidate for multiple reasons. It simply executes jobs on the runner host, therefore:

  • I just need to install Docker and Docker Compose to use their commands in CI
  • images are saved in the host registry, we can benefit from Docker layer caching
  • if SSH is needed it can be configured on the host without passing a private key in a GitLab CI custom variable

But I didn't think of one issue: concurrency.
All jobs will share the same environment, if many of them run simultaneously they might get into conflicts. We must care about:

  • naming containers uniquely
  • not stopping containers to make a « clean up » when a job ends, which could stop containers from another job
  • not using ports on the host which could prevent some other containers to start
  • ...

And let's not discuss storage management (accumulating images), running containers because the pipeline broke, etc.

In short, I reinitialized the runner host.


Back to the Docker executor

The host I chose is a Debian 10. I think it's better to use a memory optimized instance while storage is not important, especially if you don't have much to spend.

GitLab Runner installation

In a SSH connection to the host:

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
export GITLAB_RUNNER_DISABLE_SKEL=true; sudo -E apt-get install gitlab-runner

Source: Install GitLab Runner using the official GitLab repositories

As I use Debian, I followed the APT pinning recommendation.

Docker installation

We'll install Docker only, no Docker Compose:

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Source: Install Docker Engine on Debian

Registering the runner

Before registering the runner, we need to get a token to bind our runner to the project or group. To find it from GitLab UI, go in Settings > CI/CD and open the Runners section.

Note: from this section, for a project (not at the group level), you can disable shared runners to enforce usage of your own runners.

Once the token is copied, we can register the runner. The following command is interactive:

sudo gitlab-runner register

For the coordinator URL, I will use https://gitlab.com as I don't have my own GitLab instance.
Paste the token, give a description (will be shown in the CI/CD UI), tags (optional), and then the executor: docker.
A last question asks for a default image, let's choose docker:19.03.13.

Configuring the runner for DinD

The GitLab documentation shows 3 ways to use Docker in Docker.

Docker socket binding

The Docker socket binding technique means making a volume of /var/run/docker.sock between host and containers.
It may seem convenient, but it all containers would share the same Docker daemon. It means we would get the same concurrency issues we have with the shell executor (unique names, etc).

TLS disabled

Acceptable solution to use when we can't do otherwise.

GitLab: « For example, you have no control over the GitLab Runner configuration that you are using. »

But we have control over the configuration.

TLS enabled

This is the recommended way, for security reasons. Open the runner configuration file: /etc/gitlab-runner/config.toml:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "GitLab Runner"
  url = "https://gitlab.com/"
  token = "super-secret-token"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03"
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Add privileged = true in the [runners.docker] section, the privileged mode is mandatory to use DinD. In the same section, about TLS, add the "/certs/client" volume.

To avoid that the runner only run one job at a time, change the concurrent value on the first line.

The file should look like this (shortened):

concurrent = 4
...

[[runners]]
  ...
  [runners.docker]
    ...
	privileged = true
    volumes = ["/cache", "/certs/client"]
    ...

The runner should not need to be restarted but you can do it with the following commands:

gitlab-runner restart
sudo service gitlab-runner restart

Pipeline

We finished with the runner !
We now have to work on the .gitlab-ci.yml file.

Making DinD work

If needed, specify the default image to use:

image: docker:19.03.13

It is particularly useful if you want to change the image version without changing the runner configuration.

In the default variables, add one for TLS:

variables:
    DOCKER_TLS_CERTDIR: "/certs"

Finally, to use Docker in Docker, add a docker service:

services:
    -   name: docker:19.03.13-dind
        alias: docker

Using the same Docker image in a pipeline

To avoid building a Docker image at each job, it can be built in a first job, pushed to the image registry provided by GitLab, and pulled in the next jobs.

Its name will have to be unique per pipeline. As it will be used in every job, let's add its name as a variable using predefined environment variables:

variables:
	...
    DOCKER_CI_IMAGE: "$CI_REGISTRY_IMAGE:ci-$CI_COMMIT_SHORT_SHA"

As you will need to log into the registry for each job, create a YAML anchor to use in the before_script key of each job:

.docker-login: &docker-login
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

A first job creates the image to use in the whole pipeline:

docker-build:
    before_script:
        - *docker-login
    script:
        - docker build --pull -t "$DOCKER_CI_IMAGE" -f ./Dockerfile .
    after_script:
        - docker push $DOCKER_CI_IMAGE

The next jobs will only have to log into the registry and the image will be downloaded when needed:

example-job:
    before_script:
        - *docker-login
    script:
        - docker run -d $DOCKER_CI_IMAGE

Remplacing Docker Compose

For some jobs, starting a single container will suffice to test the application using docker exec.
But for others, it can get a bit more complicated: functional tests depending on a database.

« Then we add a service to the job with the services key ... »

But no, because this service will only be accessible for the job container, not for our application container which runs inside the job's one. And the possible commands to contact the service are more complex than the solution I have.

Docker Compose allows you to easily start multiple containers, but it has no more feature than Docker itself. So we can recreate an ideal environment test in a few commands.
Example:

test-suite:
    variables:
        POSTGRES_PASSWORD: postgres
        DATABASE_URL: "pgsql://postgres:postgres@postgres/postgres"
    before_script:
        - *docker-login
        - docker network create app-net
        - docker run -d --network app-net --name postgres -e POSTGRES_PASSWORD postgres:11.7-alpine
        - docker run -d --network app-net --name app -e DATABASE_URL $DOCKER_CI_IMAGE
        - docker exec app ./path/to/database-migrations
    script:
        - docker exec app ./path/to/your/tests

Conclusion

If you read until now, thank you !

Well, Docker in Docker works well, but has its drawbacks, like Docker layer caching which needs some more commands to be used.

I didn't talk about a complete pipeline structure or how to reduce its execution time but it will probably be the subject of a future post. I hope you liked this first post, at least I had fun !

Comments: 0

Robot invasion coming from robohash.org