Introduction to Gitlab CI

I'm giving a Gitlab CI workshop on September 16th (in English) and on September 23rd (in French) during our local Free Software Week and I wrote a bunch of notes for the participants.

I thought I would publish them here so others can refer to it. Feedback is welcome! I also wrote a French version of these notes.

Why is continuous integration important?

There are plenty of reasons to start doing continuous integration (CI). Here's a few:

1. Testing your code. If you are not already testing the code you write, starting to use a CI platform is good way to do so. Automated tests let you find errors you would otherwise have missed. It also lets you know if the code you wrote works.

2. Testing your code in radically different environments. Having tests you can run locally on your machine is better than nothing, but what tells you your code will work outside of your dev environment? Using a CI platform lets you easily test your code in environments you don't normally run.

3. Automatically test your code. Using a CI that works with a git hosting platform forces you to run your tests every time you push. If you are using a merge/push request workflow, this means code in your master branch will always have passed your tests.

4. Automatically deploy your code. You can go a step further and decide to use a CI platform to automatically publish code on archives like PyPi or NPM if your code passes all your tests. This is known as continuous delivery (CD).

Introduction to Gitlab CI's functionalities

Outline

To use Gitlab CI, you need to have a project on a Gitlab instance. When you push a git commit to that repository, if a .gitlab-ci.yml file is present in your repository, Gitlab will run a script to test your code in a runner, an external machine used for tests.

Most of the Gitlab instances have shared runners everyone can use. Some people prefer to setup private runners since the shared ones can be slow or have restrictions on them.

For everyone's sake, if your tests take more than an hour to run or require a lot of computational power, please run your own private runner.

The .gitlab-ci.yml file

To start using Gitlab CI, you first have to create a .gitlab-ci.yml file.

Here's a simple example of how that might look like:

---
image: debian:stable

test12345:
  script:
  - echo 'deb http://deb.debian.org/debian buster main contrib non-free' >> /etc/apt/sources.list
  - apt-get update
  - apt-get install -y -t buster rolldice
...

The first part of the file is the image parameter. This is the docker image that will be used by default for your tests. This image is fetched from the Docker Hub. To use an image, you have to specify its full tag. Thus, the stable version of Debian is debian:stable.

The second part of the file is the test we are running. It is possible to have multiple tests one after another. The test in our example is called test12345 but we could have called it something else. apt:rolldice would also have been valid.

Each test must have a script. This script is a collection of shell commands ran by bash. The commands we want to run are listed one after another.

Here's an example with multiple tests:

---
image: debian:stable

apt:rolldice:stable:
  script:
  - apt-get install -y rolldice

apt:rolldice:buster:
  script:
  - echo 'deb http://deb.debian.org/debian buster main contrib non-free' >> /etc/apt/sources.list
  - apt-get update
  - apt-get install -y -t buster rolldice
...

It's also possible to change the default image in a test:

---
image: debian:stable

apt:rolldice:stable:
  script:
  - apt-get install -y rolldice

apt:rolldice:buster:
  image: debian:buster
  script:
  - apt-get install -y rolldice
...

All the files in your git repository are accessible by your scripts as if you were working directly in that directory:

> foobar.txt

Hello World

> .gitlab-ci.yml

---
image: debian:stable

test:hello:
  script:
  - cat foobar.txt
  - cat foobar.txt | grep "Hello World"
...

The service parameter

For tests that are more complex, you sometime need an external service to run in a separate docker container. The typical example is an application using a database.

With Gitlab CI, you can run such a service for all your tests directly in the CI. Those services are also Docker images. Here a simple example of a test using a database service:

> mytest.sql

CREATE TABLE `table1` (randomvar VARCHAR(30) NOT NULL);
CREATE TABLE `table2` (randomvar VARCHAR(30) NOT NULL);
CREATE TABLE `table3` (randomvar VARCHAR(30) NOT NULL);
CREATE TABLE `table4` (randomvar VARCHAR(30) NOT NULL);

> .gitlab-ci.yml

---
image: debian:stable

services:
  - mariadb:10.1

variables:
  MYSQL_DATABASE: mytest
  MYSQL_ALLOW_EMPTY_PASSWORD: 1

test:mariadb:
  script:
  - apt-get update && apt-get -y install mariadb-client-10.1
  - mariadb -u root -h mariadb -D mytest < mytest.sql
  - mariadb -u root -h mariadb -e "SELECT * FROM information_schema.columns WHERE table_schema = 'mytest'";
...

All Docker images can be used a service. If you have specific needs, you will need to create your own Docker image.

Artifacts

If you want to use Gitlab CI to compile code or to create files that you want to keep afterwards (compiling firmware is a great use case), you will need to use the artifact parameter. Its function is to specify what file or directory will be kept by Gitlab so you can download it afterwards.

> input.md

# Hello World

This is a markdown test. It should be rendered to HTML properly using the great
[pandoc](https://pandoc.org/MANUAL.html).

> .gitlab-ci.yml

---
image: debian:stable

build:html:
  script:
  - apt-get update && apt-get -y install pandoc
  - pandoc -o output.html input.md
  artifacts:
    paths:
    - output.html
...

Multiples tests in stages

By default, tests in Gitlab CI are all simultaneous. It's useful for unit tests, but for more elaborate tests it is often useful to work step by step and create dependencies between our tests.

The stage parameter can be used to group different tasks. These groups can then be ordered.

We can also use the dependencies parameter to create dependencies between tasks. The artifacts created in a task are automatically passed to the second task if it depends on it.

> hello.c

#include 

int
main (void)
{
  printf ("Hello, world!\n");
  return 0;
}

> .gitlab-ci.yml

---
stages:
  - build
  - test

compile:debian:
  image: debian:stable
  stage: build
  script:
  - apt-get update && apt-get install -y build-essential
  - gcc -Wall hello.c -o hello
  artifacts:
    paths:
    - hello

compile:alpine:
  image: alpine
  stage: build
  script:
  - apk add --update build-base
  - gcc -Wall hello.c -o hello
  artifacts:
    paths:
    - hello

test:debian:
  image: debian:stable
  stage: test
  dependencies:
  - compile:debian
  script:
  - ./hello

test:alpine:
  image: alpine
  stage: test
  dependencies:
  - compile:alpine
  script:
  - ./hello
...

Secret variables

If you want to use Gitlab CI to publish your projects on platforms like PyPi or NPM, you will need to stock authentication information somewhere.

You should obviously not add your passwords directly in the .gitlab-ci.yml file or in your git repository: anyone could stumble upon it and compromise your accounts.

Gitlab lets you stock secret variables. To add some, you have to go in the Settings > CI/CD > Variables tab.

Once you have added a secret variable, you can refer to it in your .gitlab-ci.yml file just like a normal variable.

---
image: debian:stable

test:secret-var:
  :script:
  - apt-get update && apt-get -y install wget
  - wget https://agendadulibre.qc.ca/events/$SECRET
...

Compile and host a static website with Gitlab CI and Gitlab Pages

An interesting use case for Gitlab CI is compiling a static website. Once compiled, this website can then be hosted directly on the Gitlab instance using Gitlab Pages. This can be very useful to host a project's documentation.

Here's an example using Gitlab CI and Gitlab Pages. This project uses Sphinx with the Read the Docs theme. The compiled website can be seen here.

Once you've added a .gitlab-ci.yml that compiles your static website, you need to go in the Settings > Pages tab in your git project to put your website online. Gitlab also gives the option of using an external domain if you wish to.

More info

The complete documentation for the .gitlab-ci.yml can be found online here.

Collaborative workshop

Find a project that you like and then try to create a few tests with Gitlab CI.

Installing Gitlab CI + Docker on a private machine (advanced)

We want to use Docker and Gitlab CI on a Debian Stretch server. These steps require a basic knowledge of the command line.

Docker

The Docker executor is the most simple and flexible one. It's also the one that the shared runners on gitlab.com use.

We thus start by installing Docker on our machine. Normally, you would install Docker directly from the Debian repository. Sadly, Docker is not packaged in Debian Stretch so we need to install it from Docker's own repository.

We start by adding a source file to apt:

> /etc/apt/sources.list.d/docker.sources

Types: deb
URIs: https://download.docker.com/linux/debian/
Suites: stretch
Architectures: amd64
Components: main
Signed-By: /usr/share/keyrings/docker-archive-keyring.gpg

We then complete the installation process by downloading Docker's GPG key and by installing packages¹:

$ apt install apt-transport-https
$ curl https://download.docker.com/linux/debian/gpg | gpg --dearmor > /usr/share/keyrings/docker-archive-keyring.gpg
$ apt update
$ apt install docker-ce

Docker has a bad bug and the networking doesn't work when you reboot the machine running it. This means our CI doesn't have networking and all the tests people run on it fail.

To hotfix this, we create a cron file that reboots Docker 30 seconds after our machine has booted:

$ sudo echo '@reboot root /bin/sleep 30 && /bin/systemctl restart docker' > /etc/cron.d/docker-reboot

We also want to make sure our server doesn't fill up with the docker images people are running for their tests. We thus add a second cron file to purge Docker periodically:

$ sudo echo '0 3 * * * root /usr/bin/docker system prune -a -f > /dev/null 2>&1' > /etc/cron.d/docker-prune

Et voilà! Docker is ready to be used in Gitlab CI!

Gitlab CI

Once again, instead of using the Debian repository to install Gitlab CI, we have to use Gitlab's repository. Gitlab is still pretty young in Debian and the runner has not been packaged for Debian Stretch yet.

We start by adding a source file to apt:

> /etc/apt/sources.list.d/gitlab.sources

Types: deb
URIs: https://packages.gitlab.com/runner/gitlab-runner/debian/
Suites: stretch
Architectures: amd64
Components: main
Signed-By: /usr/share/keyrings/gitlab-archive-keyring.gpg

We then complete the installation process by downloading Gitlab's GPG key and by installing packages:

$ curl https://packages.gitlab.com/gpg.key | gpg --dearmor > /usr/share/keyrings/gitlab-archive-keyring.gpg
$ apt update
$ apt install gitlab-runner

Once gitlab-runner has been installed, we can use the command line helper to create a new runner:

$ sudo gitlab-runner register
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
> https://gitlab-instance-url
Please enter the gitlab-ci token for this runner:
> your-project-token
Please enter the gitlab-ci description for this runner:
> Your runner's description
Please enter the gitlab-ci tags for this runner (comma separated):
> name-of-the-runner, docker

To get the token to create the runner, you need to go in the Settings > CI/CD > Runners tab in your git project. The token can be found in the Setup a specific Runner manually section.

If everything works correctly, your runner should show up in the Settings > CI/CD > Runners tab.

Pro Tip

If you are using your own Gitlab CI in a production environment, it is important to make sure it works at all times.

To make sure the runner works fine, you can create a git project on the Gitlab instance you are using. This project only has to contain a basic .gitlab-ci.yml file.

You can then run this test each day or each hour by creating a scheduled task in the CI/CD > Schedules tab of your project.

---
image: debian

test-ci:
  script:
  - apt-get update && apt-get -y install rolldice
  tags:
    - name-of-your-runner
...

It's important to specify the name of the runner in the tags section to force your test to run on your runner and not on a random one.