At the DebConf videoteam, we use ansible to manage our machines. Last fall in
Cambridge, we migrated our repositories on
salsa.debian.org and I started
playing with the Gitlab CI. It's pretty powerful and helped us catch a bunch of
errors we had missed.
As it was my first time playing with continuous integration and docker, I had
trouble when our playbooks used
systemd in a way or another and I couldn't
figure out a way to have
systemd run in the Gitlab docker runner.
Fast forward a few months and I lost another day and a half working on this issue. I haven't been able to make it work (my conclusion is that it's not currently possible), but I thought I would share what I learned in the process with others. Who knows, maybe someone will have a solution!
10 steps to failure
I first stated by creating a privileged Gitlab docker runner on a machine that
is dedicated to running Gitlab CI runners. To run
systemd in docker you either
need to run privileged docker instances or to run them with the
If you were trying to run a docker container that runs with
you would do something like:
$ docker run -it --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:ro debian-systemd
I tried replicating this behavior with the Gitlab runner by mounting the right
volumes in the runner and giving it the right
The thing is, normally your docker container runs a entrypoint command such as
CMD ["/lib/systemd/systemd"]. To run its CI scripts, the Gitlab runner takes
that container but replaces the entrypoint command by:
sh -c 'if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelse\n\techo shell not found\n\texit 1\nfi\n\n'
That is to say, it tries to run
If you try to run commands that require
systemd such as
you'll end up with this error message since
systemd is not running:
Failed to get D-Bus connection: Operation not permitted
Trying to run
systemd manually once the container has been started won't work
systemd needs to be PID 1 in order to work (and PID 1 is
bash). You end up with this error:
Trying to run as user instance, but the system has not been booted with systemd.
At this point, I came up with a bunch of creative solutions to try to bypass Gitlab's entrypoint takeover. Turns out you can tell the Gitlab runner to override the container's entrypoint with your own. Sadly, the runner then appends its long bash command right after.
For example, if you run a job with this gitlab-ci entry:
image: name: debian-systemd entrypoint: "/lib/systemd/systemd" script: - /usr/local/bin/my-super-script
You will get this entrypoint:
/lib/systemd/systemd sh -c 'if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelse\n\techo shell not found\n\texit 1\nfi\n\n'
This obviously fails. I then tried to be clever and use this entrypoint:
["/lib/systemd/systemd", "&&"]. This does not work either, since docker
requires the entrypoint to be only one command.
Someone pointed out to me that you could try to use
to PID 1
systemd, but that also fails with an error telling you
the system has not been booted with
One more level down
Since it seems you can't run
systemd in the Gitlab docker runner directly,
why not try to run
systemd in docker in docker (dind)? dind is used quite a
lot in the Gitlab CI to build containers, so we thought it might work.
Sadly, we haven't been able to make this work either. You need to mount volumes
in docker to run
systemd properly and it seems docker doesn't like to mount
volumes from a docker container that already have been mounted from the docker
If you have been able to run
systemd in the Gitlab docker runner, please
Paths to explore
The only Gitlab runner executor I've used at the moment is the docker one, since
it's what most Gitlab instances run. I have no experience with it, but since
there is also an LXC executor, it might be possible to run Gitlab CI tests with
systemd this way.