Overview

This container image is built daily from this Containerfile, and made available as:

• registry.gitlab.com/qontainers/pipglr:latest

-or-

• registry.gitlab.com/qontainers/pipglr:<version>

It's purpose is to provide an easy method to execute a GitLab runner, to service CI/CD jobs for groups and/or repositories on gitlab.com. It comes pre-configured to utilize the gitlab-runner app to execute within a rootless podman container, nested inside a rootless podman container.

This is intended to provide additional layers of security for the host, when running potentially arbitrary CI/CD code. Though, the ultimate responsibility still rests with the end-user to review the setup and configuration relative to their own security situation/environment.

Operation

This image leverages the podman runlabel feature heavily. Several labels are set on the image to support easy registration and execution of the runner container. While it's possible to use the container with your own command-line, it's highly recommended to base them off of one of the labels. See the examples below for more information.

Note: Some older versions of podman don't support the container runlabel sub-command. If this is the case, you may simulate it with the following, substituting <label> with one of the predefined values (i.e. register, setupconfig, etc.):

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ eval $(podman inspect --format=json $IMAGE | jq -r .[].Labels.<label>)

Persistent containers (step 1)

By default on many distributions, regular users aren't permitted to leave background processes running after they log out. Since this is likely desired for the user running the pipglr container, systemd needs to be configured to override this policy. For this, you ($USER) will need root access on the system.

$ sudo loginctl enable-linger $USER

Expanded user-namespace (step 2)

Some nested container images need to utilize high UID/GID values like 65535. However, many distributions set that as the default max number of user-namespace IDs. Since three IDs are needed inside the pipglr container, a larger range is required on the host. As root, edit the two files /etc/subuid and /etc/subgid to expand the range. For example assuming a local user called pipglr is used to run the container, the contents of these files should be edited to allocate 65539 IDs like:

pipglr:<some number>:65539

Where <some number> was set by your OS when the pipglr user was created. If you don't make this change, you're likely to experience errors starting containers for CI jobs, such as:

E: setgroups 65534 failed - setgroups (22: Invalid argument)

Runner registration (step 3)

All runners must be connected to a project or group runner configuration on your gitlab instance (or gitlab.com). This is done using a special registration runlabel. The command can (and probably should) be run more than once (using the same config.toml) to configure and register multiple runners. This is necessary for the pipglr container to execute multiple jobs in parallel. For example, if you want to support running four jobs at the same time, you would use the register runlabel four times.

Before using the register runlabel, you must set your unique registration (a.k.a. activation) token as a podman secret. This secret may be removed once the registration step is complete. The value (below) should be replaced with the value obtained from the "runners" settings page of a gitlab group or project's CI/CD Settings. Gitlab version 16 and later refers to this value as an activation token, but the usage is the same.

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ echo '<actual registration token>' | podman secret create REGISTRATION_TOKEN -

Next, a blank config.toml file needs to be created. Without this, the reigster runlabel will return a permission-denied error. Once the empty config.toml file is created, you may register one or more runners by repeating the registration runlabel as follows:

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ touch ./config.toml  # important: file must exist, even if empty.
$ podman container runlabel register $IMAGE
...repeat as desired...
$ podman secret rm REGISTRATION_TOKEN  # if desired

Runner Configuration (step 4)

During the registration process (above), a boiler-plate (default) config.toml file will be created/updated for you. At this point you may edit the configuration if desired before committing it as a podman secret. Please refer to the gitlab runner documentation for details.

$ $EDITOR ./config.toml  # if desired
$ podman secret create config.toml ./config.toml
$ rm ./config.toml  # if desired

This may be necessary, for example, to increase the default concurrency value to reflect the number of registered runners. If you need to edit this file after committing it as a secret, there's a dumpconfig runlabel for that.

Volume setup (step 5)

Since several users are utilized inside the container volumes must be specifically configured to permit access. This is done using several runlabels as follows:

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel setupstorage $IMAGE
$ podman container runlabel setupcache $IMAGE

Note: These volumes generally do not contain any critical operational data, they may be re-created anytime to quickly free up host disk-space if it's running low. Simply remove them with the command podman volume rm pipglr-storage pipglr-cache. Then reuse the setupstorage and setupcache runlabels as in the above example.

Runner Startup (step 6)

With the runner configuration saved as a Podman secret, and the runner volumes created, the GitLab runner container may be launched with the following commands:

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel run $IMAGE

Configuration Editing

The gitlab-runner configuration contains some sensitive values which should be protected. The pipglr container assumes the entire configuration will be passed in as a Podman secret. This makes editing it slightly convoluted, so a handy runlabel dumpconfig is available. It's intended use is as follows:

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel dumpconfig $IMAGE > ./config.toml
$ $EDITOR ./config.toml
$ podman secret rm config.toml
$ podman secret create config.toml ./config.toml
$ rm ./config.toml  # if desired

Debugging

The first thing to check is the container output. This shows three things: Systemd, Podman, and GitLab-Runner output. For example:

$ podman logs --since 0 pipglr

Next, try running a pipglr image built with more verbose logging. Both the runner.service and podman.service files have a log-level option. Simply increase one or both to the "info", or "debug" level. Start the debug container, and reproduce the problem.

Building

This image may be built simply with:

$ podman build -t registry.gitlab.com/qontainers/pipglr:latest .

This will utilize the latest stable version of podman and the latest stable version of the gitlab runner.

Build-args

Several build arguments are available to control the output image:

• PRUNE_INTERVAL - A systemd.timer compatible OnCalendar value that determines how often to prune Podman's storage of disused containers and images. Defaults to "daily", but should be adjusted based on desired caching-effect balanced against available storage space and job execution rate.
• RUNNER_VERSION - Allows specifying an exact gitlab runner version. By default the latest is used, assuming the user is building a tagged image anyway. Valid versions may be found on the runner release page.
• TARGETARCH - Supports inclusion of non-x86_64 gitlab runners. This value is assumed to match the image's architecture. If using the --platform build argument, it will be set automatically. Note: as of this writing, only amd64 and arm64 builds of the gitlab-runner are available.
• NESTED_PRIVILEGED - Defaults to 'true', may be set 'false' to prevent nested containers running in --privileged mode. This will affect the ability to build container images in CI jobs using tools like podman or buildah.

Environment variables

Nearly every option to every gitlab-runner sub-command may be specified via environment variable. Some of these are set in the Containerfile for the register runlabel. If you need to set additional runtime env. vars., please do so via additional Environment optionns in the runner.service file. See the systemd.nspawn man page for important value-format details.