Overview

This container image is built daily from this Containerfile, and made available as:

• registry.gitlab.com/qontainers/pipglr:latest

-or-

• registry.gitlab.com/qontainers/pipglr:<version>

It's purpose is to provide an easy method to execute a GitLab runner, to service CI/CD jobs for groups and/or repositories on gitlab.com. It comes pre-configured to utilize the gitlab-runner app to execute within a rootless podman container, nested inside a rootless podman container.

This is intended to provide additional layers of security for the host, when running potentially arbitrary CI/CD code. Though, the ultimate responsibility still rests with the end-user to review the setup and configuration relative to their own security situation/environment.

Operation

This image supports podman container runlabel, or if your version lacks this feature, Several labels are set on the image to support easy registration and execution of a runner container using a special bash command. See the examples below for more information.

Volume Ownership Bug

Some versions of podman contain a bug where named local volumes aren't owned by the namespaced user within a rootless container (i.e. the 'podman' user). Since the podman user/group inside the pipglr container is known, it's possible to manually setup ownership ahead of time. This should be be done once, prior to registering your runners:

$ for VOLUME in pipglr-podman-root pipglr-config pipglr-podman-cache; do \
    PUPVM="podman unshare podman volume mount $VOLUME"
    podman volume create $VOLUME && \
    podman unshare chown 1000:1000 $($PUPVM) && \
    podman unshare chmod 02770 $($PUPVM) && \
    podman unshare ls -land $($PUPVM) ; \
  done

If you get podman system service startup permission-denied errors, or errors from gitlab-runner, unable to connect to the podman socket, this is likely the cause. You can fix it after-the-fact using the same commands as above, just add a -R option to the chown/chmod, and additionally target ./*.

Runner registration

Each time the registration command is run, a new runner is added into the configuration. If however, you simply need to update/modify the configuration, please edit the config.toml file directly after mounting (default) pipglr-runner-config (/home/podman/.gitlab-runner/) volume. For modern versions of podman, registration can be performed with the following commands:

$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ echo '<actual registration token>' | podman secret create REGISTRATION_TOKEN -
$ podman container runlabel register $IMAGE

Where <actual registration token> is the value obtained from the "runners" settings page of a gitlab group or project. When you're finished registering as many runners as you want, the secret is no-longer needed and may be removed:

$ podman secret rm REGISTRATION_TOKEN

Note

Some versions of podman don't support the container runlabel sub-command. If this is the case, you may simulate it with the following command (in addition to the other example commands above):

$ eval $(podman inspect --format=json $IMAGE | jq -r .[].Labels.register)

Runner Startup

With one or more runners successfully registered and configured, the GitLab runner container may be launched with the following commands:

$ podman container runlabel run $IMAGE

Note

As above, if you're missing the container runlabel sub-command, the following may be used instead (assuming $IMAGE remains set):

$ eval $(podman inspect --format=json $IMAGE | jq -r .[].Labels.run)

Runner configuration

You may inspect/modify the gitlab-runner configuration as you see fit, just be sure to use the podman unshare command-wrapper to enter the usernamespace. For example, to display the config:

$ podman unshare $(podman unshare podman volume mount pipglr-config)/config.toml

Debugging

The first thing to check is the container output:

$ podman logs --since 0 pipglr

Before starting the runner, you may export PODMAN_RUNNER_DEBUG=debug to enable debugging on the inner-podman. Whereas export LOG_LEVEL=debug can be used to debug the gitlab-runner itself.

Building

This image may be built simply with:

$ podman build -t registry.gitlab.com/qontainers/pipglr:latest .

This will utilize the latest stable version of podman and the latest stable version of the gitlab runner.

Notes

• If you wish to use the testing or upstream flavors of the podman base image, simply build with --build-arg FLAVOR=testing (or upstream).

• Additionally or alternatively, you may specify a specific podman base image tag with --build-arg BASE_TAG=<value>. Where <value> is either latest, the podman image version (e.g. v4, v4.2, v4.2.0, etc.)

Build-args

Several build arguments are available to control the output image:

• FLAVOR - Choose from 'stable', 'testing', or 'upstream'. These select the podman base-image to utilize - which may affect the podman version, features, and stability. For more information see the podmanimage README.
• BASE_TAG - When FLAVOR="stable", allows granular choice over the exact podman version. Possible values include, latest, vX, vX.Y, and vX.Y.Z (where, X, Y, and Z represent the podman semantic version numbers). It's also possible to specify an image SHA.
• CLEAN_INTERVAL - A sleep (command) compatible time-argument that determines how often to clean out podman storage of disused containers and images. Defaults to 24-hours, but should be adjusted based on desired caching-effect versus available storage space and rate of job execution.
• EXCLUDE_PACKAGES - A space-separated list of RPM packages to prevent their existence in the final image. This is intended as a security measure to limit the attack-surface should a gitlab-runner process escape it's inner-container.
• RUNNER_VERSION - Allows specifying an exact gitlab runner version. By default the latest is used, assuming the user is building a tagged image anyway. Valid versions may be found on the runner release page.
• DNFCMD - By default this is set to dnf --setopt=tsflags=nodocs -y. However, if you'd like to volume-mount in /var/cache/dnf then you'll need to use --build-arg DNFCMD="dnf --setopt=tsflags=nodocs -y --setopt keepcache=true" Note: Changing DNFCMD will cause build-time cache cleanup to be disabled.
• TARGETARCH - Supports inclusion of non-x86_64 gitlab runners. This value is assumed to match the image's architecture. If using the --platform build argument, it will be set automatically.
• RUNNER_LISTEN_ADDRESS - Disabled by default, setting this to the FQDN and port supports various observability and debugging features of the gitlab runner. For more information see the gitlab runner advanced configuration documentation.
• PRIVILEGED_RUNNER - Defaults to 'true', may be set 'true' if you're brave. However this may result in the gitlab-runner failing to launch inner-containers. Setting it false will also prevent building container images using the runner.
• RUNNER_TAGS - Defaults to podman_in_podman, may be set to any comma-separated list (with no spaces!) of tags. These show up in GitLab (not the runner configuration), and determines where jobs are run.
• RUNNER_UNTAGED - Defaults to true, may be set to false. Allows the runner to service jobs without any tags on them at all.

Environment variables

Nearly every option to every gitlab-runner sub-command may be specified via environment variable. Many important/required options are set in the Containerfile. However it's entirely possible to pass them in via either of the podman container runlabel... container commands. To discover them, simply append --help to the end of the command. For example:

podman container runlabel $IMAGE register --help