267 lines
10 KiB
Markdown
267 lines
10 KiB
Markdown
# Podmand-In-Podman Gitlab Runner
|
|
|
|
This project provides a Gitlab Runner which runs inside a container launched
|
|
with `podman`. The Gitlab Runner itself uses an independent `podman` instance
|
|
inside to launch jobs.
|
|
|
|
## Overview
|
|
|
|
This container image is built daily from this `Containerfile`, and made
|
|
available as:
|
|
|
|
- `registry.gitlab.com/qontainers/pipglr:latest`
|
|
|
|
-or-
|
|
|
|
- `registry.gitlab.com/qontainers/pipglr:<version>`
|
|
|
|
It's purpose is to provide an easy method to execute a GitLab runner, to service
|
|
CI/CD jobs for groups and/or repositories on [gitlab.com](https://gitlab.com).
|
|
It comes pre-configured to utilize the gitlab-runner app to execute within a
|
|
rootless podman container, nested inside a rootless podman container.
|
|
|
|
This is intended to provide additional layers of security for the host, when
|
|
running potentially arbitrary CI/CD code. Though, the ultimate responsibility
|
|
still rests with the end-user to review the setup and configuration relative to
|
|
their own security situation/environment.
|
|
|
|
**Note**: While this can run entirely under a regular user, it will require root
|
|
access for the first two setup steps (below).
|
|
|
|
### Operation
|
|
|
|
This image leverages the podman `runlabel` feature heavily. Several labels are
|
|
set on the image to support easy registration and execution of the runner
|
|
container. While it's possible to use the container with your own command-line,
|
|
it's highly recommended to base them off of one of the labels. See the examples
|
|
below for more information.
|
|
|
|
**_Note:_** Some older versions of podman don't support the `container runlabel`
|
|
sub-command. If this is the case, you may simulate it with the following,
|
|
substituting `<label>` with one of the predefined values (i.e. `register`,
|
|
`setupconfig`, etc.):
|
|
|
|
```bash
|
|
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
|
|
$ eval $(podman inspect --format=json $IMAGE | jq -r .[].Labels.<label>)
|
|
```
|
|
|
|
#### Persistent Containers (step 1)
|
|
|
|
By default on many distributions, regular users aren't permitted to leave
|
|
background processes running after they log out. Since this is likely desired
|
|
for running the pipglr container long-term, `systemd` needs to be configured to
|
|
override this policy. For this, you (`$USER`) will need root access on the
|
|
system.
|
|
|
|
```bash
|
|
$ sudo loginctl enable-linger $USER
|
|
```
|
|
|
|
Side-effect: This will allow your user to persist other user-level systemd
|
|
services as well. For example `podman.socket` is handy to enable for
|
|
`podman remote` access. You could also
|
|
[setup quadlet](https://www.redhat.com/sysadmin/quadlet-podman) or a systemd
|
|
unit so pipglr starts up on system boot.
|
|
|
|
#### Expanded User-Namespace (step 2) **_This is probably important_**
|
|
|
|
As an added protection/safety measure, pipglr excludes three UID/GIDs from being
|
|
used by job-level containers. One for `root`, another for `runner` and a third
|
|
for `podman`. However, some container images you may want to use for jobs
|
|
(mainly Debian/Ubuntu), assign one/more essential users a high UID/GID value
|
|
(like `65535`).
|
|
|
|
At the same time, most distributions also set `65536` as the default maximum
|
|
number (including ID `0`) of IDs to allocate for user-namespaces (via
|
|
`/etc/login.defs`). This creates a problem you won't realize until the runner
|
|
actually picks up a job 😞 The main symptom of this issue will be messages in
|
|
the pipglr containers log, similar to (abbreviated):
|
|
|
|
```text
|
|
...cut...
|
|
running `/usr/bin/newuidmap ...cut...`: newuidmap: write to uid_map failed: Operation not permitted
|
|
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
|
|
...cut...
|
|
```
|
|
|
|
or
|
|
|
|
```text
|
|
E: setgroups 65534 failed - setgroups (22: Invalid argument)
|
|
```
|
|
|
|
**_The good news is, working around this is relatively simple:_**
|
|
|
|
As root, edit the two files `/etc/subuid` and `/etc/subgid` to expand the by 3
|
|
IDs. For example assuming a user running the pipglr container is called
|
|
`johndoe`, the contents of these files should be edited to allocate `65539` IDs
|
|
like:
|
|
|
|
`johndoe:<some number>:65539`
|
|
|
|
Where `<some number>` was set by your OS when the `johndoe` user was created
|
|
(you can ignore this). Only the last number needs to be increased. This change
|
|
will be effective on next login, or immediately by running:
|
|
|
|
`podman system migrate`
|
|
|
|
_Note:_ This will stop any currently running containers.
|
|
|
|
#### Runner Registration (step 3)
|
|
|
|
All runners must be connected to a project or group runner configuration on your
|
|
gitlab instance (or `gitlab.com`). This is done using a special registration
|
|
_runlabel_. The command can (and probably should) be run more than once (using
|
|
the same `config.toml`) to configure and register multiple runners. This is
|
|
necessary for the _pipglr_ container to execute multiple jobs in parallel. For
|
|
example, if you want to support running four jobs at the same time, you would
|
|
use the `register` _runlabel_ four times.
|
|
|
|
Before using the `register` _runlabel_, you must set your unique _registration_
|
|
(a.k.a. _activation_) token as a podman _secret_. This secret may be removed
|
|
once the registration step is complete. The **`<actual registration token>`**
|
|
value (below) should be replaced with the value obtained from the `runners`
|
|
settings page of a gitlab group or project's _CI/CD Settings_. Gitlab version 16
|
|
and later refers to this value as an _activation_ token, but the usage is the
|
|
same.
|
|
|
|
```bash
|
|
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
|
|
$ echo '<actual registration token>' | podman secret create REGISTRATION_TOKEN -
|
|
```
|
|
|
|
Next, **_a blank `config.toml` file_** needs to be created. Without this, the
|
|
`reigster` _runlabel_ will return a permission-denied error. Once the empty
|
|
`config.toml` file is created, you may register one or more runners by repeating
|
|
the registration _runlabel_ as follows:
|
|
|
|
```bash
|
|
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
|
|
$ touch ./config.toml # important: file must exist, even if empty.
|
|
$ podman container runlabel register $IMAGE
|
|
# ...repeat as desired...
|
|
$ podman secret rm REGISTRATION_TOKEN # if desired
|
|
```
|
|
|
|
#### Runner Configuration (step 4)
|
|
|
|
During the registration process (above), a boiler-plate (default) `config.toml`
|
|
file will be created/updated for you. At this point you may edit the
|
|
configuration if desired before committing it as a _podman secret_. Please refer
|
|
to the
|
|
[gitlab runner documentation](https://docs.gitlab.com/runner/configuration/) for
|
|
details.
|
|
|
|
```bash
|
|
$ $EDITOR ./config.toml # if desired
|
|
$ podman secret create config.toml ./config.toml
|
|
$ rm ./config.toml # if desired
|
|
```
|
|
|
|
This may be necessary, for example, to increase the default `concurrency` value
|
|
to reflect the number of registered runners. If you need to edit this file after
|
|
committing it as a secret, there's
|
|
[a `dumpconfig` _runlabel_ for that](README.md#configuration-editing).
|
|
|
|
#### Volume Setup (step 5)
|
|
|
|
Since several users are utilized inside the container volumes must be
|
|
specifically configured to permit access. This is done using several _runlabels_
|
|
as follows:
|
|
|
|
```bash
|
|
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
|
|
$ podman container runlabel setupstorage $IMAGE
|
|
$ podman container runlabel setupcache $IMAGE
|
|
```
|
|
|
|
Note: These volumes generally do not contain any critical operational data, they
|
|
may be re-created anytime to quickly free up host disk-space if it's running
|
|
low. Simply remove them with the command
|
|
`podman volume rm pipglr-storage pipglr-cache`. Then reuse the `setupstorage`
|
|
and `setupcache` _runlabels_ as in the above example.
|
|
|
|
#### Runner Startup (step 6)
|
|
|
|
With the runner configuration saved as a Podman secret, and the runner volumes
|
|
created, the GitLab runner container may be launched with the following
|
|
commands:
|
|
|
|
```bash
|
|
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
|
|
$ podman container runlabel run $IMAGE
|
|
```
|
|
|
|
### Configuration Editing
|
|
|
|
The gitlab-runner configuration contains some sensitive values which should be
|
|
protected. The pipglr container assumes the entire configuration will be passed
|
|
in as a Podman secret. This makes editing it slightly convoluted, so a handy
|
|
_runlabel_ `dumpconfig` is available. It's intended use is as follows:
|
|
|
|
```bash
|
|
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
|
|
$ podman container runlabel dumpconfig $IMAGE > ./config.toml
|
|
$ $EDITOR ./config.toml
|
|
$ podman secret rm config.toml
|
|
$ podman secret create config.toml ./config.toml
|
|
$ rm ./config.toml # if desired
|
|
```
|
|
|
|
### Debugging
|
|
|
|
The first thing to check is the container output. This shows three things:
|
|
Systemd, Podman, and GitLab-Runner output. For example:
|
|
|
|
```bash
|
|
$ podman logs --since 0 pipglr
|
|
```
|
|
|
|
Next, try running a pipglr image built with more verbose logging. Both the
|
|
`runner.service` and `podman.service` files have a `log-level` option. Simply
|
|
increase one or both to the `info`, or `debug` level. Start the debug container,
|
|
and reproduce the problem.
|
|
|
|
## Building
|
|
|
|
This image may be built simply with:
|
|
|
|
```bash
|
|
$ podman build -t registry.gitlab.com/qontainers/pipglr:latest .
|
|
```
|
|
|
|
This will utilize the latest stable version of podman and the latest stable
|
|
version of the gitlab runner.
|
|
|
|
### Build-Arguments
|
|
|
|
Several build arguments are available to control the output image:
|
|
|
|
- `PRUNE_INTERVAL`: A systemd.timer compatible `OnCalendar` value that
|
|
determines how often to prune Podman's storage of disused containers and
|
|
images. Defaults to `daily`, but should be adjusted based on desired
|
|
caching-effect balanced against available storage space and job execution
|
|
rate.
|
|
- `RUNNER_VERSION`: Allows specifying an exact gitlab runner version. By default
|
|
the `latest` is used, assuming the user is building a tagged image anyway.
|
|
Valid versions may be found on the
|
|
[runner release page](https://gitlab.com/gitlab-org/gitlab-runner/-/releases).
|
|
- `TARGETARCH`: Supports inclusion of non-x86_64 gitlab runners. This value is
|
|
assumed to match the image's architecture. If using the `--platform` build
|
|
argument, it will be set automatically. Note: as of this writing, only `amd64`
|
|
and `arm64` builds of the gitlab-runner are available.
|
|
- `GITLAB_URL`: Defaults to `https://gitlab.com/` but can be set to point to a
|
|
self hosted instance of Gitlab.
|
|
- `NESTED_PRIVILEGED`: Defaults to `true`, may be set `false` to prevent nested
|
|
containers running in `--privileged` mode. This will affect the ability to
|
|
build container images in CI jobs using tools like podman or buildah.
|
|
|
|
### Environment Variables
|
|
|
|
Nearly every option to every gitlab-runner sub-command may be specified via
|
|
environment variable. Some of these are set in the `Containerfile` for the
|
|
`register` _runlabel_. If you need to set additional runtime env. vars., please
|
|
do so via additional `Environment` optionns in the `runner.service` file. See
|
|
the _systemd.nspawn_ man page for important value-format details.
|