pipglr/README.md

# Podmand-In-Podman Gitlab Runner

This project provides a Gitlab Runner which runs inside a container launched
with `podman`. The Gitlab Runner itself uses an independent `podman` instance
inside to launch jobs.

## Overview

This container image is built weekly from this `Containerfile`, and made
available as:

- `registry.gitlab.com/qontainers/pipglr:latest`

-or-

- `registry.gitlab.com/qontainers/pipglr:<version>`

It's purpose is to provide an easy method to execute a GitLab runner, to service
CI/CD jobs for groups and/or repositories on [gitlab.com](https://gitlab.com).
It comes pre-configured to utilize the gitlab-runner app to execute within a
rootless podman container, nested inside a rootless podman container.

This is intended to provide additional layers of security for the host, when
running potentially arbitrary CI/CD code. Though, the ultimate responsibility
still rests with the end-user to review the setup and configuration relative to
their own security situation/environment.

**Note**: While this can run entirely under a regular user, it will require root
access for the first two setup steps (below).

### Operation

This image leverages the podman `runlabel` feature heavily. Several labels are
set on the image to support easy registration and execution of the runner
container. While it's possible to use the container with your own command-line,
it's highly recommended to base them off of one of the labels. See the examples
below for more information.

*_Only podman versions 4.8.x and later are supported by pipglr._*

#### Persistent Containers (step 1)

By default on many distributions, regular users aren't permitted to leave
background processes running after they log out. Since this is likely desired
for running the pipglr container long-term, `systemd` needs to be configured to
override this policy.  Doing so requires root or `sudo` access on the
system.  Assuming the user `johndoe` will be executing the pipglr container,
linger may be enabled (as the admin user or root) with a command like:

```bash
$ sudo loginctl enable-linger `johndoe`
```

Side-effect: This will allow the user (`johndoe` for example) to persist
other user-level systemd services as well. For example `podman.socket` is
handy to enable for `podman remote` access. You could also
[setup quadlet](https://www.redhat.com/sysadmin/quadlet-podman) or a systemd
unit so pipglr starts up on system boot.

#### Expanded Host-system User-Namespace (step 2)

**_This is really important and frequently the cause of pipglr issues_**

On the host, as root, edit the two files `/etc/subuid` and `/etc/subgid` to
expand the pipglr user's ID allocation by 3.  For example if the host user
running the pipglr container is named `johndoe`, this entry in the files should
be edited like:

`johndoe:<some number>:65539`

Where `<some number>` was set by the OS when the `johndoe` user was created
(you can ignore this).  Only the last number needs to be increased by three.
This change will be effective on the user's next login, or immediately
by running (as the host pipglr user):

`podman system migrate`

_Note:_ This will stop any currently running containers.

_Details:_ As an added protection/safety measure, pipglr excludes three UID/GIDs
from being used by job-level (nested) containers. One for `root`, another for
`runner` and a third for `podman`.  This necessitates expanding the number of
available UIDs/GIDs from the host machine, to the pipglr container.  This way,
the full set of 65535 UIDs/GIDs may be utilized by job-level (nested)
sub-containers.

#### Runner Activation/Registration (step 3)

All runners must be connected to a project or group runner configuration on your
gitlab instance (or `gitlab.com`). This is done using a special registration
_runlabel_. The command can (and probably should) be run more than once (using
the same `config.toml`) to configure and register multiple runners. This is
necessary for the _pipglr_ container to execute multiple jobs in parallel. For
example, if you want to support running four jobs at the same time, you would
use the `register` _runlabel_ four times.

Before using the `register` _runlabel_, you must set your unique _registration_
(a.k.a. _activation_) token as a podman _secret_. This secret may be removed
once the registration step is complete. The **`<actual registration token>`**
value (below) should be replaced with the value obtained from the `runners`
settings page of a gitlab group or project's _CI/CD Settings_. Gitlab version 16
and later refers to this value as an _activation_ token, but the usage is the
same.

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ echo '<actual registration token>' | podman secret create REGISTRATION_TOKEN -
```

Next, **_a blank `config.toml` file_** needs to be created. Without this, the
`reigster` _runlabel_ will return a permission-denied error. Once the empty
`config.toml` file is created, you may register one or more runners by repeating
the registration _runlabel_ as follows:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ touch ./config.toml  # important: file must exist, even if empty.
$ podman container runlabel register $IMAGE
# ...repeat as desired...
$ podman secret rm REGISTRATION_TOKEN  # if desired
```

#### Runner Configuration (step 4)

During the registration process (above), a boiler-plate (default) `config.toml`
file will be created/updated for you. At this point you may edit the
configuration if desired before committing it as a _podman secret_. Please refer
to the
[gitlab runner documentation](https://docs.gitlab.com/runner/configuration/) for
details.

```bash
$ $EDITOR ./config.toml  # if desired
$ podman secret create config.toml ./config.toml
$ rm ./config.toml  # if desired
```

This may be necessary, for example, to increase the default `concurrency` value
to reflect the number of registered runners. If you need to edit this file after
committing it as a secret, there's
[a `dumpconfig` _runlabel_ for that](README.md#configuration-editing).

#### Volume Setup (step 5)

Since several users are utilized inside the container volumes must be
specifically configured to permit access. This is done using several _runlabels_
as follows:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel setupstorage $IMAGE
$ podman container runlabel setupcache $IMAGE
```

Note: These volumes generally do not contain any critical operational data, they
may be re-created anytime to quickly free up host disk-space if it's running
low. Simply remove them with the command
`podman volume rm pipglr-storage pipglr-cache`. Then reuse the `setupstorage`
and `setupcache` _runlabels_ as in the above example.

#### Runner Startup (step 6)

With the runner configuration saved as a Podman secret, and the runner volumes
created, the GitLab runner container may be launched with the following
commands:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel run $IMAGE
```

### Configuration Editing

The gitlab-runner configuration contains some sensitive values which should be
protected. The pipglr container assumes the entire configuration will be passed
in as a Podman secret. This makes editing it slightly convoluted, so a handy
_runlabel_ `dumpconfig` is available. It's intended use is as follows:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel dumpconfig $IMAGE > ./config.toml
$ $EDITOR ./config.toml
$ podman secret rm config.toml
$ podman secret create config.toml ./config.toml
$ rm ./config.toml  # if desired
```

### Debugging

The first thing to check is the container output. This shows three things:
Systemd, Podman, and GitLab-Runner output. For example:

```bash
$ podman logs --since 0 pipglr
```

Next, try running a pipglr image built with more verbose logging. Both the
`runner.service` and `podman.service` files have a `log-level` option. Simply
increase one or both to the `info`, or `debug` level. Start the debug container,
and reproduce the problem.

## Building

This image may be built simply with:

```bash
$ podman build -t registry.gitlab.com/qontainers/pipglr:latest .
```

This will utilize the latest stable version of podman and the latest stable
version of the gitlab runner.

### Build-Arguments

Several build arguments are available to control the output image:

- `BUILD_TYPE`: The build type, either `prod` or `dev`. In `dev` mode, the package
  manager is not deleted for development and debugging purposes. Please see
  [`build.sh`](scripts/build.sh) for more details.
- `PRUNE_INTERVAL`: A systemd.timer compatible `OnCalendar` value that
  determines how often to prune Podman's storage of disused containers and
  images. Defaults to `daily`, but should be adjusted based on desired
  caching-effect balanced against available storage space and job execution
  rate.
- `RUNNER_VERSION`: Allows specifying an exact gitlab runner version. By default
  the `latest` is used, assuming the user is building a tagged image anyway.
  Valid versions may be found on the
  [runner release page](https://gitlab.com/gitlab-org/gitlab-runner/-/releases).
- `TARGETARCH`: Supports inclusion of non-x86_64 gitlab runners. This value is
  assumed to match the image's architecture. If using the `--platform` build
  argument, it will be set automatically. Note: as of this writing, only `amd64`
  and `arm64` builds of the gitlab-runner are available.
- `GITLAB_URL`: Defaults to `https://gitlab.com/` but can be set to point to a
  self hosted instance of Gitlab.
- `NESTED_PRIVILEGED`: Defaults to `true`, may be set `false` to prevent nested
  containers running in `--privileged` mode. This will affect the ability to
  build container images in CI jobs using tools like podman or buildah.

Additional build-args are available as well.  See the `Containerfile` comments
for more details.

### Environment Variables

Nearly every option to every gitlab-runner sub-command may be specified via
environment variable. Some of these are set in the `Containerfile` for the
`register` _runlabel_. If you need to set additional runtime env. vars., please
do so via additional `Environment` optionns in the `runner.service` file. See
the _systemd.nspawn_ man page for important value-format details.