pipglr/README.md

## Overview

This container image is built daily from this `Containerfile`, and
made available as:

* `registry.gitlab.com/qontainers/pipglr:latest`

-or-

* `registry.gitlab.com/qontainers/pipglr:<version>`

It's purpose is to provide an easy method to execute a GitLab runner,
to service CI/CD jobs for groups and/or repositories on
[gitlab.com](https://gitlab.com).  It comes pre-configured to utilize
the gitlab-runner app to execute within a rootless podman container,
nested inside a rootless podman container.

This is intended to provide additional layers of security for the host,
when running potentially arbitrary CI/CD code.  Though, the ultimate
responsibility still rests with the end-user to review the setup and
configuration relative to their own security situation/environment.

**Note**: While this can run entirely under a regular user, it will require
root access for the first two setup steps (below).

### Operation

This image leverages the podman `runlabel` feature heavily.  Several
labels are set on the image to support easy registration and execution
of the runner container.  While it's possible to use the container
with your own command-line,  it's highly recommended to base them
off of one of the labels.  See the examples below for more information.

***Note:*** Some older versions of podman don't support the
`container runlabel` sub-command.  If this is the case, you may simulate
it with the following, substituting `<label>` with one of the predefined
values (i.e. `register`, `setupconfig`, etc.):

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ eval $(podman inspect --format=json $IMAGE | jq -r .[].Labels.<label>)
```

#### Persistent containers (step 1)

By default on many distributions, regular users aren't permitted to leave
background processes running after they log out.  Since this is likely
desired for running the pipglr container long-term, `systemd` needs to be
configured to override this policy.  For this, you (`$USER`) will need
root access on the system.

```bash
$ sudo loginctl enable-linger $USER
```

Side-effect: This will allow your user to persist other user-level systemd
services as well.  For example `podman.socket` is handy to enable for
`podman remote` access.  You could also [setup
quadlet](https://www.redhat.com/sysadmin/quadlet-podman) or a systemd unit
so pipglr starts up on system boot.

#### Expanded user-namespace (step 2) ***This is probably important***

As an added protection/safety measure, pipglr excludes three UID/GIDs
from being used by job-level containers.  One for `root`, another for
`runner` and a third for `podman`.  However, some container images
you may want to use for jobs (mainly Debian/Ubuntu), assign one/more
essential users a high UID/GID value (like `65535`).

At the same time, most distributions also set `65536` as the default maximum
number (including ID `0`) of IDs to allocate for user-namespaces (via `/etc/login.defs`).  This
creates a problem you won't realize until the runner actually picks up a job
😞 The main symptom of this issue will be messages in the pipglr containers log,
similar to (abbreviated):

```
...cut...
running `/usr/bin/newuidmap ...cut...`: newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
...cut...
```

or

```
E: setgroups 65534 failed - setgroups (22: Invalid argument)
```

***The good news is, working around this is relatively simple:***

As root, edit the two files `/etc/subuid` and `/etc/subgid` to expand the
by 3 IDs.  For example assuming a user running the pipglr container is
called `johndoe`, the contents of these files should be edited to allocate
`65539` IDs like:

`jogndoe:<some number>:65539`

Where `<some number>` was set by your OS when the `johndoe` user was created
(you can ignore this).  Only the last number needs to be increased.

#### Runner registration (step 3)

All runners must be connected to a project or group runner configuration
on your gitlab instance (or `gitlab.com`).  This is done using a special
registration *runlabel*.  The command can (and probably should) be run
more than once (using the same `config.toml`) to configure and register
multiple runners.  This is necessary for the *pipglr* container to execute
multiple jobs in parallel.  For example, if you want to support running
four jobs at the same time, you would use the `register` *runlabel*
four times.

Before using the `register`  *runlabel*, you must set your unique
*registration* (a.k.a. *activation*) token as a podman *secret*.  This
secret may be removed once the registration step is complete.  The
**<actual registration token>** value (below) should be replaced with
the value obtained from the "runners" settings page of a gitlab
group or project's *CI/CD Settings*.  Gitlab version 16 and later
refers to this value as an *activation* token, but the usage is the same.

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ echo '<actual registration token>' | podman secret create REGISTRATION_TOKEN -
```

Next, ***a blank `config.toml` file*** needs to be created.  Without this, the
`reigster` *runlabel* will return a permission-denied error.  Once the empty
`config.toml` file is created, you may register one or more runners by repeating
the registration *runlabel* as follows:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ touch ./config.toml  # important: file must exist, even if empty.
$ podman container runlabel register $IMAGE
...repeat as desired...
$ podman secret rm REGISTRATION_TOKEN  # if desired
```

#### Runner Configuration (step 4)

During the registration process (above), a boiler-plate (default) `config.toml` file
will be created/updated for you.  At this point you may edit the configuration
if desired before committing it as a *podman secret*.  Please refer to the
[gitlab runner documentation](https://docs.gitlab.com/runner/configuration/)
for details.

```bash
$ $EDITOR ./config.toml  # if desired
$ podman secret create config.toml ./config.toml
$ rm ./config.toml  # if desired
```

This may be necessary, for example, to increase the default `concurrency` value
to reflect the number of registered runners.  If you need to edit this file
after committing it as a secret, there's
[ a `dumpconfig`  *runlabel* for that](README.md#configuration-editing).

#### Volume setup (step 5)

Since several users are utilized inside the container volumes must be
specifically configured to permit access.  This is done using several
*runlabels* as follows:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel setupstorage $IMAGE
$ podman container runlabel setupcache $IMAGE
```

Note: These volumes generally do not contain any critical operational data,
they may be re-created anytime to quickly free up host disk-space if
it's running low.  Simply remove them with the command
 `podman volume rm pipglr-storage pipglr-cache`.  Then reuse the `setupstorage`
and `setupcache` *runlabels* as in the above example.

#### Runner Startup (step 6)

With the runner configuration saved as a Podman secret, and the runner volumes
created, the GitLab runner container may be launched with the following commands:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel run $IMAGE
```

### Configuration Editing

The gitlab-runner configuration contains some sensitive values which
should be protected.  The pipglr container assumes the entire configuration
will be passed in as a Podman secret.  This makes editing it slightly
convoluted, so a handy *runlabel* `dumpconfig` is available.
It's intended use is as follows:

```bash
$ IMAGE="registry.gitlab.com/qontainers/pipglr:latest"
$ podman container runlabel dumpconfig $IMAGE > ./config.toml
$ $EDITOR ./config.toml
$ podman secret rm config.toml
$ podman secret create config.toml ./config.toml
$ rm ./config.toml  # if desired
```

### Debugging

The first thing to check is the container output.  This shows three things:
Systemd, Podman, and GitLab-Runner output.  For example:

```bash
$ podman logs --since 0 pipglr
```

Next, try running a pipglr image built with more verbose logging.  Both
the `runner.service` and `podman.service` files have a `log-level` option.
Simply increase one or both to the "info", or "debug" level.  Start the
debug container, and reproduce the problem.

## Building

This image may be built simply with:

```bash
$ podman build -t registry.gitlab.com/qontainers/pipglr:latest .
```

This will utilize the latest stable version of podman and the latest
stable version of the gitlab runner.

### Build-args

Several build arguments are available to control the output image:

* `PRUNE_INTERVAL` - A systemd.timer compatible `OnCalendar` value that
  determines how often to prune Podman's storage of disused containers and
  images.  Defaults to "daily", but should be adjusted based on desired
  caching-effect balanced against available storage space and job
  execution rate.
* `RUNNER_VERSION` - Allows specifying an exact gitlab runner version.
  By default the `latest` is used, assuming the user is building a tagged
  image anyway.  Valid versions may be found on the [runner
  release page](https://gitlab.com/gitlab-org/gitlab-runner/-/releases).
* `TARGETARCH` - Supports inclusion of non-x86_64 gitlab runners.  This
  value is assumed to match the image's architecture.  If using the
  `--platform` build argument, it will be set automatically.  Note:
  as of this writing, only `amd64` and `arm64` builds of the gitlab-runner
  are available.
* `NESTED_PRIVILEGED` - Defaults to 'true', may be set 'false' to prevent
  nested containers running in `--privileged` mode.  This will affect
  the ability to build container images in CI jobs using tools like
  podman or buildah.

### Environment variables

Nearly every option to every gitlab-runner sub-command may be specified via
environment variable.  Some of these are set in the `Containerfile` for
the `register` *runlabel*.  If you need to set additional runtime
env. vars., please do so via additional `Environment` optionns in the
`runner.service` file.  See the *systemd.nspawn* man page for important
value-format details.