Fix newuidmap...`: newuidmap: open of uid_map failed: Permission denied

Within a container, caps need to be set on /usr/bin/new{uid,gid}map
which reflect the (inner) podman user's root namespaced ID of 10000.

Signed-off-by: Chris Evich <chris_gitlab@icuc.me>

.gitlab-ci.yml

@@ -1,22 +1,39 @@
 ---
 
-stages:
-    - build
-
-build:
-  stage: build
+default:
+  image: quay.io/buildah/stable:v1.28.0
   tags:
     - docker
     - linux
-  image:
-    name: gcr.io/kaniko-project/executor:v1.6.0-debug
-    entrypoint: ["/busybox/sh", "-c"]
+
+envars:
+  stage: test
+  script: |
+    echo "Select CI env. vars.:";
+    printenv | egrep '^CI_' | sort
+
+commit_check:
+  stage: test
+  variables:
+    BADRX: '^(squash!)|(fixup!)'
+  script: |
+    dnf install -y git
+    shortlogtmp=$(mktemp -p '' commit_check_tmp_XXXX)
+    git log --oneline --no-show-signature "${CI_MERGE_REQUEST_DIFF_BASE_SHA}..HEAD" > "$shortlogtmp"
+    if egrep -q "$BADRX" "$shortlogtmp"; then
+        egrep "$BADRX" "$shortlogtmp"
+        die "Found the above commits matching '$BADRX'"
+    fi
+
+build:
+  stage: deploy
+  variables:
+    BUILDAH_FORMAT: docker
+    BUILDAH_ISOLATION: chroot
+    STORAGE_DRIVER: vfs
+  before_script:
+    - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
   script:
-    - 'mkdir -p /kaniko/.docker'
-    - 'echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json'
-    - |
-        echo "Select CI env. vars.:";
-        printenv | egrep '^CI_' | sort
     # N/B: There could be more than one merge-request open with this branch's HEAD
     - |
         IMAGE_TAG="${CI_COMMIT_BRANCH}";
@@ -28,12 +45,12 @@ build:
             IMAGE_TAG="latest";
         fi
         echo "Building/Pushing to: ${CI_REGISTRY_IMAGE}:${IMAGE_TAG}";
-    - |
-        /kaniko/executor \
-            --context $CI_PROJECT_DIR \
-            --dockerfile $CI_PROJECT_DIR/Containerfile \
-            --destination "${CI_REGISTRY_IMAGE}:${IMAGE_TAG}" \
-            --label "org.opencontainers.image.source=${CI_PROJECT_URL}" \
-            --label "org.opencontainers.image.revision=$CI_COMMIT_SHA" \
-            --label "org.opencontainers.image.created=$CI_JOB_STARTED_AT" \
-            --label "org.opencontainers.image.version=${IMAGE_TAG}"
+    - >-
+      buildah build \
+        --label "org.opencontainers.image.source=${CI_PROJECT_URL}" \
+        --label "org.opencontainers.image.revision=$CI_COMMIT_SHA" \
+        --label "org.opencontainers.image.created=$CI_JOB_STARTED_AT" \
+        --label "org.opencontainers.image.version=${IMAGE_TAG}" \
+        -t "${CI_REGISTRY_IMAGE}:${IMAGE_TAG}" "$CI_PROJECT_DIR"
+    - buildah images
+    - buildah push "${CI_REGISTRY_IMAGE}:${IMAGE_TAG}"

Containerfile

@@ -57,6 +57,7 @@ RUN for rpm in ${EXCLUDE_PACKAGES}; do x+="--exclude=$rpm "; done && \
     $DNFCMD update && \
     $DNFCMD install $x $RUNNER_RPM_URL && \
     $DNFCMD upgrade && \
+    $DNFCMD reinstall shadow-utils && \
     if [[ "${DNFCMD}" == "${_DNFCMD}" ]]; then \
       dnf clean all && \
       rm -rf /var/cache/dnf; \
@@ -97,13 +98,15 @@ RUN sed -i -r \
     chown -R podman:podman /home/podman && \
     chmod u+s /usr/bin/new{uid,gid}map && \
     rm -f /home/podman/.bash* && \
-    echo DOCKER_HOST="unix:///tmp/podman-run-1000/podman/podman.sock" > /etc/profile.d/podman.sh
+    echo DOCKER_HOST="unix:///tmp/podman-run-1000/podman/podman.sock" > /etc/profile.d/podman.sh && \
+    echo "podman:10000:10000" | tee /etc/subuid > /etc/subgid && \
+    setcap -n 10000 cap_setuid+ep /usr/bin/newuidmap && \
+    setcap -n 10000 cap_setuid+ep /usr/bin/newgidmap
 
 # Runtime rootless-mode configuration
 USER podman
-VOLUME ["/home/podman/.local/share/containers/storage/",\
-        "/home/podman/.gitlab-runner/", \
-        "/cache"]
+# N/B: Volumes are cumulative with the base image
+VOLUME ["/home/podman/.gitlab-runner/", "/cache"]
 WORKDIR /home/podman
 ENTRYPOINT ["/usr/local/bin/gitlab-runner-wrapper"]
 
@@ -142,10 +145,13 @@ ENV CLEAN_INTERVAL="$CLEAN_INTERVAL" \
     DOCKER_PRIVILEGED="$PRIVILEGED_RUNNER"
 
 # Not a real build-arg.  Simply here to save lots of typing.
-ARG _pm="--systemd=true --device=/dev/fuse --security-opt label=disable --user podman --volume pipglr-podman-root:/home/podman/.local/share/containers/storage --volume pipglr-config:/home/podman/.gitlab-runner -v pipglr-podman-cache:/cache -e PODMAN_RUNNER_DEBUG -e LOG_LEVEL"
+ARG _pm="--systemd=true --device=/dev/fuse --security-opt label=disable --user podman --volume pipglr-podman-root:/home/podman/.local/share/containers --volume pipglr-config:/home/podman/.gitlab-runner -v pipglr-podman-cache:/cache --tmpfs /var/lib/containers,ro,size=1k -e PODMAN_RUNNER_DEBUG -e LOG_LEVEL"
 
 # These labels simply make it easier to register and execute the runner.
 # Define them last so they are absent should a image-build failure occur.
 LABEL register="podman run -it --rm $_pm --secret REGISTRATION_TOKEN,type=env \$IMAGE register"
 # Note: Privileged mode is required to permit building container images with inner-podman
 LABEL run="podman run -d --privileged --name pipglr $_pm \$IMAGE run"
+
+# In case it's helpful, include the documentation
+ADD /README.md /home/podman/

README.md

@@ -27,27 +27,28 @@ lacks this feature, Several labels are set on the image to support
 easy registration and execution of a runner container using a special
 bash command.  See the examples below for more information.
 
-#### [Volume Ownership Bug](https://github.com/containers/podman/issues/16576)
+#### [Volume setup]
 
-Some versions of podman contain a bug where named local volumes aren't owned
-by the namespaced user within a rootless container (i.e. the 'podman' user).
-Since the `podman` user/group inside the `pipglr` container is known, it's
-possible to manually setup ownership ahead of time. This should be be done
-once, prior to registering your runners:
+Since podman inside the container runs as user `podman`, the volumes
+used by it need to be pre-created with ownership information.  While,
+we're at it, might as well add the performance-improving `noatime`,
+option as well.
 
 ```bash
-$ for VOLUME in pipglr-podman-root pipglr-config pipglr-podman-cache; do \
-    PUPVM="podman unshare podman volume mount $VOLUME"
-    podman volume create $VOLUME && \
-    podman unshare chown 1000:1000 $($PUPVM) && \
-    podman unshare chmod 02770 $($PUPVM) && \
-    podman unshare ls -land $($PUPVM) ; \
+$ VOLOPTS="o=uid=1000,gid=1000,noatime"; \
+  for VOLUME in pipglr-podman-root pipglr-config pipglr-podman-cache; do \
+    podman volume create --opt $VOLOPTS $VOLUME || true ; \
+    VOLPTH=$(podman unshare podman volume mount $VOLUME)
+    podman unshare chown -c -R 1000:1000 $VOLPTH && \
+    podman unshare chmod -c 02770 $VOLPTH && \
+    podman unshare podman volume unmount $VOLUME ; \
   done
 ```
 
 If you get `podman system service` startup permission-denied errors, or
 errors from gitlab-runner, unable to connect to the podman socket, this is
-likely the cause.  You can fix it after-the-fact using the same commands as above, just add a `-R` option to the `chown`/`chmod`, and additionally target `./*`.
+likely the cause.  You can fix it after-the-fact using the same commands
+above.
 
 #### Runner registration
 
@@ -107,7 +108,13 @@ sure to use the `podman unshare` command-wrapper to enter the usernamespace.
 For example, to display the config:
 
 ```bash
-$ podman unshare $(podman unshare podman volume mount pipglr-config)/config.toml
+$ podman unshare cat $(podman unshare podman volume mount pipglr-config)/config.toml
+```
+
+Edit the config with your favorite `$EDITOR`:
+
+```bash
+$ podman unshare $EDITOR $(podman unshare podman volume mount pipglr-config)/config.toml
 ```
 
 #### Debugging
@@ -118,9 +125,9 @@ The first thing to check is the container output:
 $ podman logs --since 0 pipglr
 ```
 
-Before starting the runner, you may `export PODMAN_RUNNER_DEBUG=debug` to enable
-debugging on the inner-podman.  Whereas `export LOG_LEVEL=debug` can be used to
-debug the gitlab-runner itself.
+Next, try running pipglr after an `export PODMAN_RUNNER_DEBUG=debug` to enable
+debugging on the inner-podman.  If more runner detail is needed, you can instead/additionally
+set `export LOG_LEVEL=debug` to debug the gitlab-runner itself.
 
 ## Building