Isolated Image Builder¶

Q: Install the worker profile for Isolated Image Builder?

The effective GitHub labels include linux, the host architecture, image-builder, and oci-builder. The profile permits one active worker across all targets because cleanup applies to the entire BuildKit daemon.

Q: Materialize job credentials for Isolated Image Builder?

Create a job-private directory containing ca.pem, cert.pem, and key.pem, then set:

Q: Candidate evidence for Isolated Image Builder?

--candidate-output writes one atomic image-candidate.schema.json document after the authoritative post-build cleanup completes. The report contains only bounded build, digest, output, cleanup, and optional source provenance evidence.

The image-builder profile builds OCI images without giving workflow code the PitCrew host's Docker socket. Disposable workers contain buildctl, crane, and pitcrew-build-image; a rootless BuildKit service performs build execution on the same Docker host.

The built-in profile pins crane 0.21.9 and verifies its exact upstream version output, 0.21.9, before a worker image can replace the active revision. The worker image also normalizes the build helper to LF and verifies that it contains no carriage returns and parses as valid Bash before replacement.

This lane supports Dockerfile build verification and image publication. It does not provide Docker Compose, Testcontainers, service containers, or a generic Docker API.

Architecture¶

GitHub job
  -> disposable image-builder worker
  -> mTLS on pitcrew-image-builder network
  -> rootless BuildKit service
  -> optional OCI registry

The service runs as UID 1000 and receives no Docker socket or host port. Dockerfile build processes run inside the BuildKit container and see only the context streamed by buildctl; they cannot read the Actions runner filesystem or registration state.

Host prerequisites¶

Rootless BuildKit requires unprivileged user namespaces. On Ubuntu 24.04 or later, the host may require:

kernel.apparmor_restrict_unprivileged_userns=0

The service uses exactly:

seccomp=unconfined
apparmor=unconfined
systempaths=unconfined

It never uses --privileged or --oci-worker-no-process-sandbox. If these rootless settings cannot start successfully, qualification fails; do not substitute privileged BuildKit on a shared PitCrew node.

The shipped Compose contract also defaults the persistent service to:

memory: 8 GiB
memory plus swap: 8 GiB
CPU: 4 cores
PIDs: 4096

Override these through PITCREW_IMAGE_BUILDER_MEMORY_LIMIT, PITCREW_IMAGE_BUILDER_MEMORY_SWAP_LIMIT, PITCREW_IMAGE_BUILDER_CPU_LIMIT, and PITCREW_IMAGE_BUILDER_PIDS_LIMIT before running service setup. Setup rejects an effective container without all four ceilings. Include the chosen service limits when calibrating the image-builder profile's host-admission worker cost.

Generate service and client certificates¶

Run from the PitCrew checkout:

$certificates = .\services\image-builder\New-PitCrewBuildKitCertificates.ps1 `
    -OutputDirectory <operator-owned-certificate-directory>

The script creates:

authority/ca.pem
authority/ca-key.pem
server/ca.pem
server/server-cert.pem
server/server-key.pem
client/ca.pem
client/cert.pem
client/key.pem

It prints paths, expiry, and certificate fingerprints but never private material. Keep authority and server outside repositories and GitHub. Store only the three client files as protected, job-scoped repository secrets.

Start the rootless service¶

.\services\image-builder\Setup-PitCrewImageBuilderService.ps1 `
    -ServerCertificateDirectory $certificates.serverDirectory

The setup script:

creates or validates pitcrew-image-builder;
creates the exact state volume and an immutable certificate volume keyed by the server certificate fingerprint;
imports only server material into the certificate volume;
starts the pinned rootless BuildKit Compose service;
verifies that it is healthy, non-privileged, socketless, and attached only to the intended network; and
preserves service state and certificates when stopped.

The service remains running while Actions workers scale to zero. Certificate rotation creates a new volume and force-recreates the service. A failed replacement restores the previously recorded certificate volume before returning an error.

Stop it only after the image-builder profile has no active workers:

.\services\image-builder\Setup-PitCrewImageBuilderService.ps1 -Down

Install the worker profile¶

.\Setup-Runner.ps1 `
    -Profile image-builder `
    -Repos https://github.com/example/project=1

The effective GitHub labels include linux, the host architecture, image-builder, and oci-builder. The profile permits one active worker across all targets because cleanup applies to the entire BuildKit daemon.

An operator may add a repository-specific alias while preserving built-in labels:

.\Setup-Runner.ps1 `
    -Profile image-builder `
    -Labels 'oci-builder,project-image-builder' `
    -Repos https://github.com/example/project=1

Materialize job credentials¶

Create a job-private directory containing ca.pem, cert.pem, and key.pem, then set:

export BUILDKIT_HOST=tcp://buildkitd:1234
export BUILDKIT_TLS_DIR="$RUNNER_TEMP/buildkit-tls"
export DOCKER_CONFIG="$RUNNER_TEMP/docker-config"

Use an if: always() step to delete both directories. Never upload them as artifacts, print them, or include them in profile state.

Verify a pull request without publishing¶

pitcrew-build-image \
  --image-ref ghcr.io/example/project:candidate \
  --context . \
  --dockerfile . \
  --platform linux/amd64 \
  --build-arg SDK_VERSION=1.2.3 \
  --output-oci "$RUNNER_TEMP/project-verification.tar" \
  --candidate-output "$RUNNER_TEMP/image-candidate.json" \
  --recipe-id application-ci \
  --source-repository example/project \
  --source-commit "$GITHUB_SHA" \
  --workflow-run-id "$GITHUB_RUN_ID"

This builds the Dockerfile, writes an OCI tarball, verifies the digest and manifest blob declared by its OCI index, and creates no registry tag. Put toolchain assertions inside the Dockerfile so the build itself proves the resulting image contract.

Publish and verify an immutable image¶

After materializing job-scoped registry authentication:

immutable_ref="$(
  pitcrew-build-image \
    --image-ref ghcr.io/example/project:sha-"$GITHUB_SHA" \
    --context . \
    --dockerfile . \
    --platform linux/amd64 \
    --build-arg SDK_VERSION=1.2.3 \
    --label org.opencontainers.image.revision="$GITHUB_SHA" \
    --push \
    --verify-registry \
    --candidate-output "$RUNNER_TEMP/image-candidate.json" \
    --recipe-id application-ci \
    --source-repository example/project \
    --source-commit "$GITHUB_SHA" \
    --workflow-run-id "$GITHUB_RUN_ID"
)"

printf 'Published %s\n' "$immutable_ref"

The helper compares BuildKit metadata with the registry digest returned by pinned crane. A mismatch fails the job.

Candidate evidence¶

--candidate-output writes one atomic image-candidate.schema.json document after the authoritative post-build cleanup completes. The report contains only bounded build, digest, output, cleanup, and optional source provenance evidence.

Published candidates require --verify-registry; a mutable tag without independent registry digest confirmation cannot become ready. OCI verification candidates carry the verified digest but no immutable registry reference.

The report is written with owner-only permissions and must be outside the reviewed build context. Keep workflow transcripts, candidate reports, OCI output, TLS material, and Docker configuration under RUNNER_TEMP; adding generated output to the build context changes the reviewed inputs and is rejected.

A failed build writes a failed candidate report when the candidate output and recipe identity were already validated. Failure categories and details use a closed non-secret vocabulary. Raw build logs remain in the owning workflow and are never embedded in candidate evidence.

Validate an existing report before consuming or publishing it:

$candidate = ./scripts/Test-PitCrewImageCandidate.ps1 `
    -Path $env:RUNNER_TEMP/image-candidate.json

The validator accepts one UTF-8 document no larger than 16 KiB and returns the parsed candidate only after complete schema validation.

Recipe-specific toolchain assertions remain in the reviewed Dockerfile or workflow. The generic report proves whether the image build, immutable digest verification, output verification, and BuildKit cleanup boundaries succeeded; it is not an arbitrary key/value evidence envelope.

Build arguments with secret-shaped names are rejected. Use BuildKit secret mounts for future secret-bearing build inputs; do not pass secrets as ordinary build arguments.

Before and after every build, the helper deletes BuildKit history before pruning cache. It retries cleanup for up to three minutes because references from an abruptly disconnected client can be released asynchronously. The helper succeeds only after both history and disk usage are verified empty. BuildKit 0.32.2 serializes empty disk usage as JSON null; any other nonempty value is retained state and fails the job boundary.

PITCREW_BUILDER_CLEANUP_TIMEOUT_SECONDS may set the fail-closed cleanup retry window between 1 and 600 seconds; the default remains 180. Changing the timeout never permits reuse while cache or history remains.

Qualification¶

Before enabling a repository workflow, prove:

the service runs as UID 1000 and is not privileged;
all three required security options are present;
an authenticated client connects and a CA-only client is rejected;
/var/run/docker.sock and the server private key are absent inside workers;
an unrelated profile cannot resolve buildkitd;
literal build arguments and labels are not shell-expanded;
pull-request mode produces an OCI tarball and no registry tag;
push mode returns the same digest as the registry;
buildctl du is empty after each job; and
the next job removes cache/history left by an interrupted predecessor.

Update and rollback¶

Update PitCrew through the published release and replay the service setup plus profile setup. Existing assigned workers finish naturally.

If qualification fails:

keep repository image-builder workflows disabled;
stop or drain the exact image-builder profile;
stop the service without deleting its volumes;
retain service logs, configuration, certificate fingerprints, and state for diagnosis; and
restore the prior workflow and PitCrew release.