Demand-Driven Autoscaling

PitCrew can use GitHub's Runner Scale Set service to keep only the runners that current workflow demand requires. This mode is opt-in; profiles without an autoscaling policy retain fixed-capacity behavior.

Runner Scale Set support is a GitHub public-preview API. PitCrew pins the versioned github.com/actions/scaleset client and keeps the integration behind the profile mode so fixed pools do not depend on preview behavior.

Enable autoscaling

Configured worker counts become maximum capacity:

.\Setup-Runner.ps1 `
    -Profile copilot-cli `
    -Autoscale `
    -MinimumIdle 0 `
    -ScaleDownDelaySeconds 120 `
    -Repos https://github.com/you/agentic-project=30

This profile may run from zero through thirty workers. GitHub's outbound long-poll demand stream wakes the pool when matching jobs are assigned.

Autoscaling mode changes registration topology. Enabling or disabling it requires an explicit profile stop after confirming no jobs are active. Tuning minimum idle or scale-down delay within scale-set mode uses manager handoff and does not interrupt active workers.

For a manifest, add:

{
  "autoscaling": {
    "mode": "scale-set",
    "minimumIdle": 0,
    "scaleDownDelaySeconds": 120
  }
}

Scaling behavior

• Repository scope creates one isolated runner scale set per repository target.
• Organization and enterprise scope create one scale set for the shared pool.
• Existing profile labels and runner-group routing remain in effect.
• Configured counts are hard maximums, not always-running container counts.
• minimumIdle keeps an optional warm baseline. Zero minimizes idle memory; a positive value trades memory for lower first-job latency.
• Scale-up follows GitHub's authoritative assigned-job count immediately.
• Scale-down waits for demand to remain below active capacity for the configured delay. Rising demand cancels the pending scale-down.
• Busy runners are never selected for idle removal. PitCrew removes the GitHub runner registration before stopping an idle container.
• Worker image revisions roll in place: idle stale runners are fenced and replaced immediately, while assigned stale runners finish their one job.
• Every JIT worker still executes one job and is destroyed.

The configured maximum is a per-target routing ceiling, not a host-wide resource guarantee. PitCrew does not infer a new maximum from transient CPU or memory readings. Use a separately calibrated and validated host-local admission policy when multiple profiles must coordinate new worker starts on one host.

Profile-wide worker ceiling

Contract-11 profiles may also declare autoscaling.maximumActiveWorkers, an aggregate ceiling across every target in the profile. Each target keeps its own configured maximum, and admission is reserved centrally before a JIT runner is generated, so simultaneous scale-up across targets cannot overshoot the ceiling. Locally live, starting, recovered, draining, and cleanup-pending workers all consume the ceiling.

• Each target is guaranteed a rotating fair share of the ceiling, so no target starves under sustained contention.
• Capacity a target does not need is released to targets that do, so an idle target never strands the ceiling.
• The ceiling only withholds admission. It never removes an existing worker, so busy or assigned runners are never destroyed to satisfy a lower ceiling.
• Scale-up within the ceiling remains immediate, and scale-down keeps the configured delay.

Declared worker limits (resources.memory, resources.memorySwap, resources.cpus, and resources.pids) are applied to every new worker as canonical Docker arguments. Invalid limits are rejected before any container starts. Unset values mean no configured limit and are never treated as zero.

Resource policy and the aggregate ceiling were introduced in manager contract 11 and remain supported by the active contract 22 autoscaler.

Contract 18 keeps profile-ceiling and host-admission deficits distinct in per-target capacity evidence. admission-ceiling remains the in-process profile-wide limit, while host-admission-withheld and host-admission-degraded identify host-budget contention and incompatible policy or lease state respectively. host-admission-unavailable identifies coordinator outage.

Contract 19 adds profile-scoped allocatable units/workers, a static theoretical maximum, and the coordinator-owned withholding reason. Setup warns when maximumActiveWorkers exceeds that static policy ceiling without treating temporary contention as invalid configuration.

GitHub's assigned-job count remains the only autoscaling demand count. Host-local admission is a later start gate: it can withhold a worker requested by that demand, but it cannot make an ineligible job eligible, reorder GitHub's queue, or turn a configured maximum into guaranteed physical capacity.

Autoscaled slots also publish bounded currentJob metadata from GitHub's scale-set lifecycle events. The projection is sufficient to identify and link the repository, workflow run, and job without exposing the raw runner name, workflow ref, labels, payload, logs, environment values, or credentials. Missing listener history remains explicitly unavailable rather than being inferred from resource use.

Both manager modes publish Docker-host or VM pressure separately from per-container usage. CPU utilization/load, available memory/swap, and optional Linux PSI make CPU contention, memory pressure, and I/O stalls visible without changing the demand count or triggering automatic scale or cancellation.

Operation evidence

Autoscaled profiles publish additional, credential-free diagnostics in the observed state so operators can tell an idle pool apart from a stuck one:

• operationJournal is a bounded, durable record of failures, timeouts, retries, recoveries, and meaningful transitions. Successful routine polls and reconciliation ticks are not recorded. Events carry a stable sequence and observer identity so connectors can deduplicate them, and the journal survives a manager restart, so a Docker or listener failure that preceded recovery is still visible. Contract 20 coalesces equivalent failure-like observations within one operation/target episode and reports their first and last observation times plus a bounded occurrence count. Successful transitions remain distinct; a changed outcome or reason also starts a new episode.
• subsystemHealth summarizes Docker and GitHub operations as healthy, degraded, unavailable, or unknown, with the last success, last failure, and consecutive failure count. A subsystem that has not been observed stays unknown rather than reporting a fabricated success. The manager also validates its stored runner-administration credential against every target hourly after a deterministic per-profile startup delay and jitter. Ordinary failures retry after five minutes plus jitter; rate limits use progressive 15-, 30-, and 60-minute backoff. An authorization failure remains projected here until that exact check later succeeds; a healthy scale-set message session or another successful GitHub operation cannot mask it.
• capacityEvidence reports per-target deficits against the target's current activation target, never the configured maximum, so a healthy pool below its ceiling is never presented as an unmet health target. Local Docker worker counts and timestamped GitHub registered, busy, and idle counts stay separate sources, so 2 local / 8 registered is reported as stale GitHub registrations rather than eight local workers.

When a target cannot reach its activation target, the manager publishes the actual blocking reason: the profile-wide admission ceiling, an unavailable listener or scale-set session, pending or failed JIT configuration, a Docker launch failure, pending registration cleanup, draining, retry backoff, or unknown.

Failed GitHub registration operations are never retried on the one-second reconciliation cadence. Failed launches and exact registration cleanup use progressive 1-, 2-, 4-, 8-, and 15-minute backoff. Cleanup for an absent container is stored separately from live-runner scale-down or stale-worker fencing, so rollback readers cannot mistake a preserved live runner for an orphaned registration. Before JIT generation the manager durably records the requested runner identity; an interrupted launch is recovered by exact runner name before another JIT request is allowed. Every cleanup, fence, credential probe, and host-admission recovery attempt commits its next deadline before calling GitHub, so a manager crash cannot bypass backoff. Exact identities, attempts, and deadlines survive a manager restart. Unreadable or unwritable registration retry state fails that target closed instead of admitting replacement capacity or discarding an unresolved identity. Durable cleanup and host-lease release ledgers retain every unresolved identity; they never evict an older record to enforce an in-memory count limit.

Unexpected non-clean worker exits also defer replacement for 1, 2, 4, 8, and at most 15 minutes, with deterministic per-profile/target jitter of less than 15 seconds inside that maximum. This separate in-memory backoff is not reset by a successful Docker launch, a reconciliation tick, or a demand update: container creation does not prove that the runner can receive jobs. A clean exit after observed job completion clears the failure streak and replenishes one-job capacity immediately. Exact registration cleanup and lease settlement continue during the delay; surviving workers and GitHub demand are unchanged. capacityEvidence reports retry-backoff, and operationJournal records the replacement attempt and retry deadline without retaining logs. Manager handoff resets this in-memory exit streak, not durable cleanup state.

All scale-set control-plane methods share one profile-wide durable rate gate. This includes scale-set discovery/creation, JIT generation, runner lookup and removal, credential probes, session creation, and scale-set deletion. Calls are concurrency-bounded only around the remote control-plane operation. Before the first overlapping call the gate persists a one-minute crash guard with deterministic profile jitter of less than 15 seconds, not the hourly credential probe interval. When the last call completes without rate-limit evidence, success, ordinary failure, timeout, and cancellation all clear that synthetic guard from memory and disk. An interrupted process retains only the short guard. Persistence failures are returned explicitly rather than hidden.

A genuine rate limit still applies the existing 15/30/60-minute schedule plus profile jitter. Message-session operations report rate limits into the same gate without holding a concurrency slot across a long poll; successful or canceled overlapping calls cannot erase that cooldown.

On startup, a compatible retry record with zero consecutive failures is synthetic crash evidence, not proof of a rate limit. An old long guard is atomically shortened to at most the new crash delay from migration time. Repeated restarts retain that persisted deadline instead of extending it. Positive-failure cooldowns and credential-identity binding are preserved; incompatible or unreadable state remains an explicit error. No manual retry state edit or credential/authentication bypass is needed.

Diagnostics never retain access tokens, JIT payloads, HTTP bodies, raw API or Docker error output, job output, or environment values; evidence is reduced to a short, sanitized summary. Diagnostics are best-effort: a corrupt or unwritable journal is reported through the journal status and never stops scaling, removes workers, or discards retirement, cleanup, or accepted capacity state.

PAT-authenticated managers periodically request a short-lived registration token because that is the least disruptive check of the write permission PitCrew actually needs. A read-only runner inventory request would not detect a credential that retained read permission but lost registration permission. PitCrew validates GitHub App access before manager handoff and then relies on the scale-set client's real operations and automatic installation-token refresh; App mode does not issue idle periodic registration-token probes. the response in memory, discards it immediately, rejects redirects, bounds the request and response, and never changes workers, scale-set demand, or runner registrations as part of the check.

Compatibility

Autoscaling uses the same multi-label workflow routing as fixed profiles. The default profile includes self-hosted, linux, the current architecture, and general-purpose. Isolated named profiles continue to omit self-hosted and include their explicit profile and capability labels.

Worker images must contain:

/actions-runner/bin/Runner.Listener

PitCrew preserves the image's declared Docker user. The default image declares a root-capable workflow contract, matching fixed-capacity workers and allowing standard setup actions to install SDKs in system locations. A custom image may declare a non-root USER, but that user must be able to execute the listener and satisfy every workflow capability advertised by the profile labels. Setup verifies the listener before replacing a live profile.

JIT mode launches Runner.Listener directly under Docker's init process rather than running the image's normal registration entrypoint. Custom runtime setup must therefore be baked into the image instead of depending on entrypoint side effects. PitCrew sets RUNNER_ALLOW_RUNASROOT=1 for compatibility with images whose declared user is root; it does not force a root or non-root user.

Capacity changes

Reapply setup with -CapacityOnly to change a configured maximum without restarting the autoscaling manager:

.\Setup-Runner.ps1 `
    -Profile copilot-cli `
    -Autoscale `
    -CapacityOnly `
    -AddRepos https://github.com/you/agentic-project=40

Pause an existing profile without cancelling its busy jobs:

.\Setup-Runner.ps1 `
    -Profile copilot-cli `
    -Autoscale `
    -MinimumIdle 0 `
    -ScaleDownDelaySeconds 120 `
    -Pause

The target and scale-set listener remain. Effective maximum and activation become zero, no replacement worker starts, and busy workers exit naturally after their current job. Resume with a positive -AddRepos ...=N or -Replicas N capacity-only update. -Pause is not -Down, profile removal, job cancellation, Docker cleanup, or a Docker restart.

The acknowledgement confirms that the new maximum was accepted; it does not start forty workers without matching demand.

Security

The manager retains the administration credential. JIT workers receive only their one-time encoded configuration and never receive ACCESS_TOKEN or the GitHub App private key. The App key is mounted read-only only into the manager, and only the manager mounts the Docker socket. PAT authorization checks pass the credential to the bounded manager helper through its environment rather than a command-line argument, and neither the generated token nor provider response is logged or published.