Self-hosted CI/CD runners can reduce build wait times, keep workloads near private systems, and give teams more control over their build environment. They also place responsibility for isolation, patching, credentials, capacity, and cleanup on your team. This guide explains how to choose a runner architecture, register build agents safely, track operational signals, and establish a maintenance routine that remains useful as repositories and pipelines change.
Overview
A self-hosted runner is a machine, virtual machine, container, or Kubernetes-managed worker that executes jobs from a CI/CD platform. The control plane may remain hosted or may be self-managed; the runner is the execution layer that checks out code, runs tests, builds artifacts, and sometimes deploys services.
The central design question is not simply whether a runner can execute a pipeline. It is whether the runner can do so with an appropriate level of isolation, predictable capacity, limited network access, and a repeatable recovery process. A useful starting point is to separate workloads by trust and sensitivity:
- Trusted internal jobs: builds from repositories controlled by the team, with known dependencies and limited access requirements.
- Untrusted or semi-trusted jobs: pull requests, external contributions, and code that can modify pipeline configuration.
- Privileged jobs: release, deployment, signing, or infrastructure tasks that need access to production-adjacent systems.
Do not place all three categories on one broadly privileged worker. A small installation may begin with one runner pool, but it should still use labels, permissions, and pipeline rules to prevent sensitive jobs from landing on an unsuitable host. Teams evaluating broader self-hosted CI/CD options can also compare execution models in this guide to self-hosted CI/CD tools and pipeline runners.
Runner architecture comparison
| Architecture | Best fit | Advantages | Key risks or costs |
|---|---|---|---|
| Persistent host | Small, trusted workloads and simple internal builds | Easy to understand; fast warm starts; straightforward local caching | Workspace contamination, configuration drift, and difficult cleanup after failed jobs |
| Docker-based runner | Repeatable builds with container-compatible tooling | Consistent images; isolated job processes; convenient tool versioning | Container escape concerns, daemon access, image maintenance, and possible privileged operations |
| Ephemeral virtual machine | Untrusted code, sensitive builds, or stronger isolation requirements | Fresh environment per job or job group; clear destruction boundary | Slower provisioning, image maintenance, and higher infrastructure overhead |
| Kubernetes-based runner | Teams already operating a cluster with variable job demand | Scheduling, namespace controls, scalable worker creation, and declarative configuration | Cluster complexity, pod security, secret exposure, and shared-control-plane risk |
For many teams, a practical progression is to use containers for trusted builds, move sensitive or untrusted workloads to ephemeral virtual machines, and adopt Kubernetes only when its scheduling and lifecycle benefits justify the operational complexity.
What to track
Runner management becomes easier when the team records a small set of operational variables rather than relying on occasional manual inspection. Keep the inventory close to the repository or infrastructure configuration so changes are reviewable.
Capacity and performance
- Number of registered runners and the projects or groups they serve.
- Runner labels, executor type, operating system, architecture, and supported toolchains.
- Queue time, job duration, concurrency, and the frequency of capacity-related failures.
- CPU, memory, disk, and network saturation during representative builds.
- Cache hit behavior and cache storage growth.
A rising queue does not always mean more runners are needed. Long jobs, oversized artifacts, serial pipeline stages, or a cache that is not being reused may be the actual cause. Measure job duration alongside queue time before changing capacity.
Security and access
- Which repositories can schedule work on each runner.
- Whether pull requests or external contributions can reach privileged runners.
- Runner registration credentials, access tokens, deployment keys, and cloud permissions.
- Operating system and container-image patch status.
- Outbound network destinations and any inbound management paths.
- Evidence that workspaces, temporary files, logs, and caches are removed between jobs.
Track permissions as explicitly as software dependencies. A runner that can deploy, publish, or access a private package store is part of the release boundary, not just a build utility. Pair runner records with dependency and artifact controls; the SBOM and dependency scanning workflow provides a related way to track software risk across releases.
Reliability and observability
Centralize job logs and runner service logs where practical, while avoiding the accidental recording of secrets. Track offline runners, repeated registration failures, disk exhaustion, image-pull errors, expired credentials, and jobs that terminate without cleanup. Define an owner for each runner pool and document the escalation path when a worker becomes unavailable.
Security hardening checklist
- Use dedicated hosts or clearly separated worker pools for CI workloads.
- Apply least privilege to runner identities, cloud roles, repository access, and deployment credentials.
- Prefer short-lived credentials and inject them only into jobs that need them.
- Keep the runner service, operating system, executor, base images, and build tools patched.
- Disable unnecessary inbound access and restrict outbound traffic to required services where feasible.
- Do not expose a container runtime socket to untrusted jobs unless the security consequences are understood and accepted.
- Use protected branches, protected environments, approval rules, or equivalent controls for release jobs.
- Clear workspaces, temporary credentials, generated files, and sensitive caches after execution.
- Separate untrusted pull-request workloads from release and deployment runners.
- Test runner revocation and replacement instead of assuming recovery will work during an incident.
Cadence and checkpoints
A recurring review prevents runner infrastructure from becoming an invisible dependency. The exact schedule should reflect workload sensitivity and change frequency, but the following cadence is a practical baseline.
Every pipeline change
Review new commands, services, artifacts, caches, permissions, and network destinations. Treat changes to pipeline configuration as code changes. Confirm that a job still runs on the intended runner label and cannot be redirected to a more privileged pool.
Weekly
Review failed jobs, offline workers, queue time, disk usage, and repeated cleanup errors. Remove obsolete runners and investigate unexplained changes in build duration. Verify that failed or cancelled jobs do not leave sensitive material behind.
Monthly
Audit runner registration, repository assignments, identity permissions, open management ports, image versions, and cache growth. Rebuild or replace workers that have accumulated configuration drift. Confirm that backups or infrastructure definitions can recreate the runner without relying on undocumented manual steps.
Quarterly and after major platform changes
Reassess executor support, runner requirements, container-image guidance, Kubernetes policies, authentication methods, and pipeline permission behavior. Re-test a representative build, release, rollback, and runner-revocation path. Also review whether the current architecture still matches repository trust boundaries and workload volume.
How to interpret changes
Use trends and comparisons rather than a single failed job to decide what to change.
- Queue time increases while job duration is stable: check runner concurrency, offline workers, scheduling labels, and peak-hour demand.
- Job duration increases with higher cache misses: inspect cache keys, storage limits, dependency changes, and network access to package repositories.
- Failures follow image or tool updates: compare the old and new image contents, pin a known-good version temporarily, and test the update in a separate pool.
- Only external-contribution jobs fail or raise security concerns: review trust boundaries and move those jobs to isolated, lower-privilege runners rather than granting broader access.
- Disk usage grows after successful builds: inspect artifacts, Docker layers, package caches, and workspaces; add retention and cleanup controls.
- Deployment jobs pass but releases remain risky: check approval gates, credential scope, rollback readiness, and environment separation. A green build is not by itself a safe deployment.
Keep changes reversible. Add a new runner pool before removing the old one, test with a representative repository, and document the decision. For operational failures, connect runner troubleshooting with a tested CI/CD rollback and incident response playbook.
When to revisit
Revisit this guide on a monthly or quarterly schedule, and sooner when a recurring data point changes materially. A new repository, a shift from internal to external contributions, a production deployment requirement, a major executor update, or a change in build volume can invalidate earlier assumptions.
Use this short runbook at each review:
- Export the current runner inventory, labels, executor types, and repository assignments.
- Compare queue time, job duration, failure categories, capacity, and disk usage with the previous review.
- Audit credentials, network paths, image versions, operating-system patches, and privileged job access.
- Test cleanup by cancelling a job and confirming that its workspace and temporary credentials are removed.
- Replace or rebuild one worker from documented configuration to verify recovery.
- Record decisions, owners, due dates, and the next review date in the team’s infrastructure documentation.
Finally, update the runner design whenever the repository workflow changes substantially. If the team is also standardizing development environments, a reproducible dev container setup can help align local tooling with CI, but local consistency should not replace runner isolation. Safe self-hosted CI/CD is an operating practice: define boundaries, observe the system, rotate and rebuild regularly, and keep the recovery path demonstrably usable.
