A failed CI/CD run is easier to fix when you identify the failing boundary before changing code. This checklist provides a repeatable path from source checkout through deployment and rollback, with platform-neutral checks plus practical guidance for GitHub Actions, GitLab CI, and Jenkins.
Overview
Pipeline troubleshooting should begin with evidence, not assumptions. First identify the exact job, stage, commit, environment, runner, and failure timestamp. Then determine whether the problem is deterministic, intermittent, or limited to one branch, service, runner, or deployment target.
Use this decision path:
- Did the pipeline start? Check triggers, branch filters, permissions, schedules, and webhook delivery.
- Did the job obtain the expected source? Verify the commit, submodules, checkout depth, and generated files.
- Did dependencies install? Check package indexes, lockfiles, credentials, caches, and network access.
- Did validation pass? Separate compilation, unit tests, integration tests, linting, and security checks.
- Was an artifact created? Confirm its name, digest, contents, retention, and publishing destination.
- Did infrastructure or deployment fail? Inspect permissions, configuration, capacity, health checks, and rollout events.
- Is the release healthy? Compare service signals with the previous version and follow the rollback plan if required.
Preserve the failing logs and configuration before rerunning. A rerun can hide a race condition, consume a one-time token, or replace the original evidence.
What to track
Source and trigger stage
Record the event that started the pipeline: pull request, push, tag, manual run, scheduled job, or promotion from another environment. Confirm that the job used the intended revision. Common clues include “no jobs scheduled,” an unexpected branch, a missing tag, or a checkout that succeeds but lacks submodules.
For GitHub Actions, inspect the workflow event, branch and path filters, permissions, and the selected runner. For GitLab CI, check rules, variables, protected branches, and runner tags. In Jenkins, review the job parameters, webhook payload, multibranch indexing, and the node assigned to the build.
Dependencies and environment
Compare the failing run with the last successful run. Track the operating system, runtime version, package-manager version, lockfile hash, environment variables, and cache key. A cache hit is not proof of a correct environment: stale or partially restored caches can create confusing errors. Temporarily disable the cache to test that hypothesis, then repair the key or invalidation rule.
Useful checks vary by stack, but the principle is consistent:
- Print runtime versions, such as
node --version,python --version, orjava -version. - Verify the lockfile is present and that installation is running in immutable or reproducible mode where supported.
- Check DNS, proxy, registry, and certificate access from the runner.
- Confirm that credentials have the required scope without printing their values.
Tests, artifacts, and logs
Classify test failures instead of treating every red job as a code defect. Compilation failures often indicate a toolchain or dependency mismatch. A single flaky test suggests isolation and retry analysis. Widespread integration failures may point to an unavailable service, expired fixture, schema change, or shared test environment.
Track whether test reports, screenshots, coverage files, and diagnostic logs are uploaded even when a job fails. Artifact creation should include a version or commit identifier and, where practical, a checksum. This makes it possible to prove which build was promoted rather than rebuilding from the same source and hoping for identical output.
Secrets and infrastructure changes
Separate missing secrets from invalid secrets and from authorization failures. Check the target scope, environment approval, secret name, expiration, and rotation history. Never solve a secret error by echoing values into logs. Use masked output and a safe presence check instead.
For infrastructure changes, capture the plan or change summary, provider version, state-lock status, target account or project, and identity used by the job. A failed plan, a failed apply, and a successful apply followed by an unhealthy service require different owners and recovery actions.
Deployment health and rollback
After deployment, verify the rollout status, readiness or health checks, error rate, latency, logs, queue depth, and dependency connectivity. For Kubernetes DevOps workflows, inspect the workload events, pod status, image reference, configuration, probes, and recent ReplicaSet or rollout history. Avoid calling a deployment successful solely because the command returned exit code zero.
A deployment rollback checklist should answer four questions: What version is known to be healthy? Which command or promotion reverses the change? How will health be verified after rollback? Who communicates the decision and records the incident? Keep rollback actions documented beside the pipeline rather than only in personal notes.
Cadence and checkpoints
Use a small operational review rather than waiting for a major outage.
- Every failed run: Capture the stage, first meaningful error, revision, runner, environment, and whether a rerun changed the result.
- Weekly or after a release cycle: Review repeated failures, flaky tests, queue time, deployment duration, rollback frequency, and jobs that require manual intervention.
- Monthly: Check runner images, runtime versions, action or plugin versions, cache behavior, artifact retention, secret expiration dates, and pipeline permissions.
- Quarterly: Exercise rollback in a safe environment, review ownership and escalation paths, remove obsolete jobs, and test recovery when a registry, repository, or deployment target is unavailable.
- After a major platform change: Revalidate triggers, credentials, network routes, policy rules, reusable workflows, shared libraries, and environment-specific configuration.
Instrument the pipeline itself. Useful observability signals include run count, success rate, stage duration, queue time, retry count, deployment frequency, failed-change categories, and time from failure detection to recovery. Correlate pipeline identifiers with application logs and deployment events so an operator can move from a failed release to the affected service without searching several unrelated systems. An OpenTelemetry implementation can help teams standardize telemetry across the delivery and runtime sides of the system.
How to interpret changes
Look for patterns across time and dimensions. A rise in failures only on one runner suggests an image, host, capacity, or network issue. Failures across all runners after a dependency update suggest a compatibility or lockfile problem. Longer queue times with normal execution times point toward capacity or scheduling. Longer execution times with normal queues suggest tests, builds, external calls, or caches have changed.
Do not optimize for a higher pass rate by hiding failures with broad retries. Retries can be appropriate for known transient operations, but record retry counts and set limits. A job that passes only on its third attempt is an availability problem in the delivery system, even if the final status is green.
Maintain a short incident worksheet for each significant failure:
- Incident and pipeline run IDs
- Repository, commit, branch, environment, and deployment target
- First failing stage and exact error message
- Last known successful run and relevant differences
- Logs, artifacts, change records, and linked alerts
- Mitigation, rollback decision, and verification evidence
- Root cause, contributing conditions, owner, and prevention task
For broader response coordination, pair this worksheet with an incident response checklist for DevOps teams. If the failure involves credentials or exposure risk, use a defined secrets process described in the DevSecOps secrets management guide.
When to revisit
Revisit this checklist monthly for active teams and at least quarterly for stable pipelines. Return sooner after a failed production deployment, runner-image migration, CI platform change, major dependency upgrade, infrastructure-provider change, or security incident. Also revisit it when recurring data points move: rising flaky-test counts, longer queue times, more manual approvals, increased rollback frequency, or a growing gap between pipeline success and real service health.
Make the review actionable. Choose the three most repeated failure categories, assign an owner to each, and define a completion test—for example, a reproducible dependency installation, a tested rollback, or an alert linked to the deployment identifier. Compare the next review with the previous worksheet rather than relying on memory.
Finally, keep platform-specific procedures close to the workflow. Teams comparing GitHub Actions, GitLab CI, and Jenkins can use this CI/CD platform comparison, while teams redesigning the pipeline can consult the CI/CD pipeline tutorial. A troubleshooting checklist becomes valuable when it is maintained as part of release engineering—not when it is opened only after the next failed deployment.
