CI/CD on GitHub Actions Without Losing Your Mind

For three weeks, a repository I worked on had a green CI badge and no running tests.

Someone changed the test script, the step exited 0 without running anything, and every pull request merged clean. We found out when a bug shipped that an existing test would have caught on its first assertion. Nothing was broken in the ordinary sense: the pipeline reported success accurately, about work it hadn't done.

Each rule below comes with the failure it prevents, ordered by how badly that failure hurts.

Rule 1: Assert that your tests ran, not just that they passed

Prevents: a green pipeline over an empty test run.

exit 0 from a step that matched zero test files is a pass. So is a test command that skips silently because a config path moved.

Fail the build on an empty run. Jest defaults --passWithNoTests to off, so don't turn it on. pytest exits with code 5 when it collects nothing, so don't swallow it. Better, publish the test count as a job summary and set a floor: fewer than N tests, fail. Crude, and it catches the failure nobody goes looking for.

Rule 2: Pin third-party actions to a commit SHA

Prevents: a compromised maintainer account executing arbitrary code with your secrets.

uses: some-org/some-action@v3 resolves a mutable tag. Whoever controls that repository can move v3 anywhere, and your workflow runs it with whatever secrets are in scope. This has happened to widely used actions, and the blast radius was every repo pinned to a tag.

uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

Keep the version in a comment so humans can read it, and let Dependabot bump the SHAs. GitHub's own actions are lower risk; everything else gets a SHA. While you're there, set permissions: explicitly, defaulting to contents: read.

Rule 3: Don't buy a self-hosted runner to save money

Prevents: acquiring a server, and letting strangers run code on it.

Hosted Linux minutes cost a fraction of a cent each, with a free tranche on most plans. The self-hosted pitch is that you stop paying that. What you start paying is patching the box, watching its disk fill with Docker layers, and finding it wedged at 11pm before a release.

Worse, the default runner isn't ephemeral. State leaks between jobs, so you get builds that pass only on that machine. And on a public repo, a fork's pull request executes on your infrastructure.

Legitimate reasons exist: bigger machines, GPUs, access inside a private VPC. If you go there, make the runners ephemeral.

Rule 4: Use OIDC to reach your cloud, not long-lived keys

Prevents: a permanent AWS credential in repo secrets, valid until someone remembers to rotate it.

GitHub issues a short-lived OIDC token that AWS, GCP and Azure trade for temporary credentials. A trust policy on an IAM role plus permissions: id-token: write. An hour's setup.

What people get wrong is the trust condition. Scope sub to your repo and branch or environment. repo:my-org/* means any repo in the org can assume that role, including one created next week.

Rule 5: Make your cache key change when the thing it caches changes

Prevents: a build that succeeds on stale artifacts, and a cache that never hits.

Cache that never invalidates. Key on something too coarse (key: node-modules) and you serve a stale dependency tree forever. I lost most of a day to a build passing in CI and failing locally, because CI restored a node_modules from before a dependency bump. The lockfile had changed; the key hadn't.

Cache that never hits. Key on ${{ github.sha }} and every run misses, so you pay storage for nothing.

What works: hash the lockfile, use restore-keys as a prefix fallback.

key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: ${{ runner.os }}-node-

Never cache build output on a key less precise than its inputs.

Rule 6: Don't run the full matrix on every push

Prevents: a fifteen-minute feedback loop and a billing surprise.

A 3-OS × 4-version matrix is twelve jobs per push, spending real minutes on information you don't need until merge.

Run one representative combination on feature-branch pushes, the full matrix on pull requests to main. Add concurrency with cancel-in-progress: true so three quick commits don't leave two dead runs in the queue.

Billing: macOS runners bill at roughly ten times the Linux rate, Windows around double. A macOS entry in a matrix that runs on every push is the most expensive line in most CI invoices.

Rule 7: If you use path filters, understand what they do to required checks

Prevents: a pull request that can never merge.

Path filters skip a workflow when no matching file changed, and a skipped workflow reports no status. If that status is a required check in branch protection, the PR waits forever, and the way out is an admin merge or a junk commit touching a matching path.

The fix is a job that always runs and reports the status, deciding internally whether to do work. Set that up before you enable path filters, not after the first stuck PR.

Rule 8: Keep deploy logic in a script you can run locally

Prevents: debugging a deployment by pushing commits.

The worst YAML I've written was a deploy job with conditional steps, inline shell, and templated expressions computing the target environment. To test a change I pushed a commit and waited four minutes. Then again.

Put the logic in scripts/deploy.sh (or a Makefile, or a small Go binary) taking an environment argument. The workflow becomes: check out, authenticate, run the script. Now you can run it locally against staging, and the workflow stops being a language you can only execute by committing. Same for any step over five lines of shell.

Rule 9: Treat a flaky test as a CI outage

Prevents: the culture where everyone reflexively re-runs red builds.

Once "just re-run it" is the normal response, CI has stopped being a signal. People re-run through real failures without reading them. That's how the thing in my opening paragraph survives three weeks.

Quarantine flaky tests into a non-blocking job the same day, with an owner and a date. Don't add blanket retries; they hide the flakiness and keep the habit.

---

None of this is really about GitHub Actions, which is fine and roughly as good as its competitors. It's that CI is the one system whose failure mode is silence. A broken API pages you. A broken pipeline shows a green tick, and you believe it, because believing it is the entire point.