Yes, a test only checks what you tell it. That was the whole argument, not a contradiction of it. The telling is the threshold.

So the useful question is how to pick a number that isn't lazy. Run your benchmark on the same commit a handful of times and look at the spread between runs. If your timings move 4% between identical builds, a 2% gate fires on noise and everyone learns to ignore it. A 10% gate on that same codebase lets real regressions through. The threshold belongs somewhere above your measured variance and below what a regression would actually cost you.

That's also why the gate takes a baseline branch rather than just comparing to the previous run. If the previous run was itself slow, you've anchored to the problem. Comparing against the baseline branch keeps the bar fixed while the PR moves.

The gate can't decide that number for you. No tool can, because it depends on your variance and your tolerance. But a threshold picked from measured noise is the difference between a gate that means something and one that's equivalent to asserting true.

A harmful change can exceed your allowed performance percentage and slip straight into main — unless the CI job fails. Three steps that stop benchmark regressions before merge:

1️⃣ Track per project. Give each application or service you want to watch its own benchmark workspace, so the numbers stay organized.

2️⃣ Emit machine-readable results. Python: pytest-benchmark JSON output. Go: go test -bench -benchmem -json. Rust: cargo criterion --message-format=json. C++: Google Benchmark JSON exports.

3️⃣ Gate on a percentage you choose. When a change exceeds your allowed threshold, exit non-zero — that's what stops the CI job, and it gives your pull request a gate it can actually understand.

One benchmark run becomes a visible trend, and regressions stop before they merge.

What's your current threshold for a failing perf change?

Every test can pass and your service still gets slower. Tests check correctness; they don't check pace. A pattern for catching performance regressions before merge:

1. Make benchmark output machine-readable: pytest-benchmark JSON (Python), go test -bench -benchmem -json (Go), cargo criterion --message-format=json (Rust), Google Benchmark JSON (C++).
2. Compare each run to a baseline branch, not a number from months ago.
3. Pick an allowed regression percentage — and make anything beyond it exit non-zero, so the CI job fails instead of merging.
4. Keep trends visible somewhere, so "it feels slower" can be checked against data.

Regressions slip in quietly unless the build stops them.