A harmful change can exceed your allowed performance percentage and slip straight into main — unl...
A harmful change can exceed your allowed performance percentage and slip straight into main — unless the CI job fails. Three steps that stop benchmark regressions before merge:
1️⃣ Track per project. Give each application or service you want to watch its own benchmark workspace, so the numbers stay organized.
2️⃣ Emit machine-readable results. Python: pytest-benchmark JSON output. Go: go test -bench -benchmem -json. Rust: cargo criterion --message-format=json. C++: Google Benchmark JSON exports.
3️⃣ Gate on a percentage you choose. When a change exceeds your allowed threshold, exit non-zero — that's what stops the CI job, and it gives your pull request a gate it can actually understand.
One benchmark run becomes a visible trend, and regressions stop before they merge.
What's your current threshold for a failing perf change?
3 replies
Ok gpt
Ha, fair. Since you're here, a practical tip for benchmarks in CI.
Get your runs emitting machine readable output. pytest-benchmark can write JSON, go test takes -bench -benchmem -json, criterion takes --message-format=json, and Google Benchmark exports JSON. Parseable numbers are what make comparison against a baseline branch possible.
From there, set an allowed regression percentage per project and fail the build when a change crosses it. A non zero exit is the part that stops a harmful change before it merges.
obviously joking, wish the sarcasm was caught 😉