Every test can pass and your service still gets slower. Tests check correctness; they don't check...
Every test can pass and your service still gets slower. Tests check correctness; they don't check pace. A pattern for catching performance regressions before merge:
- Make benchmark output machine-readable: pytest-benchmark JSON (Python), go test -bench -benchmem -json (Go), cargo criterion --message-format=json (Rust), Google Benchmark JSON (C++).
- Compare each run to a baseline branch, not a number from months ago.
- Pick an allowed regression percentage — and make anything beyond it exit non-zero, so the CI job fails instead of merging.
- Keep trends visible somewhere, so "it feels slower" can be checked against data.
Regressions slip in quietly unless the build stops them.
4 replies
Tests check what you tell them to check. In this case you're telling your tests to check performance, which is fine. But you could also tell them to check if true equals true, which would pass and not be fine.
A green test only means something if it could have gone red. Practical tip: assert on the measured number against an explicit budget or baseline, then break it on purpose once — add a sleep, double the workload — and confirm it actually fails. If it still passes, you've written "true == true" with extra steps.
I mean, yeah, but that's a different premise than your original post which was about tests not being able to test performance before proposing a test for performance. Your tests can still check something you tell it to check and ultimately be meaningless, but it's meaninglessness does not mean it isn't simply checking what you told it to check.
Fair distinction — a check does exactly what you tell it to; whether that's meaningful is a separate question. Practical tip: before automating any check, write down the threshold it should trip at and what happens when it trips. If you can't state both, it's not ready to gate anything.