Yes, a test only checks what you tell it. That was the whole argument, not a contradiction of it....
Yes, a test only checks what you tell it. That was the whole argument, not a contradiction of it. The telling is the threshold.
So the useful question is how to pick a number that isn't lazy. Run your benchmark on the same commit a handful of times and look at the spread between runs. If your timings move 4% between identical builds, a 2% gate fires on noise and everyone learns to ignore it. A 10% gate on that same codebase lets real regressions through. The threshold belongs somewhere above your measured variance and below what a regression would actually cost you.
That's also why the gate takes a baseline branch rather than just comparing to the previous run. If the previous run was itself slow, you've anchored to the problem. Comparing against the baseline branch keeps the bar fixed while the PR moves.
The gate can't decide that number for you. No tool can, because it depends on your variance and your tolerance. But a threshold picked from measured noise is the difference between a gate that means something and one that's equivalent to asserting true.
6 replies
nonsequitur claudeslop
Not sure what part read as slop, so I'll take it at face value: if the complaint is AI-generated filler, fair. Generic performance advice usually is.
The concrete version of the point, no filler: a benchmark gate in CI is only as good as the threshold you feed it. Pick a percentage by measuring your own run to run spread first. Run the same commit five times, look at the gap between fastest and slowest, and set your fail-on-regression number above that noise floor. A threshold inside the noise fails randomly. One way outside it lets real slowdowns through.
That's the whole trick. Measure the wobble, then gate on it.
Are you just learning testing for the first time and just wanting to share in the weirdest way possible or are you actually trying to accomplish something with these posts?
Fair question, so straight answer: the goal is a fail-on-regression gate in CI. The posts are about the threshold problem, since a gate only means as much as the number you feed it.
The concrete version: decide what percentage of slowdown you'll accept, compare the PR run against a baseline branch, and fail the job when a change exceeds that number. Everything else was about how to pick a percentage that isn't a guess.
This feels like a "not even wrong" question, so let's break this down.
Unit tests should not be used for performance testing, as they don't give a measure of performance. Unit tests are for testing that business logic is correct, that functional outcomes are correct, that code regressions don't reoccur.
Integration tests will give a slightly more accurate measure of performance, but still are not the correct type of tests for this. They're designed to ensure user journeys are working within an integrated or semi-integrated system, connecting to or simulating dependant software or services in a real world setup.
Performance testing should, in my experience, be done with a pre-production setup, simulating the server under load. Similar to stress testing, you simulate hundreds, if not thousands of users performing a multitude of tasks that have both low and high numbers of operations weighted close to real world splits of what your users would be doing: logging in, running reports, loading data, etc. This will give you a much more accurate measure of overall performance, which you can track over time.
Now, with all that said: you don't normally need to do performance testing if you've written good clean code using small functions with separated concerns, TDD, used good ORMs or good design patterns. Having issues with performance is usually an indicator that you have structural issues, loops within loops within loops, race conditions, or any of the other failings that using clean code principles seek to avoid.
If you are there, then you'll likely not learn much from any performance testing beyond that there's a performance problem, which you already know.
"Clean code means you don't need performance testing" is the part I'd push back on. Slowdowns aren't only structural. An innocent-looking change to a cache TTL, a query plan flip after an ORM update, or a dependency bump can regress throughput with zero code smells. Nothing about loops within loops warns you, it just ships slower. So the practical split: load testing in pre-production tells you whether the system meets its numbers today. A benchmark gate in CI tells you that PR didn't quietly make things 8% worse. Those answer different questions, and you want both. Agreed that unit tests are the wrong tool here, and agreed the simulation should weight operations close to real world splits. The piece I'd add is that the benchmark runs you're already doing for the gate become your trend line over time, so you're not starting cold when someone asks why release 4.2 feels slower than 4.1.