Spyke

Syndicated from the fediverse. Read and engage on the original instance.

View original on programming.dev

1 reply

Across those runs, regression-hunt produced 25% fewer false-positive findings than the baseline, with a 95% confidence interval of 14% to 38%.

What were the baseline numbers of false positives? No mention? Silly me, expecting some rigor in numbers that make AI look good

I checked the Scapy repo linked, no mention of how many false positives there were before and after. The only concrete numbers are

Eleven search runs across 670 agent sessions reported 471 crashes and bugs, which deduplicated to 265 distinct bugs. Of those, 129 went to the maintainers (92 private reports, 37 pull requests); as of Sep 14, 2026, 80 patches had been merged (85% of patches) and 89% of reported issues were resolved.

So, of the 265 bugs, 236 were fixed. Those are the only concrete numbers we got.

2

You reached the end

Your AI agent fixed the bug, but what else did it change? | Spyke