Spyke

Syndicated from the fediverse. Read and engage on the original instance.

View original on lemmy.today

9 replies

lemmy.ml

Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

So you DDOS'd yourself. Nice job.

30
nebekerreply
programming.dev

To be clear, that was during recovery, not a root cause.

The closest to that is this:

Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. Since April, monthly commits have grown from 1.4 billion to 2.9 billion. That growth explains the pressure on our systems, but it does not excuse these outages.

One could argue they still DDOSd themselves by promoting tools that produce commits at a much higher rate.

12

Exactly. Imagine the board meeting where somebody has to suggest “maybe we don’t want that many more users.” I was about to type “customers,” but realistically a significant portion of this new load is on free plans.

2
piefed.social

Since April, monthly commits have grown from 1.4 billion to 2.9 billion.

Slopcoders are flooding it.

15
Dave.reply
aussie.zone

LLMs looove to commit things. Every little thing. It is handy for them as they can use commits as their own memory store that they can scan easily, but it leads to very wordy-but-minor commits.

Things that humans would just lump together in a single commit like "updated docs" gets broken into commits for every single change made over the course of a few hour session.

8
magikmwreply
piefed.social

I might be weird but I like having lots of small, concise commits.

5

Same, it's really nice to look back and do a git revert if I have to. My commit messages are not as wordy, though.

4

You reached the end

The August 17 outage, and the work ahead | Spyke