Spyke

Syndicated from the fediverse. Read and engage on the original instance.

View original on retrolemmy.com
general·GeneralbyDosDude

Botnet trouble, and what we're trying to do about it. [edit: field tested now]

There was downtime over the past 24 hours. You might not have noticed if you use an app, since the API stayed up the whole time.

We were getting swamped by a botnet. Tens of thousands of different IP addresses, each making only a couple of requests, crawling deep into old posts. That makes normal rate limiting useless. Anubis was challenging them, but at the difficulty we had it set to, the bots were solving those challenges cheap enough that it didn't slow them down.

So Anubis now raises its difficulty automatically when the server is struggling. That still wasn't quite enough on its own, so there's one more layer now. If load spikes badly, the site temporarily goes private, meaning only logged-in users can read it. Bots get nothing to scrape, and it switches itself back off once things calm down. If you're logged out when that happens you'll get sent to the login page.

Fair warning that the private mode part hasn't faced a real attack yet. It works in testing, but the real test comes next time we get hit. Considering we've been attacked almost daily, we will probably see soon enough. Here's to hoping it holds.

Edit: Got attacked for a few hours again at ~18:30 UTC and it worked. Site went private on its own, stayed up, switched back when it calmed down. Federation unaffected. Hopefully this will hold any and all future attacks.

View original on retrolemmy.com
30

6 replies

DosDudereply
retrolemmy.com

According to this post, it should not: https://github.com/LemmyNet/lemmy-ui/issues/4271

I also tested it here before turning it on. With private mode enabled, an anonymous API read returns a 400, while webfinger and nodeinfo still return 200, so federation keeps working.

It works through a script that flips the private_instance column in the local_site table directly in Postgres. Lemmy caches that value with a one second TTL, so it takes effect almost immediately, no restart needed.

6
lazysoci.al

Ah excellent. I last looked at this 2 years ago. That link totally clears this up. Thank you.

I'll be making the same automation as you :)

2

Nice. Good luck. Hope it works as good for you as it seems to run here!

1
piefed.world

Sounds like a nice solution, I hope you guys have good results when it faces it's intended purpose.

5

You reached the end

Botnet trouble, and what we're trying to do about it. [edit: field tested now] | Spyke