[Meta] Lemmy (not PieFed) changed to a Private Instance
LLM scrapers and bots are a continuing challenge. Lemmy is not behind Cloudflare and custom filtering rules using NGINX only gets you so far.
I'm trying the setting called "Private Instance".
It's my understanding that all this does is require a user to be logged in to use the instance. This will hopefully cut down or stop the constant scraping. Preliminary signs are that it helps a ton.
- walden
6 replies
dbzer0 is using haproxy with haphash successfully to mitigate.
https://github.com/dgl/haphash
He might be willing to share the uri/paths to approve in your proxy config to deploy haphash or anubis etc.
Do you want to give Anubis a try? That has helped our instances avoid the bots
Another vote here for anubis! It cut our traffic down by some 75% or so.
I've heard mostly good things about Anubis, but I've also heard it can be hard to get it right. I've never looked into it so I'm not sure what people are referring to... maybe the balance between blocking lots of bots but still letting humans through?
This Lemmy instance is mostly sticking around for posterity, so hopefully Private Instance cuts down on the B.S. so I don't have to think about it any further!
I had some challenges setting it up which were mostly related to using load balancing for Lemmy-UI. In general there isn't much of a problem letting humans through so long as you only apply it to the front end UI not to the API.
We have a rule to allow anyone with a jwt set which means logged in users are not challenged.
Finding an instance that blocks content from logged out users would likely put off new users, but if you're not looking for new users then requiring login seems like an easy solution to the problem.
I'd have to sit down and figure that all out. It'd be a good skill to have these days.