Stopping the scraper bot menace
Over the past few weeks, our little site has been slammed by web scraping bots.
There is a lot of headroom on the server so the increased load has not led to any noticeable degradation, but the numbers are pretty wild. Here are some stats for the past 2 weeks:
We may be popular, but we're not THAT popular.
The typical solution to something like this is to embrace services like those provided by Cloudflare. I absolutely do NOT want to do this, unlike many of our Fediverse peers. I believe surrendering autonomy and privacy in exchange for security is incompatible with what we're trying to do here.
Instead, I have been working on a PoW challenge system, with the inspiration coming from projects like Anubis (https://github.com/TecharoHQ/anubis)
The idea is simple. Clients need to solve a set of simple cryptographic challenges before they are allowed access to the site content. This rules out the vast majority of simple scrapers, and makes it computationally expensive for the more sophisticated ones. I'm calling it Tollbat because why not.
The trade-off is you will have noticed a Tollbat challenge screen before accessing HC. This could mean a 5-10 second load time every so often. I will continue to tune it to make it as light as possible for real users.
Federation, 3rd party apps like Jerboa and legitimate bots are not affected. So far things seem to be going well, but let me know if you notice any weird behavior.
The load reduction is significant to say the least. Here are more graphs for those who like them - see if you can spot when Tollbat was turned on:
Feel free to ask any questions, I'm happy to answer them.
