Important Notice: Possible security incident - no detected impact
Between 19:45 UTC and 19:50 UTC, there was a mistake in how information was stored temporarily (cached) on Beehaw. This mistake could have allowed some people to see and use other people's accounts without permission.
If you were using the website during that time, please check that your account settings and email address are still correct. Also, make sure that any posts or actions you made during that time are still connected to your account.
It's important to note that we don't have any proof that this error was actually used by anyone to do anything bad during the short time it happened.
I can verify that @[email protected] is one of our volunteer sysadmins and this incident was witnessed by me.
same, same :p
I can, also, verify that @[email protected] is another volunteer sysadmin here at Beehaw.
That's one of the two hard problems of programming: cache invalidation, naming things, and off-by-one errors.
thanks for alerting everyone to this aswell as reacring quickly
Do we know what could have caused this cache error?
Yes; configuration settings for the web server involving improving performance. Those settings have been changed back to the previous, non-issue ones. So this should not occur at this time, or again.
Is this a mistake that's easy to do for an inexperienced instance admin or just a consequence of too much fiddling and shouldn't be an issue for other instances?
Was a result of too much fiddling. Attempting to gain even better performance from a bottleneck issue due to recent user influx. It was not an error in the Lemmy instance or Lemmy-UI but rather the web server front-end misconfiguration.
Thanks for working through these issues and improving performance of the website! Very appreciated. I've been tempted recently to create my own Lemmy instance, was this a problem with an nginx configuration option? How much does Beehaw deviate from a standard Lemmy deployment?
Feel free to answer vaguely if you don't feel comfortable with giving away the details : )
Basically, this was
proxy_cache_keybeing configured incorrectly. If you don't use theproxy_cacheyou should be fine.The only thing we changed from the norm is
ulimits and some nginx settings. If we figure out what works well, we'll probably create a post about how to host lemmy. If you stick to the defaults, you'll be mostly fine if your instance isn't as big as Beehaw's.Awesome! Thanks for the response. I'd love a post sometime on hosting Lemmy. I'd find it very interesting and useful!
Im also considering to setup nginx caching for lemmy.ml. Did you find a configuration which works?
Not yet. Session tracking in Lemmy is pretty hard to proxy, I'll have to dive into the code to figure out why.
Yes.
Deviation is pretty minimal, related to configuration and customization.