Blahaj zone hacked
Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.
The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn't use postgres). From what we can tell, the attacker did not do anything with that access, so we don't believe any user data was accessed, but we can't be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.
Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.
We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it's as smooth as we think it will be, but we can't be sure!
As stated earlier, our writefreely instance is still up and running as it wasn't impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.
We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don't plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won't be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn't seem worth the cost/effort to run it isolated.
in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked
Edit - Piefed is back now!
https://pen.blahaj.zone/supakaity/weve-been-hackedOpen linkView original on lemmy.blahaj.zone
Is there any way to change the email address on an existing lemmy.blahaj account? I'm not the best at research, sorry about that - but I wasn't able to find an answer through a search engine and I haven't found the option on any of the listed frontends.
https://lem.lemmy.blahaj.zone/settings
Appreciate the steer - I was able to change my password, but nothing I change in the 'Settings' field saves when I click the button. Sorry for asking but really appreciate the support.
Edit: working now, not sure what was different - either way, thanks for the help, appreciated
Kaity fixed it. Sorry, forgot to let you know!
waow o.o;
i sure showed up at a doozie of a time... LITERALLY making my first comment right here!
i applied for the account like a month ago and finally managed to check on the status, only literally just now finished setting things up the rest of the way X3 feel like i lowkey dodged a bullet that I didn't put anything up before it would've been stolen.
my condolences for the hardship you're going through D:
I'm curious if they hardened their SE Linux, and I wonder what kernel bug was used and if its public. I know the recent batch of copy fail style exploits were patched. If these are zero days I fear the only way to accurately protect yourself is to run a full VM level hypervisor. And even then how long before we have zero days for hypervisor. I noticed C abd C++ applications seem more vulnerable so maybe moving to rust will help.
Yikeeeesss. Wishing you all the best, and many thanks for the work you do for the community!
I was wondering what was going on. First noticed it with matrix, then Lemmy.
The daisy-chaining of exploits tells me that AI was involved. Maybe that's why nothing malicious was done yet, because the AI was running autonomously and caught before the person could do anything?
This type of stuff is going to happen more often. Small projects are going to be hit as much as big ones because there are attackers scanning for everything and anything then telling an AI to keep trying to break it 24/7.
I've got no idea how you're supposed to defend against this. All I know is that small exploits aren't small anymore. Any ounce of leverage from each system involved will be used to pry open your servers by an agent brute force attacking all night every night.
That was kinda my feel as to what happened as well. It feels like if it was someone targeting us specifically with a multi layered attack, they'd have done something more overtly hostile. Deleted our data, defaced our site, or something. But the fact they don't seem to have done much of anything after getting in there, and the fact that there wasn't much in there of much use in the first place felt at odds with the sophistication of the attack. Which is why I am leaning towards it being driven by an LLM
The feeling when a small hobby non-profit project gets hacked and the owners quickly respond to the users and say "hey, we got hacked but don't worry, your passwords are safe because they were encrypted!!"
But a damn multi-billion company gets hacked, takes months to tell the users and their answer is: "so... a few months ago, we got hacked, but it wasn't that bad so we didn't think about telling you until someone found our database for sale in a forum. Also, change your passwords, email, physical adress, bank account, credit cards and if you sent it to us, your SSN, because we didn't think it was important so it was all stored in a plain wordpad file without any encryption".
I know this must have been awful for you guys, but damn if it feels good to know that even if the fucker got access to your database, they couldn't do shit because you were competent and took measures to protect your users in a way a multi-billion company doesn't.
I get your sentiment, but the difference is the mega-company has to worry about what they say for when they inevitably get sued.
No one is going to sue blahaj, and their currency is trust and communication, so it helps to be open.
No, the difference is that blahaj encrypts user passwords while a multi-billion dollar company stores them in a fucking plaintext file (alongside the credit card numbers and other sensitive data).
Also, under GDPR, a company must inform of a databreach ASAP, and they only do when they get caught.
I’ve been on Lemmy for a while and the sheer naive idealism on here boggles me sometimes. You guys think “megacorp is evil and stoopid”, when realistically the person who discovered the breach freaks out because he may lose his job and is the sole breadwinner of the family, then takes it to his manager, who isn’t in a much better position. It bounces through a few low level admins and teams trying to figure out the extent, since there are hundreds of applications and systems, all while the clock is ticking by what you want to see. Only once they have things does it get bubbled up because they want to make sure they can answer to the VP in hopefully a way they won’t be scapegoated and lose their jobs. Finally it may get to the people who do have control of information but by now it’s been a while and past your idealistic timeframe. All because a common working man doesn’t want to become homeless.
But sure, blame the corporation, because it’s a single monolithic entity that you can focus your ire on instead of showing some sympathy for the working people, who are also here on Lemmy, fearing for their jobs.
Uh, what entity do you think it's responsible for the environment you just described?
Corporations are responsible for him being in that position. Hope that clears it up for you.
Thank you 🫂 I've been through some disastrous code deployments, but I know those experiences could never truly compare to something like this- stress, fear, accountability, and just feeling violated. You all must have put in sooo much effort and had to make some difficult decisions. Thank you for all of your time and knowledge to creating and supporting this space for us 🩷
See, this is why I respect the hell out of you Ada. Well, one of the reasons, because there are plenty more. But this is a perfect example of the kind of person you are, as well as the kind of admin. Transparency, rapid response, and you even opened up with an apology for someone else having screwed things up.
That goes for the entire blahaj team, but you are very much the face of it, and I just wanted to say something that I very often think, that we're all damn lucky you're here.
Thank you all for everything you're doing to keep users safe and the servers functional ~
Thank you so much kaity and ada for everything you do and your moral integrity. I don't envy any of this.
Sounds like a real mess. It must be a lot of work running infrastructure like this, so you should know we appreciate all the work you guys do.
Getting hacked is never an if, it's a when.
So sorry that you've had to shoulder all this. I really do hope you took breaks and didn't overwhelm yourselves. I understand remediating the hack itself quickly was important, but I hope you took a break and got good rest before you brought everything back online. Even in such a serious situation, I want to know my admins are still caring for themselves, too. It's hard to do this stuff on such a small scale when we have literal nation state actors doing hacks, it's a literal 24/7 threat.
Anyway, please be kind to yourselves. Thanks so much for all the hard work and bringing a beautiful community together.
Fingers crossed this gets sorted out, blahaj.zone is such a blessing.
Thank you so much, Ada and Will. Appreciate the transparency! :3
To all curious, for the future: if you cannot go to your account on the Blåhaj instance or open up any stuff from there, check the desktop website of the instance, just go to lemmy.blahaj.zone (or its piefed equivalent).
Chances are, that there may be something on it. If you have an alt, I'd recommend one on an instance that's mutually federated with the Blåhaj one.
For changing passwords, your app may not support it - use the desktop environment.
Consider donating to the Blåhaj instance - kofi link!
(Might be good to put that in the sidebar too...)
This feels like an inappropriate time & place to plug my instance, but I have a bot set up to mirror instance bans from blahaj (and dbzer0), so my instance can be a safe space as a back up for folks who need one. Important differences in that regard though are that I do have downvotes enabled, federate with hexbear, and could only copy bans going back a little over a year (which is over 3000 accounts banned btw! They do so much work to keep a safe space it's jaw dropping- I donate monthly as a thanks for the ban list lol)
thank you for all your hard work Kaity, Ada, and the rest of team, and for the transparency. even tho this was a horrible thing, the honesty and work makes me hopeful in a dark world. lots of love <333
Thank you for your hard work and transparency, Blahaj Zone team!
Thank you both for working so hard to deal with this! I’ve changed my password, hopefully nothing more will come of it. I hope other instances are also on the lookout for this hack.
Thank you for bringing my favorite source of memes back online. You two are much appreciated!
I've waded through my share of critical incidents and systems recoveries. The work can be deeply stressful and infuriating as you gradually uncover inevitable missteps, find the footprints of malicious actors and dream up countless hindsight mitigations that would have prevented all this.
Bless you, kind friends. I know how hard this is. Your work and diligence has value, and this entire community appreciates it.
o7 Admiring your tenacity, welcome back.
People wishing to manage their lemmy account should use the Lemmy UI (web) frontend.
Were your services containerized? Just curious. Systems architect here. Find me on LinkedIn. Curious if you need or want a hand. - Opal Wild
Mostly no. Our smaller ones were.
Your passion project is community. You're doing well and good. Thank you for protecting me and protecting us. If you need, I'm free as in beer and sex. 💖🫶
Slightly off topic but what is the exact meaning of the 'free as in sex' phrase? I know 'free as in speech' (which is ironic because speech is not that free nowadays) and 'free as in beer' but it is first time coming across the sex version.
I suspect it is a reference to something Linus Torvalds said, something asking the lines of "software is like sex; it is better when it's free".
I'm a whore.
If that were true, the sex would most definitely not be free.
No, it's true. I'm being appreciative. Also, maybe don't try to call out transsexual sex workers offering alms to the gods of Blahaj Zone. 😕 I'm also a systems architect. You coulda just read my posts to verify and be kinder to me. 😔
Fuuuuck.
Glad everything could be straightened out, but dog damnit that sounds like a shitload of work just because someone decided to be an asshole >:-(
Thx for resurrecting us back.
Oof! I'm really sorry that happened, to our blaj kindred. Hopefully everyone and everything successfully mitigates damage and restored to the fullest extent.
Ouch, that’s rough. Good work and good luck!
This is a reminder from someone in IT with an interest in security, use discrete, unique passwords for accounts you are concerned about. Finance, health, banking, etc. use different passwords. For places you don't care about use a throw away only for those sites that don't have PII or HIPPA.
This, very much this!
Use a password manager. Keepass if you're paranoid, Bitwarden if you're slightly less paranoid.
Idk how the built-in OS and browser ones are these days. Browser seems a little loose.
Browser ones aren't to be trusted. But yeah a password manager is pretty much a requirement at this point. Just make the master password hard to crack.
+1 for Keepass. Specifically, KeepassXC.
As ever, both of your dedication to transparency and communication is both excellent and very gratefully received! I did see you said in the post above Ada that keypair rotation and all that that entails would make Piefed a tricker recovery but I was wondering if you had any updates for Piefed recovery? If you don't, you don't :) I fully appreciate how time consuming dealing with all this is, I just thought I'd ask.
We just ran out of time to get Piefed back yesterday (Australian time). We're also navigating around moving house and the Kaity's day job. It will be up as soon as we can today (It's currently 7am here)
Yeah I thought the timezone thing might be playing a part! I hope I didn't come across like I was complaining in any way as I definitely am not :)
Are IP addresses stored? And if so are they affected?
It looks like IP addresses are stored in the DB in lemmy. It's possible that the attacker had access to those IPs, but we don't believe they accessed them.
This is the sort of thing we would turn off if we could :\
I believe IP addresses are anonymized on hexbear although I dont know how it's done.
We could do that by direct DB manipulation.
I just use a VPN. Paid one, but not one that's advertised everywhere.
I would like to bring up my onion service post from earlier https://lem.lemmy.blahaj.zone/post/22655537
Neither side of the connection knows the ip address.
I don't know anything about server management.
Had a feeling that this was the fact. Glad to be back
Thanks for sharing all the technical details!
Did you have ssh keys configured between your machines, or is all of blahaj on a single server? (Wondering how they got from postgres -> root -> other servers)
We run our instances across multiple servers, but the postgres databases are all hosted together on a single server, though technically not a single server, as, at the time of the attack, we also hosted a backup database server, which was spec'd to backup our instances, but not serve them. Their access was limited to the main postgres server, but that server holds the databases for all of our instances. It looks like the script they used in the postgres exploit to give them local access interfered with the cleanup/backup process, so WOL files would get written, but not deleted, which filled up the disk on the main machine, and ultimately, caused it to fail over to the backup machine.
In theory, they could have used the same script/exploit on the backup machine, but because it wasn't spec'd to serve all of our instances at once, everything fell over at this point. That is what alerted us to the issue, and also limited the attackers available time in the system.
So happy you're back! I was so desperate as to visit Reddit a few times. It was horrible. You're amazing! ❤️❤️❤️
Status of Blahaj registration links?
Can you give me some more details on what you've run in to?
Just wondering if those links were exposed to the adversary? I know there is a statement on them when we received them telling us not to share them with anybody.
Guess we now know where the database problems where coming from.
Oof. People can be shitty. It sounds like not collecting data spared a lot of potential damage, though, so good on you for that. I know was the point, and this just proves why that was such a good decision.
Oh, minor thing: there's some piece of punctuation in the name pgcrypto that your blog software is interpreting as markdown to start and stop italics. Luckily you mention it twice in fairly quick succession, so the passage in italics is relatively short, but you might want to escape out that character. I realize this is an unbelievably low priority right now, and possibly forever. It's just something I would want pointed out to me if it were my post.
It’s good to be back
im glad that they didnt do too much damage. the person/people who did this have to truly have nothing better to do with their time. i cant imagine how sad it must be to be them.