Lemmy.ml is back up! Apologies for the downtime.
Sometime early this morning, our database started having issues, most likely due to a disk corruption issue.
Unfortunately the postgres database was unrecoverable, and we had to restore from a nightly backup, so a few hours of posts and comments will be missing.
Apologies again for the downtime, and for the disruption.
307
Comments29
We are a bit unlucky with this server
Oh holy cow, those servers are running at some pretty high temps!
Liquid cooling required
That's the power of optimized rust 🦀🦀
Holy fuck. I thought this was a joke before I saw who was posting it.
Do you know that you are doing to me?! I actually had to work while lemmy was down...
no kidding. this particular FWP was... unpleasant.
Awesome that it's back up! Thanks for all your hard work
o7
How does lemmy federation work in this case? Conceivably after being restored from backup the lemmy.ml instance could see those few hours of lost history as federated to other lemmy instances and resync it back as the host instance. Obv I'm vastly oversimplifiying things but what happens today?
I believe they're lost, as recieved apub content doesn't have the original post / comments, but I'm not totally sure. cc @[email protected]
It strikes me that there is the potential to use trusted remote servers as a means of recovering the lost data. I mean, nearly every lemmy instance except lemmy.ml will have copies of the missing data, and given the hugely redundant availability of that data (including the ability to compare from multiple sources to establish/verify trust), using that data to rebuild missing content seems like it could be useful functionality.
If I understand, federating contents through ActivityPub only works once. Sounds a good feature to re-download contents again, but may introduce additional work as there should be some way to know if a content is missing then another job to rebuild
That doesnt work because we generally cant trust remote servers. Plus we dont even know where to fetch from, so wed have to run a complete crawl of all known instances which isnt practical.
We posted in some of the lemmy matrix chats, but ya we should also use our mastodon too.
Yep.
it'd be helpful to post to sth like masto yeah, as its much more public facing than matrix.
No worries, glad to hear you got it up and running again :)
No apologies necessary. Y’all have been working overtime to get things back in order, and I appreciate it!
I AM TALKING IN ALL CAPS BECAUSE I AM MAD
HOW DARE THIS FREE SERVICE I USE GO DOWN HOW EVER WILL I POST MEMES FROM MY OPAL THRONE
Thanks for coming to my Ted talk.
also thank you for hosting us and working on the issue!looks like some replay action too. my instance just got spanked.
Welcome back!
I saw the "502 Gateway" HTTP error message, but I never even closed my browser tab. I refreshed again today and Lemmy.ml was back online! Thanks for your hard work!
So happy it's back!
Only was a cable
Please tell me you've been saving this picture just for this type of occasion. 😂
Nope, only searched Lemmy server. 🤥
Glad to see it back
Remember to make regular backups, kiddos.