[Completed] Upgrade to version 18
I will be updating the instance to v18 at 20:00 22:00 UTC.
See https://programming.dev/post/181191 for the changes
edit: Lemmy.ml updated and seems to have gone down. We're going to wait and see what the outage was caused by and then proceed from there.
edit 2: lemmy was down due to a ddos attack. We will upgrade at 22:00 UTC
edit 3: we had issues with the email setup getting overridden again. If you tried to sign up in the past 8 hours please try to just log in. If you can't, please message me (discord, matrix, or mastodon)
25 replies
After logging a daily cycle of the system base load, could you post some before and after metrics for resource usage for comparison? That'd be fun to see.
Of course, though I think a single image just of the past few hours would demonstrate the difference pretty well lol:
I'm not sure if you intended to do this, but you've doxxed yourself in this image.
If this was deliberate, please ignore this comment.
Ha, yes I did that in the first post ever 😜 https://programming.dev/post/1
Question for sysadmins: if you were looking at those metrics, without any context about an upgrade to production, what would you first suspect had happened?
Before cross-check unique visitor or page count remained steady, I'd have guessed the site got yeet'ed of every search platform or red flagged and blacklisted. 🚩💣☠️
Not a sysadmin, but I would assume that a load balancer or something had failed and prevented half of traffic from reaching the servers.
Phew I hope lemmy.ml is doing okay...
RIP
Looks like their back up and running using
UI: 0.18.0-rc.6 BE: 0.18.0.Relevant ticket for patch fix:
Yikes, looks like there are some serious query problems. Paging in postgres and ordering by an unindexed column.
Someone split off that issue into another issue with more details and yeah, it sounds bad.
https://github.com/LemmyNet/lemmy/issues/3312
If you don't mind me asking, how much does it cost to run all of the servers?
we're at like $48 a month for the main server, we have a staging server at about $6 a month, the domain is quite expensive, $580 or something for the first year, $427 for the following years. Email is $35 a month, but we had a snafu with it so we had actually gone with a different provider and that was $20 for 3 months. We also have cloudflare and I expect I'll need to set up CSAM protection eventually but that will be a bit of money too, so I'm working on getting a donation box set up first and then we'll move on with that.
Thanks for letting me know. I plan to contribute to the source code, as well as running my own server just to play around with it and getting more familiar with Rust.
I agree with the donation box, I'll donate when it's ready.
FYI, others commenting on hosting Lemmy in an AskLemmy thread:
responded. thanks!
Yikes, was wondering why it was down. It loaded just once stating a rate limiting error. Mostly just seeing 504 nginx errors.
I seem to be facing some issues after the 0.18 update. I'm unable to report comments on subs on lemmy.world, I made myself as a mod from my other account on lemmy.world, but I'm not seeing that reflect here. Are these known issues?
Hm. I'm not sure I understand the problem. I don't see anything like this in the current issues list. Maybe if you could describe it a bit more I might be able to log something?
The report issue seems to have resolved itself now, but the moderator issue is still present. I created a bunch of communities on lemmy.world, and I want to see if I can move over here as my main instance. I thought I'd make myself mod on those communities. I did that for 4 communities, but I only see two of them in my profile as being moderator. However, if I open my programming.dev profile on lemmy.world, it shows that I'm moderating 4 communities. Does that make sense?
I didn't know one could mod for communities on remote instances, that's cool! I've also noticed some slow or loss of synchronization between instances, e.g where the vote count for a remote post when viewed locally never steady states with the vote count when viewed directly from the remote instance.
One other thing I noticed now, is that if I mark a message as read, it doesn't disappear like it used to. It goes away if I refresh the page, but it used to disappear right away when I marked a comment as read.
That's because websockets were probably used to relay the state change to the browser. With websockets now removed to improve server performance, that state change may have to be conveyed using something else to replicate that UX.
That makes sense. Hope they find some alternative.