they definitely rushed the 0.18.0, first they release without waiting for the captcha to be finished and now we realize it can't even be used in production yet
0.17.4 has a lot of the same problems, so going forward is the only hope. The amount of messages and peer servers alone are causing fundamental performance problems with the code, regardless of 0.17 or 0.18.
People taking down their servers to upgrade to 0.18 could have triggered a swarm of resource problems on the central federation instance that they had never experienced when there were only a dozen(?) other Lemmy instances 2 months ago.
There is a pattern of retries built into the code, but syncing has no ability to repair failures (that get past the retry logic), and there have been hand-documented cases where even with retries, there is significant data loss: https://github.com/LemmyNet/lemmy/issues/3101
I'm posting from it to you, and it's been showing internal crashes off and on, but still mostly up. I expect cascading failures from servers trying to federate and swarming each other.
I am starting to see some partial page loads, broken pages - bug with actual data content of posting/comments on them. I expect other federation servers are swarming to the site to try and replicate - and it's causing a mess given how many communities lemmy.ml is center of.
Don't be surprised if this causes cascading failures, impacts other instances, as they try to exchange postings/comments/votes with lemmy.ml and get backed up.
1
FYI: lemmy.ml is down now, hours after the upgrade to release 0.18.0 - EDIT: over 2 hours down, but BACK UP now! | Spyke
still down as of the time of this comment you are reading.
they definitely rushed the 0.18.0, first they release without waiting for the captcha to be finished and now we realize it can't even be used in production yet
0.17.4 has a lot of the same problems, so going forward is the only hope. The amount of messages and peer servers alone are causing fundamental performance problems with the code, regardless of 0.17 or 0.18.
People taking down their servers to upgrade to 0.18 could have triggered a swarm of resource problems on the central federation instance that they had never experienced when there were only a dozen(?) other Lemmy instances 2 months ago.
So will this comment sync to the lemmy.ml instance once it back up ?
There is a pattern of retries built into the code, but syncing has no ability to repair failures (that get past the retry logic), and there have been hand-documented cases where even with retries, there is significant data loss: https://github.com/LemmyNet/lemmy/issues/3101
lemmy.ml is back now, it was a DDOS attack after an exploit was found
So that's why the page won't load :(
Still down as of time of this comment, 45 minutes since I first observed it.
An I wrong or lemmy.world is down too?
I'm posting from it to you, and it's been showing internal crashes off and on, but still mostly up. I expect cascading failures from servers trying to federate and swarming each other.
I am starting to see some partial page loads, broken pages - bug with actual data content of posting/comments on them. I expect other federation servers are swarming to the site to try and replicate - and it's causing a mess given how many communities lemmy.ml is center of.
Don't be surprised if this causes cascading failures, impacts other instances, as they try to exchange postings/comments/votes with lemmy.ml and get backed up.