Spyke
lemmyworld·Lemmy.World AnnouncementsbyRuud

[Done] New try at upgrading to 0.18.1 July 1st 20:00 CET

We'll give the upgrade new try tomorrow. I've had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.

Also there are newer RC versions with fixed issues.

Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.

We see a huge rise in new user signups (duh.. it's July 1st) which also stresses the server. Let's hope the improvements in 0.18.1 will also help with that.

View original on lemmy.world
atimholtreply
lemmy.world

I've been reading about you. Urithiru has just been attacked.

26

TFW you run a small server with an open source social platform for shits any giggles and one month it explodes a thousandfold because of one greedy pig boy.

183

You know, maybe I'm being overly dramatic, but I really feel like we're at the beginning of something special. I realize that Lemmy has existed for some years, but the whole Reddit thing is really pushing it into the limelight, and it's new to me (I joined just before the blackout).

Seeing all the developers and admins dealing with the inevitable bugs, working together to keep the federated instances operating together during a huge surge in users, like a flotilla of disparate ships lashed together facing a major storm - it makes me excited, nervous, and hopeful. I fully intend to support developers and instances financially, but also want to give a big "yay team" shout-out.

142

Not dramatic at all. This feels like starting fresh, and the enthusiasm and excitement makes it feel like the early 2010s of the internet again. I love it 😍

24
lemmy.world

Totally agree. Alas, I'll need to find time to help with time/knowledge. (that's my best chance, cash is an issue)

9
lemmy.world

Time and knowledge are probably more important. A lot of well meaning people are copying over a bunch of Reddit content into communities here, but I think that's only useful in a small subset of them. If you're looking to scan through a bunch of memes, okay that's great. But a lot of people are looking for discussions and content to interact with, and a giant number of posts with no comments isn't helping that.

So starting or contributing to real discussions (like this one!) is a big contribution.

9

There is some content on thousands of niche subreddits that are there for knowledge reasons.

Not all subs are default meme subs there are tons of small niche communities that shared and collected valuable knowledge.

For me that's the biggest loss.

2
lemmy.world

I think the solving the bugs and server issues is great to see, but I also think Lemmy is facing some deep issues about its design in the future, and it's going to take more than technical know-how to fix it.

2

Yeah, I agree. I think they can be worked out, but I don't think they have been yet. I think there's a lot of potential in the distributed architecture, and am excited to see it progress.

1

This is like that scene in a sci-fi movie where they're trying to repair the ship mid-battle. Good luck!

113
RegalPotooreply
lemmy.world

I used to work with a guy who was a sysadmin for a moderately sized webhost in the mid-2000s when things like containers and cluster orchestrators didn't exist and high availability/multi-master database systems were only really accessible to banks.

He once described patching the servers "like trying to change all the tires on a car without being allowed to pull over".

20

...as an IT guy who started my career as containers starred to be a thing, some of the stories from the late 90's early 2000's are bonkers. We have it so much easier with things like k8s handling the orchestration.

6

Watching lemmy react realtime is hella interesting, good luck y’all!

18

PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.

While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.

@[email protected] will provide a debrief once we have completed testing.

106
lemmy.world

Godspeed. I really hope this is the one and the age of randomly redirecting to sports game threads will be over!

94

Surprise MLB got you down? Too bad, we have orbital space lasers!

*zap!*

18

Yeah the random refreshes on desktop is almost unusable. Fingers crossed!

16

the age of randomly redirecting to sports game threads

Glad I wasn't the only one experiencing that.

14
lemmy.world

Is it wise to do a big deployment on the day when you may receive a massive influx of new users from Reddit? The API changes take affect on the 1st.

67

It's a bit of a lose lose situation tbh because if we don't upgrade then the first experience those new people are gonna have of Lemmy is an unusable All feed and a broken Jerboa.

My other account is on lemm.ee and the admin there contributes a lot of fixes to the software (plus has been working with Ruud to figure out our federation issues). He upgraded us a couple days ago and it's been amazingly stable given that it was only a day after the rc1 disaster.

Ultimately I trust that he and Ruud know what they're doing, but sometimes with an untestable situation like the size of .world you just have to pull the trigger and hope it works.

60
cley_fayereply
lemmy.world

Wise ? No. Useful ? Maybe. If the improvement can help with the load and the general experience, it's a good idea.

And, hm, the "site is down sometimes" experience would not be that foreign to reddit user in the first place :D

29

Probably not, but surprisingly some 3P apps still work. I can confirm that Joey for Reddit and Infinity for Reddit are still able to access reddit. Seems like they don't pull the plug in one go? Or cherry picking the most demanding apps first?

For context, it's July 1st 16:27 p.m., GMT+9 where I am.

There might be less users influx than expected, though we'll see in the next few hours or days.

11

Well website crashing all the time and jerboa crash loops. Woop, at least I've found a working alternative client now

5

Server feels a lot better now than it did an hour ago. Comments are going through quickly and upvotes are working for me.

66
Coelacanthreply
lemmy.world

With how unbelievably awful Reddit is without 3rd party apps, all Lemmy needs to thrive is content (and servers not blowing up).

39
JaasBaasreply
lemmy.world

Yeah I'm fully prepared to stay here just based off how much effort has gone into making it more accessable. Hopefully the content will be able to get up to the reddit levels.

13

The userbase is still a long way away from critical mass, but it might have grown enough to get the snowball rolling. Especially if the power users relied on their 3rd party apps, they might be more inclined to move over, and those were disproportionately responsible for producing the content.

3
lemmy.world

I don't see the point in returning to reddit. The Spez has lost the thread and lost any trust from any user who isn't too young to care or the people who don't have a brain. So anyone left, isn't worth interacting with to me. If a miraculous recovery happens, maybe, but I don't see it as likely at all.

18

I'm actually glad this has happened. Quality has been going down on average over time there as it got more popular. And I don't just mean the site itself.

8

To say I will NEVER use reddit again would be a lie, but it will be very seldom, and only when I'm at a computer. Even that will only last as long as old.reddit does, which is basically guaranteed to be a short time.

Otherwise, I'm here fully.

11
xSPYXExreply
lemmy.world

I'm down for a new social media addiction as long as there's a good app for it.

5

Both Boost and Sync are coming to Lemmy, so you'll have at least two top tier apps to choose from.

1
lemmy.world

Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.

52
lemmy.world

Good luck. This is my favorite instance so I will be eagerly awaiting. Lemmy has been incredible and strangely what I needed. I would doom scroll reddit and see the same posts and the same comment chains and just redundancy hell.

Lemmy feels so different and the community is so welcoming. Thanks again everyone for making this lost Redditor a found lemming (is that the correct term lol).

50
dot20reply
lemmy.world

I would doom scroll reddit and see the same posts and the same comment chains and just redundancy hell.

Ugh, so true. On Lemmy, it feels like it's actually worth it to qualitatively engage with people.

6
lemmy.world

Agreed. I think the user base is a big part of it. Almost by definition, this place is full of people that used to use third party Reddit apps. That group would have been dominated by people that signed up before Reddit launced their official app in 2016, so accounts that are at least seven years old, and in many cases, even older (my first Reddit account dates to 2011). That means (1) users who were teens or adults in 2016 that are now seven years older and hopefully seven years wiser, and (2) users that participated in Reddit when it was more discussion forum oriented rather than the current form of quick dopamine hit memes.

I'm nearly 40, and I suspect a decent amount of the users over here are too (i.e., the group of twentysomethings that discovered Reddit shortly after the Digg situation). Children and teenagers that signed up for Reddit within the past two or three years would be on the official app without a reason to migrate, so hopefully, that means this place is mostly adults willing to have respectful conversations with one another. We'll see if it lasts, but that's the impression I get now. It's refreshing.

4

I agree. I'm an early millenial and had a 10 year account that I primarily accessed using 3rd party apps. This feels much closer to reddit when I first started than reddit itself does right now, and I am grateful for its existence.

3

Can't you disable the post button within the upgrade progress? It's a shame to see posts to disappear due to it.

47
lemmy.world

Fingers crossed. Would love for the auto refreshing to finally stop.

43

I can only wish you the best. You are an amazing person for signing up for all of this.

40

Thanks for all your hard work Ruud and team. This place is seriously shaping up well to completely replace my Reddit addiction.

35

Heyo! Small update from someone who is watching the upgrade live; it's ongoing still. Seems like they are still facing some performance issues. So grab your popcorn and wait! 🍿 (Btw, i am not a sysadmin, just a moderator)

34

US people: This is 2p EDT today. I had to look it up, so I thought I'd save you the trouble.

EDIT: I'm assuming that Ruud means CEST (the summer daylight savings time version of the time zone that's currently being used there).

31
lemmy.world

For those that were about to Google it... 20:00 CET is 12pm PDT and 3pm EDT

30

18:00 UTC would be a better time to report for an international server. (on the other hand, people who can fund us can DDG the timezones as well ;) )

11

I'm pretty sure OP means CEST, since we're in summer. So that means you need to subtract one more hour.

9
lemmy.world

Waiting for the update so I can sign into Jerboa.

Lemmy looks like it's shaping up nicely to take over my Reddit habit

28

Same here. Thought I was doing something wrong, being a very fresh reddit-refugee.

Meanwhile I'm trying out connect for lemmy. Seems pretty good so far, too, in case anyone wants to give it a try as well.

5

I hope it works! I'm so over posts just spawning into a page that is already loaded.

27

Latest version of Jerboa on Google Play doesn't work with servers below 0.18;

For those who want to use Jerboa to access lemmy.world before the upgrade to 0.18.x happens, you can install a specific release from GitHub instead of Google Play. I found version 0.0.34-alpha to work (using it rn to post this). https://github.com/dessalines/jerboa/releases/tag/0.0.34

Edit: since the update is done, it's probably best to go back to latest version. Glad I helped someone!

25

Performance is extremely uneven. Sometimes loads instantly, sometimes I get a timeout. Upvotes don't show up until reload. Still a lot better than the last attempt.

The server hamsters seem to be on fire, but I won't hold it against the service/community/individuals (except spez; fuck you spez). Lemmy is in the unfortunate position of being forced into a development model called "fuck it, we'll do it live!"

edit: commenting and editing seems to be fine.

25

@[email protected] Your energy is amazing and commendable! Your Fediverse hobby is like what most people call a job! Thank you so much for all the hard work and attentive management, and also thanks to those that are helping! I hope more folks take a moment to look into the options for donating to the cause via Patreon and the other options you have. Excellent work ethic!

24
lemmy.world

Godspeed. I really hope this is the one and the age of randomly redirecting to sports game threads will be over!

22

I sadly couldn't use the web version of lemmy.world at all cause of the websocket issues. 0.18+ is gonna be an amazing upgrade fore this instance! All the devs and staff who made this happen have my undying respect.

18

Y'all are so nice and transparent here. This place is lovely, unlike Reddit and their admins lol.

18
lemmy.world

Hopefully third time's the charm and the upgrade will be successful.

17

Most issues were caused by browsers trying to access cached code. A recent change introduced changed paths for each new release, so browsers will fetch the new files. At least those issues should be a thing of the past.

13

Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?

15

It’s awesome to see some teamwork between server admins. Best of luck with the upgrade, hope it goes smoothly for you this time!

15

I can't login on my lemmy.world account, neither on Jerboa nor in the browser, I'm getting "Incorrect login" :(

(I'm posting from Liftoff because I was already logged in)

14

Best of luck with everything. This is the fix that would fix the instant crashing with Jerboa if it works, correct?

12

Seems I can't upvote

Edit. I can upvote, but with a lot of delay!

12

You have the support of Dragon of Dojima. We'll take out reddit ourselves. You have my word

12

Server has sped up significantly. Things are getting faster / better.

Maybe 0.18.1-RC4 will stick? Lets hope!

12

Cleared my cache and cookies as I thought it would be necessary like last time, but now I can't log in. The login button just turns into the animated wheel and then nothing happens. I'm typing this on a mobile where I still have an active session.

11

GL! And while at it, a big thank you for running (and fixing occasional server oopsies) this instance. :) I started to donate for the server costs, even if 1€ is mostly symbolic

11

If everyone on the server donated $/€1 per month we'd be Gucci.

7

Looks like the update is showing, but it's been slow since. Hopefully not permanent. Also having issues upvoting.

11

My outdoor BBQ Im hosting starts at the same time as the lemmy upgrade. See you all on the other side and good luck LW admin!

10

Just a heads-up. I ended up having to create a new login at lemm.ee because even after the improvements in speed and so on with the upgrade, for all intents and purposes it looks like my account here may have been wiped. Can't login from anywhere, getting "Record not found" whenever I try to sign in on Connect, "incorrect login" from Jerboa, and a spinning button and then nothing on a PC browser, basically treating my account as if it doesn't exist. Some folks appear to be able to login though. Is this an ongoing server issue, or did my account actually get wiped? Same username (Grangle1) as on lemm.ee.

9

Good luck! And thanks for keeping us updated with what you're all doing, I don't know why but it makes me feel part of something big :D

8
lemmy.world

Crossing my fingers for everything to work out well. I’m looking forward to being able to log into Jerboa.

8

Slow, but working.

I don't know if we're slow because of the upgrade, or because of the July rush from Reddit losing 3rd party API access.

7
lemmy.world

I am probably one of many refugees coming from Reddit. Here is hoping lemmy grows and grows well.

7

Success?! Hello from 0.18 :)

The UI is a lot smoother. Feels good man.

Edit: servers are very sluggish though. Hopefully this gets fixed without needing to roll back.

7
Yoz
lemmy.world

Something tells me it will work this time.

7

Thanks Ruud. In June I donated to the Lemmy Devs and in July I’ll donate to Lemmy.world I appreciate all of your work.

6

I am probably one of many refugees coming from Reddit. Here is hoping lemmy grows and grows well.

6

Can't login anymore. I'll just use my .ca account for now while things get sorted. Not a big deal. These things take time.

5

Great! Let's hope it goes well since many users will default to jerboa when migrating.

5

Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.

5

No problem if you have to wait, it is the best instance. Do what you have to do.

No problema si toca esperar, es la mejor instancia. Haz lo que tengas que hacer.

5
lemmy.world

Good luck!

0.18 gives us a feed that doesn’t auto refresh and moves the comment collapse button to its rightful place to the left of the username.

4

Looking forward to the update. Jerboa for Android as available on F-Droid has 0.18 as a minimum requirement. Really wanna get that one set up for this Account.

4

Do logins work for anyone else?

I created a "in private" Firefox browser session. I can't seem to login through that.

4

The last few minutes the site seems to be very fast! Let's hope it stays this way!

4

Just looked. First 1/2 loads were slow but after that it's lighting fast! I think by not everyone establishing a Websocket connection and just loading once performance should increase a tad bit.

4

My lemmy server is also running the same versions are you are now. I was getting timeouts with lemmy.world federation so I recompiled lemmy_server with timeout changes:

diff --git a/crates/utils/src/lib.rs b/crates/utils/src/lib.rs
index e5d07db2c..e2c592d82 100644
--- a/crates/utils/src/lib.rs
+++ b/crates/utils/src/lib.rs
@@ -18,7 +18,7 @@ use std::time::Duration;

 pub type ConnectionId = usize;

-pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);
+pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(13);

 #[macro_export]
 macro_rules! location_info {
diff --git a/src/lib.rs b/src/lib.rs
index cc77ca48f..45c621a7c 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -37,7 +37,7 @@ use tracing_subscriber::{filter::Targets, layer::SubscriberExt, Layer, Registry}
 use url::Url;

 /// Max timeout for http requests
-pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);
+pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(16);

 /// Placing the main function in lib.rs allows other crates to import it and embed Lemmy
 pub async fn start_lemmy_server() -> Result<(), LemmyError> {

4

Tried to logout, website on mobile won't release my session, and when I tried to login on jerboa it said password incorrect, not sure if that's a ui issue or a lemmy.world issue though

4

Good luck! I know you and the team will do the best possible work to get everything updated

3

Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.

3

The progressive web app had ui issues after the update so I uninstalled it and now I can't figure out how to reinstall it 😕 tried wefwef but that doesn't load anything and can't log in. Happily staying patient though, hoping things smooth out!

3

Best of luck! I really appreciate the transparency on here that you keep providing.

3

Thanks for all your hard work Ruud and team. This place is seriously shaping up well to completely replace my Reddit addiction.

3

What happened? Or is this not yet done? Sorry I’m kinda lost in the whole CET timings.

2

I hope it helps with the speed. It's like trudging on molasses right now, but can't complain much.

2

A bit concerning perhaps hold this off for a week or so seems like this is very important moment for the users from r**dit to decide if they want to be here or not.

2

Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?

2

Looks like DB's struggling under the load 🤔 Any chance to move DB to the separate server?

2

Logging in doesn't work - returns 404 record not found. Resetting password does log you in though,

2

A lot of improvements, but some weird problems. Jerboa finally allowed me to log in but not much else. Connect is working better, but won't allow me to reply to one particular post ("incorrect language" or something). The browser version and wefwef are slow and glitchy - my comment about birds ended up in a conversation about Lemmy apps. I don't know anything about tech stuff but I get the impression that people are working very hard to fix all the issues. And the spezidus wouldn't be making things easy. Looking forward to a bigger better Lemmy!

2

How loaded are the servers? It would be good to see it maybe in the instance sidebar or a website

2

entire instance proceeds to fucking obliterate itself

Srs though, hope it works well.

1

Pages loading slowly here and sometimes not loading content. Could be due to the number of new users.

1

Looking awesome! faster, more responsive, new sorting types, tighter layout, no auto-load of random posts 🥳 🥳

Just need to sort out the wefwef issue, and we're back in business...

1

I feel like that time. On that day. Is a bad time to do something that will likely end up with people creating accounts being turned.

0

I feel like that time. On that day. Is a bad time to do something that will likely end up with people creating accounts being turned away.

0
lemmy.world

You couldn't have picked a worse day to try something risky. There's going to be a huge flood of newcomers tomorrow, and if things aren't working, it could send them straight back to Reddit.

0
DrMarioreply
lemmy.world

Things aren’t working perfect right now either. The random sports score bug is quite confusing

6

I don't have that in Jerboa or Liftoff... Just lots of random crashes mainly

1

What happened? Or is this not yet done? Sorry I’m kinda lost in the whole CET timings.

-1

What happened? Or is this not yet done? Sorry I’m kinda lost in the whole CET timings.

-1