[Done] New try at upgrading to 0.18.1 July 1st 20:00 CET
We'll give the upgrade new try tomorrow. I've had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.
Also there are newer RC versions with fixed issues.
Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.
We see a huge rise in new user signups (duh.. it's July 1st) which also stresses the server. Let's hope the improvements in 0.18.1 will also help with that.
2224
Comments404
Good luck.
I've been reading about you. Urithiru has just been attacked.
We're all counting on you.
Fuck Moash
I heard this in the SNES Star Fox voice.
I heard this in the SNES Star Fox voice.
TFW you run a small server with an open source social platform for shits any giggles and one month it explodes a thousandfold because of one greedy pig boy.
You know, maybe I'm being overly dramatic, but I really feel like we're at the beginning of something special. I realize that Lemmy has existed for some years, but the whole Reddit thing is really pushing it into the limelight, and it's new to me (I joined just before the blackout).
Seeing all the developers and admins dealing with the inevitable bugs, working together to keep the federated instances operating together during a huge surge in users, like a flotilla of disparate ships lashed together facing a major storm - it makes me excited, nervous, and hopeful. I fully intend to support developers and instances financially, but also want to give a big "yay team" shout-out.
Not dramatic at all. This feels like starting fresh, and the enthusiasm and excitement makes it feel like the early 2010s of the internet again. I love it 😍
Totally agree. Alas, I'll need to find time to help with time/knowledge. (that's my best chance, cash is an issue)
Time and knowledge are probably more important. A lot of well meaning people are copying over a bunch of Reddit content into communities here, but I think that's only useful in a small subset of them. If you're looking to scan through a bunch of memes, okay that's great. But a lot of people are looking for discussions and content to interact with, and a giant number of posts with no comments isn't helping that.
So starting or contributing to real discussions (like this one!) is a big contribution.
There is some content on thousands of niche subreddits that are there for knowledge reasons.
Not all subs are default meme subs there are tons of small niche communities that shared and collected valuable knowledge.
For me that's the biggest loss.
I think the solving the bugs and server issues is great to see, but I also think Lemmy is facing some deep issues about its design in the future, and it's going to take more than technical know-how to fix it.
Yeah, I agree. I think they can be worked out, but I don't think they have been yet. I think there's a lot of potential in the distributed architecture, and am excited to see it progress.
This is like that scene in a sci-fi movie where they're trying to repair the ship mid-battle. Good luck!
I used to work with a guy who was a sysadmin for a moderately sized webhost in the mid-2000s when things like containers and cluster orchestrators didn't exist and high availability/multi-master database systems were only really accessible to banks.
He once described patching the servers "like trying to change all the tires on a car without being allowed to pull over".
...as an IT guy who started my career as containers starred to be a thing, some of the stories from the late 90's early 2000's are bonkers. We have it so much easier with things like k8s handling the orchestration.
Watching lemmy react realtime is hella interesting, good luck y’all!
Admins be like:
PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.
While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.
@[email protected] will provide a debrief once we have completed testing.
Godspeed. I really hope this is the one and the age of randomly redirecting to sports game threads will be over!
Surprise MLB got you down? Too bad, we have orbital space lasers!
*zap!*
Yeah the random refreshes on desktop is almost unusable. Fingers crossed!
Glad I wasn't the only one experiencing that.
That should be a conspiracy.
Thank you for the heads-up. 🤙
Is it wise to do a big deployment on the day when you may receive a massive influx of new users from Reddit? The API changes take affect on the 1st.
It's a bit of a lose lose situation tbh because if we don't upgrade then the first experience those new people are gonna have of Lemmy is an unusable All feed and a broken Jerboa.
My other account is on lemm.ee and the admin there contributes a lot of fixes to the software (plus has been working with Ruud to figure out our federation issues). He upgraded us a couple days ago and it's been amazingly stable given that it was only a day after the rc1 disaster.
Ultimately I trust that he and Ruud know what they're doing, but sometimes with an untestable situation like the size of .world you just have to pull the trigger and hope it works.
Wise ? No. Useful ? Maybe. If the improvement can help with the load and the general experience, it's a good idea.
And, hm, the "site is down sometimes" experience would not be that foreign to reddit user in the first place :D
Probably not, but surprisingly some 3P apps still work. I can confirm that Joey for Reddit and Infinity for Reddit are still able to access reddit. Seems like they don't pull the plug in one go? Or cherry picking the most demanding apps first?
For context, it's July 1st 16:27 p.m., GMT+9 where I am.
There might be less users influx than expected, though we'll see in the next few hours or days.
Well website crashing all the time and jerboa crash loops. Woop, at least I've found a working alternative client now
Server feels a lot better now than it did an hour ago. Comments are going through quickly and upvotes are working for me.
With how unbelievably awful Reddit is without 3rd party apps, all Lemmy needs to thrive is content (and servers not blowing up).
Yeah I'm fully prepared to stay here just based off how much effort has gone into making it more accessable. Hopefully the content will be able to get up to the reddit levels.
The userbase is still a long way away from critical mass, but it might have grown enough to get the snowball rolling. Especially if the power users relied on their 3rd party apps, they might be more inclined to move over, and those were disproportionately responsible for producing the content.
I don't see the point in returning to reddit. The Spez has lost the thread and lost any trust from any user who isn't too young to care or the people who don't have a brain. So anyone left, isn't worth interacting with to me. If a miraculous recovery happens, maybe, but I don't see it as likely at all.
I'm actually glad this has happened. Quality has been going down on average over time there as it got more popular. And I don't just mean the site itself.
If we spray them with glue, quite a few.
To say I will NEVER use reddit again would be a lie, but it will be very seldom, and only when I'm at a computer. Even that will only last as long as old.reddit does, which is basically guaranteed to be a short time.
Otherwise, I'm here fully.
Some of them will, some of them won't. I wouldn't worry too much about it.
I'm down for a new social media addiction as long as there's a good app for it.
Both Boost and Sync are coming to Lemmy, so you'll have at least two top tier apps to choose from.
Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.
You might want to DM @[email protected]
Sent thanks
0 years of experience willing to do the same. Me basically this meme https://i.kym-cdn.com/photos/images/newsfeed/001/446/009/d10.jpg
The lemmy code is on github.
https://github.com/LemmyNet/lemmy
Good luck. This is my favorite instance so I will be eagerly awaiting. Lemmy has been incredible and strangely what I needed. I would doom scroll reddit and see the same posts and the same comment chains and just redundancy hell.
Lemmy feels so different and the community is so welcoming. Thanks again everyone for making this lost Redditor a found lemming (is that the correct term lol).
Ugh, so true. On Lemmy, it feels like it's actually worth it to qualitatively engage with people.
Agreed. I think the user base is a big part of it. Almost by definition, this place is full of people that used to use third party Reddit apps. That group would have been dominated by people that signed up before Reddit launced their official app in 2016, so accounts that are at least seven years old, and in many cases, even older (my first Reddit account dates to 2011). That means (1) users who were teens or adults in 2016 that are now seven years older and hopefully seven years wiser, and (2) users that participated in Reddit when it was more discussion forum oriented rather than the current form of quick dopamine hit memes.
I'm nearly 40, and I suspect a decent amount of the users over here are too (i.e., the group of twentysomethings that discovered Reddit shortly after the Digg situation). Children and teenagers that signed up for Reddit within the past two or three years would be on the official app without a reason to migrate, so hopefully, that means this place is mostly adults willing to have respectful conversations with one another. We'll see if it lasts, but that's the impression I get now. It's refreshing.
I agree. I'm an early millenial and had a 10 year account that I primarily accessed using 3rd party apps. This feels much closer to reddit when I first started than reddit itself does right now, and I am grateful for its existence.
Can't you disable the post button within the upgrade progress? It's a shame to see posts to disappear due to it.
Fingers crossed. Would love for the auto refreshing to finally stop.
FYI, you can mitigate this by specifying page zero in the URL. As an example, if you're looking at the "All" page on Lemmy.world, the URL looks like this:
https://lemmy.world/home/data_type/Post/listing_type/All/sort/Active/page/1
Just change the
1at the end to a0and hit Enter. No more refresh bug.I can only wish you the best. You are an amazing person for signing up for all of this.
Thanks for all your hard work Ruud and team. This place is seriously shaping up well to completely replace my Reddit addiction.
Heyo! Small update from someone who is watching the upgrade live; it's ongoing still. Seems like they are still facing some performance issues. So grab your popcorn and wait! 🍿 (Btw, i am not a sysadmin, just a moderator)
Looking forward to it. The lag has been brutal.
Keeping my eyes crossed for you!
US people: This is 2p EDT today. I had to look it up, so I thought I'd save you the trouble.
EDIT: I'm assuming that Ruud means CEST (the summer daylight savings time version of the time zone that's currently being used there).
For those that were about to Google it... 20:00 CET is 12pm PDT and 3pm EDT
18:00 UTC would be a better time to report for an international server. (on the other hand, people who can fund us can DDG the timezones as well ;) )
I'm pretty sure OP means CEST, since we're in summer. So that means you need to subtract one more hour.
Waiting for the update so I can sign into Jerboa.
Lemmy looks like it's shaping up nicely to take over my Reddit habit
Same here. Thought I was doing something wrong, being a very fresh reddit-refugee.
Meanwhile I'm trying out connect for lemmy. Seems pretty good so far, too, in case anyone wants to give it a try as well.
Use liftoff for now
I hope it works! I'm so over posts just spawning into a page that is already loaded.
Latest version of Jerboa on Google Play doesn't work with servers below 0.18;
For those who want to use Jerboa to access lemmy.world before the upgrade to 0.18.x happens, you can install a specific release from GitHub instead of Google Play. I found version 0.0.34-alpha to work (using it rn to post this). https://github.com/dessalines/jerboa/releases/tag/0.0.34
Edit: since the update is done, it's probably best to go back to latest version. Glad I helped someone!
Performance is extremely uneven. Sometimes loads instantly, sometimes I get a timeout. Upvotes don't show up until reload. Still a lot better than the last attempt.
The server hamsters seem to be on fire, but I won't hold it against the service/community/individuals (except spez; fuck you spez). Lemmy is in the unfortunate position of being forced into a development model called "fuck it, we'll do it live!"
edit: commenting and editing seems to be fine.
o7
Godspeed you absolute madlads.
@[email protected] Your energy is amazing and commendable! Your Fediverse hobby is like what most people call a job! Thank you so much for all the hard work and attentive management, and also thanks to those that are helping! I hope more folks take a moment to look into the options for donating to the cause via Patreon and the other options you have. Excellent work ethic!
And… we’re back!
Godspeed. I really hope this is the one and the age of randomly redirecting to sports game threads will be over!
This should be some sort of conspiracy.
May the odds be ever in
yourour favor.This is our home now. May the road rise to meet us. And FUCK Spez
Good luck! Really looking forward to the new version
Good luck! Posted 19:59 CET.
I sadly couldn't use the web version of lemmy.world at all cause of the websocket issues. 0.18+ is gonna be an amazing upgrade fore this instance! All the devs and staff who made this happen have my undying respect.
Y'all are so nice and transparent here. This place is lovely, unlike Reddit and their admins lol.
Hopefully third time's the charm and the upgrade will be successful.
Most issues were caused by browsers trying to access cached code. A recent change introduced changed paths for each new release, so browsers will fetch the new files. At least those issues should be a thing of the past.
Thanks for giving me a new home.
o7
Bon courage mon ami with the upgrade. You guys rock.
Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?
Good luck! 🍀
I’m going to begin supporting this instance on Patreon starting this month, I guess symbolic of the changing times and server demands.
It’s awesome to see some teamwork between server admins. Best of luck with the upgrade, hope it goes smoothly for you this time!
I can't login on my lemmy.world account, neither on Jerboa nor in the browser, I'm getting "Incorrect login" :(
(I'm posting from Liftoff because I was already logged in)
Fingers crossed and thanks for all the transparency!!
Looks like it works, but it's very slow - 25 seconds to load the page. https://i.imgur.com/ghiofDC.png
Is the update done? I hope not because can't i upvote😅
Best of luck with everything. This is the fix that would fix the instant crashing with Jerboa if it works, correct?
o7
May god help us all
Seems I can't upvote
Edit. I can upvote, but with a lot of delay!
You have the support of Dragon of Dojima. We'll take out reddit ourselves. You have my word
Server has sped up significantly. Things are getting faster / better.
Maybe 0.18.1-RC4 will stick? Lets hope!
Cleared my cache and cookies as I thought it would be necessary like last time, but now I can't log in. The login button just turns into the animated wheel and then nothing happens. I'm typing this on a mobile where I still have an active session.
looking good so far, the instance is back, testing to post!
GL! And while at it, a big thank you for running (and fixing occasional server oopsies) this instance. :) I started to donate for the server costs, even if 1€ is mostly symbolic
If everyone on the server donated $/€1 per month we'd be Gucci.
Looks like the update is showing, but it's been slow since. Hopefully not permanent. Also having issues upvoting.
o7
My outdoor BBQ Im hosting starts at the same time as the lemmy upgrade. See you all on the other side and good luck LW admin!
Stay on target.....
Seems to have worked! Typed from Jeboa :)
Just a heads-up. I ended up having to create a new login at lemm.ee because even after the improvements in speed and so on with the upgrade, for all intents and purposes it looks like my account here may have been wiped. Can't login from anywhere, getting "Record not found" whenever I try to sign in on Connect, "incorrect login" from Jerboa, and a spinning button and then nothing on a PC browser, basically treating my account as if it doesn't exist. Some folks appear to be able to login though. Is this an ongoing server issue, or did my account actually get wiped? Same username (Grangle1) as on lemm.ee.
Posting and seems fine from Jerboa, can't log in using wefwef.
let's fucking go!!!
Good luck! And thanks for keeping us updated with what you're all doing, I don't know why but it makes me feel part of something big :D
Thanks for everything you do, [email protected]
Crossing my fingers, excited for the update
We have almost the same nickname!
So it was you took my normal username haha
Crossing my fingers for everything to work out well. I’m looking forward to being able to log into Jerboa.
Me too! I miss Jerboa.
Feels very sluggish for me
Wow, this is much faster now that caches are built.
Slow, but working.
I don't know if we're slow because of the upgrade, or because of the July rush from Reddit losing 3rd party API access.
I am probably one of many refugees coming from Reddit. Here is hoping lemmy grows and grows well.
Amen, me as well.
May the surf be calm and the sailing smooth.
Success?! Hello from 0.18 :)
The UI is a lot smoother. Feels good man.
Edit: servers are very sluggish though. Hopefully this gets fixed without needing to roll back.
Goodluck and thanks for the effort in all of this guys.
Something tells me it will work this time.
Third times the charm, baby!
o7
Looks like it worked! Thanks for all you do @Ruud!
o7
See you on the other side
Thanks Ruud. In June I donated to the Lemmy Devs and in July I’ll donate to Lemmy.world I appreciate all of your work.
Godspeed, commander.
Finally able to register. Glad to be here.
I am probably one of many refugees coming from Reddit. Here is hoping lemmy grows and grows well.
Did I just check lemmy right after the reboot? Good timing huh
I'm here to stay 🤟
Good luck!
Good luck, we’re all counting on you
Can't login anymore. I'll just use my .ca account for now while things get sorted. Not a big deal. These things take time.
Great! Let's hope it goes well since many users will default to jerboa when migrating.
What is new with the updated version later today?
Good luck!!!
Good luck.
Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.
Is this happening now? Everything seems up and running
EDIT: absolutely perfect timing
Godspeed
Seems to have worked! Typed from Jeboa :)
No problem if you have to wait, it is the best instance. Do what you have to do.
No problema si toca esperar, es la mejor instancia. Haz lo que tengas que hacer.
Thank you so much. Im happy to be here
🤘🤞
May the force be with you. Live long and prosper.
Good luck!
0.18 gives us a feed that doesn’t auto refresh and moves the comment collapse button to its rightful place to the left of the username.
thanks for keeping us informed :)
Good luck! Thanks for all your hard work
God speed
Thanks for the heads up, and good luck!
Looking forward to the update. Jerboa for Android as available on F-Droid has 0.18 as a minimum requirement. Really wanna get that one set up for this Account.
Best of luck!
Good luck with this!
Update went good so far ;)
Seems to have worked! Typed from Jeboa :)
Is the update done? The platform isn't going too well 😅
Do logins work for anyone else?
I created a "in private" Firefox browser session. I can't seem to login through that.
looking good so far, the instance is back, testing to post!
The last few minutes the site seems to be very fast! Let's hope it stays this way!
Muuuch faster now. Thanks for all the hard work guys!
Just looked. First 1/2 loads were slow but after that it's lighting fast! I think by not everyone establishing a Websocket connection and just loading once performance should increase a tad bit.
My lemmy server is also running the same versions are you are now. I was getting timeouts with lemmy.world federation so I recompiled lemmy_server with timeout changes:
Tried to logout, website on mobile won't release my session, and when I tried to login on jerboa it said password incorrect, not sure if that's a ui issue or a lemmy.world issue though
Good luck! I know you and the team will do the best possible work to get everything updated
Thanks and good luck!
Thank you so much for your hard work! I deeply appreciate it.
It's this why I can't login to the jerboa app?
Happy to be here. o7
Good luck!
o7
o7
Good luck
Can't wait
Page me if you need any halp
Hey man if y'all have planning meetings or anything for this stuff I'm willing to jump on and read the source code. 15 years engineering and dev, willing to put time in for you guys.
The progressive web app had ui issues after the update so I uninstalled it and now I can't figure out how to reinstall it 😕 tried wefwef but that doesn't load anything and can't log in. Happily staying patient though, hoping things smooth out!
New update sure is fire, post making process page is all cleared up!
Thanks for all the work you're doing, best of luck!
Working for me currently on connect for Lemmy.
Thanks
Super responsive, and I can login using Jerboa 37!
Best of luck! I really appreciate the transparency on here that you keep providing.
Best of luck!! “crossing fingers”
yay, instance is back! trying to post timed out two times, this is the third try.
Thank you for all of your hard work!
Here for it! This is the last chance to catch the wave of signups, if there's gonna be one.
yay, instance is back! trying to post timed out two times, this is the third try.
o7
Good luck!
Working super slow. Like 45 seconds per page load.
Thanks for the heads up! Fingers crossed indeed!
Hoping the update works well this time !
Thanks for all your hard work Ruud and team. This place is seriously shaping up well to completely replace my Reddit addiction.
What happened? Or is this not yet done? Sorry I’m kinda lost in the whole CET timings.
o7
my fingers are crossed for you, good luck. o7
Thank you for all of your hard work!
I hope it helps with the speed. It's like trudging on molasses right now, but can't complain much.
GLHF
A bit concerning perhaps hold this off for a week or so seems like this is very important moment for the users from r**dit to decide if they want to be here or not.
Thoughts and prayers! /s
Seriously though, good luck!!
Good luck!!!
o7
o7
You got this ruud.
Good luck, hope it works this time
May the git repos bless the server 🙏
Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?
wew. noice!
Good luck!
Hell yeah
This hasn't gone well! Fingers crossed it sorts itself out without a rollback.
Super responsive, and I can login using Jerboa 37!
Looks like DB's struggling under the load 🤔 Any chance to move DB to the separate server?
Thanks for all the work you're doing, best of luck!
It's working! Writing from jerboa.
It's working! Writing from jerboa.
I get an error message when replying to comments tho.
Thanks everyone for your hard work!
It worked this time, nice!! Feels a lot better, much more responsive.
Things look better. Thank you for the updates!
Instance got faster now, but still impossible to log in after you've cleared cookies.
Logging in doesn't work - returns 404 record not found. Resetting password does log you in though,
A lot of improvements, but some weird problems. Jerboa finally allowed me to log in but not much else. Connect is working better, but won't allow me to reply to one particular post ("incorrect language" or something). The browser version and wefwef are slow and glitchy - my comment about birds ended up in a conversation about Lemmy apps. I don't know anything about tech stuff but I get the impression that people are working very hard to fix all the issues. And the spezidus wouldn't be making things easy. Looking forward to a bigger better Lemmy!
How loaded are the servers? It would be good to see it maybe in the instance sidebar or a website
O7
entire instance proceeds to fucking obliterate itself
Srs though, hope it works well.
o7
Looking better! Let's go!
Godspeed friend haha
Good luck!
Good luck, and Godspeed.
o7
Nice work!
Was it succesfull?
I like the new layout better, just hope everything keeps working
Pages loading slowly here and sometimes not loading content. Could be due to the number of new users.
🙏
Super responsive, and I can login using Jerboa 37!
Looking awesome! faster, more responsive, new sorting types, tighter layout, no auto-load of random posts 🥳 🥳
Just need to sort out the wefwef issue, and we're back in business...
I feel like that time. On that day. Is a bad time to do something that will likely end up with people creating accounts being turned.
I feel like that time. On that day. Is a bad time to do something that will likely end up with people creating accounts being turned away.
You couldn't have picked a worse day to try something risky. There's going to be a huge flood of newcomers tomorrow, and if things aren't working, it could send them straight back to Reddit.
Things aren’t working perfect right now either. The random sports score bug is quite confusing
I don't have that in Jerboa or Liftoff... Just lots of random crashes mainly
How this goes through this time.
Oh.... that's why it's unstable right now.
What happened? Or is this not yet done? Sorry I’m kinda lost in the whole CET timings.
How this goes through this time.
How this goes through this time.
How this goes through this time.
How this goes through this time.
What happened? Or is this not yet done? Sorry I’m kinda lost in the whole CET timings.
Bailing out, you are on your own. Good luck.