Lemmy.world status update 2023-07-05
Another day, another update.
More troubleshooting was done today. What did we do:
- Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
- @[email protected] created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
- We started using this image, and saw a big drop in CPU usage and disk load.
- We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a
return 404in nginx conf for/api/v3/ws. - We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
- We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set
proxy_next_upstream timeout;max_fails=5in nginx.
Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the proxy_next_upstream timeout;max_fails=5 workaround but for now it seems to hold with 1.
Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!
And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!
And thank you all for your patience, we'll keep working on it!
Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.
Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.
server load is too low, everyone upvote more stuff so i can optimize more
edit: guess there is some more work to be done ๐
Upvote causes an endless spinner on Liftoff. ๐
I'm getting 504 gateway time outs when I try to upvote
For me it works way better than before
seems like it may have been a temporary issue. It's clearing back up.
I don't understand your graph. It says you are measuring gigabit/sec but shouldn't the true performance rating be gigabeans/sec for a Lemmy instance?
And where's the statistics for days between each core dump? A healthy instance should have at least three days between each one
Depends on whether they have fiber or not.
I see what you did there.
Beans have tons of fiber!!
heh
aye aye sir, to the upvote machine!
Double the image upload size and you will see more shitposts
I was gonna argue that you'd see more bean posts, but at this point they're the same thing, both in the pun sense and the literal sense
Web-ui is very smooth rn.. is this .world?
๐
Joke aside, the improvement is like heaven and earth. Love it!. Good work teams!
I'm on another instance, but here's some federated activity for you.
All hail @[email protected]! Today is your day. You have made the single most valuable contribution and you must be celebrated! Bravo! Hurrah!
I was just going to post a meme about choosing either creating activity or spare the server from overloading. Now the joke won't stick.
Test:
Upvote if you can see this comment. ๐
Looking good from here.
Edit: And comment rapidly going through. :)
Can't see anything...
Shame you're getting no karma. Take my upvote.
I think i was on r for a year or two before I learned what karma was. I still don't understand it's value.
Oh, I thought that was the joke ๐
Like a tongue-in-cheek, remember our silly old Reddit ways kinda thing.
Ha. I don't miss it. I upvote for good content.
I can see and upvote this comment. ๐
I can also see and updoot yours. ๐
502, sorry mate.
Damn, better luck next time.
The change is noticeable. Good job guys.
Thanks for the updates.
I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing
Yup I can even post comments first try, without getting an error! Things are working well!
Really noticeable. Cool update. Thank you, guys! โค๏ธ
This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!
This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that "small instance" will mean 50k users.
I sincerely doubt reddit will die anytime soon, it'll just exist as its own thing that it's new target audience gets bored with and moves on from in a few years when something new and flashy catches their eye in the app store. Just like they do all the other apps designed in exactly the same fashion that reddit is currently morphing into.
Meanwhile Lemmy will be slowly building it's communities up to be what reddit used to be.
I'm actually kinda waiting a few releases to start promoting my instance anywhere, letting some other brave instance admins work the kinks out a bit first.
Agreed. I decided to keep my community on lemmy.world specifically because of the community investment I see being put into it.
If this project is to stay for the long haul, we gotta load test it and stabilize it. These folks are doing the important work here. Large instances are more or less inevitable if Lemmy sticks.
Wow. So much smoother today.
Great work.
You dropped this ๐
Damn tootin.
You're gonna be when you see how fast I can bean post now.
You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.
My upvote can go through fast now
Good work
I love the smell of updates in the morning.
Thank you guys for your awesome work!
Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5โฌ, we have 25.000โฌ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!
You can pry my cup of coffee from my my cold, dead hands.
Will donate anyway, I really want this project to keep going.
euro contributed ๐ซก
Is there a good link? One for USD?
If there's a thing you really need to pay in a foreign currency, look into Revolut or Wise. Since I occasionally have to pay stuff in Turkish lira, GBP and donated to USD-only Receivers I like to keep Revolut as my secondary bank account since exchanging one currency to another is completely free!
you inspired me to serve the greater good!
For example, if you speak a second language, you can even help with translation in projects. Its very easy. E.g. I translated the Jerboa (Lemmy client for Android) in Greek 2-3 days ago. I needed only 1 hour to finish and special 15-20 minutes for fixes that I missed, yesterday.
Boy does it feel good to have those reports and understand the work you guys do. It's really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.
I'm not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!
Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?
You'd be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.
This. Most of the time, the bottleneck will be the database backend.
Curious if lemmy.world uses separate reader/writer instances.
Impressive if true!
if it runs this well on a single container, considering the amount of users it has, I fear the power it'd hold with more
Lemmy is built on an async work stealing runtime and you can get very large instances from Amazon and the like.
I'm sure the instance isn't massive but number of containers and a single "host" are not great indicators of efficiency.
So that's why it was so smooth today... Great work!
Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.
This is better optimization than most enterprise devs will see in their lifetimes.
upvoting posts is so much more stable now, we might actually see more bean posts as a result
Upvotes are still getting rejected. Replies hang so I cancel out and it turns out they did post.
That said, browsing is pretty snappy and smooth. I know the kinks will get worked out eventually. Thanks for the update.
Edit: This now appears resolved minutes later. All smooth on my end.
Hmm. Seems to work for me.. (Yes this is a test reply)
Everyone itโs a test reply, deploy the upvotes
I'm going to take the opportunity to test a reply as well.
It's a good thing the points mean even less on Lemmy.
Iโve since successfully upvoted some comments and made replies without it hanging with the spinning circle. Not sure what the issue was but it all seems to be running smoothly now. Thanks.
I was getting upvote errors earlier, too, but now it seems smoothed out.
Iโve also had errors posting but the reply still goes through.
Test comment โฆ
Not getting the upvote or comment hang anymore so far.
This is why I love open source. The fact that a community can directly debug the code that's it's being hosted on and directly contribute the improvements back is just wild. Thanks for all the hard work @[email protected] and the rest of the lemmy.world team! The site already feels much more responsive.
The server is absofuckinglutely flying today! It feels smooth and bug free!!! You guys are legends.
As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.
EDIT: It looks like ![email protected] is a good place to subscribe to for more technical info on some of these performance improvements.
Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.
Yes please
Same, my job is like 80% SQL, so it'd be cool to see what is used in the background and maybe help improve things.
Me too!
Same here! If you find a way to contribute please give us an update!
Just edited my comment!
Good to see a heavy production server taking on the scaling issues. Thank you! To discuss Lemmy performance issues, there is a dedicated community: [email protected]
Appreciate that these updates use the yyyy-mm-dd format :D
It now feels pretty good to browse and it now makes the experience of using Lemmy much more enjoyable. Having to spam the vote buttons was really annoying.
The vote button change was also the one I immediately noticed!
Thanks to all involved across the board. Great work all around ๐๐
Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastodon has said today that they don't mind sleeping with vipers when their demise / dissolution is in Meta's best interest.
Please tell me we are defederating from Meta....or let us know what to expect
EDIT: I originally stated that Mastodon told them to fuck off, but I got confused with Fosstodon (who did that). Mastodon doesn't mind being in bed with Meta
Where have you seen Mastodon formally state they have no interest in working with them?
I'm genuinely asking because I'm relatively new to Mastodon and Lemmy and want to be as informed as possible with this whole Meta situation. And just to be transparent, I'm not a Meta fan at all, to the extent I've never had an account with any of their products
I did read this official Mastodon blog post today...
https://blog.joinmastodon.org/2023/07/what-to-know-about-threads/
and my question to you is because I'm not seeing as quite aggressive a stance as you mentioned.
It's so smooth now; the speed difference is insane! You all are doing excellent work!
Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.
Thanks for this very nice report.
Huge props to everyone working on the project. It's awesome seeing everyone work together and resolving issues so quickly!
Thanks for the detailed update and all the hard work you guys are doing!
Gadzooks! These are huge fixes. Compliments to the team, you guys pulled off a small miracle today.
Lemmy's devs and the .world admins have done in a month what Reddit hasn't done in it's whole existence: having a smooth and almost bug-free experience.
Jerboa feels so damn FRESH to use now!
Not to undervalue the efforts going into this, because I appreciate the new community and the transparency, but I believe we have wildly different definitions of 'almost bug free'
Which, is also something to consider about user experience consistency. Will be a challenge with growth. Fortunately, plugged in admins and devs will help.
By almost bug free I was speaking only about my own experience. There are probably loads of things under the hood I'm not noticing, but it's been hours since I last noticed any issue.
I agree that we've still got a long way to go though: both Lemmy and Jerboa are far from their 1.0 release yet.
I don't understand anything other than you worked diligently to make things smoother. Thanks to everyone for their wonderful work!
Same! My first thought was โthatโs an impressing looking graph. I have absolutely no idea what it means.โ The proof is in the pudding though - lemmy.world is much improved!
The site is running so much better now, thanks to all. BTW: Love these updates!
I'm very curious: does single Lemmy instance have the ability to horizontally scale to multiple machines? You can only get so big of a machine. You did mention a second container, so that would suggest that the Lemmy software is able to do so, but I'm curious if I'm reading that right.
Shouldn't the correct HTTP status code for a removed API be 410? 404 indicates the domain wasn't found or doesn't exist, 410 indicates a resource being removed
How great is it to be a part of history in the making -
This is Web 3 in its fomenting -
Headlines ~5yrs:
The ending of Web 2 was unceremonious and just ugly. u/spez and moron@musk watched as their social media networks signaled the end of Web 2 and slowly dissolved. Blu birdโs value disintegrated and Redditโs hopes for IPO did likewise. Twitter and Reddit dissolved into odorous flatulence as centralization fell apart to the worldโs benefit. Decentralized/federated social media such as Mastodon and Lemmy made their convoluted progress and led Web 3โs development and growthโฆ
This is how history is made, itโs ugly and convoluted but comes out sweeetโฆ
Awesome work - things seem to be running much more smoothly today.
Do you have anything behind CDN by chance? Looking at the lemmy.world IPs, the server appears to be hosted in Europe and web traffic goes directly there? IPv4 apparently seems to be resolving to a Finland-based address, and IPv6 apparently seems to be resolving to a Germany-based address.
If you put the site behind a CDN, it should significantly reduce your bandwidth requirements and greatly drop the number of requests that need to hit the origin server. CDNs would also make content load faster for people in other parts of the world. I'm in New Zealand, for example, and I'm seeing 300-350 ms latency to lemmy.world currently. If static content such as images could be served via CDN, that would make for a much snappier browsing experience.
Is it safe to use 2FA yet?
Whilst I'm aware that too many users on one instance can be a bad thing for the wider Fediverse, I think it is a great thing at the moment in terms of how well people are banding together to fix the issues being encountered from such a surge in users.
The issues being found on lemmy.world results in better lemmy instances for everyone and improves the whole Fediverse of lemmy instances.
I'm very impressed with how well things are being debugged under pressure, well done to all those involved ๐
I agree, it's great to see Lemmy being battle-tested in large instance scenarios.
You guys are absolute legends, thanks for the update!
That's so awesome! Look at that GRAPH!
I'd volunteer to be a technical troubleshooter - very familiar with docker/javascript/SQL, not super familiar with rust - but I'm sure yall also have an abundance of nerds to lend a hand.
Hey I can upvote now!
You guys are absolutely amazing. So many thanks to you @Ruud and the entire admin/troubleshooting team! Thank you.
It blows my mind with the amount of traffic you guys must be getting that you are only running one container and not running in a k8s cluster with multiple pods (or similar container orchestration system)
Edit: misread that a second was coming up, but still crazy that this doesnโt take some multi node cluster with multiple pods. Fucking awesome
smoooooooooth! Keep up the good work!
Is it weird that Iโm always excited to read the update posts?
Not at all. The admins here are doing great work and their updates are often informative and helpful, it makes sense you'd look forward to them.
Wow it is smooth as butter now. Great job ruud and team!
Installed Jerboa again and it feels smoother than Reddit itself, great job!
It reminds me somehow thw old Google+ in the mid 2010s
@ruud crazy impressive
Agreed
Things have been super smooth lately, thanks for all the work!
Yeah this morning everything has loaded so much quicker, Iโve been able to post and vote on comments no problem! Lemmy is really starting to take form. I fucking love this whole thing.
You know there's something about dealing with the lagginess in the past few days makes me appreciate the fast and responsive of the update. It nice to see the community grows and makes the experience at Lemmy feels authentic.
I took a SM break for a few days, and it's running noticeably better today...I think. (:
Thanks a bunch for floating us degenerates.
Awesome work. Any way other devs can contribute?
Go checkout the Github issues for Lemmy
Already on it sir
Just seeing if thereโs anything else
Keep tabs on ![email protected] as well if you're not already subscribed to it.
Thanks!
You're a good person.
Really great job, guys! I know from my experience in SRE that these types of debugs, monitoring and fixes can be much pain, so you have all my appreciation. I'm even determined to donate on Patreon if it's available
It felt like Iโd jinx us all if I commented but THANK YOU! This has been a wonderful experience today. Absolutely loving it and knew you just needed some time to work out the kinks that happen with fast growth.
You guys did an awesome job.
As a Performance Engineer myself, these are the kind of performance improvements I like to see. Those graphs look wonderful. Nice job to all.
It's smooth as butter today
A lot of this stuff is pretty opaque to me, but I still read through it because I love how detailed they are in sharing what's going on under the hood and how it relates to problems users experience. Kudos to you guys!
thankyouthankyouthankyouthankyouthankyou it's working very well today...
Love the transparency. Thanks to the entire team!
Love these updates! Love the transparency!
I love reading these! Thanks for all the work
I hope to start on some small contributions sometime next week. Stability has been noticeably better the last few days and I imagine itโs only going to get better.
I like that the post goes in detail and allows us tech nerds to get hard watching this stuff instead of the regular corpo jumbo change log that consists of:
Definitely noticeable!
Also, the amount of communication here is AWESOME! Thank you for keeping us updated.
Proving that orgs do need DBAs!
Great job guys! It really feels more responsive today
Thanks for the updates! Seeing the details of how you work through these early issues is valuable to those of us thinking of starting an instance.
Great job. Everything seems to be working smoothly for me now. The past several days have been a bit rough but now it's all working.
It works so well, that's very refreshing
Great work !!
Would HAProxy work better as a load balancer? For work we switched due to some issues with NGINX; so far, the service has been much more consistent with pretty much no downtime, even when restarting server hosts.
Sweet. Yeah it is definitely snappier today. Feels great.
Ty for the hard work <3
Awesome, thanks!
I agree that everything is a lot more stable!
Compared to days prior, things are running much better today. Page load speed, reliable post/reply/upvoting. I dunno if it's just happenstance but whatever knobs, levers and keystrokes you're manipulating, keep going the things! Thanks so much for a home absent of corporate BS.
Man I thought I noticed something different. For the past week or so I've gotten nothing but network error and Java errors in Jerboa which are completely gone now. Posts load almost instantly too. Appreciate the effort guys, was going insane.
Thanks for the update and thanks for the amazing collaboration youโre fostering!
I feel like Lemmy.World is leading the way for other instances to follow, and thatโs a good thing to do. Even if itโs painful for everyone involved.
Wow I applied these PRs on my server as well, running waaay lighter now. And it seems the federation misses have cleared up! Bravo Lemmy.world team!
It's definitely a lot better today. Great work.
The responsiveness of the website has felt drastically better today.
Now that's a satisfying couple of charts!
Everytime I open a post and go back to previous page it scrolls back to top. Is this fixable? Im on windows 11, chrome.
Things have been very noticably improving every day since the big update! I've had very, very few problems today browsing on Jerboa.
It is much smoother than it was previously. Thank you!
This was cool to read.
Amazing work, thank you so much!
Awesome, no more errors on my end! Thanks for the hard work, donated as a small token of thanks for the team here.
Seems a lot faster today - great work!
Well done guys!
ROFL Back in the day, when AOL was a thing - I used to think it stood for 'Assholes Online'.
Good luck (",) appreciate the efforts you make fighting the crazy deluge.
Was just noticing how much smoother it is this morning. Great work!
Ever since I joined Lemmy I had problems posting replies but that kink seems to have been ironed out. Thanks!
This update is a difference like night and day. Very impressive!
I'll be honest, I think I understood three words of your OP. Sounds like the general gist though is Woo so great ๐
Thanks for all your efforts, glad someone knows what they're doing
It's really fun to see these updates, and get insight into what you're putting into this instance. It's real obvious that it's a labor of love.
I'll be honest I don't know what any of this means but what I can say is I absolutely love the transparency of all of this. It's so refreshing and maybe I'll start learning more about what I'm looking at because I'll keep seeing it. Great work!
@[email protected] is this docker container y'all are using available on a registry? We'd like to use it. And do you have a load balancer in front of your lemmy-ui image to allow two containers to run? or is that built in and I just never noticed it?
Well we use the
cetra3/lemmy:the-phiresky-cutimage but you can also wait for the next RC or release which will have the PRs. We load balance with nginx which works if you use max_fails=5 for each upstream.Thank you for all your great work, and I really appreciate and enjoy the updates about the gritty details!
Thank you guys, you are doing an amazing job. These are great times for the internet and you are a huge part of it!
My god, it's fast right now. Don't touch anything.
Good job.
"Don't touch anything." lol
Awesome news. Thanks for all the hard work.
Thank you so much for the hard work, time and money you spend into making lemmy.world run very smoothly. This much transparency is awesome for something that's being used so massively.
Thank you to everyone that helped.
Thank you so much for all the efforts! So happy to be part of this community.
Please recommend people to update their app in a topic title. Connect couldn't even load a topic without failing out today. An update fixed it, but I had to manually force it because it didn't apply automatically.
This will drive people away. Literally none of the communities I subscribe to on world even seemed to have a new comment.
Itโs very much helping the third party apps as well. Memmy is running way smoother now
good vibes!! thank you for your work
That's a very phallic bandwidth image Dr Freud
Good job on troubleshooting!
Have you looked into possibly migrating to kubernetes or some other form of docker container management/orchestration system to help with automatic scaling and load balancing?
Don't quote me, but I recall reading on GitHub that there are a few things to be refactored before Lemmy can support horizontal scaling approaches.
Was waiting for this update. Woke up a couple hours ago and Lemmy was working great so I knew yโall put in some time. Thanks for your efforts!
Thank you for all the hard work and transparency as always! Everything is running perfectly knocks on wood
That's some pretty heroic shit right there.
You just took lemmy from something I'm willing to live with in the short term in hope it gets better, to something I am fully satisfied with.
Now let's grow so we can fuck it up all over again!!!
Thank you so much! I will be donating a few cappuccinos your way when my next check arrives. I really appreciate how awesome of a community youโve brought together & all of the transparency with the updates (and the frequency) is astounding! Keep up the great work but donโt forget to take breaks :)
The instance seems to be much better. Posting and commenting is not taking as long and loading times are way better. I hope things can stay this good or even get better.
Woo hoo, nice graphs! Monitoring data that shows big improvements is always fun to see.
Thanks for all the work you've put into this!
The new updates made a ton of improvements. Thank you very much. It is near ideal now.
Minor thing but over night both wefwef and Memmy clients are showing the wrong comment score (karma) against my profile, and given they are showing the same amount I assume itโs related to API fed data. Value was correct yesterday. Easy for me to confirm given I have only 2 dozen posts and the value has dropped to single digits.
Not a biggie, but figured Iโd report it in case there was some issue causing that. Might be some optimisation around indexing or something has intentionally or unintentionally impacted that.
Otherwise the service feels much more stable currently. No timeouts today where itโs been very frequent the past few days. Nice job. ๐
Things are looking a lot more stable, thank you (all) for the amazing work!
The changes are very much noticeable, thanks as always everyone!
LEGEND!
Great work! Awesome to see how fast the technical side of the Lemmyverse is evolving and improving!
Upvotes appear to be working but I still can't post images. The post displays the infinite spinning loading circle.
How large is the image (both in dimensions and filesize) ? What image type is it? Is it in a post, comment?
209KB, 720x891px, jpg. It was a post.
Do you see the separate line for uploading the image? If not, can you refresh the browser?
I see the separate line for uploading images.
damn bro, yโall coming in clutch to improve stability of this lemmy instance.
Good shit bros. Hope to contribute upstream and find more performance related bugs. I browsed the code for lemmy, and could not find any performance tests.
I can probably start there.
Awesome update, great to see and feel the progress from all the hard work!
Thanks for your efforts and improvements. Keep it going โฅ๏ธ๐
Fantastic job in troubleshooting and submitting PRs! Improvements are very noticeable already :)
Thanks for the update! Things seem way speedier now ^^
You all are champions today. When I loaded this up today, I was very happy and impressed. I knew tweaks were made.
Can confirm, having way less problem browsing lemmy right now. Thank you admins!
Things feel a ton better today. Thank you for all the hard work!
Seemed to be running a lot smoother, thank you Ruud!
Running so much smoother! You guys rock!
Thank you guys for all the hard work!!
Amezing improvement! Lemmy felt so much more responsive suddenly today, now it makes sense why :3
Everything is much faster and smoother today. Thanks devs!
am i amazed by you all guys work?!
i absolutely am
Good job! ๐ค
Really appreciate all the hard work going on behind the scenes! Feels night and day different after the changes. Also appreciate the transparency. Nice to see in this day and age.
Wow! Your commitment and diligence is admirable!
Thank you!
Stuff is pretty smooth now!
Absolutely great job! Everything feels buttery smooth today, and Iโm yet to see any errors.
Amazing work @[email protected] and admin team! Good to see some really awesome progress!
I can't imagine the amount of work Lemmy's devs and ITs are under since few days, but those are important for the future of Lemmy. Keep the good work! You're awesome!
Working great thanks for all you do!
Thank you so much for all your hard work!
Awesome work! I'm seeing a lot less errors and I can actively scroll through posts quickly now.
I'm in awe of how quickly these issues are being resolved.
Thank you for all the hard work.
It's definitely snappier than before. Constantly having to refresh got old quick.
Cheers!
Are you guys able to create reports? I am not.. It keeps spinning.
Looks really good, and have definitely noticed less issues just over the last couple of hours. Great job folks!
Thank you to all the code monkeys for getting things straightened out!
System appears to be working as intended
Thank you guys so much for all you do.
Noice!
Thank you for your work people. You are doing great.
Sweet!
Thanks for all your hard work. The change is absolutely noticeable.
That is so hot!
Thanks for sharing :).
Loving the transparency.
Not that it really matters, but did anyone else notice their comment/post points get deleted after this update? Seems odd.
What points?
Its not on the default lemmy-ui on the web, but many third party apps for lemmy are showing total comment and post โscoresโ on userโs profile pages. Not exactly sure how they are calculating that, but ive noticed that those scores were wiped for at least my account today. Again, not a big deal, just thought it was strange how that would happen. It is showing like this in memmy as well as wefwef so itโs definitely some data out in the lemmy verse that went bad. it is starting to โcountโ stuff from comments ive made today though so, who knows.
Hmm that's odd. @[email protected] do you know why score counts are wiped? I have no clue how they are calculated.
It looks like on the person_view counts object API response coming back from Lemmy.world, the amounts for scores are not right. Hmmm
It's great to see such transparent and informative updates on this.
Well done!
Thanks for the tremendous work!
Thanks for the hard work!
This is a lot of work, I genuinely feel honored by the service of everyone involved. Thank you!
Awesome job as always, @[email protected]
But these grey fonts on the sidebar will take some getting used to.
So much smoother! Thank you for all your charity and hard work!
I read this post after browsing for an hour and Iโve definitely felt a change. Great work!
Amazing results! Thank you for all of your hard work. The collaboration among the admins is inspiring.
I tried to enable push notifications and the app crashed. Also, how do I create a community?
The 502 error still seems to be common by me. Vut they are less permanent. Before it stayed multiple refreshes now it is safe to say after 1 reload it is most of the time gone.
Everything is feeling great so far. The only bug I'm encountering is that when opening a thread (in Firefox on desktop) it auto-scrolls down past the content to the replies.
Thanks admins, things feel much smoother now. ๐
I love how open you guys are when it comes to running the server! Much nicer than lemmy.ml imo
Great work!
Yay, I can finally log in on Chrome!
Building history one update at a time ๐
Thank god, wasn't able to use the site lol
Can you describe your setup? Single instance or multi-node setup behind load balancer?
Sounds like we need to comment more to me. Great work all around!
You guys are amazing!
You guys are doing god's work. Thank you thank you thank you...
Amazing guys. I did donate so you can keep up with things. I really want Lemmy to last.
Awesome. Great work all!
Huge thank you for all your hard work^^
Loading is faster, thank you guys ๐
Glorious, Ruud!
I think some of the empty page issues are back now?