lemmy.ml is migrated to a new server
Its now running on a dedicated server with 6 cores/12 threads and 32 gb ram. I hope this will be enough for the near future. Nevertheless, new users should still prefer to signup on other instances.
This server is financed from donations to the Lemmy project. If you want to support it, please consider donating.
440
Comments145
Everyone liked that.
As a new user, thanks for being accepting of us migrants!
Please know that your work is genuinely appreciated in fascilitating the migration from Reddit to Lemmy. Your efforts will hopefully ensure a bright future for communities on this platform. Kudos @[email protected] !
Thank you :)
Upgrade unlocked!
Thank you so much for all the hard work, I'm really loving it here.
Youre welcome :)
⭐ lemmy star award! I agree, I'm enjoying my time jere much more than at reddit.
I just want to say, you all are doing a great job. Maybe I don’t fully understand it yet, but I have created some communities on the lemmy.ml instance and I don’t really want to move away. Is there anything I can or should do? I think that I’m currently locked in here.
I'm getting 502 error all the time on my lemmy.ml account (lemmygrad account works since i'm posting from it).
I was getting errors until I deleted my cookies. Try that and see if it works.
It worked, thanks.
Same. This is why I had to switch to lemmy.world. Now it's time to subscribe to everything all over again...
The server has become more responsive definitely. I thought my internet routing was so shitty that it took so long to load the site. Nice!
Is it possible to horizontally scale these instances instead of just upping the machine hardware? What are the main performance bottlenecks typically?
Hey, what do you mean by "scale horizontally"? There are multiple approaches to tackle this.
All of these requiere most likely a major rewrite/change of Lemmy server software I guess. They are already addressed as issues/feature requests on github In my opinion the first option would fit the most.
My comment was without knowing the topology of Lemmy at all, but my thoughts were initially that vertically scaling can have diminishing returns past a certain threshold. Since the servers seem to be struggling I'm wondering if that has been surpassed and if it's more cost-effective and reliable to scale this way? But if the application isn't written in that way, or the underlying data store isn't equipped for multiple instances then fair enough, I'd be interested as to why especially if Lemmy grows. I'll take a look at open issues and educate myself a bit more though.
Just started supporting this instance on liberapay, if other follow you'll hopefully be able to upgrade the potato soon !
I'm still amazed at how light-weight lemmy is.
Rust magic
Inferno is pretty slick too
Yeah I mean a single dedicated server doesn't sound like much, and yet performance is much better than before!
And it's not even a super beefy one!
kbin.social uses a CCX41 with 16 virtual cores (I think 8 of those are hyperthreaded though) and 64gb of ram for its ~25k users
@nutomic @liaizon 502 Bad Gateway ?
I was getting 502 Bad Gateway. When I pinged Lemmy.ml I got an IPV6 address. It disabled IPV6 on my local computer and now when I ping I get a IPV4 IP address it works now.
I am wondering if DNS is screwed up on the IPV6 network for Lemmy.ml.
~~Note. This could totally be something on my end, I really haven't done much with IPV6 but it did solve the 502 Error so I might do the same for you. ~~
Edit. Multiple people are reporting the same thing I am seeing. It is defiantly something about IPV6 on the lemmy.ml server end.
Thanks for mentioning the IPv6, I've been banging my head all day trying to figure out why I kept getting the 502 yet no one was complaining anywhere and isitdown was showing the server as Up.
I forces my DNS to resolve only IPv4 for lemmy.ml and now I can use it.
My suspicion is that nginx is misconfigured and not listening via IPv6. Or maybe the AAAA record is pointing to the wrong IPv6 address.
@nutomic Thanks for upgrading the server!
My money in on the AAAA record because "it is always DNS".
You are right, I forgot to configure IPv6. Will be fixed shortly.
Edit: Should be fixed now.
it works, thanks
I'm also unable to connect with IPV6 enabled getting the 502 error, but able to connect with it disabled
@[email protected] pls fix this. I'm not turning off IPv6 on my end.
edit: it works now :)
same its been 6 hours. seems like using a vpn to connect works.
Which country works for you on VPN? Doesn't work for me with or without VPN
it seems this new server is not configured to work with iCloud's Private Relay service as the last one was
how to configure servers to work with private relay: https://developer.apple.com/support/prepare-your-network-for-icloud-private-relay/
Me and others are getting that issue without that. Seems to be related to its IPV6 configuration or AAAA record since disabling IPV6 seems to make lemmy.ml work
How to migrate an account to another instance without losing anything including relationships ?
This is what I need to know. I also have an account on Lemmy world as a backup
I was wondering the same thing. I initially created an account on some obscure instance because I didn't fully understand what I was doing. I just abandoned it and set up a new one on lemmy world. I think I'm getting the hang of it now. I'm curious to see how Lemmy grows and matures over time. There is still a learning curve that will keep some people away.
Based
You're holding the gates of freedom open, my friend
Congrats on the smooth migration!
Are there still issues with cross-instance content? Previously, if I signed in to lemmy.ca and subscribed to a channel on .ml, I didn't see all content. Likewise, if I left a comment from my .ca account, it wouldn't necessarily show up for users on .ml.
If this is still a problem, it's a HUGE roadblock in being able to just tell people to join other instances, if we don't want to fracture an existing community.
Edit: That may have simply been a result of the excess load?
Edit 2: Okay I'm testing and it looks like this was just an issue of load; I don't seem to be having any further issues with this.
Edit 3: Comments from users on lemmy.world still don't seem to propagate correctly. I can see them from .ml but not .ca
This new machine is speedy! Getting pretty much instant loading times. Thank you to the donators, I will be joining you soon!
I really do appreciate everything you're doing for us all. Thank you.
Have we explored the possibility of "porting" the larger communities to other instances? It seems that many of us simply wish to subscribe to the largest (insert type of community here) and can do so from various home instances. Might lower demand on this specific instance at the very least.
Is that why I kept getting this error?
Same..not sure if it's caused by jerboa as it loads just fine on browser.
Probably hit the limit again and need a bigger server. It was migrated yesterday
Is most of the discussion about the infrastructure done in a community here, or do you folks have an invite only discord/slack or something @[email protected]?
I've been working in devops for 4 years now, but with mostly kubernetes and AWS, but I'd like to throw some ideas and just ask some more specific questions about the infrastructure. Like I'm curious why autoscaling the web servers and moving the database server to a dedicated instance is not the current configuration.
There is really not much to discuss. The server was overloaded so we got a bigger one. And there is no reason to mess with stuff like kubernetes when a single server works fine. After all our job is to improve the Lemmy software for everyone, not build a huge centralized platform only on lemmy.ml.
I agree with the kubernetes point. Not every problem needs it.
I was more curious about having a load balancer infront of an autoscaling group of servers with the existing images that you are running off of, minus the self hosted database server. Then you would be able to handle spikes automatically. Just curious if that has been thought of.
Having it accessible to everyone is great. For sure. But I was just thinking that having stronger pieces of infrastructure for instances that handle more traffic might be beneficial.
So far it's not necessary and there are more urgent problems to fix.
I agree, lots of application improvements and bugs are probably your priority. That should always be the focus of the application developers. I honestly want to try and experiment with this idea now.
If I get a good repository going that focuses just on the infrastructure side of things, I will post it here.
Exactly. lemmy.ml is lemmy.ml - just a single instance. there should be incentive for other to stand up instances of their own. overload on any instance is good reason for ppl to get out there and build new instances.
Hi @[email protected]. Thanks for the upgrade and for your work.
May I ask about the resources utilization now? CPU, RAM, storage? Thanks!
Load average is around 4, ram and storage are also rather low. So there are plenty of reserves for now.
If that's the load average now on a 6c/12t system with 32GB, just how bad was the previous server?!?
Was this website running on a Raspberry Pi or something?
Was running on a potato
Thanks. Thinking about hosting a Czech instance for lemmy, that's why I am asking
Add an instance migration feature, and
throw outkindly suggest it to some users when the server gets overloaded... (/jk... or not)Thank you! I can actually participate now!! I have several screen shots saved of my profile showing me logged in under random user names over the last few days.
.Shit was weird. I couldn't post, couldn't stay logged in, or I'd see a strange profile name if it actually let me click into a thread. Everything seems to be solid today.That was a software bug that was fixed with 0.17.4
I can feel the difference already. Really enjoying it here and while I wouldn’t let some technical difficulties stop me, I think it will help grow the community. Thank you!
@nutomic Sounds awesome!
Was wondering why it was offline.
Is there a way to migrate my account to a new server?
hi! I would be interested in understaning more the setup of the lemmy.ml instance. Do you use a cloud provider, a SaaS platform or a traditional hosting ? What are the costs that are incurred? Cheers!
It's a single, dedicated server from ovh.com for 60 euros per month.
Is Lemmy made with horizontal scaling (a.k.a. launching more instances and have a load balancer proxy the requests to the various instances) in mind? It could help larger instances like lemmy.ml managing the load better rather than just putting it on a beefier machine.
You should be able to do that without problems. However the main bottleneck is the database, I think some people want to experiment with read replicas. However as developers we would rather focus on optimizations which will benefit everyone, not only the largest instances.
Ah awesome. The database horizontal scaling is a solved problem already luckily, especially an enterprise database like PostgreSQL has lots of options there.
Oh sure, but being able to horizontally scale shouldn't hurt small instances 😉 Personally I'd probably host a single-user instance at some point just like I do with Mastodon, so I personally don't really have a need for horizontal scaling either but it's good to think of those things.
See matrix DMs.
Congrats! I was hoping that's what was going on!!❤️
Something isn't working with ipv6, on my phone network I get nginx 502 but on wifi it works
It's giving me a 502 error on my computer as well but is working on my phone using WiFi.
@v_krishna @nutomic not working on my phone either. Same error
@nutomic likely https://support.us.ovhcloud.com/hc/en-us/articles/4412804434195-How-to-Configure-IPv6-on-a-Dedicated-Server
Thanks for your work on this site. It feels very promising and a great place to migrate to. Reddit is about to hit their digg/myspace moment.
Thank you for your hard work! Although I kinda foresee for the future if Lemmy really would become the new "reddit" with such servers and millions of users, wouldn't that also rise the server costs and ultimately make the hosts dependent on asking money for it, maybe by a paywall or by ads? I think to make this community really be "free" without any host responsible for spending a huge amount of money for servers, the best solution would be to make the actual "servers" be a p2p cluster. Unfortunately I'm not quite sure how to realize that without losing a huge fraction of the model if a lot of nodes (i.e., the actual users) are offline. Sorry, I'm just brainstorming.
I would love a social network powered by the users that are using it. Maybe also running something like serverless functions on the client devices.
You might want to check the Earthstar project: https://earthstar-project.org/ They are working on that. Right now, it's super early; they are building the foundations. Peer-to-peer is unfortunately much more difficult to code then servers, because less people have built the building-blocks required, and because mobile phones are actively making it hard to run peer-to-peer apps.
I'm still having trouble that any comments I make on other instances through lemmy.ml are not seen on those instances, only when viewed through lemmy.ml
I've donated on https://opencollective.com/lemmy to support this :)
This is excellent news.
I'm a little curious how difficult it would be to include this status in the sidebar - we see 'users online' statistics, we should also see 'costs vs donations' statistics there plus a link for donations also.
Fantastic to see and the improvements are already being felt and very obvious.
Thank you!
Gotta say it's been a smoother experience using it in the last couple hours. Thanks!
Thank you!!
Thank you for your hard work on this site!
First! 😂
I kept refreshing and was wondering why it was down. It was showing an nginx error message or sometimes a certificate issue.
Really glad we got a new beefy server though. 🤞
This server now: I am speed
Thanks for your hard work, I really can tell the difference. Now lemmy.ml is much more responsive than before
Are the json objects related to the migration? Btw, thanks for your hard work!
Thanks! Donated 5$ - I encourage others to do the same.
Great news, and thanks for the upgrade, it is flying now. Would there be need from any devopsy work to contribute, I'd like to help.
Sexy
Can really feel the difference in speed. Thank you!
Noticeably faster! Thank you.
ive noticed already that it is much faster and responsive now it must be this new server
JFYI, it's getting an error 500 here and there when refreshing and sometimes it misses loading the css. No idea if you are using a cache somewhere, could be that as well if you don't see anything weird in the monitoring.
that's excellent! Thank you and team for all the hard work.
No time but you can do it yourself.
Are you saying beehaw does not federate with lemmy.ml? Maybe I am misunderstanding something here - I can definitely see lemmy.ml subscriptions on beehaw.
I think they were talking about lemmygrad
This sounds wonderful!
Great to hear! It's already a lot smoother than it was before! Hopefully it will continue to grow!
hey 👋 do you have some Grafana dashboard where we can check user increase?
Given the no. of posts on this instance I guess it has quite a large user base and I’m really surprised how it manages to run on that amount of resources 😄
Cool. Which provider, if you want to share?
Ovh
You feel safe that they won't take a dim look on piracy then?
No they dont seem to care at all.
very interesting. Thanks for letting me know.
I haven't dug into the bowels of lemmy, but is there anything to be gained by scaling horizontally?
Probably, but it's significantly more difficult to set up and maintain, and introduces new problems. I'm sure it'll be considered once they reach the limits of what a single node can handle.
I'm fully aware of the operational challenges involved. What I have not looked into (yet) is whether it is even feasible.
Honestly, the operational side doesn't worry me, I can do that in my sleep. Application level issues I am powerless to solve however.
I assume it's for the sake of more freedom. Without admins to make troubling choices like in the case of reddit, lemmy can't be corrupt so easily. I'm quite new to this as well, but from what I've seen, the most lemmy can do is unlink certain instances so they don't show up in your search, but instances may live or die, but the social network as a whole lives on (except if the killed instance housed your account).
I think you have misunderstood my question because I have no idea what you are talking about.
Yeah that was my bad, I thought you meant horizontal scaling utilizing multiple instances.
They meant horizontally scaling this instance (multiple servers serving the same app)
Oh, my bad. Thanks for the clarification.
Liberapay has a manual renewal option. I'd go with doing your amount you want to donate there and just ignore the renewal email.
love it. thanks for everything
I tried to run my own instance but ran into errors. Gonna try again some other time.
@nutomic thanks, but I'm still getting 502 bad gateway
Works for me. Try force reloading the page, or maybe DNS isn't updated for you yet?
@nutomic hey, just fyi, still having the issue. 502 bad gateway. Someone further down I'm the conversation that it may be IPV6 related? Not sure
@nutomic yeah, force reloading didn't work. I'll wait for dns propagation. Thanks for the confirmation though!
Noice. Now it is fast!
neat, separate hosts for lemmy.ml and lemmygrad.ml :D
They were already hosted in completely different places though??
The devops understander has logged on