Hi all,

First off, I want to apologize for all the server instability. We long ago outgrew our instance size, but I was unable to afford a larger node on our provider, Vultr. We were maxing out every part of the server whenever any even slightly significant number of users were on the fediverse.

I've finally found the time to migrate us to a new provider, which allows us to step up to a much more powerful configuration. That migration has now been completed. I actually intended to post about the downtime on this community this morning before beginning, but when I went to do so, the server was already down and struggling to come back up. So I went ahead with the migration.

Server before 4cpu/16GB/400GB NVMe Server after 8cpu/64GB/1Tb NVMe

Please update this thread if you are seeing any issues around any part of the site. This means duplicate threads, things that aren't federating, inability to load profiles, etc.

There is still database tuning that needs to occur, so you should expect some downtime here and there, but otherwise the instance should be much more stable from now on.

During this process I also improved several other aspects of operating the server, so any 'actual' downtime should be accompanied by proper maintenance pages (that hopefully don't get wiped by ansible anymore), so that will also be a good indicator of legitimate maintenance.

Once again, I really apologize for all of the downtime. It's very frustrating to use a server that operates like this, I understand.

snowe

We are going to be upgrading to 0.19.5 at 3:00 UTC on Thursday Oct 3 (10 minutes from now).

Downtime is expected to be about an hour. Hopefully it is not more.

I will no longer be able to assist with development nor debugging actual issues with the software... Quite juvenile behavior from the devs. It stemmed from this issue where the devs continuously argued in public by opening and closing an issue. Anyway, thought I would keep y'all apprised of the situation, since these are the people maintaining the software you are currently using.

Over the weekend we had a large intermittent outage, followed up by unplanned maintenance that I had put off for way too long.

Lemmy runs with several different services.

• lemmy-ui (the reactesque frontend)
• lemmy (the rust backend)
• postgres (the data store for operations, comments, posts, etc)
• pictrs (the image data store)

The outage concerns itself with the last one. We always knew we'd eventually need to migrate to an object based store, but Lemmy defaults to file based picture storage and that's what we stuck with up until now. This eventually caused the VPS that programming.dev is running on to seize up, and resulted in the outage over the weekend.

Saturday night I spent several hours testing out the object migration on the beta.programming.dev site in order to validate that it worked. During this time I struggled with some very obtuse ansible errors that I hadn't encountered before and so I was not able to start the migration that night. I delayed until the next morning (thank goodness).

I began work Sunday morning at 10:00 America/Denver time. Initially the migration started off quite well, but was moving incredibly slowly. Looking back on it now, the migration would have taken over 144 hours if I left it to do its thing. I let this run for about an hour before messaging the pictrs dev to understand why logs weren't showing up for the migration (even though objects were showing up in the store). Apparently lemmy-ansible is set to use 0.4.0 of pictrs, which not only is quite old, but doesn't have the ability to run migrations concurrently. There was the issue. I asked the dev is it was possible to stop a migration in the middle of the running, upgrade, and continue. They told me what changes I'd need to make, I made them, did the upgrade, and restarted the migration. It immediately failed. This was the start of my issues.

The server was now too full of data to do anything, including running apt update or apt install to install tools to assist me. I was able to attach more block storage, but I'm not enough of a linux guru to figure out how to mount it where the current pictrs filesystem would be able to take advantage of it. I had to result to copying the entire pictrs filesystem to a fresh ~500gb mount, fixing permissions, and then rerunning the migration from there. By the time I got to this point, it was about 12:30PM. The migration from then on took several hours.

After the migration completed, I needed to deploy the new stack with the correct settings. The ansible script needed to run apt though, and, well, that wouldn't work when the server was still full. At this point I was not confident in the migration and I also hadn't realized that you could do the migration while the site was running (big oversight from me). I therefore wanted to maintain the entire pictrs file store until I proved the object store was working. I created another block storage, copied the entire pictrs directory over to it again (another 20 minutes or so) and then deleted the original directory. I was now able to run the ansible script and deploy the new settings for pictrs, confident that I had a backup available in case something went wrong (this is not the main backup method, the server is backed up externally as well, but I didn't want to have to resort to those during the migration).

That completed the migration, some 5 hours after it originally started.

There were several things that exacerbated the issue that made it take several hours longer than I wanted.

1. I let it go so long before doing the migration to object storage that the server was too full to even perform an apt update. This resulted in me not being able to install tools I needed, along with a host of other issues as mentioned
2. pict-rs was at a very suboptimal version. If it had just been two minor versions newer it would have migrated perfectly fine, in a few hours.
3. my limited knowledge around ansible led me on wild goose chases several times

Things I would change if I had to do it again:

1. Dig in a bit deeper on the concurrency flag in the pictrs docs. It was not present in the original guide I followed (from a lemmy post on another instance), and thus I didn't realize that it wouldn't run with concurrency at all.
2. Don't wait so long so that the server is full
3. Migrate while the server is running. That would have been dumb in this case, since the server wouldn't stay up anyway, and could have caused other issues. But there was no reason to take the server down if it had been stable, and other instances have done so with no problems.

It seems like the password limit is set to 60 characters so I’m unable to login to my instance. There probably should be no limit in the app because each server could have different limits set.

There's gods for everything, but of course computers didn't exist in ancient Roman and Greek times. What God or Goddess in your opinion would personify Testing?

And yes these answers matter. 😬

Start by reading these two articles:

• https://blog.joinmastodon.org/2023/07/what-to-know-about-threads/
• https://ploum.net/2023-06-23-how-to-kill-decentralised-networks.html

Ok, now that you've done that (hopefully in the order I posted them), I can begin.

I have always been a strong supporter of Open Source Software (OSS), so much so that all of my projects (yes all) are OSS and fully open for anyone to use. And with that, I knew that things could be used for good... and bad. I took that risk. But I also made sure to build stuff that wasn't, in itself, inherently bad. I didn't build anything unethical to my eyes (I understand the nuance here).

But I've seen what unethical devs can do.

Just take a look at those implementing the ModFascismBot for Reddit (that's not its name, but that's what it is). That is an incredibly unethical thing to build. Not because it's a private company controlling what they want their site to do, no, that's fine by me. Reddit can do whatever they want. But because it's an attempt to lie about reality, to force users to do something through manipulation not through honesty. Even subreddits that voted overwhelmingly to shut down still got messaged by the bot telling them that the users (that voted for it) didn't want it and they had to open back up or they would be removed from mod position. This is not ethical. This is not right. This is not what the internet is about.

Or the unethical devs at Twitter, who:

• built in actual keywords to mark Ukraine news as misinformation
• marked Substack as unsafe when they released their own Twitter competitor
• banned Mastodon links for no reason besides the fact that they are a competitor
• marked NPR, BBC, CBC, and PBS as 'state-funded media' even though that is clearly meant to indicate something along the lines of Russia's propaganda arm RT or China Daily, the same for China. Then when there was enormous backlash, they removed the labels, but did the same for the actual government propaganda accounts like RT and China Daily. And then they removed the limits on virality of those posts from those propaganda accounts allowing them to have a massive spike in engagement, thus furthering misinformation.

It's one thing for an organization to have political lean...that is just a part of life, and that will never end. It's another to actually sow disinformation in order to accomplish nefarious things to further your profits. It is what has caused massive addiction to tobacco, the continuation of climate change, death and disfiguration from forever chemicals, ovarian cancer and mesothelioma from undisclosed exposure to asbestos, or selling 'health products' that claim to cure everything under the sun, but can "interfere with clinical lab tests, such as those used to diagnose heart attacks".

Please do not confuse this for saying that companies shouldn't be able to sell things and make a profit. If you want to sell someone something that kills them if they misuse it and you market it as such, you go for it. That's literally how every product in the cleaning aisle of your grocery store works. That's how guns work, that's how fertilizers work, that's why we have labels. But manipulation for profit is unethical, and that's why companies hide it. It hurts their bottom line. They know that their products will not be used if they reveal the truth. Instead of doing something good for humanity, they choose the subterfuge. Profits over people. Profits over Earth honestly. Profits over continuing the human race. Absolutely nothing matters to companies like this. And unethical developers enable this.

Facebook (ok, fine, Meta, still going to refer to them as FB though) is trying to join the Fediverse. We as a community, but honestly each of you as individuals, have a decision to make. Do they stay or do they go? Let's put some information on the table.

Facebook...

• lies about the amount of misinformation it removes ¹
• increased censorship of 'anti-state' posts ^{1 2 3}
• lied to Congress about social networks polarizing people, while FB's own researchers found that they do ²
• attempted to attract preteens to the platform (huh, wonder where all that "you must be 13" stuff went) ⁴
• rewards outrage and discord ³⁵

Facebook also...

• Allows for checking on friends and family in disasters ⁶
• Created and maintained some of the most popular open source software on the planet (including the software that runs the interface you're looking at right now) ⁷⁸

From my perspective... There's not much good about FB. It has single handedly caused the deaths of tens of thousands of people across the planet, if not hundreds of thousands. It continually makes people angrier and angrier. It's a launching pad for scammers, thieves, malevolent malefactors, manipulators, dictators, to push their conquests onto the world through manipulation, lies, tricks, and deceit. Its algorithms foster an echo chamber effect, exacerbating division and animosity, making civil discourse and mutual understanding all but impossible. Instead of being a platform for connection, it often serves as a catalyst for discord and misinformation. FB's propensity for prioritizing user engagement over factual accuracy has resulted in a global maelstrom of confusion and mistrust. Innocent minds are drawn into this vortex, manipulated by fear and falsehoods, consequently promoting harmful actions and beliefs. Despite its potential to be a tool for good, it is more frequently wielded as a weapon, sharpened by unscrupulous entities exploiting its vast reach and influence. The promise of a globally connected community seems to be overshadowed by its darker realities.

As a person, I believe that we need to choose things as a community. I do not believe in the 'BDFL'...the Benevolent Dictator For Life. Graydon Hoare, creator of Rust, wrote an article just recently about how things would have been different if they had stayed BDFL of Rust. From my position the BDFLs we currently have on this planet really suck. Not just politically, but even in tech. I don't think that path is good for society. It might work in specific circumstances, but it usually fails, and when it does, people get hurt. Badly.

So, with that in mind, I've been working on a polling feature for Lemmy. I seriously doubt I'll be done with it soon, but hopefully FB takes a while longer to implement federation. I understand there's a desire for me, or the other admins to just make a decision, but I really don't like doing that. If it comes down to it, I will implement defederation to start with, but I will still be holding a vote as soon as I can get this damn feature done.

I will be updating the instance to v18 at 20:00 22:00 UTC.

See https://programming.dev/post/181191 for the changes

edit: Lemmy.ml updated and seems to have gone down. We're going to wait and see what the outage was caused by and then proceed from there.

edit 2: lemmy was down due to a ddos attack. We will upgrade at 22:00 UTC

edit 3: we had issues with the email setup getting overridden again. If you tried to sign up in the past 8 hours please try to just log in. If you can't, please message me (discord, matrix, or mastodon)

I'm going to be working on getting the instance upgraded to 17.4 today. I had tried in the past, but had some issues with cloudflare and dns resolution.

First attempt will be at 17:00 UTC.

I'll update this post if that doesn't work and provide a second time.

We're seeing increased rates of memory errors in postgres, due to a default docker setting

I will be performing a fix at 23:45 UTC. This should not update the server, so it should be a very fast fix. I expect the downtime to be less than a minute.

Shopify wrote a new hand-written recursive descent parser. This looks like it will be a great improvement to the Ruby ecosystem!

At 6:49 Denver/America time today I migrated the DNS nameservers to Cloudflare. This propogated quickly, but inadvertently I had set the SSL/TLS Encryption mode to Flexible, which resulted in Cloudflare attempting to encrypt traffic between itself and the server. But programming.dev already has its own certificate. Cloudflare expects http traffic to come from the origin server, not https, so when it received https it simply tried over and over again, resulting in failure to connect.

Switching the SSL/TLS setting to Full (Strict) fixed the issue. Sorry about that everyone! I'll try to not break stuff that badly in the future.

I will be taking the server offline for an upgrade in 35 minutes at 4:00 UTC

Hi all,

Thank you for joining me here! It's great to see that we have a community that wants to grow in such a new and exciting manner.

As it is, I thought it would be a lot easier to do this by myself than it has been. So I'm asking for some help!

I have several things I need assistance with:

• setting up and moderating a chat community, for those times when users are having issues with the website. I think it's up to the community what software we use, but I would probably prefer Discord. Since this is all federation though I completely understand if others want to use something like Zulip or Matrix! So let's just use what everyone wants. If you have an opinion please post below.
• database stuff. I'm absolutely terrible at database stuff, and that is not an exaggeration. If anyone is willing to help it would be much appreciated. Currently I have a need to set up pgbouncer, or we should modify the lemmy source to allow for setting up a bouncer. I also want to set up read replicas so that we can distribute the load a bit more evenly. As it currently is, the site was simply set up with the lemmy-ansible script, so everything is running on a single box 😬. If you know Rust and want to help modify the Lemmy source code for this, or you are a Database Admin and want to help, I'd very much appreciate it.
• instance admins. I cannot be online constantly and I do have a day job. I'm getting messages and applications to join the instance along with needing to set up new communities, create and update rules, moderate, etc. I cannot handle this all alone.
• I also need some general help.
  • email admin
  • migration of server to larger VPS (will have to bring the entire site down for this, unless someone wants to help set up a load balancer, a brand new box, and have some sort of migration strategy.)

If you want to help out on the server side of things I will want to know your real life persona, but for instance admins, chat mods, etc. I would just like to see some sort of comment history from you elsewhere.

And thank you once again, for helping create an inclusive community.