Importantly, I want to issue an apology

Way I see it, family and mental health always comes before internet randos. Thanks for working hard for everyone.

I will try and reply to each comment - but you've all been really kind and that means so much ❤️

If you're interested, this graph will show you how far behind we are. We should eventually catch up, but things will likely be very delayed for up to 12 hours.

The status page did not work as expected - and I'll try and link a few more places where I post updates. If you haven't yet, definitely join the matrix space and you'll get minute by minute panic updates 🫠

I've been there. But it is my honor to bestow upon you this award to commemorate the accomplishment

Thanks for all your hard work. We missed .zip while it was gone.

Thanks for the update.

I was a bit worried for your mental health as the hours of downtime continued :)

Awesome that the backup restore procedure work that well.

One thing I have been wondering is, why status.lemmy.zip stayed all green during all of this.

entirely my fault and frankly unacceptable.

You're providing a service out of your own time, pocket and energy; you don't owe anyone.
It's the other way around, we owe you.

So thank you.
Learn from your mistakes and carry on. 👍

Dude, you're being wayyyy to harsh on yourself!
You run this awesome instance for free while caring for a newborn, you don't owe anybody nothing.
Forget the delay, forget apologies and "unacceptable". Real life comes before social media, don't beat yourself up for the outage.

People who can't stand downtime should practice personal redundancy by creating backup accounts on other instances ;)

Thank you for this post. Don't be so harsh on yourself, everyone can make a mistake!

Good to see Lemmy.zip back up!

Dude, keeping this running with a job and a newborn? You're headed for sainthood.

If you don't have one, you could start an out of band chat during updates, just in case you need some eyes on things or just some moral support. I'm sure we have at least a few subject matter experts around if you can stand us :)

No need to apologize as you have been doing a stellar job. Your family needs to always take priority no matter what. I don't care if it is down for a week as your health and kid are far more important.

One thing I will say is that I think Lemmy.zip could really benefit from a external way of communicating announcements. It doesn't need to be complicated and you could reuse your existing mastodon account to post updates when things go wrong. It also could allow for users to give advise on how to fix issues.

Things gotta fuck up sometimes, tis how we figure shit out and learn things! You got this.

No worries! Make sure you're getting enough rest!

As a new parent myself, I'm stunned you managed to find the time to restore it at all. Good on ya, fella!

Thanks for the update! I figured it must have had something to do with the baby and the busy life + the update not working as expected, so I was patient.

After 12hrs or so I did go on mastodon to look for an update (just a 'everything crashed, working on the backup' kinda message) so if this ever happens again that might be an idea?

Thanks for working so hard on getting everything back up and don't forget to rest!

Thank you for all your hard work, as an IT guy I know the feeling when production doesn't work as it should, and the feeling of relief when the backups are actually being restored and working.

Take care and make sure to take a break if you need to, we'll still be here.

I appreciate the transparency and frankly couldn't ask for more. Shit happens and this is a one-person operation. Thanks for all your effort!

thanks for the effort and also explanation

Appreciate the honesty and transparency. Thank you for your hard work maintaining the site, and hopefully you're able to restore everything to a fully working state!

Don't worry. Newborn is a trump card, but even without it you literally are a volunteer.

Anyone complaining about you volunteering your time esp with a newborn, is not a parent..but ignoring that you're doing this for free. Thanks for your time, effort, and just happy it's back up :).

Social media that can go down for a day or two is way better than a shit hole of advertising and manipulation that is Facebook, reddit, and all the rest.

Hey you’ve done a ton for all of us and I can’t thank you enough for the work and dedication. Don’t be too hard on yourself, your child and well being are both important, it’s fine. I’d rather some downtime than losing you as admin. Pretty sure most on the sever would agree.

We appreciate you very much! Take all the rest you need. 🫡

Thank you for the transparent announcement, and don't sweat it!

Hard agree with all of the other comments here. No apologies needed, you do a great job of keeping this instance going and the transparency is appreciated.

I temporarily switched over to an alt account and was back browsing Lemmy after figuring out .zip was offline, absolutely no big deal.

Hey man, you've got absolutely nothing to worry about. The fact that you have this service for us at all is quite frankly amazing and we thank you for it. As another commenter said below, I'd rather have a day worth of downtime than to be on big corporate social media and have everything fixed quicker. Because I know that I'm not the product here. When it did not come back, I checked the status page and it said it was working. So I just figured something broke and decided I'd wait until it came back.

Actually... disregard everything I said above. I'm so fucking mad right now. I could bite holes in bricks. I mean, how dare you notice that there's a problem and not get it fixed absolutely immediately. /s

Don't fret about it, things happen. You run a great service for us. A job and a new family already add up to being more than two full-time obligations. Managing Zip along with all that is a lot. Thanks for doing it.

Trial by fire. At least it was interesting(!)

Praise be to the backup strategy 🙂

How dare you interrupt my ability to look at memes and see the same news article posted in 17 places at once!

Jokes aside I appreciate the work y'all do to keep this sorta thing running without any pay or thanks for the most part.

I am greatful.

We so back

Thanks for your hard work! Remember your mental health always has priority though. Cheers mate.

I think you handled it very well. Not sure how it could’ve been handled better tbh. I figured something didn’t go as planned and I didn’t have any problems waiting for you to find a solution. No apologies needed.

I didn't realize it was a single person keeping it all running! Tech sometimes goes wonky, good job getting it back online!

Been there, done that, with my Friendica instance. 2 days of downtime while rebuilding a corrupted database, while people are tapping their feet waiting for all to return. I'm with you in spirit, my friend.

Thanks for all your hard work keeping the dream alive! And for keeping good backups

You're good bro. Sort of assumed something went wrong with the upgrade.

No worries. You’re doing a great job even though things are hard from time to time.

Thanks for your efforts. ❤️

Thank you for this post. Don't be too harsh on yourself, everyone can make a mistake!

Glad to have lemmy.zip up and running again!

Unfortunately these things can and do happen. I'm glad you were able to get things functional with a restoration. Best of luck troubleshooting and repairing the leftover gremlins.

Thanks for all you do to support Lemmy.

Congratulations on the baby. We should thank you for making us go touch grass.

lemmy.zip best instance lfg

Everyone has already said it, but don't even sweat it. I had a little time when I tried to check in on Lemmy and couldn't, so I simply set it aside to try again later. Things hum along smoothly here so much more often than not, and I wouldn't be surprised if a sizable portion of us have been in that same position ourselves of needing to straighten out a failed upgrade for some project or other. I know I certainly have

Praise be to containers \o/

I definitely felt that! Trying to check the feed and getting a 502 was not nice. Good thing I had an account with another instance for the interim. Anyway, no service is bulletproof and things absolutely go wrong when running a server, no matter how good or prepared we may be. Having working backups is instrumental and I'm glad you have that going.

Hey man its all good. I understand living the working parent life. It ain't easy

Thank you for your work!

Chiil, you do a great job managing this. There is no need to blame that way yourself.

Get some rest, enjoy first stages of parenting and take your time updating.

Thanks a lot for lemmy.zip and take care of yourself.

No worries man. It's just social media. We survived!

One thing I was curious about is I didn't know where to go to look up info on what was going on? But you mentioned you were posting links, so I'll bookmark some of that info! Anyway thanks for all your hard work.

This is FOSS, not a job. You are doing it for free + maybe some donations people give. This is social media, not some critical health thing that needs to be working 24/7. Thanks for caring, now you know for next time, but don't beat yourself up. We appreciate your efforts and the transparency you have been giving us by making these posts! Keep on going with the transparency, but take it easy on yourself.

The graph shows the Federation sink is finished after four and a half hours.

Great work and welcome back 🤗

Thanks for the hard work! Glad the server is back online.

A suggestion: Post a message on status.lemmy.zip when there is maintenance. That was where I thought to check when I found that the main site was not working. Though, it was reporting the site was fine when it was unavailable, this time.

Oh, and congratulations on the newborn!