What tools/services do people often neglect to their own detriment?
This question arose as I have come to a realization that I need to put an effort in some health check monitoring for my self-hosting and an off-site backup solution for all of my important data, and not keep putting it off, after my company's SAN went down.
I'm curious if there are other things I should consider for my self-hosted services that may not be as obvious
For me it was always a challenge to keep everything up-to-date. I couldn't check weekly or w/e because it just felt like a time suck to go to a dozen different sites, so I would let things languish.
I started putting the github releases pages of all my services in a special "updates" category on Miniflux . You can get a feed by appending the releases page with ".atom" (e.g. https://github.com/LemmyNet/lemmy/releases.atom for Lemmy) and then just get notifications whenever they're updated.
That + Watchtower for non-critical Docker containers and everything stays up-to-date.
Thanks for the .atom tip. I've been messing with Diun to try and keep up on updates but I run so many different things it ends up being useless by the time I get around to wanting to actually do the updates. I'll add the ones I super care about to Miniflux though and see if that's more doable for me
Documentation 😏
Drive health checks and regular scrubbing of raids to correct errors.
Testing and replacement of UPS batteries.
Health check is good, I need to do similar. What I've personally been missing is monitoring (passive or active). I have infouxdb/grafana for maybe two services but it's not robust enough to determine anything yet. So definitely passive monitoring.
Also, alerting. Even simple email alerts would be ideal but for the life of me I can never get email set up properly on any machine I own. It's always awkward and troublesome.
Finally probably stability in case of power outage, internet outage, or other issue. It sucks to notice you have services dependent on each other to boot properly after things reboot at 2am.