Debian admin here. Even Debian gets regular kernel upgrades that like a reboot afterwards. Security updates are more important than uptime. Also regular testing for clean recovery after a reboot is a must so a power outrage doesn't bring any new surprises with it. Also test your backup restores regularly.
Novice homelaber here, is this just a case of apt update & upgrade or is there different commands for security and kernel updates? Also what’s your preferred backup/restore software? Thanks!
You should be reading the proposed changes before pushing yes. But regardless you can get stuck upgrading if you use only "upgrade" when dependences intentionally incompatibly change by package maintainers
Your note is very interesting about the difference between the commands and how autoremove will automatically remove stuff before or after the upgrade is performed. Should it always be done after, or are there instances when running it before is more beneficial? Is there any need to do both like this:
I can't really imagine a benefit to --autoremove except for keeping old packages a bit longer before removing them.
Eg, if you run apt --update --autoremove upgrade -y once a day you'll keep your prior-to-currently-running-version kernel packages a day longer than if you ran autoremove immediately after each upgrade.
To make things more confusing: the new-ish apt full-upgrade command seems to remove most of what apt autoremove wants to... but not quite everything. 🤷
Incredible that it's not written everywhere, I always wanted to use something like this without the " update && upgrade" which looks like is not working oftentimes
Is it really not written? I saw apt upgrade --update and knew the standard shortcut would be -u, but that didn't work so I tried -U, bingo bongo off I went.
It am quite sure in the manual, but if you just look on the interwebz tutorials every command line just writes the full shebang. So you don't look up the manual and get flabbergasted when you see this post.
btw: if you are able to guess "what the standard shortcut would be", you are a wizard Harry 🥳
I am using restic and backrest on my yunohost server and I really like it! It is really set up and forget for me. Only the uploads to backblaze b2 are still triggered manually.
Also did a full recovery from the backblaze repo (downloaded locally) without problems.
Also just now heard about Zerobyte it is a backup solution based on restic and looks very good!
I'm not the person you asked the question of. I'm a fellow novice homelaber.
I use Kopia to backup my data folders and Docker container data. Works really well. The project for this weekend is to set offsite backups to be uploaded to iDrive.
Yeah, people that brag about uptimes are just bragging about the fragility of their infrastructure. If designed correctly you should be able to patch and reboot infrastructure while application availability stays up.
With an uptime of greater than 5 years I'm going to be concerned about the system potentially not coming back up after a reboot/power outage, especially for physical hardware
At a bank I worked at, we had an old IBM Power server which was at that point purely used for historical data. It had multiple years of uptime and was of course a good 10+ years old. When we went to take it offline, we actually just disabled the nic on the switch so we could reduce the number of powercycles it would see in fear that it would not power on anymore. Theoretically the data on it is purely historical, backed up and not needed, but there was enough question marks on each of those fronts we just played it safe
I haven't had a kernel update on Debian that triggered the "you should restart" message in quite some time. I was under the understanding that most newer systems now use splicing at the kernel level to not require periodic reboots.
Check for the existence of the for containing packages that recommend a reboot. Debian does not do live patching like Ubuntu does. Not least because updates to firmware are usually not applied until reboot. Also even if that were the case, regular checks for healthy reboots make sense.
Seriously, one black out and suddenly you see the need for a UPS. Now my desktop is on a USB, my work laptop and monitors are on a UPS, my homelab is on a UPS, even my modem and router are on a UPS. I just wish I could get a backup generator, but that's not happening anytime soon.
My experience with using an UPS is that they have caised an outage every few years, which is more often than we get power outages where I live, so I didn't replace the batteries last time the UPS took down my server, and are just running straight from the wall.
It might be better with a more expensive UPS, but it's not worth it for me.
A hale storm earlier this year and the power outage it caused created some bizarre issue with my home server I have yet to diagnose. All of my containers and VMs corrupted in some way, so I had to restore from backup, but my file server container has some sort of permissions issue on top of that.
Honestly the brownout before the outage is almost definitely what did it, but the cost of a UPS that also protects against brownouts is well outside of my usual hobby budget so it's hard to justify on ewaste hardware that I got a pallet of for less than what the UPS would cost used
Yeah I read up on them before I bought, APC seemed like the best. I test them at least once a year and so far I've only had to replace one battery. Depending on the application I paid between $80 and $180 for each, but the higher upfront cost seems to have paid off for me at least. I am a sample size of 1, your results may vary.
Yep, the black outs have stopped now but for a while it was a daily occurrence. My NAS took a beating and so did my desktop. I spent a ton on ups's to make sure that stuff was protected and bonus, I wouldn't loose connection while on phone calls with government officials while at work.... they get pissy when you suddenly drop off.
I sometimes have power outage in winter (snow storm, ice, etc) and working from home I need a UPS ; modem cable, router, PC, monitors, are on it, it can stand ~5h
I bought a used APC Back-UPS Pro BR1500G for $100 on market place, it was a good deal, I replaced the 2 battery inside and added 4 outside (it is supported), I'm ready!
I was introduced to homelab by trying to figure out how my uncles setup. It ran for 4 years after he died, 11 years uptime. The estate probate prevented anyone from touching the equipment for the legal fights, and I get a kick out of thinking of how smug he would have been about it.
Or if you have a UPS and backup generator or a house battery (do these need a UPS as well still?) it will tell you how long since you setup the system.
I would suspect you would still want a UPS. I don't think house "power" setups have the switch over speed even if they're automatic. Most home generator setups are manual not sure about battery setups.
My home generator is automatic but you still need an ups because the transfer switch and power on process for the generator isn’t instant. Takes like 10-30 seconds depending on how cold it is and how recently I serviced the generator.
You also ideally need a higher quality ups that can handle the shitty power coming from a generator, although the overall ups doesn’t need to be as “hefty” as a result. My ups is the kind that has extra filtering and stabilization of incoming power. My old ups was a cheaper cyberpower and it died after a few months of generator usage (we lose power here roughly every 4-6 weeks, thus the auto generator). The cheaper cyberpower would be fine in the majority of home circumstances tbh otherwise.
Pretty sure batteries can be set up for ups. One of the companies I worked for had part of the power for the building as ups. They used orange outlets to mark them. They constantly had to keep telling people not to overload it.
Battery should be automatic, but yeah not certain how quick the switchover is. At least you could go for the smallest capacity UPS there is as long as it can manage the wattage you are going with.
Yeah that’s about the only time I have to do reboots at work which are 99% linux. Well the production ones anyway.
Or the other reason is my lab having power issues due to malfunctioning UPSes, faulty NEMA L6-30 plugs, janky 240v circuit breakers or… I’m beginning to think my lab is electrically cursed.
While technically the truth, it can be a hassle to make sure you restart all relevant services after updating a given library.
I just like being able to restart underlying system to take care of any possible straggler without thinking, and the services broadly be provided by multiple systems so the "experience" is starting up through a rolling reboot
Pretty sure everybody is missing the joke. The joke is that Debian packages are so stable and stale that you likely will need a reboot before an update.
I got obsessed with uptime in the early 2000s, but for my desktop Slackware box. It ran a bunch of servers and services and crap but only for me, not heavy loads of public users. Anyway, I reached 6 years of uptime without a UPS and was aiming for 7 when a power outage got me.
Skill issue. Next time you can open up the computers power supply while it’s running, splice in a second power cable, and attach a UPS without powering down or getting electrocuted.
That's pretty impressive. I have a "big battery" that still isn't that big, but I don't consider it part of my wacky constellation of UPS devices. It's more of an "oh shit, no power, make sure the food doesn't go bad" and keep the refrigerator going for a day kind of thing.
danggg I wonder how long that bad boy takes to charge after an outage. The page says the solar powered one says < 2 hours but I don't know if I trust that. Thats an insane charge speed for solar
Can I ask, what is the advantage of a Debian server over a True Nas one? Asking because I set up True Nas and wondering if I should switch it to Debian
True nas is nas software that moonlights as a server. Debian is a linux distro commonly used as the operating system for servers due to its incredible stability and reliability among other things. So reliable infact that it's used as the operating system for true nas scale! So unless your using the core version (that runs bsd) then your already using it.
As far as rawdogging Debian on your hardware goes, id recommend against it unless you're looking to seriously up your admin game. No web interfaces, lots of time in the terminal ( command line ) and more configuration files than is anyway reasonable. And we haven't even started on virtual machines like proxmox ( also Debian based! ) or container critters like docker and kubernetes. (Iirc true nas uses kubernetes under the hood)
::: spoiler alt-text
___alt-text: The "I lied, I don't have netflix" meme template. The girl with heavy dark rings around her eyes points a gun at the observer, with various images inserted in the background. The images include references to debian, libreboot, rsync, sed&awk, cron. The text reads: "I lied, I don’t have netflix - Take off your shoes, we’re going to learn to setup a NAS with Debian customized and automated to the bone and also automate the deployment process with Kubernetes. Everything will have 3-2-1 backups and controls will be networked to the volume slider in the radio of your car. We will use the motherboard of your calculator because it’s supported by libreboot."
:::
We're chilling right now (: Don't let the gun mislead you. They sure do have a bad rep. spins gun around finger It's not actually guns that kill peop- shoots self in foot.
Might as well… we would’ve probably gone to jail instead anyway, because they think they know me better than myself. It would’ve been so romantic though: they’d come banging, shouting some gibberish about unauthorized use of emergency frequency and arson. We would’ve shot two bullets back through the door, shouting back, backup failure IS an emergency, and how you personally don’t take shit from no one.
Small correction: since the newest version there only is Trunas Scale, so the Debian derivative, which they now call Community Edition. The BSD variant has been decommissioned as far as I know.
What route do I go if I want to up my admin slowly so I eventually feel able to run pure Debian? Currently running Docker on Unraid with two minor VMs but looking to migrate away from Unraid with the intention to only run FOSS (and get a deeper understanding of everything under the hood).
I know that's little information, all I need is a nudge in the right direction so I can figure things out by consulting documentation and forums.
If you can afford it it's a good idea to buy a Raspberry Pi since Raspbian is basically just debian. Then replicate your current setup on it and just try to tinker with it without any risks of breaking things or losing data.
If you're using a lot of Docker I would recommend learning the command line since you'll be able to use Docker on basically any real OS at that point.
Realistically, comfort comes from experience. The more you use it the more you'll feel comfortable.
If you want to get a lot of exposure without dedicating too much time to it and limit the risk, I would say, spin up a Debian VM and try to configure it into the server you want the old school way. Setup ssh keys, raid pool and samba share all via ssh. Try to do it like you're actually deploying it. This will give you real world exposure to the command line and the commands you'd run. Next maintain that server like it's production, ssh in every couple of weeks to run updates and reboot. Just that muscle memory of logging in and reviewing updates will help you feel more comfortable. Do it again with another service (a VPN server would be an easy choice, a Minecraft server is also a fun one but requires a lot more memory. DNS would be good if you're feeling brave, but that's really just because DNS architecture is more complex than most realize) and maintain those servers too
Once you've setup a couple of servers and spent a couple of months monitoring and updating them your comfort level should be much higher and you might feel ready to setup some actually home production servers on Debian or the like.
You mentioned running Trunas and wanting to learn Debian and other FLOSS software, the easy button answer is to run Proxmox. Its free and open source with paid enterprise support plans available and has been rapidly improving just in the handful of years I've been running it. Proxmox is really just a modified version of Debian. They have some tweaks and custom kernels over stock Debian but impressively actually have a supported install method of installing overtop of an existing Debian install and apparently some Proxmox employees actually run it as their workstation operating system
I've actually been working my way through the proxmox-on-top-of-debian guide recently, but after installing the proxmox-ve kernel and rebooting, I was left with SSH disabled (connection refused) and no local console (more precisely no monitor output past "loading initial ramdisk"). I have so little time on my hands that troubleshooting is sometimes taking the fun out of it. Probably just going to re-install using the Proxmox ISO.
Honestly figuring out docker is 50% of the journey with the other 50% mostly being networking. For instance if your looking to start your own Jack Sparrow themed streaming service you'd want to grab a domain name, point it at your ip, open up ports 80 and 443 on your router, install a reverse proxy via docker and set up SSL ( hint: Caddy makes this easy ) and point it at your jellyfin docker container and voila, your very own streaming service you can access from anywhere! Notice the complicated part of all this is mostly the networking and docker setups, not so much the OS that your running. ( Note: don't open ports without knowing the risks )
Debian is a fine OS but most homelab stuff can be done on anything you can install docker on, even on a windows computer! That's not to say you shouldn't learn some Linux server stuff but it isn't wholly necessary. That being said...
My best advice for getting into Linux servers would be to grab an old PC, laptop or even a raspberry pi and install Debian, raspbian or any other distro on it. Figure out how to log in via ssh and get the thing running headless ( no keyboard or monitor ) and just learning to navigate and do things via the terminal. Some of the basics would be learning to use the package manager to install software, mounting the file system remotely and figuring out how to setup static IPs and such. When your ready go ahead and install docker, follow some tutorials, learn some yaml and your off to the races!
Debian is well known for maintaining established packages in its repos. This means that all of the software is thoroughly tested, and therefore (usually) stable; however, the software in question is generally older, so it also means that sometimes you'll have to find your own approach if you want to run any newer services.
Configurability? I mean Truenas Scale is also based on Debian, but it's an appliance software, if you want NAS it's purpose made for that. You need to configure Debian yourself if you want functioning NAS.
I still remember when TN doesn't have native Tailscale apps/docker yet and everytime there's a Truenas update I need to reinstall and set up Tailscale from scratch.
If you just need a NAS with basic apps/docker, there is no reason to just use Truenas.
I use both, but run a Technitium DNS and Frigate on bare Debian.
I like having a consistent update and reboot schedule. Uptime feels overrated over stability and clearing the RAM occasionally.
I definitely have some Docker containers that randomly stop working, and they are more often consistently fixed by a reboot of the machine rather than a reboot of the container or the Docker service.
At some point when I am less busy again I think I am gonna swap back to a debian based system because my experience on arch and red hat systems just hasnt been as good
(this may be because I started on Debian based systems and keep trying to use commands that dont work on the other ones out of muscle memory)
I get bored every so often and move all the important stuff to an external drive or a separate internal one and completely change my os
I am on manjaro but I have also run arch, red hat, void, mint, Debian, Ubuntu and a bunch of others that I either put on laptops or something similar as messing around with devices
Tails and slitaz have to be my favorite to run from a USB but peppermint isn't the worst
I have had minimal issues from my manjaro desktop but I just dont like it as much as my mint based systems because everything feels wrong and I can barely remember how to update my graphics drivers on manjaro vs mint where I am confident I could run my entire system mostly command line from installs to updates and random other shit that I just can't remember how to do through arch systems because I dont run them as hard for some reason
They screwed up their upgrade script in the release around then. I spent an hour manually fixing it on my system. I’d imagine they sorted it with a point release shortly after but I haven’t checked.
My RPi uptime on one project will never exceed 4 hours.
I've got a cron job set to reboot my Raspberry Pi every 4 hours because I wrote a crappy Python app that continuously creates objects during operation that I would have to recreate, but I can't delete the originals, or rather, I can delete the original parent but the child survives and keeps its memory allocation. So a full reboot with autolaunch of the application on boot is my ugly janky workaround. Its a cosmetic application, nothing critical. Its just a colorful display of data metrics.
I can hear the horror and gnashing of teeth of real developers as they read this.
You just lack imagination. Some hikvision security cameras (the large, very expensive enterprisey ones) restart every couple of days due to "buildup of cosmic radiation".
No matter how competent you are as a developer, there is no escaping cosmic radiation! 😉
Oh, there's even more jank in this thing than the reboot workaround described above!
I have 3 windows displaying different metrics on this display powered by the RPi. Because of the animation of each metric rendered on the display, higher value metrics will consume more CPU. Since each is a separate process, the animation in the displays would be different for each window by without any modifications. So to make each of the 3 display's animations operate at the same relative speed, I do a calculation of how the number of objects being displayed for the metric, then add an amount of invisible (well, black on black) objects to each window so to equal a fixed amount of the animation speed I want resulting in each window having the exact same number of objects and the animations move at the same speed.
This works surprisingly well. The only time I have to monkey with the fixed value is if I'm using it on faster or slower Raspberry Pis. For example, I'll have a lower number of final fixed objects for an RPi 3 rather than a higher number of fixed final objects for a faster RPi 4.
I've got an OpenBSD based router with ~4 hours of battery backup. If I ever stopped futzing around with it, the uptime would be fairly close to when the last version update was. (They've got a release cadence of about 6 months).
Alt+L in fish shell, works great. It runs the ls command without having to modify your current command that you're typing. Or even change your cursor position.
how long since the boss has been asleep so you can finally restart without them calling two seconds later cause they didn't bother reading the scheduled downtime email
Interesting. LMDE seems to be more like MS Windows in that things like kernel updates insist on a reboot, and certain other things are easiest restarted with a reboot too, for example, X.Org changes.
I'm sure there's still a way to bootstrap a new kernel on the bare metal without needing to reboot, likewise for restarting X.Org, but I foresee problems with any programs and daemons that were children of the original processes. For example, convincing them not to exit when their parent does and then getting them to play nice under a new session.
I mean, I guess you could just not update, or have a long period where they're unnecessary and that'd work too. That could well be what this meme is getting at. Can confirm sessions (caveat: with standby and hibernate) that have lasted well over a month.
But this all raises the question: Does anyone actually not reboot when system changes happen, and what's the workflow for bootstrapping without rebooting there?
LMDE is mostly just the apps and visual config. It is verg close to regular Debian. I know for a fact it is basically just regular Debian because I have distromorphed it into Kicksecure several times, which only works on Debian.
Those who manage the dedicated server racks service kept my stuff intact. Thankfully. Just disrupted my uptime.
[User "error" since, has cost me a TB of data. "Error", fearquoted, because it was intentional... probably unnecessary clearing of space, partly regretted since.]
I don't know how big the fire was, happened over 1000 miles away from here.
Debian admin here. Even Debian gets regular kernel upgrades that like a reboot afterwards. Security updates are more important than uptime. Also regular testing for clean recovery after a reboot is a must so a power outrage doesn't bring any new surprises with it. Also test your backup restores regularly.
Novice homelaber here, is this just a case of apt update & upgrade or is there different commands for security and kernel updates? Also what’s your preferred backup/restore software? Thanks!
Nope it's just apt update & upgrade. Iirc apt tells you when the kernel was updated and needs a reboot as well.
Only if you installed the package
needrestartNot if you use Proxmox! One has to be careful.
full-upgrade probably a better pick
Is it? Afaik that also removes incompatible packages so if you've installed some custom stuff that might not be the best option.
You should be reading the proposed changes before pushing yes. But regardless you can get stuck upgrading if you use only "upgrade" when dependences intentionally incompatibly change by package maintainers
I think you can do
apt upgrade --updatenow.WHAT. Does this do both sudo apt update and sudo apt upgrade?
see also --autoremove
Your note is very interesting about the difference between the commands and how autoremove will automatically remove stuff before or after the upgrade is performed. Should it always be done after, or are there instances when running it before is more beneficial? Is there any need to do both like this:
I can't really imagine a benefit to
--autoremoveexcept for keeping old packages a bit longer before removing them.Eg, if you run
apt --update --autoremove upgrade -yonce a day you'll keep your prior-to-currently-running-version kernel packages a day longer than if you ranautoremoveimmediately after each upgrade.To make things more confusing: the new-ish
apt full-upgradecommand seems to remove most of whatapt autoremovewants to... but not quite everything. 🤷I think so. I read it a few months back, but I don't use any apt based systems to check on.
Yup
'apt upgrade -U'
Incredible that it's not written everywhere, I always wanted to use something like this without the " update && upgrade" which looks like is not working oftentimes
Is it really not written? I saw apt upgrade --update and knew the standard shortcut would be -u, but that didn't work so I tried -U, bingo bongo off I went.
It am quite sure in the manual, but if you just look on the interwebz tutorials every command line just writes the full shebang. So you don't look up the manual and get flabbergasted when you see this post. btw: if you are able to guess "what the standard shortcut would be", you are a wizard Harry 🥳
🤯
Borg
restic (thanks @[email protected] )
backrest (restic with WebUI)
Also worth checking out restic. It's more command line oriented and is generally stateless
I configured restic once, forget about it and saved my files because it was making backups since forever.
Oh, never heard about it. A quick research showed me that restic is a very viable solution. Thanks for mentioning it, I added it to my comment.
While researching, I also came across a fancy WebUI, which is mostly what non-CLI users want: backrest
I am using restic and backrest on my yunohost server and I really like it! It is really set up and forget for me. Only the uploads to backblaze b2 are still triggered manually. Also did a full recovery from the backblaze repo (downloaded locally) without problems.
Also just now heard about Zerobyte it is a backup solution based on restic and looks very good!
Thanks just installed immich and I need a quality backup system.
I appreciate the link!
Kernel updates are usually held back and need to be selected manually. E.g. apt-get install linux-image-amd64.
I prefer rsync for private backups and employ bareos in my company for all servers.
I'm not the person you asked the question of. I'm a fellow novice homelaber.
I use Kopia to backup my data folders and Docker container data. Works really well. The project for this weekend is to set offsite backups to be uploaded to iDrive.
When I update I use this:
sudo apt update && \ sudo apt upgrade -y && \ sudo apt full-upgrade -y && \ flatpak update -y 2>/dev/null; \ sudo apt autoremove -y && \ sudo apt autoclean && \ sudo journalctl --vacuum-time=7dYou can get rid of upgrade if you also use full-upgrade
As someone running a UPS on my ubuntu server, "uptime" represents the time since the last kernel release, and not much else.
Use kexec then
Yeah, people that brag about uptimes are just bragging about the fragility of their infrastructure. If designed correctly you should be able to patch and reboot infrastructure while application availability stays up.
With an uptime of greater than 5 years I'm going to be concerned about the system potentially not coming back up after a reboot/power outage, especially for physical hardware
At a bank I worked at, we had an old IBM Power server which was at that point purely used for historical data. It had multiple years of uptime and was of course a good 10+ years old. When we went to take it offline, we actually just disabled the nic on the switch so we could reduce the number of powercycles it would see in fear that it would not power on anymore. Theoretically the data on it is purely historical, backed up and not needed, but there was enough question marks on each of those fronts we just played it safe
New fear unlocked.
The sun was angry that day, my friend...
I haven't had a kernel update on Debian that triggered the "you should restart" message in quite some time. I was under the understanding that most newer systems now use splicing at the kernel level to not require periodic reboots.
I haven't seen it in a while either, but also, if there is a kernel update,
uname -salways returns the old kernel until a reboot.Check for the existence of the for containing packages that recommend a reboot. Debian does not do live patching like Ubuntu does. Not least because updates to firmware are usually not applied until reboot. Also even if that were the case, regular checks for healthy reboots make sense.
What do you use for backup restores ?
The same tools.
HOT SWAP!
https://www.youtube.com/watch?v=SYRlTISvjww
This is why we have UPS ;-)
Seriously, one black out and suddenly you see the need for a UPS. Now my desktop is on a USB, my work laptop and monitors are on a UPS, my homelab is on a UPS, even my modem and router are on a UPS. I just wish I could get a backup generator, but that's not happening anytime soon.
My experience with using an UPS is that they have caised an outage every few years, which is more often than we get power outages where I live, so I didn't replace the batteries last time the UPS took down my server, and are just running straight from the wall. It might be better with a more expensive UPS, but it's not worth it for me.
A hale storm earlier this year and the power outage it caused created some bizarre issue with my home server I have yet to diagnose. All of my containers and VMs corrupted in some way, so I had to restore from backup, but my file server container has some sort of permissions issue on top of that.
Honestly the brownout before the outage is almost definitely what did it, but the cost of a UPS that also protects against brownouts is well outside of my usual hobby budget so it's hard to justify on ewaste hardware that I got a pallet of for less than what the UPS would cost used
Yeah I read up on them before I bought, APC seemed like the best. I test them at least once a year and so far I've only had to replace one battery. Depending on the application I paid between $80 and $180 for each, but the higher upfront cost seems to have paid off for me at least. I am a sample size of 1, your results may vary.
I got tired of my network puking every time the power went out for 5 seconds.
Edit- My NAS really dislikes having the power cut off.
Yep, the black outs have stopped now but for a while it was a daily occurrence. My NAS took a beating and so did my desktop. I spent a ton on ups's to make sure that stuff was protected and bonus, I wouldn't loose connection while on phone calls with government officials while at work.... they get pissy when you suddenly drop off.
I sometimes have power outage in winter (snow storm, ice, etc) and working from home I need a UPS ; modem cable, router, PC, monitors, are on it, it can stand ~5h
I've had good luck with APC. Just be ready to pay a bit more upfront. But so far in the last 6 or 7 years, I've only had to replace one battery.
I bought a used APC Back-UPS Pro BR1500G for $100 on market place, it was a good deal, I replaced the 2 battery inside and added 4 outside (it is supported), I'm ready!
$100 is a good deal, though with the batteries you probably paid as much as a new one. But you have extra battery life so win.
My father was an HPUX admin that had a server with an uptime of >12 years
I was introduced to homelab by trying to figure out how my uncles setup. It ran for 4 years after he died, 11 years uptime. The estate probate prevented anyone from touching the equipment for the legal fights, and I get a kick out of thinking of how smug he would have been about it.
Heard of tuptime? I've been using it for a while now, I think I like it.
System startups: 151 since 18:00:05 10/11/15 System shutdowns: 137 ok + 13 bad System life: 9yr 223d 1h 27m 47s
Longest uptime: 106d 5h 34m 28s from 14:17:10 26/03/22 Average uptime: 23d 4h 32m 0s System uptime: 99.81% = 9yr 216d 12h 31m 51s
Longest downtime: 4d 23h 30m 48s from 10:36:53 14/09/23 Average downtime: 1h 2m 46s System downtime: 0.19% = 6d 12h 55m 56s
Current uptime: 25d 0h 34m 25s since 20:25:37 15/11/25
Heard of it for the first time (as far as I can remember) a couple days ago, on Lemmy.
TIL, Lemmy's educational.
Does it work retrospectively?
it doesn't appear to
Nope it creates a little database, which you could manualy edit I suppose.
So, you never update the kernel?
Mine's not connected to the internet so I'm utilizing the if it ain't broke don't fix it strategy.
@NullPointerException @yesman debian never updates it
Or if you have a UPS and backup generator or a house battery (do these need a UPS as well still?) it will tell you how long since you setup the system.
I would suspect you would still want a UPS. I don't think house "power" setups have the switch over speed even if they're automatic. Most home generator setups are manual not sure about battery setups.
My home generator is automatic but you still need an ups because the transfer switch and power on process for the generator isn’t instant. Takes like 10-30 seconds depending on how cold it is and how recently I serviced the generator.
You also ideally need a higher quality ups that can handle the shitty power coming from a generator, although the overall ups doesn’t need to be as “hefty” as a result. My ups is the kind that has extra filtering and stabilization of incoming power. My old ups was a cheaper cyberpower and it died after a few months of generator usage (we lose power here roughly every 4-6 weeks, thus the auto generator). The cheaper cyberpower would be fine in the majority of home circumstances tbh otherwise.
Pretty sure batteries can be set up for ups. One of the companies I worked for had part of the power for the building as ups. They used orange outlets to mark them. They constantly had to keep telling people not to overload it.
Battery should be automatic, but yeah not certain how quick the switchover is. At least you could go for the smallest capacity UPS there is as long as it can manage the wattage you are going with.
At least in my experience the chances that I move or replace hardware are much higher than the chances for a power outage.
For me I won't be replacing and video cards or ram sticks for the foreseeable future.
Same... but I also remember a single outage in the last 15 or so years.
Are we not doing kernel upgrades?
Was about to say, "or if you're running Arch, the last time you updated the kernel or systemd version, so probably last week or summit."
Yeah that’s about the only time I have to do reboots at work which are 99% linux. Well the production ones anyway.
Or the other reason is my lab having power issues due to malfunctioning UPSes, faulty NEMA L6-30 plugs, janky 240v circuit breakers or… I’m beginning to think my lab is electrically cursed.
While technically the truth, it can be a hassle to make sure you restart all relevant services after updating a given library.
I just like being able to restart underlying system to take care of any possible straggler without thinking, and the services broadly be provided by multiple systems so the "experience" is starting up through a rolling reboot
Pretty sure everybody is missing the joke. The joke is that Debian packages are so stable and stale that you likely will need a reboot before an update.
Also, it's a joke....please patch your boxes, k?
I've got a patch in my boxers right now.
Oh boxes.
manNot much, time. What's up with you?
total erection time
Priapism isn't a good thing.
Never heard women complain about it.
I got obsessed with uptime in the early 2000s, but for my desktop Slackware box. It ran a bunch of servers and services and crap but only for me, not heavy loads of public users. Anyway, I reached 6 years of uptime without a UPS and was aiming for 7 when a power outage got me.
Skill issue. Next time you can open up the computers power supply while it’s running, splice in a second power cable, and attach a UPS without powering down or getting electrocuted.
For legal reasons, /s
Not sure what your signature is supposed to do here but now I have 3rd degree burns and a fireball has engulfed my office wall
But more importantly, did your uptime get reset?
“Uptime” — aka the anxiety meter for every sysadmin.
What if I have a really good UPS with like a whole day's worth of juice?
That's either an impressive UPS or an impressively low power server. Either way, congrats!
It's a giant hamster wheel connected to a generator I run in 😃
I wish my daughter's rabbit wasn't so lazy. I'd join you in the dream of animal powered infrastructure.
Anker have up to 3.8kWh stations, goals. This or Anker's equivalent of the Tesla Powerwall, wall-mounted energy storage.
That's pretty impressive. I have a "big battery" that still isn't that big, but I don't consider it part of my wacky constellation of UPS devices. It's more of an "oh shit, no power, make sure the food doesn't go bad" and keep the refrigerator going for a day kind of thing.
Oh yeah, I think maybe a bunch of smaller UPS is better than one massive one - can't always power everything from one point aha
Let me tell you though, when the power goes out it is like an unholy choir of beeping. Our pets think the apocalypse has arrived.
danggg I wonder how long that bad boy takes to charge after an outage. The page says the solar powered one says < 2 hours but I don't know if I trust that. Thats an insane charge speed for solar
In a critical environment the UPS only has to last as long as it takes to switch over to a backup generator.
Hamsters on crack, noted.
I only get about 7 minutes out of mine but that gets it through most power outages.
Can I ask, what is the advantage of a Debian server over a True Nas one? Asking because I set up True Nas and wondering if I should switch it to Debian
True nas is nas software that moonlights as a server. Debian is a linux distro commonly used as the operating system for servers due to its incredible stability and reliability among other things. So reliable infact that it's used as the operating system for true nas scale! So unless your using the core version (that runs bsd) then your already using it. As far as rawdogging Debian on your hardware goes, id recommend against it unless you're looking to seriously up your admin game. No web interfaces, lots of time in the terminal ( command line ) and more configuration files than is anyway reasonable. And we haven't even started on virtual machines like proxmox ( also Debian based! ) or container critters like docker and kubernetes. (Iirc true nas uses kubernetes under the hood)
::: spoiler alt-text
___alt-text: The "I lied, I don't have netflix" meme template. The girl with heavy dark rings around her eyes points a gun at the observer, with various images inserted in the background. The images include references to debian, libreboot, rsync, sed&awk, cron. The text reads: "I lied, I don’t have netflix - Take off your shoes, we’re going to learn to setup a NAS with Debian customized and automated to the bone and also automate the deployment process with Kubernetes. Everything will have 3-2-1 backups and controls will be networked to the volume slider in the radio of your car. We will use the motherboard of your calculator because it’s supported by libreboot." :::
cool, then we can chill?
We're chilling right now (: Don't let the gun mislead you. They sure do have a bad rep. spins gun around finger It's not actually guns that kill peop- shoots self in foot.
Ahh, another ER romance...
Might as well… we would’ve probably gone to jail instead anyway, because they think they know me better than myself. It would’ve been so romantic though: they’d come banging, shouting some gibberish about unauthorized use of emergency frequency and arson. We would’ve shot two bullets back through the door, shouting back, backup failure IS an emergency, and how you personally don’t take shit from no one.
Small correction: since the newest version there only is Trunas Scale, so the Debian derivative, which they now call Community Edition. The BSD variant has been decommissioned as far as I know.
You seem like the right person to ask this:
What route do I go if I want to up my admin slowly so I eventually feel able to run pure Debian? Currently running Docker on Unraid with two minor VMs but looking to migrate away from Unraid with the intention to only run FOSS (and get a deeper understanding of everything under the hood).
I know that's little information, all I need is a nudge in the right direction so I can figure things out by consulting documentation and forums.
If you can afford it it's a good idea to buy a Raspberry Pi since Raspbian is basically just debian. Then replicate your current setup on it and just try to tinker with it without any risks of breaking things or losing data.
If you're using a lot of Docker I would recommend learning the command line since you'll be able to use Docker on basically any real OS at that point.
Realistically, comfort comes from experience. The more you use it the more you'll feel comfortable.
If you want to get a lot of exposure without dedicating too much time to it and limit the risk, I would say, spin up a Debian VM and try to configure it into the server you want the old school way. Setup ssh keys, raid pool and samba share all via ssh. Try to do it like you're actually deploying it. This will give you real world exposure to the command line and the commands you'd run. Next maintain that server like it's production, ssh in every couple of weeks to run updates and reboot. Just that muscle memory of logging in and reviewing updates will help you feel more comfortable. Do it again with another service (a VPN server would be an easy choice, a Minecraft server is also a fun one but requires a lot more memory. DNS would be good if you're feeling brave, but that's really just because DNS architecture is more complex than most realize) and maintain those servers too
Once you've setup a couple of servers and spent a couple of months monitoring and updating them your comfort level should be much higher and you might feel ready to setup some actually home production servers on Debian or the like.
You mentioned running Trunas and wanting to learn Debian and other FLOSS software, the easy button answer is to run Proxmox. Its free and open source with paid enterprise support plans available and has been rapidly improving just in the handful of years I've been running it. Proxmox is really just a modified version of Debian. They have some tweaks and custom kernels over stock Debian but impressively actually have a supported install method of installing overtop of an existing Debian install and apparently some Proxmox employees actually run it as their workstation operating system
I've actually been working my way through the proxmox-on-top-of-debian guide recently, but after installing the proxmox-ve kernel and rebooting, I was left with SSH disabled (connection refused) and no local console (more precisely no monitor output past "loading initial ramdisk"). I have so little time on my hands that troubleshooting is sometimes taking the fun out of it. Probably just going to re-install using the Proxmox ISO.
I've never tried the Proxmox over Debian method, I just know it is an officially supported install method. Good on you for getting that far though!
Welcome to home labbing... you poor fool!
Honestly figuring out docker is 50% of the journey with the other 50% mostly being networking. For instance if your looking to start your own Jack Sparrow themed streaming service you'd want to grab a domain name, point it at your ip, open up ports 80 and 443 on your router, install a reverse proxy via docker and set up SSL ( hint: Caddy makes this easy ) and point it at your jellyfin docker container and voila, your very own streaming service you can access from anywhere! Notice the complicated part of all this is mostly the networking and docker setups, not so much the OS that your running. ( Note: don't open ports without knowing the risks )
Debian is a fine OS but most homelab stuff can be done on anything you can install docker on, even on a windows computer! That's not to say you shouldn't learn some Linux server stuff but it isn't wholly necessary. That being said...
My best advice for getting into Linux servers would be to grab an old PC, laptop or even a raspberry pi and install Debian, raspbian or any other distro on it. Figure out how to log in via ssh and get the thing running headless ( no keyboard or monitor ) and just learning to navigate and do things via the terminal. Some of the basics would be learning to use the package manager to install software, mounting the file system remotely and figuring out how to setup static IPs and such. When your ready go ahead and install docker, follow some tutorials, learn some yaml and your off to the races!
Debian is well known for maintaining established packages in its repos. This means that all of the software is thoroughly tested, and therefore (usually) stable; however, the software in question is generally older, so it also means that sometimes you'll have to find your own approach if you want to run any newer services.
Configurability? I mean Truenas Scale is also based on Debian, but it's an appliance software, if you want NAS it's purpose made for that. You need to configure Debian yourself if you want functioning NAS.
I still remember when TN doesn't have native Tailscale apps/docker yet and everytime there's a Truenas update I need to reinstall and set up Tailscale from scratch.
If you just need a NAS with basic apps/docker, there is no reason to just use Truenas.
I use both, but run a Technitium DNS and Frigate on bare Debian.
I like having a consistent update and reboot schedule. Uptime feels overrated over stability and clearing the RAM occasionally.
I definitely have some Docker containers that randomly stop working, and they are more often consistently fixed by a reboot of the machine rather than a reboot of the container or the Docker service.
Not to mention the security implications of not rebooting after certain updates.
Clearing the ram? For what purpose?
At some point when I am less busy again I think I am gonna swap back to a debian based system because my experience on arch and red hat systems just hasnt been as good (this may be because I started on Debian based systems and keep trying to use commands that dont work on the other ones out of muscle memory)
I get bored every so often and move all the important stuff to an external drive or a separate internal one and completely change my os
I am on manjaro but I have also run arch, red hat, void, mint, Debian, Ubuntu and a bunch of others that I either put on laptops or something similar as messing around with devices
Tails and slitaz have to be my favorite to run from a USB but peppermint isn't the worst
I just did the contrary. Moved from debian to arch. After the update to trixie my network stack completely died somehow, so I'm going back to arch.
I have had minimal issues from my manjaro desktop but I just dont like it as much as my mint based systems because everything feels wrong and I can barely remember how to update my graphics drivers on manjaro vs mint where I am confident I could run my entire system mostly command line from installs to updates and random other shit that I just can't remember how to do through arch systems because I dont run them as hard for some reason
Why type
uptimewhenwis sufficient?What the hell!?
?
Because in case not wanting the superfluous lines of output.
I had to reboot since i upgraded nvidea and jellyfin stopped transcoding.
They screwed up their upgrade script in the release around then. I spent an hour manually fixing it on my system. I’d imagine they sorted it with a point release shortly after but I haven’t checked.
Does NixOS apply kernel updates live? I can't recall from when I used it.
Mine doesn't. I reboot when I get a new kernel.
Huh. Only 11 days on the Raspberry Pi I'm using as a "desktop system" right now. (Arch Linux Arm, btw... though Arch Linux Arm sucks now-a-days.)
Let's check my RPi-based NAS:
Also not as long as I'd have guessed.
My RPi uptime on one project will never exceed 4 hours.
I've got a cron job set to reboot my Raspberry Pi every 4 hours because I wrote a crappy Python app that continuously creates objects during operation that I would have to recreate, but I can't delete the originals, or rather, I can delete the original parent but the child survives and keeps its memory allocation. So a full reboot with autolaunch of the application on boot is my ugly janky workaround. Its a cosmetic application, nothing critical. Its just a colorful display of data metrics.
I can hear the horror and gnashing of teeth of real developers as they read this.
You just lack imagination. Some hikvision security cameras (the large, very expensive enterprisey ones) restart every couple of days due to "buildup of cosmic radiation".
No matter how competent you are as a developer, there is no escaping cosmic radiation! 😉
Oh shit! Forget all that stuff I admitted to. My RPi reboots every 4 hours because ... uh ... cosmic radiation.
As a real developer...
I just remember that airplanes have "reboot the plane every 51 days" to prevent an overflow from crashing the plane in their maintenance manuals
So, like, yours can be improved, but it's not safety critical like other reboot requirements...
I'm a sysadmin and I'm weeping, gnashing my teeth and rending my garments. 😆 And I've never done anything janky like that. Ever.
Oh, there's even more jank in this thing than the reboot workaround described above!
I have 3 windows displaying different metrics on this display powered by the RPi. Because of the animation of each metric rendered on the display, higher value metrics will consume more CPU. Since each is a separate process, the animation in the displays would be different for each window by without any modifications. So to make each of the 3 display's animations operate at the same relative speed, I do a calculation of how the number of objects being displayed for the metric, then add an amount of invisible (well, black on black) objects to each window so to equal a fixed amount of the animation speed I want resulting in each window having the exact same number of objects and the animations move at the same speed.
This works surprisingly well. The only time I have to monkey with the fixed value is if I'm using it on faster or slower Raspberry Pis. For example, I'll have a lower number of final fixed objects for an RPi 3 rather than a higher number of fixed final objects for a faster RPi 4.
"If it works, it ain't stupid." ;)
I've got an OpenBSD based router with ~4 hours of battery backup. If I ever stopped futzing around with it, the uptime would be fairly close to when the last version update was. (They've got a release cadence of about 6 months).
That’s ridiculous. It’s much more complicated than that.
You need to check NUT.
Deez? 🌰🐿️
step 1:
sudo apt install slstep 2: fuck up
step 3: ???
step 4: profit!!!!
A must have IMO
Also 'fuck'
https://github.com/iffse/pay-respects there is a new alternative.
Won't work for me, I am more of a ";l" guy, I have the good direction, but wrong starting point 90%of the time 🥲
Alt+L in fish shell, works great. It runs the ls command without having to modify your current command that you're typing. Or even change your cursor position.
how long since the boss has been asleep so you can finally restart without them calling two seconds later cause they didn't bother reading the scheduled downtime email
I have a LXQt on my Termux, running on my android phone, and I am really happy with it (even though I use it for jack shit)
Hardware errors often cause system instability hence this is false.
How false are we talking? A couple seconds? Minutes?
A lot of windows errors are actually hardware acting up. Such as an aggressive overclock or random issues.
An operating system cannot prevent that
How do I check when the last power outage was if it's connected to a UPS?
Interesting. LMDE seems to be more like MS Windows in that things like kernel updates insist on a reboot, and certain other things are easiest restarted with a reboot too, for example, X.Org changes.
I'm sure there's still a way to bootstrap a new kernel on the bare metal without needing to reboot, likewise for restarting X.Org, but I foresee problems with any programs and daemons that were children of the original processes. For example, convincing them not to exit when their parent does and then getting them to play nice under a new session.
I mean, I guess you could just not update, or have a long period where they're unnecessary and that'd work too. That could well be what this meme is getting at. Can confirm sessions (caveat: with standby and hibernate) that have lasted well over a month.
But this all raises the question: Does anyone actually not reboot when system changes happen, and what's the workflow for bootstrapping without rebooting there?
I don't get why are you talking about LMDE under a meme about Debian
Because LMDE stands for Linux Mint Debian Edition
Yeah but it's Linux mint debian edition, not Debian
LMDE is mostly just the apps and visual config. It is verg close to regular Debian. I know for a fact it is basically just regular Debian because I have distromorphed it into Kicksecure several times, which only works on Debian.
Think of it as a gateway drug, lol
On my Gentoo server, uptime:
Solid.
Would have been double that by now if not for the fire.
You forgot to say "this is fine", I take it?
Joking aside, I hope you didn't lose anything. Was it a big fire?
Those who manage the dedicated server racks service kept my stuff intact. Thankfully. Just disrupted my uptime.
[User "error" since, has cost me a TB of data. "Error", fearquoted, because it was intentional... probably unnecessary clearing of space, partly regretted since.]
I don't know how big the fire was, happened over 1000 miles away from here.
So, it really was fine. :3
😁👍 Good to hear