What's your "base" stack of choice?
How do you set up a server? Do you do any automation or do you just open up an SSH session and YOLO? Any containers? Is docker-compose enough for you or are you one of those unicorns who had no issues whatsoever with rootless Podman? Do you use any premade scripts or do you hand craft it all? What distro are you building on top of?
I'm currently in process of "building" my own server and I'm kinda wondering how "far" most people are going, where do y'all take any shortcuts, and what do you spend effort getting just right.
I'm a lazy piece of shit and containers give me cancer, so I just keep iptables aggressive and spin up whatever on an Ubuntu box that gets upgrades when I feel like wasting a weekend in my underwear.
An honest soul
I get paid to do shit with rigor; I don't have the time, energy, or help to make something classy for funsies. I'm also kind of a grumpy old man such that while I'll praise and embrace Python's addition of f-strings which make life better in myriad ways, I eschew the worse laziness of the all the containers attitude that we see for deployment.
Maybe a day shall come when containers are truly less of a headache than just thinking shit through the first time, and I'll begrudgingly adapt and grow, but that day ain't today.
I use debian VMs and create rootless podman containers for everything. Here's my collection so far.
I'm currently in the process of learning how to combine this with ansible... that would save me some time when migrating servers/instances.
Thanks for sharing. There’s some great stuff in the repo.
After many years of tinkering, I finally gave in and converted my whole stack over to UnRAID a few years ago. You know what? It's awesome, and I wish I had done it sooner. It automates so many of the more tedious aspects of home server management. I work in IT, so for me it's less about scratching the itch and more about having competent hosting of services I consider mission-critical. UnRAID lets me do that easily and effectively.
Most of my fun stuff is controlled through Docker and VMs via UnRAID, and I have a secondary external Linux server which handles some tasks I don't want to saddle UnRAID with (PFSense, Adblocking, etc). The UnRAID server itself has 128GB RAM and dual XEON CPUs, so plenty of go for my home projects. I'm at 12TB right now but I was just on Amazon eyeing some 8TB drives...
Proxmox, then create LXC for everything (moslty debian and a bit of alpine), no automation, full yolo, if it break I have backup (problems are for future me eh)
This.
Proxmox and then LXCs for anything I need.
and yes - I cheat a bit, I use the excellent Proxmox scripts - https://tteck.github.io/Proxmox/ because I'm lazy like that haha
I used to do the same, but nowadays I just run everything in docker, within a single lxc container on proxmox. Having to setup mono or similar every time I wanted to setup a game server or even jellyfin was annoying.
Mostly the same. Proxmox with several LXC, two of which are running docker. One for my multimedia, the other for my game servers.
I use the following procedure with ansible.
People like to diss running kubernetes on your personal servers, but once you have enough services running in your servers, managing them using docker compose is no longer cut it and kubernetes is the next logical step to go. Tools such as k9s makes navigating as kubernetes cluster a breeze.
I'm doing basically the same thing with microk8s and flux. I'd probably switch to argo if it wasn't already working.
Synology with docker-compose stack
Debian + nginx + docker (compose).
That's usually enough for me. I have all my docker compose files in their respective containers in the home directory like
~/red-discordbot/docker-compose.yml.The only headache I've dealt with are permissions because I have to run docker as root and it makes a lot of messy permissions in the home directories. I've been trying rootless docker earlier and it's been great so far.
edit: I also use
rclonefor backups.Debian and docker compose
Right now, I just flash ubuntu server to whatever computer it is, ssh and yolo lmao. no containers, no managers, just me, my servers, and a vpn, raw dogging the internet lmao. The box is running a nas, jellyfin, lemmy, and a print server; the laptop a minecraft server, and the pi is running a pihole, and a website that controls gpio that controls the lights. In the pictured setup i dont have access to the apartment complex's router, so i vpn through a openvpn server i setup in a digitalocean server.
i didnt even know what a container was until i setup the lemmy server, which i just used ansible for.
i still dont really know what ansible is.
I've set up some godforsaken combination of docker, podman, nerdctl and bare metal at work for stuff I needed since they hired me. Every day I'm in constant dread something I made will go down, because I don't have enough time to figure out how I was supposed to do it right T.T
I use Unraid and their docker and VM integration, Works great for me as a home user with mixed drives. Most of the dockers i want already have unraid templates so require less configuration. Does everything i want and made it a bit easier for me with less configuration and the mixed drive support.
I love unraid! Definitely wait between updates though to let them stabilize.
I just have a pi 4 running OpenMediaVault with docker and portainer. 😅
I have a git repository with all my compose files sorted neatly into directories, i.e. my "stack". Portainer allows adding stacks using a repository, so it's essentially one click deployment once the compose file is on a remote git server.
I run unraid on my server box with a few 8tb hdd and nvme for cache. From there it is really easy to spin up Docker containers or stacks using compose, as well as VMs using your iso of choice.
For automation, I use Ansible to run one click setup machines; it is great for any cloud provider work too.
I use NixOS on almost all my servers, with declarative configuration. I can also install my config in one command with NixOS-Anywhere
It allows me to improve my setup bit by bit without having to keep track of what I did on specific machines
Sqlite where possible, nginx, linux, no containers. I hate containers.
About two years ago my set up had gotten out of control, as it will. Closet full of crap all running vms all poorly managed by chef. Different linux flavors everywhere.
Now its one big physical ubuntu box. Everything gets its own ubuntu VM. These days if I can't do it in shell scripts and xml I'm annoyed. Anything fancier than that i'd better be getting paid. I document in markdown as i go and rsync the important stuff from each VM to an external every night. Something goes wrong i just burn the vm, copy paste it back together in a new one from the mkdocs site. Then get on with my day.
For personal Linux servers, I tend to run Debian or Ubuntu, with a pretty simple "base" setup that I just run through manually in my head.
I don't automate any of this because I don't see a whole of point in doing it.
Super interesting to me that you swap between Debian and Ubuntu. Is there any rhyme or reason to why you use one over the other?
I tend to prefer installing Debian on a server, but recently I did install Ubuntu's recent LTS on a box because I was running into an issue with the latest version of Debian. I didn't want to revert to an earlier version of Debian or spend a bunch of time figuring out the problem I was having with Python, so I opted to use Ubuntu, which worked.
Ubuntu is based on Debian, so it's like using the same operating system, as far as I'm concerned.
Ansible and docker compose.
i do this, mixed with a little
docker runinside of Makefiles. i store all my ansible playbooks in a repo, along with other repos for different projects and purposes. i store all of those in git repos that i clone via ssh from a server that acts as a NAS backed by zfs.I just use Ansible + docker_container. Does it all beautifully!
I have a single desktop running Proxmox with a TrueNAS VM for handling my data and a Debian VM for my Docker containers which accesses the NAS data through NFS.
I have a base Debian template with a few tweaks I like for all my machines. Debating setting up something like terraform but I just don't spin up VMs frequently enough to wan tto do that. I do have a few Ansible playbooks I run on a fresh server to really get it to where I want though.
Generally, it’s Proxmox, debían, then whatever is needed for what I’m spinning up. Usually Docker Compose.
Lately I’ve been playing some with Ansible, but it’s use is far from common for me right now.
raspberry pi, arch linux, docker-compose. I really need to look up ansible
Debian + docker-compose
I'm currently Ubuntu, but if I were to start from scratch this would be it, simple, basic, does everything I need.
I setup my bare metal boxes and vms with ansible. Then I use ansible to provision docker containers on those.
A bunch of old laptops running Ubuntu Server and docker-compose. Laptops are great; built in screen, keyboard, and UPS (battery), and more than capable of handling the kind of light workloads I run.
I use a heterogeneous environment with some things hosted in various cloud providers and others locally. Often times, I can usually find the package I need - but if I can't, I usually go for Docker and docker-compose. This is often the case in Oracle Linux on OCI - where docker just makes things so much easier.
For my static stuff I just use Cloudflare Pages and forget about it.
On my homelab it is Arch Linux with my own set of scripts. I used to do VFIO gaming a lot (less now), so I had the host only be a hypervisor and used a separate Arch VM to host everything in a docker-compose stack. The VM makes my server operations a lot more tidy.
My RPI is using dietpi and is natively running the pihole software and a couple other things.
I know some folks swear by UnRaid and Proxmox, but I've always found those platforms limited me vs building things my way. Also borking my own system unintentionally on occasion is a thrilling opportunity to learn!
If doing a fresh server external, I'd go for debian as base(don't need to update it too often + stable)
For apps it's mostly docker-compose to set up portainer/nginx-proxy then from ther just manage the rest from portainer/nginx-proxy web-ui. ony log on the server for the occasional docker updates / pruning for space.
I see a lot of guys going the full kubernetes route and it's something I'm hoping to get into at some point but it seems like a lot to unpack for now.
Debian netinst via PXE, SSH/YOLO, docker + compose (formerly swarm), scripts are from my own library, Debian.
I do the same except I boot a usb installer instead of PXE.
I can never find a USB drive when I need one, thus my PXE server was born. lol
Xen on Gentoo with Gentoo VMs. I've scripted the provisioning in bash, it's fairly straightforward - create lvm volume, extract latest root, tell xen whick kernel to boot.
Ideally would like to netboot a readonly root off nfs and apply config from some source. Probably bash :D
Some things like opnsense are much more handcrafted because they're a kind of unicorn compared to the rest of the stuff.
That’s impressive effort for a home lab.
Hi jago,
Sorry for the delayed response. I do also see the inconsistency between looking at your post directly vs lemmy.ml. I have noticed, however, that every now and the lemmy.ml throws a bad gateway error, which would imply it's getting overloaded again. That might create situations where lemmy.ml has all comments marked as federated, while some of them were actually dropped mid transit. Same applies to lemmy.one.
I don't know of any workarounds for that, unfortunately. Feels a lot like a bug.
In regards to subscriptions - you're right, the pending state does seem to actually impact federation. Some of my subscriptions to beehaw have been pending since day one but I can see the content just fine. I've written this off for another bug in the software.
I had a look at your profile - I can definitely see the posts you've created as well as the comments. I've noticed some UI bits fail to get refreshed - things like notification status, etc. I found forcing a page refresh helps with that.
Sorry I couldn't be of more help.
Proxmox and shell scripts. I have everything automated from base install to updates.
All the VMs are Debian which install with a custom seed file. Each VM has a config script that will completely setup all users, ip tables, software, mounts, etc. SSL certs are updated on one machine with acme.sh and then pushed out as necessary.
One of these days I’ll get into docker but half the fun is making it all work. I need some time to properly set it up and learn how to configure it securely.
I'm all in on docker-compose + rootless podman. Definitely not no issues, but I've got the hang of the kinds of issues it presents at this point. They're mostly around SELinux and networking, though generally the networking only gets problematic on exotic compose setups - jitsi was a huge pain for me.
Raw server with SSH and an immutable OS too. I'm using fedora IOT for my homeserver, and apart from some initial issues with GPU drivers because of layering issues (now working) that's been basically flawless.
I was on OpenSuse MicroOS, but I had huge problems with BTRFS and decided to give it up in favour of EXT4 + XFS. That necessitated moving distro, because MicroOS uses BTRFS snapshots as the basis for its auto-updating/green/blue system. Fedora IOT uses rpm-ostree instead, and works on any filesystem.
For a while I tried to run k8s (k3s mostly), then I did run nomad for a while. Now I am just running docker compose on Ubuntu (still have one box running Proxmox, but that will be decommissioned eventually, and mostly just runs one VM running Ubuntu).
I am building a few things to solve specific problems I have with this:
Most of my server hardware is oriented toward having a bunch of disks plugged into them (I am 100% guilty of being a data hoarder), and I am running gluster to glue that all together, so that is something I install onto the servers to share their physical disks and/or mount the logical disks.
I use SSH to manage docker compose. I'm just using a raspberry pi right now so I don't have room for much more than Syncthing and Dokuwiki.
Don't underestimate a pi! If you have a 3 or up, it can easily handle a few more things.
I forgot to mention I also have a samba share running on it and it's sooooooo sloooooow. I might need to reflash the thing just to cover my bases but it's unusable for large or many files.
Fedora-server with Podman and Quadlet on btrfs drives. Although I must admit I often use rootful mode in Podman as it works better with Containers made for Docker. Ah and you might want to turn off SElinux in the beginning as it can get frustrating fast.
I use Proxmoxn then stare at the dashboard realizing I have no practical use for a home lab
So i'm not alone. I am trying to better myself.
Usually Debian as base, then ansible to setup openssh for accessandd for the longest time, I just ran docker-compose straight on bare metal, these days though, I prefer k3s.
Up until now I've been using docker and mostly manually configuring by dumping docker compose files in /opt/whatever and calling it a day. Portainer is running, but I mainly use it for monitoring and occasionally admin tasks. Yesterday though, I spun up machine number 3 and I'm strongly considering setting up something better for provisioning/config. After it's all set up right, it's never been a big problem, but there are a couple of bits of initial with that are a bit of a pain (mostly hooking up wireguard, which I use as a tunnel for remote admin and off-site reverse proxying.
Salt is probably the strongest contender for me, though that's just because I've got a bit of experience with it.
For me it’s Ubuntu Server as the OS base, swag as reverse proxy and docker-compose for the services. So mostly SSH and yolo but with containers. I’d guess having something like Portainer running would probably be useful, but for me the terminal was enough.
As folder structure I just have a
servicesdirectory with subfolders for each app/service.NixOS instances running Nomad/Vault/Consul. Each service behind Traefik with LE certs. Containers can mount NFS shares from a separate NAS which optionally gets backed up to cloud blob storage.
I use SSH and some CLI commands for deployment but only because that’s faster than CICD. I’m only running ~’nomad run …’ for the most part
The goal was to be resilient to single node failures and align with a stack I might use for production ops work. It’s also nice to be able to remove/add nodes fairly easily without worrying about breaking any home automation or hosting.
A series of VPSes running AlmaLinux, I have a relatively big Ansible playbook to setup everything after the server goes online. The idea is that I can at any time scrape the server off, install an OS, put in all the persistent data (Docker volumes and /srv partition with all the heavy data), and run a playbok.
Docker Compose for services, last time I checked Podman, podman-compose didn't work properly, and learning a new orchestration tool would take an unjustifiable amount of time.
I try to avoid shell scripts as much as possible because they are hard to write in such a way so that they handle all possible scenarios, they are difficult to debug, and they can make a mess when not done properly. Premade scripts are usually the big offenders here, and they are I nice way to leave you without a single clue how the stuff they set up works.
I don't have a selfhosting addiction.
I usually set up SSH keys and disable password login.
Then I git-pull my base docker-compose stack that sets up:
I have a handful of other docker-compose files that hook into that setup to make it easy to quickly deploy various services wherever in a modular way.
Kubernetes.
I deploy all of my container/Kubernetes definitions from Github:
https://github.com/JustinLex/jlh-h5b/tree/main/applications
I try to have most of the common parts setup with ansible. Over time, keep adding more and more. This is useful specially for things you may not do, or need, often and that is not as fresh on your mind how you set it up last time.
Any configuration management system would work; I find ansible is very approachable and fast to get productive with it.
Proxmox + mostly Debian + currently documenting my builds for future automation.
Lots of snapshots and clones/backups, for in case I want to roll back, or in case I want a head start in the future.
For example, I have a couple LAMP stack VMs backed up. If I need another LAMP VM, I clone (restore-as-unique) the backup in Proxmox, twiddle a few settings to make it actually unique, and go.
I don't do Docker or anything like it currently, and eventually I'm sure I'll learn, but having a crapload of VMs (true VM or LXC) suits me just fine for now. I will likely learn how to do my deployments with Ansible before learning Docker et al.
I have a bunch of different stuff, a dedicated server with Debian, 4 raspberry Pis + 1 micro computer that acts as a LB/Router/DHCP/DNS for the Pis.
In general I would say that my logic is as follows:
apt install Xand instead you might need to create a new playbook for it, but in the long term, it paid off multiple times. I do have some default playbook that does basic config (user, SSH key provisioning, some default packages) and hardening (SSH config, iptables).Sounds cool. ansible could never convince me, though, because playbook writing is so annoying.
Oh, I am there with you on that. I got used in my previous job, where everything was done with Ansible, but I still find myself copy pasting and changing most of the times. I actually like way more a declarative approach a-la-terraform.
Overall though there is a lot of community material, and once the playbooks are written it's quite good!
I guess if I would automate my base setups with ansible so that I have a good foundation and have learned the tool properly, I would stick to it, but it was one of the cases were I was pushed away right from the start.
Web servers: unikernel (nanos) + fastly to backblaze Mail servers: tailscale (for admin ssh), fail2ban, letsencrypt nginx, docker compose, mailu, heavy use of vps firewall Raspberry pi "servers": k3s, managed with lens desktop pro, but honestly learning kubernetes isn't exactly enjoyable and feels painful compared to a shell script that launches dead simple API deployments
I used to do a lot of Ansible to manage a lot of servers but most of the time the thing I need to fix is manual, and a setup.md in my personal notes is all I need for "how do I make a new minio backup host" or whatever.
I run Debian + Docker, and use Portainer to manage the docker stacks
Probably the odd one here with Arch Linux + docker compose with still a lot of manual labor
updating it after maximum 4 weeks is enough, container more often
For years over done an Ubuntu LTS base with docker, but I've just recently started using debian base. Moved to debian for my workstation as well.
I deploy bare-metal with a mix of Ansible and Docker Compose.
I've recently switched my entire self hosted infrastructure to NixOS, but only after a few years of evaluation, because it's quite a paradigm shift but well worth it imho.
Before that I used to stick to a solid base of Debian with some docker containers. There are still a few of those remaining that I have yet to migrate to my NixOS infra (namely mosquitto, gotify, nodered and portainer for managing them).
Cloud vps with debian. Then fix/update whatever weird or outdated image my vps provider gave me (over ssh). Then setup ssh certs instead of password. I use tmux a lot. Sometimes I have local scripts with scp to move some files around.
Usually I'm just hosting mosquitto, maybe apache2 webserver and WordPress or Flask. The latter two are only for development and get moved to other servers when done.
I don't usually use containers.
I'm better at hardware development than all this newfangled web stuff, so mostly just give me a command line without abstractions and I'm happy.
I'd like to use rootless podman, but since I include zerotier in my containers, they need access to the tunnel device and net_admin, so rootless isn't an option right now.
Podman-compose works for me. I'd like to learn how to use Ansible and Kubernetes, but right now, it's just my Lemmy VPS and my Raspberry Pi 4, so I don't have much need for automation at the moment. Maybe some day.
You can add net_admin to the user running podman, I have added it to the ambient capability mask before, which acts like an inherited override for everything the user runs.
Only ssh,nvim,htop, and screen. Rest all are whatever is required. I like to keep things minimal until i really need the server to do anything specific.
I resort to docker only if i need the application temporarily or the application setup is awkward/annoying.
I have a stupid overcomplicated networking script that never works. So every time i set up a new server I need to fix a myriad of weird issues I've never seen before. Usually I setup a server with a keyboard and mouse because SSH needs networking, if it's a cloud machine its the QEMU console or hundreds of reboots.