New server for the family, Proxmox or TrueNAS, LXC or Docker?
cross-posted from: https://sh.itjust.works/post/39436160
cross-posted from: https://sh.itjust.works/post/39436154
Hello everyone, I'm building a new server for the house, it will act as a NAS for everyone and host a few services like paperless, immich, baikal, jellyfin, syncthing probably navidrone, etc. The main reason I'm building a new one is that my current one is a HP prebuilt with a 3rd gen i5 and 8GB ram that is slowly beating the bucket, my 4TB HDD is completely full and there's no more sata ports nor space in the case.
I am fully psychologically prepared to be 24/7 tech support, but after all I already am, and in this way I have to support services for which I know how they work (and that I trust!) and not some strange Big Tech service whose UI and inner workings changes every other day.
For reference my new build is:
- CPU: Ryzen 5 PRO 4560G + stock cooler. Has integrated graphics, can use it for Jellyfin transcoding.
- RAM: Corsair Vengeance 2x8GB (from my desktop before I upgraded to 64GB RAM. If needed in the future I will upgrade the capacity and probably switch to ECC, I've chosen the CPU since it has support for it)
- SSD NVME (boot+VM storage): Verbatim VI3000 512GB
- Storage (SATA): 4x12TB Seagate Enterprise (White label) to use ZFS and Raid Z1 + 1x512GB Samsung SSD as cache.
I'm planning on using proxmox on bare metal and spin up VM/containers as needed, for which I'm wondering:
I know proxmox can manage ZFS arrays, is it better to create the array via proxmox, then share it as needed via something like openmediavault in vm/container OR to create a TrueNAS VM and passthrough the SATA controller to it, then manage everything via TrueNAS? I've done the latter in the past on another server, it's holding strong
I don't know if exposing the server to the open internet is a good idea (of course with fail2ban and a firewall properly configured) or to just keep a VPN connection to the server always open. I think the latter would be more secure, but also less user-friendly for parts of the family. I'm using wireguard currently to remote into my server when needed, and sometimes networks like eduroam in my university block it completely.
- Self signed SSL certificates might also be a problem in the latter case
Since I will experiment with this server a little bit, I was thinking of keeping:
- One VM for services for the family (exposed to internet or VPN)
- One VM for services I still want to expose (I currently expose a couple websites for friends with data archived in my NAS)
- One VM for me to experiment with before going in "production" for the family
Each VM would host its services using Docker+Portainer. My question is: is this too convoluted? Should I just use proxmox's LXC containers (which I have no experience with) and host services there?
I was also thinking of spinning up a pfsense/opnsense box and put the server into a separate VLAN from the domestic lan. But that will be a project for a second time. Unfortunately the way ethernet is wired in my house and for the physical space I have available prevent me from separating the networks by physically using another router.
Thanks!
I'll just suggest my setup. It's highly segmented - and I believe your reasons against segmentation are not too strong.
I run a very small proxmox server with a zfs raid over 2 SSD/NVME, because I dread setting it up all again from scratch if a disk fails. I run everything in dedicated, unprivileged LXCs. My reasons: very lightweight, very secure (arguably less secure than VMs for shared kernel).
My two main containers are the reverse proxy and the firewall. The firewall owns 2-3 plain interfaces for the various WAN connections (currently cable and 5g as backup, fiber soon[tm]) and one vlan supporting interface with about 20 interfaces defined on it. I threw everything into /etc/network/interfaces.d/$service.conf, so my interfaces have names like the connected services which run in lxc containers by the same name. makes it easy to write the policies and manage the interaces. ifup frigate, iptables -A FORWARD -i frigate -o wan+ -j ACCEPT (or whatever)
every lxc runs in its own /30 subnet, only having the firewall local. There are a few bigger networks at home, iot (mainly wifi, client isolated), family for our laptops/tablets/mobils, guest network, tor-access wlan and other stuff. So for anything to work, I have to allow it. Family is mainly open to the internt, my devices have some admin connections allowed, TVs can only access jellyfin, many iot stuff is denied internet.
i recently switched from isc-dhcpd to dnsmasq, mainly for their 1-line syntax for pools and reservations, as i do this manually with vim.
In my home assistant I have a dashboard where I can configure individual VPNs to individual client devices, which are brought up on demand and use ip policies to route wan packets via those.
It's a bit of a hassle to make hardware stuff work (gpu acceleartion for jellyfin transcoding, frigate needs my usb-coral-ai and the GPU as well, vpn containers need special devices) but it's doable. and 90% of services need nothing special configured.
I got an old hp 4-external-3.5"-bay server gifted, I run openmediavault on that. it mounts a few exports on the proxmox host, which passes it to the relevant LXCs (jellyfin, frigate and a few more). It also pulls the whole zfs pool every night via rrsync, and takes a snapshot afterwards (actually right before copying new data). It pulls backups from my home proxmox and two more proxmox. For this possibility alone I love the containers - compared to VMs at least.
I have an additional external usb-sata dock, where I copy the most important data once a week or month or such on rotating drives.
I run a very similar setup (container, firewall, reverseproxy) on all three of those proxmox (one shared with friends in a datacenter, one running my (small) company services). They are all in different iterations :D I started with apache2 for reverseproxy, and netfilter/iptables, at my company, which is only a few years of age, it's nft and nginx
On all systems most virtual hosts require a client certificate. The only unprotected ports/vhosts for my company are smtp and our nextcloud instance. at home it's jellyfin on a quite random hostname.
oh, and at least for the home one: put a raspberry pi (or similar, maybe even nanoKVM) into your rack, and connect it. it's awesome for maintenance. I have a rs232 connected pi and can control from bios, bootloader, shell of my proxmox.
This is basically what i ended up doing in one year of experiments, building and rebuilding.
Proxmox with unpriviledged LXCs. The only VM runs OPNsense, isolates services into 3 different vlans: internal for LAN/VPN only services, external for a couple public websites I host, and auth for authelia and lldap. Only internal can reach the others always through a reverse proxy lxc, the other vlans are blocked in opnsense and can only reach outside internet for eventual updates or api calls. No services can reach the 192.168.1.0/24 subnet with all the home devices (no smart home stuff).
Bulk storage is on a 4x12TB raid z1 array. The NAS dataset can be accessed via a cockpit lxc, with samba, the only one that sits on my LAN. Some services (like forgejo, immich, paperless) use their own dataset on the same pool.
Apart from cockpit, only opnsense sits on my lan as well. All the lan devices pass through it+reverse proxy to reach the other services.
Snapshots and backups are handled with zfs_autobackup and proxmox backup server, which runs on the repurposed old server, on which i also backup a subset of important files.
I was eyeing the nanokvm, pcie version, for that exact reason. My Pi runs pihole, but is otherwise separated from the server.
I was also looking at setting up a router/firewall with another minipc and a 2.5G pcie ethernet switch. Kinda long term project
Oh lol, didnt notice the age of your post :D makes the community look rather dead :(
Congrats on the setup, sounds solid!
I allow my internal api calls to be clear text & direct... Not perfect but its a tradeoff. Allows me to not let the proxy access everything.