What are your offsite backup solutions
I've been backing up to a dedicated hard disk within the same server for all my backups in case my disks fail. And as I run more and more services, the concern of disks failures grow bigger.
I'm looking for a cheapish off-site backup solution and I'm just curious what everyone does for their 3-2-1 backup solutions.
111
Comments91
I have a borg server in the office that takes backups of all my servers. Each server stores their applications backup that gets pulled into the repo. On top of that, the borg server pushes the backup to rsync.net.
All of this is monitored by my Zabbix server
Do you have any family or friends that are willing to let a small NAS sit around somewhere? Or host a friends backup and return they host your backup? For me, this approach works well and is probably as cheap as it can get. To just backup some data over the internet, any cheap old NAS will do. I have an old NAS sitting at my parents and just manually turn it on when I'm visiting. A small startup script runs rsync without further interaction and shuts down when finished.
Honestly, I don't. The vast majority of my data is just stuff like Linux ISOs that I could download again. Important documents and stuff like that take up so little space that I just keep them in Google Drive. Most of my personal project work is on GitHub. And while neither of those are technically backups, it's not a tragic loss if I accidentally delete everything.
Do you at least encrypt those documents?
No. They're not that sensitive. And if I did, I'd lose the ability to search their contents through the Google Drive interface.
I also use SpiderOak, and they say they use end-to-end encryption. That's where I keep my tax returns and other finance stuff.
It seems the desktop application for SpiderOak is proprietary, so you can't trust that it works they way they say it does. But apparently the mobile app is Free Software so maybe that one is safe to use.
Yeah it's weird, 10+ years ago or so I feel like I had SO MUCH DATA and it was always an issue. Now I really don't have anything. A few gigs of photos I guess, some various files, but that's it. I guess I used to have a lot more media like movies and porn, which I don't really need anymore.
Kopia to B2. Works great!
I just switched to Kopia and B2 a few months back and it is working great so far. I backup my machines with Kopia to a local Unraid box on my network running Kopia server. Then the Kopia repo on the Unraid box is synced up to Backblaze B2 nightly.
I'm only backing up around 200 GB of data so the B2 storage is something like $1 a month.
Came here to comment this "obscure" combination. That I use. Lol
Kopia is a solid bit of software. I run it on my VPS's, my homelab and my desktop/laptops. All to a single Backblaze repo.
Ah yes automated backups, on my to-do which I'll hopefully do before a failure (famous last words). People talking about backblaze b2. I just looked. Why not use the personal one? The one computer would just be the Nas if using it for cold storage/redundancy?
To copy a comment from reddit:
I use B2 with rclone and just backup "important" stuff on my NAS with cron jobs. I guess you could have rclone move the "important" stuff from NAS to a "burner" PC which uses Backblaze Personal.
I don't have enough data to warrant all that so I use B2 for now and I have around 50GB of data so the price is cheap
I'm currently backing up my 20TB Hetzner Storage box using a windows VM to Backblaze Personal backup. I'm using https://github.com/dokan-dev/dokany to mount the SMB share as a "real" local drive.
Wow ok, didn't realize it was that constrained! Sorry for late response, lemmy app on phone not sending me notifications or letting me link to replies (thought Jerboa was supposed to be good)
I use restic/borg (depending on servers) and push to a bunch of S3 buckets on Backblaze. This applies to my desktop, my NAS and in general my non-Kubernetes data.
For Kubernetes I wrote a small tool that...well does the same for PVCs. Packs up the data with restic (soon I hope to migrate to rustic, once the library gets polished) and pushes to Backblaze.
To give an idea of the pricing, for 730GB, with daily backups or more, I pay approximately $5 a month.
Restic is fantastic. It's just one binary, has support for various cloud services (including Backblaze which I use as well), snapshots which can be mounted with FUSE. It's really quite useful. Borg I believe is similar?
Either way, I feel like today there is no reason to use awkward rsync solutions when better tools are out that have proven themselves.
Yeah, borg is very very similar, at least in the context I use it! I agree with the praise of restic, very solid tool. It's always possible to use rsync...but to sync restic repos!
Define which data is from value. I got 68TB of data but realistically only 3 TB are from such value I maintain several copies (Raspi + SSD) and online backup. The rest of data is stored on a cheap server built at a family member and synchronized twice a year. Make sure your systems and drives are all encrypted. And test your backups and redeployment strategy.
Edited: typo
Restic to Wasabi.
I used to use Backblaze B2, until I did the maths on how much it would cost me to restore. B2 storage is cheap yes, but the egress is so fucking expensive. It would have cost me hundreds.
Wasabi storage is equally cheap, and restoring won't cost me an arm and a leg.
I use the following scripts for Restic: https://gitlab.com/finewolf-projects/restic-wrapper-scripts
wasabi is cheaper than B2 unless...
The 3 points above are how they can not charge egress for the majority of people.
Remember, this is for an offsite backup scenario.
Yeah, absolutely. In my case, I backup way more than 1TB.
Which is absolutely acceptable in a offsite backup scenario. The data there is present for a long time, and if you use a solution like Restic which has deduplication capabilities, this is not an issue.
This is false. You can only download the same amount as you store in a month without violating the terms of service. That said, I've been using Wasabi in a professional manner for a number of years now, and as long as it isn't a regular occurrence, you can always contact support and give them a heads up that you do need to have more egress in a month.
This only occurs if you have to do a full restore TWICE in a month, which I had to once due to our team not noticing that the SAS controlled had failed and was responsible for corrupting data; not the drives. Support was quick, and it was no issue. Still didn't pay for that egress.
Backblaze will ship you a drive up to 8TB with your restore data on it. You pay a $189 fee which includes shipping and handling and serves as a deposit to guarantee the drive while it’s in your hands. They refund the deposit when they get the drive back. Or you can keep the drive if you like.
i use duplicati to back up configs and data for docker containers to 2 cloud services. my 8 TB server is almost maxed. i need funds to buy a backup for that and expand.
I know synology (and others probably) have an app where you can back up your data to your friends NAS and vice versa, but that's taking up their storage too and cost for HDD/SSD may be prohibitive
My home "offsite" backup is a second NAS at my parents house. I plan on getting two identical NASes with identical storage setup and let them replicate themselves automatically, but no money for that now.
I don't do 3 2 1, I do 3 1 1
How much do you pay for that OVH server?
Cool, that's a beefy machine. Way more expensive than I would be comfortable with but you do get lots of power.
If you ever get raided by the Feds they'll probably raid your friends and family's houses too so it is generally advisable to avoid using friends and family for offsite storage.
Is getting raided by the FBI something most people worry about?
Given the shit I saw in Australia during the pandemic, I don't trust the police or the government at all. I do everything I can to protect myself from them. Although I'm not worried, I do take steps to protect my data.
If you host Tor nodes, maybe.
Only if you know no government has ever lasted forever, and think humans are capable of great evil. Even if not..it's just best practices..think about targeted attacks, corporate espionage, vengeance, things like that.
If your data is such valuable, I’m sure you took the time to setup a complete encrypted system (LUKS).
What is the alternative?
2 spare drives and a safe deposit box ($10/yr). Swap the bank box once a month or so. My upstream bandwidth isn't enough to make cloud saves practical, and if anything happens, retrieving the drive is faster than shipping a replacement, nevermind restoring from cloud.
Of course, my system is a few TB, not a few dozen.
Backblaze B2 sync from my NAS. All my client computers use ayncthing or Nextcloud to the NAS.
This is what I did. Family member with decent internet, I have a Synology box that my server rclones into
Hetzner storage box, and just rsync. It takes care of snapshotting via auto snapshots. I costs like $20 for 1T I think. But there are cheaper options yoo
Backblaze using qnap backup software
Locally I have a mix of SnapRAID and mirroring across 2 servers. Then I use restic to backup select directories/files into Backblaze B2 cloud storage.
edit: I also have a local Time Machine backup server and Backblaze unlimited personal computer backup for my Mac.
B2 from my NAS with duplicacy. Set it up with healchecks.io to let me know it if stops, and it works without a flaw
My server is now up to 100 and something tb of storage. About 50% used. Raid 6. (Yes raid isn’t a backup. I know) Mainly media. Movies, tv, music, Books/audiobooks.
I’ve separated our media storage vs OS.
I only backup my OS and configs. It goes to an on-site nas.
If my media library dies, I’ll just slowly re-download what people want.
If I lose my os, I have one backup, other wise I’m off to work rebuilding that too.
I’m happy to pay for iCloud at this stage to backup and store sentimental or critical things.
✌️💛
Backblaze B2 for off-site storage, and restic for automated backups with snapshots
I've never considered off-site storage. You got me thinking
I have a Hetzner storage box which also has a borgbackup server installed.
Like many others here, I back up all important data from my Truenas to B2. Have a couple hundred gigs. It's like 2 bucks a month.
Crashplan can't tell the difference between local folders and NFS mounts, and they have an unlimited size backup plan per device for like $10/month. I have 1 device with NFS mounts from many desktops and my Nas. About 9TB.
Are you saying, theoretically if I had 100s of TB (I don't... yet!) on mounted drives (local or NFS shares), I could back it all up to Crashplan, and keep the retention as long as the files still exist on my device(s)? Sounds amazing, but what's the cost of restoring the data? They're not being very loud about that part on their website.
Yes. Look here, the plan is per-device, and the capacity is unlimited: https://www.crashplan.com/pricing/ . I think the restore would be extremely painful, it's not a fast pipe, but the bigger you go that's gonna be an issue no matter what.
I use syncthing to synchronise my collection of important stuff between my laptop, local server and VPS. My laptop then gets backed-up to an USB SSD using Time Machine. Granted, it’s not a proper backup, but it’s better than nothing.
For my photo collection I burned it to a BluRay (M-disc) and asked my SO to store it at work.
I’ve got two synology NASes. My current backup strategy is to backup everything between the two NASes so I have two copies of everything locally. Then I back up documents, photos, pretty much everything except TV shows and movies to Backblaze.
I have a local backup only drive for pictures and critical laptop backups and use rsync nightly. I also do rsync nightly to Backblaze for pictures. Figure if I can grab the drive I will have it stored offsite.
Duplicati to Hetzner storage. Working on replacing duplaicati with Borg. Because mono.
I use S3 sync via the cli and use lifecycle policies to manage number of snapshots and deletion.
Some cool options for moving files to different tiers like cold and glacier but I don't know enough about it or the retrieval costs to use it just yet
Take a look at intelligent tiering for a good no-frills solution! Each item automatically determines its tier based on how often you access it.
restic hourly backup to external SSD + idrive e2
Everything local is synced to NAS.
NAS is backed up to external USB-HDD with versioning (Hyperbackup).
NAS is backed up to Hetzner Storage via Kopia with versioned Snapshots off-site.
Raspberry Pi - USB HDD - borg backup - parents home :)
I run a Synology NAS and use their backup solution Synology C2. It's e2e encrypted, pretty affordable and well integrated into the system, so it was basically a one-click setup. Also, they keep old versions for 30 days, but only the most recent versions count towards your quota, which makes the space usage very predictable.
Backblaze, move everything u want to an external attached hdd and then back that up with the backblaze client
I use rsync.net with a promo that adds 1TB free to a 680GB plan for $10/mo, which is enough for me to sync all my personal artifacts nightly from my synology. I also did a NAS share swap with a friend but it's less reliable as friends are always changing things on their setups
My personal approach to offsite is to have my NAS, running TrueNAS Core, automatically encrypt and back up its primary pool directly to an S3-compatible service called Wasabi.
I've also considered setting up a small box with a 12TB hard drive at my parents' house a few miles away for ZFS replication.
I won't go into my solution here, but the only tip I'll give you is don't use cloud based storage. Restoring is slow if we're talking terrabytes, it's expensive compared to buying a disk and your data is never truly safe.
Buy another drive, backup to it and have it on a rotation schedule. I keep my "offsite" backup in the boot of my car. If I'm not at home I'm usually away with my car.
Storing a hard disk in a car or any other moving vehicle is highly not recommended. The vibration will kill your drive. There are stories of companies moving drives on trolleys across their carpark to find the data has been lost.
they are in a shock proof and waterproof containers
I have two Proxmox servers at different sites. I use Proxmox backup server to backup full vms. Addition on-site backups as well. Just had to use my backups for the first time last week. Worked really well!
I'm sure I could do better, but my strategy right now is B2 encrypted backups. I need to start storing a hard drive with the most critical data in a drawer at the office though or something so I can do a proper 3-2-1.
NAS at my parents and OneDrive
For internal "backups", I guess you could use a RAID setup or Snapraid. For offsite, I have a custom script that compress my data and shoot them to GDrive. It's not a lot of space so it's good. I don't backup media, I only export my photos to an external drive. Though to be good it shouldn't be in the same home as my server.
Gocryptfs + Rclone sync to B2
Used to use freefilesync for offsite backups, but haven't in a while. Wanted to replace that with a native BTRFS offsite sync tool like Btrbk, but haven't got around to it yet
I run a Windows 10 vm that shares a drive with samba, I borg/kopia backup everything to it, and it runs the backblaze client which then backs up to Backblaze personal backup.
Oh, that's neat one. Do you keep the VM running always to make the backups work?
Out of laziness yeah, but it could probably pretty easily be set up to turn it off and on when needed, since you can schedule when the Backblaze backs up.
Urbackup for workstations, and Proxmox Backup Server for my 2 Proxmox hosts.
Both configured with borg backups to rsync.net.
I haven't configured it yet, but I am planning on using rsync.net for my Synology as well (Which is mostly archive storage)
Systems backup to NAS via restic
NAS restic repo is stored online on a dedicated internal drive, which is mirrored to an external drive (normally kept offline in a safe when not bein synced), and offsite is a 3rd copy to Backblaze B2 using rclone.
I've got a Tarsnap account backing up my especially important data every night, which is admittedly only a couple of gigabytes of scans of important documents, hard to replace files, etc. It's doing snapshot-style backup with a backup for every day in the last week, every week of the last month, every month of the last year, and the last three years. Paying less than a dollar a month for it, so it's working out.
That stuff also gets rsync'd each night onto my NAS, which has its own automated LVM snapshot system going on along the same lines, and I'm using syncthing to mirror it onto my other PCs as a final last-ditch backup (and in case I need it elsewhere). Finally, there's an external hard drive I keep manual backups on every once in a while.
Larger datasets that aren't really stuff I want to pay for on the cloud (14 TB worth) just get stored on the NAS and a drawer full of external hard drives. Not ideal, but it's just way too much data.
I briefly wrote about this on my blog
I'm using AWS S3. I've got a script on my RPI that runs daily and uses the AWS CLI to sync my photos etc to there, and stores it as Glacier storage.
It's about US$9 per month for 800GB of storage, at that time it was the cheapest and most convenient.
You can get a 1TB storage box at hetzner.com for €3.81 per month. For about €13 you get 5TB. I was on S3 first and moved to the 1TB box because it's significantly cheaper.
How are your speeds? I tried one but as I'm in the US it was very slow, around 20 Mbps upload. With Wasabi S3 I get around 800 Mbps.
I wasn't sure if it was the distance/latency, or if the boxes were just throttled a lot due to the low price.
No idea, really. I use SCP and incremental backups, so speed is not quite an issue for me. The initial load I just started and didn't care how long it took. I am in Germany, as are my server and the storage box.
Truenas Scale to Backblaze b2 via built in cloud sync backup which I believe is rclone. Healthchecks.io warns me when it doesn’t run. Truenas emails me when an error occurs.
8TB USB 3.1 hard drive connected to a basic bitch HP Elitebook 8440p at my dad's place, and a Mac mini at my in-laws. Tailscale, fuck yes.
follow up - i never considered offsite backups for my data. i have a ds920+ that's got some big drives in it and thats all i need. but after reading this post i considered backup. what i went with was a qnap ts233, a wifi adapter, and 2 6t's that i had replaced with bigger ones on my syno. My syno is my daily driver and the new qnap sits on my work desk at the office, syncs with the syno, and backs up my photos and docs. Happy with this back up method. Thanks for the great post.
Can I ask you how backblaze quota works? I plan to do an rsync once per night, how does it costs? And If I want to explore the storage or download the database?