I’m working on the next revision of my homelab backend. Currently I rely on an Ubuntu server with ZFS on Linux for file storage over Samba, and a separate Proxmox Virtualization Environment (PVE) server (the Minilab) with local LVM storage for virtualization. Ideally, I’d like to add a backup server to the mix, with its own storage, that can handle both the Proxmox server and Samba shares. However, the big choice ahead is what software to use for backups - TrueNAS (CORE or SCALE), or the newer Proxmox Backup Server (PBS)? Both have their advantages and disadvantages, and I’d like to explore them before deploying one as my new backup server.

My Life in Storage, A History⌗

When I started building my network, I relied on the original Raspberry Pi’s, each with a USB hard drive and each storing some files. I’d essentially manually distribute files across the Pi’s, and remember where data was stored. This was pretty terrible for my own data, but a lot of this data was media which would be indexed by XBMC (later KODI) so I didn’t really need to remember which drive it was on. I had multiple Pi’s since the original Pi had a pretty terrible processor and 100Mbit Ethernet, so using more than one meant it was possible to (on average, depending on where the files were located) access data at a reasonable speed from multiple clients. This was not a great solution, but it worked pretty well for me at the time.

Eventually I upgraded to FreeNAS, and built a new server for this purpose. I used an AMD Bulldozer based APU on an ITX board, using a pair of USB sticks to boot (at the time this was somewhat common for FreeNAS), onboard SATA for the drives, and a PicoPSU for power. This worked great for file sharing, and I was happily able to migrate all of my data from the old USB drives to the new ZFS pool. The downside to this setup was the lack of virtualization support. Back when I built this it wasn’t a huge deal, but over the years I’d try to get more into virtualization and FreeNAS (built on FreeBSD) just wasn’t great at it. What it was good at were jails (the BSD equivalent and predecessor to LXC containers), so as long as I could run software on FreeBSD, I could run it in a jail. I had a MySQL server for KODI, as well as some energy monitoring software for the first one-wire network, but it was always a struggle for me due to my inexperience with BSD at the time.

Eventually all of my USB sticks died and I got super sick of dealing with USB sticks in general (even ‘brand name’ ones can be absolutely awful quality, even for a read-only boot drive). Initially, I wanted to run ZFS on Root on Ubuntu to avoid the need for a separate boot drive and bring me back to a more familiar Linux environment (also with better software and driver support), but Ubuntu support just wasn’t quite ready for this, so I ended up getting a cheap SSD to run Ubuntu on EXT4 and then mount the ZFS pool. This had the side effect of limiting expansion of my ZFS pool, since I no longer had enough free SATA ports to add another mirror vdev. I setup this configuration, using LXC containers for my virtual environments, and it worked well. The Bulldozer processor really couldn’t handle proper virtualization anyway, and all of the software I wanted to virtualize ran in Linux, so containerization was great. I upgraded to a 10G NIC rather painlessly, and was pretty happy. I also eventually replaced the Bulldozer system with a new AMD Ryzen 3400G, so I could use the GPU for hardware acceleration of ffmpeg with Zoneminder. Being in a container, hardware passthrough is really easy (at least compared to PCIe passthrough with VMs) and all of this worked great.

After this, I started using my mini desktop (Asrock A300 Deskmini with Ryzen 2400G) as an XCP-NG and later Proxmox host, with a single 500G SSD for both the OS and VM storage (using LVM-Thin). This brings us to the present day, where I need to backup both the Ubuntu server (or its replacement) and the PVE host.

My Backup Needs and Options⌗

Currently, my ZFS mirror is essentially my only backup. I regularly scrub it, but the data on it isn’t all stored somewhere else. That said, a large portion of the data stored on the pool is either nearly worthless shortly after it’s created (such as the security camera recordings from Frigate) or is using the ZFS pool as a backup location (i.e. all of my 3D printers, HomeAssistant, etc. push backups to the ZFS pool), so the data is still stored in the original working location. I’d certainly be able to recover from a failure of the storage server without much loss, but it’s not a friendly idea. At the same time, I’m kinda sick of dealing with command line Ubuntu for managing my Samba users and shares. So, upgrades are needed again.

What data do I have?

  • ‘Bulk’ data in the Samba shares, including project files, raw video for editing, etc, some of this is very large and some is very important, but it’s mixed
  • Media, which was previously used with KODI and may be used with something like Plex or Jellyfin in the future (although realistically streaming services have improved a lot in recent years)
  • Backups of devices on the network such as HomeAssistant, and Raspberry Pi’s which push data over Samba periodically
  • Virtual machine drives from Proxmox

For me, the ideal setup would be to use TrueNAS (which is great at user management for sharing) for storage and Proxmox (which is great at virtualization), each running on their own hardware and with their own local ZFS pool doing what they do best. The challenge now is which to use for the single backup server?

  • TrueNAS has great features to push/pull from other ZFS systems using zfs senc/recv, as well as cloud sync features such as S3 which might be useful for me for very critical data. In theory, I might be able to pull snapshots from a PVE server which is using ZFS, but that’s not a feature built in to PVE.
  • Proxmox has PBS, which integrates well with PVE to backup virtual disks as well as filesystems and deal with archival, including tape backup. I’m particularly interested in tape backup although I know it’s not cost effective for me, but it’s cool and PBS seems to have a good solution for dealing with tapes.
  • TrueNAS as a backup server can also be a storage host on its own, allowing devices which support NFS/SMB to push their own backups to the backup server directly (i.e. Home Assistant can push a configuration backup to an SMB share, and this could be on the backup server instead of the working data server). PBS has no option to operate this way.
  • PVE doesn’t natively support delta backups when using local storage without using PBS, although I could store the VM disks in TrueNAS and mount them over NFS so I can do snapshots on the TrueNAS side, or use TrueNAS to pull snapshots from the PVE server.
  • PBS doesn’t support pull-style backups at all, it relies on the PBS client to push data to the backup server, so I’d need to run the PBS client on TrueNAS SCALE to use PBS as my backup system.
  • TrueNAS backup works mostly by using ZFS send/recv, so I’d need a completely separate solution to handle tape backup if I ever want to go there.
  • PBS doesn’t handle cloud/S3 backup like TrueNAS does, although I don’t plan on archiving a lot of data to the cloud, having the ability to back up a critical data share would be nice. I could of course do this entirely on the TrueNAS instance which stores the working data instead of the archival instance.

To try and answer these questions, I’m going to try and setup TrueNAS, PVE, and PBS all in virtual machines (PVE doesn’t run well in nested virtualization unfortunately) and try to see which set of interactions works best for data storage, PVE, and backups. I don’t want to virtualize TrueNAS given how much I work directly from the file shares, and I also don’t really want PVE to rely on remote storage, but those might be compromises I’ll have to deal with to get a single working backup solution.

Which backup solution would you use, and why?