As I outlined in my first blog post on the topic, I want to build a new backup server, and I want to explore the different options I have. This project included a lot of testing, and will eventually culminate in actually building and setting up a proper backup system. It’s definitely an important and often overlooked part of a homelab, or even small business networks. Ideally, I can also get a functional offsite backup working, but that might be a future project.

Small Scale Tests⌗

I’d like to back up bulk data stored in TrueNAS, as well as virtual machines and their associated storage stored in Proxmox. Ideally, the backup server will contain mostly lower cost spinning rust, store the complete contents of both the TrueNAS and Proxmox servers, and manage the long term archival and offsite backup. I have tried the software below, each has its own longer blog post on the topic and a short summary is listed here (More may be added in the future as I test more):

I have a few decisions I can make on the primary server side which affect the backup options in the backup server, such as:

  • Proxmox could use shared storage on a TrueNAS system, so backups are primarily done by TrueNAS instead of Proxmox. I could also use ZFS over iSCSI for this instead of NFS or CIFS so the images are zvols in TrueNAS instead of files on a dataset.
  • Proxmox in an cluster requires all of the backup and storage to be defined entirely in Proxmox, there is no way to reliably pull ZFS snapshots from Proxmox servers to TrueNAS in a cluster since the VM could potentially migrate from host to host.

TrueNAS⌗

TrueNAS is a great solution for network storage, but is it also a good backup server?

Data Transfer to Backup Server⌗

For TrueNAS -> TrueNAS, the feature is built in and works great. ZFS datasets and their snapshots are easily replicated from one system to another. All is golden.

For Proxmox -> TrueNAS, there are two options and neither are perfect. Proxmox supports a snapshot export, which makes a clone of the VM disk and compresses it, along with an archive of the VM configuration. This can optionally be pushed to remote storage (Even if the VM disk is on local storage). The downside to this is the TrueNAS backup server will end up with a dataset full of files, each of which are compressed VM disks. I can either deduplicate this dataset and hope it can recover some of this massive space penalty, leave the VM disks uncompressed and hope deduplication works better on uncompressed data, or waste a ton of space with the compressed images as files. The other option is the path I went down in my “Can TrueNAS backup a Proxmox host using ZFS replication?” blog post - create native ZFS snapshots of the zvol backing the VM, and pull them from PVE using TrueNAS’s builtin replication function. This works great for the VM disks, we get a history of snapshots natively in ZFS without the overhead of storing image files cloned from the zvol on a dataset, and incremental transfer of the backups. But, we don’t get the configuration file of the VM, and would either need to separately back them up or recreate them to restore.

Offsite Backup Options⌗

The options you have here are to replicate to another offsite ZFS system, or to a supported cloud storage provider. Most people would probably use an S3 compatible provider or Backblaze B2, neither of which are particularly cheap if you are storing a lot of data, but the software support in TrueNAS is good if you are using a compatible service. While I didn’t make a blog post about it, I did make a quick video about using Linode Object Storage with TrueNAS, although I only pay for this for my most critical data.

Summary⌗

In summary, TrueNAS is an excellent backup server when the primary server is also using ZFS and send/recv can be used to transfer snapshots. It’s less excellent when the primary server isn’t as cooperative or doesn’t run ZFS at all. If you are using shared storage on TrueNAS to host your PVE VMs anyway, then you can manage backups entirely in TrueNAS aside from the Proxmox configuration. If you are using local ZFS storage, then the method I tried in my blog post will work. If you are using anything else, you’re stuck with Proxmox compressing disk images and dumping them on a NFS/CIFS share on the backup server.

Proxmox Backup Server (PBS)⌗

Proxmox Backup Server is a new offering from Proxmox, and designed for backup and archival storage of Proxmox VMs, CTs, and file storage (‘hosts’). It internally stores the data using its own archival / chunk format with deduplication, and client-side incremental sending, even in cases where the client’s underlying filesystem doesn’t support snapshots. It also has support for tape archival and tape loader management.

Data Transfer to Backup Server⌗

For PVE -> PBS, the process is entirely automated in PVE. Proxmox supports scheduled backups to PBS, incremental backups, deduplication on the PBS side, and also live restores! The live restore function starts the VM as soon as it starts copying data, and transfers blocks from the server as the VM requests them. This means the VM is up and running immediately, even if it’s running slowly until all of the blocks have been read.

For TrueNAS -> PBS, the process requires us to run TrueNAS SCALE (Debian based) and run the Proxmox Backup Client through the command line via a cron job. I went through an exercise to script this backup in my blog post “Can TrueNAS backup to Proxmox Backup Server?”, where I created a shell script to backup ZFS datasets in TrueNAS to the Proxmox Backup Server. Since the backup isn’t done with zfs send/recv, the ZFS attributes and dataset properties aren’t copied as they are in TrueNAS -> TrueNAS. This means we are backing up the files, their contents, and their UNIX permissions, but not the associated ZFS dataset attributes (quotas, special properties, and dataset layout itself). We can separately backup TrueNAS’s configuration database (it’s in /data on the boot pool and copied to the system dataset nightly) as well as any ssh keys we might need to rebuilt the server, but this is something that a TrueNAS -> TrueNAS backup wouldn’t include either.

Offsite Backup Options⌗

PBS offers two options to replicate backups - replication to another PBS server (a ‘remote’), and tape backups. The tape backup management can deal with autoloaders, scheduled backups to tape, and multiple sets of tapes. It seems fairly fully featured, although I haven’t personally tried using it.

Summary⌗

PBS is the first choice for backing up a Proxmox cluster. It’s also capable of backing up Linux filesystems, but the client currently only supports x64 Linux and is only precompiled for Debian and Ubuntu. This is fine for use with TrueNAS SCALE (which is Debian based), but the backup client currently doesn’t compile for Raspberry Pi. Work is in progress to expand the client support to the Pi and it looks like a patch set is in place to build on AArch64 (ARMv8), and several other distros are beginning to compile it on for x64 for their own package repositories. The restore features with Proxmox are excellent (including live-restoring a VM!), and the Linux client can mount a backup as a filesystem to allow file recovery in a simple way. The tape archival support seems full-featured and easy to understand, especially compared to the other open source tape solutions I’ve read the documentation for.