As part of my series exploring backup options, I’d like to see if I can use Proxmox Backup Server to archive both datasets and zvols of a TrueNAS SCALE server. Why would you want to do this? In my case, I’m trying to choose the best starting point for my new backup server, and one potential option is to use Proxmox Backup Server (PBS), but I’d like to store data outside of the Proxmox Virtual Environment (PVE) ecosystem. So, I need to make sure I can setup backups from TrueNAS to PBS in a way that makes sense and isn’t too difficult to manage. Based on how both PVE and TrueNAS handle backups, I’m not going to get everything I want out of either TrueNAS or PBS on the backup server, but I’d like to know how much manual scripting I’ll have to do with whichever option I choose. This test is part of the Backup Software Project, click there for other related projects.

The Video⌗

Here’s a video of the process. Feel free to watch it before or after you read the rest of the post. Click the thumbnail to go to Youtube. Video

What Is Proxmox Backup Server?⌗

Read their description here. Basically, it’s a backup system designed to integrate with Proxmox Virtual Environment (PVE), a hypervisor. However, it really manages 3 types of backups - virtual machines, containers, and hosts. The host backup option is designed to backup the entire filesystem of a host system by using a backup client on the system, proxmox backup client. With the host backup method, PBS can archive the entire contents of a file system, and the user can view the backup archive and download individual files or mount the entire archive as a read-only filesystem. This is pretty handy, and it’s the method I’m going to use to try to backup the TrueNAS storage server.

What are the downsides?⌗

Well, TrueNAS has no integration for this, so I’m on my own with a shell script and cron job to ensure it runs. PBS deals with deduplication, snapshot management, etc. entirely on its own, so I could just run the cron job often and let PBS deal with the rapid influx of data. Another downside is that the PBS client only runs on Debian or Ubuntu Linux, which limits me to using TrueNAS SCALE instead of CORE (the traditional FreeBSD based version).

Another downside is that PBS doesn’t include mount points in the backup (so backing up /mnt on a TrueNAS system won’t backup anything, since all of the ZFS datasets are separate mount points). However, we can get around this using the --all-file-systems flag, if we actually want to backup all of the datasets as one folder, but this merges all of the ZFS datasets as subfolders instead of as separate pxar archives. You have the tradeoff between explicitly listing each dataset in the shell script (with the possibility of forgetting some and not backing them up) vs backing up all file systems into a single archive.

PBS is also unaware of ZFS on the client side, so it’s reading the live filesystem intead of a coherent snapshot and copying that. TrueNAS’s native duplication tasks work by taking a snapshot in ZFS and then copying that using the ZFS native send/recv methods, and none of that is done with PBS.

So, for a disaster recovery type backup, this seems like a functional solution, but ZFS snapshots are still much more convenient to deal with user-induced file deletion, ransomware, etc. Depending on how badly you want the PBS integration in PVE and number of datasets being backed up in PVE vs TrueNAS in your environment, it might make sense to deal with these limitations and continue using PBS to backup everything.

How do I set it up?⌗

First, I need to install the proxmox backup client on the TrueNAS system. They only have install guides and binaries for Debian and Ubuntu, and we can use this on TrueNAS SCALE (which is based on Debian Bullseye as of this writing). So, you add the Proxmox public key and sources.list entry, run apt-get update, then apt-get install proxmox-backup-client. Fairly simple.

After this, I need to write a backup script for TrueNAS to call to initiate the backup. I created a new dataset on TrueNAS specifically to hold the backup script, which is itself part of the dataset which will be backed up. This way, the script itself can be easily restored along with the data. Everything is self-contained, including setting the environment variables for the API key / secret and host of the PBS server. Here’s a version of the script I use to do the backup

#Authentication and host information for PBS server
export PBS_REPOSITORY=<apikey>@pbs@<IP or hostname>:<datastore>
export PBS_PASSWORD=<apisecret>

# Backup Specifications
SPEC=""

#Backup dataset media on pool tank
SPEC="$SPEC tank.pxar:/mnt/tank/media"

#Backup dataset backup on pool tank
SPEC="$SPEC backup.pxar:/mnt/tank/backup"

#Backup a zvol
#I don't use this personally, so I can't test it, but this is the syntax for the backup client
#Don't try to back up a zvol while it is mounted / in use
SPEC="$SPEC zvol.img:/dev/zvol/tank/volume"

#Perform backup
#You may add your own client-side encryption if you wish
#By default, the client hostname is used as the backup id in PBS
#You can optionally specify your own using --backup-id <name>
#If you want each pool to have a unique backup ID, you'll need to call the client
#one time for each backup ID, each with a different spec.
echo SPEC is $SPEC
proxmox-backup-client backup $SPEC --all-file-systems true

Finally, I need to tell TrueNAS to run the script routinely. PBS will deduplicate at the client level (so it won’t send the entire dataset over the network if nothing has changed), so it’s safe to call this essentially as often as you want and let PBS deal with pruning on its own.

To do this, I went to System Settings -> Advanced and added a Cron Job. The job calls the backup.sh script hourly as root. Root should have the required permissions to read all of the files in the zfs pool, so it should be able to archive them as well. We could also create a new user for this job, and give them read-only permissions to the entire dataset.

Restore Process⌗

The process for restoring a file backup, after rebuilding the TrueNAS system and assuming the PBS system is still fine, is to do the following:

  1. Recreate the datasets and permissions (the UNIX permissions of files and folders are retained by PBS, but the ZFS datasets permissions are not).
  2. Make a new mount point in /mnt to mount the backup using the backup client mkdir /mnt/backup
  3. Mount the pxar archive using proxmox backup client’s FUSE abilities proxmox-backup-client mount host/<hostname>/<time> <archive>.pxar /mnt/backup
  4. rsync the contents of the FUSE-mounted backup to the new ZFS pool

Conclusions⌗

This method is good for backing up data, but we lose the ZFS attributes and ACLs for the datasets and will need to recreate them before we copy files off the backup.

It seems like running backup jobs from TrueNAS to PBS is functional with relatively minor console work, if you just want the contents of the filesystem and not the ZFS snapshots to be archived. It’s not as clean as the replication tasks within TrueNAS, but backing up a TrueNAS dataset to a PBS server seems like a functional solution. You have to use TrueNAS SCALE at this point, since the PBS client only runs on Linux, but the TrueNAS community seems to be moving to SCALE over CORE anyway. It’s not a bad solution