I’ll be honest, if disaster struck I’d lose a lot of data. I don’t have proper backups in place. This is part 1 of my backup journey, in this one I’ll tackle virtual machine backups.

Firstly, I don’t really care if something happens to a VM itself. Everything exists in IaC and can be quickly rebuilt. Most applications either run in Kubernetes or I have an ansible playbook to spin it all back up. I do however, care about config files or databases.

Tools

I had a look at some tools that could get the job done. I considered a plain rsync script but I’d need to take care of de-dupe, expirations, etc manually. I’ve initially implemented Restic purely because it seemed simple to me. I will take a look at Borg if I run into problems with Restic.

The setup

As part of my provisioning playbooks, I’ve added Restic to the software installation steps. This means all servers in the fleet have Restic installed on them. Each server has an /etc/backup-paths file. This file is just a simple list of files to be backed up. This is because each server will have different requirements. It also makes it easy to manage with Ansible. At 2am every night Rundeck will run a backup job that runs Restic on a loop using the path file.

#!/bin/bash
while read p; do
RESTIC_PASSWORD=@option.RESTIC_PASSWORD@ /usr/bin/restic -r sftp://truenas:/mnt/tank/backups
restic_backups backup $p; 
done < /etc/backup-paths

Limitations

Everything seems to be running ok. The only thing that frustrated me is how Restic deals with symlinks. Initially I wanted to symlink files and directories into to a single directory that Restic would backup. The problem is Restic will backup the actual symlink instead of following it, there are a couple of issues on their github page but it hasn’t been looked at yet.

All in all I’m happy with this setup for now. In the next one I’ll look at backing up files from the NAS.