So, Enhance backups are wonderful.. btrfs and the way it happens incrementally.. amazing.. how it should be. Next, to level up.. I am looking to replicate the /backups btrfs volume to another location. I'd prefer either S3 (backblaze) or even another storage location via ssh.
I have my Enhance backup server in my physical office, so I have an "offline" backup so to speak already.. I'l just looking to "enhance" the backup system now.
Any thoughts?
I'd love for my Enhance Backups to reside in two (or more) locations natively.
As it is we do a near real time replication of the server and to S3 and it works... it's just not elegant.
https://btrfs.readthedocs.io/en/latest/btrfs-send.html
https://btrfs.readthedocs.io/en/latest/btrfs-receive.html
These let you do incremental (or full) backups. A script could sync the new snapshots enhance makes each time the backup runs which would replicate the backup offsite. There are also some tools I ran across that seem to keep a source and target btrfs filesystem in sync by automatically transfering new snapshots.
You can also "send" to a file. Each snapshot would be a file that is dependant on a base/parent file. You could send those to backblaze. Restore would be to restore ("receive") the base file first
@Adam Do you guys have some documentation on the best way to do this? My concern is how enhance could handle the restoration if the backup server was lost, new backup server added and btrfs-receive is run.
+1 to the idea of having a replication of the backups.
I'm not so worried about this myself.. just having the backups somewhere else is enough for my backup strategy. I could always spin up a new enhance instance, configure backups and manually copy over things in the unlikely event.
https://github.com/digint/btrbk seems to be the best tool I can find.. still not ready to set this all up myself without Enhance input though.
I use Kopia to do incremental backups of Enhance's backups. A set-and-forget routine. Kopia will send backups to two different backup servers once a day, and one of them will send its backups to Backblaze B2 once every three days as cold backups.
xyzulu You could try following as well, which are important or useful according to my test:
If you need to create a few different repos, don't forget to disconnect one repo before connecting to another.
When you are connected to one repo, use this command kopia repository status -t -s | grep token | head -1 | awk '{print $7}' to get the token. Then you can use kopia repo connect from-config --token-file=/path/to/tokenfile to connect a repo. This way, you don't need to pass your repo password through the command line.
Use kopia repo sync to duplicate one repo to another backup location.
I have tested mounting a kopia snapshot and copying files from there.
JohnB Sorry, no, I still haven't tested this.
However, I've once tested destroying the control panel server completely, then restoring it from proxmox snapshot, everything's working fine. So I'm thinking, if we keep a snapshot of the backup server's filesystem and the block storage separately, we do have the chance to get it back
We keep both a local snapshot (taken 4 times a day) and an offsite full image of our control server (well, all our Enhance servers). We have done disaster testing with restoration 2 times in the last 3 months. So far, that has worked well.
We don't utilize ProxMox, instead we use XCP-ng, but either will have good backup/restore options for entire VM.
My only concern is any database(s) utilized by Enhance itself. I could see that getting out of sync.
I will look into Kopia, it sounds like a good option.
@bgeek I'm finding kopia is taking a long time to take a snapshot (backup) of the btrfs Enhance uses for backups. Worse still (but I am not sure why), Enhance backups don't happen when kopia is running.. the logs seem to indicate the btrfs transactions cause Enhance control server to think a backup is in process and things are put on hold.
xyzulu Sorry buddy, I'm still in testing phase with Enhance, so I don't have too many websites on the cluster, totally around 10. So with such a low load, I haven't met the problem you are now facing.
And, regarding kopia's speed, according to all the reviews I've read and the results of my own tests, kopia's incremental snapshot is faster than restic and much much faster than rclone. Anyway, rclone is not a backup tool.
It's related to the btrfs transactions .. it works.. I have backed up around 300Gb just fine.. but it takes a LONG time (around 30 hrs I think) .. and Enhance backups are put on hold while it's running (though I am not why this is the case).
