Enhance backups - replication - Enhance Control Panel

Enhance backups - replication

xyzulu

So, Enhance backups are wonderful.. btrfs and the way it happens incrementally.. amazing.. how it should be. Next, to level up.. I am looking to replicate the /backups btrfs volume to another location. I'd prefer either S3 (backblaze) or even another storage location via ssh.

I have my Enhance backup server in my physical office, so I have an "offline" backup so to speak already.. I'l just looking to "enhance" the backup system now.

Any thoughts?

jeff

I'd love for my Enhance Backups to reside in two (or more) locations natively.

As it is we do a near real time replication of the server and to S3 and it works... it's just not elegant.

JohnB

https://btrfs.readthedocs.io/en/latest/btrfs-send.html
https://btrfs.readthedocs.io/en/latest/btrfs-receive.html

These let you do incremental (or full) backups. A script could sync the new snapshots enhance makes each time the backup runs which would replicate the backup offsite. There are also some tools I ran across that seem to keep a source and target btrfs filesystem in sync by automatically transfering new snapshots.

You can also "send" to a file. Each snapshot would be a file that is dependant on a base/parent file. You could send those to backblaze. Restore would be to restore ("receive") the base file first

@Adam Do you guys have some documentation on the best way to do this? My concern is how enhance could handle the restoration if the backup server was lost, new backup server added and btrfs-receive is run.

xyzulu

JohnB @Adam Do you guys have some documentation on the best way to do this?

I'm not so worried about this myself.. just having the backups somewhere else is enough for my backup strategy. I could always spin up a new enhance instance, configure backups and manually copy over things in the unlikely event.

https://github.com/digint/btrbk seems to be the best tool I can find.. still not ready to set this all up myself without Enhance input though.

Rich

+1 to the idea of having a replication of the backups.

bgeek

I use Kopia to do incremental backups of Enhance's backups. A set-and-forget routine. Kopia will send backups to two different backup servers once a day, and one of them will send its backups to Backblaze B2 once every three days as cold backups.

Andreas

bgeek thanks for the share. I will look into this myself. Im looking for a solution to backup my backups.

xyzulu

bgeek I use Kopia to do incremental backups of Enhance's backups.

Thanks for the tip.. so you use this to "backup" the Enhance backups on the btrfs right?

bgeek

Andreas No problem!

I used to use restic. But I came across Kopia last November. After researching and testing, I'm now fully moved to Kopia.

bgeek

xyzulu Yes.

xyzulu

Oh wow.. Kopia seems amazing.. after creating the repository and connection just

$ kopia snapshot create /backups
.. is needed.

Any other tips you picked up @bgeek

bgeek

xyzulu You could try following as well, which are important or useful according to my test:

If you need to create a few different repos, don't forget to disconnect one repo before connecting to another.
When you are connected to one repo, use this command kopia repository status -t -s | grep token | head -1 | awk '{print $7}' to get the token. Then you can use kopia repo connect from-config --token-file=/path/to/tokenfile to connect a repo. This way, you don't need to pass your repo password through the command line.
Use kopia repo sync to duplicate one repo to another backup location.

JohnB

bgeek

Kopia looks great! I'm curious have you tested taking the backup server offline (to simulate failure), setting up a new backup server, restoring from Kopia backup and seeing if backups show up?

bgeek

JohnB Sorry, no, I still haven't tested this.

However, I've once tested destroying the control panel server completely, then restoring it from proxmox snapshot, everything's working fine. So I'm thinking, if we keep a snapshot of the backup server's filesystem and the block storage separately, we do have the chance to get it back

xyzulu

I have tested mounting a kopia snapshot and copying files from there.

8Dweb

We keep both a local snapshot (taken 4 times a day) and an offsite full image of our control server (well, all our Enhance servers). We have done disaster testing with restoration 2 times in the last 3 months. So far, that has worked well.

We don't utilize ProxMox, instead we use XCP-ng, but either will have good backup/restore options for entire VM.

My only concern is any database(s) utilized by Enhance itself. I could see that getting out of sync.

I will look into Kopia, it sounds like a good option.

xyzulu

@bgeek I'm finding kopia is taking a long time to take a snapshot (backup) of the btrfs Enhance uses for backups. Worse still (but I am not sure why), Enhance backups don't happen when kopia is running.. the logs seem to indicate the btrfs transactions cause Enhance control server to think a backup is in process and things are put on hold.

bgeek

xyzulu Sorry buddy, I'm still in testing phase with Enhance, so I don't have too many websites on the cluster, totally around 10. So with such a low load, I haven't met the problem you are now facing.

And, regarding kopia's speed, according to all the reviews I've read and the results of my own tests, kopia's incremental snapshot is faster than restic and much much faster than rclone. Anyway, rclone is not a backup tool.

xyzulu

It's related to the btrfs transactions .. it works.. I have backed up around 300Gb just fine.. but it takes a LONG time (around 30 hrs I think) .. and Enhance backups are put on hold while it's running (though I am not why this is the case).

bgeek

xyzulu That's a bit too long. How many files and where you are backing up to will determine the time cost. Last time I backed up around 230G for merely 2 hours, totally around 40k files, between two US servers both with 1Gb ports.