Multiple Backup Targets / Replication

Zoinkies

Adam Understood, still, would be nice to have on site(block storage), and off site/provider, S3 backups simultaneously.

WebInfra

Adam Will the upcoming S3 feature not support differential backup?

Adam

It's a tar.gz upload, which I believe is also what other competing control panels do. This is the first iteration, in future it might be possible to do something more advanced.

WebInfra

Adam Yes, that is correct. That being said I know JetBackup does offer differential backup with S3.

badboybackagain

WebInfra Yes, i am using jetbackup and thats correct, they offer differential backup and its a MUST requirement if we are planning to use s3 backups. This will save a lot of storage, time and cost.

Isaia-Arknet_PTY_LTD

I am not going to pretend I know it all or that I am right, but from what I remember can share:

cPanel backup is a tar.gz (it requires you to restore the full account) (fast restore as restore is copying a big file to the server and restoring it locally). I remember someone saying how good this option was as it is fast on restore.

A very bad thing about this is that every backup will push a full account .tar.gz to the backup target. It is super very slow on big accounts. Also, the Production server is under a bit of stress when backups are created. Also, a lot of bandwidth is used for transferring the backups every night. (this is a no-go for my personality) i care more about my production server being available and using the least bandwidth possible.

Jet backup was created around cpanel and other panels to allow differential backups on various targets. (the restore is slower as it is moving file by file) it is granular, and it can be faster if you know what you want to restore. You can restore only a few needed files, but it can be slower if you restore a big account with a lot of very small files

Enhance it is using a jet backup-like, (almost as at this point in time can not restore a few files only) system, and the benefit is that it can be installed on very cheap hardware and can be improved in time. And with a bit of effort, enhance can beat both previous systems.

At the moment (enhance backup) it is a differential backup, and files are moved one by one.

What can happen with a bit of effort is:
Keep and improve the differential backup.

Improve restore options by:
If a Granular restore is selected, you restore file by file. This is the path that Enhance already took.
If the full account or full Server Restore is selected, then the backup server should locally create archives and deliver big payloads for fast transfer.
This would make any other backup system obsolete. And stop the backup system war 🙂

At the same time, S3 or B2B can not compare on price with a Hetzner or OVH 4 x 10 TB server for 60 Euro price.
Dedicated servers are super affordable considering the price/tb.
It is Ok to be a fanboy of S3 or B2B, but if you do the math it is impossible and actually futile to go that path, but i am not against it, as I am not wasting my time creating tools for s3 or b2b or any other storage target.

Yes, there is some work, but with the correct direction and do it once do it right attitude, we all can win from it, fast.
If the above is wrong (it will not be for the first time 🙂 ) take only what is good, and filter the bad.
If this is possible (Adam can confirm), and if you read this and like the idea, vote it up. 🙂

george

Isaia-Arknet_PTY_LTD +1 agree!

Adrien

Isaia-Arknet_PTY_LTD I support your view. However, I prefer focusing on the technology as a main criterion (what you've explained about differential backup that would be improved over time) to pricing when it comes to backup.

When you go the Hetzner or OVH route, you need to manage scalability and availability yourself. We saw the effect of that drawback when OVH Strasbourg burned in 2021.

WebInfra

Isaia-Arknet_PTY_LTD You're not wrong but there's no one-size-fit-all for everyone as they will each have different business plans.

With a 4x 10 TB server, you will need to use RAID to ensure some redundancy. If you use RAID 5 which can tolerate 1 disk failure out of the 4, your effective usable space is down to 30 TB only and your cost is around $0.0022.

Besides that, you will need to pay for the full amount upfront which is space that you may never use, be limited to a physical hardware so it is difficult to scale and re-allocate resources along with single point of failure for many of the hardware components. You haven't taken into the account that multiple servers backing up into a single backup server can result in high I/O and overall slowness in backup.

Now compare that to an S3 solution like Backblaze (or Wasabi but that has its own caveat) which cost $0.005 per GB. Yes, the price may be doubled but you only pay for what you use (so nothing is paid upfront) and you don't have all the problems associated with a dedicated server like redundancy, scalability and performance. If your business plan allows it and you make good margins, it is not unthinkable to go this route but that's up to you.

The important thing to note is that the S3 route is only possible if and only if it supports differential backup like JetBackup and not the cPanel style solution (will be great to have choice for both).

Isaia-Arknet_PTY_LTD

Adrien Enhance in itself. It is scalable already. S3 or B2B do not help with this.
Regarding S3, we have to understand that only tar.gz will be available, and this is a big problem when backing up every night hundred of GB per night per server. For me, it is unthinkable in such a scenario 🙂
(I would not use such a backup form even if it is local from SSD to SSD) it is a waste of TBW and CPU utilisation.
It is a difference between what we think we what and what we can get in the immediate future, for remote backups.

The DC fire in Europe was bad luck, and if that would happen, it is fine. you rebuild the backup server in 30 minutes and start backing up again.

On another note, I guess that's why Microsoft tested DC underwater (in some ocean warming up) so when it burns you do not see the smoke 🙂

Isaia-Arknet_PTY_LTD

🙂 Hehe, nice choice of numbers 🙂 You make the cloud sound more affordable than it is.

In reality, b2b is 0.005; also, it is 0.01 for download on top of the storage price.
S3 is 0.026 + 0.09 for downloads.
And so on... source: https://www.backblaze.com/b2/cloud-storage-pricing.html

I also forgot to say that B2B tends to be slow on uploads compared to a dedicated server. I guess you can not overload a server like B2B and S3, as the pipe is the bottleneck. (i might be wrong on this one.)

The difference is that you can get 4TB (S3 storage) for the price of 30Tb in raid5 of a dedicated server!

WebInfra

Isaia-Arknet_PTY_LTD S3 solutions consistently perform faster than dedicated server in our experience when multiple servers are backing up at the same time (which is unavoidable when you have many servers). This is more so when you have servers in various locations around the world because S3 providers tend to have more locations than say OVH or Hetzner.

Anyway like I said, not everyone have the same business plan so if you want to save on this particular cost, that's your business plan but some of us have different business plans and margins so we can easily absorb the increase in cost for the various benefits.

ss88us

Adam Hi Adam,

Could you not look into tar and snar for incremental backups? It might not be 'ideal' but would save on bandwidth of a 20GB website every day to S3.

This backup script will create the first (large) .tar.gz, then incremental tar.gz's and then every Monday restart (make a large backup again)... you get the point:

P.S. no judging... made many years ago lol

DATE=`date +%m-%d-%Y`
DAY=$(date +%a)
 
for dir in /home/*/webapps/*
do
        base=$(basename "$dir") 
        SNAR=$base-backup.snar

        if [ $DAY = 'Mon' ]
                then
                        test -e /root/$SNAR && mv /root/$SNAR /root/${SNAR}.$(date --date '7 days ago' +%F)
        fi

        tar -czg /root/$SNAR -f "/tmp/${base}-${DATE}.tar.gz" $dir

done