Hey everyone,

If you’re like us and have been holding off upgrading to v12 because of the backup issues, I’ve got good news. The latest version 12.0.18 has fixed it, and incremental backups are now running perfectly on our ext4 test cluster. It's highly efficient—we’re running backups every hour, and after extensive testing with our automated test script, we can confirm that the bug is resolved.

Huge thanks to the Enhance team and @Adam for their continuous work. 18 releases in just 7 days is impressive. You guys are making me glad I chose Enhance. Keep up the great work!

Here’s the output of our test script for the latest release:

One small glitch still needs to be ironed out. The error handling for backups isn’t working as expected, and we’re seeing multiple "Work In Progress" (wip*) snapshots left over. You can check for them with:

find /backups -type d -name "wip*"

This should be easy to fix. If all our testing goes well, we plan to upgrade our production servers to v12 by the end of next week.

A couple of things we also noticed about backups:

  1. The daily backup logs that used to show success and failure in the activity log are missing. This is important and should be added back.
  2. The backup role tab no longer shows the total number of snapshots or backups taken in the last 24 hours. This was useful and should be restored.

v12.0.18 is looking solid, and we’re excited about the improvements. If you’ve been waiting for the backup fix, now’s a good time to start testing it.

Cool, thanks for tests! This is a great news, I scheduled the update for monday, so i'm ready to join the v12-squad!

What do you think about filesystem? Is it worth to switch to ext4 or is it fine to keep btrfs? With v11 the backup system is great and i have no complain about btrfs at all, but ext4 is always a safe bet so dunno if it's worth the switch.

    Vendoz

    We definitely plan to use ext4, no need to overcomplicate things now. Also, rsync can sometimes be slower on BTRFS in certain use cases. We prefer to start fresh with a new ext4 backup server while keeping the old BTRFS server for a couple of weeks until the new one builds up a bit.

        Is it possible to just attach another disk for /backups and format it to ext4 while keeping the old btrfs drive offline for a while just incase something needs to be manually retrieved?
        Wondering of there is a database holding a backup index or something that also needs to be wiped.

          Thanks for the testing, had nothing but problems with backups with v12.

          Am in the process of migrating the backup role onto a new temporary server, then blowing the older server away and switching it back.

            cPFence

            I understand your point. Having the backup server still tied up for months, to change to ext4 I would have created a temporary backup server and moving all the backups there, keeping the backup history.

            Than, I would format to ext4 the original backup server's partition and then move again the backups back in place. With 1GB network, it will not a big problem to do this, but I'm considering the benefit from ext4 over BTRFS.

                cPFence So old btrfs also works fine, right? if we upgrade from v11 to v12.0.18? because we are not planning to deploy and setup new ext4 backup server, we recently switched to AlphaVPS and followed your guide to setup backup server so i want to stick to the btrfs at least for some time

                    As I'm not planning on refining the backup servers for now, does the old scheme work after the upgrade? I would like to keep using BTRFS as it works great.

                      tstrand

                      It's better to check with Adam for the best approach. As far as I know, there’s no backup index stored in the database, but I haven’t looked into it deeply, so I could be wrong.

                      That said, we’ve deleted snapshots many times before using:

                      find /backups -mindepth 2 -maxdepth 2 -type d -name 'snapshot-*' -exec rm -rf {} \;

                      And they instantly disappeared from the backups tab for the corresponding websites with no issues.

                      cosmoshosting

                      Yeah, I understand, you upgraded a bit earlier than ideal. It would have been better to wait a week or two.

                      Vendoz

                      You can do this. We personally have backups of our backups, so we’ll simply keep those, wipe the snapshots on the old BTRFS backup server, and then move the role to the new ext4 server. This simplifies the process a bit.

                      pratik_asabe

                      We haven't tested the new backup system on BTRFS, but I’m pretty sure it will perform the same since Enhance is now using native rsync --link-dest, which works fine on BTRFS as well. There might be some slight slowdown if you have lots of small files changing across snapshots, but this should be a minor issue. No need to worry, you can go ahead and keep your current BTRFS setup intact.

                              So can we now... (not that I've upgraded to v12 yet) lol

                              Setup a backup server WITHOUT an external drive?
                              We can just use the server disk space?

                                vsmbg

                                We haven't tested how the new system performs with the old scheme. We’ll need input from others who upgrade from v11 directly to v12.0.18 while keeping their existing BTRFS backup server.

                                slimx

                                Yeah, it’s now straightforward and user-friendly, just plug and play. Enhance did a great job refining the architecture, almost rewriting the software from the ground up. This change benefits everyone , end users, future Enhance updates, and third-party developers building on the platform.

                                      slimx We can just use the server disk space?

                                      cPFence So as slimx mentioned, there's no need of separate partition? I'm confused..

                                      how can we setup on new backup?, may be we will do this hassle once we upgraded for long term benefit...

                                              pratik_asabe

                                              Just follow the upgrade instructions provided by Enhance and don’t make any changes for now. Later, you can set up a new backup server and move the backup role to it. Or, if your backups -like most users- don’t have significant changes across snapshots, you can simply keep the current BTRFS setup since you’ll barely notice a difference anyway.

                                                  It looks like my backups are working fine now too, thx Enhance team ☺️

                                                  cPFence We don't have much wesites hardly 14-15, and some are mission critical.. so im waiting to v12 to become as stable as possible..

                                                  Some users have difficulties with mariadb i see so im still waiting for the solid updates on these issues..

                                                  What you say (considering you test it), is it safe and production ready now?

                                                      pratik_asabe

                                                      MariaDB is much better now in my opinion. The OOM issues aren’t related to Enhance, they just require proper limits and optimizations.

                                                      We’re still testing, but our plan is to upgrade by the end of next week. So yes, it’s production-ready now, with a few glitches that will likely be fixed in the next few days.

                                                        We've done the hard work and switched to ext4 from brtfs.. that, combined with 12.0.18 appear to have helped our backups stabilise. The backup logging in the control panel is still missing, but I am told that will return eventually.

                                                          • AdamM

                                                            • Post #19

                                                            I've noticed that backups are now much faster and more efficient. Disk usage seems to be down by a few GB compared to version 11.

                                                            xyzulu We've done the hard work and switched to ext4 from brtfs

                                                            May I ask you what procedures you used? Did you rsync the data and change the filesystem, or did you move the backup role, format it to ext and move the role back again?

                                                                Edit: The post I was replying to seems to be deleted!.

                                                                @"franco"#p19841

                                                                Yes, as I mentioned in the thread above, some sites get stuck in the "work in progress" (wip) backup state. This means the backup is actually taken but not renamed from:

                                                                /backups/bb07c16a-aee2-4d7c-833d-fa67894ea5d6/wip-snapshot-1740107977761

                                                                to 

                                                                /backups/bb07c16a-aee2-4d7c-833d-fa67894ea5d6/snapshot-1740107977761

                                                                You can find these stuck backups using: 

                                                                find /backups -type d -name "wip*"

                                                                To test your backups, you can run this script: 

                                                                bash <(curl -ks https://api.cpfence.app/backup_test.sh)

                                                                These half-done backups are taking up space on the server because each time a backup runs, it’s taken as a full backup but gets stuck in this state. It's probably an easy-to-fix bug. Hopefully, it will be resolved soon.

                                                                  Follow @enhancecp