Enhance v12.0.0 Released 🎉: Page 22 - Enhance Control Panel

Enhance v12.0.0 Released 🎉

anturk

Update: Okay pushing staging site to live seems to work now i guess? Maybe i did something wrong.

But other issue still exist when pushing staging site to live on existing site making a back-up and restoring it doesn't work anymore on that site. This also happens when you delete the existing site and push the staging to a new site.

Normal backup/restore on sites that didn't come from staging stage does still work.

Edit: Made a ticket for this

cPFence

AdamM SharedGrid @8Dweb

We tested the new backup system in v12.0.16 today and noticed inconsistencies with how backups are being taken. Specifically, incremental backups don’t always work as expected, it’s hit or miss, and we haven’t been able to identify a clear pattern.

Upon further investigation, we found that the new Enhance backup system is sometimes creating full copies of files instead of using hard links for unchanged files. We checked the inodes of identical files across multiple snapshots for a test WordPress site (with no changes made), and the results showed that most snapshots had different inodes for the same files. This confirms that in many cases, files are being fully copied instead of hard-linked, leading to higher disk usage than expected.

We’ve seen this behavior for both automated and manual backups, though occasionally, a snapshot will correctly use hard links. It seems inconsistent.

AdamM

cPFence Yes I noticed that too, disk usage is increasing too fast compared to version 11.

xyzulu

cPFence Yes, I have reported some of this to Enhance via a ticket as well. I'm sure they will sort it out. I am thinking about resetting the backups.. ie set retention to 1 day.. let them all delete and then see if things look better after that.

Side point, now backing up your backups requires copying/syncing etc the 'current' folder... which is a symlink, so needs rsync -L (if using rsync). I'm currently testing the best way to backup my backups to another server.

mbe81

cPFence
Did you have a separate mail server in your test setup? See my earlier post: https://community.enhance.com/d/2415-enhance-v1200-released/343

This was confirmed by Adam and they are working on a fix.

LuisMiguel

It happens to me (in a new installation from scratch) that the backups are saved and I can restore them, but if I try to change to another backup server I get an error and the copies become unusable...

CloudyBlake

xyzulu I did this, doesn't seem to work any better. (set to 1 day and let them age out)

cPFence

xyzulu

Yeah, we’ll probably write a new script for backing up the backups and share it with the community since the old BTRFS script we shared earlier is no longer useful for v12 users.

mbe81

No, we don’t have an email-only server, and we tested it on an ext4 setup, not BTRFS. I’m sure the Enhance team will fix this backup issue soon.

anturk

xyzulu

Do you use rsync to just make another copy for "just in case" scenario?
Didn't think about using rsync since there is already S3 integration and the backup server is on a separate server.

xyzulu

cPFence Yeah, we’ll probably write a new script for backing up the backups and share it with the community since the old BTRFS script we shared earlier is no longer useful for v12 users.

I think you will want to use rsync.. we have the working now on v12 backups. It's a little messy (--include, --exclude etc) but it's working for us now.

cPFence

xyzulu

Yes, rsync fits this, but you’ll probably want to use -H instead of -L to preserve hard links. I haven’t fully dug into the structure yet, but it’ll likely be something like:
rsync -aHAX --numeric-ids
That should handle hard links, permissions, and everything else properly.

xyzulu

cPFence I've not used -H before.. will need to compare the differences with what I currently see.

cPFence

xyzulu

It’s a must if you want to handle the hard links properly. Also, no need to bother with excludes, just mirror the full backups folder using the command above, and you’re good to go.

At least there’s a positive side to moving away from Btrfs, backing up the backups is now much easier and more straightforward.

xyzulu

v12 upgrade note (I have opened a ticket): There are a number of logs/large files in /var/local/enhance that should be removed (in my opinion) after the upgrade. Perhaps this was already planned? In any case, until we can move the control panel role (waiting for documentation) keep an eye on your control server disk space.

Edit: I've had a reply to my ticket.. not as much of a concern as I thought.. but still, there is still logs files in there that are not needed (if space is an issue for you).

xyzulu

cPFence it’s still a bit of a mess (disk usage, leftovers from btrfs etc).. but perhaps once the backup bugs are sorted it will be a little more straightforward.

PS our backup of the backups might be different to your use case. We’re seeing only copying the ‘current’ folder using -L as being suitable to our needs. But, we will continue to explore our options.

cPFence

xyzulu

We’re planning to start fresh on ext4, it’s not worth the hassle. We’ll keep the old Btrfs backup server running alongside the new one until the backups build up, then retire Btrfs for good.

And yes, if your goal is just to backup the current files, that’s a different story.

xyzulu

Screenshots working for others? They are not working for me. I have reported it as a bug.

anturk

xyzulu

Some sites it works and on some it doesn't but the ones that do are maybe because these came from with v11 to v12 upgrade.

Kosta

xyzulu on old upgraded cluster my screenshots working but on my new cluster where I installed v12 they doesn’t seems to work yet.

xyzulu

cPFence We’re planning to start fresh on ext4, it’s not worth the hassle.

I still think btrfs offers some benefits for this kind of file storage (with some limitations as well). We will stick with btrfs for now, but if we run into any issues, we will switch to ext4 (reformat the attached hard disk) for our /backups partition (now you only need to specify a mount point, not a device).

Thinking out loud: I wonder how Enhance would cope if you deleted the current /backup partition and reformatted? Really, the correct ownership and users etc should be set with the first backup sync.. I'd probably check with Enhance first if we did try this in the future.

xyzulu

anturk pressing the refresh screenshot works for you?

anturk

xyzulu

Nope i only see hourglass on the ones that don't have a screenshots and the ones that have screenshot don't update to a new screenshot of the site when i press refresh

Kosta

xyzulu its keep spinning to update on upgraded cluster

mbe81

xyzulu

Yes, I have it working. It was a problem with ownership. I used the following command to fix it:

chown orchd:orchd-redis /var/www/control-panel/screenshots/*

xyzulu

anturk Do you use rsync to just make another copy for "just in case" scenario?
Didn't think about using rsync since there is already S3 integration and the backup server is on a separate server.

We use rsync to push a snapshot of the Enhance backups to another server in our local datacenter.. and then we back that server up each night to a remote location. So, we have backups and a copy of the backups stored elsewhere. rsync works well for this purpose and it's fast since the data is all local.

« Previous Page Next Page »