Enhance v12.0.18 – Incremental Backups Now Working as Expected: Page 3 - Enhance Control Panel

Enhance v12.0.18 – Incremental Backups Now Working as Expected

gmakhs

lamosty what I wonder now is if I can mount S3 endpoint , and use it

cPFence

Version 12.0.19 seems to have completely broken the backup system. After deleting all snapshots and testing this version using our testing script, de-duplication is completely off for all sites!!

=========================================
Summary of Backups for Enhance version 12.0.19:
-----------------------------------------
Successful (hard links): 0 sites
Full copies (no hard links): 21 sites
Skipped (empty, no files, or single snapshot): 3 sites
=========================================

Vendoz

cPFence I wanted to update in a few hours, but reading this I'm having second thoughts 😂

SharedGrid

cPFence Are you seeing this on both EXT4 and BTRFS?

rdbf

cPFence Version 12.0.19 seems to have completely broken the backup system.

Decided to run the test script on 12.0.19 on the actual backups, turns out 50% of the sites work fine and 50% don't. That's BTRFS separate partition. Can't find out what it correlates with though....

Adam

cPFence

Do you definitely have 12.0.19 on every server in your cluster? If so, something very unusual is happening. Can you send a support ticket?

slimx

cPFence
Thank you for this:
https://community.enhance.com/d/2468-serious-errors-reported-but-not-yet-resolved-in-version-v12/8

So based on what @rdbf said i ran your script and got a fair few warnings and info results.

Checking snapshots taken from 2025-02-20 12:00:00
[WARNING] Site ID = 026df538-d11d-4e8c-b243-1914b816de81 – Full copies detected (Checked 3 sample files: inc                                      remental = 0, full copies = 3) (Total Snapshots size = 1.7G, Snapshots count = 14)
[SUCCESS] Site ID = 06ec4dc7-1ff5-44b2-b6ea-ddcb20a699b8 – Incremental backups working (Checked 3 sample fil                                      es: incremental = 3, full copies = 0) (Total Snapshots size = 531M, Snapshots count = 13)
[SUCCESS] Site ID = 0847cab2-9b65-402f-8ce5-e146ab36494d – Incremental backups working (Checked 1 sample fil                                      es: incremental = 1, full copies = 0) (Total Snapshots size = 20G, Snapshots count = 14)
[INFO] Site ID = 0c804ebd-e078-423f-a107-c09b1d041bbf – Skipped (No files found in public_html in first or s                                      econd snapshot) (Total Snapshots size = 60G, Snapshots count = 11)
[SUCCESS] Site ID = 0cfea927-8070-45c9-9acf-88285bd17a22 – Incremental backups working (Checked 3 sample fil                                      es: incremental = 3, full copies = 0) (Total Snapshots size = 312M, Snapshots count = 7)
[SUCCESS] Site ID = 0e0fdf5d-89c0-48ae-a554-1a06381a4af9 – Incremental backups working (Checked 3 sample fil                                      es: incremental = 3, full copies = 0) (Total Snapshots size = 352M, Snapshots count = 14)
[WARNING] Site ID = 1248d79a-716d-4a4b-9b64-29ff749c33de – Full copies detected (Checked 3 sample files: inc                                      remental = 0, full copies = 3) (Total Snapshots size = 1.3G, Snapshots count = 12)
[WARNING] Site ID = 13231ed9-b7b1-420d-a585-098bce6a5fc3 – Full copies detected (Checked 3 sample files: inc                                      remental = 0, full copies = 3) (Total Snapshots size = 1.7G, Snapshots count = 12)
[INFO] Site ID = 14b9f910-1ec3-48fb-838b-cfe0d6b5b374 – Skipped (No files found in public_html in first or s                                      econd snapshot) (Total Snapshots size = 8.5G, Snapshots count = 13)
[WARNING] Site ID = 158ce794-1ec0-45bf-a673-5b5e2799d815 – Full copies detected (Checked 3 sample files: inc                                      remental = 0, full copies = 3) (Total Snapshots size = 1.9G, Snapshots count = 13)
[INFO] Site ID = 16fe11ee-6b13-44e6-9ef6-5ebb4815f1fd – Skipped (No files found in public_html in first or second snapshot) (Total Snapshots size = 568M, Snapshots count = 12)
[WARNING] Site ID = 18eb8537-337f-4cb1-a239-07d25d9cb6ae – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 146G, Snapshots count = 13)
[WARNING] Site ID = 1b6a381b-412c-4eff-81bd-ad45888de1f8 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 6.0G, Snapshots count = 12)
[WARNING] Site ID = 1f3a40ea-0ca8-4861-91b6-6a1ba75fbbd2 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 5.2G, Snapshots count = 12)
[WARNING] Site ID = 1ffafad1-1e45-43be-8b82-c0297f6667b7 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 12G, Snapshots count = 12)
[WARNING] Site ID = 21df7d8e-78bc-46ad-8c9f-38c91a7ffa22 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 70G, Snapshots count = 14)
[SUCCESS] Site ID = 2301bf5c-f29b-45c1-9fac-7dc1dfc2e052 – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 1.3G, Snapshots count = 14)
[SUCCESS] Site ID = 23813722-2abd-4cf7-aaf8-9f7892591874 – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 438M, Snapshots count = 7)
[WARNING] Site ID = 258c51ac-616a-4841-af76-eae6299a3306 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 2.0G, Snapshots count = 12)
[SUCCESS] Site ID = 262aea26-6433-49db-8f0b-a087750fe3a7 – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 230M, Snapshots count = 6)
[SUCCESS] Site ID = 2d7663b4-9bbb-4287-9458-00294976141c – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 518M, Snapshots count = 7)
[SUCCESS] Site ID = 330d3eef-4c8d-431b-b098-6daf5787a790 – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 2.4G, Snapshots count = 13)
[WARNING] Site ID = 33c07c18-896b-4f1d-8b32-7ff5c177e92c – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 14G, Snapshots count = 14)
[WARNING] Site ID = 350d9263-352a-46b5-916c-d4f1ea859fee – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 4.7G, Snapshots count = 14)
[WARNING] Site ID = 35e586d8-18f5-43a8-b8be-ba5c31518b44 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 979M, Snapshots count = 13)
[SUCCESS] Site ID = 3c21a002-7907-4259-9dcb-3fef80de2b4a – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 256M, Snapshots count = 7)
[WARNING] Site ID = 3d4e7052-5ff7-4123-a24f-02fad3eac62b – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 3.7G, Snapshots count = 11)
[WARNING] Site ID = 3ebc8809-18d3-4ce5-b204-7ea91c1c92a3 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 3.6G, Snapshots count = 12)
[SUCCESS] Site ID = 3eed1651-4317-4704-89a1-edf30e87ed8a – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 194M, Snapshots count = 7)
[SUCCESS] Site ID = 4136ebfd-b688-4d6d-b296-b9855095a5ac – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 483M, Snapshots count = 7)
[SUCCESS] Site ID = 4230f5cc-85c4-41dc-9137-d33fab6fdcce – Incremental backups working (Checked 3 sample files: incremental = 3, full copies = 0) (Total Snapshots size = 322M, Snapshots count = 13)
[WARNING] Site ID = 42b93735-0961-4219-b1fc-6cb8af9cbf5f – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 38G, Snapshots count = 14)
[WARNING] Site ID = 43d81831-a121-46cd-96ce-fc8b6acf8771 – Full copies detected (Checked 3 sample files: incremental = 0, full copies = 3) (Total Snapshots size = 579M, Snapshots count = 12)

And it's still running.
Safe to assume de-duplication is not working fully?

cPFence

Vendoz

🙂 Yeah, we also decided to delay the upgrade until the backup system stabilizes for a week or two. With each new version, we’ll reset the backups, test again, and update this thread.

cPFence

SharedGrid

We tested it only on ext4. Is it working well on BTRFS?

cPFence

rdbf

Make sure you change the time so you only check snapshots taken after upgrading to 12.0.19.

Example:

bash <(curl -ks https://api.cpfence.app/backup_test.sh) "2025-02-24 12:00:00"

Also, for more accurate BTRFS data, try downloading the script and replacing this line:

size=$(du -sh "$site" 2>/dev/null | cut -f1)

With:

size=$(btrfs filesystem du -s "$site" 2>/dev/null | awk '{print $2}')

Haven’t tested it on BTRFS yet.

cPFence

The script has been updated to provide accurate information and work for both BTRFS and EXT4 file systems. After testing it, the correct method is:

size=$(btrfs filesystem du -s "$site" 2>/dev/null | awk 'NR==2 {print $2 + $3 " MiB"}')

To use the script:

bash <(curl -ks https://api.cpfence.app/backup_test.sh)

This will test all snapshots taken today only.

For custom timing, use:

bash <(curl -ks https://api.cpfence.app/backup_test.sh) "2025-02-24 12:00:00"

It's open-source, feel free to download, edit, and share your testing results here. It’s helping everyone, especially those using BTRFS.

MarkD

xyzulu Not sure if this will work for v12 as we are still on v11 but this is how I check for failed backups each day

Access the postgres container
sudo docker exec -it orchd-postgres psql -U enhance

And then run
select website_id,started_at from (select website_id, started_at,finished_at, snapshot_dir_name, row_number() over(partition by website_id order by finished_at desc) as p from backups) as foo where p=1 and snapshot_dir_name is null;

This gives me the website id for any website that did not backup when it last ran. I am "guessing" for v12 you might need to use something like this to access postgres

orchd-postgres psql -U enhance

cPFence

Adam

Yes, I have a cron job set to run updates daily, and I just double-checked all servers using appcd-cli version.

I'll send you a ticket now.

cPFence

I opened a ticket but haven’t received a reply yet. After investigating further, I think I found the issue. We used to delete the old snapshots to reset the backup server for testing new versions, but we didn’t delete the 'current' symbolic link, so it was still pointing to a non-existent snapshot. As soon as the 'current' symlink was deleted, the backup system got back on track.

Sorry for the false alarm.

That said, the issue with stuck wip backups is still not fixed.

=========================================
Summary of Backups for Enhance version 12.0.19:
-----------------------------------------
Successful (hard links): 20 sites
Full copies (no hard links): 0 sites
Skipped (empty, no files, or single snapshot): 4 sites
=========================================
root@dev3:~# find /backups -type d -name "wip*"
/backups/f000be85-98a4-44ea-b8e8-013b6612ab8f/wip-snapshot-1740420578586
/backups/73fe844e-5e36-410d-8721-0b5ecb4236f7/wip-snapshot-1740420270706
/backups/73fe844e-5e36-410d-8721-0b5ecb4236f7/wip-snapshot-1740420843030
/backups/c7e4e4ac-aa4f-4285-9644-56ce687313bb/wip-snapshot-1740420864225
/backups/c7e4e4ac-aa4f-4285-9644-56ce687313bb/wip-snapshot-1740420472607
root@dev3:~#

Hopefully, this one gets resolved soon so we can finally join the v12 club.

AdamM

cPFence What do you mean if I have a “wip-snapshot” folder in each folder of the backup sites?

cPFence

AdamM

The backup system uses temporary wip-snapshot folders and then renames them to snapshot-{unix_timestamp} once completed. For some reason, some sites are getting stuck in this wip (work in progress) state. It’s an issue that needs to be fixed.

xyzulu

My stuck website was fixed by running the v12 backup locations fix tool again.

rdbf

Looks like I solved my de-duplication problem, manually.

All websites that were not properly de-duplicating, had their "current" symlink pointing to the email backup, and didn't have a "current_email_only" symlink. And somehow, every backup cycle the "current" one would be updated to the new wrong "snapshot".

So I edited all the wrong symlinks, and added the current_email_only ones to their correct folders. I'll check in after a few cycles to report on whether it now properly de-duplicates everything, and if it keeps pointing the symlinks to the correct folders. If not, it's something that will need an update/fix, but even if it now works, it ended up wrong for no reason, so it would still need an update/fix or some command to run once, as advised by Adam.

rdbf

rdbf I'll check in after a few cycles to report on whether it now properly de-duplicates everything

Already I can say that the backup is properly de-duplicating with the correct symlinks in place, but after the backup it messes up the symlinks again, pointing the "current" to the new email backup, and leaving the "current_email_only" on the previous email backup. So this is clearly a bug in handling backups with separate web and email setups.

@Adam, do you need a ticket for this? Or is it enough info already?

xyzulu

rdbf

What if you:

clear out an entire websiteid backup folder? (don't delete the websiteid folder of course)
run the v12 backup locations fix script again on the particular server that has the website in question
try to run a manual backup

Perhaps you have tried already.. but this is where my mind goes for troubleshooting..

cPFence

rdbf

He’s swamped and very busy, so any issue that isn’t reported and documented in a ticket most likely won’t get fixed anytime soon.

rdbf

xyzulu run the v12 backup locations fix script again on the particular server that has the website in question

This can only be run on the control panel, which was already done during upgrading, then by Adam himself, and again now by me.

xyzulu try to run a manual backup

Same result. Something decides some websites should have both the website and emails backed up, trigger both servers to send data to the backup server, properly de-duplicating them if the symlinks are pointing correctly, and then only symlink the "current" for the website to the last email backup, while ignoring the update for the email symlink, hence breaking the de-duplication for the next backups.

xyzulu

rdbf I hope Adam can help you with this soon. I get the feeling he is doing his best, but it still feels like there are other priorities for Enhance (other than us). While so many bugs have been fixed now, all good, the small niggling issues have not even started to be addressed. Anyway.. I need to keep my emotions in check.. easy to become impatient 😉

« Previous Page Next Page »