Since upgrading from 12.6.0 > 12.6.2, I've started seeing intermittent backup errors in the activity logs:
"message": "S3 backup failed: Error { kind: Internal, msg: \"Task had error: invalid argument: gzip error: \\"\\"\" }"
The error seems to be reported for 3-4 sites every day. They seem to be different sites each day and there doesn't seem to be a connection with the size of the accounts.
I know logging was re-enabled in 12.6.x so I'm not sure how long it's been happening. During the upgrade from 11 > 12 I had some issues with backups requiring a script to purge older backups to get it working again.
I also recently moved backup roles to new server so I thought this could be related. The new backup server is in the EU. It's not particularly quick and my S3 endpoint is in the US.
I also tried disconnecting S3 and setting up a new bucket but that didn't help. Interestingly, if I download the "failed" backups from either bucket they do appear to be valid.
I'm also seeing an sqldump on a few sites:
"message": "Enhance backup failed: Error { kind: Internal, msg: \"mysqldump command failed for sconelit1_live\" }"
I don't know if these errors are related, but if they are it seems logical that something might be timing out if the backup server is low on resources or the connection is too slow?
Appreciate anyone's advice on where to look next!