Backups Failure On Backups Subdomain

twest

I'm pretty regularly getting 3 logs for my backups server backup. Not really sure what there is to backup for the backups website backup, and not sure if the erroring is causing other issues (maybe my backups load spike problem?). Anyone else getting these errored backups? It's always 3 entries and 2 of 3 fail with codes like this in the logs section:

{
"id": "8c740f28-485a-44d4-b1a6-5ba7de1fdffd",
"orgId": "e403d6bb-67c1-4317-95ad-b90974b8354f",
"kind": "backupError",
"activityObject": {
"content": {
"detail": {
"ok": {
"domain": "backups.hosting.com",
"orgId": "e403d6bb-67c1-4317-95ad-b90974b8354f",
"subscriptionId": null
}
},
"id": "93a5df1b-7867-439b-8434-cbc883452842"
},
"type": "website"
},
"context": {
"error": {
"content": {
"detail": {
"code": "internal",
"message": "Grpc Transport failed: \"transport error\""
}
},
"type": "error"
}
},
"message": "MySQL backup failed",
"createdAt": "2024-01-27T22:50:17.367524Z"
}

Activity detail
{
"id": "0eba1506-9984-4736-a351-6c98ab37e8c9",
"orgId": "e403d6bb-67c1-4317-95ad-b90974b8354f",
"kind": "backupError",
"activityObject": {
"content": {
"detail": {
"ok": {
"domain": "backups.hosting.com",
"orgId": "e403d6bb-67c1-4317-95ad-b90974b8354f",
"subscriptionId": null
}
},
"id": "93a5df1b-7867-439b-8434-cbc883452842"
},
"type": "website"
},
"context": {
"error": {
"content": {
"detail": {
"code": "internal",
"message": "Grpc Transport failed: \"transport error\""
}
},
"type": "error"
}
},
"message": "Home dir backup failed",
"createdAt": "2024-01-27T22:48:06.293580Z"
}

ordinary

twest I'm getting similar errors for multiple customer sites.

I have native and Backblaze backups enabled:

{
"id": "b23746b8-7ccc-4d66-acb2-XXX",
"orgId": "0b384bcd-ff3f-4782-a8f3-XXX",
"kind": "backupError",
"activityObject": {
"content": {
"detail": {
"ok": {
"domain": "XXX.com",
"orgId": "d428de2c-c02a-48cc-a165-XXX",
"subscriptionId": 7
}
},
"id": "74b0b56f-fd44-4a06-b8fe-XXX"
},
"type": "website"
},
"context": {
"error": {
"content": {
"detail": {
"code": "internal",
"message": "Grpc Transport failed: \"transport error\""
}
},
"type": "error"
}
},
"message": "Home dir backup failed",
"createdAt": "2024-01-28T23:37:05.641625Z"
}

twest

ordinary if you check the backups section on those websites that are failing, are they consistently failing?

I had one consistently failing because there was a folder with file permissions of 000.

I do have some randomly failing pretty regularly, but not consistently - just one-off failure of homedir sometimes.

The inconsistent failures don't bother me much, it's when they consistently fail that I'm worried as something is borked.

ordinary

twest It appears to have started right after the 10.2.0 update.

This is typical of what I'm seeing on the failing sites now: https://www.dropbox.com/scl/fi/94k90dn73p6eqlg61n3ts/Screenshot-2024-01-29-at-1.10.35-pm.jpg?rlkey=8rgorkbi273cvoixj5h7d4nst&dl=0

I think this warrants opening a ticket!

xyzulu

Check this: https://community.enhance.com/d/1158-docker-2500
I had this issue on one server of mine.. and it affected the backups just as you are seeing.

twest

xyzulu I'm not sure if that's the same issue. Does fixing it require restarting all servers? I had an extremely buggy control panel, but after I restarted the docker containers on the control panel everything went back to normal.

xyzulu

twest Does fixing it require restarting all servers?

No, but that may be the simplest solution. Just check the Mac addresses of the docker containers to see if you have this issue.

perhaps: docker container inspect
.. on a few of the containers that are having issues.

Or.. a ticket is also a surefire way to confirm the issue.