Some Backups Failed

CloudyBlake

Been running backups for a week now no issues. Today in the activity logs a couple of small sites, the backups are showing failed. Plenty of free space on backup server.

How do I dive into this to investigate? Are there backup logs stored somewhere with more info?

Edit: I did find in the log the raw event data, says this:

{
"id": "f30e7524-518f-4310-80ae-b71535bba07b",
"orgId": "4447a064-0cb5-4bb2-9f50-7ce463119db0",
"kind": "backupError",
"activityObject": {
"content": {
"detail": {
"ok": {
"domain": "xxxx.com",
"orgId": "0b0421a7-02c7-4426-8895-cd3548e44c7c",
"subscriptionId": 2
}
},
"id": "3155733e-ed13-440d-a35a-31efe64861bf"
},
"type": "website"
},
"context": {
"error": {
"content": {
"detail": {
"code": "internal",
"message": "No such file or directory (os error 2)"
}
},
"type": "error"
}
},
"message": "MySQL backup failed",
"createdAt": "2024-07-26T10:16:15.482835Z"

All of these seem to be the MySQL backup failed. I just did the Enhance software update earlier that day, did something change regarding where the db files are?

The other websites with databases are backing up fine, it's just these couple of sites.

Vortex

Same here.

CloudyBlake

Manual backups fail with the same error on the same sites. It's specific to mysql. They are still failing, was not one time thing. But not all websites.

Adam

Do you have the latest version of appcd? There was an issue with 4.0.1 where this error was thrown during the cleanup of the ephemeral container used for the backups but this was fixed in 4.0.2. The backup was still completed despite the error.

CloudyBlake

Adam I wasn't aware there is another place to go update things other than the platform section under settings. I had to find it under servers. It says I was still running 3.1.3. So when updating you have to hit both these places?

CloudyBlake

Adam So I think I know what happened. I ran the platform update, was unaware there was, in a separate place, the app updated needed. (this really needs to be all unified in one place. I understand it may need to be in different areas, but there should be a system alert or something of all updates needed in one place, with links to go deal with them)

After doing the platform update, I went and removed the /var/enhance_container_images (because release notes), which is what broke the things, because I was still on app 3.1.3.

For us new to the platform, there needed to be something in the release notes, or again, something in the UI to alert about the app update.

All seems to be fixed and working now.

Edit: to follow up on this, realizing now that the Update and Release notes were about the App and not the Enhance platform in general, which is what the UI alerted me to first when App was released.

twest

CloudyBlake reading the entire enhance documentation would probably be a big help to you. You should also start learning some basic sysadmin skills because enhance isn't like cPanel where you can click everything with a button, you need to manage updates for most of your enhance cluster (all servers individually) manually via command line.

JohnB

twest +1 and even with cpanel you shouldn't be hosting anything important without sysadmin skills. Its like a critical transport company without qualified mechanics. It will eventually end in disaster.

rdbf

twest

Makes me wonder, hasn't there been an apt get update && apt get upgrade since appcd 3.1.3 on that system?

CloudyBlake

twest Thanks for the suggestion.

I have read the entire documentation, and been working with Linux for over 20yrs.

I have the skills, some of it is being used to knowing where to find the info and Enhance is different critter.

xyzulu

CloudyBlake Enhance is different critter.

Different to cPanel.. yes.. but not different if you are used to linux server admin. Enhance is on top of linux.. so.. manage your server as your normally would (enhance has installed and setup ufw for you).

If you read the Enhance roadmap you will get a sense of more being dine via ssh/cli.. ie in the near future even the panel will be updatable via ssh/cli.