MySQL Self Recovery For Crashes/Shutdowns: Page 2 - Enhance Control Panel

MySQL Self Recovery For Crashes/Shutdowns

xyzulu

That log should tell you the exact reason .. you can even use: --since "06:00" or --since "2 hours ago" etc to make it easier to find.

twest

Welp, journalctl didn't help, it showed no entries. I analyzed the syslog and looking at the time stamps from when the server was last shut down found these entries:

Mar 19 14:27:26 s988 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Mar 19 14:27:26 s988 systemd[1]: Stopped MySQL Community Server.
Mar 19 14:27:26 s988 systemd[1]: mysql.service: Consumed 1d 1h 59min 7.545s CPU time.
Mar 19 14:27:26 s988 systemd[1]: Starting MySQL Community Server...
Mar 19 14:27:26 s988 kernel: [37918439.152631] audit: type=1400 audit(1742408846.980:35): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/mysqld" pid=1476683 comm="apparmor_parser"
Mar 19 14:27:31 s988 srs_milter[1339927]: CLOSE
Mar 19 14:27:31 s988 systemd[1]: Started MySQL Community Server.

That appears to indicate there's a scheduled task restarting mysql. I checked systemctl active timers, and crons - nothing strange there.

I tried checking timestamps in syslog to match up with the SHUTDOWN entry in the mysql error log (this was initiated by <via user signal>), but the only entries in syslog around that timestamp are hundreds and hundreds of website screenshot entries...

Speaking of website screenshot entries... I seem to be generating about 1 million entries daily on my syslog, and it's almost entirely website screenshots. The last time I updated the settings in the control panel I set it to update once a week or maybe once a month. I went to check on that setting, but it appears to be gone from the control panel now?

The docs says screenshot interval is at Settings>Service>App, but it's not there for me. I can't find it anywhere now?
https://enhance.com/docs/application-role/application-settings.html#screenshot-interval

I think I need to disable the screenshot service, it appears to be running non-stop.

xyzulu

twest Scheduled restart job

Anything just before this entry? This entry could be from your auto restart MySQL setting.

Jordan

@twest you might want to look at Netdata, it will alert on MySQL service being restarted or stopped as well as capture a ton over observability metrics.

wizardeur I noticed the same thing. I'm trying to investigate, log rotation is one of my suspects. Disabled it and will observe if it happens again.

Log rotation shouldn't restart/reload MySQL it should be sending flush logs;

twest Error log is good/clean,
Double check that it's still enabled. Default is disabled with Enhance. With the upgrade to v12 it might have been disabled.

twest Welp, journalctl didn't help, it showed no entries. I analyzed the syslog and looking at the time stamps from when the server was last shut down found these entries:

Enable the mysql general log, might get more details, might get the same.

MarkD

twest Speaking of website screenshot entries... I seem to be generating about 1 million entries daily on my syslog, and it's almost entirely website screenshots. The last time I updated the settings in the control panel I set it to update once a week or maybe once a month. I went to check on that setting, but it appears to be gone from the control panel now?

We have this problem as well - the logs are full of "Checking "/var/local/enhance/screenshots/..."

cPFence

twest

If Restart=always is set in the MySQL systemd unit file, then any crash or forced kill (like from OOM or a segfault) can trigger a fast restart. Logs about the crash get missed in journalctl -u mysql because the service is started again almost immediately. Systemd marks it as a normal restart if the service comes back up fine, so it doesn't treat it as a failure worth logging at the systemd level. To troubleshoot better, you will need restart=no or maybe try to enable persistent journaling. Hopefully, logs won't get lost.

twest

xyzulu just nonsense entries about the screenshot service.

twest

Jordan I trialed netdata before, but it was too heavy on cpu for my liking. I'll try the general log if another shutdown happens.

Currently running at 27 hours without a shutdown on the trouble server, guessing it's not a scheduled service crapping it up. Only change I've made so far is killing screenshots on all servers.

Jordan

twest Jordan I trialed netdata before, but it was too heavy on cpu for my liking. I'll try the general log if another shutdown happens.

I haven't had CPU issues, I also make sure to size servers correctly to allow for CPU/Memory for Netdata and backups.

twest Currently running at 27 hours without a shutdown on the trouble server, guessing it's not a scheduled service crapping it up. Only change I've made so far is killing screenshots on all servers.

I'm having 2-3 days and then mysql has stopped. I was wrong, you don't need enable the general log, as it will show all queries which may be helpful but will take up disk space. Instead just enable log_error.

log_error = /var/log/mysql/error.log

cPFence Logs about the crash get missed in journalctl -u mysql because the service is started again almost immediately

Unfortunately with MySQL depending on syslog for logging isn't ideal, enabling log_error should be a default configuration and journtalctl might capture stdout/stderr for the service via systemd.

Jordan

Just had another server have a MySQL server randomly stop. Still trying to track the reason down.

twest

My problem server hasn't had another shutdown yet, all servers going good. The only thing I changed since the last shutdown was log rotating postfix, and disabling screenshots. It's a big mystery.

cPFence

twest

Most likely it was a spike caused by a user running a bloated WordPress theme or plugin with stupid queries that heavily abuses MySQL. These kinds of issues are a pain in the neck to troubleshoot and can stay quiet for a week or two, then suddenly resurface. Best to review and optimize your MySQL settings to reduce the risk of it happening again.

« Previous Page