Tracing brief server load spikes

AdamR

Ever since we switched to Enhance we have experienced these strange and very brief server load spikes that we would never have on cPanel. It happens on all of our application role servers to one degree or another. Doesn't matter if it's a cloud server or a bare metal dedicated, they sometimes appear to temporarily seem to cripple the machine for a few seconds (you may click something and the wheel spins for 5 seconds like its down) and then everything goes back to normal and is lightning fast again. I've tried monitoring 'top -c' and looking for D state processes, high iowait, monitoring mysql processes, etc but I still can't seem to pinpoint what happens. We run a combination of Litespeed and OLS and it does it on both, vanilla configuration on these as well as MySQL. Same on both Ubuntu 22.04 and 24.04.

Every single website on every single server is assigned to a customer and a package with resource restrictions for CPU, memory, Nproc etc.

Even a brand new bare metal server (14 cores 20 threads) idling with no websites seems to have these little blips with nothing but Enhance + Application Role installed. Rebooting it doesn't change anything.

Here are current screenshots from a few minutes ago of 4 different Application Role servers in 4 different datacenters to provide some context:

Litespeed, MariaDB 11, bare metal:

OLS, MariaDB 11, cloud:

OLS, MariaDB 11, bare metal:

EMPTY / IDLE: OLS, MariaDB 11, bare metal, 0 websites:

Does anyone else have similar spikes in your clusters? Any suggestions to try, or things to check?

Adam

Enhance does run periodic scheduled tasks but none of them should cause load spikes like this. There is a disk usage collector which can create a spike in i/o activity.

It's a low tech approach but if you want a log over time to help pinpoint the cause, you could set up a cron job like this:

* * * * * top -b -n 1 >> /var/log/top.log

This will give you the process list as well as memory/swap usage, iowait, cpu steal, etc on a minute by minute basis in chronological order. Obviously don't forget to remove the cron job when you're done as the log could grow very large.

AdamR

Adam Thanks, I will set these up and follow up!

Vendoz

The empty one is the best one to debug. If it's empty (no sites or applications), than the load must come from OS/enhance. It seems a spike every 3 hours: could be a backup software like acronis or veam?

Do you have software like atop installed on the machine? Atop is great because it takes a snapshot of top every 10 minutes, so you can understand what the server was doing in that moment, to debug better.

twest

I've made a few threads about my own issues trying to figure out these kind of spikes, might want to go over those threads to look for ideas. I think in the end I did a combination of fixes including reducing the number of concurrent backups run at a time to 1, and functionally disabling swap by reducing swapiness to 0 or 1 (the load spikes seemed to correspond to increase in swapping activity, indicating some kind of IO bottleneck, possibly due to IO limits in website packages too, so I set IO/IOPS to unlimited), There might have been a few other little tweaks I tried, can't really remember - it's all documented in those old threads though lol

AdamR

Thanks for all the suggestions, turned out we had NPROC set too high on Packages.

This server had the most significant reduction (from load spikes of 120+ to a load average under 1), surprisingly enough it was still functioning pretty well under those crazy load spikes, high iowait (between 60-95%) was causing the load:

The following commands were helpful in narrowing the problem down in case anyone else runs into this:

Monitor CPU / RAM usage:
top -c

Input/output disk statistics by process / thread:
iotop

Virtual memory statistics:
vmstat

Monitor system input / output statistics:
iostat

View the processes when a load spike hits, ultimately for us there were a bunch of D state processes causing this:
ps -e v