Tracing brief server load spikes

Adam

Enhance does run periodic scheduled tasks but none of them should cause load spikes like this. There is a disk usage collector which can create a spike in i/o activity.

It's a low tech approach but if you want a log over time to help pinpoint the cause, you could set up a cron job like this:

* * * * * top -b -n 1 >> /var/log/top.log

This will give you the process list as well as memory/swap usage, iowait, cpu steal, etc on a minute by minute basis in chronological order. Obviously don't forget to remove the cron job when you're done as the log could grow very large.

AdamR

Adam Thanks, I will set these up and follow up!

Vendoz

The empty one is the best one to debug. If it's empty (no sites or applications), than the load must come from OS/enhance. It seems a spike every 3 hours: could be a backup software like acronis or veam?

Do you have software like atop installed on the machine? Atop is great because it takes a snapshot of top every 10 minutes, so you can understand what the server was doing in that moment, to debug better.

twest

I've made a few threads about my own issues trying to figure out these kind of spikes, might want to go over those threads to look for ideas. I think in the end I did a combination of fixes including reducing the number of concurrent backups run at a time to 1, and functionally disabling swap by reducing swapiness to 0 or 1 (the load spikes seemed to correspond to increase in swapping activity, indicating some kind of IO bottleneck, possibly due to IO limits in website packages too, so I set IO/IOPS to unlimited), There might have been a few other little tweaks I tried, can't really remember - it's all documented in those old threads though lol

AdamR

Thanks for all the suggestions, turned out we had NPROC set too high on Packages.

This server had the most significant reduction (from load spikes of 120+ to a load average under 1), surprisingly enough it was still functioning pretty well under those crazy load spikes, high iowait (between 60-95%) was causing the load:

The following commands were helpful in narrowing the problem down in case anyone else runs into this:

Monitor CPU / RAM usage:
top -c

Input/output disk statistics by process / thread:
iotop

Virtual memory statistics:
vmstat

Monitor system input / output statistics:
iostat

View the processes when a load spike hits, ultimately for us there were a bunch of D state processes causing this:
ps -e v

knireis

I have some of these spikes as well.
You mention NPROC value too high, default it is set to unlimited.
What is a decent value for a wordpress with woocommerce website?

adil

knireis
Start with 50 and adjust up/down as you monitor the server, also it depends on the websites if they are big or low on traffic.

cPFence

knireis

Try these limits , they might help stabilize things.

Additionally, be selective when choosing WordPress themes and plugins. Many poorly optimized ones can cause MySQL-oriented load spikes, especially on WooCommerce sites.

Andreas

cPFence Hey cpfence, thank you for the suggestions.

Playing around with the settings and running loader.io tests (10K hits in 1 minute), I found these settings worked best so far.

ram 256
cpu .5
ops 1024
IO 100
nproc 100

load .25
peaked .3

The test VPS server specs are

1CPU Ryzen CPU
3GB of ram
Using apache

The server didn't even break a sweat with the settings above.

My test site setup was

CMS - Wordpres
Kadence theme - https://www.kadencewp.com/kadence-theme/starters/soap-shop/
Cache plugin Surge - https://wordpress.org/plugins/surge/

I'm going to test other themes as well.

adil

Andreas

Thanks for sharing, Can you also try with WooCommerce ?

Andreas

adil

This theme, I picked has WooCommerce plugin installed. (That is what I normally try to pick/use for testing)
Is there a specific test you want me to do?

adil

Andreas

Oh, Then you did great job optimizing the limits!

We are using Woodmart + Elementor + WooCommerce, its little heavy and requires a little more resources.