Diagnosing Load Spikes Regarding Backups

Rich

Don't forget if you've placed limits on packages for resources, say 2GB limit on a user. When it's performing whatever backups it needs, even though the server has tons of free RAM I am seeing swap being used. That's due to a memory leak within the website, that's causing the memory pressure to go up towards 7GB, the spike is handled and brought back down. But I feel this user-level restriction is causing the bottleneck over the server having enough free resources.

Just another 2 pence thrown out there.

twest

Rich hmmm that's an interesting idea. I wouldn't expect the server backups to be consuming the website/package resources, I would expect it to be operating outside that level on its own. Maybe we need to consult @Adam for confirmation (do backups consume the website/package resource limits?).

When I set the concurrent backups setting to 10, I assumed it meant that would dedicate up to 10 cpu too, so maybe I was wrong on that too.

So far today haven't had a spike, we'll see how the next 8 hours go as traffic load picks up @_@

twest

deydod Nah, it's all premium/enterprise hardware, NVMe, etc., and there's +200GB free RAM available lol

Andreas

twest When I set the concurrent backups setting to 10, I assumed it meant that would dedicate up to 10 cpu too, so maybe I was wrong on that too.

I understood that to be number of backups running at the same time.

twest

Andreas yes "number of backups running" is the simple way to think about it, but when considering how much resources each backup process requires it's most common in control panels and backups systems that 1 concurrent process = 1 cpu... It could be Enhance does things different though, maybe each process is allowed to run wild and consume max resources to get the run completed as fast as possible. A rate limiter might be very useful if that's the case.

twest

Welp, dialing back the backups aggressiveness seemed to do the trick, not a load spike nor memory swap made all day. Okey dokey then 🙂

SharedGrid

@twest - is the spike on your web server or the backup server?

SharedGrid

I got a bit bored this morning, probably have many other things I should be doing but after seeing this thread here and on WHT; I thought I'd take a look.

What happens when a backup is kicked off, in simple terms: The source server creates a SSH connection to the backup server and starts a rsync server, the source server starts a rsync transfer of the website files using archive and compress modes. Archive mode keeps all file and directory timestamps, ownership, etc. Compress mode compresses the data ready for transfer over the network - this is why you're seeing a CPU spike as it takes CPU cycles to compress this data.

The rsync process runs within the website user account, I haven't looked into the cgroup implementation so it's a 'more than likely' correction assumption that the rsync process is contained to the limits you set on the package. However, by default rsync is a single threaded application so it can theoretically only use 100% of 1 CPU/Thread anyway. But... if you are running multiple backups at the same time then you're going to see this spike higher as each rsync process will use up to 100%, therefore, 10 concurrent backups * 100% = 1,000% or 10 CPU/Threads.

You will not only be limited to the 1,000% CPU though as there's network throughput and disk throughput happening at the same time which increases the CPU load so you're actually using more than 1,000%.

The Disk I/O graph you've posted is Disk I/O and not a memory graph so the Swap you're seeing isn't memory swap. Disk I/O Swap isn't very accurate, it's more of a generalisation and there are better ways to graph disk I/O. Disk I/O Swap counts all input/output operations including memory usage, file read/write operations and any other interactions with the storage so it's pretty useless to drill down into any troubleshooting using Disk I/O swap.

I monitored my source server while performing a 19GB backup and there was minimal memory usage and no memory swap usage from the process.

For reference, here's a screenshot of the rsync process while transferring the backup from top:

These are the rsync commands mentioned earlier:

`root@redacted:/var/www/152074ae-14f1-449c-82f1-c428c7969de5/public_html# ps -ef | grep rsync

testweb+ 1852504 132209 98 09:23 ? 00:00:17 rsync -e ssh -p 222 -i /tmp/41949d97-4bdf-45c9-8c1a-fc0ae8461077 -o StrictHostKeyChecking=no -o BatchMode=yes --archive --verbose --compress --delete /var/www/152074ae-14f1-449c-82f1-c428c7969de5/ testwebs1@redacted:/backups/152074ae-14f1-449c-82f1-c428c7969de5/backup-subvolume/home

testweb+ 1852505 1852504 0 09:23 ? 00:00:00 ssh -p 222 -i /tmp/41949d97-4bdf-45c9-8c1a-fc0ae8461077 -o StrictHostKeyChecking=no -o BatchMode=yes -l testwebs1 redacted rsync --server -vlogDtprze.iLsfxC --delete . /backups/152074ae-14f1-449c-82f1-c428c7969de5/backup-subvolume/home`

twest

SharedGrid wow thanks for the in-depth testing and analysis! That's sad to think the backup run is consuming the client's CPU, I might need to update all my packages to bump up an extra CPU to compensate for that. Maybe the user accounts are thrashing from lack of available CPU or something.

I wonder if disabling the compression on the backup would make it more efficient, I'm not really interested in spending CPU in exchange for slightly smaller sized backups. I know Adam mentioned more control/features for the backups would be coming out soon, so I may just need to wait to see what that entails.

So far today the backups seem to be running fine, no spikes, though I won't really know if reducing down to 3 concurrent backups helped or not until the peak of traffic load later today... With 28 cores/56 threads I didn't think I would need to reduce the concurrent backups down so low, not with so much available CPU, but it seems the backups aren't really that efficient after all.

Rich

twest I think you'll find the compression is just gzip for rsync transport, it will decompress the other side. The backups can be same server or across ssh to another server. So "for slightly smaller sized backups." isn't the reason for turning that off, it's your bandwidth, if your sever is remote.

BTRFS is the reason backups are small as they use snap-shot differential storage. I assume it's rsync the target vs the backup sub-voume and the files are stored in snapshot, with a little extra btrfs dedup magic happening somewhere.

twest

Welp, so far so good, I think the reduced swapiness is helping some, and I think reducing down to 3 concurrent backups is helping some, and I did also increase all hosting packages by 1 CPU to help offset any potential steal by the backup run (I do have a small number of sites on 1cpu, so I imagine their backups were totally locking up their containers).

I'm still getting some swaping going on, but it's tiny, <10 ops/s, and so far no spikes in anything.

I'll wait another day or two before declaring victory though, because you know how it is lol 🙂

twest

Rich Yeesh, well I'd rather spare my cpu and blow out bandwidth. I have 1g and 10G ports on everything, actual usage of those is extremely limited. I think most people have magnitudes of more bandwidth to spare than they do precious cpu cycles :/

Ah well, if weakening my backups fixes the issue I'm happy. Maybe the upcoming updates with more backups controls will solve this problem too 🙂

twest

For anyone interested, or future problem havers looking for a fix, I implemented another test today.

I went ahead and disabled swap entirely.

Here's my thinking... Something with the backups is causing a load spike, which coincides with swap usage. This next test aims to rule out a potential I/O bottleneck with the swap. I have swap as part of my raid1 config, so I'm wondering if that's causing a bottleneck.

Not the root cause of the bottleneck, but the cause of things getting backed up... Like if my swap wasn't part of the raid array, then maybe it would have enough IO to handle whatever is causing swap in the first place. I know the transfer speed in my raid is capped (100Mbps I think), which is well below what my NVMe drives can handle, but maybe there's some other weird things happening between the backups, the user accounts, cgroups, software raid...

With 200gb of unused ram, I'm really not worried about there ever not being enough, so swap is really unnecessary in my setup (if 7gb of swap is actually needed on my 250gb server, then I've got much much bigger problems).

Thus the next test begins 🙂

twest

Alright boys I'm calling this issue fixed. Ever since completely killing swap all my servers have been running smooth as butter, load has been reduced substantially, no spikes of any kind.

I'm sure it had something to do with a bug between the backups and my swap file being on raid, but whatever the case swap is really meaningless in servers of larger size - I would say anything with over 8gb ram really shouldn't need any swap at all. I'll still use swap on tiny little 1-2 gb servers so there's some wiggle room to use slow virtual memory, but everything else I'm going swap-free!

If you're interested in turning off swap yourself and reading this post, it's really easy:

View your swap:
swapon --show

Disable swap temporarily (resets on reboot):
sudo swapoff -a

Disable swap permanently:
sudo nano /etc/fstab

Find the line with your swap entry, something like this:
/swapfile none swap sw 0 0

And comment it out like this:
#/swapfile none swap sw 0 0

Ctrl + x to exit, Y to save changes.

Now the next time you reboot, the swap won't start up. For me, I used the temporary swap disable to turn it off on the live servers, then updated fstab so when they get rebooted some day (hopefully years from now) the swap will stay disabled.