You Guys Limit I/O Or Nah?

twest

I get some high iowait at seemingly random times and wondering what it's coming from. All accounts are limited for iops and io bandwidth, and it's enterprise nvme, so there should be plenty of io to never bottleneck.... Maybe the io cgroups itself is causing iowait?

Do you guys limit io, or just leave it open/unlimited on all packages?

PradeepC

We had it limited for quite sometime, and it caused some high io wait at random times. We removed it totally and now the issue doesn't occur. Never got the time to mention it on the forums though.

twest

PradeepC wow thanks for the reply! That's very reassuring, and really makes me want to test it on my cluster as well.

Anyone else have similar experience? This is very interesting 🙂

twest

I went ahead and cut the i/o limits, let's see how it goes 🙂

Adam

Generally the i/o limit is only required if you have slow underlying storage. If you have modern NVMe/SSD then setting i/o limits may be detrimental because it will cause user processes to run for longer which will consume more RAM as well as providing a worse customer experience.

If you are limiting i/o or CPU then I would recommend setting a low nproc limit in conjunction with this.

Rich

I've gone and removed all limits except disk size now. Lets see as I was getting some user swapiness going on at times.

twest

Rich truly unlimited hosting lol

charlesalmeida

Yes, strange, this happening today out of nowhere, the machine is down.

twest

charlesalmeida did you try reboot to bring it back up?

wbknght

yep,

https://prnt.sc/NT_BQc4DfUc7

MarkD

We have also had the same experience with trying to limit a single websites IO but it ends up affecting the whole server.

It appears that if 1 website is under load and using all it's allowed memory it causes a swap to disk and then the IO limit kicks in causng the server IO wait time to rocket to 90%+.

We came from a cPanel+Cloudlinux background so originally we tried to use the same limits that we have had for years but clearly we can't do that.

We are going to try removing the IO and storage bandwidth limits and see how we go.

twest

MarkD it seemed to help me quite a bit. Only getting a large spike of load about once per day on one server now, versus a couple times a day in every server before.

I've been troubleshooting this problem for months, testing things like disable swap, changing swapiness to 0, changing webservers, reducing backups, etc., etc... Removing IO limits seems to be the best fix so far.

Rich

twest My spikes seem to have gone since I adjusted my settings. overall compute is up a little, but I'd take that any day. I did nothing at a kernel level. Just removed package restrictions in enhance. IO probably is the culprit.

I also agree with @MarkD that's what I was seeing too. I initially thought It was bad mysql setup, but since that setting change, it's smoother. Hopefully for you too.