LiteSpeed Enterprise and 503 error

sikmo

Hi, we are having a bit of an issue with RAM resource and 503 error. The RAM starts to build up in specific user, even if there is not many people accessing the website.

We are investigating and current idea is, that since we saw idle LSPHP processes with lifetime of minutes to tens of minutes, we need to limit them via "Max Idle Time".

Once we restart the container, it is fine and returns back to reasonable RAM consumtion. We also set IOPS to 1000 but that does not seem to help when RAM limit is reached. NPROC is set to 24 - 100 (we tried many variants).

Does someone has LSE and solved this? Are we alone.

gmakhs

sikmo so, litespeed has a fix in the settings for such scenarios .

It's called auto fix 503.

It doesn't work on enhance due to the way enhance is handling php, so litespeed can't restart php to reduce ram usage and fix 503 error .

Enhance is the only panel that says that supports litespeed and that option doesn't work, have reported the issue many many times, they don't seem eager to fix it , litespeed said enhance needs to fix it .

So as an alternative, if you are willing you can use monit, set it to monitor each websites ram usage and when the usage reaches 90%, use enhance API to restart that php container .

You can do the same check for nproc per website container, it does work but it's not optimal in any way and since is not a graceful restart as litespeed does and data loss might occur .

If it will get fixed I can't tell you, but it's not even on the roadmap, it seems the "support litespeed" in Enhance is just misleading advertisement, they just use to get market share .

As is now, enhance can't handle high traffic websites with an economy on resources .

If you can afford leave that website with unlimited or very high CPU and maybe you will manage to balance out processes, again not ideal .....

All that being said I would love to see that fixed ASAP since is a critical issue for anyone using litespeed, and it's core function of litespeed
@Adam @Aliysa_Enhance

sikmo

We already have instant monitoring for RAM on our custom panel + custom code per server, so restarting PHP container is easy for us.

We are also exploring idle limit and deleting long idle processes, so I wanted to know if someone tackled this one.

Adam

The default timeout for idle workers is 5 minutes. We never made this a configurable because this setting is perfect for 99% of use cases.

Are you sure the workers with lifetimes of longer than this haven't received a request in the last 5 minutes? The easiest way to check is to strace the process.

@gmakhs as explained previously, Enhance containerises website processes including PHP, SSH, cron, su, etc. This means that Enhance needs to start PHP rather than Litespeed and is why the Litespeed settings for lsphp including "auto fix 503" have no effect. This isn't something we are likely to change.

Adam

One thing to add, the master lsphp process for a website will always be running - it is responsible for the creation of worker processes. However version 12.9.0 only starts the PHP SAPI when the first request is received following container start/restart. If you have customers who create websites that never receive traffic, it will save significant memory.

sikmo

Well, we can trace the processes - not sure how, but I will report it to our team to do it somehow. Problem is, that this issue comes to light when the memory is running low. So at that point, we have to restart and not waste time. But I am sure we can do something.

As for the limit, it seems weird to me, that processes can run for so long even though we know at that point users are mostly on cached pages of website or accessing pages that will be cached right after. And for some websites, we saw quite long time for some processes as well, where we know for sure that is like 2-5 people per day at most.

processes

Adam

The idle counter starts again every time the process receives a request so the worker has to be idle for 5 minutes before the master process will shut it down. lsphp will re-use an existing worker before creating a new one if the site is idle.

If the sites are genuinely low traffic they shouldn't have so many workers. Is it possible you have set CPU or i/o limits too low and that requests are hanging? If so, when the visitor refreshes the page (or a bot retries their request) this will spawn another worker since the original worker is still busy with the first request.

sikmo

IOPS = 1000 (should be enough)
NPROC = 24 (should be enough)
IO bandwidth = 300 TB/s (unlimited basically)
CPU = 1 (but that does not seem to be issue)
RAM = 1 000 MB

gmakhs

sikmo
Nproc 24 is very low if customer has 2-3 wordpress websites in one container .

Try 50 or 100.

Also is important to make sure the site is not hacked, low traffic shouldn't open so many processes, but low traffic with malware does .

As for the issue of full ram, create an automation to check it as I mentioned, it will solve your issue if it's from traffic .

@Adam is your choice and I respect it, if you could at least replicate how it works it would save a lot of headaches.

sikmo

gmakhs we have always 1 website per 1 cotainer. No malware, checked for that. And yes, we already have that in place (for the RAM issue)

gmakhs

sikmo I use 50 on low end and 100 nproc on the high end for all websites , it works well .

High traffic websites tent to reach limits on enhance reasons I mentioned above, and will always hit the limit sooner or later ... Specially with high traffic spikes that login etc

Best is to add in your automation, number of nproc .

Also is not wise to limit disk usage if you are on nvme and disk usage ain't big, because it just throttles the websites without reason.

Also if the container limit is 1000 mb memory, makes sure.phh ini setting is lower than 256, if php ini is set to 1000 also that can cause the issue