Connection Limit somewhere

MarkD

Hi Forum

I am looking for some guidance on where to look or check for a connection limit or timeout setting with OpenLitespeed.

To brefly explain, I have been running some load testing for a PHP website (Craft CMS 4) and it appears that no matter what size server I run, 4 Core with 16GB RAM, 8 Core with 32GB RAM or 16 cores with 64GB RAM. (All dedicated resources on Azure) the connection count seems to cap out at around 180 clients/sec as reported by loader.io.

We have even tried moving the database off to it's own Azure managed instance and connected it to the Enhance server via a private network so all database connections were still using internal IPs. The website always loads quickly when you are browsing it but it does not seem to be able to go past 180 clients/sec.

When testing with the 16 Core server I could see that all of the cores were in use but only around 30% each with plenty of headroom and the used memory was only ever around 4GB in total so plenty of memory left on every size. The disk queue was not high and the disk IOPS were not high either.

With each change in server size I tried tweaking the config by increasing the "Server Configuration > General > Number of Workers" and more importantly the "Server Configuration > External App > LiteSpeed SAPI App > Max Connections" setting.

I even tried the auto configuration from cPFence (https://gist.github.com/cPFence/d829366b95f8abd4d4ac2501b7be425d#file-ols_optimize-sh) and set the following in the script but still capped out at 180 ish clients/sec

Server Configuration > General > Number of Workers = 4

The Enhance package itself has no limits on it so thats not limiting it.

If I review the openlitespeed error_log I see multiple lines like this

[NOTICE] [3224] [18.204.4.113:34502#xxxxxx.xxxxxxxx.com] oops! 503 Service Unavailable
or
[NOTICE] [3224] [44.197.182.180:41572#xxxxxx.xxxxxxxx.com] ExtConn timed out while connecting.
or
No request delivery notification has been received from LSAPI application, possible dead lock.

I did try switching to LiteSpeed but that had the exact same issue so switched back to OpenLiteSpeed.

What am I missing?

Any help is much appreciated!

ivansalloum

Did you try that on a server running only OLS? I mean selfhosting WP without Enhance.

MarkD

ivansalloum No, not yet, so far all our tests have been within the Enhance platform but that could be our next step (or wait for v12 next week)

ivansalloum

MarkD I recommend testing that on a server running only OLS. I always conduct my tests on a clean, raw server.

MarkD

ivansalloum Yes, I will try that next week if the update to v12 yields the same result

twest Yes, I did increase Max Conn + Max SSL Conn and the highest database connection count was 252 during all testing which was well below the default max of 683 (Azure controls this based on the Database server size). I did see lots of OLS processes running but I didn't get a total count at the time. I have now wiped the test machine but on the next test I will take this into account. Based on that database connection count of 252 I am starting to think that the count of running OLS processes was at or near the "Server Configuration > External App > LiteSpeed SAPI App > Max Connections" but wouldn't that show in the OLS logs that the External App count was exceeded?

josedieguez Thanks for the link, I was not aware of this setting but after checking it doesn't look like that is the problem in this instance

twest

What are you seeing in TOP while running the test? Are the OLS workers getting maxed out? Have you increased Tuning > Connections > Max Conn + Max SSL Conn? Did you increase your database max connections?

josedieguez

Have you checked this?

https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:config:nf-conntrack-table-full

Andreas

Are you running cpfence? If so turn off DDOS. I had the same issue. cpfence --disable-DDos

cPFence

Andreas

You can simply whitelist the loader.io IPs to bypass the 100 connections limit and allow proper testing. No need to disable DDoS protection entirely.