Optimize Apache + mpm_event + php-fpm

twest

I was going to dump this in the ModSecurity thread, but I think it's gonna blow up with a ton of config dumps so I guess a new thread will work better for organizing things. I spent some time optimizing Apache and PHP today and got some good results = everything in the backend of sites is much snappier, initial connections to sites is much faster, TTFB dropped to 30ms to 50ms (previously 200ms avg, and was 150ms on OLS tests).

I did find one kind of critical flaw in Enhance though = we need a way to set php-fpm config on a per-website (account) basis, OR even better would be a php-fpm config on a per-package basis... There's just no way to have a one-size-fits-all config that runs on a per-website (account) basis when we can have websites (accounts) with 2GB ram allocation and 10gb ram allocation on the same server.

Important locations of files for reference:
Apache default config: /var/local/enhance/apache
php-fpm default config: I just su into a user and run php-fpm -tt
ModSecurity config file (this is the same file that gets updated from the ModSec config in Enhance UI): /etc/modsecurity.d/modsecurity.customisations.conf

Where I'm adding customizations:
For the most part in the network wide areas in Enhance UI in "Settings>Service" for php-fpm directives. Then for Apache/MPM configs I'm adding them in the ModSecurity config for each server as it's an easy way to alter Apache config through the UI. I do keep FTP connected and viewing the modsec folder any time I'm editing that file in Enhance, because if it gets a bad config it will crash and need to fix the file directly via FTP, so best just have the window ready to go.

Some rambling about rationale for config changes
I would say many of the default configs are probably okay for small/medium sized servers. For example the ServerLimit is default 256, which would likely only be suitable for a server up-to 8cpu/32gb ram. For a larger server, it will need to be opened way up. More limiting factors for php-fpm = defaults aren't optimized for best performance as you can see in the default config I will paste below. It's set to 25 max children which could handle a pretty good amount of traffic for a small site - if we figured 50mb per process it would consume a bit over 1gb ram, it may be a safe setting for small hosting accounts - but what if you have a large Package with 10gb ram? Well now that package is going to perform very poorly and have a whole lot of unused ram, if it's a busy site then the traffic will be slow... This is why I mentioned that per-website php-fpm config is necessary, you can't have a very large mix of Packages on the same server since they currently all have to use the same php-fpm config.

More impact on pagespeed/TTFB = pm is set to ondemand and start_servers and min/max_spare_servers are set to 0. That's probably Enhance trying to make slow/dormant hosting accounts not consume very much resources due to people complaining about that so much (derp). For me, I want my accounts to be tuned for PERFORMANCE, so I switched the pm to dynamic so that I can have processes warmed up and ready to go as soon as a request comes in. I also set a start servers and min/max spare servers.

Apache/mpm_event optimizations:
HostnameLookups Off
LimitRequestFieldSize 16380
MaxConnectionsPerChild 5000
ServerLimit 2000
StartServers 10
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestWorkers 2000
MaxKeepAliveRequests 500
KeepAlive On
KeepAliveTimeout 5

That's my config for a 50cpu/250gb ram server, that config is going to probably use about up-to 100GB ram if needed. Note I still need to optimize SSL/TLS, which I'll get back to later. LimitRequestFieldSize was increased because on Apache it's default is I think 4kb, while on other systems like nginx it's 16kb, and I've found a lot of clients with ads on their sites will get errors if the http headers are too limited as the ads can sometimes have too large of cookies. And of course hostnamelookup + keepalive are just more speed things.

Next on to php-fpm, this one is a bit more simple as these configs are mostly just affecting things on a smaller level (per-"website"). As I said before I set to dynamic because I want my workers warmed up and ready to go - they'll consume more ram while idle, but that's fine, my clientele want the best and fastest, and I have 250gb of ram that I want to be USED not just sitting unused all the time. Maybe for someone selling low-end hosting packages they would want the config to stay on ondemand to keep resource usage lower so they can cram a few hundred more customers on their server.

pm = dynamic
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 1000
pm.max_children = 100

Now this config is more suitable for a Package that has around 10GB ram allocation. Unfortunately, as noted earlier we can't set php-fpm config on per-website (account) basis, so I'm just dumping the config that works for my largest Package and calling it good (smaller sites might run into OOM scenarios if they get a burst of traffic that consumes all memory on their account, at which point it will need to be a conversation about upgrading their package or something I guess). It's a sucky tradeoff having to make one php-fpm config for all sites on a server, but if you have to choose I'd go with optimizing for the biggest package, and the smaller ones will hopefully just upgrade if running into OOM situations.

It will be nice if Enhance will give a per-website (account) php-fpm config. I'd guess it's something that will happen eventually, it kind of has-to.

Final thoughts:
I'd encourage everyone wanting to optimize their config to read about all the settings I've listed above, find out what they all mean and how they work/affect each other. My next step is to optimize SSL/TLS, after that it's going to be testing my config at high load.

For reference, here's the default php-fpm config:
[global]
pid = undefined
error_log = /dev/null
syslog.ident = php-fpm
syslog.facility = 24
log_buffering = yes
log_level = unknown value
log_limit = 1024
emergency_restart_interval = 0s
emergency_restart_threshold = 0
process_control_timeout = 0s
process.max = 0
process.priority = undefined
daemonize = yes
rlimit_files = 0
rlimit_core = 0
events.mechanism = epoll

[www]
prefix = undefined
user = undefined
group = undefined
listen = 0.0.0.0:9000
listen.backlog = -1
listen.owner = undefined
listen.group = undefined
listen.mode = undefined
listen.allowed_clients = undefined
process.priority = undefined
process.dumpable = no
pm = ondemand
pm.max_children = 25
pm.start_servers = 0
pm.min_spare_servers = 0
pm.max_spare_servers = 0
pm.max_spawn_rate = 32
pm.process_idle_timeout = 10
pm.max_requests = 0
pm.status_path = undefined
pm.status_listen = undefined
ping.path = undefined
ping.response = undefined
access.log = undefined
access.format = undefined
slowlog = undefined
request_slowlog_timeout = 0s
request_slowlog_trace_depth = 20
request_terminate_timeout = 0s
request_terminate_timeout_track_finished = no
rlimit_files = 0
rlimit_core = 0
chroot = undefined
chdir = undefined
catch_workers_output = no
decorate_workers_output = yes
clear_env = yes
security.limit_extensions = .php .phar

Aliysa_Enhance

twest we need a way to set php-fpm config on a per-website (account)

Hey twest this is possible on Enhance. If you head to Websites > '...' > Overrides you can set custom php-fpm on a per site basis

LBJ

twest For example the ServerLimit is default 256, which would likely only be suitable for a server up-to 8cpu/32gb ram. For a larger server, it will need to be opened way up.

Assuming Apache with MPM Event as per the title of this post, a ServerLimit setting of 256 is already overkill on most boxes.

Remember that for MPM Event and MPM Worker, unlike MPM Prefork, the basic tuning calculation is...

MaxRequestWorkers = ServerLimit * ThreadsPerChild

For Apache configured for MPM Event, generally ServerLimit would be set to CPU cores or lower.

twest I'd encourage everyone wanting to optimize their config to read about all the settings I've listed above, find out what they all mean and how they work/affect each other.

Excellent advice. You'll find a very comprehensive article related to Apache MPM tuning for both Event/Worker and Prefork at...

[https://www.liquidweb.com/blog/apache-performance-tuning-mpm-directives/]

Jordan

twest catch_workers_output = no

If you want to catch 500 errors in the browser, setting catch_workers_output to yes is important.

twest

For reference, here's a general glossary of settings I used in my config, with simple explanation about them:

HostnameLookups: Determines whether Apache does a reverse DNS lookup to get the hostname of clients. Setting it 'Off' improves performance by avoiding DNS lookups.

LimitRequestFieldSize: Sets the maximum size of HTTP request headers. Increasing it can be useful for handling requests with large headers, but be mindful of security implications.

MaxConnectionsPerChild: Specifies the number of requests a child process serves before it is terminated and replaced. Used to prevent memory leaks.

ServerLimit: Sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache server. Important for scaling in high-traffic environments.

StartServers: The number of child server processes created at startup. Optimizing it helps manage server load right from the start.

MinSpareThreads: The minimum number of idle threads Apache will maintain. Ensures readiness for incoming requests without creating new threads each time.

MaxSpareThreads: The maximum number of idle threads Apache will keep. Prevents resource wastage by reducing the number of unnecessary idle threads.

ThreadsPerChild: Number of threads each child process will handle. Balances the load each process carries, affecting memory usage and concurrency.

MaxRequestWorkers: Maximum number of child processes/threads that will be created to serve requests. Key for optimizing server capacity and performance.

MaxKeepAliveRequests: The maximum number of requests to serve on a persistent connection. High numbers improve performance for clients with multiple requests.

KeepAlive: Determines if persistent connections are used, enabling multiple requests per connection. Improves efficiency for clients making multiple requests.

KeepAliveTimeout: Time the server will wait for subsequent requests on a persistent connection. Balancing it helps maintain server performance.

pm = dynamic: PHP-FPM setting for how child processes are managed. ‘Dynamic’ creates children based on traffic, balancing performance and resource usage. Unlike the default 'ondemand' mode, which creates child processes only when requests arrive, thus saving resources but potentially increasing response time, 'dynamic' maintains a pool of idle processes ready to serve requests immediately, offering faster responsiveness under variable traffic conditions.

pm.start_servers: In PHP-FPM 'dynamic' mode, the number of child processes created at startup. Affects how quickly the server can respond to initial traffic.

pm.min_spare_servers: Minimum number of idle PHP-FPM child processes. Ensures readiness for handling PHP requests without delay.

pm.max_spare_servers: Maximum number of idle PHP-FPM child processes. Prevents having too many idle processes consuming resources.

pm.max_requests: The number of requests a PHP-FPM process handles before it's recycled. Helps in mitigating potential memory leaks.

pm.max_children: Maximum number of PHP-FPM child processes that can be active at a time. Crucial for resource allocation and managing PHP load.

DracoBlue

In my opinion, Apache or the MySQL/MariaDB database should be pre-optimized for performance. Default settings don't always suffice, and not everyone has the knowledge to configure everything as it should be. Therefore, I believe Enhance should consistently implement optimization for various functions or create a calculator that does this based on server parameters.

twest

DracoBlue it's pretty much impossible to do that since everyone has a different business model and config. There's no panel that does it fully, they all require optimization. Even the ones that claim to optimize are very poor out of the box, and they only fit certain use case scenarios. There's too many variables per use case - sizes of servers and network restrictions, and then what the server itself is even hosting... Basically = don't hold your breathe on that.

@everyone, I stumbled across the fact that Enhance did build in a tool to optimize php-fpm on a per-website (account) basis. It's in the Websites section, just tick the 3 dots next to the customer account you want to edit php-fpm and hit "Overrides". This is kind of a pain in the arse to do per-account, but it's better than nothing 😃

JohnB

Thanks twest for the great writeup!

twest Even the ones that claim to optimize are very poor out of the box, and they only fit certain use case scenarios

A lot of panels (and hosts) completely rely on caching for performance. Smaller sites with sporadic traffic or requests that can't be cached has really bad performance. If reviewers just benchmark the homepage they think its so fast! No real traffic, plugins, crons means wp-admin will be fast in their test. Meanwhile real-world site backend cries in pain and login takes 20 seconds if the site gets remotely busy.

They still get marketed as being so fast and optimized!

twest

JohnB yeeeeep, I couldn't count the number of noob hosts I've seen blindly jump into this industry expecting to make easy money only to end up folding after they couldn't manage a profit. So many times people think they can puke up a cPanel server and call it good, only to find out after a hundred customers that their server was bogged down or crashing constantly.

I always tell noobs the best place to start is with an unmanaged VPS with a panel like ispconfig = force them to learn how to use a command line and basic sysadmin skills. Once you know how to run a server from ssh then you can gracefully transition to something that offers more automation. You really need a good base of skills to build on so servers and the forthcoming customer accounts are setup for success. Optimizing for performance is only one of the things I do to set my services apart from the Goshittys and Blueballs mega hosts of the world 😁

I want to support Enhance development for my own greedy uses (lower license costs). This forum is a great tool to help share knowledge so that others can be successful with their own servers. If we all contribute to each other's strategies/success then we can help increase Enhance adoption, ensure its longevity 💪

JohnB

twest You really need a good base of skills to build on so servers and the forthcoming customer accounts are setup for success. Optimizing for performance is only one of the things I do to set my services apart from the Goshittys and Blueballs mega hosts of the world 😁

I think the worst I saw was a server that couldn't handle default wordpress theme homepage without caching at 10 concurrent requests with no rampup. They were hosting on the most garbage google cloud tier you can get and no doubt oversold and unoptimized.

I've seen so many people across many pretty panel forums that have no concepts about security, backups and basic sysadmin. Of course they call you an @hole if you are honest and say they have no business even being in a terminal. "Everyone starts somewhere" ... yeah by not hosting real customers with no backups is a good start.

Sadly even large companies often have horrible security and backups. This is why I always preach to never get vendor locked and always have verified and frequent backups outside the provider but many think a cloud provider VPS snapshot is a backup. When the provider incorrectly suspends service and they lose access to snapshots or the snapshot was corrupt their customers are screwed because the niave customer didn't know they needed their own backups... Annoys me a lot.

twest

Aliysa_Enhance I know, I already posted about that in this thread. Can't edit old posts tho. Maybe that should be another feature request "let forum posts be editable forever".

josedieguez

twest a time window of let say 24hrs or something, and a legend that says "edited" is better than forever.

John-P

twest Maybe that should be another feature request "let forum posts be editable forever".

The problem with that is that editing something can often make the whole conversation that follows it make no sense.

A short editable time limit is fine, forever (for non-admin) users can just get very messy.

DracoBlue

twest

LBJ do you work for liquidweb? They're a trash company and I wouldn't waste my time reading any garbage they may publish.

JohnB

twest Yup.

From that page a perfect example. Title randomly right aligned and shortcodes showing. This is the level of care and quality you get with liquid web.

Also

Set MaxKeepAliveRequests to double that of the largest count of elements on common pages.

What. Seriously what. I think they mean to say double the count of elements such as images on common page which also is bad advice. Who wrote this? This is why "professionals" who blindly trust chatGPT are idiots. All ChatGPT can do is parrot its training data in its own words. Sometimes its right sometimes its not but it always has a confident tone. Its not magic its not "AI" in the sense that it understands anything. Its a language robot.