PHP apparently hangs indefinitely

matrix

I'm already running some websites with Enhance. Each website is on a dedicated Hetzner cloud instance.
All websites are migrated from the same Plesk server which was hosting them shared.
Everything was fine till now, when recently I've been struck with a serious issue.
Some websites refuse to work. With these symptoms:
1) The website start load infintely till timeout
2) The PHP process reach 99% CPU
3) With strace the PHP process is doing nothing, no activity.
4) With ltrace the PHP process is reporting in loop something like (not sure if related)
memcpy(0x7bc9b1a860b2, "95430879", 8)
strdup("Undefined offset: 95430879")
strdup("/var/www/siteid/public_html/alwaysthesamefile.php")
this is related to a symfony dependency (XmlFileLoader.php) which anyway is used by all the other websites working too.
5) Statics file are served just fine
6) It happens no matter which webserver I use (tried OLS and Apache)
7) Nothing wrong is logged, even with debug active in OLS. Without debug logs are completely empty.
8) Tried with different instances types, no luck.

Additional info: The looping file in ltrace is always the same but again not sure if relevant.

I'm seriously out of ideas as of why it happens. Especially since it works on plesk with the same setup (LSWS and php.ini). Tried everything. Any ideas?

matrix

After lots of debugging I got the culprit.
The PHP 7.2 version bundled with ECP is bugged and not the latest.
The opcache, when active after STEP3, breaks for loops when using conditions like
for ( $i = 1, $c = count( $textarr ); $i < $c; $i += 2 )
The conditions always return true and never exit the for cycle, resulting in an infinite loop.
Lowering the opcache.optimization_level to 0x00000003 fixes it, but the whole opcache is almost useless then.
Not sure how to report this bug directly to ECP, so they can recompile and update their PHP 7.2 package.

At the moment I'm trying recompiling it from source within OLS webadmin using the same prefixes. I'll let you know.

mendozal

matrix you should open a ticket with Ehnhance.

SharedGrid

matrix The PHP 7.2 version bundled with ECP is bugged and not the latest.

PHP 7.2 on Enhance is the latest available, 7.2.34... which went EOL in 2020.

Vendoz

matrix The PHP 7.2 version bundled with ECP is bugged and not the latest.

Have you by any chance tested whether this problem also occurs with other versions of PHP 7.X?

matrix

mendozal Done, let's see

matrix

In the meantime the solution, if anyone needs it, is setting opcache.optimization_level=0x7FFFB77F

matrix

SharedGrid I know but must be missing some latest patches (custom or not).
Plesk/CPanel/DA version. which is the same, doesn't have this error.
And unfortunately many customers still have to stick to 7.2 for their CMS version.
I tried cloning a wensite from the current live version to enhance and they get stuck in this problem.
Which is not a problem in Plesk/CPanel/DA, but is a big problem for me on Enhance since I can't move around 100 customers to my new Enhance infrastructure because of this.

I don't know why the OPcache behaves differently between the these panels (well actually only on ECP it acts wrong). I only know it's because of Pass 8 (Inlining) of the opcache, which makes some specific 'for' loops loop infinitely.

SharedGrid

matrix You said lowering the opcache optimisation level fixed the issue, does disabling opcache completely also fix the issue?

matrix

Vendoz I can't with the same exact website, because the CMS doesn't work with higher PHP.
I have however recent version of the CMS running fine on enhance on PHP 8.1+ for other customers.
So it's definitely a PHP 7.2 bug (or at least before 8.x, i can't test 7.4).

matrix

SharedGrid Yes, turning it off completely fixes it. But to fix it only one pass of the opcache need to be disabled. The pass 8 (inlining), using the bitmask 0x7FFFB77F, which disables only that one.

Also I tested with different webservers. I use Litespeed Enterprise where it works.
Both OLS and Apache have the same problem on ECP. So it's not only the lsphp bin, but the fpm too.

gmakhs

SharedGrid enhance offers old php version but they are not patched like cloudlinux ones or hardened so I wouldn't expect them to work properly, and I wouldn't call it enhances fault

matrix

gmakhs Never said it's their fault. But not only CloudLinux, even the base Plesk (Debian 13), DirectAdmin and CPanel on plain CentOS works. There must be some backported patch, however I wasn't able to track down which one.
Tomorrow I'll try isntalling directly the Plesk PHP debs with dpkg and change the lsphp executeable in OLS, to see if it works that way,

matrix

So we had a long conversation wtih Adam trough tickets. We found the problem.
It is indeed a problem in how the ECP php package were compiled.
The problem is that (in this case, but may apply to anything else too) the PHP version are compiled based on which Ubuntu version you are on. So php72 in Noble is compiled on Noble, in Jammy on Jammy, and so on.
But older PHP like 7.2 had to be compiled with old GCC, otherwise compiler bug appears, probably because of new compiler code which didn't exists at that time. In this case the new gcc causes this bug in php 7.2 opcache.

The solutions (pick one):
1) The easiest way: install Ubuntu 22.04 and use the Jammy repo for ECP if you host older websites.
2) The harder way if you want to stick to 24.04:

Add the Jammy ECP repo to your apt sources.list:
deb [arch=amd64, signed-by=/usr/share/keyrings/enhance.gpg] https://apt.enhance.com jammy main
Add a pin to avoid the package to be upgrade in the future (at least till the 24.04 version is fixed): create a new file /etc/apt/preferences.d/ecp-php72 and add inside this:
```
Package: ecp-php72
Pin: release o=Enhance, n=jammy
Pin-Priority: 1001

Package: *
Pin: release o=Enhance, n=jammy
Pin-Priority: -10
```
then save.
Run apt update and upgrade, it will ask to downgrande the ecp-php72 package. Confirm.

Wget the following missing libraries and install them with dpkg -i in the following order:

http://launchpadlibrarian.net/790503544/libldap-2.5-0_2.5.19+dfsg-0ubuntu0.22.04.1_amd64.deb
http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.24_amd64.deb
http://launchpadlibrarian.net/820811009/libtiff5_4.3.0-6ubuntu0.12_amd64.deb
http://launchpadlibrarian.net/740689248/libmagickcore-6.q16-6_6.9.11.60+dfsg-1.3ubuntu0.22.04.5_amd64.deb
http://launchpadlibrarian.net/740689251/libmagickwand-6.q16-6_6.9.11.60+dfsg-1.3ubuntu0.22.04.5_amd64.deb

Restart PHP

Anyway, the problem was solved.
Now Adam will look into recompiling old ECP PHP packages on Noble with the old compilers, to avoid this problem. But will probably take some time till we can see them in the repos.

I want to make a big THANK YOU to Adam for the time spent together to reproduce, confirm and fix this problem, which is kinda hidden and not easy to spot, unless you have a very specific PHP code. The debug took some time.
I've never encountered such skilled support in similar occasions, usually you get into a wall of level1/2/3 support and waste a week to get a response which doesn't sound like an AI. And when you get it they didn't even read your question correctly.
With Adam, instead, it was a pleasure to track down this problem. He immediately understood the problem and even proposed to write a patched code to the PHP source if we were to found the bug was there. Really a pleasure to talk with competent people, real sysadmin and coders, that get what you are talking about instantly. This is what makes this project very special.

matrix

Updated the previous post with a better formatted guide.

gmakhs

matrix with Adam is a pleasure to talk , and track issues he is helpful, but enhance development doesn't live up to the same quality and speed.....