php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #75579 All Interned Strings Free memory used and PHP crashes
Submitted: 2017-11-27 10:32 UTC Modified: 2017-12-22 10:30 UTC
Votes:5
Avg. Score:5.0 ± 0.0
Reproduced:5 of 5 (100.0%)
Same Version:4 (80.0%)
Same OS:2 (40.0%)
From: post at minhost dot no Assigned: dmitry (profile)
Status: Closed Package: opcache
PHP Version: 7.1.12 OS: CentOS 7.4
Private report: No CVE-ID: None
 [2017-11-27 10:32 UTC] post at minhost dot no
Description:
------------
We are running PHP-FPM with OPCache. The servers are shared hosting servers with many WordPress sites. After upgrading from PHP 7.1.11 to PHP 7.1.12, several PHP sites started to crash, and after a while, they was all down. When downgrading to PHP 7.1.11, all works correct again.

It seems to be opcache interned_strings_buffer, that is the cause. For example when still running PHP 7.1.11 it looks like this in a PHP info page:

Interned Strings Used memory	369608
Interned Strings Free memory	3824696

After upgrading to PHP 7.1.12 it looks like this (and PHP sites stops working):
Interned Strings Used memory	4194272
Interned Strings Free memory	32

I have not tested this on PHP 7.0.26, but I suspect the problem is the same on that version.

Conclusion: So, after upgrade toi PHP 7.1.12, ALL available "Interned Strings Free memory" is used, and PHP sites start going down. Downgrading to PHP 7.1.11 solves the problem.

In apache error log for the domains, I had lines like this:
Sun Nov 26 12:00:53.161278 2017] [proxy_fcgi:error] [pid 25870:tid 139679914243840] [client 176.74.214.18:8405] AH01067: Failed to read FastCGI header

[Sun Nov 26 12:00:53.161304 2017] [proxy_fcgi:error] [pid 25870:tid 139679914243840] (104)Connection reset by peer: [client 176.74.214.18:8405] AH01075: Error dispatching request to :

In php-fpm.log I had lines like this:
[26-Nov-2017 11:52:04] NOTICE: [pool USERNAME] child 25329 started

[26-Nov-2017 11:52:04] WARNING: [pool USERNAME] child 24302 exited on signal 11 (SIGSEGV) after 1.864571 seconds from start


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-11-28 08:28 UTC] post at minhost dot no
If it helps, here is my opcache settings (please note this is servers with 128 GB RAM):

opcache.memory_consumption=32768
opcache.interned_strings_buffer=4
opcache.max_accelerated_files=1000000
opcache.revalidate_freq=0
opcache.validate_timestamps=1
opcache.fast_shutdown=1
opcache.enable_cli=0
opcache.validate_permission=1
opcache.validate_root=1
opcache.use_cwd=1
opcache.revalidate_path=1
opcache.enable_file_override=1
opcache.file_cache=/var/www/tmp/.opcache
opcache.file_cache_only=0
opcache.max_wasted_percentage=10
 [2017-11-29 10:31 UTC] post at minhost dot no
Still waiting hoping someone could look into this. Currently I am not able to upgrade from PHP 7.1.11 because of this, as it causes all PHP sites to go down. Also when looking at the changelog for PHP 7.1.12 http://www.php.net/ChangeLog-7.php#7.1.12 it has this bugfix for opcache: "Fixed bug #75373 (Warning Internal error: wrong size calculation)." However the bug was reported for PHP 7.2.x, so maybe it does not apply to PHP 7.1.x? Could it be this fix that cause the new bug I experience?: https://bugs.php.net/bug.php?id=75373
 [2017-11-29 10:40 UTC] spam2 at rhsoft dot net
opcache.memory_consumption=64
opcache.interned_strings_buffer=8

http://blog.jpauli.tech/2015/03/05/opcache.html

OPCache allocates one segment of shared memory when PHP starts, once for all, and never frees it nor fragments it
 [2017-11-29 10:47 UTC] post at minhost dot no
@spam2 at rhsoft dot net: And how does that information help me? All I know is that on both my production servers, when upgrading from PHP 7.1.11 to 7.1.12, on both servers PHP sites start crashing. When downgrading to PHP 7.1.11, everything works fine again. As sait previous, I suspect that opcache.interned_strings_buffer is the cause (but it might be something else), because on PHP 7.1.12 Interned Strings Free memory is all used up with nothing left like this:
Interned Strings Used memory	4194288
Interned Strings Free memory	16

But when I downgrade to PHP 7.1.11, there is always plenty Interned Strings Free memory left (and PHP sites works without going down). But the only thing I know for sure, is that there is a change in PHP 7.1.12 that cause PHP sites to go down.
 [2017-11-29 10:54 UTC] spam2 at rhsoft dot net
it helps you in the way that your server no longer crashs and you can assume that with 7.2/7.3 you will need higher values - most of the performance improvements happen in the opcache area

it depends heavily on the numer of scripts called due runtime how high that values need to be - on our mainserver with 8000 scripts avg cached anything below 256/8 don't work proper at all and when you edit a file it's opcached version won't be reused at all because of "OPCache allocates one segment of shared memory when PHP starts, once for all, and never frees it nor fragments it" which is a design decision in favour of performance

https://git.php.net/?p=php-src.git&a=search&h=HEAD&st=commit&s=interned
 [2017-11-29 11:07 UTC] post at minhost dot no
@spam2 at rhsoft dot net: You are wrong about this. I have allocated enough memory to opcache (32 GB). I am talking about "opcache.interned_strings_buffer", it should not crash PHP sites when it is to low. And in PHP 7.1.11 it does not. Suddenly in PHP 7.1.12 it uses all "Interned Strings Free memory" and sites start to go down.

I just now was able to reproduce this in a test VPS server wich only contain a few test sites. This is what I did:

I was using this value wich is default: opcache.interned_strings_buffer=4

Then I installed 8 vanilla WordPress sites, and visited them all. After that it looked like this in PHP info page:

Interned Strings Used memory	2094032
Interned Strings Free memory	2100272

Then on ONE of the WordPress sites I installed the following plugins:
All In One SEO Pack
Contact Form 7
Jetpack by WordPress.com
MetaSlider
TinyMCE Advanced
WooCommerce
Wordfence Security

Then it looked like this in PHP info page:
Interned Strings Used memory	4194288
Interned Strings Free memory	16

As you can see all interned strings free memory was suddenly used, if this would have been on PHP 7.1.11 that would NOT happen. And at this point sites start to stop working. Suddenly I can't log into wordpress admin area on some of the sites, they only give "HTTP ERROR 500", and apache error log give this:

[Wed Nov 29 11:53:08.623533 2017] [proxy_fcgi:error] [pid 10182:tid 139804087817984] [client 176.74.214.18:32283] AH01071: Got error 'PHP message: PHP Fatal error:  require_once(): Failed opening required '\x02' (include_path='.:/usr/local/php71/lib/php') in...

However on this test there was no errors in /usr/local/php71/var/log/php-fpm.log

So I am able to duplicate this on a test VPS server without any clients, only by installing a few WordPress sites and som extra plugins in one of them.
 [2017-11-29 11:16 UTC] nikic@php.net
PHP definitely shouldn't crash just because the interned string buffer runs full. The only recent change to interning in 7.0 I see is https://github.com/php/php-src/commit/d82805f097564558d94e3062e87b17d6ccae893f, which does not look problematic to me.

It might be that this is a round-about side-effect of bug #75573, which is a memory corruption issue that also seems to affect some WP plugins.
 [2017-11-29 11:20 UTC] post at minhost dot no
@nikic@php.net: Thank you. (Please note that I am running PHP 7.1.11 trying to upgrade to 7.1.12.) What can I do to give you more information so this could be solved?
 [2017-11-29 18:08 UTC] post at minhost dot no
@nikic@php.net: I have now applied the patch from bug #75573, I applied this patch: http://git.php.net/?p=php-src.git;a=commit;h=3b9ba7b6bd9e24bdbeca8e8e3f24cee2fccc51d8 - on my small test vps server, this seem to have solved the problem with crashing php sites. They no longer seem to crash and seem to function correctly!

However "Interned Strings Free memory" still runs full very quick, wich it never did in PHP 7.1.11. I can only speculate that in PHP 7.1.11 it did not correctly display what was actually used, and the reason it runs full in 7.1.12 is because this is fixed in this version so that it shows whats really is used? Would be nice if someone could confirm that?

I think we can consider closing my bug report, as it seems fixed in bug #75573. :)
 [2017-11-30 14:49 UTC] post at minhost dot no
I spoke to soon. After more testing on PHP 7.1.12 with the patch from bug #75573 applied: http://git.php.net/?p=php-src.git;a=commit;h=3b9ba7b6bd9e24bdbeca8e8e3f24cee2fccc51d8 - still the bug is present! After visiting about nine wordpress sites on the test server, the interned strings used memory became full and PHP sites started to crash and go down. From apache error log for one of the domains after it stopped working:

[Thu Nov 30 15:40:40.273796 2017] [proxy_fcgi:error] [pid 17990:tid 140238684755712] [client 176.74.214.18:42343] AH01071: Got error 'PHP message: PHP Fatal error:  Cannot declare interface JsonSerializable, because the name is already in use in /home/USER/domains/DOMAIN.TLD/public_html/wp-includes/compat.php on line 428\n'

[Thu Nov 30 15:40:41.697018 2017] [proxy_fcgi:error] [pid 17990:tid 140238693148416] [client 176.74.214.18:52760] AH01071: Got error 'PHP message: PHP Fatal error:  Cannot declare interface JsonSerializable, because the name is already in use in /home/USER/domains/DOMAIN.TLD/public_html/wp-includes/compat.php on line 428\n'

So this still prevents us from upgrading to PHP 7.1.12. Remeber the bug is not present in PHP 7.1.11
 [2017-11-30 15:09 UTC] post at minhost dot no
Looking closer at apache error log for the domains when PHP sites start to go down I also get errors like this:

[Thu Nov 30 15:38:32.577627 2017] [proxy_fcgi:error] [pid 17992:tid 140239062427392] [client 176.74.214.18:52690] AH01067: Failed to read FastCGI header

[Thu Nov 30 15:38:32.577665 2017] [proxy_fcgi:error] [pid 17992:tid 140239062427392] (104)Connection reset by peer: [client 176.74.214.18:52690] AH01075: Error dispatching request to :
 [2017-12-15 19:50 UTC] post at minhost dot no
I just now today installed the latest 7.1 development branch from here https://github.com/php/php-src/branches and tested it. I am sorry to see the problem is still not solved with any of the patches that have been made so far. PHP still crashes for me after visiting a few WordPress sites. This time it seemed to happen right after opcache got a automatic OOM restart.
 [2017-12-15 20:46 UTC] post at minhost dot no
Finnaly I made some progress: When I deactivate/remove opcache.file_cache= in my settings and only use opcache in memory, then PHP no longer crashes! So the bug is only visible if you use physical file cache in opcache in addition to opcache in memory. I was using file cache as a second level fallback cache (useful when opcache get empty during restarts etc).

Now that I have pinpointed the bug to be when you have enabled both normal opcache and physical file cache, can someone investigate this bug? Please? I have already posted the complete opcache settings above in this bug report.
 [2017-12-17 21:11 UTC] nikic@php.net
-Status: Open +Status: Analyzed
 [2017-12-17 21:11 UTC] nikic@php.net
The issue (presumably) is https://github.com/php/php-src/blob/master/ext/opcache/zend_file_cache.c#L229. If the script in the file cache uses an interned string, the string is not already in the interned string buffer, and the interned string buffer has run full, this simply directly returns the string from the string segment of the cached script. However, the string segment will be freed in https://github.com/php/php-src/blob/master/ext/opcache/zend_file_cache.c#L1467.

The reason why you are only seeing this issue now is due to https://github.com/php/php-src/commit/d82805f097564558d94e3062e87b17d6ccae893f and the fact that you use revalidate_path=1. Prior to this fix, if revalidate_path=1 opcache accidentally did not use the interned string buffer for most things, which is why the file cache also did not contain interned strings.
 [2017-12-17 21:19 UTC] nikic@php.net
-Assigned To: +Assigned To: dmitry
 [2017-12-17 21:19 UTC] nikic@php.net
@dmitry: Can you please take a look at this issue? I'm not sure what the best way to fix this is. Can we just zend_shared_alloc() the string in case the accel_new_interned_string() fails?

An easy way to reproduce this is to enable the file cache, cache some files with opcache.interned_strings_buffer=8 and then load them using opcache.interned_strings_buffer=0.
 [2017-12-18 18:10 UTC] dmitry@php.net
I committed a fix into the master branch https://github.com/php/php-src/commit/ce4fb228e033ef896517aea3d01eb8d9ac055366
Could you please verify, if this completely fixes the problem.
Then, I'll backport it into PHP-7.*
 [2017-12-19 11:17 UTC] post at minhost dot no
Thank you both, @nikic and @dmitry! I want to test the patch, however can I apply the patch from @dmitry directly into PHP 7.1.12? Or is it needed to use the 7.1.13 branch from https://github.com/php/php-src/tree/PHP-7.1.13 ?
 [2017-12-20 17:27 UTC] post at minhost dot no
@dmitry: I applied your patch to PHP 7.1.13RC1, but I get errors when compiling PHP with your patch, so I am not able to compile it. Please try to compile it your self to see the problem. Please look into this. Here is the complete output with errors:

/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c: In function 'zend_file_cache_serialize_ast':
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:286:49: error: 'ZEND_AST_CONSTANT' undeclared (first use in this function)
  if (ast->kind == ZEND_AST_ZVAL || ast->kind == ZEND_AST_CONSTANT) {
                                                 ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:286:49: note: each undeclared identifier is reported only once for each function it appears in
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c: In function 'zend_file_cache_serialize_zval':
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:349:5: warning: passing argument 1 of 'zend_file_cache_serialize_ast' makes pointer from integer without a cast [enabled by default]
     zend_file_cache_serialize_ast(GC_AST(ast), script, info, buf);
     ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:278:13: note: expected 'struct zend_ast *' but argument is of type 'int'
 static void zend_file_cache_serialize_ast(zend_ast                 *ast,
             ^
In file included from /usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:21:0:
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c: In function 'zend_file_cache_serialize_op_array':
/usr/local/directadmin/custombuild/php-7.1.13RC1/Zend/zend_compile.h:657:27: error: 'zend_op' has no member named 'literals'
  RT_CONSTANT_EX((op_array)->literals, node)
                           ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/Zend/zend_compile.h:622:20: note: in definition of macro 'RT_CONSTANT_EX'
  ((zval*)(((char*)(base)) + (node).constant))
                    ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:423:28: note: in expansion of macro 'RT_CONSTANT'
     opline->op1.constant = RT_CONSTANT(opline, opline->op1) - literals;
                            ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/Zend/zend_compile.h:657:27: error: 'zend_op' has no member named 'literals'
  RT_CONSTANT_EX((op_array)->literals, node)
                           ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/Zend/zend_compile.h:622:20: note: in definition of macro 'RT_CONSTANT_EX'
  ((zval*)(((char*)(base)) + (node).constant))
                    ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:426:28: note: in expansion of macro 'RT_CONSTANT'
     opline->op2.constant = RT_CONSTANT(opline, opline->op2) - literals;
                            ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:479:29: error: 'zend_arg_info' has no member named 'type'
     if (ZEND_TYPE_IS_CLASS(p->type)) {
                             ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:480:51: error: 'zend_arg_info' has no member named 'type'
      zend_bool allow_null = ZEND_TYPE_ALLOW_NULL(p->type);
                                                   ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:481:47: error: 'zend_arg_info' has no member named 'type'
      zend_string *type_name = ZEND_TYPE_NAME(p->type);
                                               ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:484:7: error: 'zend_arg_info' has no member named 'type'
      p->type =
       ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:485:27: error: 'zend_type' undeclared (first use in this function)
       (Z_UL(1) << (sizeof(zend_type)*8-1)) | /* type is class */
                           ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:487:18: error: expected ';' before 'type_name'
       (zend_type)type_name;
                  ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c: In function 'zend_file_cache_unserialize_ast':
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:908:49: error: 'ZEND_AST_CONSTANT' undeclared (first use in this function)
  if (ast->kind == ZEND_AST_ZVAL || ast->kind == ZEND_AST_CONSTANT) {
                                                 ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c: In function 'zend_file_cache_unserialize_op_array':
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1024:64: error: macro "ZEND_PASS_TWO_UPDATE_CONSTANT" passed 3 arguments, but takes just 2
     ZEND_PASS_TWO_UPDATE_CONSTANT(op_array, opline, opline->op1);
                                                                ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1024:5: error: 'ZEND_PASS_TWO_UPDATE_CONSTANT' undeclared (first use in this function)
     ZEND_PASS_TWO_UPDATE_CONSTANT(op_array, opline, opline->op1);
     ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1027:64: error: macro "ZEND_PASS_TWO_UPDATE_CONSTANT" passed 3 arguments, but takes just 2
     ZEND_PASS_TWO_UPDATE_CONSTANT(op_array, opline, opline->op2);
                                                                ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1079:10: error: 'zend_arg_info' has no member named 'type'
     if (p->type & (Z_UL(1) << (sizeof(zend_type)*8-1))) { /* type is class */
          ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1079:39: error: 'zend_type' undeclared (first use in this function)
     if (p->type & (Z_UL(1) << (sizeof(zend_type)*8-1))) { /* type is class */
                                       ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1080:31: error: 'zend_arg_info' has no member named 'type'
      zend_bool allow_null = (p->type & (Z_UL(1) << (sizeof(zend_type)*8-2))) != 0; /* type allow null */
                               ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1081:47: error: 'zend_arg_info' has no member named 'type'
      zend_string *type_name = (zend_string*)(p->type & ~(((Z_UL(1) << (sizeof(zend_type)*8-1))) | ((Z_UL(1) << (sizeof(zend_type)*8-2)))));
                                               ^
/usr/local/directadmin/custombuild/php-7.1.13RC1/ext/opcache/zend_file_cache.c:1084:7: error: 'zend_arg_info' has no member named 'type'
      p->type = ZEND_TYPE_ENCODE_CLASS(type_name, allow_null);
       ^
make: *** [ext/opcache/zend_file_cache.lo] Error 1
make: *** Waiting for unfinished jobs....
 [2017-12-20 17:38 UTC] nikic@php.net
@post at minhost dot no: How did you apply the patch? It looks to me like you copy&pasted the entire file from the master branch maybe?

Here's a variant of the patch that should work on 7.1: https://gist.github.com/nikic/31e8e4510a0ed84cea543b50d78ae431 It can be applied using "git apply" or "patch".
 [2017-12-20 17:38 UTC] spam2 at rhsoft dot net
are you aware that you can download tarballs up to a specfici commit at every point in time?

https://git.php.net/?p=php-src.git;a=shortlog;h=refs/heads/PHP-7.1
 [2017-12-20 17:44 UTC] post at minhost dot no
@nikic: Yes I copied the entire patch from here: https://github.com/php/php-src/commit/ce4fb228e033ef896517aea3d01eb8d9ac055366 and replaced it in 7.1.rc1 at ext/opcache/zend_file_cache.c

Thank you for helping. I will try again with the variant you created for me at https://gist.github.com/nikic/31e8e4510a0ed84cea543b50d78ae431
 [2017-12-20 19:07 UTC] post at minhost dot no
I have tested now by using the patch from @nikic: https://gist.github.com/nikic/31e8e4510a0ed84cea543b50d78ae431 - I used the patch on PHP 7.1.13 RC1

PHP no longer crashes, so that problem seem to be solved. However there is a new problem with physical file cache when using this patch. Previous all PHP files would be physical cached in /home/USERNAME/.opcache/ - but after applying the patch, it seems PHP files is only physical cached up to the point when there is not enough free memory in opcache left.

On test server I have this in PHP info page when physical of new PHP files stop to cache new files:

Used memory	129591600
Free memory	592

Also so far there seem to be now more OOM restarts when there is not enough free memory. It just does not empty the cache anymore it seems.

Back to the new problem with physical. On the test server I have 9 wordpress sites. After applying the patch, I started visiting one after the other, including logging into wordpress control panel on each of them. Wordpress site 1 to 4 was cached in physical file cache, then after visiting the rest of the wordpress sites, none of them would be cached in physical opcache.

So, please enable both normal opcache in memory and physical file cache, then visit enough sites to use almost all opcache memory, then opcache no longer restart and free up the memory, and also new PHP pages you visit will no longer be cached in physical file cache.

So this patch solved PHP crashes, but create new bugs.
 [2017-12-21 08:46 UTC] dmitry@php.net
OOM shouldn't necessary lead to "restart". Only in case of "wasted" memory. Otherwise you would fill and trash SHM continuously.


Anyway, thanks for perseverance and try the next version of the patch
https://gist.github.com/dstogov/c84a2579600db45282e49cbd4624f7a0 (it's for PHP-7.1)
 [2017-12-21 09:16 UTC] post at minhost dot no
@dmitry: Thanks for the patch for PHP 7.1, however your patch: https://gist.github.com/dstogov/c84a2579600db45282e49cbd4624f7a0 looks identical to the patch @nikic created at https://gist.github.com/nikic/31e8e4510a0ed84cea543b50d78ae431 - it was already @nikic patch I tested. So there is now new change in your patch for PHP 7.1

At the moment I am not 100% sure that previous behaviour of PHP 7.1.11 was to cache all files in disk even when there was no opcache memory left. So I will test it in PHP 7.1.11 now to see if it is a new bug with the patch or not. I will let you know.
 [2017-12-21 09:35 UTC] spam2 at rhsoft dot net
> OOM shouldn't necessary lead to "restart". Only in case 
> of "wasted" memory. Otherwise you would fill and trash 
> SHM continuously

the problem with this is if something spiders over the server and calls a ton of normally not often called URLs opcache may get filled with irrelevant stuff while there is no space for current real workload and that degrades performance serious

automatic fill and trash would fix that - with the current behavior you need to manually reload
 [2017-12-21 09:44 UTC] post at minhost dot no
@spam2: I am happy with the current way OOM and restart works. Every night opcache memory is completely emptied anyhway when apache and php-fpm is reloaded. Also that discussion is off-topic to this bug report.
 [2017-12-21 09:50 UTC] post at minhost dot no
@dmitry: I have now tested in PHP 7.1.11, and the behaviour is the same as with PHP 7.1.13 RC1 + your patch. When there is not enough memory left in opcache, then  new files will also not be cached in file cache on disk. So that is not a bug.

I have tested your patch thorough now, and it seems to work correct, PHP no longer crashes. It works! Please add your patch to the 7.x branch. It would be fantastic if we could get the patch already in PHP 7.1.13 and PHP 7.0.27 in january 2018! Thanks!
 [2017-12-21 11:32 UTC] dmitry@php.net
Oh sorry, I posted invalid patch because of mess with branched.
This is the patch https://gist.github.com/dstogov/69e9a08038dcaf3c2e72ea4b1676242c
 [2017-12-21 11:40 UTC] post at minhost dot no
@dmitry: Thank you for a new patch. But what are the new patch supposed to fix or improve? Because the first patch seemed to work correct as explained in my previous reply.

Anyway, I will of course test you new patch, but I don't have time until several hours later.
 [2017-12-21 11:47 UTC] dmitry@php.net
It's going to load scripts cached in files into regular memory if SHM is full. Of course, it would require more extra work on each request, but this should be cheaper then recompile everything.
 [2017-12-21 11:52 UTC] post at minhost dot no
I am not sure I completely understand you correct. However it makes me worry. Please note that the first patch fixed the problem reported in this bug. Then I made a mistake in a reply, wich I corrected in a new reply, confirming that is now works correct. File cache on disk works correct with your first patch, it works correct and the same way as in PHP 7.1.11 wich do not have the bug.

So I don't understand what else you are trying to fix? The file cache on disk should be exactly that, on disk, not in memory. And it works correct with your first patch.
 [2017-12-21 11:55 UTC] post at minhost dot no
I think your new patch is not related to this bug, I think it is a new feature you are adding, and I don't like it. It has been enough trouble now. Please consider to not add this new feature, then we have to study and test your new feature a lot before using it in production. It works correct with your first patch, no need to change behaviour with a new feature.
 [2017-12-21 11:55 UTC] spam2 at rhsoft dot net
> The file cache on disk should be exactly that, on disk, not in memory

the primary opcache is always memory - normally SHM - the file cache is *optional* to survive restarts/reboots without the need to compile each script again

the point here is: what to do when the SHM is full
 [2017-12-21 12:00 UTC] post at minhost dot no
And the answer is: When SHM is full, don't do anything new regarding file cache on disk, but keep it the way it was working in PHP 7.1.11
 [2017-12-21 12:01 UTC] dmitry@php.net
This is not a new feature. file cache worked in this way, if it wasn't able to allocate enough shared memory for the script at once, but it didn't allocate additional blocks for "overflowing" interned strings before (this was the bug).
With the first patch, if there is no enough memory for "overflowing" interned string, we fall-back to recompilation from source. With the second patch we still load precompiled version from file, but not to shared but regular memory.
 [2017-12-21 12:05 UTC] post at minhost dot no
Ok, thanks for explainging it to me, @dmitry. Sorry I misunderstood the change. I will test your newest patch later tonight, and let you know if it works correct for me.
 [2017-12-21 17:10 UTC] post at minhost dot no
@dmitry: I have now tested your newest patch against PHP 7.1.13RC1, and I did not find any problems. PHP does not crash. The behaviour seemed to be the same as with your previous patch. Anyway, it seems to work fine! Can you please apply your newest patch to the 7.x branch? I expect it will be applied to all of PHP 7.0.x, 7.1.x and 7.2.x? Hopefully it will be in time for the release of PHP 7.0.27 and 7.1.13? Thanks!
 [2017-12-21 20:38 UTC] nikic@php.net
@dmitry: With your latest patch, if the interned string buffer is full, but SHM is not full, won't we end up repeatedly allocating SHM memory that's not going to be used?
 [2017-12-21 20:49 UTC] dmitry@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=37bf8bdc1494abb2ce5cac40e0be80e23682f851
Log: Fixed bug #75579 (Interned strings buffer overflow may cause crash)
 [2017-12-21 20:49 UTC] dmitry@php.net
-Status: Analyzed +Status: Closed
 [2017-12-21 20:59 UTC] dmitry@php.net
@nikic, Sorry, I missed your last comment.
I think, the patch is OK.
If we have free SHM memory, "interned" strings are going to be allocated in additional "SHM" blocks and then reused on next requests.
If there are no enough SHM, we fall back to process memory.

If you think, something is wrong, please, explain.
 [2017-12-21 21:05 UTC] post at minhost dot no
@dmitry: I am frustrated to see you applied the patch to PHP 7.1.14 and not to PHP PHP 7.1.13 - We have production servers with a lot of shared hosting clients running on PHP 7.1.11 - and now we will not be able to upgrade before february 2018. Remember this bug completely makes PHP useless for us that is using file cache as secondary fallback cache. Also remember I reported the bug already 27 november. I feel it is not a priority because you think not so many are using this setup. :(
 [2017-12-21 21:16 UTC] post at minhost dot no
@dmitry: My understanding of interned strings is that there should never be saved more interned strings then the limit set in .ini setting:

opcache.interned_strings_buffer=

So I am confused why it seems you have made it so that it will save interned strings in memory when it exceed the limit we set in .ini setting "opcache.interned_strings_buffer="

Please see: http://php.net/manual/en/opcache.configuration.php

Quote from that link:

"opcache.interned_strings_buffer
The amount of memory used to store interned strings, in megabytes. This configuration directive is ignored in PHP < 5.3.0."

So why do you make it so that it stores more interned strings then allocated in this setting?
 [2017-12-21 21:22 UTC] nikic@php.net
@dmitry: Yes sorry, I had a logic error. The bailout only happens if both the interned string buffer *and* SHM are full, so there is no problem.
 [2017-12-21 21:39 UTC] dmitry@php.net
I just committed patch to PHP-7.1 and above branches.
I can't commit into branches already detached by release managers.

If interned string buffer is overflown, we have to keep extra strings somewhere. In case we load script into SHM, we have to copy these extra strings to SHM as well (or we will crash). Or we can load both script and strings into process memory, and then reload again on each request.
 [2017-12-21 21:50 UTC] nikic@php.net
I've opened bug #75720 to track the other issue that was mentioned in here, regarding the file cache not being populated after SHM runs full.
 [2017-12-21 22:47 UTC] spam2 at rhsoft dot net
> @dmitry: I am frustrated to see you applied the patch 
> to PHP 7.1.14 and not to PHP PHP 7.1.13 

the ship for 7.1.13 has already sailed, RC is out and release on 2017/01/04

> We have production servers with a lot of shared hosting clients 
> running on PHP 7.1.11 and now we will not be able to upgrade 
> before february 2018

this is not true!

first opcache works pretty fine without the file cache at all and we even build PHP with --disable-opcache-file all the time - the SHM cache becomes quickly hot for relevant scripts where it matters

second you can apply the patch at your own at build-time, that's the great thing about opensource - with your argumentation we could not run Kernel 4.14 on our Trafficserver while 4.13 on Fedora recieves no longer security updates but we run 4.14.7 in production (https://github.com/apache/trafficserver/issues/2908)
 [2017-12-22 09:26 UTC] nikic@php.net
@post at minhost dot no: I've written Anatol to check if this fix can be cherry-picked into the release branches. I think it would be good to do this, as 7.0.27 is going to be the last active support release of PHP 7.0, so if it doesn't go in there, it will probably not be fixed at all.

@rhsoft: People have different requirements. Just because your setup does not benefit from the file cache, does not mean that it cannot be beneficial in other configurations. Shared hosting with too many instances to cache everything in SHM sounds like exactly the kind of setup that benefits from the secondary file cache.
 [2017-12-22 10:26 UTC] dmitry@php.net
@nikic, the patch wasn't committed into PHP-7.0 at all (the branch was already closed).
 [2017-12-22 10:30 UTC] nikic@php.net
@dmitry: Yeah, I know. That's why I asked Anatol if he can cherry-pick it. I think it would make sense.
 [2017-12-22 18:02 UTC] ab@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=da61c7a2a47555c6ecb1289154f3e27b2989324f
Log: Fixed bug #75579 (Interned strings buffer overflow may cause crash)
 [2017-12-22 18:04 UTC] ab@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=790f2061abf89e8c3891077ed16bd992a4fa2698
Log: Fixed bug #75579 (Interned strings buffer overflow may cause crash)
 [2017-12-28 17:03 UTC] pollita@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=841e7a192259e3e8ec34645176a2f4565e7e8877
Log: Fixed bug #75579 (Interned strings buffer overflow may cause crash)
 [2017-12-29 09:40 UTC] post at minhost dot no
Now that is seems the patch for this bug is added to the coming PHP 7.2.1 and PHP 7.0.27, could I please ask that someone also add this to the coming PHP 7.1.13, so that is not the only version that does not get the fix for the next release?

PHP 7.1.13 is the main version we are using, and on production servers we really do not like to patch PHP, but use the latest stable. However if this fix is not added to PHP 7.1.13, there will not be any stable PHP 7.1.x until end of february when PH 7.1.14 is released.
 [2018-01-03 02:03 UTC] pollita@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=b44a1556aa861863ddc721de56f12642579adfd0
Log: Fixed bug #75579 (Interned strings buffer overflow may cause crash)
 [2018-01-13 11:58 UTC] post at minhost dot no
Please see Bug #75795 wich is a new bug report related to this one.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Nov 21 09:01:32 2024 UTC