php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #66460 PHP 5.5.7 and 5.5.8 horrible unstable
Submitted: 2014-01-10 18:39 UTC Modified: 2014-01-16 16:48 UTC
Votes:1
Avg. Score:5.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:1 (100.0%)
Same OS:0 (0.0%)
From: spam2 at rhsoft dot net Assigned: rasmus
Status: Closed Package: opcache
PHP Version: 5.5.8 OS: Linux
Private report: No CVE-ID:
 [2014-01-10 18:39 UTC] spam2 at rhsoft dot net
Description:
------------
PHP 5.5.6 is running fine with opcache
PHP 5.5.7 and 5.5.8


[Fri Jan 10 19:31:12.026088 2014] [core:notice] [pid 17100] AH00052: child pid 17108 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:13.027956 2014] [core:notice] [pid 17100] AH00052: child pid 17142 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:14.030639 2014] [core:notice] [pid 17100] AH00052: child pid 17150 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:14.030679 2014] [core:notice] [pid 17100] AH00052: child pid 17151 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:15.034780 2014] [core:notice] [pid 17100] AH00052: child pid 17152 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:15.034820 2014] [core:notice] [pid 17100] AH00052: child pid 17153 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:15.034827 2014] [core:notice] [pid 17100] AH00052: child pid 17154 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:15.034834 2014] [core:notice] [pid 17100] AH00052: child pid 17155 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:16.038616 2014] [core:notice] [pid 17100] AH00052: child pid 17156 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:16.038652 2014] [core:notice] [pid 17100] AH00052: child pid 17157 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:17.042936 2014] [core:notice] [pid 17100] AH00052: child pid 17159 exit signal Segmentation fault (11)
[Fri Jan 10 19:31:17.042968 2014] [core:notice] [pid 17100] AH00052: child pid 17160 exit signal Segmentation fault (11)
_________________________________________

on one of my testing-machines disable "opcache.enable_file_override" which was enabled with PHP 5.4 all the time makes it better, on another server no way - permenently segfaults until i stop to load opcache

the really strange is that with "ab -c 100 -n 50000" i am unable to reprocude a single segfault, sometimes depending on luck also not with browsing, the next time nearly any request is answered with a Firefox error that the server unexpectly closed the connection


Test script:
---------------
currently i have no easy test-script, only cms pages and complexer libraries used and as said: it may take hours to reproduce and in other cases i update a production server to PHP > 5.5.6 and it gets terrible unstable shortly after restart httpd


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2014-01-10 18:54 UTC] spam2 at rhsoft dot net
maybe the real configuration helps to reproduce

i would appreciate a unified diff for "rpmbuild" with reverts
the latest opcache changes to the state of 5.5.6 and matching
the 5.5.8 tarball to provide feedback 

in the meantime i try to reproduce it with a small script instead 
a large application, at the moment i am out of ideas

[zend]
zend_extension                            = "/usr/lib64/php/modules/opcache.so"
opcache.enable                            = 1
opcache.enable_cli                        = 0
opcache.load_comments                     = 0
opcache.save_comments                     = 0
opcache.use_cwd                           = 1
opcache.validate_timestamps               = 1
opcache.revalidate_freq                   = 5
opcache.revalidate_path                   = 0
opcache.max_accelerated_files             = 200
opcache.memory_consumption                = 16
opcache.max_file_size                     = 262144
opcache.enable_file_override              = 0
opcache.fast_shutdown                     = 1
opcache.error_log                         = "/www/_logs/php_error.log"
opcache.log_verbosity_level               = 1
 [2014-01-10 19:00 UTC] spam2 at rhsoft dot net
OK, i come tighter to the circumstances

on the machine where it permanently crahed for now re-enable "opcache.revalidate_path" seems to workarund the problem, but
i am far away to trust because it's too unpredictable on different
machines / websites using the same core-codebase of our applications
___________________________________________________

for now stable or at least more stable

[zend]
zend_extension                            = "/usr/lib64/php/modules/opcache.so"
opcache.enable                            = 1
opcache.enable_cli                        = 0
opcache.load_comments                     = 0
opcache.save_comments                     = 0
opcache.use_cwd                           = 1
opcache.validate_timestamps               = 1
opcache.revalidate_freq                   = 5
opcache.revalidate_path                   = 1
opcache.max_accelerated_files             = 200
opcache.memory_consumption                = 16
opcache.max_file_size                     = 262144
opcache.enable_file_override              = 0
opcache.fast_shutdown                     = 1
opcache.error_log                         = "/www/_logs/php_error.log"
opcache.log_verbosity_level               = 1
___________________________________________________

previous config which was stable over months

[zend]
zend_extension                            = "/usr/lib64/php/modules/opcache.so"
opcache.enable                            = 1
opcache.enable_cli                        = 0
opcache.load_comments                     = 0
opcache.save_comments                     = 0
opcache.use_cwd                           = 1
opcache.validate_timestamps               = 1
opcache.revalidate_freq                   = 5
opcache.revalidate_path                   = 0
opcache.max_accelerated_files             = 200
opcache.memory_consumption                = 16
opcache.max_file_size                     = 262144
opcache.enable_file_override              = 1
opcache.fast_shutdown                     = 1
opcache.error_log                         = "/www/_logs/php_error.log"
opcache.log_verbosity_level               = 1
 [2014-01-10 19:05 UTC] spam008 at evil dot net dot pl
I have EXEACLY same observation. It only happens on three vhosts running Typo3 (multiple versions). It happens on both of my WWW servers. I`m going back to 5.5.5 right now. I`m using opcache too.

PHP-FPM self compiled on Slackware64 14.1.

One of my opcache`s config:

zend_extension = "opcache.so"
opcache.memory_consumption = 256
opcache.interned_strings_buffer = 8
opcache.max_accelerated_files = 10000
opcache.revalidate_freq = 0
opcache.validate_timestamps = On
opcache.fast_shutdown = Off
opcache.enable = On
opcache.enable_cli = Off
opcache.max_file_size = 0
opcache.use_cwd = On
opcache.log_verbosity_level = 2
opcache.load_comments = On
opcache.save_comments = On
 [2014-01-10 19:21 UTC] spam2 at rhsoft dot net
crap - on the machine in front of me it makes no 

differnce how i configure opcache, it crashed right after restart 
and never generated a single page of our self-developed cms, after
restart httpd and open phpMyAdmin i wondered that this did not 
crash and after that for now it appears that my cms-page also runs
stable

this is completly unpredictable and i doubt that the other reporter
which confirmed this behavior and me are the only persons facing this

whatever changes are done to "opcache" they are broken

[zend]
zend_extension                            = "/usr/lib64/php/modules/opcache.so"
opcache.enable                            = 1
opcache.enable_cli                        = 0
opcache.load_comments                     = 0
opcache.save_comments                     = 0
opcache.use_cwd                           = 1
opcache.validate_timestamps               = 1
opcache.revalidate_freq                   = 5
opcache.revalidate_path                   = 1
opcache.max_accelerated_files             = 1000
opcache.memory_consumption                = 128
opcache.interned_strings_buffer           = 8
opcache.max_file_size                     = 262144
opcache.enable_file_override              = 0
opcache.fast_shutdown                     = 1
opcache.error_log                         = "/var/log/php_error.log"
opcache.log_verbosity_level               = 1
 [2014-01-11 12:17 UTC] spam008 at evil dot net dot pl
Setting opcache.revalidate_path = 1 didn`t help. I will try 5.5.8 without loading opcache. Just to be 100% sure it is opcache issue.
 [2014-01-11 12:17 UTC] spam008 at evil dot net dot pl
Setting opcache.revalidate_path = 1 didn`t help. I will try 5.5.8 without loading opcache. Just to be 100% sure it is opcache issue.
 [2014-01-11 13:33 UTC] spam2 at rhsoft dot net
confirmed

changes in "opcache.enable_file_override" and "opcache.revalidate_path" change things only unpredictable and far away from get the server stable

opcache.enable=0 for sure disables the bytecode-cache and the segfaults of httpd-workers are going away - so opcache *is broken* and the changelog of 5.5.7 and 5.5.8 are shoing *both* changes in opcache, 5.5.8 is even more broken then 5.5.7 because with 5.5.7 on my workstation it was mostly useable, with 5.5.8 and opcache no way 


why you would downgrade to 5.5.5 is unclear for me
5.5.6 is clean and stable with opcache
5.5.7 is the first broken release
 [2014-01-11 16:08 UTC] spam2 at rhsoft dot net
take tar "ext/opcache" from 5.5.6 tarball, unpack the 5.5.8 tarball, replace the folder with the older opcache, make a tarball and feed rpmbuild, after that you are done, have a stable webserver without known security issues or degraded performance

[root@asterisk:~]$ rpm -q --changelog php
* Sa Jan 11 2014
- replace "ext/opcache" in 5.5.8 tarball with sources from 5.5.6
  because PHP 5.5.7/5.5.8 are completly unstable with opcache

* Fr Jan 10 2014 
- update to 5.5.8
 [2014-01-11 21:43 UTC] spam008 at evil dot net dot pl
No segfaults after turning opcache off - confirmed.

I`ve downgraded to 5.5.5 (not to 5.5.6) becouse I`m hitting this "bug" https://bugs.php.net/bug.php?id=66107 (fixed in 5.5.8) with Typo3. So 5.5.6 and 5.5.7 are usless for me at the moment.

Going to try your solution with "ext/opcache".
 [2014-01-12 12:21 UTC] spam008 at evil dot net dot pl
No segfaults after replacing 5.5.8 ext/opcache with 5.5.6 ext/opcache. It would be nice to pinpoint this bug.
 [2014-01-13 02:12 UTC] phpdev at ehrhardt dot nl
Try combining the block_pass.c from the 5.5.6 tarball with the other files from the 5.5.8 sources. See https://bugs.php.net/bug.php?id=66474

I encountered that bug on Windows, but maybe it is the same problem.
 [2014-01-13 04:53 UTC] phpdev at ehrhardt dot nl
Are you using Superfish in your Typo3? Superfish and the changes in block_pass.c did not go along in my Drupal7 site, on Windows and on Centos 5.
 [2014-01-13 05:57 UTC] rasmus@php.net
If it is block_pass-related, why not just dial down the optimization level to check? Set it to 0 to turn off all optimizations and see if that makes a difference. Much easier than recompiling.
 [2014-01-13 06:23 UTC] phpdev at ehrhardt dot nl
I did not even know what block_pass.c was for, so for me recompiling was easier ;)

But you are right: with the opcache.optimization_level set to 0 it does not fail. The question is: what causes the failure after the changes in block_pass.c in this patch:
https://github.com/zendtech/ZendOptimizerPlus/commit/b73b6a5559d6ca4925a09ed284a4e93be06726ed

If you have followed the php-windows list: I have first bisected to find that patch and then split up the patch to find out which part of that patch did the harm.

I now have a Centos6 server where I can easily switch between PHP 5.5 NTS as mod_php and PHP 5.4 NTS as mod_fcgid. Only the mod_fcgid PHP fails when the optimization level is set to its default.

BTW, we are trespassing on another, possibly related bug. Mine is this one:
https://bugs.php.net/bug.php?id=66474
 [2014-01-13 09:14 UTC] spam2 at rhsoft dot net
> If it is block_pass-related

it does not matter opcache 5.5.6 is not broken

the much more interesting question is how long it takes that someone of the PHP developers realizes this bugreport or make the damage biug enough that they no longer can ignore it
 [2014-01-13 09:45 UTC] phpdev at ehrhardt dot nl
This commit solved my problems:
https://github.com/zendtech/ZendOptimizerPlus/commit/6bbd955b01baeb6957e41717bd2dda07bf1ab1e0

Download the new sources from https://github.com/zendtech/ZendOptimizerPlus/archive/master.zip and compile then to see if your problems are solved as well.
 [2014-01-15 19:54 UTC] spam008 at evil dot net dot pl
I`m going to add one patch per day to opcache starting with 5.5.6 as a base to pinpoint this problem. I will post my results here.
 [2014-01-15 20:21 UTC] spam008 at evil dot net dot pl
One shot - one kill ;)

This patch causes segfault:

http://git.php.net/?p=php-src.git;a=commit;h=d85ac7fb3f498455cfc1b051f8d48c9ba8ed8fdd

Going to test opcache 5.5.8 without this patch.
 [2014-01-15 21:14 UTC] rasmus@php.net
Yes, we know it was the block_pass change as mentioned in previous comments here and https://bugs.php.net/bug.php?id=66474 has a reproducing PHP code snippet. This issue was fixed, so if you want to help, make sure you are building current HEAD (triple-check that you are replacing the newly .so correctly and restarting) because everyone else who had this same issue is reporting it is solved in the current code. If you can still make it crash with the current code, try to find the code snippet that does it and verify that it doesn't crash with opcache.optimization_level=0
 [2014-01-15 21:30 UTC] spam008 at evil dot net dot pl
#66474 made my opcache stable again. Thank you.
 [2014-01-15 22:03 UTC] spam2 at rhsoft dot net
> #66474 made my opcache stable again. Thank you

unconfirmed, see my commetn at that bugreport

i replaced the "opcache" folder from 5.5.8 tarball with the linked
master and from the first request on our asterisk-backend segfaults

how can it be that *two* following stable releases are that broken
 [2014-01-16 01:05 UTC] rasmus@php.net
-Status: Open +Status: Closed -Assigned To: +Assigned To: rasmus
 [2014-01-16 01:05 UTC] rasmus@php.net
Look at the dates please. This was fixed after 5.5.8 was released which is why I asked you to try the current code, not the 5.5.8 release. And it wasn't noticed because many people run with the optimization_level set to 0 and it is also a weird edge-case script that triggers the bug.
 [2014-01-16 01:33 UTC] spam2 at rhsoft dot net
> Look at the dates please. This was fixed after 5.5.8 was released 
> which is why I asked you to try the current code, not the 5.5.8 
> release. 

i took the source of https://github.com/zendtech/ZendOptimizerPlus/archive/master.zip and replaced the "opcache" folder of 5.5.8 tarball the same way as i described above i did with the "opcache" folder from 5.5.6 to replace the broken 5.5.8 "ext/opcache"

> And it wasn't noticed because many people run with the 
> optimization_level set to 0 

i did not change the optimization_level setting anywhere

> and it is also a weird edge-case script that triggers the bug

PHP after 5.5.6 as well as https://github.com/zendtech/ZendOptimizerPlus/archive/master.zip triggers segfaults not only with edge-case scripts, it triggers them with any code we run the last 10 years, the only interesting is that apache benchmarks do not trigger it - but why do i need explain that all again?
 [2014-01-16 02:05 UTC] phpdev at ehrhardt dot nl
Did you ever try the block_pass.c from the 5.5.6 tarball and the other files from the 5.5.8 tarball?
 [2014-01-16 02:17 UTC] spam2 at rhsoft dot net
> Did you ever try the block_pass.c from the 5.5.6 tarball 
> and the other files from the 5.5.8 tarball?

no, only 5.5.6 opcache on 5.5.8 source and head opcache on 5.5.8 source
both have the same segfault-behavior obviously introduced in 5.5.7

i noticed this a few times before christmas on local machines but
not predictable to reproduce and calling the same page 100000 times
with a benchmark after restart httpd may not trigegr a single segfault

due a lot of updates in system libraries in the first front i was unsure
if there was something broken, starting with 2014/01/07 i upgraded servers
from Fedora 18/PHP 5.4 to Fedora 19 PHP/5.5, tests looked fine and after
teh first low-trafficserver the segfaults in the apache-errorlog started
heavily, so i downgraded to PHP 5.5.6 quickly

the day before i found something with such sgefaults in context of
"zend.enable_gc", disabled it and for a short time it looked OK

finally after 5.5.8 had the same issues and no predictable way to
reproduce i started playing around with configuration, one of the 
first things was to disable opcache, not a single segfault

since downgrade to 5.4 would have been way too much work i tried
the 5.5.6 opcache because i had PHP 5.5 over months on my development
machines and faced the problem not until december

so the only thing i currently can say is that in case of the Linux 
kernel *any* commit to opcache after 5.5.6 would have been *reverted*
completly 

> because many people run with the optimization_level set to 0

i doubt that because it's not default
 [2014-01-16 02:25 UTC] phpdev at ehrhardt dot nl
Sounds like a different bug. The only way to make sure is, really, trying the block_pass.c from the 5.5.6 tarball and the other files from the 5.5.8 tarball.
 [2014-01-16 02:29 UTC] spam2 at rhsoft dot net
> Sounds like a different bug

pretty sure because on https://bugs.php.net/bug.php?id=66474 you said
solved by https://github.com/zendtech/ZendOptimizerPlus/archive/master.zip

and that is why any change after 5.5.6 in ext/opcache should be reverted
because not a single bug there could have the same bad impact as the "fixes"
 [2014-01-16 02:34 UTC] rasmus@php.net
> > because many people run with the optimization_level set to 0
> i doubt that because it's not default

Nevertheless, it is true. It is certainly why I never noticed it on any of my production boxes.

And you still haven't answered the question of whether setting it to 0 affects what you are seeing. I know you didn't change it and that it worked in 5.5.6, but it would still help us narrow it down and determine if it is the same bug.
 [2014-01-16 02:56 UTC] phpdev at ehrhardt dot nl
I found the 'guilty' commit by bisecting the commits since 5.5.6. Then I found out that Superfish was the package that triggered the bug. After that Dmitry applied a fix, which solved my problems with Superfish. You can never be sure the fix covered all possible cases. That is why I asked you to mix the two tarballs.

The info you are supplying now will not give Rasmus or Dmitry enough to determine what goes wrong. And simply saying that all commits since 5.5.6 have to be reverted is a no-go either. There were bugs in 5.5.6. You did not notice them, but others did. These bugs have to be fixed and the only way to do that is applying similar fixes as the commits since then. As long as you cannot pinpoint what goes wrong in your case, the devs can never be sure what they can and what they can't change.
 [2014-01-16 11:55 UTC] spam2 at rhsoft dot net
> And you still haven't answered the question of whether 
> setting it to 0 affects what you are seeing

https://github.com/zendtech/ZendOptimizerPlus/archive/master.zip built standalone with "opcache.optimization_level = 0" does *not* crash

with default settings *it does crash* as with 5.5.7 / 5.5.8 sources

[root@asterisk:~]$ rpm -q --filesbypkg php-opcache 
php-opcache /usr/lib64/php/modules/opcache.so

what i *really* do not understand is that the first here happened is set the bugreport to "closed" - based on waht?
 [2014-01-16 16:46 UTC] rasmus@php.net
Well, others reported the base issue fixed in this bug. It appears you have a separate problem, but "horrible unstable" is not a useful bug report. If you can get segfaults, get a back trace. Narrow it down to a specific bit of code and open a report we can do something with.
 [2014-01-16 16:48 UTC] rasmus@php.net
-Block user comment: No +Block user comment: Yes
 
PHP Copyright © 2001-2015 The PHP Group
All rights reserved.
Last updated: Mon Jun 15 18:01:54 2015 UTC