php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #75744 Sporadic crashes when opcache enabled
Submitted: 2017-12-28 19:47 UTC Modified: 2018-01-02 20:56 UTC
Votes:2
Avg. Score:4.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:0 (0.0%)
Same OS:1 (100.0%)
From: kochuev at gmail dot com Assigned:
Status: Open Package: opcache
PHP Version: 7.0.26 OS: Oracle Linux
Private report: No CVE-ID: None
View Add Comment Developer Edit
Have you experienced this issue?
Rate the importance of this bug to you:

 [2017-12-28 19:47 UTC] kochuev at gmail dot com 
Description:
------------
There are sporadic crashed with the traces:

Program terminated with signal 11, Segmentation fault.
#0  0xfffffff80000001a in ?? ()
#0  0xfffffff80000001a in ?? ()
#1  0x00005563484baa4d in zend_hash_destroy (ht=0x7f9987280ab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1264
#2  0x00005563484bad19 in zend_array_destroy (ht=0x7f9987280ab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1324
#3  0x00005563484bac98 in i_zval_ptr_dtor (zval_ptr=0x7f998726bf60) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#4  zend_array_destroy (ht=0x7f9987280150) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1338
#5  0x00005563484dfed4 in i_zval_ptr_dtor (zval_ptr=0x7f9987281118) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#6  zend_object_std_dtor (object=0x7f99872810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:69
#7  0x00005563484e4c11 in zend_objects_store_del (object=0x7f99872810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:178
#8  0x000055634849d140 in i_zval_ptr_dtor (zval_ptr=0x7f99872696a0) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#9  zend_cleanup_user_class_data (ce=0x7f998720a800) at /usr/src/debug/php-7.0.25/Zend/zend_opcode.c:167
#10 0x0000556348499415 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:336
#11 0x00005563484a9598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#12 0x0000556348448411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#13 0x00005563482e8cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995


Another example:

Program terminated with signal 11, Segmentation fault.
#0  zend_string_release (s=0x4) at /usr/src/debug/php-7.0.25/Zend/zend_string.h:269
269		if (!ZSTR_IS_INTERNED(s)) {
#0  zend_string_release (s=0x4) at /usr/src/debug/php-7.0.25/Zend/zend_string.h:269
#1  free_zend_constant (zv=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_constants.c:41
#2  0x00005563484bbf85 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=<optimized out>, ht=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1025
#3  _zend_hash_del_el (p=0x7f998be85b90, idx=3164, ht=0x55634a3265a0) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1049
#4  zend_hash_reverse_apply (ht=0x55634a3265a0, apply_func=apply_func@entry=0x556348497440 <clean_non_persistent_constant>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1631
#5  0x00005563484977c3 in clean_non_persistent_constants () at /usr/src/debug/php-7.0.25/Zend/zend_constants.c:161
#6  0x00005563484997a5 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:347
#7  0x00005563484a9598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#8  0x0000556348448411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#9  0x00005563482e8cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
/var/www/corefiles_storage/core.php-fpm.6071.web138.eta.prod.ed2.etadirect.com.80.1514298742

Crashes are NOT gone if fast_shutdown is set to 0
Increasing interned_strings_buffer to a large values or settign it to zero does not help.
Disabling opcache makes crashes go

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-12-28 19:57 UTC] spam2 at rhsoft dot net 
PHP Version: 7.0.26
zend_hash_destroy (ht=0x7f9987280ab8) at /usr/src/debug/php-7.0.25/

i guess 7.0.26 is a lie then and so the question is if it still happens with the latest version
 [2017-12-28 20:07 UTC] kochuev at gmail dot com 
Yes, correct version for the traces given is 7.0.25 which is still on our prod servers, crashes started at 7.0.12 and still in place up to 7.0.26 (but I don't have core dumps for it).
 [2017-12-28 20:15 UTC] nikic@php.net 
These shutdown crashes are likely triggered by some form of memory corruption at an earlier point in time, so it will be hard to track down the root cause of this issue.

Two things to try would be:
 * Set opcache.optimization_level=0, to exclude one relatively common source of issues.
 * Set opcache.protect_memory=1 to immediately abort if the SHM memory is corrupted.
 [2017-12-29 13:21 UTC] kochuev at gmail dot com 
Tried opcache.optimization_level=0 - no noticeable change in the frequency of crashes (may be just a bit).

Traces for the crashes with opcache.protect_memory=1 (fast_shutdown => 1, optimization_level => 0x7FFFBFFF) are given below:

Program terminated with signal 11, Segmentation fault.
#0  php_stream_notification_free (notifier=0xffffffff) at /usr/src/debug/php-7.0.25/main/streams/streams.c:2211
2211        if (notifier->dtor) {
#0  php_stream_notification_free (notifier=0xffffffff) at /usr/src/debug/php-7.0.25/main/streams/streams.c:2211
#1  0x000055619d392227 in php_stream_context_free (context=0x7f7eb2c7dae0) at /usr/src/debug/php-7.0.25/main/streams/streams.c:2186
#2  0x000055619d3ed5d1 in zend_resource_dtor (res=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_list.c:76
#3  0x000055619d3ed623 in zend_close_rsrc (zv=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_list.c:230
#4  0x000055619d3eaeb2 in zend_hash_reverse_apply (ht=0x55619d7d8558 <executor_globals+536>, apply_func=apply_func@entry=0x55619d3ed610 <zend_close_rsrc>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1628
#5  0x000055619d3edc0c in zend_close_rsrc_list (ht=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_list.c:238
#6  0x000055619d3c8793 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:351
#7  0x000055619d3d8598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#8  0x000055619d377411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#9  0x000055619d217cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995

Program terminated with signal 11, Segmentation fault.
#0  0xfffffff80000001a in ?? ()
#0  0xfffffff80000001a in ?? ()
#1  0x000055619d3e9a4d in zend_hash_destroy (ht=0x7f7eb2c7eab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1264
#2  0x000055619d3e9d19 in zend_array_destroy (ht=0x7f7eb2c7eab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1324
#3  0x000055619d3e9c98 in i_zval_ptr_dtor (zval_ptr=0x7f7eb2c6ef60) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#4  zend_array_destroy (ht=0x7f7eb2c7e150) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1338
#5  0x000055619d40eed4 in i_zval_ptr_dtor (zval_ptr=0x7f7eb2c7f118) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#6  zend_object_std_dtor (object=0x7f7eb2c7f0c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:69
#7  0x000055619d413850 in zend_objects_store_free_object_storage (objects=objects@entry=0x55619d7d8670 <executor_globals+816>) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:99
#8  0x000055619d3c86e3 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:361
#9  0x000055619d3d8598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#10 0x000055619d377411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#11 0x000055619d217cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
 [2017-12-29 15:07 UTC] kochuev at gmail dot com 
More crashes with opcache.fast_shutdown=0 + opcache.protect_memory=1

Program terminated with signal 11, Segmentation fault.
#0  _zval_dtor_func (p=0x600000001) at /usr/src/debug/php-7.0.25/Zend/zend_variables.c:33
33      if (--GC_REFCOUNT(p)) {
#0  _zval_dtor_func (p=0x600000001) at /usr/src/debug/php-7.0.25/Zend/zend_variables.c:33
#1  0x00005642fb1064f5 in _zval_dtor (zvalue=0x7f0d8de7f660, zvalue=0x7f0d8de7f660) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:44
#2  free_zend_constant (zv=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_constants.c:36
#3  0x00005642fb12af85 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=<optimized out>, ht=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1025
#4  _zend_hash_del_el (p=0x7f0d92b6b710, idx=2872, ht=0x5642fd0835a0) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1049
#5  zend_hash_reverse_apply (ht=0x5642fd0835a0, apply_func=apply_func@entry=0x5642fb106440 <clean_non_persistent_constant>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1631
#6  0x00005642fb1067c3 in clean_non_persistent_constants () at /usr/src/debug/php-7.0.25/Zend/zend_constants.c:161
#7  0x00005642fb1087a5 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:347
#8  0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#9  0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#10 0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  php_stream_notification_free (notifier=0xffffffff) at /usr/src/debug/php-7.0.25/main/streams/streams.c:2211
2211        if (notifier->dtor) {
#0  php_stream_notification_free (notifier=0xffffffff) at /usr/src/debug/php-7.0.25/main/streams/streams.c:2211
#1  0x00005642fb0d2227 in php_stream_context_free (context=0x7f0d8de7db40) at /usr/src/debug/php-7.0.25/main/streams/streams.c:2186
#2  0x00005642fb12d5d1 in zend_resource_dtor (res=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_list.c:76
#3  0x00005642fb12d623 in zend_close_rsrc (zv=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_list.c:230
#4  0x00005642fb12aeb2 in zend_hash_reverse_apply (ht=0x5642fb518558 <executor_globals+536>, apply_func=apply_func@entry=0x5642fb12d610 <zend_close_rsrc>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1628
#5  0x00005642fb12dc0c in zend_close_rsrc_list (ht=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_list.c:238
#6  0x00005642fb108793 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:351
#7  0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#8  0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#9  0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  zend_object_std_dtor (object=0x7f0d8de80b60) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:59
59          if (EXPECTED(!(GC_FLAGS(object->properties) & IS_ARRAY_IMMUTABLE))) {
#0  zend_object_std_dtor (object=0x7f0d8de80b60) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:59
#1  0x00005642fb153c11 in zend_objects_store_del (object=0x7f0d8de80b60) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:178
#2  0x00005642fb129bc7 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de97568) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#3  zend_array_destroy (ht=0x7f0d8de801f8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1334
#4  0x00005642fb14eed4 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de81108) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#5  zend_object_std_dtor (object=0x7f0d8de810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:69
#6  0x00005642fb153c11 in zend_objects_store_del (object=0x7f0d8de810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:178
#7  0x00005642fb10c140 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de696a0) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#8  zend_cleanup_user_class_data (ce=0x7f0d8de0a800) at /usr/src/debug/php-7.0.25/Zend/zend_opcode.c:167
#9  0x00005642fb108415 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:336
#10 0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#11 0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#12 0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  0xfffffff80000001a in ?? ()
#0  0xfffffff80000001a in ?? ()
#1  0x00005642fb129a4d in zend_hash_destroy (ht=0x7f0d8de7eab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1264
#2  0x00005642fb129d19 in zend_array_destroy (ht=0x7f0d8de7eab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1324
#3  0x00005642fb129c98 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de6ef60) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#4  zend_array_destroy (ht=0x7f0d8de7e150) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1338
#5  0x00005642fb14eed4 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de7f118) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#6  zend_object_std_dtor (object=0x7f0d8de7f0c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:69
#7  0x00005642fb153c11 in zend_objects_store_del (object=0x7f0d8de7f0c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:178
#8  0x00005642fb10c140 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de6c6a0) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#9  zend_cleanup_user_class_data (ce=0x7f0d8de0a800) at /usr/src/debug/php-7.0.25/Zend/zend_opcode.c:167
#10 0x00005642fb108415 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:336
#11 0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#12 0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#13 0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  0xfffffff80000001a in ?? ()
#0  0xfffffff80000001a in ?? ()
#1  0x00005642fb129a4d in zend_hash_destroy (ht=0x7f0d8de80ab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1264
#2  0x00005642fb129d19 in zend_array_destroy (ht=0x7f0d8de80ab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1324
#3  0x00005642fb129c98 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de6bf60) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#4  zend_array_destroy (ht=0x7f0d8de80150) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1338
#5  0x00005642fb14eed4 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de81118) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#6  zend_object_std_dtor (object=0x7f0d8de810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:69
#7  0x00005642fb153c11 in zend_objects_store_del (object=0x7f0d8de810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:178
#8  0x00005642fb10c140 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de696a0) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#9  zend_cleanup_user_class_data (ce=0x7f0d8de0a800) at /usr/src/debug/php-7.0.25/Zend/zend_opcode.c:167
#10 0x00005642fb108415 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:336
#11 0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#12 0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#13 0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  0xfffffff80000001a in ?? ()
#0  0xfffffff80000001a in ?? ()
#1  0x00005642fb129a4d in zend_hash_destroy (ht=0x7f0d8de80ab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1264
#2  0x00005642fb129d19 in zend_array_destroy (ht=0x7f0d8de80ab8) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1324
#3  0x00005642fb129c98 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de6bf60) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#4  zend_array_destroy (ht=0x7f0d8de80150) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1338
#5  0x00005642fb14eed4 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de81118) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#6  zend_object_std_dtor (object=0x7f0d8de810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects.c:69
#7  0x00005642fb153c11 in zend_objects_store_del (object=0x7f0d8de810c0) at /usr/src/debug/php-7.0.25/Zend/zend_objects_API.c:178
#8  0x00005642fb10c140 in i_zval_ptr_dtor (zval_ptr=0x7f0d8de696a0) at /usr/src/debug/php-7.0.25/Zend/zend_variables.h:58
#9  zend_cleanup_user_class_data (ce=0x7f0d8de0a800) at /usr/src/debug/php-7.0.25/Zend/zend_opcode.c:167
#10 0x00005642fb108415 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:336
#11 0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#12 0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#13 0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  zend_string_release (s=0x6) at /usr/src/debug/php-7.0.25/Zend/zend_string.h:269
269     if (!ZSTR_IS_INTERNED(s)) {
#0  zend_string_release (s=0x6) at /usr/src/debug/php-7.0.25/Zend/zend_string.h:269
#1  free_zend_constant (zv=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_constants.c:41
#2  0x00005642fb12af85 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=<optimized out>, ht=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1025
#3  _zend_hash_del_el (p=0x7f0d92b6c650, idx=2994, ht=0x5642fd0835a0) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1049
#4  zend_hash_reverse_apply (ht=0x5642fd0835a0, apply_func=apply_func@entry=0x5642fb106440 <clean_non_persistent_constant>) at /usr/src/debug/php-7.0.25/Zend/zend_hash.c:1631
#5  0x00005642fb1067c3 in clean_non_persistent_constants () at /usr/src/debug/php-7.0.25/Zend/zend_constants.c:161
#6  0x00005642fb1087a5 in shutdown_executor () at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:347
#7  0x00005642fb118598 in zend_deactivate () at /usr/src/debug/php-7.0.25/Zend/zend.c:985
#8  0x00005642fb0b7411 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/debug/php-7.0.25/main/main.c:1856
#9  0x00005642faf57cde in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1995
---
Program terminated with signal 11, Segmentation fault.
#0  0x0000001500000001 in ?? ()
#0  0x0000001500000001 in ?? ()
#1  0x00005642fb1100d5 in _zval_get_string_func (op=op@entry=0x7f0d8de130d0) at /usr/src/debug/php-7.0.25/Zend/zend_operators.c:857
#2  0x00005642fb169d65 in ZEND_FAST_CONCAT_SPEC_TMPVAR_CONST_HANDLER () at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:42227
#3  0x00005642fb1579eb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:414
#4  0x00005642fb1a220f in zend_execute (op_array=0x7f0d8de5e0e0, op_array@entry=0x7f0d1ae06b68, return_value=return_value@entry=0x7f0d8de13030) at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:458
#5  0x00005642fb1188f3 in zend_execute_scripts (type=type@entry=8, retval=0x7f0d8de13030, retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-7.0.25/Zend/zend.c:1445
#6  0x00005642fb0b88a8 in php_execute_script (primary_file=primary_file@entry=0x7ffdb36cbfc0) at /usr/src/debug/php-7.0.25/main/main.c:2518
#7  0x00005642faf57c74 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1967
 [2017-12-29 17:52 UTC] nikic@php.net 
Not seeing anything obvious in these traces. They do not look like something triggered by protect_memory, so it's probably no simple SHM corruption.

Could it be that you are using a ZTS build of PHP?

Another thing that might or might not work is to run PHP under valgrind (using USE_ZEND_ALLOC=0 ZEND_DONT_UNLOAD_MODULES=1 valgrind php [args]) and see if this produces reliable memory errors.
 [2018-01-01 17:51 UTC] kochuev at gmail dot com 
No, we are not using ZTS build and pthreads. Also as the problem appears only in the heavily loaded situation it's hard to run valgrind on a sample script. Also, this is not a cli script, but complex web system, so I'm not sure how to run php-fpm under valgrind.

One thing that I noticed is that the number of crashes a bit reduces (-30%) with the following settings:

opcache.fast_shutdown=1
opcache.interned_strings_buffer=256
opcache.memory_consumption=1024

Btw all the crashes happen in the endpoints dealing with soap extension.

With opcache disabled number of crashes reduces -80%, but still, there are some, ex:

Program terminated with signal 11, Segmentation fault.
#0  zend_mm_alloc_small (size=<optimized out>, bin_num=6, heap=0x7f3ca6200040) at /usr/src/debug/php-7.0.25/Zend/zend_alloc.c:1306
1306            heap->free_slot[bin_num] = p->next_free_slot;
#0  zend_mm_alloc_small (size=<optimized out>, bin_num=6, heap=0x7f3ca6200040) at /usr/src/debug/php-7.0.25/Zend/zend_alloc.c:1306
#1  _emalloc_56 () at /usr/src/debug/php-7.0.25/Zend/zend_alloc.c:2380
#2  0x000055f14ae7e990 in _array_init (arg=arg@entry=0x7fff1c86aaf0, size=0) at /usr/src/debug/php-7.0.25/Zend/zend_API.c:1059
#3  0x000055f14ae60a08 in zend_try_ct_eval_array (result=0x7fff1c86aaf0, ast=0x7f3c4cc46a58) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:5933
#4  0x000055f14ae600e8 in zend_eval_const_expr (ast_ptr=ast_ptr@entry=0x7fff1c86ab68) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7577
#5  0x000055f14ae60c80 in zend_const_expr_to_zval (result=result@entry=0x7fff1c86abb0, ast=ast@entry=0x7f3c4cc46a58) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7036
#6  0x000055f14ae61470 in zend_compile_prop_decl (ast=ast@entry=0x7f3c4cc46ac0) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:5036
#7  0x000055f14ae676f2 in zend_compile_stmt (ast=0x7f3c4cc46ac0) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7154
#8  0x000055f14ae682e7 in zend_compile_stmt_list (ast=ast@entry=0x7f3c4cd50878) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:4429
#9  0x000055f14ae6786e in zend_compile_stmt (ast=ast@entry=0x7f3c4cd50878) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7095
#10 0x000055f14ae685f2 in zend_compile_class_decl (ast=ast@entry=0x7f3c4cdb9558) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:5376
#11 0x000055f14ae67a08 in zend_compile_stmt (ast=ast@entry=0x7f3c4cdb9558) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7163
#12 0x000055f14ae6a05a in zend_compile_top_stmt (ast=0x7f3c4cdb9558) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7069
#13 0x000055f14ae6a09f in zend_compile_top_stmt (ast=0x7f3c4cc46018) at /usr/src/debug/php-7.0.25/Zend/zend_compile.c:7064
#14 0x000055f14ae4512d in compile_file (file_handle=<optimized out>, type=<optimized out>) at Zend/zend_language_scanner.l:608
#15 0x00007f3c96c75920 in phar_compile_file (file_handle=<optimized out>, type=<optimized out>) at /usr/src/debug/php-7.0.25/ext/phar/phar.c:3337
#16 0x000055f14ae4524d in compile_filename (type=8, filename=filename@entry=0x7f3ca6214ba0) at Zend/zend_language_scanner.l:649
#17 0x000055f14af02777 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:29518
#18 0x000055f14aebb9eb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:414
#19 0x000055f14ae6d59a in zend_call_function (fci=fci@entry=0x7fff1c86b4d0, fci_cache=0x7f3ca6258700, fci_cache@entry=0x7fff1c86b4a0) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:867
#20 0x000055f14ae9a793 in zend_call_method (object=0x7f3ca6256c68, obj_ce=<optimized out>, fn_proxy=0x7f3ca6256c60, function_name=0x7f3ca625ff68 "autoloader::loadclass\001", function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=param_count@entry=1, arg1=0x7f3ca6214b20, arg2=arg2@entry=0x0) at /usr/src/debug/php-7.0.25/Zend/zend_interfaces.c:104
#21 0x000055f14ad80131 in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>) at /usr/src/debug/php-7.0.25/ext/spl/php_spl.c:421
#22 0x000055f14ae6d61c in zend_call_function (fci=fci@entry=0x7fff1c86b760, fci_cache=fci_cache@entry=0x7fff1c86b730) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:887
#23 0x000055f14ae6dcea in zend_lookup_class_ex (name=name@entry=0x7f3ca62a4560, key=0x7f3ca62f1820, use_autoload=use_autoload@entry=1) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:1049
#24 0x000055f14ae6e6a8 in zend_fetch_class_by_name (class_name=0x7f3ca62a4560, key=<optimized out>, fetch_type=fetch_type@entry=512) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:1395
#25 0x000055f14af00a54 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:5679
#26 0x000055f14aebb9eb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:414
#27 0x000055f14ae6d59a in zend_call_function (fci=fci@entry=0x7fff1c86b970, fci_cache=0x7f3ca6368a20, fci_cache@entry=0x0) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:867
#28 0x000055f14ae6dad9 in call_user_function_ex (function_table=function_table@entry=0x0, object=object@entry=0x7fff1c86bae0, function_name=function_name@entry=0x7fff1c86bac0, retval_ptr=retval_ptr@entry=0x7fff1c86bad0, param_count=<optimized out>, params=<optimized out>, no_separation=no_separation@entry=1, symbol_table=symbol_table@entry=0x0) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:675
#29 0x000055f14ae6db19 in call_user_function (function_table=function_table@entry=0x0, object=object@entry=0x7fff1c86bae0, function_name=function_name@entry=0x7fff1c86bac0, retval_ptr=retval_ptr@entry=0x7fff1c86bad0, param_count=<optimized out>, params=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_execute_API.c:657
#30 0x00007f3c95ef41fb in zim_SoapServer_handle (execute_data=<optimized out>, return_value=<optimized out>) at /usr/src/debug/php-7.0.25/ext/soap/soap.c:1863
#31 0x000055f14aefa4fb in ZEND_DO_FCALL_SPEC_HANDLER () at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:842
#32 0x000055f14aebb9eb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:414
#33 0x000055f14af0620f in zend_execute (op_array=0x7f3ca6273000, op_array@entry=0x7f3ca62ddea0, return_value=return_value@entry=0x7f3ca6213420) at /usr/src/debug/php-7.0.25/Zend/zend_vm_execute.h:458
#34 0x000055f14ae7c8f3 in zend_execute_scripts (type=type@entry=8, retval=0x7f3ca6213420, retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-7.0.25/Zend/zend.c:1445
#35 0x000055f14ae1c8a8 in php_execute_script (primary_file=primary_file@entry=0x7fff1c86e060) at /usr/src/debug/php-7.0.25/main/main.c:2518
#36 0x000055f14acbbc74 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.0.25/sapi/fpm/fpm/fpm_main.c:1967
 [2018-01-01 20:50 UTC] nikic@php.net 
You mention that this may be related to SOAP -- do you have the WSDL memory cache enabled? In that case this might be bug #75502, which may cause SOAP shutdown segfaults. Your traces don't quite look like this problem though :/
 [2018-01-02 20:44 UTC] kochuev at gmail dot com 
We do have soap.wsdl_cache_enabled=1 but after a short test, it seems that setting that to 0 have no effect on segfaults. Any idea what to try else?
 [2018-01-02 20:56 UTC] nikic@php.net 
Are you using any 3rd party extensions? Especially something SHM based like apcu?
 
PHP Copyright © 2001-2019 The PHP Group
All rights reserved. 		Last updated: Sun Jun 16 01:01:28 2019 UTC