php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #62418 php-fpm master process crashes
Submitted: 2012-06-26 05:56 UTC Modified: 2013-02-18 00:35 UTC
Votes:6
Avg. Score:4.2 ± 0.9
Reproduced:5 of 5 (100.0%)
Same Version:1 (20.0%)
Same OS:1 (20.0%)
From: php at scratchspace dot com Assigned:
Status: No Feedback Package: FPM related
PHP Version: 5.3.14 OS: CentOS 6.2 x86_x64
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: php at scratchspace dot com
New email:
PHP Version: OS:

 

 [2012-06-26 05:56 UTC] php at scratchspace dot com
Description:
------------
php-fpm master process crashes randomly on lightly loaded server running 
nginx/0.8.54 with various CMS applications. It is unknown how to reproduce the 
crash, but occurs in both php 5.3.13 and 5.3.14.

Actual result:
--------------
5.3.14 backtrace

Core was generated by `php-fpm:'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000016b5300 in ?? ()
#1  0x00000000006e02f3 in fpm_event_epoll_wait (queue=<value optimized out>, 
timeout=<value optimized out>)
    at /usr/src/debug/php-5.3.14/sapi/fpm/fpm/events/epoll.c:143
#2  0x00000000006d508f in fpm_event_loop (err=0) at /usr/src/debug/php-
5.3.14/sapi/fpm/fpm/fpm_events.c:401
#3  0x00000000006d01f7 in fpm_run (max_requests=0x7fffd103136c) at 
/usr/src/debug/php-5.3.14/sapi/fpm/fpm/fpm.c:110
#4  0x00000000006d79a4 in main (argc=1, argv=0x7fffd10334d8) at 
/usr/src/debug/php-5.3.14/sapi/fpm/fpm/fpm_main.c:1778


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2012-06-26 06:10 UTC] rasmus@php.net
Could you check if it is related to epoll? In your php-fpm.conf file, set:

events.mechanism = poll

to try with "poll" instead. You can also try setting it to "select"
 [2012-06-26 06:10 UTC] rasmus@php.net
-Status: Open +Status: Feedback
 [2012-06-27 18:28 UTC] php at scratchspace dot com
We've been up and running for 24 hours with "events.mechanism = poll" and no 
problems. Let's give it a couple of days to be sure this workaround works. Is this 
issue specific to epoll and the way we have php-fpm configured (to use minimal 
resources), or is this is a bug in php-fpm that can ultimately be fixed?
 [2012-06-27 18:28 UTC] php at scratchspace dot com
-Status: Feedback +Status: Open
 [2012-06-27 20:10 UTC] rasmus@php.net
Not sure. Many people out there run with epoll without problems.
 [2012-07-04 10:30 UTC] fat@php.net
Thank you for this bug report. To properly diagnose the problem, we
need a backtrace to see what is happening behind the scenes. To
find out how to generate a backtrace, please read
http://bugs.php.net/bugs-generating-backtrace.php for *NIX and
http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32

Once you have generated a backtrace, please submit it to this bug
report and change the status back to "Open". Thank you for helping
us make PHP better.

If you were able to give us a complete and detailed stack trace it would be 
useful. 

thx
 [2012-07-04 10:30 UTC] fat@php.net
-Status: Open +Status: Feedback
 [2013-02-12 10:59 UTC] webmaster at jbfavre dot org
Hello,
We have the same problem, with PHP 5.3.21 behind Nginx 1.2.6.
Happens, sometimes, at PHP-FPM reload, after updating the code.
We're running a Symfony web app, on Ubuntu 10.04 Lucid with home-made PHP packages.

Enabled modules
---------------
amqp 1.0.9
apc 3.1.9
curl 5.3.21
gd 5.3.21
geoip 1.0.8
igbinary 1.1.1
intl 5.3.21
memcached 2.1.0
mongo 1.3.0
mysql 5.3.21
pinba 1.1.0
redis 2.2.2

Have just modified FPM configuration to use 'poll' instead of default 'epoll'.
Will see what happen.

2 backtraces available (I can provide coredumps upon request)
------------------------------------------------------------
###############################
#0  0x0000444c48434749 in ?? ()
#1  0x0000000000761703 in fpm_event_epoll_wait (queue=0x1f270e8, timeout=3) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/events/epoll.c:143
#2  0x0000000000755e0f in fpm_event_loop (err=0) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm_events.c:403
#3  0x0000000000750d67 in fpm_run (max_requests=0x7fff3704f79c) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm.c:113
#4  0x000000000075872e in main (argc=3, argv=0x7fff37051908) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm_main.c:1838


###############################
#0  0x00000000007558d6 in fpm_event_fire (ev=0x2ae4d78) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm_events.c:459
#1  0x0000000000761703 in fpm_event_epoll_wait (queue=0x2ae4d78, timeout=3) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/events/epoll.c:143
#2  0x0000000000755e0f in fpm_event_loop (err=0) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm_events.c:403
#3  0x0000000000750d67 in fpm_run (max_requests=0x7fff94e7286c) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm.c:113
#4  0x000000000075872e in main (argc=3, argv=0x7fff94e749d8) at /tmp/buildd/php5-5.3.21/sapi/fpm/fpm/fpm_main.c:1838


Regards,
Jean Baptiste Favre
 [2013-02-18 00:35 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Open". Thank you.
 [2013-02-25 16:45 UTC] webmaster at jbfavre dot org
Hello,
Little feedback to this issue.
After 2 weeks running PHP-FPM using events.mechanism=poll, I did not experienced any segfault.

Seems that Rasmus was right.

Regards,
Jean Baptiste Favre
 [2013-05-06 08:53 UTC] ck at claudiokuenzler dot com
I experience a similar problem on a SmartOS smartmachine (like Solaris zone) where the master process seems to crash. Difference is that events.mechanism is set to /dev/poll, not to poll.

With activated debug log, the logs are full with such entries:
-------------
[06-May-2013 09:28:15.974674] WARNING: pid 7144, fpm_event_devpoll_wait(), line 166: /dev/poll: ioctl() returns 22
[06-May-2013 09:28:15.974699] WARNING: pid 7144, fpm_event_devpoll_wait(), line 166: /dev/poll: ioctl() returns 22
[06-May-2013 09:28:15.974726] WARNING: pid 7144, fpm_event_devpoll_wait(), line 166: /dev/poll: ioctl() returns 22
[06-May-2013 09:28:15.974747] WARNING: pid 7144, fpm_event_devpoll_wait(), line 166: /dev/poll: ioctl() returns 22

Eventually, php-fpm crashes.

Dtrace shows the following stack info:
-------------
 12   6130                     getpid:entry
              libc.so.1`getpid+0xa
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

 12   6430              clock_gettime:entry
              libc.so.1`__clock_gettime+0x28
              php-fpm`fpm_clock_get+0x24
              php-fpm`fpm_event_loop+0x1b7
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

12   6130                     getpid:entry
              libc.so.1`getpid+0xa
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

 12   6430              clock_gettime:entry
              libc.so.1`__clock_gettime+0x28
              php-fpm`fpm_clock_get+0x24
              php-fpm`fpm_event_loop+0xa6
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

 12   6186                      ioctl:entry
              libc.so.1`ioctl+0xa
              php-fpm`fpm_event_loop+0x16f
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

 12   6116                      gtime:entry
              libc.so.1`__time+0xa
              libc.so.1`tzset+0x26
              libc.so.1`strftime+0x2d
              php-fpm`zlog_print_time+0x3f
              php-fpm`zlog_ex+0x348
              php-fpm`fpm_event_devpoll_wait+0x106
              php-fpm`fpm_event_loop+0x16f
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

 12   6100                      write:entry
              libc.so.1`__write+0xa
              php-fpm`zlog_ex+0x189
              php-fpm`fpm_event_devpoll_wait+0x106
              php-fpm`fpm_event_loop+0x16f
              php-fpm`fpm_run+0x67
              php-fpm`main+0x581
              php-fpm`_start+0x6c

Not sure, if this the same source of bug, but the relevant function's name is the same (just replace epoll through devpoll): fpm_event_devpoll_wait.
 [2019-01-21 08:09 UTC] mnikulin at plesk dot com
The issue is likely similar to
Bug #77114 php-fpm master segfaults in fpm_event_epoll_wait/fpm_event_fire
that is present even in 7.3.1

For example, reproduced on php-7.1.26 CentOS-7

(gdb) bt
#0  0x0000000000000002 in ?? ()
#1  0x00000000006c9396 in fpm_event_epoll_wait (queue=<optimized out>, timeout=<optimized out>) at /usr/src/debug/sapi/fpm/fpm/events/epoll.c:147
#2  0x00000000006bf221 in fpm_event_loop (err=err@entry=0) at /usr/src/debug/sapi/fpm/fpm/fpm_events.c:410
#3  0x00000000006b97a7 in fpm_run (max_requests=max_requests@entry=0x7ffdde6142d8) at /usr/src/debug/sapi/fpm/fpm/fpm.c:114
#4  0x00000000004301ce in main (argc=2, argv=0x7ffdde614618) at /usr/src/debug/sapi/fpm/fpm/fpm_main.c:1893
(gdb) frame 1
#1  0x00000000006c9396 in fpm_event_epoll_wait (queue=<optimized out>, timeout=<optimized out>) at /usr/src/debug/sapi/fpm/fpm/events/epoll.c:147
147	/usr/src/debug/sapi/fpm/fpm/events/epoll.c: No such file or directory.
(gdb) set $i = 0
(gdb) while ($i < ret)
 >print $i
 >print epollfds[$i]
 >print *(struct fpm_event_s *)epollfds[$i].data.ptr
 >set $i = $i + 1
 >end
$1 = 0
$2 = {events = 1, data = {ptr = 0xa4aec0 <signal_fd_event.13524>, fd = 10792640, u32 = 10792640, u64 = 10792640}}
$3 = {fd = 7, timeout = {tv_sec = 0, tv_usec = 0}, frequency = {tv_sec = 0, tv_usec = 0}, callback = 0x6be700 <fpm_got_signal>, arg = 0x0, flags = 2, index = 7, which = 2}
$4 = 1
$5 = {events = 16, data = {ptr = 0xdf9a58, fd = 14654040, u32 = 14654040, u64 = 14654040}}
$6 = {fd = 0, timeout = {tv_sec = 0, tv_usec = 7104752}, frequency = {tv_sec = 14653904, tv_usec = 304942678018}, callback = 0x2, arg = 0x3c00000000, flags = 71, index = 0, which = 0}
$7 = 2
$8 = {events = 16, data = {ptr = 0xdf9aa0, fd = 14654112, u32 = 14654112, u64 = 14654112}}
$9 = {fd = 0, timeout = {tv_sec = 0, tv_usec = 75879187218432}, frequency = {tv_sec = 3, tv_usec = 0}, callback = 0x61, arg = 0x7f672a8e27b8 <main_arena+88>, flags = 713959352, index = 32615, which = 2}
$10 = 3
$11 = {events = 16, data = {ptr = 0xdf9118, fd = 14651672, u32 = 14651672, u64 = 14651672}}
$12 = {fd = 62, timeout = {tv_sec = 0, tv_usec = 0}, frequency = {tv_sec = 0, tv_usec = 0}, callback = 0x6c68f0 <fpm_stdio_child_said>, arg = 0xdf90f0, flags = 2, index = -1, which = 2}
$13 = 4
$14 = {events = 16, data = {ptr = 0xdf9160, fd = 14651744, u32 = 14651744, u64 = 14651744}}
$15 = {fd = 71, timeout = {tv_sec = 0, tv_usec = 0}, frequency = {tv_sec = 0, tv_usec = 0}, callback = 0x6c68f0 <fpm_stdio_child_said>, arg = 0xdf90f0, flags = 2, index = -1, which = 2}
(gdb) p i
$16 = <optimized out>
(gdb) info locals
ret = 5
i = <optimized out>
__func__ = "fpm_event_epoll_wait"

Notice signal_fd_event/fpm_got_signal that could bury some children, so files for fpm_stdio_child_said() are likely closed (index = -1) and the memory previously reserved for events has been freed and overwritten (epollfds[1].data, epollfds[2].data).
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Tue Mar 19 08:01:29 2024 UTC