php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #77443 FPM hangs on reload due to children handling SIGTERM since PHP 7.1
Submitted: 2019-01-11 07:47 UTC Modified: 2020-01-12 16:41 UTC
Votes:1
Avg. Score:3.0 ± 0.0
Reproduced:0 of 0 (0.0%)
From: emayoral at arsys dot es Assigned: bukka (profile)
Status: Closed Package: FPM related
PHP Version: 7.2.14 OS: CentOS 6
Private report: No CVE-ID: None
 [2019-01-11 07:47 UTC] emayoral at arsys dot es
Description:
------------
I run a system with a relatively large number of pfp-fpm pools. Whenever a pool is added or removed a php-fpm reload is issued.

The behaviour of php-fpm on reload is to terminate children with SIGQUIT, wait process_control_timeout (10 seconds in my config), then issue SIGTERM to the remaining children.

Now, some of my pool children will not die on SIGQUIT when opcache is used, but that is a different bug. Point is until PHP 7.0, SIGTERM would finish the remaining children, the reload would complete in 10 seconds and everything would be fine.

But starting with PHP 7.1 the php processes also handle SIGTERM (I suspect it uses the same signal handler as for SIGQUIT, although I am not 100% sure of that).

Now the children which did not die on SIGQUIT do not die on SIGTERM either and the reload will not complete until those children are manually issued a SIGKILL.

I think php-fpm 7.1 and higher should do its first round of SIGQUIT to its children on reload, but after process_control_timeout, issue a SIGKILL instead of a SIGTERM, since SIGTERM is not an effective way of ending rogue children any more.


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2019-01-11 10:26 UTC] emayoral at arsys dot es
Digging further into the code, I think this case is covered in:

sapi/fpm/fpm/fpm_process_ctl.c

static void fpm_pctl_action_next() /* {{{ */
{
        int sig, timeout;

        if (!fpm_globals.running_children) {
                fpm_pctl_action_last();
        }

        if (fpm_signal_sent == 0) {
                if (fpm_state == FPM_PCTL_STATE_TERMINATING) {
                        sig = SIGTERM;
                } else {
                        sig = SIGQUIT;
                }
                timeout = fpm_global_config.process_control_timeout;
        } else {
                if (fpm_signal_sent == SIGQUIT) {
                        sig = SIGTERM;
                } else {
                        sig = SIGKILL;
                }
                timeout = 1;
        }

        fpm_pctl_kill_all(sig);
        fpm_signal_sent = sig;
        fpm_pctl_timeout_set(timeout);
}

(Send SIGQUIT, wait process_control_timeout , send SIGTERM, wait 1 second, send SIGKILL)

But my experience is that after the SIGQUIT and SIGTERM, the SIGKILL never happens (Which didn't matter until php 7.1 as SIGTERM would do the job). Not sure why, though.
 [2019-01-22 09:55 UTC] mnikulin at plesk dot com
Please, have a look at
Bug #76601 "Partially working php-fpm after incomplete reload"
that happens for both 5.6 and 7.x versions and at
Bug #76895 "PHP-FPM7.x can't finish reloading after encountering a fatal error"
that is specific to 7.x versions and easily reproducible

The following one-line patch fix SIGKILL so the problem becomes invisible,
however the proper way would be to avoid ignoring of SIGQUIT and SIGTERM
by child processes
https://bugs.php.net/patch-display.php?bug_id=76601&patch=php-76601_kill-not-rescheduled_7.2.11_2018-09-28.patch&revision=latest
 [2019-11-20 04:54 UTC] mnikulin at plesk dot com
Issue with signal blocked by opcache has been likely solved in PHP-7.2.22, see
https://github.com/php/php-src/pull/4461
https://github.com/php/php-src/commit/38f1288b6427fc9e2fa2b5ad9912745ded923ee7
and Bug #76895

SIGKILL should be fixed in 7.4
https://github.com/php/php-src/commit/0ed6c3714087b254c49185568df96a31df76b2ce
 [2020-01-12 16:41 UTC] bukka@php.net
-Status: Open +Status: Closed -Assigned To: +Assigned To: bukka
 [2020-01-12 16:41 UTC] bukka@php.net
This issue should be fixed in 7.4.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Nov 21 12:01:29 2024 UTC