php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #77023 PHP-FPM cannot shutdown processes
Submitted: 2018-10-16 11:30 UTC Modified: 2022-02-14 07:19 UTC
Votes:1
Avg. Score:4.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:0 (0.0%)
Same OS:1 (100.0%)
From: azurit at pobox dot sk Assigned: bukka (profile)
Status: Closed Package: FPM related
PHP Version: 7.2.11 OS: Debian 9
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If this is not your bug, you can add a comment by following this link.
If this is your bug, but you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: azurit at pobox dot sk
New email:
PHP Version: OS:

 

 [2018-10-16 11:30 UTC] azurit at pobox dot sk
Description:
------------
I'm not albe to reproduce this problem but it is sometimes happening to pool of one of our customers. Pool si set like this:

pm = dynamic
pm.max_children = 30
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1

I want from it to keep only one IDLE process at a time but FPM sometimes fails to kill IDLE processes (kill because of pm.max_spare_servers) which results into having 30 IDLE processes (or other number, depends on pm.max_children). When i start killing FPM processes by hand, FPM is starting them back BUT there are always few stucked FPM processes (probably the ones which were running for longest time) which, if are killed, FPM itself starts killing also other IDLE processes (like it was waiting for them to shutdown before start shutdowning other processes) and everything goes to normal. I was able to get a strace output from one such process:


socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 0
rt_sigreturn({mask=[]})                 = 43
accept(26, 0x7ffee4d3f860, [112])       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} ---
close(0)                                = 0
...




Look on this:


# ps aux | grep customer1 | grep fpm | wc -l
11

# ps aux | grep customer1 | grep fpm
customer1   2345  0.0  0.1 398280 20640 ?        S    12:34   0:00 php-fpm: pool customer1
customer1   3295  0.4  0.2 493180 47964 ?        S    12:34   0:12 php-fpm: pool customer1
customer1  10887  0.4  0.2 489452 43628 ?        S    12:35   0:12 php-fpm: pool customer1
customer1  12018  0.6  0.3 497272 50852 ?        S    12:35   0:18 php-fpm: pool customer1
customer1  12707  0.4  0.3 496552 51748 ?        S    12:35   0:14 php-fpm: pool customer1
customer1  13157  0.5  0.4 511312 65676 ?        S    12:35   0:15 php-fpm: pool customer1
customer1  15860  0.6  0.3 797176 53464 ?        S    12:36   0:17 php-fpm: pool customer1
customer1  15941  0.5  0.3 788804 50856 ?        S    12:36   0:16 php-fpm: pool customer1
customer1  23635  0.0  0.1 398280 20636 ?        S    13:24   0:00 php-fpm: pool customer1
customer1  23636  0.0  0.1 398280 20636 ?        S    13:24   0:00 php-fpm: pool customer1
customer1  23637  0.0  0.1 398280 20636 ?        S    13:24   0:00 php-fpm: pool customer1

// the one, which is stucked here, is PID 2345
// let's try to kill few processes except 2345

# skill -kill 10887 12018 13157 23635 23637

# ps aux | grep customer1 | grep fpm | wc -l
11

# ps aux | grep customer1 | grep fpm
customer1   2345  0.0  0.1 398280 20640 ?        S    12:34   0:00 php-fpm: pool customer1
customer1   3295  0.4  0.3 508332 63120 ?        S    12:34   0:14 php-fpm: pool customer1
customer1  12707  0.5  0.3 494504 49876 ?        S    12:35   0:15 php-fpm: pool customer1
customer1  15860  0.6  0.4 813560 68608 ?        S    12:36   0:19 php-fpm: pool customer1
customer1  15941  0.5  0.3 795164 56780 ?        R    12:36   0:17 php-fpm: pool customer1
customer1  23636  0.4  0.2 485492 37128 ?        R    13:24   0:00 php-fpm: pool customer1
customer1  26450  2.7  0.2 416048 43144 ?        S    13:26   0:00 php-fpm: pool customer1
customer1  26451  2.5  0.2 416048 43144 ?        S    13:26   0:00 php-fpm: pool customer1
customer1  26452  2.6  0.2 416048 43156 ?        S    13:26   0:00 php-fpm: pool customer1
customer1  26453  2.5  0.2 416048 43140 ?        S    13:26   0:00 php-fpm: pool customer1
customer1  26454  0.0  0.1 398660 26028 ?        S    13:26   0:00 php-fpm: pool customer1

// now let's kill 2345

# skill -kill 2345

# ps aux | grep customer1 | grep fpm | wc -l
1

# ps aux | grep customer1 | grep fpm
customer1  27650  0.1  0.1 398660 26760 ?        S    13:27   0:00 php-fpm: pool customer1



Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2022-02-13 20:52 UTC] bukka@php.net
Apology for very late replay.


I looked a bit to the code and it seems to me that this might happen if FPM is failing to kill the process using the SIGQUIT that is used for that purpose. In other cases this is actually handled by trying SIGKILL if SIGQUIT wasn't successful but not for the dynamic idle check which seems like an omission to me. Will try to put together a patch in the coming months and see if I can also find some way to recreate this.
 [2022-02-13 20:53 UTC] bukka@php.net
-Assigned To: +Assigned To: bukka
 [2022-02-14 07:19 UTC] azurit at pobox dot sk
Thanks, i'm really happy someone is going to look into this. Meanwhile, i bypassed the problem using this:

 - i'm periodically running a script which is looping through the all pools status pages, searching for processes which are idling more than idle timeout and killing them

 - patched FPM and disabled process respawn if they are killed (by the way, this would be a great official feature)
 [2022-04-22 22:08 UTC] git@php.net
Automatic comment on behalf of bukka
Revision: https://github.com/php/php-src/commit/d8612fb6b7496a4f17e8250037a00b26623c1c77
Log: Fix bug #77023: FPM cannot shutdown processes
 [2022-04-22 22:08 UTC] git@php.net
-Status: Assigned +Status: Closed
 
PHP Copyright © 2001-2022 The PHP Group
All rights reserved.
Last updated: Sat Jul 02 18:03:35 2022 UTC