php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #80739 PHP-FPM status page shows listen queue 0
Submitted: 2021-02-12 15:21 UTC Modified: 2022-11-13 15:42 UTC
Votes:3
Avg. Score:3.7 ± 0.9
Reproduced:3 of 3 (100.0%)
Same Version:2 (66.7%)
Same OS:2 (66.7%)
From: micha at dietpi dot com Assigned: bukka (profile)
Status: Assigned Package: FPM related
PHP Version: 8.0.10 OS: Debian Bullseye
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: micha at dietpi dot com
New email:
PHP Version: OS:

 

 [2021-02-12 15:21 UTC] micha at dietpi dot com
Description:
------------
I enabled the PHP-FPM status page and configured a handler in Apache2:

```
<Location /status>
        SetHandler "proxy:unix:/run/php/php8.0-fpm.sock|fcgi://localhost/status"
</Location>
```

This is the result:

```
pool:                 www
process manager:      static
start time:           28/Jan/2021:15:53:44 +0100
start since:          1147547
accepted conn:        1054191
listen queue:         0
max listen queue:     0
listen queue len:     0
idle processes:       11
active processes:     1
total processes:      12
max active processes: 12
max children reached: 0
slow requests:        0
```

As can be seen, while all running processes have been used concurrently (this is the case after a few hours already, so not a very rare event), the listen queue stays at zero. Especially the `listen queue len` stays at zero, even that `listen.backlog = 1024` is applied in the pool configuration and as well system-wide a sufficient backlog is permitted:

```
net.core.somaxconn = 2048
net.ipv4.tcp_max_syn_backlog = 1024
```

This is on PHP8.0.2, but it was the same before on PHP8.0.1 and PHP7.4.

Steps to reproduce the behavior:

1. Install Debian Bullseye
2. Install Apache2 and PHP-FPM
3. Enable the PHP-FPM status page and the related handler in Apache2
4. Cause more concurrent requests than `pm.max_children`
5. Watch PHP-FPM status page to show `listen queue len` and `max listen queue` being zero.

Probably related bug report: https://bugs.php.net/bug.php?id=76323

Expected result:
----------------
I would expect that `listen queue len` matches `listen.backlog` and that I do see a non-zero value at `max listen queue`, when the process limit is hit regularly. But probably in this setup it is handled differently? At least connections are not dropped from what I can say, no related error messages appear in either PHP or Apache2 logs, so in fact requests are queried somewhere.

Actual result:
--------------
listen queue:         0
max listen queue:     0
listen queue len:     0

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2021-05-28 02:02 UTC] yuta at adachi dot life
I reproduced this as well. Checking the implementation, we see that "getsockopt" is used for getting the queue length. We can use the "SO_LISTENINCQLEN" option in FreeBSD to get the status of the UNIX domain socket, but I think there is no option in Linux to get it.
https://github.com/php/php-src/blob/php-8.0.6/sapi/fpm/fpm/fpm_sockets.c#L487-L563


The original implementation did not support UNIX domain sockets, so I suppose this is more of an undocumented specification than a but.
http://svn.php.net/viewvc/php/php-src/branches/PHP_5_4/sapi/fpm/fpm/fpm_sockets.c?sortby=date&r1=305266&r2=305267&pathrev=312922&

We can use "netlink" to get the status of all sockets, but I think there are some privilege and performance concerns. Any ideas?
 [2021-09-16 12:27 UTC] micha at dietpi dot com
-PHP Version: 8.0.2 +PHP Version: 8.0.10
 [2021-09-16 12:27 UTC] micha at dietpi dot com
Indeed SO_LISTENINCQLEN seems to not exist on Linux: https://manpages.debian.org/bullseye/manpages/socket.7.en.html#Socket_options

I tried to find out other ways to get current and max queue length on Linux: https://serverfault.com/a/930677/577419

```
# ss -lx | sed -n '1p;/php/p'
Netid State  Recv-Q Send-Q                              Local Address:Port      Peer Address:Port     Process
u_str LISTEN 0      1024                     /run/php/php8.0-fpm.sock 157408202            * 0
```

If I understood it right, this means current queue length is zero and max is 1024? The latter is true, changing "listen.backlog" to 2048 makes "Send-Q" change accordingly. I'll now monitor whether "Recv-Q" is ever different from zero. Checking the internals of "ss" may then provide the options for the FPM as well, I hope.

Best regards,

Micha
 [2021-11-08 13:45 UTC] phpbugs at icounsellor dot co dot uk
Further to Micha's comment:

>> I'll now monitor whether "Recv-Q" is ever different from zero. 

This weekend I wrote a quick-and-dirty set of scripts to see if Micha's plan would show positive results (details below). 

RESULTS:
Day: https://icounsellor.co.uk/phpbugs/phpfpm_queue-day.png
Week: https://icounsellor.co.uk/phpbugs/phpfpm_queue-week.png
Month: https://icounsellor.co.uk/phpbugs/phpfpm_queue-month.png
(These URLs are soft-linked to my live Munin data)

CONCLUSION: Although my PHP-FPM site is of low activity, we do have positive results here showing we can get the value of Receive-Q on Linux

RECOMMENDATION: As Micha suggests earlier, the PHP Development Team will be able to gain knowledge from Linux source code (reference: https://serverfault.com/a/930677/577419 - internals of "ss") to attain the "how to" with respect to gaining Receive-Q data for PHP-FPM.


--------------------------------
FYI - SCRIPTS.

SCRIPT #1: Quick & Dirty Bash Script to collate maximum Receive-Q value for php-fpm7.4.

Purpose: Munin runs only once every 5 minutes so asking it to call ss to show receive-q values for php-fpm once every 5 minutes isn't very detailed. Hence, this script runs (... repeatedly via cron) collating, once a second, the maximum value of phpfpm-7.4 Receive-Q via ss (in periods of 5 minutes).

Source:

#!/bin/bash

FileRoot="php7.4-fpm";
Socket=$FileRoot".sock";
LoopDelay=1;

SharedFile="/dev/shm/"$FileRoot".count";

# Run for 5 minutes... then quit
let RunLoop=300/$LoopDelay;

COUNTER=0
Max=0;

if [ ! -f $SharedFile ]; then
        # https://unix.stackexchange.com/questions/107038/obtain-exclusive-read-write-lock-on-a-file-for-atomic-updates
        flock -x -w 5 $SharedFile echo 0 > $SharedFile;
fi

while [  $COUNTER -lt $RunLoop ]; do

        SendQCount=$(ss -ln | grep $Socket | awk '{print $3}');

        if (( SendQCount > Max )); then
                Max=$SendQCount;
        fi

        sleep $LoopDelay;
        let COUNTER=COUNTER+1;
done
# Write the latest results...
flock -x -w 5 $SharedFile echo $Max > $SharedFile;

-----
SCRIPT #2: Quick and Dirty Munin plugin

Purpose: Munin-node calls this script, reads the data left by the above script (once every 5 minutes), and graphs the information

Source:

#!/bin/bash 
if [ "$1" = "config" ]; then
        MaxQueue=$(ss -ln | grep php7.4-fpm.sock | awk '{print $4}');
        echo 'graph_title PHP-FPM Socket Queue Statistics'
        echo 'graph_vlabel Count'
        echo 'graph_category php'
        echo "graph_info This graph counts the number connections queued. Max queue configured to be: "$MaxQueue
        echo "Current.label Currently Queued Requests";

        exit 0
fi

        CurrentQueue=$(cat /dev/shm/php7.4-fpm.count);
        echo Current.value $CurrentQueue;

exit 0;
 [2021-11-08 15:59 UTC] micha at dietpi dot com
Many thanks for providing the scripts. Indeed I can also verify with PHP8.0 that this method works fine and "Recv-Q" indeed shows the current query length.

I did monitoring with this one-liner from shell:
```
while sleep 1; do value=$(ss -lxn | mawk '/php/{print $3}'); (( $value )) && echo "$value"; done
```

However, this is of course everything else than precise as it only takes a snapshot of the current query length at defined interval, which, considering the speed in which PHP requests are handled, will unlikely ever show the real maximum reached.

For the current query length ("listen queue:" on status page) this however works fine. Of course PHP-FPM shouldn't execute any external command but try to obtain the information with the methods "ss" uses ;).

I wonder whether /proc/<PID>/net/netstat contains what we are looking for. But so far I couldn't identify any of the long value list a maximum reached query length. Probably there are other stats provided by the kernel in /sys or /proc somewhere?

Best regards,

Micha
 [2022-11-13 15:42 UTC] bukka@php.net
-Assigned To: +Assigned To: bukka
 [2022-11-13 15:42 UTC] bukka@php.net
I had a closer look on this. The ss (iproute2) uses netlink to get this info so we could use the same. I created an issue with more info: https://github.com/php/php-src/issues/9943
 
PHP Copyright © 2001-2022 The PHP Group
All rights reserved.
Last updated: Fri Dec 09 23:05:55 2022 UTC