php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #69637 stream_select() warns: recompile php with FD_SETSIZE>1024 but it already is
Submitted: 2015-05-15 02:08 UTC Modified: 2016-06-01 14:03 UTC
Votes:13
Avg. Score:4.5 ± 0.6
Reproduced:12 of 13 (92.3%)
Same Version:4 (33.3%)
Same OS:3 (25.0%)
From: bousset1 at gmail dot com Assigned:
Status: Verified Package: Sockets related
PHP Version: 5.6.8 OS: linux (tested on ubuntu 14.04)
Private report: No CVE-ID: None
Have you experienced this issue?
Rate the importance of this bug to you:

 [2015-05-15 02:08 UTC] bousset1 at gmail dot com
Description:
------------
With more than 1024 sockets it produces a warning using stream_select(). It asks to recompile php with --enable-fd-setsize higher than the current value (which says is 1024). However, even after recompiling php with a higher value (2048, 4096, 65536, 9999999, etc) it gives the same error.

My system (ubuntu 14.04) is tuned to allow a high number of opened files (I globally followed instructions described here https://rtcamp.com/tutorials/linux/increase-open-files-limit/ and here http://kb.odin.com/en/967).
Then it seems not to be a problem from the system limits.

Here attached a cli script that you can use to reproduce this problem (see the actual result section for details).

It seems that I'm not the only one to encounter this problem : http://www.programering.com/a/MTO4cTMwATA.html

If it can helps, here is my current configure line :
./configure '--enable-exif' '--enable-ftp' '--enable-sockets' '--enable-shmop' '--enable-mbstring' '--enable-mbregex' '--enable-bcmath' '--enable-calendar' '--with-gd' '--enable-gd-native-ttf' '--with-gettext' '--with-curl' '--disable-debug' '--enable-fpm' '--with-fpm-user=www-data' '--with-fpm-group=www-data' '--with-mysql' '--with-pdo-mysql' '--disable-phpdbg' '--enable-opcache' '--with-zlib' '--with-bz2' '--with-iconv' '--with-mcrypt' '--enable-zip' '--with-tidy' '--with-mysqli' '--with-openssl' '--with-jpeg-dir=/usr/lib/x86_64-linux-gnu' '--enable-fd-setsize=65536'

This issue (not a php bug) is very close to mine : https://bugs.php.net/bug.php?id=37025 however I changed the fd_setsize value in typesizes.h already.

Test script:
---------------
<?php
$pid = getmypid();
$sockets = array();
$loopNbr = 1025; // This normally could go up to the compile --enable-fd-setsize value, but the limit will be anyway 1024, like if it has been hardcoded somewhere
$sysLimit = min(posix_getrlimit()['soft openfiles'], posix_getrlimit()['hard openfiles']);
$nbrInitialOpenedFiles = (int)exec("ls /proc/$pid/fd | wc -l");
echo "There is initially $nbrInitialOpenedFiles opened files".PHP_EOL."The sysLimit is $sysLimit".PHP_EOL."In some few seconds, we will try to open $loopNbr sockets".PHP_EOL;
sleep(5);
for ($i = 0; $i < $loopNbr; $i++) {
    echo $i > ($sysLimit - $nbrInitialOpenedFiles) ? 'Normal crash : openfiles system limitation. Edit /etc/security/limits.conf, increasing soft & hard values' . PHP_EOL : null;
    $s = stream_socket_client("127.0.0.1:80", $errno, $errstr, 30, STREAM_CLIENT_ASYNC_CONNECT | STREAM_CLIENT_CONNECT);
    if ($s) {
        $sockets[$i] = $s;
        echo 'Stream n°' . $i . ' opened' . PHP_EOL;
        echo(exec("lsof -i -a -p $pid") . PHP_EOL);
    } else {exit;}
}
echo ($loopNbr + $nbrInitialOpenedFiles) > 1024 ? 'Weird crash : ($loopNbr + $nbrInitialOpenedFiles) is > 1024' . PHP_EOL : 'Nbr of files descriptor <= 1024 <= sysLimit, then all should be ok' . PHP_EOL;
$read = $sockets;
stream_select($read, $w = null, $e = null, null, null);

Expected result:
----------------
As long as the number of files descriptor is lower than --enable-fd-setsize and lower than the system limits, no error should occurs.

Actual result:
--------------
1/ By default $loopNbr is set to 1025, that is > 1024. With this value, it produces the following error :
"PHP Warning:  stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE.
It is set to 1024, but you have descriptors numbered at least as high as 1027.
 --enable-fd-setsize=2048 is recommended, but you may want to set it
to equal the maximum number of open files supported by your system,
in order to avoid seeing this error again at a later date."

2/ By changing $loopNbr to a value < 1024, such 1000, everything is ok.

3/ By changing your system limits (edit /etc/security/limits.conf then logout / login to apply), for exemple with a value of 512 it produces another error, which is a normal error.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2015-08-27 23:57 UTC] cmb@php.net
After changing the definition of __FD_SETSIZE in bits/typesizes.h
the script runs fine for me (Debian Jessie) without warning,
except for the spurious "weird crash" error message. Did you
recompile *everything* after changing __FD_SETSIZE? And of course,
check whether FD_SETSIZE really changed.

Anyhow, I consider it a bug that --enable-fd-setsize might not
have any effect, without producing at least a meaningful warning
message. A respective check in configure.in would be appropriate.
 [2016-05-31 21:05 UTC] deriamis at gmail dot com
The reason this appears to be happening is that sys/select.h on Linux is overriding FD_SETSIZE by setting it to the value of __FD_SETSIZE, which is set statically to 1024, without checking that it's already defined. The --enable-fd-setsize is not cross-platform and, people who need this either have to patch PHP source in several places (bad) or system include files (worse). I think a good way to do this is to set a dummy value and then in source set both FD_SETSIZE and __FD_SETSIZE to the dummy value immediately before the sys/select.h include.
 [2016-06-01 14:03 UTC] cmb@php.net
-Status: Open +Status: Verified
 [2016-11-08 10:45 UTC] kexianbin at diyism dot com
1.vi /usr/include/x86_64-linux-gnu/bits/typesizes.h:
#define __FD_SETSIZE  65536

2.compile:
./configure --with-curl --enable-fd-setsize=65536 CFLAGS='-D_FORTIFY_SOURCE=0'
make && sudo make install

3.vi /etc/security/limits.conf:
* soft nofile 65535
* hard nofile 65535
 [2017-04-24 19:05 UTC] erw at macrovita dot com dot br
If PHP process has more open file descriptors than FD_SETSIZE, it seems stream_select() will fail even when trying to monitor just a couple file descriptors.

I get the error even when trying to stream_select() only 2 file descriptors, apparently because the apache process has thousands of file descriptors open.

Environment: apache-2.4.23 event mpm, php-7.0.11, Gentoo Linux

// BTM11121, BTM10159
 [2019-01-30 11:34 UTC] stormdja at gmail dot com
@[2016-11-08 10:45 UTC] - directly modifiying system headers may not be a good idea.
Here's a simple patch (tested on php 7.2.14):
--- main/php.h    2019-01-08 09:59:17.000000000 +0000
+++ main/php.h    2019-01-30 09:03:18.698899000 +0000
@@ -515,6 +515,11 @@
 
 #endif
 
+#undef __FD_SETSIZE
+#define __FD_SETSIZE 32768
+#undef FD_SETSIZE
+#define FD_SETSIZE 32768
+
 /*
  * Local variables:
  * tab-width: 4

-----------
Result:
# php-7.2.14-009/bin/php -r "print_r(get_defined_constants(true));"|grep FD_SETSIZE
            [PHP_FD_SETSIZE] => 32768
 
PHP Copyright © 2001-2019 The PHP Group
All rights reserved.
Last updated: Tue Apr 23 01:01:26 2019 UTC