Reproduce Code:

<?
// test-fcgi.php - a sample FCGI client

define('FCGI_VERSION_1', 1);
define('FCGI_BEGIN_REQUEST', 1);
define('FCGI_ABORT_REQUEST', 2);
define('FCGI_END_REQUEST', 3);
define('FCGI_PARAMS', 4);
define('FCGI_STDIN', 5);
define('FCGI_STDOUT', 6);
define('FCGI_STDERR', 7);
define('FCGI_DATA', 8);
define('FCGI_GET_VALUES', 9);
define('FCGI_GET_VALUES_RESULT', 10);

define('FCGI_RESPONDER', 1);
define('FCGI_KEEP_CONN', 1);

function FCGI_Packet($type, $content)
{
  $len=strlen($content);
  $packet=chr(FCGI_VERSION_1).chr($type).chr(0).chr(1).chr((int)($len/256)).chr($len%256).chr(0).chr(0).$content;
  return($packet);
}

function FCGI_NVPair($name, $value)
{
  $nlen = strlen($name);
  $vlen = strlen($value);

  if ($nlen < 128)
    $nvpair = chr($nlen);
  else
    $nvpair = chr(($nlen >> 24) | 0x80) . chr(($nlen >> 16) & 0xFF) . chr(($nlen >> 8) & 0xFF) . chr($nlen & 0xFF);

  if ($vlen < 128)
    $nvpair .= chr($vlen);
  else
    $nvpair .= chr(($vlen >> 24) | 0x80) . chr(($vlen >> 16) & 0xFF) . chr(($vlen >> 8) & 0xFF) . chr($vlen & 0xFF);
  return $nvpair . $name . $value;
}

function FCGI_Decode($data)
{
  if( strlen($data) < 8 )
    die("Packet too small " . strlen($data) . "\n");
  $length = (ord($data{4}) << 8)+ord($data{5});
  $packet = Array( 'version' => ord($data{0}),
                   'type'    => ord($data{1}),
                   'length'  => $length,
                   'content' => substr($data, 8, $length) );

  return $packet;

}

function FCGI_Connect($host, $port) {

  // Connect to FastCGI server
  $socket = fsockopen($host, $port, $errno, $errstr, 5);
  if( !$socket )
    die("Failed to connect to $host:$port\n");
  return $socket;
}
function FCGI_Test($socket)
{
  // Begin session
  $packet = '';
  $packet .= FCGI_Packet(FCGI_BEGIN_REQUEST, chr(0).chr(FCGI_RESPONDER).chr(FCGI_KEEP_CONN).chr(0).chr(0).chr(0).chr(0).chr(0) );

  // Build params

  $params = '';
  $params .= FCGI_NVPair('GATEWAY_INTERFACE', 'FastCGI/1.0');
  $params .= FCGI_NVPair('REQUEST_METHOD', 'GET');
  $params .= FCGI_NVPair('SCRIPT_FILENAME', '/tmp/test.php');

  $packet .= FCGI_Packet(FCGI_PARAMS, $params);
  $packet .= FCGI_Packet(FCGI_PARAMS, null);
  $packet .= FCGI_Packet(FCGI_STDIN, null);

  fwrite($socket, $packet);
}

function FCGI_Response($socket)
{
  // Read answers from fastcgi server
  while(true)
  {
    if(feof($socket))
      die("Socket closed\n");
    $packet = fread($socket, 8);
    if( $packet === false )
      die("Read failed\n");
    $header = FCGI_Decode($packet);
    //print_r($header);
    $len=$header['length']%8;
    $padlen=($len?(8-$len):0);
    $packet .= fread($socket, $header['length']+$padlen);
    $response = FCGI_Decode($packet);
    if( $response['type'] == FCGI_END_REQUEST )
      break;
    else
      print "[{$response['type']}] [{$response['content']}]\n";
  }
}

$socket1 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket1);
FCGI_Response($socket1);
$socket2 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket2);
FCGI_Response($socket2);
?>


<?
// /tmp/test.php - a sample cgi
echo "Hello World\n";
?>

Then start php-cgi in single process mode. 

php-cgi -b 1234

Run test-fcgi.php. The second request will never return. If you only open up 1 socket, and run multiple requests it works fine.

In your example you use persistent FastCGI connections (FCGI_KEEP_CONN). It means web server connects to PHP and sends SEVERAL requests using the SAME socket then it can close connection. You can correct your example in the following way to use persistent conection:

$socket1 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket1);
FCGI_Response($socket1);
FCGI_Test($socket1);
FCGI_Response($socket1);

or you may not to use persistent connection and then you must close connection 

$socket1 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket1);
FCGI_Response($socket1);
fclose($socket1);
$socket2 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket2);
FCGI_Response($socket2);
fclose($socket2);

In case of non-persistent connection usgage of shutdown() right after sending request is much better then close() after reading response.

I know if you open up 1 socket to one child, it works:

> If you only open up 1 socket, and run multiple requests > it works fine.

That's not the bug. The bug is PHP doesn't handle persistent connections (FCGI_KEEP_CONN), when the number of persistent connections exceedes the number of php children. The fcgi spec states that if the application doesn't have enough resoures to complete the request (e.g database handles, or in the case of PHP enough children), that it should return that it's overloaded. PHP does not do this; it simply accepts/ignores them. What PHP does is rely on the connection queueing, which doesn't solve the KEEP_CONN problem. Constantly opening up connections is inefficient.



Regards,

Erik Osterman

> it simply accepts/ignores them

PHP process DOESN'T accept() new connections if it already has persistent connection opened. Note that php/fastcgi is one-process-one-connection server that doesn't implement multiplexion (like apache 1.3). 

PHP doesn't try to manage persistent connection itself, however FastCGI module may do it (especially in multithreaded environment).

> PHP process DOESN'T accept() new connections 
> if it already has persistent connection opened. 
> Note that php/fastcgi is one-process-one-connection
> server that doesn't implement multiplexion

If this were actually the case, I'd be satisfied. That would mean the FCGI clients would get "connection refused" when there are no more sockets/children available. But what actually happens is that the connection is established, meaning accept() does get called.

To test this, modify the example like this

// Open up the first connection
$socket1 = FCGI_Connect('localhost', 1234);
// Send a request with FCGI_KEEP_CONN
FCGI_Test($socket1);
// Open up the second connection (should be refused)
$socket2 = FCGI_Connect('localhost', 1234);

printf("socket1:%d socket2:%d\n", feof($socket1), feof($socket2));


Expected output:
socket1:0 socket2:1

Actual output:
socket1:0 socket2:0

In otherwords, both connections are established => accept() was called.


Am I making sense?

Regards,

Erik Osterman

No you are :)
But you made a great job writting test script, and I think this discussion may lead us to some good result.

The fact that connect() returns file descriptor doesn't mean that accept() was really called. See the folllowing script. You can even write to socket befor it is really accepted.

<?php
	$server = stream_socket_server('tcp://127.0.0.1:1234');
	if (!$server) {
		die('Unable to create AF_INET socket [server]');
	}


	$socket1 = fsockopen('127.0.0.1', 1234, $errno, $errstr, 5);
	if (!$socket1)
		die("Failed to connect to 127.0.0.1:1234\n");
	fwrite($socket1, "Hello\n");


	$socket2 = fsockopen('127.0.0.1', 1234, $errno, $errstr, 5);
	if (!$socket2)
		die("Failed to connect to 127.0.0.1:1234\n");
	fwrite($socket2, "World\n");

	$socket = stream_socket_accept($server);
	$data = fgets($socket, 1024);
	echo($data);
	fclose($socket);

	$socket = stream_socket_accept($server);
	$data = fgets($socket, 1024);
	echo($data);
	fclose($socket);

	fclose($socket2);
	fclose($socket1);
	fclose($server);
?>

Dimitry,

You're example shocked me. I tried something similar making a simple stream_socket_server, but didn't make the client in PHP.

<?
   $server = stream_socket_server('tcp://127.0.0.1:1234');
    if (!$server) {
        die('Unable to create AF_INET socket');
    }
    sleep(10);
?>

# telnet localhost 1234
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

I had expected the same behavior in PHP's fsockopen, but indeed, as you said fsocketopen returns a valid resource even though the connection has not yet been accept()'d by the server. More over, feof returns false and fwrite to the socket returns the correct number of bytes "written". stream_get_metadata returns nothing interesting either. 

How is a PHP fsockopen/stream_socket_client client to know when a connection truely has been accept()'d?

The reason why this is all so important to us, is that we have a cluster of frontend servers that use a pool of FCGI backend servers. We need the busier frontend servers to dynamically open up more persistent connections (FCGI_KEEP_CONN) to FCGI servers (and release them as demand subsides). When a particular FCGI server is at capacity (PHP_FCGI_CHILDREN), we need the client to try (in a round robin fashion) another FCGI server. The current problem, as you understand by now, is that the client get's stuck when connecting to an FCGI server at capacity and timeout. 

As you've now suggested, it's apparently b/c they getting stuck waiting for the accept(), but never get it.

What recourse does the client have?


Regards,

Erik Osterman

For what it's worth, this shows some conflicting data. Perhaps I'm just interpretting it wrong.

Start theh FCGI server in single process mode
# php-cgi -b 1234

Modify the test script as follows:
<?
$socket1 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket1);
FCGI_Response($socket1);
sleep(30);
fclose($socket1);
?>

Now, the single process php-cgi server should not be accept()ing any more connections for 30 seconds.

# telnet localhost 1234
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.

Per my previous post, when a stream_socket_server server does not call accept(), the 'telnet' session gets "connection refused". In this example, it's not getting refused by the php-cgi server, which leads me to believe that php-cgi might actually be calling accept. This could be a implementation difference.. but I don't know.

Regards,

Erik Osterman

> <?
> $socket1 = FCGI_Connect('localhost', 1234);
> FCGI_Test($socket1);
> FCGI_Response($socket1);
> sleep(30);
> fclose($socket1);
> ?>

this code is not perferct, and this is the reason of bad behavior. You should call shutdown() "man 2 shutdown" after writing request into socket, but PHP doesn't implement such function.

<?
$socket1 = FCGI_Connect('localhost', 1234);
FCGI_Test($socket1);
shutdown($socket1, SHUT_WR); //we don't have this function
FCGI_Response($socket1);
sleep(30);
fclose($socket1);
?>

I cannot repeat your behavior with telnet. I always get "Connection closed" after a timeout.

BTW you can insert debug output into fastcgi.c right after accept() and try it.

By changing the example code to use PHP's sockets (e.g. socket_create/connect/read/write) allows one to use the socket_shutdown as you suggested. But calling shutdown after the request defeats the purpose of persistent connections, since that just negates the effect of passing FCGI_KEEP_CONN. Connections cannot be reused to send requests once they're shutdown after a request. 

So, it sounds like PHP's design decision is to support PHP_KEEP_CONN, but when PHP_FCGI_CHILDREN is reached, all new connections are queued/blocked. When a client releases its connection to a php-cgi child process, then the next connection in the queue is handed over to the next available child. While this makes a lot of sense when you're dealing with only non-persistent connections or only a single php-cgi server, it's not desireable when you have lots of persistent connections or lots of fcgi servers. For example, in HTTP/1.1, servers respond "503 Server too busy". If the client were notified in some way, either by way of a closed connection, refused connection, or FCGI_OVERLOADED packet, then the client could try another FCGI server. But since that doesn't happen, the client must just sit and wait. Perhaps, indefinitely or just timeout =( 

What is your take on this?


Best Regards,

Erik Osterman

Agree. shutdown() should not be used together with FCGI_KEEP_CONN. 

PHP cannot return FCGI_OVERLOADED, because all PHP processes are busy and nobody accepts new connection. The only way to detect this situation - use connection timeout.

Thanks for your constructive input. I guess we'll have to rely on creatively implemented timeouts w/in our application, as simply doing connection timeouts won't work (php returns connected immediately, even though the socket is not accepted).