php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #65584 recv() failed (104: Connection reset by peer) while reading response ...
Submitted: 2013-08-29 13:43 UTC Modified: 2015-08-02 04:22 UTC
Votes:19
Avg. Score:4.6 ± 0.7
Reproduced:17 of 18 (94.4%)
Same Version:5 (29.4%)
Same OS:4 (23.5%)
From: michael dot heuberger at binarykitchen dot com Assigned:
Status: No Feedback Package: Scripting Engine problem
PHP Version: 5.5.3 OS: Ubuntu 13.04
Private report: No CVE-ID: None
 [2013-08-29 13:43 UTC] michael dot heuberger at binarykitchen dot com
Description:
------------
Hi there

I recently apt-get upgraded my PHP disto on my server, adjusted, tested few lines of the current website but there is one last problem: PHP-FPM crashes always. With these log lines in NGINX:

2013/08/30 01:34:06 [error] 12282#0: *51 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 78.10.247.202, server: binarykitchen.com, request: "GET /user/register/action/why/language/de/ HTTP/1.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "www.binarykitchen.com", referrer: "http://www.binarykitchen.com/user/register/action/why/language/de/"

There are many of those. Sometimes the page loads, sometimes not.

I tried hard to adjust nginx and php-fpm but no success. I am using sockets.

Tell me, how can I see what is exactly resetting the connection? Why it is crashing? What commands should I try?

And here more details:

$ php -v
PHP 5.5.3-1+debphp.org~raring+2 (cli) (built: Aug 27 2013 09:16:12) 
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2013 Zend Technologies
    with Zend OPcache v7.0.3-dev, Copyright (c) 1999-2013, by Zend Technologies

Thanks for your help,
Michael


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2013-08-29 15:17 UTC] laruence@php.net
-Status: Open +Status: Feedback
 [2013-08-29 15:17 UTC] laruence@php.net
seems your php-fpm is crashing, you need try to get a backtrace.. 

http://bugs.php.net/bugs-generating-backtrace.php for *NIX and
http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32


and you could also try to test with the latest snapshot of php-5.5 branch: 
http://snaps.php.net/php5.4-latest.tar.gz
 [2013-08-29 23:06 UTC] michael dot heuberger at binarykitchen dot com
-Status: Feedback +Status: Open
 [2013-08-29 23:06 UTC] michael dot heuberger at binarykitchen dot com
Thank you but I am unable to create a backtrace on my server.

I have set /proc/sys/kernel/core_pattern with
echo "/var/cores/core-%e.%p" > /proc/sys/kernel/core_pattern 

Then I made /var/cores/ writeable for user www-data and chmodded it with 777. Then sudo service php5-fpm restart and reloaded the page. It crashed but no core was dumped. Why? ulimit is unlimited.

By the way, I am on nginx. Your instructions are for Apache.
 [2013-08-29 23:15 UTC] michael dot heuberger at binarykitchen dot com
Ah, I managed it. I have the core dumps now.

You forgot to mention that I have to adjust /etc/php5/fpm/pool.d/www.conf with a higher value than zero, i.E. rlimit_core = 10

I have 10 core dumps, all 119 MB large. How can I attach them here or hand them over to you?
 [2013-08-29 23:19 UTC] michael dot heuberger at binarykitchen dot com
PS: The gdb command does not exist on my machine.
 [2013-08-29 23:21 UTC] rasmus@php.net
-Status: Open +Status: Feedback
 [2013-08-29 23:21 UTC] rasmus@php.net
apt-get install gdb
 [2013-08-29 23:26 UTC] michael dot heuberger at binarykitchen dot com
-Status: Feedback +Status: Open
 [2013-08-29 23:26 UTC] michael dot heuberger at binarykitchen dot com
Thx!

I have the dump here. An clues?

$ sudo gdb /usr/sbin/php5-fpm /var/cores/core-php5-fpm.31743
GNU gdb (GDB) 7.5.91.20130417-cvs-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/php5-fpm...(no debugging symbols found)...done.
[New LWP 31743]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www                                                       '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000006f8459 in gc_remove_zval_from_buffer ()
(gdb) bt
#0  0x00000000006f8459 in gc_remove_zval_from_buffer ()
#1  0x0000000000763ff8 in ?? ()
#2  0x00000000007480e8 in execute_ex ()
#3  0x00000000006dad99 in zend_execute_scripts ()
#4  0x00000000006799ac in php_execute_script ()
#5  0x0000000000466844 in main ()
 [2013-08-30 00:27 UTC] michael dot heuberger at binarykitchen dot com
I also see a file in /var/crash/
_usr_sbin_php5-fpm.33.crash

It is full with binary data. I tried to run it with gdb but no success.
 [2013-08-30 00:31 UTC] michael dot heuberger at binarykitchen dot com
I tried to parse it with apport-retrace but it is not helpful:

$ apport-retrace -R -g /var/crash/_usr_sbin_php5-fpm.33.crash
GNU gdb (GDB) 7.5.91.20130417-cvs-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Reading symbols from /usr/sbin/php5-fpm...(no debugging symbols found)...done.
[New LWP 2530]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www                                                       '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000006f8459 in gc_remove_zval_from_buffer ()
(gdb)
 [2013-08-30 03:08 UTC] laruence@php.net
hmm, the backtrace is too generic, I can not get any useful info from there.

are you able to get a reproduce test script?
 [2013-08-30 03:28 UTC] laruence@php.net
-Status: Open +Status: Feedback
 [2013-08-30 03:28 UTC] laruence@php.net
Please try using this snapshot:

  http://snaps.php.net/php5.5-latest.tar.gz
 
For Windows:

  http://windows.php.net/snapshots/


we fixed some segfaults issues recently, maybe you could have a try..thanks
 [2013-08-30 04:31 UTC] michael dot heuberger at binarykitchen dot com
-Status: Feedback +Status: Open
 [2013-08-30 04:31 UTC] michael dot heuberger at binarykitchen dot com
Happy to give your snapshots a try but do not how to install these. I installed PHP-FPM via apt-get.
 [2013-08-30 05:12 UTC] michael dot heuberger at binarykitchen dot com
I managed to get more logging information.

In the log I see this fatal error:

[30-Aug-2013 17:05:50 Pacific/Auckland] PHP Fatal error:  Cannot redeclare class SGL_Request in /opt/bk/lib/SGL/Request.php on line 53

But in my spl_autoload_register function I already test with this to make sure classes won't be redeclared:

    public static function isLoaded($name)
    {
        return class_exists($name, false) || interface_exists($name, false);
    }

But when, why doesn't it work?
 [2013-08-30 05:30 UTC] michael dot heuberger at binarykitchen dot com
I also found out, that when I comment out logic around $_COOKIE (auto-login feature), then my site is back up working.

Does PHP 5.5.3 have issues with $_COOKIE?
 [2013-12-19 10:56 UTC] hornung dot frank at gmail dot com
OS: Debian GNU/Linux 7 \n \l
PHP Version: 5.5.7 (git tag)
Lua Version 5.2.3
Lua extension: 1.1.1-dev

Core File is not possible.

This Bug occurs with Lua::registerCallback(). It's not necessary to call the callback from Lua.

On CLI seems to be all fine.
 [2013-12-19 12:04 UTC] hornung dot frank at gmail dot com
This problem only occours with sockets. If i etablish the connection over a network port is all ok.
 [2014-03-30 22:19 UTC] vlad dot rusu at gmail dot com
I am experience the same problems on Ubuntu Server 12.04 / NGINX 1.4.7 & PHP 5.5.10 FPM (installed from ppa:ondrej/php5).

More specifically, I have two BIG problems, which made me revert to using PHP 5.4 (which works ok, btw):
1. A LOT of "recv() failed (104: Connection reset by peer)" in NGINX's error.log
This is when using 127.0.0.1:9000 to listen. If I use sockets, I receive also connect() failed erros.
2. No matter what options tried, PHP-FPM does not log errors. 
All that is logs is "NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful" lines.

This is in spite having error_log = /var/log/php-fpm/fpm.log, log_level = notice, and catch_workers_output = yes !!
(Opcache disabled/enabled... no difference).

I tried all the possible settings but to no avail.
Reverting to PHP 5.4.26 fixes this issues.
 [2014-04-07 12:42 UTC] craig_carnell at hotmail dot com
I've also started encountering this issue. I already have php installed so I downloaded the 5.5.3 source and re-compiled with --enable-debug.

Here is my debug output:

root@web:/tmp# gdb /usr/sbin/php5-fpm /tmp/core-php5-fpm.11309
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/php5-fpm...Reading symbols from /usr/lib/debug/usr/sbin/php5-fpm...done.
done.
[New LWP 11309]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Core was generated by `php-fpm: pool www                                                       '.
Program terminated with signal 11, Segmentation fault.
#0  _zend_mm_free_int (heap=0x178f300, p=0x3f0f1a8) at /build/buildd/php5-5.5.3+dfsg/Zend/zend_alloc.c:2104
2104	/build/buildd/php5-5.5.3+dfsg/Zend/zend_alloc.c: No such file or directory.
(gdb)
 [2014-04-07 12:46 UTC] craig_carnell at hotmail dot com
And again. The same code seems to throw an error randomly. It doesn't always happen. There is log anywhere apart from my nginx logs which say:

2014/04/07 12:28:37 [error] 16336#0: *390 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: **.co.uk, request: "GET /registration/form/success/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "www.**.co.uk", referrer: "http://www.**.co.uk/registration/form/"
2014/04/07 12:31:05 [error] 16336#0: *429 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: **.co.uk, request: "GET /registration/form/success/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "www.**.co.uk", referrer: "http://www.**.co.uk/registration/form/"

GDB output:

#0  _zend_mm_free_int (heap=0x178f300, p=0x3ecd330) at /build/buildd/php5-5.5.3+dfsg/Zend/zend_alloc.c:2104
2104	/build/buildd/php5-5.5.3+dfsg/Zend/zend_alloc.c: No such file or directory.
(gdb) quit
root@web:/tmp# gdb /usr/sbin/php5-fpm /tmp/core-php5-fpm.10563
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/php5-fpm...Reading symbols from /usr/lib/debug/usr/sbin/php5-fpm...done.
done.
[New LWP 10563]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Core was generated by `php-fpm: pool www                                                       '.
Program terminated with signal 11, Segmentation fault.
#0  _zval_ptr_dtor (zval_ptr=0x2f7374736f68762f) at /build/buildd/php5-5.5.3+dfsg/Zend/zend_execute_API.c:426
426	/build/buildd/php5-5.5.3+dfsg/Zend/zend_execute_API.c: No such file or directory.
(gdb)
 [2014-04-08 11:37 UTC] craig_carnell at hotmail dot com
I have found the specific line in Varien_Io_File. Sometimes it will crash, sometimes it wont on chdir. The value of $this->_iwd is a valid path - :

/var/www/vhosts/magento-capistrano-dev/releases/20140408102656

Code below:

public function cd($dir)
    {
        if( is_dir($dir) ) {
            @chdir($this->_iwd);
            $this->_cwd = realpath($dir);
            return true;
        } else {
            throw new Exception('Unable to list current working directory.');
            return false;
        }
    }
 [2014-04-09 14:33 UTC] craig_carnell at hotmail dot com
You can forget my previous entries.. this is wall down to opcache module being loaded twice by duplicate ini files.
 [2015-07-22 13:10 UTC] mike@php.net
-Status: Open +Status: Feedback -Package: FPM related +Package: Scripting Engine problem
 [2015-07-22 13:10 UTC] mike@php.net
Please try a current version.
 [2015-08-02 04:22 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Sun Dec 22 06:01:30 2024 UTC