php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #59909 Gearman Workers Run Away causing gearmand cpu load to 99%
Submitted: 2011-08-22 17:12 UTC Modified: 2013-02-18 00:35 UTC
Votes:1
Avg. Score:4.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:0 (0.0%)
Same OS:0 (0.0%)
From: mike at digitalstruct dot com Assigned: hradtke (profile)
Status: No Feedback Package: gearman (PECL)
PHP Version: 5.3.5 OS: RHEL 5.6
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If this is not your bug, you can add a comment by following this link.
If this is your bug, but you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: mike at digitalstruct dot com
New email:
PHP Version: OS:

 

 [2011-08-22 17:12 UTC] mike at digitalstruct dot com
Description:
------------
See the following bug report: 
https://bugs.launchpad.net/gearmand/+bug/802850

It seems like it happens at random and only under heavier 
load.  When the workers are killed the server returns to 
normal and 
after the restart of the worker it seems to function fine.

What I've noticed is two main things between a working worker 
and a non-working worker (basically when things are fried).  
The 
working ones block and the non-working ones stop blocking but 
simply DDoS the gearmand process.


Expected result:
----------------
I would think that the duration of time should not matter 
that the worker is running.  The fact that it stops blocking 
should never happen while it is sitting in it's while loop.

The fact that it DDoS the gearmand process means that it is 
likely losing some pointer to the connection and unable to 
resolve itself.  It does not give any useful error message 
but continues in the way above until killed.

Actual result:
--------------
Working:
poll([{fd=13, events=POLLIN}], 1, 1000) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
sendto(13, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 
0) = 12
recvfrom(13, "\0RES\0\0\0\n\0\0\0\0", 8192, 0, NULL, NULL) = 
12
sendto(13, "\0REQ\0\0\0\4\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 
0) = 12
poll([{fd=13, events=POLLIN}], 1, 1000 <unfinished ...>

Non-Working:
getsockopt(13, SOL_SOCKET, SO_ERROR, [284636038280773632], 
[4]) = 0
sendto(13, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 
0) = 12
sendto(13, "\0REQ\0\0\0\4\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 
0) = 12
poll([{fd=13, events=POLLIN}], 1, 1000) = 1 ([{fd=13, 
revents=POLLIN}])
getsockopt(13, SOL_SOCKET, SO_ERROR, [284636038280773632], 
[4]) = 0
sendto(13, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0 
<unfinished ...>

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2011-08-22 20:07 UTC] hradtke@php.net
Did we rule out libgearman?
 [2011-08-22 22:50 UTC] mike at digitalstruct dot com
Unfortunately it is extremely hard to rule out either the 
extension or libgearman at this point.  I've gone back and 
forth between both areas but I do not see any clear 
definition between the two in where it is breaking.

Any ideas to help diagnose?
 [2011-10-12 21:31 UTC] felipe@php.net
-Status: Open +Status: Assigned
 [2012-09-24 01:20 UTC] hradtke@php.net
-Status: Assigned +Status: Feedback
 [2012-09-24 01:20 UTC] hradtke@php.net
Thank you for this bug report. To properly diagnose the problem, we
need a short but complete example script to be able to reproduce
this bug ourselves. 

A proper reproducing script starts with <?php and ends with ?>,
is max. 10-20 lines long and does not require any external 
resources such as databases, etc. If the script requires a 
database to demonstrate the issue, please make sure it creates 
all necessary tables, stored procedures etc.

Please avoid embedding huge scripts into the report.


 [2013-02-18 00:35 UTC] pecl-dev at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Open". Thank you.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Tue Mar 19 06:01:30 2024 UTC