php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #63648 Gearman worker stops with segfault after 1-2 hour of working
Submitted: 2012-11-29 13:47 UTC Modified: 2017-01-10 08:23 UTC
Votes:5
Avg. Score:5.0 ± 0.0
Reproduced:5 of 5 (100.0%)
Same Version:1 (20.0%)
Same OS:1 (20.0%)
From: fruit dot dev at gmail dot com Assigned:
Status: Suspended Package: gearman (PECL)
PHP Version: 5.4.9 OS: FreeBSD 8.1 x64
Private report: No CVE-ID: None
Have you experienced this issue?
Rate the importance of this bug to you:

 [2012-11-29 13:47 UTC] fruit dot dev at gmail dot com
Description:
------------
My worker can't work for a long period of time, because it terminates with a 
segfault.

Jobs were added every minute with: 
  gearman -b -f testcase_01 1010101

After 2 hours of normal working, suddenly I got the message "Failed to set 
exception option".

This bug refers to: https://bugs.php.net/bug.php?id=63041

PECL Gearman:
  gearman support => enabled
  extension version => 1.1.0
  libgearman version => 1.1.3 (Compiled with --enable-assert and --enable-debug)
  Default TCP Host => localhost
  Default TCP Port => 4730
PHP: v5.4.8 (compiled with --enable-debug)


# Solution

After one week of investigations, I found the problem.

My gearmand server was running with "--verbose DEBUG", and in such case, it 
behavior is unpredictable. After I changed verbosity to INFO ("--verbose INFO") 
the issue disappears and my workers now works without stops.

To confirm, that it not a PECL bug, I have created a simple worker on C/C++ ( 
http://pastie.org/5452943 ). When I start it on gearmand with verbosity 
DEBUG, it starts to overload gearmand server with 100% of CPU. When is start it 
on gearmand server with verbosity INFO, my C/C++ worker and gearmand server 
didn't increased more than 1-2% of CPU and works continuously.

Test script:
---------------
<?php

  function testcase (GearmanJob $job)
  {
    printf("%s: %s|%s\n", date(DATE_ISO8601), $job->unique(), $job->workload());
  }

  $worker = new GearmanWorker();
  $worker->addServer();
  $worker->addFunction('testcase_01', 'testcase');
  $worker->setTimeout(1000);

  while ($worker->work() || $worker->returnCode() == GEARMAN_TIMEOUT)
  {
    if (GEARMAN_TIMEOUT == $worker->returnCode()) continue;
    if (GEARMAN_SUCCESS == $worker->returnCode()) continue;

    printf("return code: %d\n", $worker->returnCode());
    break;
  }

Expected result:
----------------
I'm expecting to test script never stops.

Actual result:
--------------
Program received signal SIGSEGV, Segmentation fault.
0x0000000801e3160b in gearman_packet_free (packet=0x802466098) at 
libgearman/packet.cc:289
289	    if (packet->universal->packet_list == packet)
Current language:  auto; currently c++
#0  0x0000000801e3160b in gearman_packet_free (packet=0x802466098) at 
libgearman/packet.cc:289
#1  0x0000000801e2af28 in gearman_connection_st::free_private_packet 
(this=0x802466000) at libgearman/connection.cc:338
#2  0x0000000801e34e87 in connection_loop (universal=@0x80245ac60, 
message=@0x7fffffffc390, check=@0x7fffffffc4e0) at 
libgearman/universal.cc:511
#3  0x0000000801e34f85 in gearman_request_option (universal=@0x80245ac60, 
option=@0x7fffffffc560) at libgearman/universal.cc:576
#4  0x0000000801e36a11 in gearman_worker_set_server_option 
(worker_shell=0x801a505a8, option_arg=0x801d16d3e "exceptions", 
option_arg_size=10) at libgearman/worker.cc:1297
#5  0x0000000801d12a2c in zif_gearman_worker_work (ht=0, 
return_value=0x801a4f190, return_value_ptr=0x0, this_ptr=0x801a4c390, 
return_value_used=1)
    at /usr/home/fruit/make/php-ext-gearman-1.1.0/php_gearman.c:3646
#6  0x00000000007f27dc in zend_do_fcall_common_helper_SPEC 
(execute_data=0x801a170d8) at zend_vm_execute.h:642
#7  0x00000000007f39b5 in ZEND_DO_FCALL_BY_NAME_SPEC_HANDLER 
(execute_data=0x801a170d8) at zend_vm_execute.h:752
#8  0x00000000007f1001 in execute (op_array=0x801a4d170) at 
zend_vm_execute.h:410
#9  0x00000000007b5a4a in zend_execute_scripts (type=8, retval=0x0, 
file_count=3) at /usr/home/fruit/make/php-
5.4.8/Zend/zend.c:1309
#10 0x0000000000729f03 in php_execute_script (primary_file=0x7fffffffe660) at 
/usr/home/fruit/make/php-5.4.8/main/main.c:2482
#11 0x0000000000902de4 in do_cli (argc=2, argv=0x7fffffffe928) at 
/usr/home/fruit/make/php-5.4.8/sapi/cli/php_cli.c:988
#12 0x0000000000903c79 in main (argc=2, argv=0x7fffffffe928) at 
/usr/home/fruit/make/php-5.4.8/sapi/cli/php_cli.c:1364

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2013-05-20 21:44 UTC] phplist at brandoncheckets dot com
Same problem on Ubuntu 12.04,  PHP 5.3.10, pecl gearman version 1.1.0

Worker is talking to gearmand server version 1.0.3.

The worker process segfaults *significantly* more often when the gearman server is run with verbosity=DEBUG, but even with gearman server verbosity set to NOTICE, it still segfaults, but is more difficult to 'catch'.

In my case the Segfault causes a loop where the PHP process consumes 100% of a single CPU core.

A bit from my strace of the processes as it crashed:


sendto(6, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
recvfrom(6, 0x1cdc461, 8192, 16384, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=4, events=POLLIN}], 3, 5000) = 1 ([{fd=6, revents=POLLIN}])
recvfrom(6, "\0RES\0\0\0\n\0\0\0\0", 8192, MSG_NOSIGNAL, NULL, NULL) = 12
sendto(7, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
recvfrom(7, 0x1cd7e61, 8192, 16384, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=7, events=POLLIN}, {fd=4, events=POLLIN}], 2, 5000) = 1 ([{fd=7, revents=POLLIN}])
recvfrom(7, "\0RES\0\0\0\n\0\0\0\0\0RES\0\0\0\n\0\0\0\0", 8192, MSG_NOSIGNAL, NULL, NULL) = 24
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb)                       = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb)                       = 0
 [2017-01-10 08:23 UTC] kalle@php.net
-Status: Open +Status: Suspended
 [2017-01-10 08:23 UTC] kalle@php.net
The gearman extension have not had much activity in the past few years, so I'm taking the safe bet that this is no longer under active development (as I don't consider the typo fixes on the github repo an active development), besides that it also targets unsupported PHP versions. Please unsuspend this report in case it begins to blossom with life once more
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Sun Oct 24 19:03:33 2021 UTC