php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #58406 semaphores not cleaned up after
Submitted: 2008-11-04 11:53 UTC Modified: 2009-03-14 00:30 UTC
From: oliver at realtsp dot com Assigned:
Status: Closed Package: APC (PECL)
PHP Version: 5.2.5 OS: FreeBSd 7.0
Private report: No CVE-ID: None
 [2008-11-04 11:53 UTC] oliver at realtsp dot com
Description:
------------
sysV IPC semaphores are not cleaned up upon apc shutdown.

initially I thought it might be this bug:
http://bugs.php.net/bug.php?id=45423

patch to 5.2.6 realease here:
http://cvs.php.net/viewvc.cgi/php-
6

but actually the behaviour before and after applying that
patch to php5.2.6 is identical. apc_sem_destroy does appear
to be called both with and without that patch.

i added the following debug to apc_sem_destroy:

void apc_sem_destroy(int semid)
{
    /* we expect this call to fail often, so we do not check
*/
    union semun arg;
    int semPid = semctl(semid, 1, GETVAL, 0);
    apc_wprint("semPid=%d, getpid()=%d, semid=%d", semPid,
getpid(), semid);
    if (semPid == getpid()) {
        apc_wprint("apc_sem_destroy: semctl(%d, 0, IPC_RMID,
arg)\n", semid);
        semctl(semid, 0, IPC_RMID, arg);
    }
}

code never gets inside that if statement. ie semPid ==
getpid() is never true. in fact

semctl(semid, 1, GETVAL, 0) seems to return a value which is
unrelated to any process id?? Is this the right call?

the FreeBSD 7.0 man page for semctl says:

GETVAL       Return the value of semaphore number semnum.

and when SETVAL is called during 

apc_sem_create(const char* pathname, int proj, int initval)

it is set to initval which is passed as 

apc_lck_create(NULL, 0, 1, lock)

where

#define apc_lck_create(a,b,c,d) d=apc_sem_create(NULL,(b),

ie initval=1 always? therefore GETVAL should always return
1...??

This is not what I am getting in my debug, but I am also not
getting a valid pid and therefore the "pid" never matches
and the semaphores are never removed.

we are currently getting around the problem with some bash
scripting on restart:

/usr/local/etc/rc.d/fastcgi-php.sh stop

# clean up the semaphores
for id in `ipcs | grep www | awk '{print $2}'`
do
  ipcrm -s $id
done

/usr/local/etc/rc.d/fastcgi-php.sh start

Help anyone...am at the end of my C...

Reproduce code:
---------------
restart fastcgi server multiple times and monitor semaphore usage with 

root@torbay# ipcs
Message Queues:
T           ID          KEY MODE        OWNER    GROUP

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP

Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s      5242880            0 --rwarwarwa www      www
s      5242881            0 --rwarwarwa www      www
s      5242882            0 --rwarwarwa www      www
s      1966083            0 --rwarwarwa www      www
s      1966084            0 --rwarwarwa www      www
s      1179653            0 --rwarwarwa www      www




Expected result:
----------------
semaphore are removed on shutdown and left lying around

Actual result:
--------------
semphores build up and eventually you get:

[Mon Nov  3 23:04:58 2008] [apc-error] apc_sem_unlock:
semop(3080192) failed: No space left on device

due to the ultimately limited number available.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2008-11-08 22:11 UTC] shire@php.net
These semaphores should be cleaned up on module shutdown, I'm able to verify this does happen on a CLI run.  Perhaps you could build a CLI build of your PHP and verify that simply running a CLI script with APC enabled cleans up the semaphores on exit.

If you are then perhaps we should take a look at what the return values are for the semaphore deletion operation, (which we don't normally check as we have multiple processes trying to destroy the same semaphore).  And if we can limit your environment (FastCGI) to one process to test this out on that would simplify things as well.

In your code below i believe you want your line:

semctl(semid, 1, GETVAL, 0);

to be:

semctl(semid, 0, GETVAL, 0);

which is likely causing the seemingly random numbers you see.
 [2008-11-10 09:50 UTC] oliver at realtsp dot com
thanks for your feedback.

With apc.enable_cli="1" in my php.ini, i can reproduce the
problem on cli as shown below:

clearing any exsiting semaphores:

root@torbay# ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     10485760            0 --rwarwarwa root     wheel
s     10485761            0 --rwarwarwa root     wheel
s     10485762            0 --rwarwarwa root     wheel
s      6750211            0 --rwarwarwa root     wheel
s      6750212            0 --rwarwarwa root     wheel

root@torbay# ipcs | awk "{ if (\$5 == \"root\") print \$2}"
| xargs -n1 ipcrm -s


proving there are none:

root@torbay# ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP


run php cli

root@torbay# php -v
PHP 5.2.6 (cli) (built: Nov  4 2008 16:18:13)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend
Technologies

5 semaphores are left over

root@torbay# ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     10551296            0 --rwarwarwa root     wheel
s     10551297            0 --rwarwarwa root     wheel
s     10551298            0 --rwarwarwa root     wheel
s      6815747            0 --rwarwarwa root     wheel
s      6815748            0 --rwarwarwa root     wheel

I am running php 5.2.6stable with ACP 3.0.19 + the patch you
developed for

http://pecl.php.net/bugs/bug.php?id=14949

and that patch is the reason there are 5 semaphores (without
it there are 3).

with regards to the line:

semctl(semid, 1, GETVAL, 0);

it is part of APC-3.0.19 not my code.
I just printed that value to debug it, but was getting those
"seemingly random values".

Can you clarify what debug output you would like me to
produce?

Thanks
 [2008-11-10 10:59 UTC] oliver at realtsp dot com
I have added slightly different debug like this:

void apc_sem_destroy(int semid)
{
    /* we expect this call to fail often, so we do not check
*/
    union semun arg;
    int semPid = semctl(semid, 1, GETVAL, 0);
    apc_wprint("apc_sem_destroy: checking if semaphore %d
matches our pid: %d", semid, getpid());

    if (semPid == getpid()) {
        apc_wprint("apc_sem_destroy: removing semaphore %d",
semid);
        semctl(semid, 0, IPC_RMID, arg);
    }
    else
    {
        apc_wprint("apc_sem_destroy: semaphore %d is not
ours it has semPid=%d", semid, semPid);
    }
}


and get this output, which is inline with previous results,
ie the pid never matches.

root@torbay# ipcs | awk "{ if (\$5 == \"root\") print \$2}"
| xargs -n1 ipcrm -s
root@torbay# php -v
PHP 5.2.6 (cli) (built: Nov  4 2008 16:18:13)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend
Technologies
[Mon Nov 10 15:54:34 2008] [apc-warning] apc_sem_destroy:
checking if semaphore 10944513 matches our pid: 76206
[Mon Nov 10 15:54:34 2008] [apc-warning] apc_sem_destroy:
semaphore 10944513 is not ours it has semPid=10670
[Mon Nov 10 15:54:34 2008] [apc-warning] apc_sem_destroy:
checking if semaphore 7208963 matches our pid: 76206
[Mon Nov 10 15:54:34 2008] [apc-warning] apc_sem_destroy:
semaphore 7208963 is not ours it has semPid=10670
[Mon Nov 10 15:54:34 2008] [apc-warning] apc_sem_destroy:
checking if semaphore 10944512 matches our pid: 76206
[Mon Nov 10 15:54:34 2008] [apc-warning] apc_sem_destroy:
semaphore 10944512 is not ours it has semPid=10670
root@torbay# ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     10944512            0 --rwarwarwa root     wheel
s     10944513            0 --rwarwarwa root     wheel
s     10944514            0 --rwarwarwa root     wheel
s      7208963            0 --rwarwarwa root     wheel
s      7208964            0 --rwarwarwa root     wheel
 [2008-11-10 13:24 UTC] shire@php.net
I looked at APC-3.0.19 for every occurrence of semctl and there are no calls that pass '1' as the second argument.  So the line semctl(semid, 1, GETVAL, 0); is not valid.

For your CLI test, are you running *without* your debug modifications?
 [2008-11-10 14:11 UTC] oliver at realtsp dot com
ok firstly to clear up the source of that call..it's not me,
it's the freebsd port patching ACP:

http://www.freebsd.org/cgi/cvsweb.cgi/ports/www/pecl-

and it add 2 of those lines..one in create and one in
destroy it seems.

http://www.freebsd.org/cgi/cvsweb.cgi/ports/www/pecl-

the patch originally comes from here:
http://pecl.php.net/bugs/bug.php?id=5280

Since it never made it into APC cvs i will remove that patch
now and retest...stay tuned.
 [2008-11-10 14:13 UTC] oliver at realtsp dot com
right.... so the behaviour is different but not fixed for me
yet...

sorry for causing confusion by pasting all that Freebsd port
patched code earlier...here is my new debug code which
prints the retval of the semctl call as you suggested:



void apc_sem_destroy(int semid)
{
    /* we expect this call to fail often, so we do not check
*/
    union semun arg;
    int retval;
    retval = semctl(semid, 0, IPC_RMID, arg);
    apc_wprint("semctl retval= %d", retval);
}

and this is the test run with that debug:

root@torbay# ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP

root@torbay# php -v
PHP 5.2.6 (cli) (built: Nov  4 2008 16:18:13)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend
Technologies
[Mon Nov 10 19:10:36 2008] [apc-warning] semctl retval= 0
[Mon Nov 10 19:10:36 2008] [apc-warning] semctl retval= 0
[Mon Nov 10 19:10:36 2008] [apc-warning] semctl retval= 0
root@torbay# ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     11272194            0 --rwarwarwa root     wheel
s      7536644            0 --rwarwarwa root     wheel

so just 2 semaphore rather than 5 (which seems more
logical...one blocking and one non-blocking)...but they
still hand around...

next suggestion?
 [2008-11-10 14:31 UTC] oliver at realtsp dot com
more info...

tried to reduce fastcgi server to one process but min i
could get is 3, 1 parent + 2 children..so sticking to cli
and running:

php -r 'sleep(10);'

five semaphores are still used and then 3 are removed
successfully (hence the 0 retvals in previous debug) but 2
remain:

while php-cli is running:

oliver@torbay$ ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     12451840            0 --rwarwarwa root     wheel
s     12451841            0 --rwarwarwa root     wheel
s     11927554            0 --rwarwarwa root     wheel
s      8454147            0 --rwarwarwa root     wheel
s      8192004            0 --rwarwarwa root     wheel

after it finishes:

oliver@torbay$ ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     11927554            0 --rwarwarwa root     wheel
s      8192004            0 --rwarwarwa root     wheel

so that points the finger to the calling code I guess?

    for (i = 0; i < sma_numseg; i++) {
        apc_lck_destroy(((header_t*)sma_shmaddrs[i])-
 [2008-11-10 14:33 UTC] oliver at realtsp dot com
also wondering whether removing that patch in the freebsd
port given here originally:

http://pecl.php.net/bugs/bug.php?id=5280

is going to cause problems...testing now..
 [2008-11-10 14:58 UTC] oliver at realtsp dot com
16000 requests later with siege and the APC seems be ok
without that dodgy patch...

also when I remove the recent non-blocking lock which was
committed to CVS from this bug:

http://pecl.php.net/bugs/bug.php?id=14949

only 3 semaphores are created (as we already knew) and all
three are correctly cleaned up:

while php -r 'sleep(10);' is running


oliver@torbay$ ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP
s     13238272            0 --rwarwarwa root     wheel
s     13238273            0 --rwarwarwa root     wheel
s     12320770            0 --rwarwarwa root     wheel

after it quits

oliver@torbay$ ipcs -s
Semaphores:
T           ID          KEY MODE        OWNER    GROUP

so we are getting closer:

a) the patch from here:
http://pecl.php.net/bugs/bug.php?id=5280
is buggy and since it caused 3 semaphores to be left lying
around, and it is dubious that it helps at all?? (i don't
quite understand what it is trying to do)

b) the more recent patch committed by shire from here:
http://pecl.php.net/bugs/bug.php?id=14949
works to prevent "cache slam" but creates 2 extra semaphores
(correct), and fails to clean them up.

the latter makes sense when we look at the calling for loop
which is basically 
"foreach (shared memory segment) { clear its semaphore) }"
no mention of non-blocking semaphores 

kind of obvious cause for the extra 2 left over...

shire can you confirm that the clear cli runs which you had
were *without* your recent patch from 
http://pecl.php.net/bugs/bug.php?id=14949

and perhaps suggest a way that these can be recorded and
then cleaned up?

thanks
 [2008-11-10 15:49 UTC] oliver at realtsp dot com
this simple patch seems to clean up the non-blocking
semaphores

http://www.realtsp.com/download/patch-

works for you?
 [2008-11-10 15:50 UTC] shire@php.net
Bad URL :-(
 [2008-11-10 15:52 UTC] oliver at realtsp dot com
always corrupting long urls!

http://www.realtsp.com/download/patch-apc_cache_destroy_nb.c
 [2008-11-10 15:54 UTC] shire@php.net
Sorry, already in CVS.  Can you try the latest CVS build and see if it fixes your problems?
 [2008-11-11 02:43 UTC] oliver at realtsp dot com
sigh....

yes latest CVS works just fine!

Is there an easy way to get a snapshot for pecl/APC? short
of checking it out from annonymous CVS in php and then
making your own tarball? The php snapshots here:

http://snaps.php.net/

don't seem to contain APC in the /ext directory.


Anyway I guess we have a solution for this bug:

a) Don't use the patch from:
http://pecl.php.net/bugs/bug.php?id=5280

b) use the latest CVS of APC which has DESTROY(wrlock)

Does this help justify a new point release? looking at the
changelog there seem to be quite a few bug fixes and tidy
ups, plus the semaphore based non-blocking lock feature for
those on BSD style OSs.

Thanks for your help.
 [2008-11-17 14:17 UTC] shire@php.net
Thanks for the verification.

I'm not sure on the snaps builds, to make sure you have our latest code I would suggest using the cvs checkout yourself.

Yep, we're actively working towards a new release.
 [2009-06-15 11:41 UTC] rasmusrnielsen at gmail dot com
>> Yep, we're actively working towards a new release.

Can we expect a 3.0.20 anytime soon or do you recommend to 
build from cvs?
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Fri Sep 24 03:03:36 2021 UTC