php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #78092 Opcache shared lockfile and fcntl()
Submitted: 2019-05-31 12:11 UTC Modified: 2021-09-19 04:22 UTC
Votes:6
Avg. Score:4.3 ± 0.5
Reproduced:6 of 6 (100.0%)
Same Version:3 (50.0%)
Same OS:2 (33.3%)
From: stefan dot djordjevic at humanity dot com Assigned: cmb (profile)
Status: No Feedback Package: opcache
PHP Version: Irrelevant OS: Centos 7.6
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: stefan dot djordjevic at humanity dot com
New email:
PHP Version: OS:

 

 [2019-05-31 12:11 UTC] stefan dot djordjevic at humanity dot com
Description:
------------
Hi,

We got some strange behaviour which seems related to shared lockfile stored on a default path (/tmp/.ZendSem.04tkPY)
Randomly our Apache server with forked workers enter in some strange "loop", which slows them down and brings our load and number of apache workers to max.
I mange to strace them and what I see is:

cntl(10, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(10, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
.
.

At one point in time it just passed this "condition" and execute, finish request.

File Descriptor is actually pointing to opcode lock files:
ls -l  /proc/30174/fd/10 
lrwx------ 1 root root 64 May 30 13:35 /proc/30174/fd/10 -> /tmp/.ZendSem.04tkPY (deleted)

As this condition happens pretty randomly, it's hard to identify real cause of this, but from what I see is that it entering some kind of loop, or race condition with locking - unlocking pattern.

I am trying to figure out how we get to this condition, as there are many "variables" included in our new setup (docker containers etc.) I am wondering if you guys have any clue if this can be triggered by some kind of bug in Zend opcode cache-ing mechanism or similar ?

Thanks.





Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2021-09-09 12:22 UTC] cmb@php.net
-Status: Open +Status: Feedback -Assigned To: +Assigned To: cmb
 [2021-09-09 12:22 UTC] cmb@php.net
Is this still an issue with any of the actively supported PHP
versions[1]?  Note that "Irrelevant" is no longer supported.

[1] <https://www.php.net/supported-versions.php>
 [2021-09-19 04:22 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
 [2023-07-02 09:54 UTC] alexis at madrzejewski dot com
Hi,

I think I just encountered the same error.

We have customers complaining that they can't login to their WordPress websites. 

After some debugging, I found out that it's caused by a PHP process with a 100% CPU usage on wp-admin/load-styles.php, lopping over two system calls :


fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0
fcntl(3, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = 0

The file descriptor seems deleted too : 

[root@ruby fd]# ls -lha /proc/26052/fd/3
lrwx------ 1 xxxx xxxx 64  2 juil. 11:30 3 -> /tmp/.ZendSem.Tj7ZFx (deleted)

I think I only have this problem with PHP 8.1. 
If I change the PHP version to 8.2 or 8.0, it seems to fix the problem (at least, I don't have the customer complaining about this issue anymore)

The current version of PHP I'm using is : 

[root@ruby fd]# /opt/alt/php81/usr/bin/php -v 
PHP 8.1.20 (cli) (built: Jun 14 2023 08:52:43) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.20, Copyright (c) Zend Technologies

I'm using the litespeed (lsphp) : 

[root@ruby fd]# /opt/alt/php81/usr/bin/lsphp -v 
PHP 8.1.20 (litespeed) (built: Jun 14 2023 08:56:47)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v4.1.20, Copyright (c) Zend Technologies

I think it's related to opcache because if I stay on PHP 8.1 AND disable opcache, it also seems to fix the problem.

I'm not familiar with PHP internals, it seems to be a locking error of some kind.

It also could be a problem related to CloudLinux and their cageFS system because when I do a cagefsctl --remount {user} it seems to temporarily fix the problem for a few minutes. I'm not 100% sure it's a PHP bug.

Thanks.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Mar 29 11:01:29 2024 UTC