php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #76280 Performance degrades permantly when overwriting PHP files, until apache restart
Submitted: 2018-04-27 19:09 UTC Modified: 2021-03-02 13:21 UTC
Votes:3
Avg. Score:3.7 ± 0.9
Reproduced:2 of 2 (100.0%)
Same Version:2 (100.0%)
Same OS:1 (50.0%)
From: teo8976 at gmail dot com Assigned: cmb (profile)
Status: Not a bug Package: opcache
PHP Version: 5.6.36 OS: debian
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: teo8976 at gmail dot com
New email:
PHP Version: OS:

 

 [2018-04-27 19:09 UTC] teo8976 at gmail dot com
Description:
------------
I have a website (well, it's not mine actually) that gets about 1k requests per minute.

Sometimes I upload updated PHP files. Obviously this means I'm overwriting files at the very same time they are being executed. I expect that to cause errors (experienced by the users) during the short period of time while files are being overwritten (usually a few seconds).

However, what I wouldn't expect and happens VERY often is that: after that, general performance degrades, meaning that the execution of EVERY SINGLE script in every single request is significantly slower. Sometimes it's just a few dozens % increase in execution time, sometimes it explodes to the point that the whole websites goes down.

Restarting apache returns performance to normal every time, whether there had been a total collapse or just a degradation in performance.

Expected result:
----------------
When you overwrite PHP files in a website that is being visited by thousands of people, you'd expect a small percentage of them to experience random errors, if a file is being overwritten just when it's being executing.

However, as soon as all the requests that were being handled are done, everything should go back to working normally, and there shoul be no need to restart Apache

Actual result:
--------------
See description.

Seriously, this is fucking pathetic.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2018-04-27 19:18 UTC] teo8976 at gmail dot com
Here's some example performance data (oh come on, I can't attach a screenshot??) https://ibb.co/h3J6bx

This is avarage execution time (measured on the server side) of all requests, but graphs for any given single url would show exactly the same pattern.
 [2018-04-27 19:44 UTC] nikic@php.net
-Status: Open +Status: Feedback
 [2018-04-27 19:44 UTC] nikic@php.net
Please check whether the deployment of new files triggered an opcache restart (because the opcache SHM memory limit was reached). Generally any slowdown should only be temporary though.

In any case, PHP 5 is no longer supported and PHP 7 has quite a lot of changes to improve performance, including in opcache. Additionally, the second-level file cache can be used to reduce the performance impact of opcache restarts.
 [2018-04-28 14:54 UTC] teo8976 at gmail dot com
> Please check whether the deployment 
> of new files triggered an opcache restart

How can I check that?

> Generally any slowdown should only be temporary though

It isn't. When this happens, the slowdown is permanent until I restart apache.
In the screenshot I attached (where I unintentionally cut off the part that showed the times), it went on for several hours. It was also surprisingly "stable", everything being consistently X% slower (of course it is possible that the slowdown was affecting a specific piece of code being called by all scripts)

> In any case, PHP 5 is no longer supported

http://php.net/supported-versions.php says "security support until 31 december" this year. Something that can cause a website to go down permanently because of a performance degradation with no good reason, I would call a security issue.
 [2018-04-28 15:00 UTC] spam2 at rhsoft dot net
>> Please check whether the deployment 
>> of new files triggered an opcache restart
>>
> How can I check that?

look at the opcache vars and in doubt feed "opcache status panel" into Google and ask yourself why you don't have such tool installed - not knowing what a server does would i personally call a security issue (and no such a panel out-of-the-box is no good idea because *that* would be a security issue)

> Something that can cause a website to go down 
> permanently because of a performance degradation 
> with no good reason, I would call a security issue

nosense - following your logic running a server without opcache would be insecure at all
 [2018-04-28 15:13 UTC] teo8976 at gmail dot com
-Status: Feedback +Status: Open
 [2018-04-28 15:13 UTC] teo8976 at gmail dot com
> nosense - following your logic running a server without 
> opcache would be insecure at all

No, that is not following my logic. I meant something that can cause a SUDDEN crash of a website just because you do something as trivial as upload some files. I thought "security support" implicitly covered fixes for bugs so serious as to cause a system to go completely down triggered by trivial events.

I know *strictly speaking* this is not security related, but I thought in the context of support version lifecycle "security" was short for "is secure and at least barely usable".
 [2018-04-28 15:20 UTC] spam2 at rhsoft dot net
the problem is pretty sure that your "opcache.memory_consumption" setting is too low and you are trahsing opcache with a lot of changes - so either increase you config or reload the webserver after chaning lot of files

the most lkiley problem is that your configuration don't suite your workload and since you even don't know how to look pretty clear says you never gave a damn about your configs
 [2018-04-28 15:27 UTC] teo8976 at gmail dot com
> the most lkiley problem is that your configuration don't suite your workload

Let's assume this is the case.

That would explain a TEMPORARY degradation in performance.

Even if that is justified by a bad configuration which is my fault, if that doesn't recover on its own after a while that no file is being changed, then there's a bug.

> pretty clear says you never gave a damn about your configs

Indeed, I never did. But again, if I need to worry about fine-tuning my configs just for PHP to be stable, then it's ill-designed.
 [2020-11-09 17:28 UTC] cmb@php.net
-Status: Open +Status: Feedback -Assigned To: +Assigned To: cmb
 [2020-11-09 17:28 UTC] cmb@php.net
Is this still an issue with any of the actively supported PHP
versions[1]?

If so: if updated files are being cached by OPcache, the old files
will never be removed from the shared memory (this is reporteded
as wasted memory).  This wasted memory may sum up to the point
where you no longer have sufficient free memory, so no more files
can be cached, what would cause them to be executed uncached.
That might explain the behavior you've described.

[1] <https://www.php.net/supported-versions.php>
 [2020-11-22 04:22 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
 [2020-11-23 17:49 UTC] php4fan at gmail dot com
> if updated files are being cached by OPcache, the old files
> will never be removed from the shared memory (this is reporteded
> as wasted memory).  This wasted memory may sum up to the point
> where you no longer have sufficient free memory, so no more files
> can be cached, what would cause them to be executed uncached.

That is the current behavior?? If that wasted memory never gets freed, that sounds like a bug to me.

I haven't tested PHP 7 yet.
 [2020-11-23 18:06 UTC] cmb@php.net
Consider to read <http://blog.jpauli.tech/2015-03-05-opcache-html/>.
 [2020-11-23 19:31 UTC] rtrtrtrtrt at dfdfdfdf dot dfd
> That is the current behavior?? 
> If that wasted memory never gets freed, that sounds like a bug to me.

no, that's by design to avoid memory fragmentation which has it's own problems like APC had with it - in theory a ton of cache memory free but no free segment large enough to cache anything useful anymore
 [2021-03-01 19:16 UTC] php4fan at gmail dot com
Recently upgraded to 7.4, still observing the issue.
 [2021-03-01 19:25 UTC] svegeg at sgegegeg dot gh
why didn't you read http://blog.jpauli.tech/2015-03-05-opcache-html/ in the meantime to fix your buggy expectations?
 [2021-03-02 13:21 UTC] cmb@php.net
-Status: No Feedback +Status: Not a bug
 [2021-03-02 13:21 UTC] cmb@php.net
Feedback has been given, but still, this behaves as intended[1].

[1] <https://github.com/php/php-src/blob/php-7.4.16/ext/opcache/ZendAccelerator.c#L2118-L2126>
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Tue Apr 16 15:01:29 2024 UTC