php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #80341 OPCache wasted memory grows rapidly and no restart after max_wasted_percentage
Submitted: 2020-11-09 00:09 UTC Modified: 2020-11-17 16:57 UTC
From: alex at ndros dot com Assigned:
Status: Open Package: opcache
PHP Version: 7.4.12 OS: Windows 2019
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: alex at ndros dot com
New email:
PHP Version: OS:

 

 [2020-11-09 00:09 UTC] alex at ndros dot com
Description:
------------
I run PHP 7.4.12 on a Windows 2019 IIS10 server.

The wasted memory value in PHP Opcache grows rapidly, fills up my entire 2GB allocated to Opcache, and then opcache stops working.

For example, these are the statistics after 30mins since the last webserver reset:

total memory: 2.00GB
used memory: 89.00MB
free memory: 1.66GB
wasted memory: 261.36MB (12.76%)

And these are my settings:

opcache.memory_consumption=2048
opcache.cache_id=oc1
opcache.error_log="D:\temp\php\opcache_errors.log"
opcache.validate_timestamps=1
opcache.revalidate_freq=90
opcache.interned_strings_buffer=32
opcache.save_comments=0
opcache.max_file_size=0
opcache.file_update_protection=0
opcache.file_cache_consistency_checks=0
opcache.file_cache="D:\temp\php\opcache_filecache"
opcache.max_accelerated_files=100000

If I leave it to run more time, the wasted memory fills up my entire 2GB allocated to Opcache, and opcache will simply stop to work. I can verify this be updating .php files and the changes show up instantly (where normally they take 90 seconds or less, based on my opcache.revalidate_freq).


Thanks
Alex


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2020-11-09 00:10 UTC] alex at ndros dot com
Also, the default opcache.max_wasted_percentage of 5% does not seem to work. Nothing happens when 5% is exceeded.
 [2020-11-09 10:27 UTC] cmb@php.net
-Status: Open +Status: Feedback -Assigned To: +Assigned To: cmb
 [2020-11-09 10:27 UTC] cmb@php.net
> I run PHP 7.4.12 […]

Assuming it is about a build from windows.php.net, which one
exactly?  php-7.4.12-nts-Win32-vc15-x64.zip?

> I can verify this be updating .php files and the changes show up
> instantly (where normally they take 90 seconds or less, based on
> my opcache.revalidate_freq).

Are these files stored on some file share, or are they somehow
linked (e.g. behind a junction)?

Is there any possibly relevant info in the OPcache error log?
 [2020-11-09 14:36 UTC] alex at ndros dot com
Hi there,

Yes, I'm using php-7.4.12-nts-Win32-vc15-x64.zip

We're using a load balance with multiple IIS server. Yes, the files are located in a symlink pointing to an SMB share.

But it seems that I have solved the problem with the wasted memory. I switched off opcache.file_cache and now after 12 hours the wasted memory is at 0%. So whatever it was, it was somehow related to the file_cache.

However, the problem that Opcache not reseting after opcache.max_wasted_percentage is reached still remains. I did several tests for this, and it never resets when the value was reached.
 [2020-11-09 16:05 UTC] cmb@php.net
-Status: Feedback +Status: Assigned
 [2020-11-09 16:05 UTC] cmb@php.net
Thanks for the swift reply!

I'll have a closer look.
 [2020-11-11 16:13 UTC] cmb@php.net
-Status: Assigned +Status: Feedback
 [2020-11-11 16:13 UTC] cmb@php.net
Hmm, I still cannot reproduce any of the reported behavior.

> Yes, the files are located in a symlink pointing to an SMB
> share.

This might be reason for the misbehavior.  Could you please
post what

    fsutil reparsePoint query <path>

reports for the symlink and the SMB share?

Could you also please check whether realpath() resolves any of the
files on the SMB share correctly, and also what filemtime() and
stat()['mtime'] reports?  I'm not so much interested in the exact
timestamps, but rather whether these are "in the future" what could
explain the premature updates.
 [2020-11-11 17:08 UTC] cmb@php.net
Also, which version of SMB is in use there?  Version 1 is insecure
and has other known issues, so you should use at least v2 or
better v3.

Furthermore, I'm assuming that the network connection to that SMB
share is very stable; otherwise an unstable connection might be
the root cause for "arbitrary" issues in this regard.  And likely,
you are better of deploying all the PHP files to the individual
machines instead of hosting them on an SMB share, anyway.
 [2020-11-12 15:33 UTC] alex at ndros dot com
-Status: Feedback +Status: Assigned
 [2020-11-12 15:33 UTC] alex at ndros dot com
Hi there,

We're using SMB3. The network connection to the SMB share is perfectly stable.

The reason for the pre-mature updates was that OPcache stopped running altogether. All opcache statistics froze. After wasted_memory filled my entire 2GB allocation, Opcache stopped running.

The wasted memory problem is gone since we removed filecache. Right now, Opcache is much healthier:

total memory: 256.00MB
used memory: 92.91MB
free memory: 163.09MB
wasted memory: 0.00b (0%)

number of cached files: 3,229
number of hits: 1,623,566,737
number of misses: 16,133
blacklist misses: 1,732,646
number of cached keys: 5,683
max cached keys: 130,987
interned strings usage
buffer size: 24.00MB
used memory: 7.59MB
free memory: 16.41MB
number of strings: 109,680

However, the problem that opcache will not reset after the max_wasted_percentage value is reached still remains.

The output of that command is:

fsutil reparsePoint query D:\inetpub\wwwroot
Reparse Tag Value : 0xa000000c
Tag value: Microsoft
Tag value: Name Surrogate
Tag value: Symbolic Link

Reparse Data Length: 0x00000068
Reparse Data:
0000:  28 00 34 00 00 00 28 00  00 00 00 00 5c 00 5c 00  (.4...(.....\.\.
0010:  31 00 30 00 2e 00 31 00  2e 00 31 00 2e 00 31 00  1.0...1...1...1.
0020:  30 00 30 00 5c 00 77 00  77 00 77 00 72 00 6f 00  0.0.\.w.w.w.r.o.
0030:  6f 00 74 00 5c 00 3f 00  3f 00 5c 00 55 00 4e 00  o.t.\.?.?.\.U.N.
0040:  43 00 5c 00 31 00 30 00  2e 00 31 00 2e 00 31 00  C.\.1.0...1...1.
0050:  2e 00 31 00 30 00 30 00  5c 00 77 00 77 00 77 00  ..1.0.0.\.w.w.w.
0060:  72 00 6f 00 6f 00 74 00                           r.o.o.t.

I cannot publish much more information here. If you want, I can give you access to a VM in the load balance, with the initial settings that caused the problem so that you can debug.

Thanks
Alex
 [2020-11-13 15:48 UTC] cmb@php.net
There are general issues regarding OPcache restarts on Windows.  A
long running process attached to the same OPcache instance
apparently completely blocks restarts, even if
opcache.force_restart_timeout has expired.  While a long running
request should probably use its own OPcache instance, not
regarding opcache.force_restart_timeout appears to be bug.

And if opcache.file_cache is enabled, restarts take considerable
time between the schedule and the actual restart, contrary to
opcache.file_cache being disabled.

> I cannot publish much more information here.

Okay, but actually I'm only interested in the "Reparse Tag Value"
of the SMB share, since we do not support arbitrary reparse tags,
and this might explain some issues.
 [2020-11-17 10:51 UTC] cmb@php.net
> […] , not regarding opcache.force_restart_timeout appears to be
> bug.

Well, on a closer look, it doesn't.  While on non Windows
platforms, all processes which share an OPcache instance are
forked childs of the same process, on Windows arbitrary processes
may share an OPcache instance, so forcefully terminating these
processes may not even be possible. At least, there is currently
no provision to track the relevant processes, so for now this is a
documentation problem, fixed with
<http://svn.php.net/viewvc?view=revision&revision=351404>.
 [2020-11-17 16:57 UTC] cmb@php.net
-Assigned To: cmb +Assigned To:
 [2020-11-17 16:57 UTC] cmb@php.net
> However, the problem that opcache will not reset after the
> max_wasted_percentage value is reached still remains.

It is important to note that a restart is only triggered, if there
actually is not enough free memory (i.e. some SHM allocation
failed).  I've just updated the manual accordingly[1].

That said, for me OPcache restarts in that case.

[1] <http://svn.php.net/viewvc?view=revision&revision=351415>
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Thu Aug 05 06:01:23 2021 UTC