php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #72322 High sys CPU usage
Submitted: 2016-06-03 06:58 UTC Modified: 2019-07-28 04:22 UTC
Votes:10
Avg. Score:4.7 ± 0.5
Reproduced:9 of 9 (100.0%)
Same Version:4 (44.4%)
Same OS:2 (22.2%)
From: jai at vidyamantra dot com Assigned:
Status: No Feedback Package: Performance problem
PHP Version: 7.0.7 OS: Centos 6.8 x64
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: jai at vidyamantra dot com
New email:
PHP Version: OS:

 

 [2016-06-03 06:58 UTC] jai at vidyamantra dot com
Description:
------------
Upgrade to php 7 and saw high CPU usage.
http://serverfault.com/q/780555/126826 (CPU graphs and sar output).

==== PHP 7 ====
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 31.24    0.000973           1      1204           munmap
 15.25    0.000475           0      1523         3 open
 12.52    0.000390           0      4012       216 lstat
  7.70    0.000240           0      1363           mmap
  5.81    0.000181           0      1528           close
  5.36    0.000167           1       219           getdents
  5.01    0.000156           0      1618           stat
  4.33    0.000135           0      1433       535 access
  3.69    0.000115           0      3256           fstat
  2.47    0.000077           0       547           write
  1.67    0.000052           0       224           uname
  1.22    0.000038           0       276           sendto
  0.87    0.000027           0       638           read
  0.83    0.000026           0       752           flock
  0.58    0.000018           0       760           poll

==== PHP 5 ====
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 23.32    0.001103           5       206           sendto
 17.45    0.000825           1      1650           getdents
 14.95    0.000707           0      4638       651 stat
 10.59    0.000501           0      3181      1708 access
  7.27    0.000344           0      4918       172 lstat
  6.07    0.000287           0      2391         3 open
  4.76    0.000225           1       314           poll
  3.66    0.000173           0      1526           munmap
  2.56    0.000121           0      4633           fstat
  2.11    0.000100           0      1704           mmap
  1.54    0.000073           1        97           rename
  1.40    0.000066           0       923         2 lseek
  1.08    0.000051           0      2395           close
  0.80    0.000038           0       562        49 ioctl
  0.68    0.000032           0       831           read
  0.57    0.000027           0       406           write
  0.34    0.000016           0       141           mprotect


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2016-12-17 14:51 UTC] nikic@php.net
-Status: Open +Status: Feedback
 [2016-12-17 14:51 UTC] nikic@php.net
Do you still experience this problem with a current version of PHP 7? I'd assume the issue is due to huge pages (see https://bugs.php.net/bug.php?id=70984), which are no longer enabled by default.
 [2016-12-20 07:33 UTC] jai at vidyamantra dot com
Tested on version 7.0.10 and problem is still present. It happens more often when system memory is near full (swap still empty), for e.g. system has 256GB of RAM and this happens when about only 10GB of RAM is used. At this stage, sys time increase significantly.
 [2016-12-25 04:22 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
 [2016-12-25 10:45 UTC] nikic@php.net
-Status: No Feedback +Status: Re-Opened
 [2017-01-20 09:16 UTC] jai at vidyamantra dot com
It has been two days since when we modified following kernel values.

vm.extra_free_kbytes
vm.min_free_kbytes

Since then, we are not facing this problem. It appears the problem is with the way how PHP7 allocates or uses memory.

We were able to reproduce this problem on two entirely different and isolated production servers that had no problem with php5 but this problem started within few hours after we upgraded to PHP7.

Our servers have 256GB of RAM and this issue appears when free memory is low. Clearing system buffers/cache (echo 3 > /proc/sys/vm/drop_caches) always fixes this problem but Linux gradually starts caching files from disk and free memory starts dropping.
 [2017-09-21 02:32 UTC] wym1 at meitu dot com
set opcache.huge_code_pages = 0 , it will work fine
 [2017-09-21 03:07 UTC] jai at vidyamantra dot com
opcache.huge_code_pages is already 0. If we don't use high values at vm.extra_free_kbytes and vm.min_free_kbytes we face this problem.
 [2017-12-06 11:44 UTC] pavel2000 at ngs dot ru
Catch the same issue with high sys CPU usage.

Opcache module completely disabled, only php-cli is in use when we see this issue.

Sys CPU usage reduced to "normal" state after command

#echo "never" >/sys/kernel/mm/transparent_hugepage/enabled


OS: Debian 8, PHP packages are from Debian testing.

Current kernel version is 

# uname -a
Linux web 3.2.0-4-amd64 #1 SMP Debian 3.2.82-1 x86_64 GNU/Linux

# php7.1 -v
PHP 7.1.11-1 (cli) (built: Oct 27 2017 13:47:15) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies

# php7.1 -m
[PHP Modules]
bcmath
calendar
Core
ctype
curl
date
dom
enchant
exif
fileinfo
filter
ftp
gd
gearman
gettext
gmp
hash
iconv
igbinary
intl
json
libxml
mbstring
mcrypt
memcached
msgpack
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_pgsql
pgsql
Phar
posix
readline
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
zip
zlib

[Zend Modules]
 [2019-07-16 10:37 UTC] nikic@php.net
-Status: Re-Opened +Status: Feedback
 [2019-07-16 10:37 UTC] nikic@php.net
Is anyone still seeing this issue? At some point we also disabled the use of transparent huge pages (via madvise rather than mmap), which I think may have been the cause here.
 [2019-07-28 04:22 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
 
PHP Copyright © 2001-2022 The PHP Group
All rights reserved.
Last updated: Fri Dec 09 02:05:54 2022 UTC