php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #75366 Raising memory_limit causes script to fail with no error.
Submitted: 2017-10-12 16:55 UTC Modified: 2017-10-12 20:07 UTC
From: vasilios at betoglou dot com Assigned:
Status: Closed Package: Reproducible crash
PHP Version: 7.0.24 OS: Linux helios 4.4.0-89-generic #1
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: vasilios at betoglou dot com
New email:
PHP Version: OS:

 

 [2017-10-12 16:55 UTC] vasilios at betoglou dot com
Description:
------------
I have a script that searches through files on the local system for a particular string and returns the results.  The form page does an ajax call to the function page.  This uses shell_exec to run the command locally.

If, while the command is running, the results get too large, an error like this is thrown:

[Thu Oct 12 12:21:49.963798 2017] [:error] [pid 16001] [client 172.19.32.179:50338] PHP Fatal error:  Allowed memory size of 41943040 bytes exhausted (tried to allocate 23068704 bytes) in /var/www/html/wp-content/themes/twentyfourteen-child/results.php on line 28, referer: https://<url>/metadata-search/

I upped the memory limit from 512M to 1024M I get an OOM error.  I upped the limit to 2048M, no error is logged, but the script just stops as if it had crashed.  Any value above ~1400M seems to be causing this issue, where the script returns no results and nothing is logged.

If I search for a term that returns smaller results, everything works fine:

(find command run CLI for example as to what the script is dealing with)

<user>@<host>:~$ find /home/metadata/arl.data/arlindex.xml -type f  | xargs grep -E 'host' /dev/null -nr |wc -l
863439 > OOM with memory_limit < ~1400M, script just dies if > ~1400M, no error thrown.

<user>@<host>:~$ find /home/metadata/arl.data/arlindex.xml -type f  | xargs grep -E 'betoglo' /dev/null -nr |wc -l
11 > WORKS FINE


Test script:
---------------
$cmd="find $directory$filename -type f  | xargs grep -E '$terms' $range /dev/null -nr";
$link=shell_exec($cmd);

Expected result:
----------------
I expect to see an OOM error or correct results.

Actual result:
--------------
I see no OOM error in /var/log/apache/error.log, the script dies with no results.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-10-12 17:05 UTC] spam2 at rhsoft dot net
memory_limit = -1

at least try if it makes any difference (and when you deal with many hundret MB for a single rqeuest the memory limit is useless anyway because some requests parallel would blow away the machine in any case)

then you have timeouts and so on....

the application is broken by design using that much resources and shell_exec() raises a red flag for a webserver
 [2017-10-12 17:21 UTC] vasilios at betoglou dot com
I've tried it with -1 with the same results.  This isn't really an issue with OOM, which is expected.  This is an issue where memory_limit is above x value, the script dies silently and no errors are logged.
 [2017-10-12 17:24 UTC] spam2 at rhsoft dot net
than "Raising memory_limit causes script to fail with no error" is simply wrong because -1 disbales memory_limit
 [2017-10-12 17:40 UTC] vasilios at betoglou dot com
I'm going to give you the benefit of the doubt in your slightly hostile response based on semantics of the "simply wrong" summary title (raising above x value, or disabling - raising to infinite value - causes an issue).

Is this a show stopper into confirming or explaining the issue I am seeing?
 [2017-10-12 17:53 UTC] spam2 at rhsoft dot net
i am not a php upstream developer but without a reproducer and with a wrong title at all it will be hard to get any further, PHP using gigabytes of RAM is no problem and since shell_exec() is part of the game the issue is probably completly out of php scope at all

in any case a reprdocuer is needed because nobody can fix something he can't see acting
 [2017-10-12 19:38 UTC] vasilios at betoglou dot com
Ok, so I tried taking shell_exec out of the equation by running something simple like this:

<?php
$var="ABCDEFGHIJKLMNOPQRSTUVWXYZ";
while(true){
        $var.=$var;
        echo memory_get_usage() . "<br>";
}

?>

It does throw an OOM error, and if I up the value to 4096M (higher than the value I saw the issue previously), it throws a different OOM error.  So you are right, it seems to be a link between shell_exec and memory_limit.

Output:

358856
358936
359032
359224
359672
360568
362872
366968
375160
387448
416120
469368
575864
788856
1214840
2066808
3770768
7178640
13994384
27625872
54888848
109414800
218466704
436570512
872778128
1745193360

error.log:
[Thu Oct 12 14:21:21.770973 2017] [:error] [pid 29059] [client 172.19.32.179:58966] PHP Fatal error:  Allowed memory size of 4294967296 bytes exhausted (tried to allocate 3489660960 bytes) in /var/www/html/test.php on line 4

Something like this, though, worked fine:

<?php
$var="";
while(true){
        $var.=shell_exec("echo $var");
        echo memory_get_usage() . "<br>";
}

?>

I.E. it threw an OOM error... so it's not necessarily the shell_exec but what it's doing.
 [2017-10-12 20:07 UTC] vasilios at betoglou dot com
-Status: Open +Status: Closed
 [2017-10-12 20:07 UTC] vasilios at betoglou dot com
I'll figure a workaround using PHP native commands - thanks for the feedback.
 [2017-10-12 20:20 UTC] spam2 at rhsoft dot net
> So you are right, it seems to be a link between shell_exec and memory_limit

both have nothing to do with each other because memory_limit is about PHP memory manager and anything allocated with shell-commands or libmysql (in case of mysql without mysqlnd) is *not* part of the memory managment of PHO at all

hence it makes no sense looking at memory_get_usage() when a large potion of your workload happens outside the PHP core
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Mar 29 12:01:27 2024 UTC