php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #77879 Usable memory
Submitted: 2019-04-12 01:48 UTC Modified: 2019-04-15 08:14 UTC
From: xxalfa at gmail dot com Assigned:
Status: Not a bug Package: Testing related
PHP Version: 7.3.4 OS: Windows
Private report: No CVE-ID: None
 [2019-04-12 01:48 UTC] xxalfa at gmail dot com
Description:
------------
The following script fills the memory with pointless data. Now the problem: From one point, the function "memory_get_usage" returns the same value, although the level has changed. This is important if you want to process data and save it only when a certain number of results have been generated. The second problem: Why are only 40% usable?

Test script:
---------------
<?php

    // C:\Users\User\Desktop\php-on-limit.cmd
    // @echo off
    // title PHP Development Server
    // cd "%cd%"
    // rem "C:\php\php.exe" "php-on-limit.php"
    // "C:\php\php.exe" "-d memory_limit=16M" "php-on-limit.php"
    // pause

    // C:\Users\User\Desktop\php-on-limit.php [PHP:7.3.3][PID:6336]

    //-------------------------------------------------
    // HEAD
    //-------------------------------------------------

    declare( strict_types = 1 );

    header( 'Content-Type:text/plain' );

    error_reporting( E_ALL );

    ini_set( 'display_errors', '1' );

    ini_set( 'html_errors', '0' );

    define( 'CORE_DIR', dirname( __FILE__ ) . DIRECTORY_SEPARATOR );

    isset( $argv ) or trigger_error( 'This is a command terminal application.', E_USER_ERROR );

    echo __FILE__ . ' [PHP:' . phpversion() . '][PID:' . getmypid() . ']' . PHP_EOL . PHP_EOL;

    //-------------------------------------------------
    // CODE
    //-------------------------------------------------

    $measuring_point_of_time = microtime( true );

    $memory_limit = conversion_back_to_bytes( ini_get( 'memory_limit' ) );

    echo 'memory_limit -- ' . human_readable_file_size( $memory_limit ) . ' -- defined limit' . PHP_EOL;

    // $memory_limit *= 0.48; // 48% of 128M can only be used

    // $memory_limit *= 0.47; // 47% of 64M can only be used

    $memory_limit *= 0.40; // 40% of 16M can only be used

    // If the 40% is exceeded, it comes to a fatal error, which is not desirable.

    // Fatal error: Allowed memory size of 16 MByte exhausted (tried to allocate 6 MByte).

    echo 'memory_limit -- ' . human_readable_file_size( $memory_limit ) . ' -- usable limit' . PHP_EOL;

    echo 'memory_allocated -- ' . human_readable_file_size( memory_get_usage( true ) ) . ' -- memory_used -- ' . memory_get_usage() . PHP_EOL;

    $something_has_to_be_calculated = true;

    $results_of_the_calculations = null;

    $last_memory_usage = memory_get_usage();

    while ( $something_has_to_be_calculated )
    {
        $results_of_the_calculations .= str_repeat( '1', 1024 );

        $currently_memory_usage = memory_get_usage();

        if ( $currently_memory_usage === $last_memory_usage )
        {
            echo PHP_EOL . 'Process recording: Memory function has not changed. currently_memory_usage and last_memory_usage are ' . $currently_memory_usage . PHP_EOL;

            break;
        }

        // echo chr( 0xD );

        echo 'memory_allocated -- ' . human_readable_file_size( memory_get_usage( true ) ) . ' -- memory_used -- ' . memory_get_usage() . ' -- data_length -- ' . strlen( $results_of_the_calculations );

        echo PHP_EOL;

        if ( memory_get_usage() >= 20 * 1024 * 1024 )
        {
            echo PHP_EOL . 'Process recording: Normal break.' . PHP_EOL;

            break;
        }

        if ( memory_get_usage() >= $memory_limit )
        {
            echo PHP_EOL . 'Process recording: Memory limit reached. Save calculated results so far.' . PHP_EOL;

            $results_of_the_calculations = null;
        }

        $last_memory_usage = memory_get_usage();
    }

    echo PHP_EOL;

    $results_of_the_calculations = null;

    echo 'Process recording: Calculations completed. Results are saved.' . PHP_EOL . PHP_EOL;

    echo 'Processing time: ' . elapsed_time( $measuring_point_of_time ) . ' seconds.' . PHP_EOL . PHP_EOL;

    //-------------------------------------------------
    // FUNCTIONS
    //-------------------------------------------------

    function elapsed_time( $measuring_point_of_time )
    {
        return number_format( microtime( true ) - $measuring_point_of_time, 3 );
    }

    function human_readable_file_size( $file_size, $precision = 2 )
    {
        $label = array( 'Bytes', 'kByte', 'MByte', 'GByte', 'TByte', 'PByte', 'EByte', 'ZByte', 'YByte' );

        return $file_size ? round( $file_size / pow( 1024, ( $index = floor( log( $file_size, 1024 ) ) ) ), $precision ) . ' ' . $label[ $index ] : '0 Bytes';
    }

    function conversion_back_to_bytes( $value )
    {
        $unit = strtolower( substr( $value, -1 ) );

        if ( $unit == 'k' ): return (integer) $value * 1024; endif;

        if ( $unit == 'm' ): return (integer) $value * 1048576; endif;

        if ( $unit == 'g' ): return (integer) $value * 1073741824; endif;

        return $value;
    }

?>

Expected result:
----------------
The "memory_get_usage" function must continually display the current memory usage to effectively operate at the memory limit.

Actual result:
--------------
C:\Users\User\Desktop\php-on-limit.php [PHP:7.3.4][PID:1224]

memory_limit -- 16 MByte -- defined limit
memory_limit -- 6.4 MByte -- usable limit
memory_allocated -- 2 MByte -- memory_used -- 410408
memory_allocated -- 2 MByte -- memory_used -- 411688 -- data_length -- 1024
memory_allocated -- 2 MByte -- memory_used -- 412968 -- data_length -- 2048
memory_allocated -- 2 MByte -- memory_used -- 414504 -- data_length -- 3072
memory_allocated -- 2 MByte -- memory_used -- 418600 -- data_length -- 4096

Process recording: Memory function has not changed. currently_memory_usage and last_memory_usage are 418520

Process recording: Calculations completed. Results are saved.

Processing time: 0.001 seconds.

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2019-04-12 02:26 UTC] requinix@php.net
Your script is quitting because PHP didn't need to allocate more memory to handle another 1k characters. Why is that a bug?
 [2019-04-12 02:58 UTC] xxalfa at gmail dot com
The script executes successfully because it has detected because the level of memory has not changed. However, the level increases by 1024 per cycle, if the function were to work properly, the script would continue to work. That's why it's a bug.
 [2019-04-12 03:06 UTC] requinix@php.net
No, the memory does not increase by 1024 each time. It increases by whatever amount PHP allocates in order to expand the string by another 1024 bytes. As your output shows, the last step it went up by 4096 - enough for approximately 4 increases.

Should it have only allocated 1k? Is it bad that PHP is reserving more memory than the precise amount it needs at that given moment?
 [2019-04-12 07:48 UTC] nikic@php.net
-Status: Open +Status: Not a bug
 [2019-04-12 07:48 UTC] nikic@php.net
Memory allocations are restricted to certain sizes for performance reasons. Small allocations are rounded up the next largest bin size. When the memory is reallocated with a size that still fits within the same bin, the reallocation is performed in-place and the reported memory usage will not change.
 [2019-04-14 03:20 UTC] xxalfa at gmail dot com
Reply to requinix@php.net
I am glad of every answer, only we both do not speak the same language. PHP can allocate as much memory as it needs. The amount of data I want to process requires knowing the amount of memory consumed to initiate paging operations. Any write access to the hard drive takes time, so I want to use as much memory as possible.

Reply to nikic@php.net
I thank you for your answer, which I now understand the way PHP works. I hope you can briefly answer my second question? When I open the task manager, I can keep track of how much memory a script consumes. Although PHP reserves full RAM, I can only fill it up to 40% at 16 MB. If I set the limit to 41% and fill the memory, I get a fatal error. In the Task Manager, a value of about 12 MB is given, why does not "memory_get_usage" provide the true consumption of 12 MB?
 [2019-04-15 08:14 UTC] nikic@php.net
@xxalfa With your script, this is the last output I get:

memory_allocated -- 10 MByte -- memory_used -- 6740856 -- data_length -- 6290432

Fatal error: Allowed memory size of 16777216 bytes exhausted at /home/nikic/php-7.4/Zend/zend_string.h:205 (tried to allocate 6291488 bytes) in /home/nikic/php-7.4/t022.php on line 65

As you can see, the attempt is to allocate 6MB additional memory, which ends up going above the memory limit. Even though the string doesn't take up the full 16MB, in order to reallocate it (if it's not possible to do so in-place) we first have to create a new larger allocation, copy the string and then release the old one. So you may need about 2x more memory to handle the reallocation.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Mon Dec 30 14:01:28 2024 UTC