php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #81092 Calling fflush before stream_filter_remove results in corrupted stream
Submitted: 2021-05-31 18:34 UTC Modified: 2021-06-07 14:08 UTC
From: mark at klb dot jp Assigned: cmb (profile)
Status: Closed Package: Bzip2 Related
PHP Version: 7.4 OS: Linux
Private report: No CVE-ID: None
 [2021-05-31 18:34 UTC] mark at klb dot jp
Description:
------------
When using filters to output bzip2 compressed data, one may be tempted to call fflush() before stream_filter_remove() to ensure all data has been written.

This used to work fine, however since php8, this causes the generated stream to be corrupt.

Test script:
---------------
<?php
  
if ($_SERVER['argv'][1]??'' == 'output') {
        $stream = fopen('php://output', 'wb+');
        $filter = stream_filter_append($stream, 'bzip2.compress', STREAM_FILTER_WRITE, ['blocks' => 9, 'work' => 0]);
        fwrite($stream, random_bytes(8192));
        fflush($stream);
        stream_filter_remove($filter); // should call dtor
        exit;
}

$script = $_SERVER['argv'][0];

$res = shell_exec('php '.escapeshellarg($script).' output'); // should output bz encoded data

$data = bzdecompress($res);
var_dump(strlen($data));


Expected result:
----------------
int(8192)


Actual result:
--------------
int(0)


Patches

Add a Patch

Pull Requests

Pull requests:

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2021-05-31 23:17 UTC] mark at klb dot jp
A couple notes:

I am not sure this is a bz2 issue, this could come from filters too. Outputting to php://output using a filter and fflush()ing before removing the filter are conditions for this bug to happen.

The bz2 stream generated is cut partially, using bzcat will for example output:

php bug.php output | bzcat -tvv
  (stdin): 
    [1: huff+mtf file ends unexpectedly
 [2021-06-03 08:08 UTC] nikic@php.net
-Assigned To: +Assigned To: cmb
 [2021-06-03 12:42 UTC] cmb@php.net
-Status: Assigned +Status: Verified -Package: Bzip2 Related +Package: Streams related -PHP Version: 8.0.6 +PHP Version: 7.4
 [2021-06-03 12:42 UTC] cmb@php.net
I can confirm that the script outputs int(0), but I get the same
result with PHP-7.4 either (regardless of explicitly calling
fflush()).  Interestingly, using a str_repeat("*", 8192) instead
of random_bytes(8192) gives the expected result.  If I use the
zlib.deflate and gzinflate(), I get an explicit:

    Warning: gzinflate(): data error

So apparently a general stream (compression) filter issue.
 [2021-06-03 16:50 UTC] mark at klb dot jp
Somehow this issue does not happen for me with php 7.4, and we started to have that issue as we upgraded from 7.4 to 8.0.6, so I assumed it was a php8 issue.

With:

$ php -v
PHP 7.4.14 (cli) (built: Mar  6 2021 04:12:14) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.14, Copyright (c), by Zend Technologies

(php 7.4.14 from Linux Gentoo, so might contain some patches)

$ php bug.php 
int(8192)

This seems to be related to large output of compressed data, so outputting something that compresses well (str_repeat, etc) will not cause this issue. random_bytes() generate a string with a lot of entropy, which will be difficult to compress and will result in larger compressed block(s).
 [2021-06-04 17:33 UTC] cmb@php.net
-Package: Streams related +Package: Bzip2 Related
 [2021-06-04 17:33 UTC] cmb@php.net
My first assessment of this bug report was done on Windows, where
I got more strange results, so I switched to Linux, where it
quickly turned out that this regression has been introduced by the
fix for bug #75776 (available as of PHP 7.4.17), and that there
are no issues with zlib.deflate.

> random_bytes() generate a string with a lot of entropy, which
> will be difficult to compress and will result in larger compressed
> block(s).

Indeed, that's the issue.

I'll investigate the Windows issues, and file bug reports if
appropriate, but lets keep these out of this ticket.
 [2021-06-07 14:05 UTC] cmb@php.net
The following pull request has been associated:

Patch Name: Fix #81092: fflush before stream_filter_remove corrupts stream
On GitHub:  https://github.com/php/php-src/pull/7113
Patch:      https://github.com/php/php-src/pull/7113.patch
 [2021-06-07 14:08 UTC] cmb@php.net
Argh, the erratic behavior on Windows was caused by shell_exec()
which uses text mode[1].  I've used a file for the regression
test; that fails without the patch, and works on Windows as well.

[1] <https://www.php.net/shell_exec>
 [2021-06-08 13:41 UTC] git@php.net
Automatic comment on behalf of cmb69
Revision: https://github.com/php/php-src/commit/a1738d8bd103b02e55a36e38aaee65964300889c
Log: Fix #81092: fflush before stream_filter_remove corrupts stream
 [2021-06-08 13:41 UTC] git@php.net
-Status: Verified +Status: Closed
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Sun Sep 19 20:03:37 2021 UTC