php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #79013 Content-Length missing when posting a curlFile with curl
Submitted: 2019-12-21 16:19 UTC Modified: 2020-03-03 08:25 UTC
Votes:4
Avg. Score:5.0 ± 0.0
Reproduced:4 of 4 (100.0%)
Same Version:4 (100.0%)
Same OS:0 (0.0%)
From: christian at klemmer dot io Assigned: cmb (profile)
Status: Closed Package: cURL related
PHP Version: 7.4.1 OS: debian 10.2
Private report: No CVE-ID: None
 [2019-12-21 16:19 UTC] christian at klemmer dot io
Description:
------------
When posting a curlFile with curl, I get different results with the same code (and curlFile) if I run it with PHP 7.3 or PHP 7.4.

"Content-Length" is missing and instead replaced by "Transfer-Encoding: chunked". I query an API which is behind an haproxy server. Missing content-lenght seems to remove all files posted on their side.
Nevertheless I would've expected the same behaviour with PHP 7.4 running with the same curl-version.

I'm using the deb.sury.org repository on a debian 10.2 server.
dpkg -l | grep curl
ii  curl                                 7.64.0-4
ii  libcurl3-gnutls:amd64                7.64.0-4
ii  libcurl4:amd64                       7.64.0-4
ii  php7.3-curl                          7.3.13-1+0~20191218.50+debian10~1.gbp23c2da
ii  php7.4-curl                          7.4.1-1+0~20191218.8+debian10~1.gbp21c50e
ii  python3-pycurl                       7.43.0.2-0.1

curl-information out of phpinfo (7.3 testenv and 7.4 testenv are exactly the same):

cURL support	enabled
cURL Information	7.64.0
Age	4
Features:
AsynchDNS	Yes
CharConv	No
Debug	No
GSS-Negotiate	No
IDN	Yes
IPv6	Yes
krb4	No
Largefile	Yes
libz	Yes
NTLM	Yes
NTLMWB	Yes
SPNEGO	Yes
SSL	Yes
SSPI	No
TLS-SRP	Yes
HTTP2	Yes
GSSAPI	Yes
KERBEROS5	Yes
UNIX_SOCKETS	Yes
PSL	Yes
HTTPS_PROXY	Yes
MULTI_SSL	No
BROTLI	No
Protocols	dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, ldaps, pop3, pop3s, rtmp, rtsp, scp, sftp, smb, smbs, smtp, smtps, telnet, tftp
Host	x86_64-pc-linux-gnu
SSL Version	OpenSSL/1.1.1d
ZLib Version	1.2.11
libSSH Version	libssh2/1.8.0

Test script:
---------------
$filename = 'testfile.txt';
if(file_exists($filename) && is_file($filename) && (int)filesize($filename) > 0) {
    try {
        $ch = curl_init('https://example.com/') or die('API down');
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_HEADER, true);
        curl_setopt($ch, CURLINFO_HEADER_OUT, true);
        curl_setopt($ch, CURLOPT_VERBOSE, true);
        curl_setopt($ch, CURLOPT_POST, true);
        curl_setopt($ch, CURLOPT_POSTFIELDS, array('test' => new CurlFile($filename)));
        curl_exec($ch);
        $curlInfo = curl_getinfo($ch);
        echo '<pre>'; print_r($curlInfo["request_header"]); echo '</pre>';
        curl_close($ch);
    } catch(Exception $e) {
        echo $e->getMessage();
    }
}

Expected result:
----------------
With PHP 7.3.13:
POST / HTTP/2
Host: example.com
Accept: */*
Content-Length: 245
Content-Type: multipart/form-data; boundary=------------------------c129897ae3ed733b

Actual result:
--------------
With PHP 7.4.1:
POST / HTTP/2
Host: example.com
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=------------------------ef5ab6d1e2455a47

Patches

Add a Patch

Pull Requests

Pull requests:

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2019-12-21 20:46 UTC] requinix@php.net
-Assigned To: +Assigned To: cmb
 [2019-12-21 20:46 UTC] requinix@php.net
Note that there's nothing inherently wrong with this behavior. Honestly, I think the bug is that your proxy somehow isn't allowing the file upload when it comes chunked instead of in full. I suggest looking into that.

Anyway, looks like this change happened during request #77711
  https://github.com/php/php-src/commit/c68dc6b5e37e74d89e0a387079139c054c8faa81
where using a CURLFile opts into cURL's support for streams, and it can't know the stream length so it has no choice but to go chunked.
 [2019-12-22 12:49 UTC] christian at klemmer dot io
Reading a bit more about it, I'm not sure if "Transfer-Encoding: chunked" is allowed, when using HTTP/2.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
"HTTP/2 doesn't support HTTP 1.1's chunked transfer encoding mechanism, as it provides its own, more efficient, mechanisms for data streaming."
 [2019-12-22 20:39 UTC] requinix@php.net
But are you actually using HTTP/2? Sure, your example says "HTTP/2" in it, but the headers themselves are HTTP/1.

Is your repro script the same as what you posted here?
 [2019-12-22 20:47 UTC] christian at klemmer dot io
Yes, it's 100% the same script.
After writing down my reposcript I actually posted a little file to https://example.com/ to test if it's working there also. It is.

And https://example.com/ is also showing with HTTP/2 in Google Chrome, so I think it's capable of doing so.
 [2019-12-22 21:15 UTC] requinix@php.net
Thing is, this doesn't make any sense.

1. I can't imagine cURL will use HTTP/2 without an Upgrade or explicit foreknowledge about remote support for it
2. I would also expect it to use HTTP/2 only if the passed options supported it (at least until it's more common)
3. Like I said, the headers are completely wrong

I only really see two possibilities: either something in this bug report is incorrect, or cURL has some sort of massive bug(s) regarding HTTP/2 support that nobody seems to have noticed until now. No offense but from where I'm sitting, only one of those is believable.

As for reproducing,

On Ubuntu with libcurl 7.58 I get a chunked HTTP/1.1 request with PHP 7.4.1 and an unchunked HTTP/1.1 request with PHP 7.3.13, so if something changed in cURL it was between 7.58 and 7.64.
 [2019-12-22 21:21 UTC] bugreports at gmail dot com
a) Curl prefers HTTP/2 when the sevrer supports it
b) the server of the reporter support HTTP2 as he statet
c) that below is curl against a http2 proxy with a http1 backend

the origin sned all the headers in the style of "Cache-Control" and the proxy talking http2 transofrms them to lowercase

curl --head https://localhost/

HTTP/2 200
date: Sun, 22 Dec 2019 21:19:35 GMT
strict-transport-security: max-age=31536000
x-frame-options: SAMEORIGIN
etag: 786c872ae3730afa5623a43d8ee3e38b
cache-control: private
last-modified: Mon, 28 Nov 2016 16:55:29 GMT
vary: Accept-Encoding,User-Agent
content-type: text/html; charset=ISO-8859-1
age: 0
 [2019-12-22 21:38 UTC] requinix@php.net
Ah, I didn't know HTTP/2 support was indicated in the TLS handshake. That explains some.

So I'm back to the headers being wrong. Maybe I just don't have the right environment to test this.
 [2019-12-22 22:14 UTC] christian at klemmer dot io
I did some additional testing with some VMs:

Ubuntu 19.10 - PHP 7.4.1 - CURL 7.65.3 (7.65.3-1ubuntu3):
-----------------------
POST / HTTP/2
Host: example.com
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=------------------------d498818044360b85



Debian 9.11 - PHP 7.4.1 - CURL 7.52.1 (7.52.1-5+deb9u9):
-----------------------
POST / HTTP/1.1
Host: example.com
Accept: */*
Content-Length: 210
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------a4980090fbc4beae



Debian 10.2 - PHP 7.4.1 - CURL 7.64.0 (7.64.0-4) (same versions as my initial bug report, but different machine):
-----------------------
POST / HTTP/2
Host: example.com
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=------------------------f8e60d58d7f0df39



Here are the steps to reproduce my exact test environment:
Install Debian 10.2 (buster).
apt install apt-transport-https lsb-release ca-certificates
curl -fsSL https://packages.sury.org/php/apt.gpg | apt-key add -
sh -c 'echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" >> /etc/apt/sources.list'
apt update
apt install php7.4-bcmath php7.4-bz2 php7.4-cgi php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-fpm php7.4-gd php7.4-imagick php7.4-imap php7.4-intl php7.4-json php7.4-ldap php7.4-mbstring php7.4-mysql php7.4-readline php7.4-soap php7.4-xml php7.4-zip
 [2019-12-22 22:41 UTC] christian at klemmer dot io
And as addition tests with PHP 7.3.13 with different curl versions.
All result in the expected behaviour.

Ubuntu 19.10 - PHP 7.3.13 - CURL 7.65.3 (7.65.3-1ubuntu3):
-----------------------
POST / HTTP/2
Host: example.com
Accept: */*
Content-Length: 210
Content-Type: multipart/form-data; boundary=------------------------4d274c175c3e77e5



Debian 9.11 - PHP 7.3.13 - CURL 7.52.1 (7.52.1-5+deb9u9):
-----------------------
POST / HTTP/1.1
Host: example.com
Accept: */*
Content-Length: 210
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------5652b6f1c9e7a818



Debian 10.2 - PHP 7.3.13 - CURL 7.64.0 (7.64.0-4):
-----------------------
POST / HTTP/2
Host: example.com
Accept: */*
Content-Length: 210
Content-Type: multipart/form-data; boundary=------------------------0b86b20895500f64

So it seems to be a combination of several CURL versions with PHP 7.4
 [2019-12-23 08:54 UTC] cmb@php.net
Thanks for reporting and digging into the details!

In my opinion, no longer sending a Content-Length header but
instead falling back to chunked transfer encoding is a bug at
least for PHP 7.3, where request #77711 has recently been
back-ported to[1].  For PHP 7.4, this change might be acceptable,
but would at least have to be documented.

However, while having a closer look at this issue, I've noticed
that the current implementation can't really work wrt.
curl_copy_handle()[2], what has to be addressed first, since fixing
that will result in a quite different implementation, which
renders the obvious trivial fix for this issue moot.

> So it seems to be a combination of several CURL versions with
> PHP 7.4

To clarify, this issue affects only libcurl >= 7.56.0, since older
versions still use the "classic" file upload.

[1] <http://git.php.net/?p=php-src.git;a=commit;h=17a9f1401aeb35fe1e3657b38102a410d151d42f>
[2] <https://bugs.php.net/bug.php?id=79019>
 [2020-01-23 17:10 UTC] sdmarshall73 at gmail dot com
Have you actually tested to see if the files are being received on the remote server? I have an application that uploads files and the files aren't being received on the remote server when using PHP7.4. I wanted to see if this bug is related before filing my own bug report.
 [2020-01-23 18:05 UTC] sdmarshall73 at gmail dot com
My apologies for the previous post ( and this one ). You did mention the files were being removed on the remote side. Seems like the same issue I'm experiencing. I've voted on the bug and I'll just follow this post. Please remove these comments if possible.
 [2020-01-24 13:11 UTC] cmb@php.net
-Status: Assigned +Status: Not a bug
 [2020-01-24 13:11 UTC] cmb@php.net
For me, using current PHP-7.4 the test script outputs:

Linux, curl 7.64.0:

    POST / HTTP/2
    Host: example.com
    Accept: */*
    Transfer-Encoding: chunked
    Content-Type: multipart/form-data; boundary=------------------------df0efab1e65ad7e7

Linux, curl 7.68.0:

    POST / HTTP/2
    Host: example.com
    accept: */*
    content-type: multipart/form-data; boundary=------------------------f55b79edfe19cb82

Windows, curl 7.67.0

    POST / HTTP/2
    Host: example.com
    accept: */*
    content-type: multipart/form-data; boundary=------------------------1e758282c5b98dfb

So obviously a bug in libcurl which has been fixed in the meantime.
 [2020-02-13 18:53 UTC] andrewscaya at yahoo dot ca
I can confirm that when PHP 7.4 reverts to using chunked file uploads, PHP scripts that were working fine on PHP 7.2 or 7.3, no longer post anything to the server.

EXAMPLE SCRIPT
--------------

/* http://example.com:
<?php var_dump($_FILES); ?>
*/

// Create a cURL handle
$ch = curl_init('http://example.com/index.php');

$cfile = curl_file_create('example.txt','text/plain','test_name');

// Assign POST data
$postdata = ['file' => $cfile];
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);

// Execute the handle
echo curl_exec($ch);

ACTUAL RESULTS
--------------

Using cURL 7.68.0 and PHP 7.3
-----------------------------

$_FILES:
/srv/www/index.php:5:
array (size=1)
  'file' =>
    array (size=5)
      'name' => 'test_name' (length=9)
      'type' => 'text/plain' (length=10)
      'tmp_name' => '/tmp/php69gfI5' (length=14)
      'error' => 0
      'size' => 25

POST /index.php HTTP/1.1
Content-Type: multipart/form-data; boundary=------------------------63a96cbe43cf28f8
Content-Length: 212
Accept: */*
Host: example.com

Using cURL 7.68.0 and PHP 7.4
-----------------------------

$_FILES:
/srv/www/index.php:5:
array (size=0)
  empty

POST /index.php HTTP/1.1
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------46680d0956db253f
Transfer-Encoding: chunked
Accept: */*
Host: example.com

EXPECTED RESULTS
----------------

Should be the same.
 [2020-02-14 10:55 UTC] cmb@php.net
-Status: Not a bug +Status: Re-Opened
 [2020-02-28 10:12 UTC] cmb@php.net
The following pull request has been associated:

Patch Name: Fix #79013: Content-Length missing when posting a curlFile with curl
On GitHub:  https://github.com/php/php-src/pull/5219
Patch:      https://github.com/php/php-src/pull/5219.patch
 [2020-03-02 15:10 UTC] carlospauluk at gmail dot com
Is there any alternative to who is running with 7.4.3?
 [2020-03-02 18:01 UTC] cmb@php.net
> Is there any alternative to who is running with 7.4.3?

If you can enforce HTTP/2, that should be sufficient.
 [2020-03-02 18:13 UTC] cmb@php.net
Automatic comment on behalf of cmbecker69@gmx.de
Revision: http://git.php.net/?p=php-src.git;a=commit;h=fc8b3ab7cbb4f5e77584babeaf25b9bf16f524cd
Log: Fix #79013: Content-Length missing when posting a curlFile with curl
 [2020-03-02 18:13 UTC] cmb@php.net
-Status: Re-Opened +Status: Closed
 [2020-03-02 23:40 UTC] christian at klemmer dot io
Debian 10.3 - PHP 7.4.3 - CURL 7.64.0 (7.64.0-4):
-----------------------
POST / HTTP/2
Host: example.com
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=------------------------1698b5ad232372f0

With debian buster and the current CURL-version of debian buster I get Transfer-Encoding: chunked with HTTP/2, which is forbidden:
"HTTP/2 doesn't support HTTP 1.1's chunked transfer encoding mechanism, as it provides its own, more efficient, mechanisms for data streaming." (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding)

Is this caused by PHP or libcurl? You stated earlier that 7.64 might have a bug. Still this is the current libcurl-version of debian stable.
Making a HTTP/2 request with Transfer-Encoding: chunked won't receive the file for some servers.



With libcurl 7.68.0-1 out of debian bullseye (testing) it's working as excepted.

Debian 10.3 - PHP 7.4.3 - CURL 7.68.0 (7.68.0-1):
-----------------------
POST / HTTP/2
Host: example.com
accept: */*
content-type: multipart/form-data; boundary=------------------------399326925cec236c
 [2020-03-03 08:25 UTC] cmb@php.net
> Is this caused by PHP or libcurl? You stated earlier that 7.64
> might have a bug.

Indeed, this looks like a bug in libcurl 7.64.0.

Anyhow, the commit which landed yesterday explicitly passes the
length of the stream to libcurl (if the stream is seekable, and no
filters are attached), so this should even work with libcurl
7.64.0.

You can either test a snapshot, or wait for PHP 7.4.4RC1 which is
scheduled for Thursday.
 
PHP Copyright © 2001-2020 The PHP Group
All rights reserved.
Last updated: Fri Sep 18 17:01:53 2020 UTC