php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #55438 race condition: curlwapper is not sending http header randomly
Submitted: 2011-08-17 11:38 UTC Modified: 2012-12-19 16:57 UTC
Votes:3
Avg. Score:5.0 ± 0.0
Reproduced:3 of 3 (100.0%)
Same Version:0 (0.0%)
Same OS:0 (0.0%)
From: xuefer at gmail dot com Assigned: pierrick (profile)
Status: Closed Package: cURL related
PHP Version: 5.3.6 OS: gentoo
Private report: No CVE-ID: None
 [2011-08-17 11:38 UTC] xuefer at gmail dot com
Description:
------------
background: php is configured with curl wrapper, which make file_get_contents 
use curl. i haven't tested with calling curl functions directly

php unset curl header too soon before curl make the request

expected order: set header, build and send request. unset header
actual order 1: set header, build and send request. unset header (good)
actual order 2: set header, unset header, build and send request (bad)
"send request" comes after php "unset header"
curl behavior randomly, by sending request before or after php unset the header


Test script:
---------------
#!/usr/lib/php5.3/bin/php
<?php
for (;;) {
        $username = 'test1';
        $password = mt_rand(0, 99999) . '....................................................................................';
        $authUrl = "http://localhost/";
        $context = stream_context_create(array(
                                'http' => array(
                                        'header'  => "Authorization: Basic " . base64_encode("$username:$password")
                                        )
                                ));
        $http_response_header = array();
        $data = file_get_contents($authUrl, false, $context);
        sleep(1);
}
?>

tcpdump -nilo dst port 80 -w- -s0

Expected result:
----------------
GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*
Authorization: Basic 
dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4=

GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*
Authorization: Basic 
dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4=


GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*
Authorization: Basic 
dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4=

GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*
Authorization: Basic 
dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4=


Actual result:
--------------
GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*
Authorization: Basic 
dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4=

GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*
Authorization: Basic 
dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4=

GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*


GET / HTTP/1.1
User-Agent: PHP/5.3.6-pl0-gentoo
Host: localhost
Accept: */*


gdb --args /usr/lib/php5.3/bin/php.debug test.php

(gdb) l /usr/src/debug/dev-lang/php-5.3.6/sapis-build/cli/ext/curl/streams.c:367
360                                             slist = curl_slist_append(slist, 
trimmed);
361                                             efree(trimmed);
362                                             p = php_strtok_r(NULL, "\r\n", 
&token);
363                                     }
364                                     efree(copy_ctx_opt);
365                             }
366                             if (slist) {
367                                     curl_easy_setopt(curlstream->curl, 
CURLOPT_HTTPHEADER, slist);
368                             }
369                     }
370                     if (SUCCESS == php_stream_context_get_option(context, 
"http", "method", &ctx_opt) && Z_TYPE_PP(ctx_opt) == IS_STRING) {
371                             if (strcasecmp(Z_STRVAL_PP(ctx_opt), "get")) {
372                                     if (!strcasecmp(Z_STRVAL_PP(ctx_opt), 
"head")) {
373                                             curl_easy_setopt(curlstream-
>curl, CURLOPT_NOBODY, 1);
374                                     } else {
(gdb) l /usr/src/debug/dev-lang/php-5.3.6/sapis-build/cli/ext/curl/streams.c:501
494                     if (msg_found) {
495                             goto exit_fail;
496                     }
497             }
498
499             /* context headers are not needed anymore */
500             if (slist) {
501                     curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, 
NULL);
502                     curl_slist_free_all(slist);
503             }
504             return stream;
505
506     exit_fail:
507             php_stream_close(stream);
508             if (slist) {
(gdb) br 367
Breakpoint 1 at 0x1c59c9: file /usr/src/debug/dev-lang/php-5.3.6/sapis-
build/cli/ext/curl/streams.c, line 367.
(gdb) br 501
Breakpoint 2 at 0x1c61bc: file /usr/src/debug/dev-lang/php-5.3.6/sapis-
build/cli/ext/curl/streams.c, line 501.
(gdb) br Curl_send_plain
Breakpoint 4 at 0xb7698ca0: file sendf.c, line 279.


(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/lib/php5.3/bin/php ./test
[Thread debugging using libthread_db enabled]
warning: Lowest section in /usr/lib/libicudata.so.48 is .hash at 000000f4
[New Thread 0xb55e76f0 (LWP 31857)]
[Switching to Thread 0xb55e76f0 (LWP 31857)]

Breakpoint 1, php_curl_stream_opener (wrapper=0x80aacd44, filename=0x80c45c24 
"http://localhost/vbb/authapi.php",
    mode=0x80a2a01b "rb", options=4, opened_path=0x0, context=0x80c47224)
    at /usr/src/debug/dev-lang/php-5.3.6/sapis-build/cli/ext/curl/streams.c:367
367                                     curl_easy_setopt(curlstream->curl, 
CURLOPT_HTTPHEADER, slist);
(gdb) c
Continuing.

Breakpoint 2, php_curl_stream_opener (wrapper=0x80aacd44, filename=0x80c45c24 
"http://localhost/",
    mode=0x80a2a01b "rb", options=4, opened_path=0x0, context=0x80c47224)
    at /usr/src/debug/dev-lang/php-5.3.6/sapis-build/cli/ext/curl/streams.c:501
501                     curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, 
NULL);
(gdb) c
Continuing.

Breakpoint 4, Curl_send_plain (conn=0x80cd0628, num=0, mem=0x80cd1120, len=82, 
code=0xbff162f4) at sendf.c:279
279     {

(gdb) p ((char*)mem)[0]@len
$1 = "GET /vbb/authapi.php HTTP/1.1\r\nUser-Agent: PHP/5.3.6-pl0-gentoo\r\nHost: 
localhost\r\nAccept: */*\r\n\r\n"

Authorization header is not there

===========================
to reproduce this bug, curl is built with ares and without threads
(in another case http request is not even send, when curl is built with threads 
and without ares, was a curl issue i guess)

# equery uses net-misc/curl
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for net-misc/curl-7.21.4:
 U I
 + + ares        : Enabled c-ares dns support
 + + gnutls      : Prefer gnutls over nss and openssl as the crypto engine
 - - idn         : Enable support for Internationalized Domain Names
 + + ipv6        : Adds support for IP version 6
 + + kerberos    : Adds kerberos support
 - - ldap        : Adds LDAP support (Lightweight Directory Access Protocol)
 - - libssh2     : Enabled SSH urls in curl using libssh2
 - - nss         : Prefer NSS over openssl as the crypto engine
 + + ssl         : Enable crypto engine support (via openssl if USE='-gnutls -
nss')
 - - static-libs : Build static libraries
 - - test        : Workaround to pull in packages needed to run with 
FEATURES=test. Portage-2.1.2 handles this internally,
                   so don't set it in make.conf/package.use anymore
 - - threads     : Adds threads support for various packages. Usually pthreads


Patches

curl-wrapper-header-fix (last revision 2012-12-19 04:24 UTC by phpnet at lostreality dot org)

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2011-08-17 11:45 UTC] xuefer at gmail dot com
sorry for the mismatch http url string. i was trying to remove some string for 
privacy
 [2012-12-18 19:29 UTC] phpnet at lostreality dot org
I think I am seeing this same problem too (On 5.3.10, but nothing has changed in the source in 5.4.9 either).

Can you explain how this is happening, or suggest a work-around? I was digging into the PHP source, expecting that the --with-curlwrappers option was basically broken and incomplete. I was surprised to find the line:
curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, slist);

Because that code all seems to indicate that the headers should be sent, but no matter what I try, nothing I put in headers ever appears in the actual request. I keep running tcpdump but I never see the headers I put in http->header.
 [2012-12-19 04:33 UTC] phpnet at lostreality dot org
I submitted a patch that moves the slist from a local variable in php_curl_stream_opener() into the php_curl_stream struct. The headers are no longer cleared and freed at the end of php_curl_stream_opener(). The code to free the slist is moved into php_curl_stream_close() instead. I'm not sure if this is the best approach, but it clearly gives me a 100% success rate with having headers get sent, where as I had a literal 0% success rate before (not sure if there is really a race condition or not, just that the headers get cleared and the slist freed before they get used.)

The test code I used was as follows (Actual cookie and URL redacted)
<?php
$opt = array('http' => array('method' => 'GET', 'header' => 'Cookie: foo=bar'));
$ctx = stream_context_create($opt);
$f = fopen('http://www.example.com/', 'r', false, $ctx);
fread($f, 1); //work-around curl-wrappers bug where meta_data doesn't exist until the stream is read
$data = stream_get_meta_data($f);
fclose($f);
var_dump($data);
?>

I compiled PHP with the following flags (not that I think anything matters to this bug other than --with-curlwrappers):
--enable-static --with-mcrypt --with-ldap --with-iconv --enable-mbstring --with-gd --enable-mbregex --with-zlib --with-imap --enable-ftp --with-gettext --enable-sockets --with-mysql=/usr --enable-cgi --with-imap-ssl --enable-sockets --with-pdo-mysql --with-openssl --with-kerberos --with-curl --with-curlwrappers --with-tidy --with-pcre-regex --with-bz2 --enable-zip --with-libdir=/lib64
 [2012-12-19 05:50 UTC] pierrick@php.net
-Assigned To: +Assigned To: pierrick
 [2012-12-19 05:50 UTC] pierrick@php.net
I tried to reproduce this bug but wasn't able to do it.

Could you give me more details on the libcurl version used by your PHP instance ? 
And also, how do you make sure that the headers are not properly sent ?
 [2012-12-19 06:01 UTC] phpnet at lostreality dot org
I have curl-7.15.5-15.el5 according to rpm -q, but I can only locate /usr/lib/libcurl.so.3.0.0 and /usr/lib64/libcurl.so.3.0.0 on my machine I'm testing on (CentOS 5.8). The binary says: /usr/bin/curl -V
curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

Normally, I use the stock RPMs for PHP, but a recent project I was working on failed to run properly on another machine, where the owners also use CentOS 5.8, but use CPanel/WHM instead of the CentOS RPMs for PHP. CPanel uses the --with-curlwrappers option, where as the stock CentOS and RHEL RPMs have never used that option on any of their builds. It took a lot of digging before I realized that it was the --with-curlwrappers option that caused the scripts to fail on that machine while working perfectly on mine.

To verify if the headers were actually sent, I used: tcpdump -i eth1 -Als0 host www.example.com
I had two PuTTY windows open, one with tcpdump, the other running the test script I mentioned before with: ./php-5.4.9/sapi/cli/php ./test.php

It was pretty clear to me that the headers were never sent before the patch, and always sent after the patch.
 [2012-12-19 07:51 UTC] pierrick@php.net
Ok, I finally reproduced the problem.

I was trying the code snippet on my local network and everything was fine, once 
I modified the code to fetch an URL on a slower network I had the problem. 

Since curl multi is used, it sometime happen that the resource is freed before 
the curl multi really execute the query. The patch looks good, I'll have a 
second look tomorrow and will commit it.

Thanks for your help on this one :)
 [2012-12-19 15:17 UTC] phpnet at lostreality dot org
Great. Any chance this can make it to 5.3.20 also?
 [2012-12-19 16:57 UTC] pierrick@php.net
Unfortunately, it's to late for 5.3.20 and 5.4.10, sorry.
It will only be available in 5.3.21 and 5.4.11
 [2012-12-19 17:15 UTC] phpnet at lostreality dot org
Ok, thanks for the info. I was mostly wondering if it would make 5.3.x at all. Well now it seems I just have to wait for 5.3.21 to be released, and CPanel to upgrade, and so on :)
 [2012-12-20 00:57 UTC] pierrick@php.net
Automatic comment on behalf of pierrick
Revision: http://git.php.net/?p=php-src.git;a=commit;h=c46e1cdcae70254cfc0b7d5781f2c71162a3734d
Log: Fixed bug #55438 (Curlwapper is not sending http header randomly)
 [2012-12-20 00:57 UTC] pierrick@php.net
-Status: Assigned +Status: Closed
 [2013-01-12 16:39 UTC] derick@php.net
Automatic comment on behalf of pierrick
Revision: http://git.php.net/?p=php-src.git;a=commit;h=c46e1cdcae70254cfc0b7d5781f2c71162a3734d
Log: Fixed bug #55438 (Curlwapper is not sending http header randomly)
 [2014-10-07 23:20 UTC] stas@php.net
Automatic comment on behalf of pierrick
Revision: http://git.php.net/?p=php-src-security.git;a=commit;h=c46e1cdcae70254cfc0b7d5781f2c71162a3734d
Log: Fixed bug #55438 (Curlwapper is not sending http header randomly)
 [2014-10-07 23:31 UTC] stas@php.net
Automatic comment on behalf of pierrick
Revision: http://git.php.net/?p=php-src-security.git;a=commit;h=c46e1cdcae70254cfc0b7d5781f2c71162a3734d
Log: Fixed bug #55438 (Curlwapper is not sending http header randomly)
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Tue Mar 19 04:01:31 2024 UTC