php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #29761 token_get_all()
Submitted: 2004-08-19 17:48 UTC Modified: 2005-02-03 14:18 UTC
Votes:11
Avg. Score:4.3 ± 0.9
Reproduced:10 of 10 (100.0%)
Same Version:2 (20.0%)
Same OS:3 (30.0%)
From: steemann at globalpark dot de Assigned:
Status: Closed Package: Scripting Engine problem
PHP Version: 5.0.3 OS: Linux 2.4.22
Private report: No CVE-ID:
 [2004-08-19 17:48 UTC] steemann at globalpark dot de
Description:
------------
Don't know if it's a tokenizer issue or something else: 
 
I just wanted to check out the token_get_all() function. 
I tried the test code from the manual page (see code 
below). I am using token_get_all() with a constant input. 
 
Unfortunately, on my box, the result for the test code is 
varying, i. e. tokening a constant string has two 
different results. I found this out it by simple hitting 
reload in the browser multiple times. I've put both 
results I've gotten in the "Actual result" textarea. The 
possibility for each result seems to be about 50%. 
 
 
Environment: 
Configure Command =>  './configure' 
'--with-apxs=/opt/pdp/apache/bin/apxs' 
'--prefix=/opt/pdp/php' '--with-mysql=/opt/pdp/mysql/' 
'--with-config-file-path=/opt/pdp/' 
'--with-iconv=/opt/pdp/iconv/' 
'--with-imagick=/opt/pdp/ImageMagick/' '--with-zlib' 
'--with-gd' '--with-freetype-dir=/usr' 
'--enable-gd-native-ttf' '--enable-gd-imgstrttf' 
'--with-jpeg-dir=/usr' '--with-png-dir=/usr' '--with-dom' 
'--with-dom-xslt' '--with-dom-exslt' '--enable-sysvsem' 
'--enable-sysvshm' '--disable-debug' '--enable-sockets' 
PHP API => 20020918 
PHP Extension => 20020429 
Zend Extension => 20021010 
Debug Build => no 
Thread Safety => disabled 
 
 
Maybe my box is f**ked up?? 

Reproduce code:
---------------
$data="<?php ob_start(); ?>";
foreach(token_get_all($data) as $c)
{
  if(is_array($c))
  {
    print(token_name($c[0]) . ": '".htmlentities($c[1])."'\n");
  }
  else
  {
    print("$c\n");
  }
}


Expected result:
----------------
T_OPEN_TAG: '&lt;?php ' 
T_STRING: 'ob_start' 
( 
) 
; 
T_WHITESPACE: ' ' 
T_CLOSE_TAG: '?&gt;' 

Actual result:
--------------
In about 50% I get: 
---------------------------- 
< 
? 
T_STRING: 'php' 
T_WHITESPACE: ' ' 
T_STRING: 'ob_start' 
( 
) 
; 
T_WHITESPACE: ' ' 
T_CLOSE_TAG: '?&gt;' 
 
 
 
 
the other 50% I get: 
---------------------------- 
T_OPEN_TAG: '&lt;?php ' 
T_STRING: 'ob_start' 
( 
) 
; 
T_WHITESPACE: ' ' 
T_CLOSE_TAG: '?&gt;' 
 

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2004-10-04 11:11 UTC] guth at fiifo dot u-psud dot fr
Same problem with PHP 5.0.*, with any PHP script.
 [2004-10-19 21:58 UTC] bloudon at townnews dot com
Bug is reproducible only when tokenizer is loaded dynamically.
If tokenizer is compiled into PHP then the bug does not occur.

Same behavior on Linux 2.4.24 and Linux 2.6.8.1.
 [2004-12-02 17:31 UTC] fpoeydomenge at free dot fr
Same problem with PHP 5.0.2, but I only get the version where :

T_OPEN_TAG: '&lt;?php ' 

is replaced by :

< 
? 
T_STRING: 'php'
 [2005-01-13 17:05 UTC] randy at rcs-comp dot com
This might help...

PHP 5.0.2, Apache 2.0.52

Test code:
----------------------------
<?php
echo '<pre>';
print_r( token_get_all('<?php $a=$b;'));
echo '</pre>';
?> 

Expected result (I think):
----------------------------
Array
(
    [0] => Array
        (
            [0] => 365
            [1] => <?php
        )

    [1] => Array
        (
            [0] => 308
            [1] => $a
        )

    [2] => =
    [3] => Array
        (
            [0] => 308
            [1] => $b
        )

    [4] => ;
) 

Result:
------------------------
Array
(
    [0] => <
    [1] => ?
    [2] => Array
        (
            [0] => 307
            [1] => php
        )

    [3] => Array
        (
            [0] => 369
            [1] => 
        )

    [4] => Array
        (
            [0] => 309
            [1] => $a
        )

    [5] => =
    [6] => Array
        (
            [0] => 309
            [1] => $b
        )

    [7] => ;
) 

Result after restarting Apache:
---------------------------------
Array
(
    [0] => Array
        (
            [0] => 366
            [1] =>  Array
        (
            [0] => 309
            [1] => $a
        )

    [2] => =
    [3] => Array
        (
            [0] => 309
            [1] => $b
        )

    [4] => ;
)

Hope it helps...
 [2005-02-03 14:10 UTC] steemann at globalpark dot de
Can not test this with PHP 4.3.8 any more because I moved 
to PHP 5. 
 
With PHP 5.0.3 everything is fine (token_get_all returns 
the same result for every try). The problem seems fixed 
there.
 [2005-02-03 14:18 UTC] derick@php.net
Considered closed then.
 
PHP Copyright © 2001-2014 The PHP Group
All rights reserved.
Last updated: Fri Apr 18 03:02:48 2014 UTC