php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #29761 token_get_all()
Submitted: 2004-08-19 17:48 UTC Modified: 2005-02-03 14:18 UTC
Votes:11
Avg. Score:4.3 ± 0.9
Reproduced:10 of 10 (100.0%)
Same Version:2 (20.0%)
Same OS:3 (30.0%)
From: steemann at globalpark dot de Assigned:
Status: Closed Package: Scripting Engine problem
PHP Version: 5.0.3 OS: Linux 2.4.22
Private report: No CVE-ID: None
View Developer Edit
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: steemann at globalpark dot de
New email:
PHP Version: OS:

 

 [2004-08-19 17:48 UTC] steemann at globalpark dot de 
Description:
------------
Don't know if it's a tokenizer issue or something else: 
 
I just wanted to check out the token_get_all() function. 
I tried the test code from the manual page (see code 
below). I am using token_get_all() with a constant input. 
 
Unfortunately, on my box, the result for the test code is 
varying, i. e. tokening a constant string has two 
different results. I found this out it by simple hitting 
reload in the browser multiple times. I've put both 
results I've gotten in the "Actual result" textarea. The 
possibility for each result seems to be about 50%. 
 
 
Environment: 
Configure Command =>  './configure' 
'--with-apxs=/opt/pdp/apache/bin/apxs' 
'--prefix=/opt/pdp/php' '--with-mysql=/opt/pdp/mysql/' 
'--with-config-file-path=/opt/pdp/' 
'--with-iconv=/opt/pdp/iconv/' 
'--with-imagick=/opt/pdp/ImageMagick/' '--with-zlib' 
'--with-gd' '--with-freetype-dir=/usr' 
'--enable-gd-native-ttf' '--enable-gd-imgstrttf' 
'--with-jpeg-dir=/usr' '--with-png-dir=/usr' '--with-dom' 
'--with-dom-xslt' '--with-dom-exslt' '--enable-sysvsem' 
'--enable-sysvshm' '--disable-debug' '--enable-sockets' 
PHP API => 20020918 
PHP Extension => 20020429 
Zend Extension => 20021010 
Debug Build => no 
Thread Safety => disabled 
 
 
Maybe my box is f**ked up?? 

Reproduce code:
---------------
$data="<?php ob_start(); ?>";
foreach(token_get_all($data) as $c)
{
  if(is_array($c))
  {
    print(token_name($c[0]) . ": '".htmlentities($c[1])."'\n");
  }
  else
  {
    print("$c\n");
  }
}


Expected result:
----------------
T_OPEN_TAG: '&lt;?php ' 
T_STRING: 'ob_start' 
( 
) 
; 
T_WHITESPACE: ' ' 
T_CLOSE_TAG: '?&gt;' 

Actual result:
--------------
In about 50% I get: 
---------------------------- 
< 
? 
T_STRING: 'php' 
T_WHITESPACE: ' ' 
T_STRING: 'ob_start' 
( 
) 
; 
T_WHITESPACE: ' ' 
T_CLOSE_TAG: '?&gt;' 
 
 
 
 
the other 50% I get: 
---------------------------- 
T_OPEN_TAG: '&lt;?php ' 
T_STRING: 'ob_start' 
( 
) 
; 
T_WHITESPACE: ' ' 
T_CLOSE_TAG: '?&gt;'

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2004-10-04 11:11 UTC] guth at fiifo dot u-psud dot fr 
Same problem with PHP 5.0.*, with any PHP script.
 [2004-10-19 21:58 UTC] bloudon at townnews dot com 
Bug is reproducible only when tokenizer is loaded dynamically.
If tokenizer is compiled into PHP then the bug does not occur.

Same behavior on Linux 2.4.24 and Linux 2.6.8.1.
 [2004-12-02 17:31 UTC] fpoeydomenge at free dot fr 
Same problem with PHP 5.0.2, but I only get the version where :

T_OPEN_TAG: '&lt;?php ' 

is replaced by :

< 
? 
T_STRING: 'php'
 [2005-01-13 17:05 UTC] randy at rcs-comp dot com 
This might help...

PHP 5.0.2, Apache 2.0.52

Test code:
----------------------------
<?php
echo '<pre>';
print_r( token_get_all('<?php $a=$b;'));
echo '</pre>';
?> 

Expected result (I think):
----------------------------
Array
(
    [0] => Array
        (
            [0] => 365
            [1] => <?php
        )

    [1] => Array
        (
            [0] => 308
            [1] => $a
        )

    [2] => =
    [3] => Array
        (
            [0] => 308
            [1] => $b
        )

    [4] => ;
) 

Result:
------------------------
Array
(
    [0] => <
    [1] => ?
    [2] => Array
        (
            [0] => 307
            [1] => php
        )

    [3] => Array
        (
            [0] => 369
            [1] => 
        )

    [4] => Array
        (
            [0] => 309
            [1] => $a
        )

    [5] => =
    [6] => Array
        (
            [0] => 309
            [1] => $b
        )

    [7] => ;
) 

Result after restarting Apache:
---------------------------------
Array
(
    [0] => Array
        (
            [0] => 366
            [1] =>  Array
        (
            [0] => 309
            [1] => $a
        )

    [2] => =
    [3] => Array
        (
            [0] => 309
            [1] => $b
        )

    [4] => ;
)

Hope it helps...
 [2005-02-03 04:51 UTC] sniper@php.net 
Please try using this CVS snapshot:

  http://snaps.php.net/php4-STABLE-latest.tar.gz
 
For Windows:
 
  http://snaps.php.net/win32/php4-win32-STABLE-latest.zip
 [2005-02-03 14:10 UTC] steemann at globalpark dot de 
Can not test this with PHP 4.3.8 any more because I moved 
to PHP 5. 
 
With PHP 5.0.3 everything is fine (token_get_all returns 
the same result for every try). The problem seems fixed 
there.
 [2005-02-03 14:18 UTC] derick@php.net 
Considered closed then.
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved. 		Last updated: Fri Oct 31 04:00:01 2025 UTC