|
|
php.net | support | documentation | report a bug | advanced search | search howto | statistics | random bug | login |
[2005-11-19 01:53 UTC] cellog@php.net
Description:
------------
The tokenizer extension is ignoring a final comment when tokenizing a script that does not contain a closing ?>
Reproduce code:
---------------
var_dump(token_get_all("<?php print 'foo'; # you'll see it
print 'bar'; # but not this one"));
Expected result:
----------------
array(13) {
[0]=>
array(2) {
[0]=>
int(366)
[1]=>
string(6) "<?php "
}
[1]=>
array(2) {
[0]=>
int(266)
[1]=>
string(5) "print"
}
[2]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[3]=>
array(2) {
[0]=>
int(315)
[1]=>
string(5) "'foo'"
}
[4]=>
string(1) ";"
[5]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[6]=>
array(2) {
[0]=>
int(364)
[1]=>
string(16) "# you'll see it
"
}
[7]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[8]=>
array(2) {
[0]=>
int(266)
[1]=>
string(5) "print"
}
[9]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[10]=>
array(2) {
[0]=>
int(315)
[1]=>
string(5) "'bar'"
}
[11]=>
string(1) ";"
[12]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[13]=>
array(2) {
[0]=>
int(364)
[1]=>
string(18) "# but not this one"
}
Actual result:
--------------
array(13) {
[0]=>
array(2) {
[0]=>
int(366)
[1]=>
string(6) "<?php "
}
[1]=>
array(2) {
[0]=>
int(266)
[1]=>
string(5) "print"
}
[2]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[3]=>
array(2) {
[0]=>
int(315)
[1]=>
string(5) "'foo'"
}
[4]=>
string(1) ";"
[5]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[6]=>
array(2) {
[0]=>
int(364)
[1]=>
string(16) "# you'll see it
"
}
[7]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[8]=>
array(2) {
[0]=>
int(266)
[1]=>
string(5) "print"
}
[9]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
[10]=>
array(2) {
[0]=>
int(315)
[1]=>
string(5) "'bar'"
}
[11]=>
string(1) ";"
[12]=>
array(2) {
[0]=>
int(369)
[1]=>
string(1) " "
}
}
PatchesPull RequestsHistoryAllCommentsChangesGit/SVN commits
|
|||||||||||||||||||||||||||
Copyright © 2001-2025 The PHP GroupAll rights reserved. |
Last updated: Sat Jul 05 04:01:35 2025 UTC |
Doesn't drop anything here: The code: <?php $arr = token_get_all("<?php print 'foo'; # you'll see it print 'bar'; # but not this one "); foreach ($arr as $token) { if (is_array($token)) var_dump($token[1]); } ?> The output: string(6) "<?php " string(1) " " string(5) "print" string(1) " " string(5) "'foo'" string(2) " " string(16) "# you'll see it " string(5) "print" string(1) " " string(5) "'bar'" string(2) " " string(19) "# but not this one "This patch fixes the issue with no negative side effects Index: zend_language_scanner.l =================================================================== RCS file: /repository/ZendEngine2/zend_language_scanner.l,v retrieving revision 1.131.2.3 diff -u -r1.131.2.3 zend_language_scanner.l --- zend_language_scanner.l 15 Nov 2005 13:29:28 -0000 1.131.2.3 +++ zend_language_scanner.l 19 Nov 2005 03:11:22 -0000 @@ -1465,6 +1465,7 @@ yymore(); } +<ST_ONE_LINE_COMMENT><<EOF>> | <ST_ONE_LINE_COMMENT>{NEWLINE} { zendlval->value.str.val = yytext; /* no copying - intentional */ zendlval->value.str.len = yyleng;better patch - this one does not increment CG(zend_lineno) Index: zend_language_scanner.l =================================================================== RCS file: /repository/ZendEngine2/zend_language_scanner.l,v retrieving revision 1.131.2.3 diff -u -r1.131.2.3 zend_language_scanner.l --- zend_language_scanner.l 15 Nov 2005 13:29:28 -0000 1.131.2.3 +++ zend_language_scanner.l 19 Nov 2005 05:52:01 -0000 @@ -1465,6 +1465,12 @@ yymore(); } +<ST_ONE_LINE_COMMENT><<EOF>> { + zendlval->value.str.val = yytext; /* no copying - intentional */ + zendlval->value.str.len = yyleng; + zendlval->type = IS_STRING; + return T_COMMENT; +} <ST_ONE_LINE_COMMENT>{NEWLINE} { zendlval->value.str.val = yytext; /* no copying - intentional */ zendlval->value.str.len = yyleng;