php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #76437 token_get_all with TOKEN_PARSE flag fails to recognise close tag
Submitted: 2018-06-09 11:26 UTC Modified: 2018-06-17 21:00 UTC
Votes:10
Avg. Score:4.7 ± 0.6
Reproduced:6 of 8 (75.0%)
Same Version:6 (100.0%)
Same OS:2 (33.3%)
From: nathanielzimmermann at gmail dot com Assigned: dmitry (profile)
Status: Closed Package: Scripting Engine problem
PHP Version: 7.3.0alpha1 OS:
Private report: No CVE-ID: None
View Developer Edit
 [2018-06-09 11:26 UTC] nathanielzimmermann at gmail dot com 
Description:
------------
token_get_all with the TOKEN_PARSE flag fails to recognise the T_CLOSE_TAG token

Test script:
---------------
<?php

var_dump(token_get_all('<?php echo 1; ?>', TOKEN_PARSE)[6]);

Expected result:
----------------
array(3) {
  [0]=>
  int(381)
  [1]=>
  string(2) "?>"
  [2]=>
  int(1)
}

Actual result:
--------------
string(2) "?>"

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2018-06-09 13:38 UTC] cmb@php.net
-Status: Open +Status: Verified -Assigned To: +Assigned To: dmitry
 [2018-06-09 13:38 UTC] cmb@php.net 
It seems that this behavioral change is an unintended side effect
of commit 8afb91c[1].  Dmitry, could you have a look at this
issue, please?

[1] <http://git.php.net/?p=php-src.git;a=commit;h=8afb91cdadaa5a40ee972dec278d89c76bf454e3>
 [2018-06-17 14:32 UTC] laruence@php.net 
I am afraid that the special cases should be handled in tokenizer side now :<

diff --git a/ext/tokenizer/tokenizer.c b/ext/tokenizer/tokenizer.c
index ef9d136..9524c69 100644
--- a/ext/tokenizer/tokenizer.c
+++ b/ext/tokenizer/tokenizer.c
@@ -191,8 +191,16 @@ void on_event(zend_php_scanner_event event, int token, int line, void *context)
 
        switch (event) {
                case ON_TOKEN:
-                   if (token == END) break;
-                   add_token(token_stream, token, LANG_SCNG(yy_text), LANG_SCNG(yy_leng), line);
+                 {
+                         if (token == END) break;
+                         /* Specical cases */
+                         if (token == ';' && LANG_SCNG(yy_leng) == 2) {
+                                 token = T_CLOSE_TAG;
+                         } else if (token == T_ECHO && LANG_SCNG(yy_leng) == 3) {
+                                 token = T_OPEN_TAG_WITH_ECHO;
+                         }
+                         add_token(token_stream, token, LANG_SCNG(yy_text), LANG_SCNG(yy_leng), line);
+                 }
                        break;
                case ON_FEEDBACK:
                        tokens_ht = Z_ARRVAL_P(token_stream);
 [2018-06-17 21:00 UTC] cmb@php.net 
Thanks!  A small improvement might be to replace the integer
literals with sizeof("?>")-1 and sizeof("<?=")-1, respectively.
 [2018-06-18 03:35 UTC] laruence@php.net 
Automatic comment on behalf of laruence@gmail.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=4d69bbeee79c6a94348935ad071b6c7c05dd8eae
Log: Fixed bug #76437 (token_get_all with TOKEN_PARSE flag fails to recognise close tag)
 [2018-06-18 03:35 UTC] laruence@php.net
-Status: Verified +Status: Closed
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved. 		Last updated: Fri Oct 31 12:00:01 2025 UTC