php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Doc Bug #71207 token_get_all does not recognize context-sensitive keywords
Submitted: 2015-12-23 22:33 UTC Modified: 2015-12-24 10:24 UTC
From: kontakt at kukulich dot cz Assigned: tpunt (profile)
Status: Closed Package: Unknown/Other Function
PHP Version: 7.0.1 OS:
Private report: No CVE-ID: None
 [2015-12-23 22:33 UTC] kontakt at kukulich dot cz
Description:
------------
Thanks to https://wiki.php.net/rfc/context_sensitive_lexer it's possible in PHP7 to use a lot of keywords as constants, methods etc.

However token_get_all doesn't return the right tokens.

Test script:
---------------
<?php

$tokens = token_get_all('<?php class Test {const PUBLIC=0;}');

foreach ($tokens as $token) {
	if (is_array($token)) {
		echo token_name($token[0]) . "\n";
	}
}


Expected result:
----------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_STRING
T_LNUMBER

Actual result:
--------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_PUBLIC
T_LNUMBER

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2015-12-23 22:37 UTC] nikic@php.net
-Summary: token_get_all doesn't work correctly +Summary: token_get_all does not recognize context-sensitive keywords -Status: Open +Status: Not a bug
 [2015-12-23 22:37 UTC] nikic@php.net
If you want these to be recognized, you must pass the TOKEN_PARSE flag as the second argument to token_get_all().
 [2015-12-23 22:38 UTC] nikic@php.net
-Status: Not a bug +Status: Open -Type: Bug +Type: Documentation Problem
 [2015-12-23 22:38 UTC] nikic@php.net
Switching to doc bug as the new parameter seems to be undocumented.
 [2015-12-24 10:17 UTC] tpunt@php.net
-Assigned To: +Assigned To: tpunt
 [2015-12-24 10:23 UTC] tpunt@php.net
Automatic comment from SVN on behalf of tpunt
Revision: http://svn.php.net/viewvc/?view=revision&amp;revision=338345
Log: Resolve doc bug #71207
 [2015-12-24 10:24 UTC] tpunt@php.net
-Status: Assigned +Status: Closed
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Apr 19 09:01:27 2024 UTC