php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Doc Bug #71207 token_get_all does not recognize context-sensitive keywords
Submitted: 2015-12-23 22:33 UTC Modified: 2015-12-24 10:24 UTC
From: kontakt at kukulich dot cz Assigned: tpunt (profile)
Status: Closed Package: Unknown/Other Function
PHP Version: 7.0.1 OS:
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: kontakt at kukulich dot cz
New email:
PHP Version: OS:

 

 [2015-12-23 22:33 UTC] kontakt at kukulich dot cz
Description:
------------
Thanks to https://wiki.php.net/rfc/context_sensitive_lexer it's possible in PHP7 to use a lot of keywords as constants, methods etc.

However token_get_all doesn't return the right tokens.

Test script:
---------------
<?php

$tokens = token_get_all('<?php class Test {const PUBLIC=0;}');

foreach ($tokens as $token) {
	if (is_array($token)) {
		echo token_name($token[0]) . "\n";
	}
}


Expected result:
----------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_STRING
T_LNUMBER

Actual result:
--------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_PUBLIC
T_LNUMBER

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2015-12-23 22:37 UTC] nikic@php.net
-Summary: token_get_all doesn't work correctly +Summary: token_get_all does not recognize context-sensitive keywords -Status: Open +Status: Not a bug
 [2015-12-23 22:37 UTC] nikic@php.net
If you want these to be recognized, you must pass the TOKEN_PARSE flag as the second argument to token_get_all().
 [2015-12-23 22:38 UTC] nikic@php.net
-Status: Not a bug +Status: Open -Type: Bug +Type: Documentation Problem
 [2015-12-23 22:38 UTC] nikic@php.net
Switching to doc bug as the new parameter seems to be undocumented.
 [2015-12-24 10:17 UTC] tpunt@php.net
-Assigned To: +Assigned To: tpunt
 [2015-12-24 10:23 UTC] tpunt@php.net
Automatic comment from SVN on behalf of tpunt
Revision: http://svn.php.net/viewvc/?view=revision&amp;revision=338345
Log: Resolve doc bug #71207
 [2015-12-24 10:24 UTC] tpunt@php.net
-Status: Assigned +Status: Closed
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved.
Last updated: Tue Jul 01 13:01:35 2025 UTC