php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #78978 Tokenizer tokenizes hex numbers as floats on Windows
Submitted: 2019-12-17 11:17 UTC Modified: 2019-12-17 17:18 UTC
From: bugs dot php dot net_nospam at adviesenzo dot nl Assigned: cmb (profile)
Status: Not a bug Package: Scripting Engine problem
PHP Version: 7.4.0 OS: Windows 7
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: bugs dot php dot net_nospam at adviesenzo dot nl
New email:
PHP Version: OS:

 

 [2019-12-17 11:17 UTC] bugs dot php dot net_nospam at adviesenzo dot nl
Description:
------------
Noticed this while working on something for PHP_CodeSniffer where I wasn't getting the same results for unit tests on Travis versus my local Windows machine.

It appears that hexidecimal integers using the new PHP 7.4 numeric literal separator are tokenized differently on Linux versus Windows.

They should be tokenized as `T_LNUMBER` and are on Linux. See: https://3v4l.org/vlH5j

However on Windows, I'm seeing them get tokenized as `T_DNUMBER`.







Test script:
---------------
$tokens = token_get_all(
	'<?php
$foo = 0xCAFE_F00D;
$foo = 0x42_72_6F_77_6E;'
);

foreach ($tokens as $token) {
    if (is_array($token)) {
        echo "Line {$token[2]}: ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
    }
}

Expected result:
----------------
Line 1: T_OPEN_TAG ('<?php
')
Line 2: T_VARIABLE ('$foo')
Line 2: T_WHITESPACE (' ')
Line 2: T_WHITESPACE (' ')
Line 2: T_LNUMBER ('0xCAFE_F00D')
Line 2: T_WHITESPACE (' ')
Line 3: T_VARIABLE ('$foo')
Line 3: T_WHITESPACE (' ')
Line 3: T_WHITESPACE (' ')
Line 3: T_LNUMBER ('0x42_72_6F_77_6E')

Actual result:
--------------
Line 1: T_OPEN_TAG ('<?php
')
Line 2: T_VARIABLE ('$foo')
Line 2: T_WHITESPACE (' ')
Line 2: T_WHITESPACE (' ')
Line 2: T_DNUMBER ('0xCAFE_F00D')
Line 2: T_WHITESPACE ('
')
Line 3: T_VARIABLE ('$foo')
Line 3: T_WHITESPACE (' ')
Line 3: T_WHITESPACE (' ')
Line 3: T_DNUMBER ('0x42_72_6F_77_6E')

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2019-12-17 11:24 UTC] nikic@php.net
-Package: PHP Language Specification +Package: Scripting Engine problem
 [2019-12-17 15:03 UTC] requinix@php.net
Are you using the x86 or x64 build?
 [2019-12-17 15:42 UTC] cmb@php.net
Thank you for taking the time to write to us, but this is not
a bug. Please double-check the documentation available at
http://www.php.net/manual/ and the instructions on how to report
a bug at http://bugs.php.net/how-to-report.php

This is not related to numeric literal seperators (you get the
same result without them), but rather to integers overflowing to
float (you can var_dump() these values to confirm).
 [2019-12-17 15:42 UTC] cmb@php.net
-Status: Open +Status: Closed -Assigned To: +Assigned To: cmb
 [2019-12-17 17:18 UTC] requinix@php.net
-Status: Closed +Status: Not a bug
 [2019-12-17 18:46 UTC] bugs dot php dot net_nospam at adviesenzo dot nl
Oh cricky... I should have thought of this myself... Let's just say it's one of those days.

Sorry for wasting your time and thank you for your kind response.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Sun May 19 02:01:35 2024 UTC