php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #81632 Shuffled parser tokens on PHP 8.1?
Submitted: 2021-11-17 16:11 UTC Modified: 2021-11-18 07:15 UTC
From: ondrej at mirtes dot cz Assigned: cmb (profile)
Status: Closed Package: *General Issues
PHP Version: 8.1Git-2021-11-17 (Git) OS: Ubuntu
Private report: No CVE-ID: None
View Developer Edit
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: ondrej at mirtes dot cz
New email:
PHP Version: OS:

 

 [2021-11-17 16:11 UTC] ondrej at mirtes dot cz 
Description:
------------
I have a really weird bug to report. I have a case of test failure that's happening on PHP 8.1 nightly in GitHub Actions and only on Ubuntu.

I tracked it down to weird value returned from `token_get_all` when parsing this file: https://github.com/phpstan/phpstan-src/blob/master/src/Rules/Debug/DumpTypeRule.php

On other systems and versions, the exported tokens look correct: https://github.com/phpstan/phpstan-src/runs/4240472132?check_suite_focus=true

This is generally what I'd expect:

array (
  0 => 
  array (
    0 => 'T_OPEN_TAG',
    1 => '<?php ',
    2 => 1,
  ),
  1 => 
  array (
    0 => 'T_DECLARE',
    1 => 'declare',
    2 => 1,
  ),
  2 => '(',
  3 => 
  array (
    0 => 'T_STRING',
    1 => 'strict_types',
    2 => 1,
  ),
...



But on PHP 8.1 on Ubuntu, this is how the tokens look like (https://github.com/phpstan/phpstan-src/runs/4240473383?check_suite_focus=true):

array (
  0 => 
  array (
    0 => 'T_END_HEREDOC',
    1 => '<?php ',
    2 => 1,
  ),
  1 => 
  array (
    0 => 'T_ENUM',
    1 => 'declare',
    2 => 1,
  ),
  2 => '(',
  3 => 
  array (
    0 => 'T_RETURN',
    1 => 'strict_types',
    2 => 1,
  ),
...

It's also possible it's a build issue that should be reported to the setup-php action, but I decided to start here to track it down. Thank you.

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2021-11-17 16:21 UTC] cmb@php.net
-Status: Open +Status: Feedback -Assigned To: +Assigned To: cmb
 [2021-11-17 16:21 UTC] cmb@php.net 
Looks good here: <https://3v4l.org/MJA67>.

How do you map the token values to string representations?
 [2021-11-17 16:28 UTC] ondrej at mirtes dot cz 
This is the debug code that outputs the log: https://github.com/phpstan/phpstan-src/blob/ec1ea7c50bcc35d9506e86746a0a2f46171599e7/NetteReflection.php#L307
 [2021-11-17 16:50 UTC] cmb@php.net 
It seems <https://github.com/shivammathur/setup-php/issues/527> is back. <https://github.com/phpstan/phpstan-src/runs/4240473383?check_suite_focus=true#step:3:13> mentions commit c02aa461268843270a6897c20ef3fb4b00d9de8a, but that appears to be on master, not on PHP 8.1.
 [2021-11-17 18:14 UTC] contact at shivammathur dot com 
I have rechecked that correct branches are used.
c02aa461268843270a6897c20ef3fb4b00d9de8a is also in PHP-8.1

$ git branch -a --contains c02aa461268843270a6897c20ef3fb4b00d9de8a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/PHP-8.1
  remotes/origin/master
 [2021-11-17 18:57 UTC] cmb@php.net 
https://bugs.php.net/bug.php?id=81632

Right, c02aa461268843270a6897c20ef3fb4b00d9de8a is on PHP-8.1, but
not on master (it had been merged into master, though).  I have
been fooled by GH UI (I thought that had been fixed some time
ago).

Anyhow, I did a fresh build of that commit, and got the expected
results.  I have no idea what might be wrong.
 [2021-11-17 21:10 UTC] ondrej at mirtes dot cz 
I just tried reproducing it on a simple example and couldn't, it works as expected, even in GitHub Actions: https://github.com/ondrejmirtes/php-bug-81632/actions/runs/1473635299 (the failures on pre-7.4 are because of typed properties)

So to me it looks like some stateful thing where a series of actions corrupts how token_get_all behaves.
 [2021-11-17 21:42 UTC] cmb@php.net
-Status: Feedback +Status: Open
 [2021-11-18 07:15 UTC] ondrej at mirtes dot cz
-Status: Assigned +Status: Closed
 [2021-11-18 07:15 UTC] ondrej at mirtes dot cz 
The problem was in ext-ds: https://github.com/shivammathur/setup-php/issues/527#issuecomment-972558744
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved. 		Last updated: Fri May 09 13:01:28 2025 UTC