php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #81632 Shuffled parser tokens on PHP 8.1?
Submitted: 2021-11-17 16:11 UTC Modified: 2021-11-18 07:15 UTC
From: ondrej at mirtes dot cz Assigned: cmb (profile)
Status: Closed Package: *General Issues
PHP Version: 8.1Git-2021-11-17 (Git) OS: Ubuntu
Private report: No CVE-ID: None
View Developer Edit
 [2021-11-17 16:11 UTC] ondrej at mirtes dot cz 
Description:
------------
I have a really weird bug to report. I have a case of test failure that's happening on PHP 8.1 nightly in GitHub Actions and only on Ubuntu.

I tracked it down to weird value returned from `token_get_all` when parsing this file: https://github.com/phpstan/phpstan-src/blob/master/src/Rules/Debug/DumpTypeRule.php

On other systems and versions, the exported tokens look correct: https://github.com/phpstan/phpstan-src/runs/4240472132?check_suite_focus=true

This is generally what I'd expect:

array (
  0 => 
  array (
    0 => 'T_OPEN_TAG',
    1 => '<?php ',
    2 => 1,
  ),
  1 => 
  array (
    0 => 'T_DECLARE',
    1 => 'declare',
    2 => 1,
  ),
  2 => '(',
  3 => 
  array (
    0 => 'T_STRING',
    1 => 'strict_types',
    2 => 1,
  ),
...



But on PHP 8.1 on Ubuntu, this is how the tokens look like (https://github.com/phpstan/phpstan-src/runs/4240473383?check_suite_focus=true):

array (
  0 => 
  array (
    0 => 'T_END_HEREDOC',
    1 => '<?php ',
    2 => 1,
  ),
  1 => 
  array (
    0 => 'T_ENUM',
    1 => 'declare',
    2 => 1,
  ),
  2 => '(',
  3 => 
  array (
    0 => 'T_RETURN',
    1 => 'strict_types',
    2 => 1,
  ),
...

It's also possible it's a build issue that should be reported to the setup-php action, but I decided to start here to track it down. Thank you.

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2021-11-17 16:21 UTC] cmb@php.net
-Status: Open +Status: Feedback -Assigned To: +Assigned To: cmb
 [2021-11-17 16:21 UTC] cmb@php.net 
Looks good here: <https://3v4l.org/MJA67>.

How do you map the token values to string representations?
 [2021-11-17 16:28 UTC] ondrej at mirtes dot cz 
This is the debug code that outputs the log: https://github.com/phpstan/phpstan-src/blob/ec1ea7c50bcc35d9506e86746a0a2f46171599e7/NetteReflection.php#L307
 [2021-11-17 16:50 UTC] cmb@php.net 
It seems <https://github.com/shivammathur/setup-php/issues/527> is back. <https://github.com/phpstan/phpstan-src/runs/4240473383?check_suite_focus=true#step:3:13> mentions commit c02aa461268843270a6897c20ef3fb4b00d9de8a, but that appears to be on master, not on PHP 8.1.
 [2021-11-17 18:14 UTC] contact at shivammathur dot com 
I have rechecked that correct branches are used.
c02aa461268843270a6897c20ef3fb4b00d9de8a is also in PHP-8.1

$ git branch -a --contains c02aa461268843270a6897c20ef3fb4b00d9de8a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/PHP-8.1
  remotes/origin/master
 [2021-11-17 18:57 UTC] cmb@php.net 
https://bugs.php.net/bug.php?id=81632

Right, c02aa461268843270a6897c20ef3fb4b00d9de8a is on PHP-8.1, but
not on master (it had been merged into master, though).  I have
been fooled by GH UI (I thought that had been fixed some time
ago).

Anyhow, I did a fresh build of that commit, and got the expected
results.  I have no idea what might be wrong.
 [2021-11-17 21:10 UTC] ondrej at mirtes dot cz 
I just tried reproducing it on a simple example and couldn't, it works as expected, even in GitHub Actions: https://github.com/ondrejmirtes/php-bug-81632/actions/runs/1473635299 (the failures on pre-7.4 are because of typed properties)

So to me it looks like some stateful thing where a series of actions corrupts how token_get_all behaves.
 [2021-11-17 21:42 UTC] cmb@php.net
-Status: Feedback +Status: Open
 [2021-11-18 07:15 UTC] ondrej at mirtes dot cz
-Status: Assigned +Status: Closed
 [2021-11-18 07:15 UTC] ondrej at mirtes dot cz 
The problem was in ext-ds: https://github.com/shivammathur/setup-php/issues/527#issuecomment-972558744
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved. 		Last updated: Fri Mar 14 08:01:32 2025 UTC