(yyless(1) could just be used before the goto...)

Anyway, did you actually try that? AFAIK it still won't work, at least with your single line example (which there's already been at least one report about). While the local code fix is correct, the re2c code/logic seems flawed to me. (Maybe this bug report can be about that instead, in general, since I didn't get around to sending a follow-up message to the internals@ list yet, explaining things. :-))

In this example, it will still be broken because of the YYFILL() check -- each time it checks if the next character can match, even when it's at the end of the input. YYFILL() then makes it return, completely ignoring anything that has matched up to that point!

I'm not sure if this explanation is 100% correct, but I believe this wrong behavior happens when EOF is encountered while trying to match the variable length part of ANY rule; or something close to that. :-) It's been over a month since I tried to track and figure out what was happening. Granted, most of the cases (unlike yours), where the match is aborted because of YYFILL(), it's with invalid code, but it shouldn't happen. BTW, I think the part with the inline_char_handler label where it looks for opening PHP tags in the HTML, while a good optimization (using memchr() to find < etc.), was actually added as a workaround for this re2c/YYFILL() behavior. I didn't try it, but from what I've observed, I think whatever plain HTML was at the end of a file would have been lost if a regular rule (like in Flex) was used to match it...

Oh, there are also some more bugs in the code that looks for opening PHP tag, but they wouldn't be found as easily as this (and haven't been reported so far). I think I know how it can be fixed nicely, along with some more other scanner optimizations (for inline HTML and comments, basically). But I haven't done anything yet since some of it won't even work with these re2c/YYFILL() issues. :-/

Finally, to simplify what I think is the basic, underlying flaw with the code of re2c and YYFILL() now, here's a super easy example. Say you have one rule:

[a-z]+

It will NEVER match any input that a person would think, such as the string "foo" -- seems pretty messed up to me!?

This should work like in older releases, Marcus please check it!

Duplicated... Bug #45147

Not sure why re2c needs to deal with the #bang situation
looking at the code it would be better to eat that line outside of the lexer..


Something like:

int ini_lex(zval *ini_lval TSRMLS_DC)
{
     if ((YYCTYPE*)yytext == SCNG(yy_start) && *yych == '#') {
         while(*yych != '\n' && *yych != '\n' && yych < yyend) {
            yych++; 
         }
         while((*yych == '\n' || *yych == '\n') && yych < yyend) {
            yych++; 
         }
         YYCURSOR = yych;
     }
.....

This bug has been fixed in CVS.

Snapshots of the sources are packaged every three hours; this change
will be in the next snapshot. You can grab the snapshot at
http://snaps.php.net/.
 
Thank you for the report, and for helping us make PHP better.

This is not fixed, actually (is it OK to change Status back to Open?).

Nuno, I saw your commit yesterday, which didn't seem like it would help (as it wasn't related to what I said above), but wanted to wait until I could check again to make sure I wasn't crazy with my above description. :-)

I just tried the latest Windows snapshot and it's still generating a parse error with the *single line file* (no newline at the end, which .+ won't match, therefore won't trigger the YYFILL() "return 0" thing) descibed in this report (and the CLI example in Bug #44654). Can't be only broken on Windows since everything uses the same generated scanner code...

Something like Alan's scanning loop could be done after just matching #, BUT that's just another workaround for that underlying re2c/YYFILL() problem (also affecting other things). I believe if you use the tokenizer extension, you can see that if the last token of code is matched by a variable length rule, it won't be returned. e.g. my example of a simple rule, [a-z]+ not matching input "foo"

Nuno, fix not correct?

Ok, I think it is really fixed now. I even fixed other related bug.
Please test and let me know if you can still break it :-)