Description:
------------
Previously, token_get_all would handle even invalid input. This is no longer the case, since some people added unrecoverable errors to the lexer (lex_scan) for unicode escapes and invalid octal literals.

This means that token_get_all can't be used for custom highlighting anymore, because invalid scripts won't produce any output. Interestingly highlight_string is still working, but I couldn't see why.

The commits that probably cause the problem:

commit bae46f307c2d0cdef9b8f5426adcc46920776700
Author: Andrea Faulds <ajf@ajf.me>
Date:   Fri Dec 19 00:40:59 2014 +0000

    Unicode Codepoint Escape Syntax

commit 5f29b980514867f1a09969ca6a1c1f5fb00c3027
Author: Andrea Faulds <ajf@ajf.me>
Date:   Fri Jan 9 06:32:36 2015 +0000

    Error on invalid octal (fixes PHPSadness #31)

commit a8bf1c5d8f5755b53492e58040cfe88150eb57b6
Author: Nikita Popov <nikic@php.net>
Date:   Sat Mar 21 20:10:19 2015 +0100

    Throw ParseException from lexer


Test script:
---------------
<?php
var_dump(token_get_all('<?php 09 "\u{bar}"'));

Expected result:
----------------
Some meaningful tokens such as:
T_OPEN_TAG <?php
T_LNUMBER 09
T_CONSTANT_ENCAPSED_STRING "\u{bar}"

or
T_OPEN_TAG <?php
T_LNUMBER_INVALID 09
T_CONSTANT_ENCAPSED_STRING_INVALID "\u{bar}"

or even
T_OPEN_TAG <?php
T_ERROR 09
T_ERROR "\u{bar}"


Actual result:
--------------
ParseException: Invalid numeric literal
ParseException: Invalid UTF-8 codepoint escape sequence