|  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Request #65323 improvement for counting ill-formed byte sequences
Submitted: 2013-07-24 10:59 UTC Modified: 2013-10-15 11:54 UTC
From: masakielastic at gmail dot com Assigned: cataphract (profile)
Status: No Feedback Package: Strings related
PHP Version: 5.5.1 OS:
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
Block user comment
Status: Assign to:
Bug Type:
From: masakielastic at gmail dot com
New email:
PHP Version: OS:


 [2013-07-24 10:59 UTC] masakielastic at gmail dot com
Consider the number of substitute characters (U+FFFD)
when the range of UTF-8 string of second byte is narrow (such as 0xA0 - 0xBF)

//      Code Points   First Byte Second Byte Third Byte Fourth Byte
//   U+0800 -   U+0FFF   E0         A0 - BF     80 - BF
//   U+D000 -   U+D7FF   ED         80 - 9F     80 - BF
//  U+10000 -  U+3FFFF   F0         90 - BF     80 - BF    80 - BF
// U+100000 - U+10FFFF   F4         80 - 8F     80 - BF    80 - BF

If you follow the recommended policy describled in "Table 3-8. Use of U+FFFD in 
UTF-8 Conversion" of The Unicode Standard,
"\xE0\x80" should be converted to "\xEF\xBF\xBD"."\xEF\xBF\xBD".
The actual result is "\xEF\xBF\xBD".

The one of solution for that purpose is introducing a macro that checks second 
byte by first byte.

Test script:
function str_scrub($str)
    return htmlspecialchars_decode(htmlspecialchars($str, ENT_SUBSTITUTE, 'UTF-8'));

$ufffd_x2 = "\xEF\xBF\xBD"."\xEF\xBF\xBD";
$ufffd_x3 = $ufffd_x2."\xEF\xBF\xBD";

    $ufffd_x2 === str_scrub("\xE0\x80"),
    $ufffd_x3 === str_scrub("\xE0\x80\x80")

Expected result:

Actual result:


Add a Patch

Pull Requests

Add a Pull Request


AllCommentsChangesGit/SVN commitsRelated reports
 [2013-07-24 11:20 UTC] masakielastic at gmail dot com
Table 3-8. Use of U+FFFD in UTF-8 Conversion" of The Unicode Standard
 [2013-07-26 00:43 UTC]
Thank you for the report.
This seems good.

We are also discussing about mb_scrub() as mb_convert_encoding() alias. i.e. 
calling converter internally like mb_convert_encoding().

On master branch mbfl converter fix has been committed.
We appreciate if you could check the current implementation.
 [2013-07-31 22:15 UTC]
-Assigned To: +Assigned To: cataphract
 [2013-07-31 23:00 UTC]
-Status: Assigned +Status: Feedback
 [2013-07-31 23:00 UTC]
Can you test this branch?

I basically rewrote the parser; it was getting too complicated.
 [2013-08-01 07:05 UTC]
Unfortunately, it's also significantly slower... I have to look more closely.
 [2013-10-15 11:54 UTC] php-bugs at lists dot php dot net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Sun Jul 14 02:01:28 2024 UTC