It's currently possible to get a partial output using JSON_PARTIAL_OUTPUT_ON_ERROR. This will replace invalid UTF8 strings with NULL though. It probably would make sense to have an alternative option that inserts the substitution character.

Here is a proposal fo this issue
https://github.com/remicollet/pecl-json-c/commit/5a499a4550d1f29f1f8eeb1b4ca0b01a33c64779

This add 2 new options to json_encode

- JSON_NOTUTF8_SUBSTITUTE (name seems better, at least to me), to replace not-utf8 char with the replacement char.

- JSON_NOTUTF8_IGNORE to ignore not-utf8 char (remove in escaped mode, keep without any check in unescaped mode)

Hi, thanks nikic and remi.

After several considering, I changed my mind.
I think the behavior of substituting U+FFFD 
for ill-formed sequences should be default.

How do you think?

We might need the discussion about the consitency for Escaper API. 
htmlspecialchars's ENT_SUBSTITUTE option is adopted 
by Symfony and Zend Framework.

https://wiki.php.net/rfc/escaper

Although the behavior breaks 2 test suites, it don't break user's codebases.

A lot of people don't use any option looking in github.

https://github.com/search?l=PHP&q=json_encode&ref=advsearch&type=Code
https://github.com/search?l=PHP&q=json_decode&ref=advsearch&type=Code

The same problem can be seen in htmlspecialchars.

https://github.com/search?l=PHP&q=htmlspecialchars&ref=advsearch&type=Code

New options complicate the situation 
when using JSON_UNESCAPED_UNICODE option and json_decode.

[two option]
json_encode
  JSON_NOTUTF8_SUBSTITUTE
  JSON_NOTUTF8_IGNORE
  JSON_UNESCAPED_UNICODE | JSON_NOTUTF8_SUBSTITUTE
  JSON_UNESCAPED_UNICODE | JSON_NOTUTF8_IGNORE

json_decode
  JSON_NOTUTF8_SUBSTITUTE
  JSON_NOTUTF8_IGNORE


If JSON_NOTUTF8_SUBSTITUTE is default behavior, 
the problem we need to consider is only JSON_NOTUTF8_IGNORE option.

[one option]
json_encode
  JSON_NOTUTF8_IGNORE
  JSON_UNESCAPED_UNICODE | JSON_NOTUTF8_IGNORE

json_decode
  JSON_NOTUTF8_IGNORE

I don't think changing the current behavior is a good idea, the reason why I really prefer some new options.

Hi remi, could you test my patch for PHP_JSON_UNESCAPED_UNICODE option?
The patch adopts JSON_NOTUTF8_SUBSTITUTE and JSON_NOTUTF8_IGNORE options.

https://gist.github.com/masakielastic/5973095

Hi, I fixed my patch and added test case for json_decode.

I posted a patch for handling surrogate pairs 
since the range (U+D800 - U+DFFF) is not allowed in UTF-8 (RFC 3629).
Someone's help is needed for handling high surrogate pairs and the options.

https://gist.github.com/masakielastic/5985383

json_decode produces invalid byte-sequences
https://bugs.php.net/bug.php?id=62010

I created new feature request for preveting XSS attack and I withdraw my option 
about the change of default behavior.

new function for preventing XSS attack
https://bugs.php.net/bug.php?id=65257

Hi, nikic, I posted a document request for the mission option and error codes.

https://bugs.php.net/bug.php?id=65259

Your opinion about the consistency among 
JSON_PARTIAL_OUTPUT_ON_ERROR and JSON_NOTUTF8_SUBSTITUTE 
and JSON_NOTUTF8_IGNORE is needed.

I nominate other names from the view of consistency with JSON_ERROR_UTF8.

JSON_UTF8_SUBSTITUTE
JSON_UTF8_IGNORE

Hi, nikic, sorry, ignore my last comment.

I added small change in json.c
https://gist.github.com/masakielastic/5973095#file-02-small_refactaring-patch

As for JSON_NOTUTF8_IGNORE, the description for security is needed in the manual 
like htmlspecialchars's ENT_IGNORE

http://www.php.net/manual/en/function.htmlspecialchars.php

That's why I didn't sugguest JSON_IGNORE in the draft and showed Escaping RFC's 
link
as resource.

UNICODE SECURITY CONSIDERATIONS
http://www.unicode.org/reports/tr36/#Deletion_of_Noncharacters
IDS11-J. Eliminate noncharacter code points before validation
https://www.securecoding.cert.org/confluence/display/java/IDS11-
J.+Eliminate+noncharacter+code+points+before+validation

> Hi remi, could you test my patch for PHP_JSON_UNESCAPED_UNICODE option?
> The patch adopts JSON_NOTUTF8_SUBSTITUTE and JSON_NOTUTF8_IGNORE options.

The PHP_JSON_UNESCAPED_UNICODE + JSON_NOTUTF8_IGNORE already works with my patch.

Yes, PHP_JSON_UNESCAPED_UNICODE + JSON_NOTUTF8_SUBSTITUTE doesn't work for now, but converting to utf16, then back to utf8 seems really... messy. Need something simpler.

Notice: this bug is only for json_encode. Other issue have their own bug for tracking (especially the json_decode one, as I dont plan to alter it)

I agree with you on isolated surrogate pairs.

The test cases for json_decode and JSON_NOTUTF8_SUBSTITUTE and 
JSON_NOTUTF8_IGNORE must be contained 
since json_decode uses json_utf8_to_utf16. 

https://github.com/php/php-src/blob/master/ext/json/json.c#L673

I already posted the test cases.

https://gist.github.com/masakielastic/5973095#file-04-test-php-L26

"a\xEF\xBF\xBD" === json_decode('"'."a\x80".'"', false, 512, 
JSON_NOTUTF8_SUBSTITUTE),
"a" === json_decode('"'."a\x80".'"', false, 512, JSON_NOTUTF8_IGNORE)


The one way of perfomance improvement is adding json_utf8_to_utf32. 
I posted  another patch.

https://gist.github.com/masakielastic/5973095#file-02-json_unescaped_unicode-
patch

I created unsigned int *utf32 data type 
for not changing unsigned short *utf16 data type.

If you want to provide a common variable  
for json_utf8_to_utf16 and json_utf8_to_utf32, 
the modification for JSON_parser.c is also needed.

The one of candidate for the name of variable is 
unsigned int *code_codes.

http://www.unicode.org/glossary/#code_unit


I also updated the previous patch.
https://gist.github.com/masakielastic/5973095#file-01-json_unescaped_unicode-
patch

if (options & PHP_JSON_UNESCAPED_UNICODE) {
+    if (us < 0x20) {
+        smart_str_appendl(buf, "\\u", 2);
+        smart_str_appendc(buf, digits[(us >> 12) & 0xf]);
+        smart_str_appendc(buf, digits[(us >> 8) & 0xf]);
+        smart_str_appendc(buf, digits[(us >> 4) & 0xf]);
+        smart_str_appendc(buf, digits[(us & 0xf)]);
+    } else if (us < 0x80) {

Another way of perfomance improvemnet is using php_next_utf8_char directly 
in json_escape_string on the condition of PHP_JSON_NOTUTF8_SUBSTITUTE 
and PHP_JSON_NOTUTF8_IGNORE. 
This way reduces one loop compared with using json_utf8_to_utf16.

I created a repo for the patches and the report of benchmarks

https://github.com/masakielastic/patches/tree/master/php_bugs_65082

The difference between json_utf8_to_utf16 and json_utf8_to_utf32 isn't seen.

the use of json_utf8_to_utf32 or the direct use of php_next_utf8_char 
in json_escape_string is better choice for 
JSON_NOTUTF8_SUBSTITUTE and JSON_NOTUTF8_SUBSTITUTE|JSON_UNESCAPED_UNICODE.

php_next_utf8_char in json_escape_string is a bit faster than
json_utf8_to_utf32 for JSON_NOTUTF8_SUBSTITUTE.

https://github.com/masakielastic/patches/blob/master/php_bugs_65082/04_php_next_
utf8_char_in_json_escape_string.patch
https://github.com/masakielastic/patches/blob/master/php_bugs_65082/04_php_next_
utf8_char_in_json_escape_string.c

I'm working on it and the first part (json_encode) can be seen in here:

https://github.com/bukka/php-src/commit/0a5fe6e0dc709985752ac82794f0f45de0ead26f

This has been addressed and will be part of PHP 7.2