Assigned to the mbstring maintainer.

mb_detect_encoding does not support the UTF-16/UTF-16BE 
encoding detection. Because UTF-16 isn't byte stream encoding like UTF-8, we cannot detect the encoding as other byte stream encoding.

The file encoded in UTF-16 can be detected easily using BOM, 
it is like,

if ($content[0]==chr(0xff) && $content[1]==chr(0xfe)) {
  echo 'UTF-16';
} else if ($content[0]==chr(0xfe) && $content[1]==chr(0xff)) {
  echo 'UTF-16BE';
}

Of course, comparing the beginning of a file with the UTF-16 BOM can be used to detect UTF-16 encoding. But what do you do with UTF-16 encoded files where no BOM is set?

Bug with correct encoding detection.

function detect_enc($str)
{
	$awe = mb_list_encodings();
	unset($awe[0], $awe[1], $awe[2]);
	foreach ($awe as $enctype)
	{
		if (mb_check_encoding($str, $enctype) === true) return $enctype;
	}
	return false;
}

echo detect_enc('String_encoded_to_Windows-1251'); // Return 'byte2be'. It's a fail.

I came here to report essentially this same bug. In fact I think this bug is directly related to bugs 51563, 64667, 63433, and even 38138.

I have a UTF-16LE encoded CSV file and fgetcsv() was failing on it. So I got to learn all about different character encodings today!

I read on another bug report from one of the PHP devs that the purpose of mb_detect_encoding() is to "detect which multibyte encoding is in use." It's failing at that right now. When I run mb_detect_encoding() on a UTF-16 encoded string, either it says ASCII (if it is not the first line of the file), or it just returns FALSE (if it is the first line, which includes the BOM). On the other hand, if I run mb_check_encoding($str, 'UTF-16') then it seems I get TRUE for all except the first line.

I'm using PHP 5.5.10 by the way.

So there seems to be some regression here between 5.5 and 5.6. I have a unit test for a project. It took a UTF-16 encoded file (with BOM), copied to a tmp dir, then detects encoding and requests mb_convert_encoding($fileContent,'UTF-8') the file. On php 5.5 the file is converted to UTF-8 properly. On 5.6 (on linux and windows) and 7 (linux) it fails. The BOM becomes ?? and then the file is detected as ASCII.

Using

if ($content[0]==chr(0xff) && $content[1]==chr(0xfe)) {
  echo 'UTF-16';
}

does detect it as UTF-16, but I'd like to be able to detect the files that are multi-byte and convert them to UTF-8.

> mb_detect_encoding does not seem to recognize UTF-16 encoded
> files properly.

That is expected behavior, that's already documented[1]:

| For UTF-16, UTF-32, UCS2 and UCS4, encoding detection will fail
| always.

I'm therefore chaning to feature request.

> The file encoded in UTF-16 can be detected easily using BOM,

Albeit not reliably, because 0xFE and 0xFF are valid ISO-8859-*
characters, for instance. Furthermore, a BOM is optional for
UTF-16.

@nathanael at gnat dot ca: that would be a different issue, so
please open a separate ticket.

[1] <http://php.net/manual/en/function.mb-detect-order.php>