I did more testing, and I was a bit wrong about the strlen() part.  Manually padding the keys with '\0' actually works, but the result does not match the ciphertext provided in RFC-2144 B.1 anymore.  Additionally to that I was wrong re: the keysize requirement for that cipher should be 16 bytes (128-bit) as cipher's name 'cast-128' suggests.

Once the keys are properly padded with '\0' to be 128-bit the test returns the following differences:

002+ 80-bit: 753de29f5d167d03
003+ 40-bit: f00b0530833d7444
002- 80-bit: eb6a711a2c02271b
003- 40-bit: 7ac816d16e9b302e

So, something else was also changed that the mcrypt extension no longer conforms to RFC-2144 B.1.

This is a bug in your mcrypt library, which does not properly support Cast-128 with keys <= 80 bits. Padding the keys with NUL bytes is not the same, because  Cast-128 uses only 12 rounds (instead of 16) if the key is <= 80 bits.

Updating your libmcrypt version will probably fix this.

@nikic, I'm running the latest available mcrypt, which is 2.5.8.  The library passed all its internal tests.  Also, PHP 5.5.12 is working on the same box and does not have any issues with mcrypt_encode().

Additionally to this there are other bug reports pointing to the same issue, e.g. bug #67286 , so I would not dismiss this bug this easy.

Finally, the gcov test at php.net itself is failing the test on mcrypt.  The following tests was performed less than 2 days ago: http://gcov.php.net/viewer.php?version=PHP_HEAD&func=tests&file=ext%2Fmcrypt%2Ftests%2Fbug62102_rfc2144.phpt

So, let's try to pinpoint the issue? :)  Or at least let's fix the test case...

The bug you linked has nothing to do with issue, it is a generic complaint from someone using broken crypto code, which is rejected by the stricter input requirements in PHP 5.6.

When you say that mcrypt_encrypt on your PHP 5.5.12 build works correctly, do you mean that it provides the correct output for this test script (whereas PHP 5.6 does not)? This test was only added in PHP 5.6, so just running `make test` will not be enough to check it - you have to manually run the file from the 5.6 tree.

Re: bug #67286 - well, the change to the behaviour as it is now is incompatible with the documentation at http://php.net/manual/en/function.mcrypt-encrypt.php, so there people will report bugs against this behaviour.  Moreover, the change is breaking things -- a warning message is one thing, but the function was changed to bail out with a failure upon a discovery of incorrect key size.  This is the problem, actually.

Re: test case.  Yes, the test case was added on Mar 5th (you committed it 2 days later), but it's currently broken in the HEAD, so I'm curious how one could do proper unit testing there :).

Re: 5.5.12 - no, I didn't run the exact same test, but OctoberCMS I'm running on that box stopped to work after switching to 5.6.1.  By default that application was using the MCRYPT_RIJNDAEL_128 cipher and that cipher was no longer working (the same application, the same mcrypt library, and the only change was PHP).  This is the reason I started to investigate.

Finally, I'm going to run the same test case on my PHP 5.5.12 and will report back shortly.  Still, it's strange to have a broken test for something that important as encryption.

Thank you for pointing out the outdated information in the documentation. I have just pushed some updates to the mcrypt_encrypt and mcrypt_decrypt docs; however it will take a few days to propagate to mirrors.

The test works on my machine (using libmcrypt 2.5.8-3.1 on Ubuntu) and also works on Travis (which is our CI platform). As libmcrypt is currently not maintained, it might be that the Cast-128 fixes have been applied by the distro vendor and may not be present in libmcrypt packages of other distros. There is nothing we (PHP) can do about this.

That your code using Rijndael-128 no longer works is unrelated to this problem (which only applies to Cast-128 with <= 80 bit keys, not to other ciphers like Rijndael). Presumably your code uses incorrect key sizes. If that is the case, you should either modify it to explicitly emulate the old \0 byte padding behavior or - which I would strongly recommend - fix your encryption code to use correctly sized keys.

I've just confirmed that libmcrypt 2.5.8 (compile from sources) has an issue with key sizes <= 80-bits.  I was getting exactly the same encrypted as PHP was reporting in the test.  I'll look into Ubuntu patches, thanks for the pointers!