Description:
------------
I'm unable to reproduce it in a small snippet, the problem occurred in a huge code base at "random" places trough out the application. However after setting "zend.enable_gc = Off" the problem disappeared.

The problem varied, but were all related to memory_issues of massive allocations. Some up to 3GB:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 3061525857 bytes) in [...]


Sometimes the errors were more informative:

PHP Fatal error:  Allowed memory size of 209715200 bytes exhausted at /usr/src/php-5.3.2/Zend/zend_operators.c:1243 (tried to allocate 1515870818 bytes) in [..]

And all errors were string related. They all happened on lines where some sort of concatenation was going on, like:

$string = $value. $string ;

or with functions like str_replace():

$string = str_replace("[".$field."]", $value, $string);


* APC was not installed on the machine.
* We tried both mysqlnd and libmysql, since initially that was the biggest difference between the development and production servers.
* PHP was source build on a debian server, the following parameter were used: ./configure --prefix=/usr/local --with-apxs2=/usr/local/apache2/bin/apxs --enable-soap --with-freetype-dir=/usr --enable-gd-native-ttf --with-gd --with-zlib --with-bz2 --enable-zip --with-mcrypt --enable-ftp --with-openssl --with-curl --enable-mbstring --with-mysql=mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --with-ldap --with-pdo-mysql --with-sqlite --with-tidy --enable-calendar


Actual result:
--------------
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/local/apache2/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/local/apache2/lib/libaprutil-1.so.0
Reading symbols from /usr/local/apache2/lib/libexpat.so.0...done.
Loaded symbols for /usr/local/apache2/lib/libexpat.so.0
Reading symbols from /usr/local/apache2/lib/libapr-1.so.0...done.
Loaded symbols for /usr/local/apache2/lib/libapr-1.so.0
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/local/apache2/modules/mod_authn_file.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authn_file.so
Reading symbols from /usr/local/apache2/modules/mod_authn_dbm.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authn_dbm.so
Reading symbols from /usr/local/apache2/modules/mod_authn_anon.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authn_anon.so
Reading symbols from /usr/local/apache2/modules/mod_authn_dbd.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authn_dbd.so
Reading symbols from /usr/local/apache2/modules/mod_authn_default.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authn_default.so
Reading symbols from /usr/local/apache2/modules/mod_authz_host.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authz_host.so
Reading symbols from /usr/local/apache2/modules/mod_authz_groupfile.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authz_groupfile.so
Reading symbols from /usr/local/apache2/modules/mod_authz_user.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authz_user.so
Reading symbols from /usr/local/apache2/modules/mod_authz_dbm.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authz_dbm.so
Reading symbols from /usr/local/apache2/modules/mod_authz_owner.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authz_owner.so
Reading symbols from /usr/local/apache2/modules/mod_authz_default.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_authz_default.so
Reading symbols from /usr/local/apache2/modules/mod_auth_basic.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_auth_basic.so
Reading symbols from /usr/local/apache2/modules/mod_auth_digest.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_auth_digest.so
Reading symbols from /usr/local/apache2/modules/mod_dbd.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_dbd.so
Reading symbols from /usr/local/apache2/modules/mod_dumpio.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_dumpio.so
Reading symbols from /usr/local/apache2/modules/mod_reqtimeout.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_reqtimeout.so
Reading symbols from /usr/local/apache2/modules/mod_ext_filter.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_ext_filter.so
Reading symbols from /usr/local/apache2/modules/mod_include.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_include.so
Reading symbols from /usr/local/apache2/modules/mod_filter.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_filter.so
Reading symbols from /usr/local/apache2/modules/mod_substitute.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_substitute.so
Reading symbols from /usr/local/apache2/modules/mod_deflate.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_deflate.so
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/local/apache2/modules/mod_log_config.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_log_config.so
Reading symbols from /usr/local/apache2/modules/mod_log_forensic.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_log_forensic.so
Reading symbols from /usr/local/apache2/modules/mod_logio.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_logio.so
Reading symbols from /usr/local/apache2/modules/mod_env.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_env.so
Reading symbols from /usr/local/apache2/modules/mod_mime_magic.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_mime_magic.so
Reading symbols from /usr/local/apache2/modules/mod_cern_meta.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_cern_meta.so
Reading symbols from /usr/local/apache2/modules/mod_expires.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_expires.so
Reading symbols from /usr/local/apache2/modules/mod_headers.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_headers.so
Reading symbols from /usr/local/apache2/modules/mod_ident.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_ident.so
Reading symbols from /usr/local/apache2/modules/mod_usertrack.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_usertrack.so
Reading symbols from /usr/local/apache2/modules/mod_unique_id.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_unique_id.so
Reading symbols from /usr/local/apache2/modules/mod_setenvif.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_setenvif.so
Reading symbols from /usr/local/apache2/modules/mod_version.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_version.so
Reading symbols from /usr/local/apache2/modules/mod_ssl.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_ssl.so
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
Reading symbols from /usr/local/apache2/modules/mod_mime.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_mime.so
Reading symbols from /usr/local/apache2/modules/mod_dav.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_dav.so
Reading symbols from /usr/local/apache2/modules/mod_status.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_status.so
Reading symbols from /usr/local/apache2/modules/mod_autoindex.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_autoindex.so
Reading symbols from /usr/local/apache2/modules/mod_asis.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_asis.so
Reading symbols from /usr/local/apache2/modules/mod_info.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_info.so
Reading symbols from /usr/local/apache2/modules/mod_cgi.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_cgi.so
Reading symbols from /usr/local/apache2/modules/mod_dav_fs.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_dav_fs.so
Reading symbols from /usr/local/apache2/modules/mod_vhost_alias.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_vhost_alias.so
Reading symbols from /usr/local/apache2/modules/mod_negotiation.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_negotiation.so
Reading symbols from /usr/local/apache2/modules/mod_dir.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_dir.so
Reading symbols from /usr/local/apache2/modules/mod_imagemap.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_imagemap.so
Reading symbols from /usr/local/apache2/modules/mod_actions.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_actions.so
Reading symbols from /usr/local/apache2/modules/mod_speling.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_speling.so
Reading symbols from /usr/local/apache2/modules/mod_userdir.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_userdir.so
Reading symbols from /usr/local/apache2/modules/mod_alias.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_alias.so
Reading symbols from /usr/local/apache2/modules/mod_rewrite.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_rewrite.so
Reading symbols from /usr/local/apache2/modules/libphp5.so...done.
Loaded symbols for /usr/local/apache2/modules/libphp5.so
Reading symbols from /usr/lib/libtidy-0.99.so.0...done.
Loaded symbols for /usr/lib/libtidy-0.99.so.0
Reading symbols from /usr/lib/libmcrypt.so.4...done.
Loaded symbols for /usr/lib/libmcrypt.so.4
Reading symbols from /usr/lib/libltdl.so.3...done.
Loaded symbols for /usr/lib/libltdl.so.3
Reading symbols from /usr/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libfreetype.so.6...done.
Loaded symbols for /usr/lib/libfreetype.so.6
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /lib/libbz2.so.1.0...done.
Loaded symbols for /lib/libbz2.so.1.0
Reading symbols from /usr/lib/libcurl.so.4...done.
Loaded symbols for /usr/lib/libcurl.so.4
Reading symbols from /usr/lib/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient.so.15
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/libssh2.so.1...done.
Loaded symbols for /usr/lib/libssh2.so.1
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libtasn1.so.3...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/libgcrypt.so.11...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
[New process 19268]
#0  0xb753ebef in gc_remove_zval_from_buffer (zv=0x892d308) at /usr/src/php-5.3.2/Zend/zend_gc.h:189
189		root->next->prev = root->prev;
warning: not using untrusted file "/usr/src/php-5.3.2/.gdbinit"
(gdb) bt
#0  0xb753ebef in gc_remove_zval_from_buffer (zv=0x892d308) at /usr/src/php-5.3.2/Zend/zend_gc.h:189
#1  0xb750d464 in _zval_ptr_dtor (zval_ptr=0x904fc1c, __zend_filename=0xb7a56fe0 "/usr/src/php-5.3.2/Zend/zend_variables.c", __zend_lineno=178)
    at /usr/src/php-5.3.2/Zend/zend_execute_API.c:438
#2  0xb751b810 in _zval_ptr_dtor_wrapper (zval_ptr=0x904fc1c) at /usr/src/php-5.3.2/Zend/zend_variables.c:178
#3  0xb752b85f in zend_hash_destroy (ht=0x9035e20) at /usr/src/php-5.3.2/Zend/zend_hash.c:526
#4  0xb751b43f in _zval_dtor_func (zvalue=0x892fef0, __zend_filename=0xb7a55e38 "/usr/src/php-5.3.2/Zend/zend_execute_API.c", __zend_lineno=439)
    at /usr/src/php-5.3.2/Zend/zend_variables.c:43
#5  0xb750d237 in _zval_dtor (zvalue=0x892fef0, __zend_filename=0xb7a55e38 "/usr/src/php-5.3.2/Zend/zend_execute_API.c", __zend_lineno=439)
    at /usr/src/php-5.3.2/Zend/zend_variables.h:35
#6  0xb750d481 in _zval_ptr_dtor (zval_ptr=0x892ec04, __zend_filename=0xb7a56fe0 "/usr/src/php-5.3.2/Zend/zend_variables.c", __zend_lineno=178)
    at /usr/src/php-5.3.2/Zend/zend_execute_API.c:439
#7  0xb751b810 in _zval_ptr_dtor_wrapper (zval_ptr=0x892ec04) at /usr/src/php-5.3.2/Zend/zend_variables.c:178
#8  0xb752b85f in zend_hash_destroy (ht=0x93c5290) at /usr/src/php-5.3.2/Zend/zend_hash.c:526
#9  0xb751b43f in _zval_dtor_func (zvalue=0x9158d98, __zend_filename=0xb7a55e38 "/usr/src/php-5.3.2/Zend/zend_execute_API.c", __zend_lineno=439)
    at /usr/src/php-5.3.2/Zend/zend_variables.c:43
#10 0xb750d237 in _zval_dtor (zvalue=0x9158d98, __zend_filename=0xb7a55e38 "/usr/src/php-5.3.2/Zend/zend_execute_API.c", __zend_lineno=439)
    at /usr/src/php-5.3.2/Zend/zend_variables.h:35
#11 0xb750d481 in _zval_ptr_dtor (zval_ptr=0x95c5694, __zend_filename=0xb7a56fe0 "/usr/src/php-5.3.2/Zend/zend_variables.c", __zend_lineno=178)
    at /usr/src/php-5.3.2/Zend/zend_execute_API.c:439
#12 0xb751b810 in _zval_ptr_dtor_wrapper (zval_ptr=0x95c5694) at /usr/src/php-5.3.2/Zend/zend_variables.c:178
#13 0xb752b85f in zend_hash_destroy (ht=0x914cdf8) at /usr/src/php-5.3.2/Zend/zend_hash.c:526
#14 0xb751b43f in _zval_dtor_func (zvalue=0x93c9640, __zend_filename=0xb7a55e38 "/usr/src/php-5.3.2/Zend/zend_execute_API.c", __zend_lineno=439)
    at /usr/src/php-5.3.2/Zend/zend_variables.c:43
#15 0xb750d237 in _zval_dtor (zvalue=0x93c9640, __zend_filename=0xb7a55e38 "/usr/src/php-5.3.2/Zend/zend_execute_API.c", __zend_lineno=439)
    at /usr/src/php-5.3.2/Zend/zend_variables.h:35
#16 0xb750d481 in _zval_ptr_dtor (zval_ptr=0x93c4bc4, __zend_filename=0xb7a56fe0 "/usr/src/php-5.3.2/Zend/zend_variables.c", __zend_lineno=178)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/php-5.3.2/Zend/zend_execute_API.c:439
#17 0xb751b810 in _zval_ptr_dtor_wrapper (zval_ptr=0x93c4bc4) at /usr/src/php-5.3.2/Zend/zend_variables.c:178
#18 0xb752b85f in zend_hash_destroy (ht=0x892e930) at /usr/src/php-5.3.2/Zend/zend_hash.c:526
#19 0xb751b43f in _zval_dtor_func (zvalue=0xbfb9b16c, __zend_filename=0xb7a5ba0c "/usr/src/php-5.3.2/Zend/zend_execute.c", __zend_lineno=678)
    at /usr/src/php-5.3.2/Zend/zend_variables.c:43
#20 0xb7546780 in _zval_dtor (zvalue=0xbfb9b16c, __zend_filename=0xb7a5ba0c "/usr/src/php-5.3.2/Zend/zend_execute.c", __zend_lineno=678)
    at /usr/src/php-5.3.2/Zend/zend_variables.h:35
#21 0xb7573fe0 in zend_assign_to_variable (variable_ptr_ptr=0x93c37dc, value=0x8323c48, is_tmp_var=1) at /usr/src/php-5.3.2/Zend/zend_execute.c:678
#22 0xb75df7b4 in ZEND_ASSIGN_DIM_SPEC_CV_CV_HANDLER (execute_data=0x8323a78) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:29444
#23 0xb7548674 in execute (op_array=0x8388798) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:104
#24 0xb750f810 in zend_call_function (fci=0xbfb9b4d4, fci_cache=0xbfb9b4c0) at /usr/src/php-5.3.2/Zend/zend_execute_API.c:947
#25 0xb74166f3 in zif_call_user_func_array (ht=2, return_value=0x8c1ba68, return_value_ptr=0x0, this_ptr=0x0, return_value_used=1)
    at /usr/src/php-5.3.2/ext/standard/basic_functions.c:4782
#26 0xb754926d in zend_do_fcall_common_helper_SPEC (execute_data=0x831f130) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:313
#27 0xb754df72 in ZEND_DO_FCALL_SPEC_CONST_HANDLER (execute_data=0x831f130) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:1603
#28 0xb7548674 in execute (op_array=0x8388798) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:104
#29 0xb750f810 in zend_call_function (fci=0xbfb9b7d4, fci_cache=0xbfb9b7c0) at /usr/src/php-5.3.2/Zend/zend_execute_API.c:947
#30 0xb74166f3 in zif_call_user_func_array (ht=2, return_value=0x89350d8, return_value_ptr=0x0, this_ptr=0x0, return_value_used=1)
    at /usr/src/php-5.3.2/ext/standard/basic_functions.c:4782
#31 0xb754926d in zend_do_fcall_common_helper_SPEC (execute_data=0x831e4d4) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:313
#32 0xb754df72 in ZEND_DO_FCALL_SPEC_CONST_HANDLER (execute_data=0x831e4d4) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:1603
#33 0xb7548674 in execute (op_array=0x8388798) at /usr/src/php-5.3.2/Zend/zend_vm_execute.h:104
#34 0xb751d99e in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/php-5.3.2/Zend/zend.c:1194
#35 0xb74b4a69 in php_execute_script (primary_file=0xbfb9dc04) at /usr/src/php-5.3.2/main/main.c:2260
#36 0xb75e2a7b in php_handler (r=0x8313c28) at /usr/src/php-5.3.2/sapi/apache2handler/sapi_apache2.c:655
#37 0x0807c4c9 in ap_run_handler (r=0x8313c28) at config.c:158
---Type <return> to continue, or q <return> to quit---
#38 0x0807f829 in ap_invoke_handler (r=0x8313c28) at config.c:372
#39 0x0808ad56 in ap_process_request (r=0x8313c28) at http_request.c:282
#40 0x08087dc8 in ap_process_http_connection (c=0x830dc38) at http_core.c:190
#41 0x08083949 in ap_run_process_connection (c=0x830dc38) at connection.c:43
#42 0x0808fa0d in child_main (child_num_arg=<value optimized out>) at prefork.c:662
#43 0x0808fd23 in make_child (s=0x80b2fa0, slot=0) at prefork.c:758
#44 0x080903aa in ap_mpm_run (_pconf=0x80b10a8, plog=0x80df160, s=0x80b2fa0) at prefork.c:776
#45 0x08068f30 in main (argc=Cannot access memory at address 0x0
) at main.c:740