Description:
------------
We are connecting to an Oracle 10g RAC cluster using load-balancing and failover.

When the apache 2.2 (prefork mpm) server is running under heavy load, segfaults are observed in the apache error_log when in php.ini we set oci8.events = On. 

We then 
1.ran httpd in gdb.
2. ran apachebench on another server 
 ab -c20 -n10000 http://appserver/test.php
3. httpd does not crash straight away - only after the first 100 requests, the seg fault occured. See 1st backtrace below.

We are using Oracle Instant Client 11.1.0.7.0.
  
When oci8.events is disabled (commented out), the crashes do not occur.

Also when apachectl restart is called when the apachebench test is running we will sometimes get a memory corruption message in glibc. 

See 2nd backtrace generated in apache error_log. The crash need not happen in oci_pconnect(). I have also seen the apachectl restart cause a crash in other places, such as ociexecute() -- i sent that backtrace to Chris Jones. 

Thanks John Lim



Reproduce code:
---------------
<?php

$connstr = "(DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = RAC1)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = RAC2)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = XXX)
      (FAILOVER_MODE =
        (TYPE = SELECT)
        (METHOD = BASIC)
        (RETRIES = 180)
        (DELAY = 5)
      )
    )
  )";

test($connstr);
die();


function test($connstr)
{
        echo $connstr, "\n";
        $db = oci_pconnect('XXX','XXX',$connstr) ;
        if ($db) echo "Connected !-------\n\n";
        else die();

        ## IF THE FOLLOWING LINES ARE REMOVED, NO CRASH

        $stmt = oci_parse($db,"ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD'");
        $ok = oci_execute($stmt);
        if ($ok) echo "ALTER SESS WORKED ".rand();
        ocifreestatement($stmt);

        ## END CRASH CODE
}
?>

Expected result:
----------------

(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = RAC1)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = RAC2)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XXX) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) Connected !------- ALTER SESS WORKED 1260846108

Actual result:
--------------
BACKTRACE 1 FROM GDB:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 47767150408976 (LWP 25640)]
0x000000389c4082e9 in pthread_mutex_lock () from /lib64/libpthread.so.0
(gdb) bt
#0  0x000000389c4082e9 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x00002b71b049a159 in sltsmna ()
   from /usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1
#2  0x00002b71afbc4a99 in kpceDeferFreeHdl ()
   from /usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1
#3  0x00002b71aebc96ae in kpufhndl0 ()
   from /usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1
#4  0x00002b71aebc42ff in kpufhndl ()
   from /usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1
#5  0x00002b71aeb9b539 in OCIHandleFree ()
   from /usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1
#6  0x00002b71ae42e437 in php_oci_spool_close (session_pool=0x1e72b110)
    at /installs/oci8-1.3.5/oci8.c:3066
#7  0x00002b71ae42f253 in php_oci_connection_close (connection=0x1e72aed0)
    at /installs/oci8-1.3.5/oci8.c:2181
#8  0x00002b71ae42f501 in php_oci_pconnection_list_dtor (
    entry=<value optimized out>) at /installs/oci8-1.3.5/oci8.c:1305
#9  0x00002b71a88b403e in plist_entry_destructor ()
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#10 0x00002b71a88b13e2 in zend_hash_apply_deleter ()
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#11 0x00002b71a88b1658 in zend_hash_graceful_reverse_destroy ()
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#12 0x00002b71a88a6f0b in zend_shutdown ()
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#13 0x00002b71a886513a in php_module_shutdown ()
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#14 0x00002b71a88651d9 in php_module_shutdown_wrapper ()
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#15 0x00002b71a8927eb1 in php_apache_child_shutdown ()
---Type <return> to continue, or q <return> to quit---
   from /usr/local/Zend/Core/modules/apache22/libphp5.so
#16 0x00002b71a7e5920d in apr_pool_cleanup_run ()
   from /usr/local/Zend/apache2/lib/libapr-1.so.0
#17 0x00002b71a7e59c97 in apr_pool_destroy ()
   from /usr/local/Zend/apache2/lib/libapr-1.so.0
#18 0x000000000047a1ce in ap_graceful_stop_signalled ()
#19 0x000000000047a606 in ap_graceful_stop_signalled ()
#20 0x000000000047a769 in ap_graceful_stop_signalled ()
#21 0x000000000047b402 in ap_mpm_run ()
#22 0x000000000042b0f5 in main ()


BACKTRACE 2 from APACHE error_log AFTER apachectl restart


[Thu Jun 11 19:55:54 2009] [notice] Apache/2.2.10 (Unix) mod_ssl/2.2.10 OpenSSL/0.9.8g Zend Core/2.5.2 PHP/5.2.6 configured -- resuming normal operations
[Thu Jun 11 19:56:08 2009] [notice] child pid 21276 exit signal Segmentation fault (11)
[Thu Jun 11 19:56:10 2009] [notice] child pid 21286 exit signal Segmentation fault (11)
*** glibc detected *** /usr/local/Zend/apache2/bin/httpd: double free or corruption (!prev): 0x000000001c1ca9a0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x389b871634]
/lib64/libc.so.6(cfree+0x8c)[0x389b874c5c]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(sltsmxd+0x33)[0x2b38afae30db]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(ltster+0x5f)[0x2b38afb1c2c7]
*** glibc detected *** /usr/local/Zend/apache2/bin/httpd: double free or corruption (!prev): 0x000000001c1ca9a0 ***
*** glibc detected *** /usr/local/Zend/apache2/bin/httpd: double free or corruption (!prev): 0x000000001c1ca9a0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x389b871634]
/lib64/libc.so.6(cfree+0x8c)[0x389b874c5c]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(sltsmxd+0x33)[0x2b38afae30db]
======= Backtrace: =========
/lib64/libc.so.6[0x389b871634]
*** glibc detected *** /usr/local/Zend/apache2/bin/httpd: double free or corruption (!prev): 0x000000001c1ca9a0 ***
/lib64/libc.so.6(cfree+0x8c)[0x389b874c5c]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(ltster+0x5f)[0x2b38afb1c2c7]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(kpufhndl0+0x4b9b)[0x2b38af85be9f]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(kpufhndl+0xb)[0x2b38af8572ff]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(sltsmxd+0x33)[0x2b38afae30db]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(ltster+0x5f)[0x2b38afb1c2c7]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(OCIHandleFree+0x13)[0x2b38af82e539]
/usr/local/Zend/Core/lib/php/20060613/oci8-1.3.5.so[0x2b38af0c1437]
/usr/local/Zend/Core/modules/apache22/libphp5.so(plist_entry_destructor+0x7e)[0x2b38a954703e]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a95443e2]
/usr/local/Zend/Core/modules/apache22/libphp5.so(zend_hash_graceful_reverse_destroy+0x18)[0x2b38a9544658]
/usr/local/Zend/Core/modules/apache22/libphp5.so(zend_shutdown+0x1b)[0x2b38a9539f0b]
/usr/local/Zend/Core/modules/apache22/libphp5.so(php_module_shutdown+0x2a)[0x2b38a94f813a]
/usr/local/Zend/Core/modules/apache22/libphp5.so(php_module_shutdown_wrapper+0x9)[0x2b38a94f81d9]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a95baeb1]
/usr/local/Zend/apache2/lib/libapr-1.so.0[0x2b38a8aec20d]
/usr/local/Zend/apache2/lib/libapr-1.so.0(apr_pool_destroy+0x47)[0x2b38a8aecc97]
/usr/local/Zend/apache2/bin/httpd[0x47a1ce]
/usr/local/Zend/apache2/bin/httpd[0x47a1fb]
/lib64/libpthread.so.0[0x389c40de70]
/lib64/libpthread.so.0(read+0x4b)[0x389c40cc8b]
======= Backtrace: =========
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(snttread+0xe)[0x2b38b111583e]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1[0x2b38afa3c053]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nsprecv+0x231)[0x2b38af927cc1]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nscon+0x8e6)[0x2b38afa05496]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nsdo+0xb95)[0x2b38af8e57f9]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1[0x2b38af9fa222]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nsrecv+0x5f)[0x2b38af9fa571]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nscall3+0xa9)[0x2b38af8e24bf]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nscall+0x45a)[0x2b38af8df922]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(niotns+0x73f)[0x2b38af979c17]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(nigcall+0x39)[0x2b38afa176d5]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(osncon+0x452)[0x2b38af985252]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(kpuadef+0x37)[0x2b38af843097]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(upiini+0x30b)[0x2b38af82b6d9]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(upiah0+0x24)[0x2b38af8232e6]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(kpuatch+0x577)[0x2b38af842617]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1[0x2b38b02c4658]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1[0x2b38b02c6664]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(kpuspgetpooledsession+0x1da)[0x2b38b02c3a38]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(kpuspsessionget+0x7bc)[0x2b38b02bf19c]
/usr/lib/oracle/11.1/client64/lib/libclntsh.so.11.1(OCISessionGet+0x118)[0x2b38b0122540]
/usr/local/Zend/Core/lib/php/20060613/oci8-1.3.5.so[0x2b38af0c1b29]
/usr/local/Zend/Core/lib/php/20060613/oci8-1.3.5.so(php_oci_do_connect_ex+0xe01)[0x2b38af0c4431]
/usr/local/Zend/Core/lib/php/20060613/oci8-1.3.5.so(php_oci_do_connect+0xf8)[0x2b38af0c5d08]
/usr/local/Zend/Core/lib/php/20060613/oci8-1.3.5.so(zif_oci_connect+0x13)[0x2b38af0d5d93]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a9568ce2]
/usr/local/Zend/Core/modules/apache22/libphp5.so(execute+0x163)[0x2b38a9558aa3]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a9568781]
/usr/local/Zend/Core/modules/apache22/libphp5.so(execute+0x163)[0x2b38a9558aa3]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a9568781]
/usr/local/Zend/Core/modules/apache22/libphp5.so(execute+0x163)[0x2b38a9558aa3]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a9568781]
/usr/local/Zend/Core/modules/apache22/libphp5.so(execute+0x163)[0x2b38a9558aa3]
/usr/local/Zend/Core/modules/apache22/libphp5.so(zend_execute_scripts+0x168)[0x2b38a95399c8]
/usr/local/Zend/Core/modules/apache22/libphp5.so(php_execute_script+0x1ed)[0x2b38a94f7fad]
/usr/local/Zend/Core/modules/apache22/libphp5.so[0x2b38a95bbea5]
/usr/local/Zend/apache2/bin/httpd(ap_run_handler+0x7a)[0x43dc0a]
/usr/local/Zend/apache2/bin/httpd(ap_invoke_handler+0x7c)[0x44104c]
/usr/local/Zend/apache2/bin/httpd(ap_process_request+0x1ae)[0x46922e]
/usr/local/Zend/apache2/bin/httpd[0x466440]
/usr/local/Zend/apache2/bin/httpd(ap_run_process_connection+0x72)[0x444ea2]
/usr/local/Zend/apache2/bin/httpd[0x47a4de]



CONFIGURE:
This is a standard Zend Core release. I disabled Zend specific Optimizer/Debugger stuff in php.ini before i reproduced the bug for you:


'./configure' '--prefix=/usr/local/Zend/Core' '--with-config-file-path=/etc' '--enable-force-cgi-redirect' '--enable-fastcgi' '--disable-debug' '--enable-inline-optimization' '--enable-memory-limit' '--disable-all' '--enable-ctype' '--enable-dom' '--enable-libxml' '--with-libxml-dir=/usr/local/Zend/Core' '--with-openssl=/usr/local/Zend/Core' '--with-pcre-regex' '--enable-session' '--enable-simplexml' '--enable-spl' '--enable-wddx' '--enable-xml' '--enable-hash' '--enable-reflection' '--with-zlib=/usr/local/Zend/Core' '--with-pear' '--with-apxs2=/usr/local/Zend/apache2/bin/apxs' '--with-layout=GNU' '--enable-json' '--enable-filter'