Description:
------------
Background:
- Compiled using Solaris Studio 12.3
- Compiled with httpd 2.4.16
- Crashes seem random/intermittent, so it's unknown how to reproduce or even test.
- Other enabled modules:
  - GeoIP 1.1.0
  - ibm_db2 1.9.6
  - oci8 2.0.8
  - spl_types 0.4.0
  - xcache 3.0.4

PHP Build steps:
export CC=cc
export CXX=CC
export CFLAGS="-m64 -I/usr/include/openldap -I/usr/local/include -I/usr/include"
export CPPFLAGS="-m64 -I/usr/include/openldap -I/usr/local/include -I/usr/include"
export LIBS="-lldap-2.4 -llber-2.4"
export LDFLAGS="-L/opt/mysql/mysql/lib -R/opt/mysql/mysql/lib -L/usr/local/lib -R/usr/local/lib"
export LD_PRELOAD_64=/usr/local/lib/preloadable_libiconv.so
export LD_LIBRARY_PATH=/usr/local/lib
export IBM_DB2="/home/db2inst1/sqllib"

(Hacked up the configure script to change "-lldap" to "-lldap-2.4", and "-llber" to "-llber-2.4", to link in the proper OpenLDAP libraries that come with Solaris)
./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql --with-mysqli --with-pdo-mysql --with-iconv=/usr/local --with-openssl --with-curl --with-ldap
--with-mhash --with-mcrypt --with-gd --with-jpeg-dir --with-png-dir --with-xsl --enable-inline-optimization --enable-ftp --with-zlib-dir --enable-soap --enable-sockets --enable-mbstring --with-gettext --enable-intl --enable-zip --enable-gd-native-ttf --with-freetype-dir=/usr --with-xpm-dir=/usr --with-ibm-db2=$IBM_DB2 --with-pdo-odbc=ibm-db2,/opt/IBM/db2/V10.5

The php.ini options that differ from the out-of-the-box php.ini-production:
short_open_tag = On
output_buffering = Off
highlight.string  = #CC0000
highlight.comment = #FF9900
highlight.keyword = #006600
highlight.bg      = #FFFFFF
highlight.default = #0000CC
highlight.html    = #000000
expose_php = Off
date.timezone = "America/Chicago"
max_execution_time = 300
memory_limit = 1024M
error_reporting  =  E_ALL & ~E_NOTICE
error_log=/var/httpd/logs/php_errors
warn_plus_overloading = Off
variables_order = "EGPCS"
max_input_vars = 100000
register_argc_argv = On
post_max_size = 100M
gpc_order = "GPC"
include_path
enable_dl = On
upload_max_filesize = 100M
extension=oci8.so
extension=xcache.so
extension=ibm_db2.so
extension=spl_types.so
extension=geoip.so
ibm_db2.instance_name
sendmail_from
session.gc_maxlifetime
session.use_trans_sid = 1
mcrypt.algorithms_dir
mcrypt.modes_dir
geoip.custom_directory
(various xcache directives)

Backtrace:
root# adb core.httpd.1444161809.13571
$c
core file = core.httpd.1444161648.13571 -- program ``
/usr/local/apache2/bin/httpd'' on platform i86pc
SIGSEGV: Segmentation Fault
libphp5.so`zend_hash_move_forward_ex+0x4e()
libphp5.so`apply_config+0x12c()
libphp5.so`php_handler+0x28a()
ap_run_handler+0x7e()
ap_invoke_handler+0x1b1()
ap_process_async_request+0x4df()
ap_process_http_async_connection+0xbd()
ap_process_http_connection+0x39()
ap_run_process_connection+0x7e()
process_socket+0x450()
worker_thread+0x455()
libapr-1.so.0.5.2`dummy_worker+0x30()
libc.so.1`_thrp_setup+0xa5()
libc.so.1`_lwp_start()
zend_hash_move_forward_ex ::dis
libphp5.so`zend_hash_move_forward_ex:   pushq  %rbp
libphp5.so`zend_hash_move_forward_ex+1: movq   %rsp,%rbp
libphp5.so`zend_hash_move_forward_ex+4: subq   $0x30,%rsp
libphp5.so`zend_hash_move_forward_ex+8: movq   %rdi,-0x8(%rbp)
libphp5.so`zend_hash_move_forward_ex+0xc:       movq   %rsi,-0x10(%rbp)
libphp5.so`zend_hash_move_forward_ex+0x10:      movq   -0x10(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x14:      cmpq   $0x0,%r8
libphp5.so`zend_hash_move_forward_ex+0x18:      je     +0xa     <libphp5.so`zend_hash_move_forward_ex+0x24>
libphp5.so`zend_hash_move_forward_ex+0x1a:      movq   -0x10(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x1e:      movq   %r8,-0x28(%rbp)
libphp5.so`zend_hash_move_forward_ex+0x22:      jmp    +0xc     <libphp5.so`zend_hash_move_forward_ex+0x30>
libphp5.so`zend_hash_move_forward_ex+0x24:      movq   -0x8(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x28:      leaq   0x18(%r8),%r8
libphp5.so`zend_hash_move_forward_ex+0x2c:      movq   %r8,-0x28(%rbp)
libphp5.so`zend_hash_move_forward_ex+0x30:      movq   -0x28(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x34:      movq   %r8,-0x20(%rbp)
libphp5.so`zend_hash_move_forward_ex+0x38:      movq   -0x20(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x3c:      movq   0x0(%r8),%r8
libphp5.so`zend_hash_move_forward_ex+0x40:      cmpq   $0x0,%r8
libphp5.so`zend_hash_move_forward_ex+0x44:      je     +0x1e    <libphp5.so`zend_hash_move_forward_ex+0x64>
libphp5.so`zend_hash_move_forward_ex+0x46:      movq   -0x20(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x4a:      movq   0x0(%r8),%r8
libphp5.so`zend_hash_move_forward_ex+0x4e:      movq   0x20(%r8),%r9
libphp5.so`zend_hash_move_forward_ex+0x52:      movq   -0x20(%rbp),%r8
libphp5.so`zend_hash_move_forward_ex+0x56:      movq   %r9,0x0(%r8)
libphp5.so`zend_hash_move_forward_ex+0x5a:      movl   $0x0,-0x14(%rbp)
libphp5.so`zend_hash_move_forward_ex+0x61:      jmp    +0x8     <libphp5.so`zend_hash_move_forward_ex+0x6b>
libphp5.so`zend_hash_move_forward_ex+0x63:      nop
libphp5.so`zend_hash_move_forward_ex+0x64:      movl   $-0x1,-0x14(%rbp)        <0xffffffff>
libphp5.so`zend_hash_move_forward_ex+0x6b:      movl   -0x14(%rbp),%eax
libphp5.so`zend_hash_move_forward_ex+0x6e:      leave
libphp5.so`zend_hash_move_forward_ex+0x6f:      ret
$r
%rax = 0x0000000000000000       %r8  = 0x0000000000000000
%rbx = 0xffff80ffbea7aa40       %r9  = 0x00000000061339e0
%rcx = 0x0000000000000001       %r10 = 0xffff80fd756395f0
%rdx = 0x0000000000000001       %r11 = 0x000000000bae78d0
%rsi = 0x0000000000000000       %r12 = 0x0000000000511688
%rdi = 0x0000000000949680       %r13 = 0x0000000000000000
                                %r14 = 0x0000000000000000
                                %r15 = 0x0000000000000000

%cs = 0x0053    %fs = 0x0000    %gs = 0x0000
%ds = 0x004b    %es = 0x004b    %ss = 0x004b

%rip = 0xffff80fd75822f9e libphp5.so`zend_hash_move_forward_ex+0x4e
%rbp = 0xffff80ff9a4136d0
%rsp = 0xffff80ff9a4136a0

%rflags = 0x00000206
  id=0 vip=0 vif=0 ac=0 vm=0 rf=0 nt=0 iopl=0x0
  status=<of,df,IF,tf,sf,zf,af,PF,cf>

%gsbase = 0x0000000000000000
%fsbase = 0xffff80ffbea7aa40
%trapno = 0xe
   %err = 0x4
0xffff80ff9a4136a0 ::dump -e -q -w 3
ffff80ff9a4136a0:  50828e00 00000000 98969400 00000000 98969400 00000000 f8679500 00000000 00000000 00000000 80969400 00000000

Initial Analysis:
Since register %r8 is 0x0 at the time of the crash, this a null pointer dereference issue. Based on the branching, I believe it is occurring near line 1126 of Zend/zend_hash.c.

Further Analysis:
The zend_hash_move_forward_ex function is used throughout PHP core, and if the problem were there, I would expect to see diversity of core dumps. However, we have only seen core dumps with the apply_config function in the stack trace. That's why I suspect this may be an apache2 handler/filter issue.


Test script:
---------------
N/A