The apache module with optimiser enabled, and which up to now hasn't been crashing, now does so reproducably. A b/t is below although maybe not a lot to go on I'm afraid, but the events around the crash are *very significant*. The point of failure is always the same, and after starting a single apache process with -X, was always crashing on the 22nd request. A *most significant* thing is that adding effectively a dummy line of code, e.g. $x = true; changed the period of crashing from being the 22nd time to always the 16th time. Adding a second line the crashing stopped. Note that once determined, the crash interval is ALWAYS the same for a given state of my application.

Clearly this is a serious, although in the general case possibly rare, problem, and probably one of those bugs that only emmerges for a very specific set of circumstances. I'll happily arrange access to my machine for an official developer to help diagnose this now, but I'll take a snapshot of my DB, code base and shared memory anyway for a chance of future analysis.

Clearly something to focus on is why the crash would always occur after: 

1) a specific number of apache requests, and
2) this number depends on the code structure.

For the trace below, assuming that the zend_file_handle structures being passed are already initialised, then they're corrupt with bogus fd's and file file names for example.  As an aside, I did a purify run on Solaris and there are a significant number UMR instances, which whilst not always indicative of a problem, typically are, and so I suspect some fundamental memory problems that one typically 'gets away with'. I'll look at this more next week.

Anyway, the trace is:

(gdb) where
#0  0x80f5b39 in php_if_array_reduce (ht=3, return_value=0x8279dc4, this_ptr=0x0, return_value_used=1) at array.c:2886
#1  0x40253f84 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#2  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#3  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#4  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#5  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#6  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#7  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#8  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#9  0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#10 0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#11 0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#12 0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#13 0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#14 0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#15 0x40254183 in zend_reverse_table () from /usr/local/Zend/lib/ZendOptimizer.so
#16 0x4025f01d in zstd__mhm () from /usr/local/Zend/lib/ZendOptimizer.so
#17 0x80dde0b in zend_execute_scripts (type=8, file_count=3) at zend.c:752
#18 0x806a6fb in php_execute_script (primary_file=0xbffff768) at main.c:1206
#19 0x80eaa1b in apache_php_module_main (r=0x8246a2c, display_source_mode=0) at sapi_apache.c:89
#20 0x8067bee in send_php ()
#21 0x8067c36 in send_parsed_php ()
#22 0x8111609 in ap_invoke_handler ()
#23 0x8126a1f in process_request_internal ()
#24 0x8126a92 in ap_process_request ()
#25 0x811d8a6 in child_main ()
#26 0x811da85 in make_child ()
---Type <return> to continue, or q <return> to quit---
#27 0x811dc06 in startup_children ()
#28 0x811e27d in standalone_main ()
#29 0x811eacc in main ()
#30 0x400c4a5e in __libc_start_main () at ../sysdeps/generic/libc-start.c:93
(gdb)              

Nick