Do you have a reproducible testcase?

This appears to be a race condition - so I am unable to reproduce.  I am however 
able to make the problem go away by modifying pdo_dbh.c to the following:

static void pdo_dbh_free_storage(pdo_dbh_t *dbh TSRMLS_DC)
{
	if (dbh->in_txn && dbh->methods && dbh->methods->rollback) {
		dbh->methods->rollback(dbh TSRMLS_CC);
		dbh->in_txn = 0;
	}
	
	if (dbh->is_persistent && dbh->methods && dbh->methods-
>persistent_shutdown) {
		dbh->methods->persistent_shutdown(dbh TSRMLS_CC);
	}
	//uncomment below to cause zend_mm_heap corrupted
	//zend_object_std_dtor(&dbh->std TSRMLS_CC);
	//dbh->std.properties = NULL;
	dbh_free(dbh TSRMLS_CC);
}

If I recompile this into PHP it works - however now there is most likely a 
memory leak.  I checked and this code is also new from PHP 5.3.  So definitely 
it is causing the fault.

Don't know what the real solution is though.

TJ

I just want to emphasize - that commenting out the code not a solution - since it 
causes errors later down the line.  Also, when stepping / breaking at problem 
area through the code - the project starts loading in bits and pieces, no 
segfaults occur.  Only when left to run without breakpoints does it crash - 
therefor this really does seem like a concurrency problem.

I can't reproduce this on my machine.

Apparently your PHP is not compiled in threaded mode (no tsrm_ls parameters in the stacktrace) so I assume you're not in threaded mode, so no race conditions.

Can you share more details on your setup and code?

That is an old backtrace - here is the newest:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd8fe9700 (LWP 31920)]
0x00007fffeb6a5722 in zval_delref_p (pz=0x5a5a5a5a5a5a5a5a) at /home/tj/php-
5.4.14/Zend/zend.h:395
395		return --pz->refcount__gc;
(gdb) backtrace 
#0  0x00007fffeb6a5722 in zval_delref_p (pz=0x5a5a5a5a5a5a5a5a) at /home/tj/php-
5.4.14/Zend/zend.h:395
#1  0x00007fffeb6a7d06 in _zval_ptr_dtor (zval_ptr=0x7fffd6d39378, 
__zend_filename=0x7fffebb88468 "/home/tj/php-5.4.14/Zend/zend_objects.c", 
__zend_lineno=54)
    at /home/tj/php-5.4.14/Zend/zend_execute_API.c:432
#2  0x00007fffeb6f258a in zend_object_std_dtor (object=0x7fffd00f56c0, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-5.4.14/Zend/zend_objects.c:54
#3  0x00007fffeb3e0056 in pdo_dbh_free_storage (dbh=0x7fffd00f56c0, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-5.4.14/ext/pdo/pdo_dbh.c:1577
#4  0x00007fffeb6fac18 in zend_objects_store_del_ref_by_handle_ex (handle=122, 
handlers=0x7fffebeb8a20 <pdo_dbh_object_handlers>, tsrm_ls=0x7fffd0017170)
    at /home/tj/php-5.4.14/Zend/zend_objects_API.c:221
#5  0x00007fffeb6fa759 in zend_objects_store_del_ref (zobject=0x7fffd6d240e0, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-5.4.14/Zend/zend_objects_API.c:173
#6  0x00007fffeb6baacd in _zval_dtor_func (zvalue=0x7fffd6d240e0, 
__zend_filename=0x7fffebb83be8 "/home/tj/php-5.4.14/Zend/zend_execute_API.c", 
__zend_lineno=438)
    at /home/tj/php-5.4.14/Zend/zend_variables.c:54
#7  0x00007fffeb6a58c1 in _zval_dtor (zvalue=0x7fffd6d240e0, 
__zend_filename=0x7fffebb83be8 "/home/tj/php-5.4.14/Zend/zend_execute_API.c", 
__zend_lineno=438)
    at /home/tj/php-5.4.14/Zend/zend_variables.h:35
#8  0x00007fffeb6a7da9 in _zval_ptr_dtor (zval_ptr=0x7fffd6bee268, 
__zend_filename=0x7fffebb84cb0 "/home/tj/php-5.4.14/Zend/zend_variables.c", 
__zend_lineno=182)
    at /home/tj/php-5.4.14/Zend/zend_execute_API.c:438
#9  0x00007fffeb6baef5 in _zval_ptr_dtor_wrapper (zval_ptr=0x7fffd6bee268) at 
/home/tj/php-5.4.14/Zend/zend_variables.c:182
#10 0x00007fffeb6d3281 in zend_hash_destroy (ht=0x7fffd6d39768) at /home/tj/php-
5.4.14/Zend/zend_hash.c:560
#11 0x00007fffeb6baa76 in _zval_dtor_func (zvalue=0x7fffd7d18be8, 
__zend_filename=0x7fffebb83be8 "/home/tj/php-5.4.14/Zend/zend_execute_API.c", 
__zend_lineno=438)
    at /home/tj/php-5.4.14/Zend/zend_variables.c:45
#12 0x00007fffeb6a58c1 in _zval_dtor (zvalue=0x7fffd7d18be8, 
__zend_filename=0x7fffebb83be8 "/home/tj/php-5.4.14/Zend/zend_execute_API.c", 
__zend_lineno=438)
    at /home/tj/php-5.4.14/Zend/zend_variables.h:35
#13 0x00007fffeb6a7da9 in _zval_ptr_dtor (zval_ptr=0x7fffd7d18d98, 
__zend_filename=0x7fffebb84228 "/home/tj/php-5.4.14/Zend/zend_opcode.c", 
__zend_lineno=165)
    at /home/tj/php-5.4.14/Zend/zend_execute_API.c:438
#14 0x00007fffeb6aef6a in cleanup_user_class_data (ce=0x7fffd7d185d0, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-5.4.14/Zend/zend_opcode.c:165
#15 0x00007fffeb6af1c8 in zend_cleanup_user_class_data (pce=0x7fffd00d6ad8, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-5.4.14/Zend/zend_opcode.c:198
#16 0x00007fffeb6d3ce3 in zend_hash_reverse_apply (ht=0x7fffd001a770, 
apply_func=0x7fffeb6af194 <zend_cleanup_user_class_data>, 
tsrm_ls=0x7fffd0017170)
    at /home/tj/php-5.4.14/Zend/zend_hash.c:799
#17 0x00007fffeb6a71e8 in shutdown_executor (tsrm_ls=0x7fffd0017170) at 
/home/tj/php-5.4.14/Zend/zend_execute_API.c:289
#18 0x00007fffeb6be217 in zend_deactivate (tsrm_ls=0x7fffd0017170) at 
/home/tj/php-5.4.14/Zend/zend.c:938
#19 0x00007fffeb601c90 in php_request_shutdown (dummy=0x0) at /home/tj/php-
5.4.14/main/main.c:1800
#20 0x00007fffeb777c6d in php_apache_request_dtor (r=0x7fffd000f068, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-
5.4.14/sapi/apache2handler/sapi_apache2.c:507
#21 0x00007fffeb7787cf in php_handler (r=0x7fffd000f068) at /home/tj/php-
5.4.14/sapi/apache2handler/sapi_apache2.c:679
#22 0x0000000000447e40 in ap_run_handler (r=0x7fffd000f068) at config.c:169
#23 0x000000000044827b in ap_invoke_handler (r=r@entry=0x7fffd000f068) at 
config.c:432
#24 0x000000000045b1bc in ap_internal_redirect (new_uri=<optimised out>, r=
<optimised out>) at http_request.c:644
#25 0x00007fffebed6658 in handler_redirect (r=0x7fffd0002970) at 
mod_rewrite.c:5051
#26 0x0000000000447e40 in ap_run_handler (r=0x7fffd0002970) at config.c:169
#27 0x000000000044827b in ap_invoke_handler (r=r@entry=0x7fffd0002970) at 
config.c:432
#28 0x000000000045bc5a in ap_process_async_request (r=0x7fffd0002970) at 
http_request.c:317
#29 0x0000000000458b27 in ap_process_http_async_connection (c=0x7fffe4037410) at 
http_core.c:143
#30 ap_process_http_connection (c=0x7fffe4037410) at http_core.c:228
#31 0x0000000000450ff0 in ap_run_process_connection (c=0x7fffe4037410) at 
connection.c:41
#32 0x000000000046284a in process_socket (my_thread_num=24, my_child_num=0, 
cs=0x7fffe4037398, sock=0x7fffe4037190, p=0x7fffe4037118, thd=0x6bf770) at 
event.c:964
#33 worker_thread (thd=0x6bf770, dummy=<optimised out>) at event.c:1812
#34 0x00007ffff7531e9a in start_thread () from /lib/x86_64-linux-
gnu/libpthread.so.0
#35 0x00007ffff725ecbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#36 0x0000000000000000 in ?? ()

The project relies heavily on PDO and runs okay on PHP 5.3.24.

Also - some additional info which may help:

(gdb) frame 3
#3  0x00007fffeb3e0056 in pdo_dbh_free_storage (dbh=0x7fffd00f56c0, 
tsrm_ls=0x7fffd0017170) at /home/tj/php-5.4.14/ext/pdo/pdo_dbh.c:1577
1577		zend_object_std_dtor(&dbh->std TSRMLS_CC);
(gdb) print dbh->std
$1 = {ce = 0x7fffd6d3afc0, properties = 0x0, properties_table = 0x7fffd6d39378, 
guards = 0x0}
(gdb)

and

for source_code/Zend/zend_objects.c:37 to 59:

ZEND_API void zend_object_std_dtor(zend_object *object TSRMLS_DC)
{
	if (object->guards) {
		zend_hash_destroy(object->guards);
		FREE_HASHTABLE(object->guards);
	}
	if (object->properties) {
		zend_hash_destroy(object->properties);
		FREE_HASHTABLE(object->properties);
		if (object->properties_table) {
			efree(object->properties_table);
		}
	} else if (object->properties_table) {
		int i;

		for (i = 0; i < object->ce->default_properties_count; i++) {
			if (object->properties_table[i]) {
				zval_ptr_dtor(&object->properties_table[i]);
			}
		}
		efree(object->properties_table);
	}
}


(gdb) print object->properties_table[0]
$2 = (zval *) 0x5a5a5a5a5a5a5a5a
(gdb) print &object->properties_table[0]
$3 = (zval **) 0x7fffd6d39378
(gdb) print object->ce->default_properties_count
$4 = 2
(gdb) print i
$5 = 0
(gdb)

Not sure if this loop is thread safe:

		for (i = 0; i < object->ce->default_properties_count; i++) {
			if (object->properties_table[i]) {
				zval_ptr_dtor(&object->properties_table[i]);
			}
		}

Thanks for your help!

so, the new backtrace has tsrm symbols, so what environment are you using?8which web server,sapi, ...) Why threaded context?

And please try using helgrind (valgrind --tool=helgrind) with the server, this should show details on race conditions.

Ok - I just recompiled apache with prefork (It was supposed to be the default, 
instead it defaulted to event) - and recompiled PHP, and it is no longer 
multithreaded - and the problem persists:

Apache information now:

Server version: Apache/2.4.4 (Unix)
Server built:   Apr 30 2013 17:41:49
Server's Module Magic Number: 20120211:11
Server loaded:  APR 1.4.6, APR-UTIL 1.5.2
Compiled using: APR 1.4.6, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/usr/local/apache2"
 -D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

PHP Information :

System	Linux dev 3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 
x86_64
Build Date	Apr 30 2013 17:54:17
Configure Command	 './configure' '--with-
apxs2=/usr/local/apache2/bin/apxs' '--enable-mbstring' '--with-config-file-
path=/etc/php5/' '--with-gettext=/usr/bin/gettext' '--with-config-file-scan-
dir=/etc/php5/mods-enabled/' '--with-mysql=mysqlnd' '--with-mysqli=mysqlnd' '--
with-pdo-mysql=mysqlnd' '--with-openssl' '--with-libdir=/lib/x86_64-linux-gnu/' 
'--enable-debug'
Server API	Apache 2.0 Handler
Virtual Directory Support	disabled
Configuration File (php.ini) Path	/etc/php5/
Loaded Configuration File	/etc/php5/php.ini
Scan this dir for additional .ini files	/etc/php5/mods-enabled/
Additional .ini files parsed	/etc/php5/mods-enabled/xdebug.ini
PHP API	20100412
PHP Extension	20100525
Zend Extension	220100525
Zend Extension Build	API220100525,NTS,debug
PHP Extension Build	API20100525,NTS,debug
Debug Build	yes
Thread Safety	disabled
Zend Signal Handling	disabled
Zend Memory Manager	enabled
Zend Multibyte Support	provided by mbstring
IPv6 Support	enabled
DTrace Support	disabled
Registered PHP Streams	https, ftps, php, file, glob, data, http, ftp, phar
Registered Stream Socket Transports	tcp, udp, unix, udg, ssl, sslv3, tls
Registered Stream Filters	convert.iconv.*, string.rot13, string.toupper, 
string.tolower, string.strip_tags, convert.*, consumed, dechunk

Apache PHP Information

Apache Version	Apache/2.4.4 (Unix) OpenSSL/1.0.1c PHP/5.4.14
Apache API Version	20120211
Server Administrator	you@example.com
Hostname:Port	dev:0
User/Group	www-data(33)/33
Max Requests	Per Child: 0 - Keep Alive: on - Max Per Connection: 100
Timeouts	Connection: 60 - Keep-Alive: 5
Virtual Server	No
Server Root	/usr/local/apache2
Loaded Modules	core mod_so http_core prefork mod_authn_file mod_authn_dbm 
mod_authn_anon mod_authn_dbd mod_authn_socache mod_authn_core mod_authz_host 
mod_authz_groupfile mod_authz_user mod_authz_dbm mod_authz_owner mod_authz_dbd 
mod_authz_core mod_access_compat mod_auth_basic mod_auth_form mod_auth_digest 
mod_allowmethods mod_file_cache mod_cache mod_cache_disk mod_socache_shmcb 
mod_socache_dbm mod_socache_memcache mod_dbd mod_dumpio mod_buffer mod_ratelimit 
mod_reqtimeout mod_ext_filter mod_request mod_include mod_filter mod_substitute 
mod_sed mod_deflate mod_mime mod_log_config mod_log_debug mod_logio mod_env 
mod_expires mod_headers mod_unique_id mod_setenvif mod_version mod_remoteip 
mod_proxy mod_proxy_connect mod_proxy_ftp mod_proxy_http mod_proxy_fcgi 
mod_proxy_scgi mod_proxy_ajp mod_proxy_balancer mod_proxy_express mod_session 
mod_session_cookie mod_session_dbd mod_slotmem_shm mod_ssl 
mod_lbmethod_byrequests mod_lbmethod_bytraffic mod_lbmethod_bybusyness 
mod_lbmethod_heartbeat mod_unixd mod_dav mod_status mod_autoindex mod_info 
mod_cgid mod_dav_fs mod_vhost_alias mod_negotiation mod_dir mod_actions 
mod_speling mod_userdir mod_alias mod_rewrite mod_php5

This is running on a Ubuntu 12.10 virtual server.

Unfortunately I have to go now but I'll check in again on Thursday.

Keep well,

TJ

if you can reproduce it in a 100% chance, please run it with valgrind, let's see 
what valgrind says about this..

thanks

Hey guys - I have run various tests - using valgrind I did not find anything.

I have debugged the code, and I have narrowed down the problem.

It is not a race condition, the problem is dangling pointers are getting freed:

below is still okay:

//php_source/ext/pdo/pdo_dbh.c
static void pdo_dbh_free_storage(pdo_dbh_t *dbh TSRMLS_DC)
{
...	
	zend_object_std_dtor(&dbh->std TSRMLS_CC);
	dbh->std.properties = NULL;
...
}

The problem arrives here in zend_object_std_dtor, in 
php_source/Zend/zend_objects.c

The situation can be recreated when a zend_object has properties AND a properties_table

Look at the code below:

In my case, the below function gets called TWICE:

ZEND_API void zend_object_std_dtor(zend_object *object TSRMLS_DC)
{
	if (object->guards) {
		zend_hash_destroy(object->guards);
		FREE_HASHTABLE(object->guards);
	}
	if (object->properties) {
		zend_hash_destroy(object->properties);
		FREE_HASHTABLE(object->properties);
		if (object->properties_table) {
			efree(object->properties_table);
		}
	} else if (object->properties_table) {
		int i;

		for (i = 0; i < object->ce->default_properties_count; i++) {
			if (object->properties_table[i]) {
				zval_ptr_dtor(&object->properties_table[i]);
			}
		}
		efree(object->properties_table);
	}
}

The important bits are:

...
if (object->properties)
{
...
        if (object->properties_table) {
                 efree(object_properties_table); //this creates dangling pointers 0x5a5a5a5a5a5a5a5a
                 //Now, object->properties_table[0, 1, 2, ... n] are all dangling pointers
        }
...
} else if (object->properties_table)
{
...
//now the second time this function gets called, at some point during the zval_ptr_dtor call on object-
>properties_table[i], a function tries to decrease the ref val on the dangling pointer, and this causes SIGSEGV 11 
(zend_mm_heap corrupted)
...
}

I do not know what the correct behavior should be.

I have tried removing the "else if" and then setting the properties_table to NULL after de-allocation and forcing de-
allocation of both, however different modifications to the code have lead to different problems and I do not seem to 
find a 100% correct solution.  The solution I had in mind to work perfect lead to for example:

/home/tj/php-5.4.14/Zend/zend_API.c(1127) :  Freeing 0x7FFFE89B6418 (16 bytes), script=/.../index.php
=== Total 1 memory leaks detected ===

I do not know if the function should indeed be called twice, or whether the object should only contain properties or 
properties_table and not both, and why the properties_table does not get set to NULL after being de-allocated, as 
well as the other properties_table[] pointers.

In either way - it turns out that this situation is created because we are extending the PDO class in PHP, and adding 
custom properties and methods.

I will still investigate further - but I will really appreciate some advice / help!

In my mind, the correct function should look like this (without knowing the 
exact context in which it runs)

//source_code/Zend/zend_objects.c:

ZEND_API void zend_object_std_dtor(zend_object *object TSRMLS_DC)
{
	if (object->guards) {
		zend_hash_destroy(object->guards);
		FREE_HASHTABLE(object->guards);
	}

	if (object->properties) {
		zend_hash_destroy(object->properties);
		FREE_HASHTABLE(object->properties);
	}

	if (object->properties_table) {
		int i;
		for (i = 0; i < object->ce->default_properties_count; i++) {
			if (object->properties_table[i]) {
				zval_ptr_dtor(&object->properties_table[i]);
				object->properties_table[i] = NULL;
			}
		}
		efree(object->properties_table);
		object->properties_table = NULL;
	}
}

However, this appears in my apache logs, only when running in debug mode:

[Thu May 02 16:41:28.476657 2013] [auth_digest:notice] [pid 23396] AH01757: 
generating secret for digest authentication ...
[Thu May 02 16:41:29.052276 2013] [ssl:warn] [pid 23396] AH01873: Init: Session 
Cache is not configured [hint: SSLSessionCache]
[Thu May 02 16:41:29.052747 2013] [lbmethod_heartbeat:notice] [pid 23396] 
AH02282: No slotmem from mod_heartmonitor
[Thu May 02 16:41:29.141345 2013] [core:warn] [pid 23396] AH00098: pid file 
/usr/local/apache2/logs/httpd.pid overwritten -- Unclean shutdown of previous 
Apache run?
[Thu May  2 16:41:35 2013]  Script:  '/.../index.php'
/home/tj/php-5.4.14/Zend/zend_API.c(1127) :  Freeing 0x7FFFE89B6050 (16 bytes), 
script=/usr/local/apache2/htdocs/www/plista/www.2.3.2/app/webroot/index.php
=== Total 1 memory leaks detected ===

But the project loads without problems.

Even more correct:

ZEND_API void zend_object_std_dtor(zend_object *object TSRMLS_DC)
{
	if (object->guards) {
		zend_hash_destroy(object->guards);
		FREE_HASHTABLE(object->guards);
		object->guards = NULL;
	}

	if (object->properties) {
		zend_hash_destroy(object->properties);
		FREE_HASHTABLE(object->properties);
		object->properties = NULL;
	}

	if (object->properties_table) {
		int i;
		for (i = 0; i < object->ce->default_properties_count; i++) {
			if (object->properties_table[i]) {
				zval_ptr_dtor(&object->properties_table[i]);
				object->properties_table[i] = NULL;
			}
		}
		efree(object->properties_table);
		object->properties_table = NULL;
	}
}

This issue can be closed - There is a problem between the combination of ( or 
rather lack of using imagick ), pdo, and the error handling, and possibly the 
logging system, unique to the site and not 100% reproducible using a reduced 
test script (and I also do not want spend a whole day writing one)

using imagick 3 RC 2 with pdo enabled works for now.

If you want to reproduce the behavior, attempt to imagine the circumstances in 
which zend_object_std_dtor gets called twice for the same object.

Ok - so I managed to create a script which breaks it:

namespace SomeName\db;

class PDO extends \PDO {

	public $name;

	public function setName($name) {
		$this->name = $name;
	}	
}

class DatabaseManager {

	private static $connections;

	private static $options = array(
		PDO::ATTR_PERSISTENT => 1
	);

	public static function createConnection($name)
	{
		$pdo = null;

		$pdo = new 
PDO("mysql:host=localhost;port=3306;dbname=db_youfilter", "root", "lmnop", 
self::$options);

		self::$connections[$name] = $pdo;
	}
}

DatabaseManager::createConnection("foo");
DatabaseManager::createConnection("bar");

Turns out this is a similar bug to

https://bugs.php.net/bug.php?id=63176

And this issue is still open.

We will probably work around this problem by not extending PDO.

Hey guys - thanks so much for fixing this - I checked out git commit 
1143f58a7094ed9a4960bfb714fa2773dda7c704, built it and can confirm its no longer 
segfaulting.

cool thanks :)