php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #80730 Random segfault PDO fetch_value
Submitted: 2021-02-10 18:23 UTC Modified: 2021-02-16 14:51 UTC
Votes:4
Avg. Score:4.5 ± 0.9
Reproduced:4 of 4 (100.0%)
Same Version:4 (100.0%)
Same OS:2 (50.0%)
From: ricardo dot branco at covermg dot com Assigned:
Status: Open Package: PDO MySQL
PHP Version: 7.4.15 OS: Centos 7
Private report: No CVE-ID: None
 [2021-02-10 18:23 UTC] ricardo dot branco at covermg dot com
Description:
------------
We are seeing random segfaults in PDO module when running script and not able to diagnose what the issue is. Looking at where the script crashes. If we restart the script it goes does the same process again without issue.

Could this be unhandled loss of DB connection in the PDO driver?

```
(gdb) bt
#0  0x0000563eb45950d1 in _emalloc ()
#1  0x00007fb0b0318c5e in fetch_value () from /opt/remi/php74/root/usr/lib64/php/modules/pdo.so
#2  0x00007fb0b031b2df in do_fetch.constprop.9 () from /opt/remi/php74/root/usr/lib64/php/modules/pdo.so
#3  0x00007fb0b031c673 in zim_PDOStatement_fetchAll () from /opt/remi/php74/root/usr/lib64/php/modules/pdo.so
#4  0x0000563eb4640e5c in execute_ex ()
#5  0x0000563eb45ac016 in zend_call_function ()
#6  0x0000563eb45ac215 in _call_user_function_ex ()
#7  0x00007fb0acc32b52 in _php_worker_function_callback () from /opt/remi/php74/root/usr/lib64/php/modules/gearman.so
#8  0x00007fb0aca09167 in FunctionV1::callback(gearman_job_st*, void*) () from /lib64/libgearman.so.8
#9  0x00007fb0aca11ffc in gearman_worker_work () from /lib64/libgearman.so.8
#10 0x00007fb0acc33e9d in zif_gearman_worker_work () from /opt/remi/php74/root/usr/lib64/php/modules/gearman.so
#11 0x0000563eb4640e5c in execute_ex ()
#12 0x0000563eb4641961 in zend_execute ()
#13 0x0000563eb45ba9b3 in zend_execute_scripts ()
#14 0x0000563eb4559170 in php_execute_script ()
#15 0x0000563eb4643adc in do_cli ()
#16 0x0000563eb43bf7cb in main ()

```


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2021-02-10 20:56 UTC] dharman@php.net
-Status: Open +Status: Feedback
 [2021-02-10 20:56 UTC] dharman@php.net
There doesn't seem to be a lot of information in this bug report that could help us reproduce the issue. Could you provide any more information, please? Can you at least identify minimal reproducible code example? Which MySQL version are you running? The error comes from emalloc so can you let us know your memory limit settings in INI and your system memory (total/free). How large is the result that you are expecting?
 [2021-02-10 21:57 UTC] ricardo dot branco at covermg dot com
-Status: Feedback +Status: Open
 [2021-02-10 21:57 UTC] ricardo dot branco at covermg dot com
The problem does not happen 100% of the time nor is it reproducible on demand, I see it randomly without any specific reason and even when I run it against the same job it crashed on before, it does it successfully.g
This machine is currently only running 3 PHP scripts, they are long running but restarted once per hour just incase of any memory leaks.

The only portion of code I can track this down too is a part that just requests a single row of data from the DB, its nothing large. This is going though ZF1 Zend_Db lib but should not matter here as it seems to be going wrong in the PDO lib.

MariaDB 10.4.15
memory_limit = 64M
No persistent DB connections (long running script, it starts and disconnects conn within the Gearman loop as needed)

Ive registered a tick handler to monitor the memory and additional logging to confirm the area it stopping on.


```
              total        used        free      shared  buff/cache   available
Mem:        1882004      338520      275664      102100     1267820     1260372
Swap:        524284       68392      455892
```


```
[2244970.641111] traps: php[22014] general protection ip:56336a0fb0d1 sp:7ffe3944afc8 error:0
[2249179.087888] traps: php[24166] general protection ip:56407225f0d1 sp:7ffd703631d8 error:0 in php[564072200000+1ff000]
[2249213.888219] traps: php[23898] general protection ip:55b6ea2ba492 sp:7ffe9cf2a290 error:0
[2261617.050534] traps: php[24174] general protection ip:563eb45950d1 sp:7ffd2efb4548 error:0
```
 [2021-02-11 15:12 UTC] ricardo dot branco at covermg dot com
Ive seen the problem again but now it shows up else where but still in PDO.

The first line shows the memory usage just before it started the job which is about 15Mb, this is fairly average usage.
The action immediately after `destfile/loadasset` just pulls one row of data from the DB via Zend_Db, it literally row of data, same as that would be pulled thousands of times in the system, the rows do not vary much in like, only a couple of KB max.


I found this ticket on MariaDB, they seem to say its limited scope of error, but could it be unhandled in 7.4 and cause this issue rather than just throw PDO_Exception?
https://jira.mariadb.org/browse/MDEV-24121


```
15572376 bytes
destjob|11423280|start
1613053297.491174 INFO (6): Starting job545547|djob11423280
1613053297.502392 INFO (6): job545547|djob11423280 - start dispatch
1613053297.503353 INFO (6): job545547|djob11423280 - start transmission s3 (4 queued files)
1613053297.508910 DEBUG (7): job545547|djob11423280 - go() - WN_Upload_Adapter_S3
1613053297.509007 DEBUG (7): job545547|djob11423280 - start() - start - WN_Upload_Adapter_S3
1613053297.509466 INFO (6): job545547|djob11423280 - process_assets() - start
1613053297.510501 INFO (6): job545547|djob11423280 - process_assets() - reset destfiles (0)
1613053297.511303 INFO (6): job545547|djob11423280 - process_assets() - destfile - ttdf_id:202513337 - ast_id:41225350
1613053297.512917 DEBUG (7): job545547|djob11423280 - process_assets() - destfile/loadasset - ttdf_id:202513337 - ast_id:41225350

Program received signal SIGSEGV, Segmentation fault.
0x000055555582d0d1 in _emalloc ()
Missing separate debuginfos, use: debuginfo-install ImageMagick-libs-6.9.11.60-1.el7.remi.x86_64 boost-system-1.53.0-28.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 expat-2.1.0-12.el7.x86_64 fastlz-0.1.0-0.1.20070619svnrev12.el7.remi.x86_64 fftw-libs-double-3.3.3-8.el7.x86_64 fontconfig-2.13.0-4.3.el7.x86_64 freetype-2.8-14.el7_9.1.x86_64 fribidi-1.0.2-1.el7_7.1.x86_64 gd-last-2.3.1-1.el7.remi.x86_64 glib2-2.56.1-8.el7.x86_64 glibc-2.17-323.el7_9.x86_64 gmp-6.0.0-15.el7.x86_64 graphite2-1.3.10-1.el7_3.x86_64 harfbuzz-1.7.5-2.el7.x86_64 jbigkit-libs-2.0-11.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-50.el7.x86_64 lcms2-2.6-3.el7.x86_64 libICE-1.0.9-9.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.7-3.el7_9.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXpm-3.5.12-1.el7.x86_64 libXt-1.1.5-3.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libcurl-7.29.0-59.el7_9.1.x86_64 libedit-3.0-12.20121213cvs.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgearman-1.1.12-18.el7.x86_64 libgomp-4.8.5-44.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu65-65.1-1.el7.remi.x86_64 libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 liblzf-3.6-7.el7.x86_64 libmcrypt-2.5.8-13.el7.x86_64 libmemcached-opt-libs-1.0.18-2.el7.remi.x86_64 libpng-1.5.13-8.el7.x86_64 libraqm-0.7.0-4.el7.x86_64 libselinux-2.5-15.el7.x86_64 libssh2-1.8.0-4.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 libtiff-4.0.3-35.el7.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 libwebp7-1.0.3-1.el7.remi.x86_64 libxcb-1.13-1.el7.x86_64 libxml2-2.9.1-6.el7.5.x86_64 libxslt-1.1.28-6.el7.x86_64 libyaml-0.1.4-11.el7_0.x86_64 libzip5-1.7.3-1.el7.remi.x86_64 libzstd-1.4.7-1.el7.x86_64 lz4-1.8.3-1.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 nspr-4.25.0-2.el7_9.x86_64 nss-3.53.1-3.el7_9.x86_64 nss-pem-1.0.3-7.el7.x86_64 nss-softokn-3.53.1-6.el7_9.x86_64 nss-softokn-freebl-3.53.1-6.el7_9.x86_64 nss-sysinit-3.53.1-3.el7_9.x86_64 nss-util-3.53.1-1.el7_9.x86_64 oniguruma5php-6.9.6-1.el7.remi.x86_64 openldap-2.4.44-22.el7.x86_64 openssl-libs-1.0.2k-21.el7_9.x86_64 pcre-8.32-17.el7.x86_64 php74-php-bcmath-7.4.15-1.el7.remi.x86_64 php74-php-common-7.4.15-1.el7.remi.x86_64 php74-php-gd-7.4.15-1.el7.remi.x86_64 php74-php-gmp-7.4.15-1.el7.remi.x86_64 php74-php-intl-7.4.15-1.el7.remi.x86_64 php74-php-json-7.4.15-1.el7.remi.x86_64 php74-php-ldap-7.4.15-1.el7.remi.x86_64 php74-php-mbstring-7.4.15-1.el7.remi.x86_64 php74-php-mysqlnd-7.4.15-1.el7.remi.x86_64 php74-php-opcache-7.4.15-1.el7.remi.x86_64 php74-php-pdo-7.4.15-1.el7.remi.x86_64 php74-php-pecl-gearman-2.1.0-1.el7.remi.x86_64 php74-php-pecl-igbinary-3.2.1-1.el7.remi.x86_64 php74-php-pecl-imagick-3.4.4-10.el7.remi.x86_64 php74-php-pecl-mcrypt-1.0.4-1.el7.remi.x86_64 php74-php-pecl-memcache-4.0.5.2-1.el7.remi.x86_64 php74-php-pecl-memcached-3.1.5-1.el7.remi.x86_64 php74-php-pecl-msgpack-2.1.2-1.el7.remi.x86_64 php74-php-pecl-redis5-5.3.3-1.el7.remi.x86_64 php74-php-pecl-ssh2-1.2-1.el7.remi.x86_64 php74-php-pecl-yaml-2.2.1-1.el7.remi.x86_64 php74-php-pecl-zip-1.19.2-1.el7.remi.x86_64 php74-php-process-7.4.15-1.el7.remi.x86_64 php74-php-xml-7.4.15-1.el7.remi.x86_64 php74-php-xmlrpc-7.4.15-1.el7.remi.x86_64 sqlite-3.7.17-8.el7_7.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64
(gdb) bt
#0  0x000055555582d0d1 in _emalloc ()
#1  0x00007fffdfc66885 in pdo_mysql_stmt_describe () from /opt/remi/php74/root/usr/lib64/php/modules/pdo_mysql.so
#2  0x00007fffe2b1a89d in pdo_stmt_describe_columns () from /opt/remi/php74/root/usr/lib64/php/modules/pdo.so
#3  0x00007fffe2b1aeec in zim_PDOStatement_execute () from /opt/remi/php74/root/usr/lib64/php/modules/pdo.so
#4  0x00005555558d8e5c in execute_ex ()
#5  0x0000555555844016 in zend_call_function ()
#6  0x0000555555844215 in _call_user_function_ex ()
#7  0x00007fffdf432b52 in _php_worker_function_callback () from /opt/remi/php74/root/usr/lib64/php/modules/gearman.so
#8  0x00007fffdf209167 in FunctionV1::callback(gearman_job_st*, void*) () from /lib64/libgearman.so.8
#9  0x00007fffdf211ffc in gearman_worker_work () from /lib64/libgearman.so.8
#10 0x00007fffdf433e9d in zif_gearman_worker_work () from /opt/remi/php74/root/usr/lib64/php/modules/gearman.so
#11 0x00005555558d8e5c in execute_ex ()
#12 0x00005555558d9961 in zend_execute ()
#13 0x00005555558529b3 in zend_execute_scripts ()
#14 0x00005555557f1170 in php_execute_script ()
#15 0x00005555558dbadc in do_cli ()
#16 0x00005555556577cb in main ()

```
 [2021-02-16 14:51 UTC] nikic@php.net
A crash in _emalloc usually indicates that some kind of memory corruption happened at an earlier point in time -- hard to guess where it comes from though.

If this is a standalone script, running it through "USE_ZEND_ALLOC=0 valgrind php script.php" usually provides more actionable information.
 [2021-04-14 20:55 UTC] standus at post dot cz
I have very similar big problem which I reported here.
https://jira.mariadb.org/browse/MDEV-25414
 [2021-04-15 07:37 UTC] standus at post dot cz
On PHP 8.x its same problem.
 [2021-04-16 08:57 UTC] standus at post dot cz
Somebody has similar problem too.
https://laracasts.com/discuss/channels/laravel/pdo-hangs-on-executing-query-on-heavy-queue-system-timeout-does-not-work
 [2021-04-20 04:37 UTC] standus at post dot cz
I replaced PDo with mysqli library and its same problem.
It look like some middleware centos library between php and madiadb is wrong.
 [2021-04-20 13:04 UTC] standus at post dot cz
I'm probably on the trail of that problem.
From phpinfo I see: 

On problematic CentOS 7.x:
PDO Driver for MySQL	enabled
Client API version	mysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $
Which is very very old.

On my develop machine (it works well) is:
PDO Driver for MySQL	enabled
Client API version	mysqlnd 7.4.6

Another test look onCentOS good when I use: 
PDO::ATTR_EMULATE_PREPARES = true;
 [2023-07-31 09:29 UTC] quanganhkop01 at gmail dot com
Thanks for the information.  (https://github.com.php)(https://www.mythdhr.ltd/)
 [2024-05-31 09:36 UTC] mike158stamp at outlook dot com
Thanks for sharing this amazing post. (https://github.com)(https://www.receiptify.net)
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Nov 21 12:01:29 2024 UTC