|
php.net | support | documentation | report a bug | advanced search | search howto | statistics | random bug | login |
[2021-02-28 19:41 UTC] theultramage at gmail dot com
Description: ------------ apache 2.4 with Event MPM + mod_php 8.0.2 with ZTS currently does not work on FreeBSD 12.0-12.2, while trying to load libphp.so the dynamic loader reports "No space available for static Thread Local Storage". PHP 7.3.20 on the same system does not exhibit this problem. So it seems to be caused by code changes or perhaps build system changes. I found one related php bugreport - https://bugs.php.net/bug.php?id=71189 - but it was talking about php 7.0 on FreeBSD 8.1, a super outdated OS from 2009, which makes me wonder if that would even build, so maybe that part was entered wrong. There is no developer feedback in that thread and the only suggestion is to fall back to single-threaded prefork mode. This issue was brought up on the freebsd bugtracker for mod_php80 last year - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250652 - but the maintainer dismissed it as not a php bug and no further attention was given to it. Followup comments imply that php-fpm is also affected so it's not just a mod_php thing. Another comment suggests recompiling kernel+world with a bigger RTLD_STATIC_TLS_EXTRA constant, based on an old issue in one bsd fork. There are very few results when searching for that error message. The issue is somehow related to thread local storage and dynamically loaded modules. I tried a small C++ test case that involved dynamically loading a shared library with threading and large arrays declared as thread_local, but it worked fine. Whatever the issue is, it's not as simple. PatchesPull RequestsHistoryAllCommentsChangesGit/SVN commits
|
|||||||||||||||||||||||||||||||||||||
Copyright © 2001-2025 The PHP GroupAll rights reserved. |
Last updated: Fri Oct 24 06:00:01 2025 UTC |
Through further reading and testing, I have identified the cause and prepared a small code sample. It is indeed tied to dynamic loading and thread local storage, specifically the use of the initial-exec tls model by TSRM and modules that rely on it. TSRM provides a 'cache' in the form of a TLS void* (8 bytes) marked as __attribute__((tls_model("initial-exec"))), and a collection of macros to work with it. The initial-exec model is meant for shared libraries that load along with the main executable, and not for dynamic loading, but FreeBSD does reserve a bit of extra space (128 bytes) if for some obscure reason a shared library flagged as DF_STATIC_TLS needs to be loaded dynamically. This value is a compile-time constant in the dynamic linker, and applies to the total static tls size across all loaded modules. That is not a lot of room to work with. In my opinion, it feels like it's not meant for general-purpose use, and using a dynamically loaded library like this may mean wandering into undefined behavior territory. I'd study the specs real hard before choosing this approach. In my custom installation, using 'readelf' I counted 11 ext modules flagged as STATIC_TLS and having an 8-byte TLS section. Adding 16 bytes used by mod_php itself, that's 104. So if 4 more modules like this are added, or the existing ones start using TSRMLS_CACHE, mod_php will fail to load. The initial-exec shenanigans were added in https://github.com/php/php-src/commit/2aefd112114bd150f5dbba0be6d0f8601561da4e with minimal information provided in the commit. So at first glance it seems like dubious premature optimization that may also be touching on undefined behavior. The whole tls pointer cache comes from edits 7 years ago and is also where that one overlong out-of-place define that litters all the makefiles came from. See https://github.com/php/php-src/commit/8aeffdd74c826b5012c9b052848cfa8e593776ed https://github.com/php/php-src/commit/b3aebda9eaf55706af2e21178f229a171725a168 At this point in time, I would question if this is still providing any performance improvement over what clang/gcc can do, or if it's hindering performance. It's definitely making the code harder to read. And then there are ignored bugreports like https://bugs.php.net/bug.php?id=50238 which seems to imply that the array juggling macros in TSRM.h alone might have been causing a measurable performance drop. I also happened to find this 2009 page https://wiki.php.net/rfc/tls which may be the origin of all this. The more immediate issue is this: libphp.so contains a 328-byte TLS section and the module is marked as STATIC_TLS. This file is thus guaranteed to fail to load dynamically on current FreeBSD. And now I can't start my webserver anymore. Through test code, I have determined that if any thread-local variable is flagged as initial-exec, all of the other thread-local variables become like that as well - I'm guessing the ELF format doesn't support multi-mode TLS. This means that all the other otherwise innocent globals, marked as ZEND_TLS, get counted toward the static tls limit. The .symtab section of libphp.so is not present even in debug builds, so I can't check to make sure, but the number sounds about right based on what I've seen in the code. I was not able to get rid of the STATIC_TLS flag on libphp.so just by patching TSRM.h, even though that's the only place that uses this attribute, and none of the makefiles pass -ftls-model. Turns out there is hand-crafted assembly code that directly references GOTTPOFF in one helper function in TSRM.c that is used in ext/opcache jit code. It was added by the same person who did the initial-exec stuff, in https://github.com/php/php-src/commit/9a06876072b9ccb023d4a14426ccb587f10882f3 Once that was dealt with, libphp.so became usable again for me. Test script: --------------- // test.c: cc test.c -o test #include <dlfcn.h> #include <stdio.h> int main() { void* h = dlopen("./ext.so", RTLD_LAZY); if( h == NULL ) { puts(dlerror()); return 0; } return 0; } // ext.c: cc -shared -fpic ext.c -o ext.so static __thread char v[129] __attribute__((tls_model("initial-exec"))); void dummy() { v[0] = 0; } // force referenceAh. I see. Your comment showed that the situation is trickier than I originally considered. I will try to clarify things based on what I've learned so far. tls_model("initial-exec") is meant to be used in code whose memory layout can be resolved during program initialization. So ordinary executables, static .a libraries, or dynamic .so modules which are hard-bound to the main executable via import table. It's not meant for true dynamic libraries, loaded via dlopen(). Or so I've read. OSes do seem to have a way of making this work anyway, in a limited manner. FreeBSD prepares 128 bytes of space in every process. I also tested Ubuntu 20, and found that the limit was 1720 bytes. I do not know if this is just an improvised compat mechanism, or if it is something documented and fully endorsed by the ABI specification. I have not consulted this with core devs on the mailing list yet. The opinion from #clang is that this should not be used. The limit is global for the whole process. If each module needs 8 bytes, then at most 16 such modules can be loaded. Currently I am counting 33 php extensions that actively use ZEND_TSRMLS_CACHE_DEFINE() (and 18 that #include TSRM.h but don't use it?), so I don't believe it's currently possible to have them all dynamically loaded on FreeBSD. Though that is not a good enough reason to disable the optimization entirely, if it indeed leads to observable performance improvement. I believe that for cli/cgi/fpm, a workaround would be to have a bunch of the extensions compiled-in. But then again, I'm realizing that these three appear to be single-threaded static executables that do not need ZTS or thread local variables, so non of this matters for them. The other problem that I described (libphp.so requiring 328 bytes of static tls storage) thus turns out to be limited to mod_php, which is a dynamically loaded shared library. The reason is that if any piece of code requires the "initial-exec" static tls model, then the whole shared library is flagged as such. Here, not only does zend.c's use of TSRMLS_CACHE do it, but so will any threaded extension that gets compiled in. Furthermore, the mere presence of 'gottpoff' (and perhaps 'ntpoff') in inline assembly code will force static tls mode. Regarding your patch, it looks very close to what I arrived at on my end. For opcache I also patched out the i386 branch just in case. It makes me wonder though, why does tsrm_get_ls_cache_tcb_offset() in TSRM.c even exist, when opcache completely replicates it? An alternative way, that keeps the extensions as STATIC_TLS, would be to remove that assembly code from TSRM.c, and disable that TSRM_TLS_MODEL_ATTR on libphp.so itself and any extensions that get compiled in, while still applying it on shared extensions somehow.That workaround might be incomplete. We are seeing lots of random crashes like below with ZEND mod_php 8.1: Program terminated with signal SIGSEGV, Segmentation fault. Sent by kill() from pid 55159 and user 80. #0 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca870, context=0x7fffd99ca500) at Zend/zend_signal.c:96 96 if (EXPECTED(SIGG(active))) { [Current thread is 1 (LWP 112544)] (gdb) bt #0 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca870, context=0x7fffd99ca500) at Zend/zend_signal.c:96 #1 0x0000000800676cbe in ?? () from /lib/libthr.so.3 #2 0x000000080067627f in ?? () from /lib/libthr.so.3 #3 <signal handler called> #4 0x000000080093f6aa in _kevent () from /lib/libc.so.7 #5 0x0000000800674103 in ?? () from /lib/libthr.so.3 #6 0x00000008006366e0 in ?? () from /usr/local/lib/libapr-1.so.0 #7 0x0000000800ff5fdb in listener_thread (thd=0x804cbfc90, dummy=0x804b8b0c8) at event.c:1786 #8 0x0000000000277171 in thread_start (thread=0x804cbfc90, data=0x804cbfc80) at util.c:3210 #9 0x00000008006710ec in ?? () from /lib/libthr.so.3 (gdb) info threads Id Target Id Frame * 1 LWP 112544 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca870, context=0x7fffd99ca500) at Zend/zend_signal.c:96 2 LWP 101671 0x00007ffffffff190 in ?? () 3 LWP 110679 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 4 LWP 110681 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 5 LWP 110684 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 6 LWP 110693 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 7 LWP 110696 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 8 LWP 110701 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 9 LWP 110703 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 10 LWP 110707 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 11 LWP 110726 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 12 LWP 110807 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 13 LWP 110808 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 14 LWP 111904 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 15 LWP 111907 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 16 LWP 112023 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 17 LWP 112024 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 18 LWP 112032 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 19 LWP 112214 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 20 LWP 112290 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 21 LWP 112398 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 22 LWP 112399 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 23 LWP 112400 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 24 LWP 112401 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 25 LWP 112402 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 26 LWP 112403 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 27 LWP 112404 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 28 LWP 112406 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 29 LWP 112408 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 30 LWP 112410 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 31 LWP 112411 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 32 LWP 112412 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 33 LWP 112470 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 34 LWP 112471 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 35 LWP 112473 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 36 LWP 112476 0x00000008046b99c1 in libunwind::CFI_Parser<libunwind::LocalAddressSpace>::decodeFDE(libunwind::LocalAddressSpace&, unsigned long, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info*, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info*) () from /lib/libgcc_s.so.1 37 LWP 112481 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 38 LWP 112482 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 39 LWP 112483 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 40 LWP 112484 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 41 LWP 103438 0x0000000800672ff0 in _thr_umtx_timedwait_uint () from /lib/libthr.so.3 42 LWP 112533 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 43 LWP 102886 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 44 LWP 112534 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 45 LWP 112535 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 46 LWP 112536 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 47 LWP 112537 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 48 LWP 112538 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 49 LWP 112539 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 50 LWP 112541 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 51 LWP 112542 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 52 LWP 112543 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 Program terminated with signal SIGSEGV, Segmentation fault. Sent by kill() from pid 55931 and user 80. #0 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca5f0, context=0x7fffd99ca280) at Zend/zend_signal.c:96 96 if (EXPECTED(SIGG(active))) { [Current thread is 1 (LWP 112652)] (gdb) bt #0 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca5f0, context=0x7fffd99ca280) at Zend/zend_signal.c:96 #1 0x0000000800676cbe in ?? () from /lib/libthr.so.3 #2 0x000000080067627f in ?? () from /lib/libthr.so.3 #3 <signal handler called> #4 0x0000000800956d4a in _read () from /lib/libc.so.7 #5 0x0000000800673c46 in ?? () from /lib/libthr.so.3 #6 0x000000080062b36a in apr_file_read () from /usr/local/lib/libapr-1.so.0 #7 0x000000080063839e in apr_poll_drain_wakeup_pipe () from /usr/local/lib/libapr-1.so.0 #8 0x000000080063681f in ?? () from /usr/local/lib/libapr-1.so.0 #9 0x0000000800ff5fdb in listener_thread (thd=0x804cbfc90, dummy=0x804b8b0c8) at event.c:1786 #10 0x0000000000277171 in thread_start (thread=0x804cbfc90, data=0x804cbfc80) at util.c:3210 #11 0x00000008006710ec in ?? () from /lib/libthr.so.3 #12 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffd99cb000 (gdb) info threads Id Target Id Frame * 1 LWP 112652 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca5f0, context=0x7fffd99ca280) at Zend/zend_signal.c:96 2 LWP 104804 0x00007ffffffff190 in ?? () 3 LWP 100778 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 4 LWP 100808 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 5 LWP 102432 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 6 LWP 104965 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 7 LWP 111908 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 8 LWP 111909 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 9 LWP 111910 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 10 LWP 111912 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 11 LWP 111913 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 12 LWP 111914 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 13 LWP 111915 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 14 LWP 111916 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 15 LWP 111917 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 16 LWP 111919 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 17 LWP 111998 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 18 LWP 111999 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 19 LWP 112001 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 20 LWP 112003 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 21 LWP 112009 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 22 LWP 112013 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 23 LWP 112016 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 24 LWP 112019 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 25 LWP 112021 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 26 LWP 112033 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 27 LWP 112034 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 28 LWP 112624 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 29 LWP 112625 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 30 LWP 112626 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 31 LWP 112627 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 32 LWP 112628 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 33 LWP 112630 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 34 LWP 112631 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 35 LWP 112632 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 36 LWP 112633 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 37 LWP 112634 0x000000080067c2bc in ?? () from /lib/libthr.so.3 38 LWP 112635 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 39 LWP 112636 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 40 LWP 112637 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 41 LWP 112638 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 42 LWP 112639 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 43 LWP 112640 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 44 LWP 112641 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 45 LWP 112642 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 46 LWP 112643 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 47 LWP 112644 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 48 LWP 112645 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 49 LWP 112646 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 50 LWP 112647 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 51 LWP 112650 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 52 LWP 112651 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 Sent by kill() from pid 5954 and user 80. #0 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca4f0, context=0x7fffd99ca180) at Zend/zend_signal.c:96 96 if (EXPECTED(SIGG(active))) { [Current thread is 1 (LWP 106356)] (gdb) bt #0 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca4f0, context=0x7fffd99ca180) at Zend/zend_signal.c:96 #1 0x0000000800676cbe in ?? () from /lib/libthr.so.3 #2 0x000000080067627f in ?? () from /lib/libthr.so.3 #3 <signal handler called> #4 0x00000008008b55c4 in ?? () from /lib/libc.so.7 #5 0x00000008008b6ab9 in ?? () from /lib/libc.so.7 #6 0x000000080087e575 in __je_tcache_bin_flush_small () from /lib/libc.so.7 #7 0x00000008008beb8a in free () from /lib/libc.so.7 #8 0x000000080062f45d in apr_pool_destroy () from /usr/local/lib/libapr-1.so.0 #9 0x000000000028c7e8 in ap_queue_info_free_idle_pools (queue_info=0x804d2f3c8) at mpm_fdqueue.c:293 #10 0x0000000800ff7c78 in close_listeners (closed=0x7fffd99caf7c) at event.c:1280 #11 0x0000000800ff5477 in listener_thread (thd=0x804d30c90, dummy=0x804bf80c8) at event.c:1683 #12 0x0000000000277171 in thread_start (thread=0x804d30c90, data=0x804d30c80) at util.c:3210 #13 0x00000008006710ec in ?? () from /lib/libthr.so.3 #14 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffd99cb000 (gdb) info threads Id Target Id Frame * 1 LWP 106356 0x00000008029ad9ca in zend_signal_handler_defer (signo=1, siginfo=0x7fffd99ca4f0, context=0x7fffd99ca180) at Zend/zend_signal.c:96 2 LWP 109808 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 3 LWP 106065 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 4 LWP 106066 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 5 LWP 106070 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 6 LWP 106072 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 7 LWP 106082 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 8 LWP 106083 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 9 LWP 106092 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 10 LWP 106094 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 11 LWP 106095 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 12 LWP 106096 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 13 LWP 106108 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 14 LWP 106111 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 15 LWP 106116 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 16 LWP 106117 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 17 LWP 106122 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 18 LWP 106123 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 19 LWP 106127 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 20 LWP 106129 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 21 LWP 106131 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 22 LWP 106136 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 23 LWP 106152 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 24 LWP 106156 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 25 LWP 106166 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 26 LWP 106171 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 27 LWP 106175 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 28 LWP 106203 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 29 LWP 106204 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 30 LWP 106209 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 31 LWP 106212 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 32 LWP 106215 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 33 LWP 106260 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 34 LWP 106284 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 35 LWP 106288 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 36 LWP 106291 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 37 LWP 106293 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 38 LWP 106301 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 39 LWP 106313 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 40 LWP 106327 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 41 LWP 106328 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 42 LWP 106330 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 43 LWP 106332 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 44 LWP 106334 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 45 LWP 106338 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 46 LWP 106339 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 47 LWP 106340 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 48 LWP 106341 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 49 LWP 106345 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 50 LWP 106349 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 51 LWP 106350 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3 52 LWP 106353 0x000000080067f7cc in _umtx_op_err () from /lib/libthr.so.3