shibd Aborted, No Error in Logs
Muthukumar Manohar, Emily (ELS-CON)
e.muthukumarmanohar at elsevier.com
Wed Jul 16 14:29:40 UTC 2025
Dear Shibboleth Community,
I am seeking advice regarding an issue with our Shibboleth deployment (v3.5.0 on Amazon Linux aarch64 (AL3)). Our SHibboleth SP instances are going unhealthy and upon investigation, I dont find any erro/exception in both tomcat and application logs. But when I executed sudo netstat -tulnp, I found that shibd that was running on 25009 port got terminated. We have observed that the `shibd` process occasionally terminates with SIGABRT while idle in the `select()` system call. The core dump analysis shows all threads waiting (either in `select()` or on condition variables), and there are no error or abort messages in `shibd.log`—only normal startup and configuration load entries.
**Commands Executed and Results:**
1.
Enabled core dumps and analyzed with gdb:
gdb /usr/sbin/shibd /path/to/core (gdb) bt
Output:
1.
#0 0x0000ffffa8323328 in __GI___select (nfds=4, readfds=0xffffd0f0b568, writefds=0x0, exceptfds=0x0, timeout=0xffffd0f0b558) at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x0000ffffa8f4aa24 in shibsp::SocketListener::run (this=0xaaaae16931b0, shutdown=<optimized out>) at remoting/impl/SocketListener.cpp:255
#2 0x0000aaaad11807ac in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/shibboleth-3.5.0-1.amzn2023.aarch64/shibd/shibd.cpp:465
1.
Inference from copilot:
The crash happened in __GI___select (the system select() call). Called from shibsp::SocketListener::run in SocketListener.cpp:255. Ultimately from main in shibd.cpp:465. shibd was waiting for socket activity (likely handling requests or inter-process communication). The process was aborted (SIGABRT) while inside the select() call. This often points to a deadlock, resource exhaustion, or an explicit abort triggered by the application (e.g., via abort() or a failed assertion).
1.
Checked all threads:
(gdb) info threads
Output:
1.
Id Target Id Frame
1 Thread 0xffffa905d020 (LWP 874440) 0x0000ffffa8323328 in __GI___select (nfds=4, readfds=0xffffd0f0b568, writefds=0x0, exceptfds=0x0, timeout=0xffffd0f0b558)
at ../sysdeps/unix/sysv/linux/select.c:69
2 Thread 0xffff9efdef40 (LWP 874445) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9efde548, op=393, expected=0,
futex_word=0xaaaae1ac74f0) at futex-internal.c:57
3 Thread 0xffff9e7cef40 (LWP 874452) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9e7ce548, op=393, expected=0,
futex_word=0xaaaae1ab32c0) at futex-internal.c:57
4 Thread 0xffff9dfbef40 (LWP 874453) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-310656592, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaaed7bc1b0) at futex-internal.c:57
5 Thread 0xffff9d7aef40 (LWP 874454) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-310572768, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaaed7d0920) at futex-internal.c:57
6 Thread 0xffff7fffef40 (LWP 876847) 0x0000ffffa8323328 in __GI___select (nfds=5, readfds=0xffff7fffe558, writefds=0x0, exceptfds=0x0, timeout=0xffff7fffe548)
at ../sysdeps/unix/sysv/linux/select.c:69
7 Thread 0xffffa4c4df40 (LWP 874441) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-513560400, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaae163b0b0) at futex-internal.c:57
8 Thread 0xffff9fffef40 (LWP 874442) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9fffe5e8, op=393, expected=0,
futex_word=0xaaaae1699cf0) at futex-internal.c:57
9 Thread 0xffff9f7eef40 (LWP 874443) 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-513509696, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaae16476c0) at futex-internal.c:57
1.
Inference from copilot:
Your info threads output shows: Thread 1 (main) is blocked in select(). Thread 6 is also blocked in select(). All other threads are waiting in __futex_abstimed_wait_common64 (futex wait), which means they are waiting for locks or events. Interpretation: The process is mostly idle or waiting for events. The main thread is blocked in select(), which is normal for a server waiting for socket activity. The crash (SIGABRT) happened while waiting, not during active processing.
1.
(gdb)thread apply all bt
Output:
1.
Thread 9 (Thread 0xffff9f7eef40 (LWP 874443)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-513509696, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae16476c0) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=-513509696, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaae16476c0) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaae16476c0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0x0, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd2c8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff90000b68, cond=0xaaaae1647698) at pthread_cond_wait.c:427
#4 ___pthread_cond_wait (cond=0xaaaae1647698, mutex=0xffff90000b68) at pthread_cond_wait.c:459
#5 0x0000ffffa79030c4 in xmltooling::CondWaitImpl::wait (mutex=0xffff90000b60, this=<optimized out>) at util/PThreads.cpp:99
#6 xmltooling::CondWaitImpl::wait (mutex=0xffff90000b60, this=<optimized out>) at util/PThreads.cpp:95
#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae172cbc0) at util/ReloadableXMLFile.cpp:304
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0aa2f) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 8 (Thread 0xffff9fffef40 (LWP 874442)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9fffe5e8, op=393, expected=0, futex_word=0xaaaae1699cf0) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff9fffe5e8, clockid=0, expected=0, futex_word=0xaaaae1699cf0) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaae1699cf0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0xffff9fffe5e8, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd4c8 in __pthread_cond_wait_common (abstime=0xffff9fffe5e8, clockid=0, mutex=0xffff98000b68, cond=0xaaaae1699cc8) at pthread_cond_wait.c:427
#4 ___pthread_cond_timedwait64 (cond=0xaaaae1699cc8, mutex=0xffff98000b68, abstime=0xffff9fffe5e8) at pthread_cond_wait.c:484
#5 0x0000ffffa792cc04 in xmltooling::CondWaitImpl::timedwait (delay_seconds=<optimized out>, mutex=0xffff98000b60, this=0xaaaae1699cc0) at util/PThreads.cpp:110
#6 xmltooling::CondWaitImpl::timedwait (delay_seconds=<optimized out>, mutex=0xffff98000b60, this=0xaaaae1699cc0) at util/PThreads.cpp:103
#7 xmltooling::MemoryStorageService::cleanup_fn (pv=0xaaaae163b240) at impl/MemoryStorageService.cpp:172
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0ad0f) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 7 (Thread 0xffffa4c4df40 (LWP 874441)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-513560400, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae163b0b0) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=-513560400, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaae163b0b0) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaae163b0b0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0x0, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd2c8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa0000b68, cond=0xaaaae163b088) at pthread_cond_wait.c:427
#4 ___pthread_cond_wait (cond=0xaaaae163b088, mutex=0xffffa0000b68) at pthread_cond_wait.c:459
#5 0x0000ffffa79030c4 in xmltooling::CondWaitImpl::wait (mutex=0xffffa0000b60, this=<optimized out>) at util/PThreads.cpp:99
#6 xmltooling::CondWaitImpl::wait (mutex=0xffffa0000b60, this=<optimized out>) at util/PThreads.cpp:95
#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae15778a8) at util/ReloadableXMLFile.cpp:304
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0ac5f) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 6 (Thread 0xffff7fffef40 (LWP 876847)):
#0 0x0000ffffa8323328 in __GI___select (nfds=5, readfds=0xffff7fffe558, writefds=0x0, exceptfds=0x0, timeout=0xffff7fffe548) at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x0000ffffa8f4c2d4 in shibsp::ServerThread::run (this=this at entry=0xaaaaed79a670) at remoting/impl/SocketListener.cpp:544
#2 0x0000ffffa8f4c5c4 in server_thread_fn (arg=0xaaaaed79a670) at remoting/impl/SocketListener.cpp:495
--Type <RET> for more, q to quit, c to continue without paging--c
#3 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0b1ef) at pthread_create.c:443
#4 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 5 (Thread 0xffff9d7aef40 (LWP 874454)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-310572768, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaaed7d0920) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=-310572768, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaaed7d0920) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaaed7d0920, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0x0, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd2c8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff80000b68, cond=0xaaaaed7d08f8) at pthread_cond_wait.c:427
#4 ___pthread_cond_wait (cond=0xaaaaed7d08f8, mutex=0xffff80000b68) at pthread_cond_wait.c:459
#5 0x0000ffffa79030c4 in xmltooling::CondWaitImpl::wait (mutex=0xffff80000b60, this=<optimized out>) at util/PThreads.cpp:99
#6 xmltooling::CondWaitImpl::wait (mutex=0xffff80000b60, this=<optimized out>) at util/PThreads.cpp:95
#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaaed7ba2b8) at util/ReloadableXMLFile.cpp:304
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0a61f) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 4 (Thread 0xffff9dfbef40 (LWP 874453)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=-310656592, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaaed7bc1b0) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=-310656592, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaaed7bc1b0) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaaed7bc1b0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0x0, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd2c8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff8c000b68, cond=0xaaaaed7bc188) at pthread_cond_wait.c:427
#4 ___pthread_cond_wait (cond=0xaaaaed7bc188, mutex=0xffff8c000b68) at pthread_cond_wait.c:459
#5 0x0000ffffa79030c4 in xmltooling::CondWaitImpl::wait (mutex=0xffff8c000b60, this=<optimized out>) at util/PThreads.cpp:99
#6 xmltooling::CondWaitImpl::wait (mutex=0xffff8c000b60, this=<optimized out>) at util/PThreads.cpp:95
#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaaed0a96f8) at util/ReloadableXMLFile.cpp:304
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0a61f) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 3 (Thread 0xffff9e7cef40 (LWP 874452)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9e7ce548, op=393, expected=0, futex_word=0xaaaae1ab32c0) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff9e7ce548, clockid=0, expected=0, futex_word=0xaaaae1ab32c0) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaae1ab32c0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0xffff9e7ce548, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd4c8 in __pthread_cond_wait_common (abstime=0xffff9e7ce548, clockid=0, mutex=0xffff88000b98, cond=0xaaaae1ab3298) at pthread_cond_wait.c:427
#4 ___pthread_cond_timedwait64 (cond=0xaaaae1ab3298, mutex=0xffff88000b98, abstime=0xffff9e7ce548) at pthread_cond_wait.c:484
#5 0x0000ffffa7903174 in xmltooling::CondWaitImpl::timedwait (delay_seconds=<optimized out>, mutex=0xffff88000b90, this=<optimized out>) at util/PThreads.cpp:110
#6 xmltooling::CondWaitImpl::timedwait (delay_seconds=<optimized out>, mutex=0xffff88000b90, this=<optimized out>) at util/PThreads.cpp:103
#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae182ea70) at util/ReloadableXMLFile.cpp:306
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0abaf) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 2 (Thread 0xffff9efdef40 (LWP 874445)):
#0 0x0000ffffa82ba994 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9efde548, op=393, expected=0, futex_word=0xaaaae1ac74f0) at futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff9efde548, clockid=0, expected=0, futex_word=0xaaaae1ac74f0) at futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0xaaaae1ac74f0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0xffff9efde548, private=private at entry=0) at futex-internal.c:139
#3 0x0000ffffa82bd4c8 in __pthread_cond_wait_common (abstime=0xffff9efde548, clockid=0, mutex=0xffff94000c48, cond=0xaaaae1ac74c8) at pthread_cond_wait.c:427
#4 ___pthread_cond_timedwait64 (cond=0xaaaae1ac74c8, mutex=0xffff94000c48, abstime=0xffff9efde548) at pthread_cond_wait.c:484
#5 0x0000ffffa7903174 in xmltooling::CondWaitImpl::timedwait (delay_seconds=<optimized out>, mutex=0xffff94000c40, this=<optimized out>) at util/PThreads.cpp:110
#6 xmltooling::CondWaitImpl::timedwait (delay_seconds=<optimized out>, mutex=0xffff94000c40, this=<optimized out>) at util/PThreads.cpp:103
#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae16bff20) at util/ReloadableXMLFile.cpp:306
#8 0x0000ffffa82bdd78 in start_thread (arg=0xffffd0f0abaf) at pthread_create.c:443
#9 0x0000ffffa832addc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread 1 (Thread 0xffffa905d020 (LWP 874440)):
#0 0x0000ffffa8323328 in __GI___select (nfds=4, readfds=0xffffd0f0b568, writefds=0x0, exceptfds=0x0, timeout=0xffffd0f0b558) at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x0000ffffa8f4aa24 in shibsp::SocketListener::run (this=0xaaaae16931b0, shutdown=<optimized out>) at remoting/impl/SocketListener.cpp:255
#2 0x0000aaaad11807ac in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/shibboleth-3.5.0-1.amzn2023.aarch64/shibd/shibd.cpp:465
1.
Inference from copilot:
Main thread and one other are blocked in select() (waiting for socket activity). All other threads are waiting on condition variables in xmltooling::CondWaitImpl::wait or timedwait, mostly inside ReloadableXMLFile::reload_fn and MemoryStorageService::cleanup_fn.No thread is actively processing; all are waiting for events or signals.
1.
The abort (SIGABRT) happened while the main thread was idle in select().No evidence of a deadlock or crash in application code—just waiting.
1.
The process was likely terminated by an external signal (e.g., kill -6/SIGABRT), or by an internal assertion/abort in the code not shown in the stack trace.
1.
This is consistent with resource exhaustion, watchdog timeout, or a deliberate abort due to a failed check.
1.
Reviewed shibd logs:
Only normal startup and config load messages, no errors or aborts.
2.
Checked system logs for OOM killer, kernel, or watchdog events:
sudo dmesg | grep -i -E 'oom|kill|error|fail|watchdog'
sudo journalctl -xe | grep -i -E 'oom|kill|error|fail|watchdog'
sudo grep -i -E 'oom|kill|error|fail|watchdog' /var/log/messages
Output: No OOM killer or watchdog events found.
We have verified EFS/NFS mount health and system resources, and have enabled core dumps.
But still we haven't found out the root cause for shibd termination. Has anyone encountered similar behavior, or can suggest additional diagnostics or configuration changes to help identify the root cause? Any guidance would be greatly appreciated.
Thank you,
Emily Sherline M.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20250716/1dae0764/attachment.htm>
More information about the users
mailing list