<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Dear Shibboleth Community,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I am seeking advice regarding an issue with our Shibboleth deployment (v3.5.0 on Amazon Linux aarch64 (AL3)). Our SHibboleth SP instances are going unhealthy and upon investigation, I dont find any erro/exception in both tomcat and application logs. But when
I executed <i>sudo netstat -tulnp</i>, I found that shibd that was running on 25009 port got terminated. We have observed that the `shibd` process occasionally terminates with SIGABRT while idle in the `select()` system call. The core dump analysis shows all
threads waiting (either in `select()` or on condition variables), and there are no error or abort messages in `shibd.log`—only normal startup and configuration load entries.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
**Commands Executed and Results:**</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="1">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div role="presentation" class="elementToProof"><b><u>Enabled core dumps and analyzed with gdb:</u></b></div>
<div role="presentation" class="elementToProof"><i>gdb /usr/sbin/shibd /path/to/core (gdb) bt</i></div>
<div role="presentation" class="elementToProof"><b>Output</b>:</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="0">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="background-color: rgb(31, 31, 31);" role="presentation" class="elementToProof">
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa8323328</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__GI___select</span><span style="color: rgb(204, 204, 204);"> (nfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">4</span><span style="color: rgb(204, 204, 204);">,
readfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffffd0f0b568</span><span style="color: rgb(204, 204, 204);">, writefds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
exceptfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">, timeout</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffffd0f0b558</span><span style="color: rgb(204, 204, 204);">)
at ..</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">sysdeps</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">unix</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">sysv</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">linux</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(156, 220, 254);">select</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">69</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#</span><span style="color: rgb(181, 206, 168);">1</span><span style="color: rgb(204, 204, 204);">
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa8f4aa24</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(78, 201, 176);">shibsp</span><span style="color: rgb(204, 204, 204);">::</span><span style="color: rgb(78, 201, 176);">SocketListener</span><span style="color: rgb(204, 204, 204);">::</span><span style="color: rgb(220, 220, 170);">run</span><span style="color: rgb(204, 204, 204);"> (</span><span style="color: rgb(86, 156, 214);">this</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaae16931b0</span><span style="color: rgb(204, 204, 204);">,
shutdown</span><span style="color: rgb(212, 212, 212);">=<</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">) at remoting</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">impl</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(156, 220, 254);">SocketListener</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">cpp</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">255</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#</span><span style="color: rgb(181, 206, 168);">2</span><span style="color: rgb(204, 204, 204);">
</span><span style="color: rgb(181, 206, 168);">0x0000aaaad11807ac</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">main</span><span style="color: rgb(204, 204, 204);"> (argc</span><span style="color: rgb(212, 212, 212);">=<</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">,
argv</span><span style="color: rgb(212, 212, 212);">=<</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">) at
</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">usr</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">src</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">debug</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">shibboleth</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(244, 71, 71);">3.5.0</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(244, 71, 71);">1.amzn2023.aarch64</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">shibd</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(156, 220, 254);">shibd</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">cpp</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">465</span></div>
</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="1">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); display: block;">
<div role="presentation" class="elementToProof"><b>Inference from copilot:</b></div>
<div role="presentation" class="elementToProof">The crash happened in __GI___select (the system select() call). Called from shibsp::SocketListener::run in SocketListener.cpp:255. Ultimately from main in shibd.cpp:465. shibd was waiting for socket activity (likely
handling requests or inter-process communication). The process was aborted (SIGABRT) while inside the select() call. This often points to a deadlock, resource exhaustion, or an explicit abort triggered by the application (e.g., via abort() or a failed assertion).</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="2">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div role="presentation" class="elementToProof"><b><u>Checked all threads:</u></b></div>
<div role="presentation" class="elementToProof"><i>(gdb) info threads</i></div>
<div role="presentation" class="elementToProof"><b>Output: </b></div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="0">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="background-color: rgb(31, 31, 31);" role="presentation" class="elementToProof">
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
Id Target Id Frame</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">1</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffffa905d020</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874440</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa8323328</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__GI___select</span><span style="color: rgb(204, 204, 204);"> (nfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">4</span><span style="color: rgb(204, 204, 204);">,
readfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffffd0f0b568</span><span style="color: rgb(204, 204, 204);">, writefds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
exceptfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">, timeout</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffffd0f0b558</span><span style="color: rgb(204, 204, 204);">)</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">at ..</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">sysdeps</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">unix</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">sysv</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">linux</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(156, 220, 254);">select</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">69</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">2</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9efdef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874445</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffff9efde548</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaae1ac74f0</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">3</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9e7cef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874452</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffff9e7ce548</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaae1ab32c0</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">4</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9dfbef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874453</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=-</span><span style="color: rgb(181, 206, 168);">310656592</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaaed7bc1b0</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">5</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9d7aef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874454</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=-</span><span style="color: rgb(181, 206, 168);">310572768</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaaed7d0920</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">6</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff7fffef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">876847</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa8323328</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__GI___select</span><span style="color: rgb(204, 204, 204);"> (nfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">5</span><span style="color: rgb(204, 204, 204);">,
readfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffff7fffe558</span><span style="color: rgb(204, 204, 204);">, writefds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
exceptfds</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">, timeout</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffff7fffe548</span><span style="color: rgb(204, 204, 204);">)</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">at ..</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">sysdeps</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">unix</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">sysv</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(204, 204, 204);">linux</span><span style="color: rgb(212, 212, 212);">/</span><span style="color: rgb(156, 220, 254);">select</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">69</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">7</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffffa4c4df40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874441</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=-</span><span style="color: rgb(181, 206, 168);">513560400</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaae163b0b0</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">8</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9fffef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874442</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xffff9fffe5e8</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaae1699cf0</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(181, 206, 168);">9</span><span style="color: rgb(204, 204, 204);"> Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9f7eef40</span><span style="color: rgb(204, 204, 204);"> (LWP
</span><span style="color: rgb(181, 206, 168);">874443</span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in
</span><span style="color: rgb(220, 220, 170);">__futex_abstimed_wait_common64</span><span style="color: rgb(204, 204, 204);"> (private</span><span style="color: rgb(212, 212, 212);">=-</span><span style="color: rgb(181, 206, 168);">513509696</span><span style="color: rgb(204, 204, 204);">,
cancel</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(86, 156, 214);">true</span><span style="color: rgb(204, 204, 204);">, abstime</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0x0</span><span style="color: rgb(204, 204, 204);">,
op</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">393</span><span style="color: rgb(204, 204, 204);">, expected</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0</span><span style="color: rgb(204, 204, 204);">,</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">futex_word</span><span style="color: rgb(212, 212, 212);">=</span><span style="color: rgb(181, 206, 168);">0xaaaae16476c0</span><span style="color: rgb(204, 204, 204);">) at futex</span><span style="color: rgb(212, 212, 212);">-</span><span style="color: rgb(156, 220, 254);">internal</span><span style="color: rgb(204, 204, 204);">.</span><span style="color: rgb(156, 220, 254);">c</span><span style="color: rgb(204, 204, 204);">:</span><span style="color: rgb(181, 206, 168);">57</span></div>
</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="2">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); display: block;">
<div role="presentation" class="elementToProof"><b>Inference from copilot:</b></div>
<div role="presentation" class="elementToProof">Your info threads output shows: Thread 1 (main) is blocked in select(). Thread 6 is also blocked in select(). All other threads are waiting in __futex_abstimed_wait_common64 (futex wait), which means they are
waiting for locks or events. Interpretation: The process is mostly idle or waiting for events. The main thread is blocked in select(), which is normal for a server waiting for socket activity. The crash (SIGABRT) happened while waiting, not during active processing.</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="3">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div role="presentation" class="elementToProof"><i>(gdb)thread apply all bt</i></div>
<div role="presentation" class="elementToProof"><b>Output:</b></div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="0">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="background-color: rgb(31, 31, 31);" role="presentation" class="elementToProof">
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">9</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9f7eef40</span><span style="color: rgb(204, 204, 204);"> (LWP 874443)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=-513509696, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaae16476c0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=-513509696, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaae16476c0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaae16476c0, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0x0, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd2c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff90000b68, cond=0xaaaae1647698)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_wait (cond=0xaaaae1647698, mutex=0xffff90000b68)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:459</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa79030c4</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::wait (mutex=0xffff90000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:99</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::wait (mutex=0xffff90000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:95</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae172cbc0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/ReloadableXMLFile.cpp:304</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0aa2f)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">8</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9fffef40</span><span style="color: rgb(204, 204, 204);"> (LWP 874442)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9fffe5e8, op=393, expected=0,
futex_word=0xaaaae1699cf0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff9fffe5e8, clockid=0, expected=0, futex_word=0xaaaae1699cf0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaae1699cf0, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xffff9fffe5e8, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd4c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0xffff9fffe5e8, clockid=0, mutex=0xffff98000b68, cond=0xaaaae1699cc8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_timedwait64 (cond=0xaaaae1699cc8, mutex=0xffff98000b68, abstime=0xffff9fffe5e8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:484</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa792cc04</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::timedwait (delay_seconds=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">, mutex=0xffff98000b60, this=0xaaaae1699cc0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:110</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::timedwait (delay_seconds=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">,
mutex=0xffff98000b60, this=0xaaaae1699cc0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:103</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::MemoryStorageService::cleanup_fn (pv=0xaaaae163b240)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> impl/MemoryStorageService.cpp:172</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0ad0f)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">7</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffffa4c4df40</span><span style="color: rgb(204, 204, 204);"> (LWP 874441)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=-513560400, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaae163b0b0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=-513560400, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaae163b0b0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaae163b0b0, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0x0, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd2c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa0000b68, cond=0xaaaae163b088)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_wait (cond=0xaaaae163b088, mutex=0xffffa0000b68)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:459</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa79030c4</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::wait (mutex=0xffffa0000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:99</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::wait (mutex=0xffffa0000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:95</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae15778a8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/ReloadableXMLFile.cpp:304</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0ac5f)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">6</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff7fffef40</span><span style="color: rgb(204, 204, 204);"> (LWP 876847)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa8323328</span><span style="color: rgb(204, 204, 204);"> in __GI___select (nfds=5, readfds=0xffff7fffe558, writefds=0x0, exceptfds=0x0, timeout=0xffff7fffe548)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/select.c:69</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa8f4c2d4</span><span style="color: rgb(204, 204, 204);"> in shibsp::ServerThread::run (this=this</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaaed79a670)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> remoting/impl/SocketListener.cpp:544</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa8f4c5c4</span><span style="color: rgb(204, 204, 204);"> in server_thread_fn (arg=0xaaaaed79a670)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> remoting/impl/SocketListener.cpp:495</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">--Type </span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">RET</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">
</span><span style="color: rgb(197, 134, 192);">for</span><span style="color: rgb(204, 204, 204);"> more, q to quit, c to continue without paging--c</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0b1ef)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">5</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9d7aef40</span><span style="color: rgb(204, 204, 204);"> (LWP 874454)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=-310572768, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaaed7d0920) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=-310572768, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaaed7d0920)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaaed7d0920, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0x0, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd2c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff80000b68, cond=0xaaaaed7d08f8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_wait (cond=0xaaaaed7d08f8, mutex=0xffff80000b68)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:459</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa79030c4</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::wait (mutex=0xffff80000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:99</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::wait (mutex=0xffff80000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:95</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaaed7ba2b8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/ReloadableXMLFile.cpp:304</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0a61f)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">4</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9dfbef40</span><span style="color: rgb(204, 204, 204);"> (LWP 874453)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=-310656592, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0xaaaaed7bc1b0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=-310656592, abstime=0x0, clockid=0, expected=0, futex_word=0xaaaaed7bc1b0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaaed7bc1b0, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0x0, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd2c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffff8c000b68, cond=0xaaaaed7bc188)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_wait (cond=0xaaaaed7bc188, mutex=0xffff8c000b68)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:459</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa79030c4</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::wait (mutex=0xffff8c000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:99</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::wait (mutex=0xffff8c000b60, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:95</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaaed0a96f8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/ReloadableXMLFile.cpp:304</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0a61f)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">3</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9e7cef40</span><span style="color: rgb(204, 204, 204);"> (LWP 874452)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9e7ce548, op=393, expected=0,
futex_word=0xaaaae1ab32c0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff9e7ce548, clockid=0, expected=0, futex_word=0xaaaae1ab32c0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaae1ab32c0, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xffff9e7ce548, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd4c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0xffff9e7ce548, clockid=0, mutex=0xffff88000b98, cond=0xaaaae1ab3298)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_timedwait64 (cond=0xaaaae1ab3298, mutex=0xffff88000b98, abstime=0xffff9e7ce548)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:484</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa7903174</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::timedwait (delay_seconds=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">, mutex=0xffff88000b90, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:110</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::timedwait (delay_seconds=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">,
mutex=0xffff88000b90, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:103</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae182ea70)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/ReloadableXMLFile.cpp:306</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0abaf)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">2</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffff9efdef40</span><span style="color: rgb(204, 204, 204);"> (LWP 874445)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82ba994</span><span style="color: rgb(204, 204, 204);"> in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xffff9efde548, op=393, expected=0,
futex_word=0xaaaae1ac74f0) </span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:57</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xffff9efde548, clockid=0, expected=0, futex_word=0xaaaae1ac74f0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:87</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xaaaae1ac74f0, expected=expected</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0,
clockid=clockid</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0, abstime=abstime</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0xffff9efde548, private=private</span><span style="color: rgb(212, 212, 212);">@</span><span style="color: rgb(204, 204, 204);">entry=0)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> futex-internal.c:139</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#3 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bd4c8</span><span style="color: rgb(204, 204, 204);"> in __pthread_cond_wait_common (abstime=0xffff9efde548, clockid=0, mutex=0xffff94000c48, cond=0xaaaae1ac74c8)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:427</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#4 ___pthread_cond_timedwait64 (cond=0xaaaae1ac74c8, mutex=0xffff94000c48, abstime=0xffff9efde548)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_cond_wait.c:484</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#5 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa7903174</span><span style="color: rgb(204, 204, 204);"> in xmltooling::CondWaitImpl::timedwait (delay_seconds=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">, mutex=0xffff94000c40, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:110</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#6 xmltooling::CondWaitImpl::timedwait (delay_seconds=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">,
mutex=0xffff94000c40, this=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/PThreads.cpp:103</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#7 xmltooling::ReloadableXMLFile::reload_fn (pv=0xaaaae16bff20)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> util/ReloadableXMLFile.cpp:306</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#8 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa82bdd78</span><span style="color: rgb(204, 204, 204);"> in start_thread (arg=0xffffd0f0abaf)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> pthread_create.c:443</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#9 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa832addc</span><span style="color: rgb(204, 204, 204);"> in thread_start ()
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">Thread </span><span style="color: rgb(181, 206, 168);">1</span><span style="color: rgb(204, 204, 204);"> (Thread
</span><span style="color: rgb(181, 206, 168);">0xffffa905d020</span><span style="color: rgb(204, 204, 204);"> (LWP 874440)):</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#0 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa8323328</span><span style="color: rgb(204, 204, 204);"> in __GI___select (nfds=4, readfds=0xffffd0f0b568, writefds=0x0, exceptfds=0x0, timeout=0xffffd0f0b558)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> ../sysdeps/unix/sysv/linux/select.c:69</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#1 </span><span style="color: rgb(181, 206, 168);">0x0000ffffa8f4aa24</span><span style="color: rgb(204, 204, 204);"> in shibsp::SocketListener::run (this=0xaaaae16931b0, shutdown=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> remoting/impl/SocketListener.cpp:255</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px;" class="elementToProof">
<span style="color: rgb(204, 204, 204);">#2 </span><span style="color: rgb(181, 206, 168);">0x0000aaaad11807ac</span><span style="color: rgb(204, 204, 204);"> in main (argc=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized
out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">, argv=</span><span style="color: rgb(212, 212, 212);"><</span><span style="color: rgb(204, 204, 204);">optimized out</span><span style="color: rgb(212, 212, 212);">></span><span style="color: rgb(204, 204, 204);">)
</span><span style="color: rgb(86, 156, 214);">at</span><span style="color: rgb(204, 204, 204);"> /usr/src/debug/shibboleth-3.5.0-1.amzn2023.aarch64/shibd/shibd.cpp:465</span></div>
<div style="line-height: 19px; font-family: Consolas, "Courier New", monospace; font-size: 14px; color: rgb(204, 204, 204);" class="elementToProof">
<br>
</div>
</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="3">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); display: block;">
<div role="presentation" class="elementToProof"><b>Inference from copilot:</b></div>
<div role="presentation" class="elementToProof">Main thread and one other are blocked in select() (waiting for socket activity). All other threads are waiting on condition variables in xmltooling::CondWaitImpl::wait or timedwait, mostly inside ReloadableXMLFile::reload_fn
and MemoryStorageService::cleanup_fn.No thread is actively processing; all are waiting for events or signals.</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="3">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); display: block;">
<div role="presentation" class="elementToProof">The abort (SIGABRT) happened while the main thread was idle in select().No evidence of a deadlock or crash in application code—just waiting.</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="3">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); display: block;">
<div role="presentation" class="elementToProof">The process was likely terminated by an external signal (e.g., kill -6/SIGABRT), or by an internal assertion/abort in the code not shown in the stack trace.</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="3">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); display: block;">
<div role="presentation" class="elementToProof">This is consistent with resource exhaustion, watchdog timeout, or a deliberate abort due to a failed check.</div>
</li></ol>
<ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;" data-editing-info="{"applyListStyleFromLevel":true}" start="4">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div role="presentation" class="elementToProof"><b><u>Reviewed shibd logs:</u></b></div>
<div role="presentation" class="elementToProof">Only normal startup and config load messages, no errors or aborts.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div role="presentation" class="elementToProof"><b><u>Checked system logs for OOM killer, kernel, or watchdog events:</u></b></div>
<div role="presentation" class="elementToProof"><i>sudo dmesg | grep -i -E 'oom|kill|error|fail|watchdog'</i></div>
<div role="presentation" class="elementToProof"><i>sudo journalctl -xe | grep -i -E 'oom|kill|error|fail|watchdog'</i></div>
<div role="presentation" class="elementToProof"><i>sudo grep -i -E 'oom|kill|error|fail|watchdog' /var/log/messages</i></div>
<div role="presentation" class="elementToProof"><b>Output</b>: No OOM killer or watchdog events found.</div>
</li></ol>
<div style="margin: 0px 0px 16px;" class="elementToProof">
<div style="line-height: 20px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="border-radius: 3px; max-width: 70%;" class="elementToProof"></div>
</div>
<div style="margin: 0px 0px 16px;" class="elementToProof">
<div style="text-align: left; text-indent: 0px; line-height: 20px; white-space: nowrap; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
We have verified EFS/NFS mount health and system resources, and have enabled core dumps.</div>
<div style="text-align: left; text-indent: 0px; line-height: 20px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
But still we haven't found out the root cause for shibd termination. Has anyone encountered similar behavior, or can suggest additional diagnostics or configuration changes to help identify the root cause? Any guidance would be greatly appreciated.</div>
<div style="text-align: left; text-indent: 0px; line-height: 20px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="text-align: left; text-indent: 0px; line-height: 20px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thank you,</div>
<div style="text-align: left; text-indent: 0px; line-height: 20px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Emily Sherline M.</div>
<div style="text-align: left; text-indent: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
</body>
</html>