<div dir="ltr">Hi all,<div><br></div><div>I just subscribed to this mailing list after reading through this thread. I have the same issue as <span style="color:rgb(0,0,0)">David Stava on this configuration</span></div><div><font color="#000000"> OS: Debian 12.1</font></div><div><font color="#000000"> Container: Jetty 11.0.20</font></div><div><font color="#000000"> JVM: OpenJDK 17</font></div><div><font color="#000000"> Idp: Shibboleth 5.1.3</font></div><div><font color="#000000"><br></font></div><div><font color="#000000">Then I changed the JVM: to Amazon-corretto 17 and there was no difference.</font></div><div><font color="#000000">Then I changed the container to Jetty 12.0.13 and there was no difference. So my current configuration is </font></div><div><div><font color="#000000"> OS: Debian 12.1</font></div><div><font color="#000000"> Container: Jetty 12.0.13</font></div><div><font color="#000000"> JVM: amazon-corretto 17</font></div><div><font color="#000000"> Idp: Shibboleth 5.1.3</font></div>my jetty-base configuration is downloaded from <a href="https://git.shibboleth.net/view/?p=java-idp-jetty-base.git">https://git.shibboleth.net/view/?p=java-idp-jetty-base.git</a></div><div><br></div><div>My suspicion is that <span style="font-family:monospace">java.lang.Thread.run(java.base@17.0.12/Thread.java:840)</span> seems to be stuck.</div><div><br></div><div>Here are the output of some commands that I run trying to diagnose thread pool leak </div><div> <font face="monospace"><b>systemctl status idp</b></font><br></div><div><font face="monospace"> idp.service - Shibboleth identity provider<br> Loaded: loaded (/lib/systemd/system/idp.service; enabled; preset: enabled)<br> Active: active (running) since Thu 2024-10-03 03:00:02 MDT; 10h ago<br> Process: 1115415 ExecStart=/etc/init.d/idp start (code=exited, status=0/SUCCESS)<br> Tasks: 5634 (limit: 19135)<br> Memory: 8.9G<br> CPU: 10min 23.821s<br> CGroup: /system.slice/idp.service<br> └─1115417 /usr/bin/java -Xms8192m -Xmx14336m -Didp.home=/opt/shibboleth-idp -Djetty.home=/opt/jetty/jetty-home-12.0.13 -Djetty.base=/opt/jetty/jetty-base -Djava.io.tmpd><br></font></div><div><br></div><div>Tasks and Memory start at about 25 and 450M and increase over time until either I restart the service or Tasks reach 19135 and the system can not create new threads and the IdP stops responding and it has to be restarted. (I threw all that RAM to keep the service up for about two days before restarting it as a work around)</div><div><br></div><div>Looking at the problematic threads </div><div> <b><font face="monospace">ps -Lu jetty | grep pool</font></b></div><div><br></div><div><font face="monospace">1115417 1115815 ? 00:00:00 pool-3-thread-1<br>1115417 1115823 ? 00:00:00 pool-4-thread-1<br>1115417 1116042 ? 00:00:00 pool-5-thread-1<br>1115417 1116046 ? 00:00:00 pool-6-thread-1<br>1115417 1116657 ? 00:00:00 pool-7-thread-1<br>1115417 1116661 ? 00:00:00 pool-8-thread-1<br></font></div><div><font face="monospace"><span style="color:rgb(0,0,0)"></span></font><div><font face="monospace">.... (thousands of lines )</font></div><div><font face="monospace">1115417 1178375 ? 00:00:00 pool-5523-
thread-1<br>1115417 1178382 ? 00:00:00 pool-5524-
thread-1<br>1115417 1178386 ? 00:00:00 pool-5525-
thread-1<br>1115417 1178389 ? 00:00:00 pool-5526-
thread-1<br>1115417 1178397 ? 00:00:00 pool-5527-
thread-1<br>1115417 1178399 ? 00:00:00 pool-5528-
thread-1<br>1115417 1178407 ? 00:00:00 pool-5529-
thread-1<br>1115417 1178431 ? 00:00:00 pool-5530-
thread-1<br>1115417 1178438 ? 00:00:00 pool-5531-
thread-1<br>1115417 1178442 ? 00:00:00 pool-5532-thread-1</font><br></div><div><br></div><span class="gmail_signature_prefix"><div><span class="gmail_signature_prefix">New thread pools keep on creating while the old ones are left sleeping.</span></div><div><span class="gmail_signature_prefix"><br></span></div><div><span class="gmail_signature_prefix">Looking at the pool-<num>-1 with jstack</span></div><div><span class="gmail_signature_prefix">(the oldest two thread pools - elapsed time over 10 hours)</span></div><div><span class="gmail_signature_prefix"><font face="monospace">"pool-3-thread-1" #105 prio=5 os_prio=0 cpu=36.14ms elapsed=38177.32s tid=0x00007f52f819fe80 nid=0x1106a7 waiting on condition [0x00007f52d6ffe000]<br> java.lang.Thread.State: WAITING (parking)<br> at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)<br> - parking to wait for <0x00000005aeab6258> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)<br> at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:341)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.12/AbstractQueuedSynchronizer.java:506)<br> at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.12/ForkJoinPool.java:3465)<br> at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.12/ForkJoinPool.java:3436)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.12/AbstractQueuedSynchronizer.java:1625)<br> at java.util.concurrent.ArrayBlockingQueue.take(java.base@17.0.12/ArrayBlockingQueue.java:420)<br> at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.12/ThreadPoolExecutor.java:1062)<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.12/ThreadPoolExecutor.java:1122)<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.12/ThreadPoolExecutor.java:635)<br> at java.lang.Thread.run(java.base@17.0.12/Thread.java:840)<br><br></font></span></div><div><span class="gmail_signature_prefix"><font face="monospace">--<br>"pool-4-thread-1" #109 prio=5 os_prio=0 cpu=0.62ms elapsed=38175.95s tid=0x00007f52f81a2ff0 nid=0x1106af waiting on condition [0x00007f52d69f8000]<br> java.lang.Thread.State: WAITING (parking)<br> at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)<br> - parking to wait for <0x00000005aeab64b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)<br> at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:341)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.12/AbstractQueuedSynchronizer.java:506)<br> at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.12/ForkJoinPool.java:3465)<br> at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.12/ForkJoinPool.java:3436)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.12/AbstractQueuedSynchronizer.java:1625)<br> at java.util.concurrent.ArrayBlockingQueue.take(java.base@17.0.12/ArrayBlockingQueue.java:420)<br> at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.12/ThreadPoolExecutor.java:1062)<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.12/ThreadPoolExecutor.java:1122)<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.12/ThreadPoolExecutor.java:635)<br> at java.lang.Thread.run(java.base@17.0.12/Thread.java:840)<br></font></span></div><div><span class="gmail_signature_prefix"><br></span></div><div><span class="gmail_signature_prefix">(the newest two thead pools - elapsed time < 3 sec.)</span></div><div><span class="gmail_signature_prefix"><font face="monospace">"pool-5845-thread-1" #27300 prio=5 os_prio=0 cpu=0.72ms elapsed=2.88s tid=0x00007f52e8527d80 nid=0x120712 waiting on condition [0x00007f516882d000]<br> java.lang.Thread.State: WAITING (parking)<br> at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)<br> - parking to wait for <0x00000004af189970> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)<br> at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:341)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.12/AbstractQueuedSynchronizer.java:506)<br> at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.12/ForkJoinPool.java:3465)<br> at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.12/ForkJoinPool.java:3436)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.12/AbstractQueuedSynchronizer.java:1625)<br> at java.util.concurrent.ArrayBlockingQueue.take(java.base@17.0.12/ArrayBlockingQueue.java:420)<br> at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.12/ThreadPoolExecutor.java:1062)<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.12/ThreadPoolExecutor.java:1122)<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.12/ThreadPoolExecutor.java:635)<br> at java.lang.Thread.run(java.base@17.0.12/Thread.java:840)<br><br>--<br>"pool-5846-thread-1" #27303 prio=5 os_prio=0 cpu=0.21ms elapsed=1.83s tid=0x00007f52f45c6180 nid=0x120716 waiting on condition [0x00007f516852a000]<br> java.lang.Thread.State: WAITING (parking)<br> at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)<br> - parking to wait for <0x00000004ae4a6360> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)<br> at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:341)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.12/AbstractQueuedSynchronizer.java:506)<br> at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.12/ForkJoinPool.java:3465)<br> at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.12/ForkJoinPool.java:3436)<br> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.12/AbstractQueuedSynchronizer.java:1625)<br> at java.util.concurrent.ArrayBlockingQueue.take(java.base@17.0.12/ArrayBlockingQueue.java:420)<br> at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.12/ThreadPoolExecutor.java:1062)<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.12/ThreadPoolExecutor.java:1122)<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.12/ThreadPoolExecutor.java:635)<br> at java.lang.Thread.run(java.base@17.0.12/Thread.java:840)</font><br></span></div><div><span class="gmail_signature_prefix"><br></span></div><div><span class="gmail_signature_prefix">Thanks,</span></div>-- </span><br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><table width="320" cellspacing="0" cellpadding="0" style="width:320px;margin:0;padding:0;font-family:Verdana,Geneva,sans-serif;font-size:13px;line-height:10px;color:#173d6f;border-collapse:collapse">
<tbody><tr>
<td style="margin:0;padding:0">
<input type="hidden" name="CUECONCTABLENAMEID" value="123412341234123412341234">
<table width="320" cellspacing="0" cellpadding="0" style="width:320px;margin:0;padding:0;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:10px;line-height:10px;color:#000;border-collapse:collapse">
<tbody><tr>
<td style="margin:0;padding:0;font-family:Verdana,Geneva,sans-serif;white-space:nowrap">
<p style="margin:0px;padding:0;margin-bottom:5px;margin-top:5px">
<a href="mailto:equbay.kiflay@concordia.ab.ca" style="border:none;text-decoration:none;color:#173d6f;font-size:15px;font-weight:bold;margin:0;padding:0" target="_blank">Equbay Kiflay</a>
</p>
<p style="margin:0px;margin-bottom:5px">
<span style="color:#173d6f;font-size:12px">IT Operations Coordinator</span>
</p>
<p style="margin:0px;margin-bottom:5px">
<span style="color:#173d6f;font-size:12px">Information Technology Services</span>
</p>
</td>
</tr>
<tr>
<td style="width:320px;margin:0;padding:0;font-family:Verdana,Geneva,sans-serif;font-size:11px;line-height:15px;color:#1173d6f;border-collapse:collapse">
<p style="margin:0px;margin-bottom:5px;margin-right:10px">Toll-Free: +1 866 479 5200</p>
</td>
</tr>
<tr>
<td style="width:320px;margin:0;padding:0;font-family:Verdana,Geneva,sans-serif;font-size:10px;line-height:15px;color:#1173d6f;border-collapse:collapse">
<p style="margin:0px;margin-bottom:5px;margin-right:10px">
<a href="https://concordia.ab.ca/" style="border:none;text-decoration:none;color:#173d6f;font-size:13px;font-weight:bold;margin:0;padding:0" target="_blank"></a></p>
<p style="margin:0px;margin-bottom:1px;margin-right:10px">
<span style="width:120px;margin:0;padding:0">
<a href="https://concordia.ab.ca/" style="border:none;text-decoration:none;color:#000;margin:0;padding:0" target="_blank">
<img src="https://static.concordia.ab.ca/img/esignaturelogo2021031701.jpg" alt="Concordia University of Edmonton" height="82" width="365" align="top">
</a>
</span>
</p>
<p style="margin:0px;margin-bottom:1px;margin-right:10px">
<a href="https://concordia.ab.ca/" style="border:none;text-decoration:none;color:#173d6f;font-size:13px;font-weight:bold;margin:0;padding:0" target="_blank">concordia.ab.ca</a>
</p>
</td>
</tr>
<tr>
<td style="width:320px;margin:0;padding:0;font-family:Verdana,Geneva,sans-serif;font-size:10px;line-height:1px;color:#1173d6f;border-collapse:collapse">
<p style="margin:0px;margin-bottom:2px;margin-right:10px"><br>
<a href="https://twitter.com/CUEdmonton" style="border:none;text-decoration:none;color:#000;font-size:14px;margin:0;padding:0" target="_blank"><img src="https://static.concordia.ab.ca/img/twitterlogo2016062701.png" alt="Concordia University of Edmonton Twitter" height="25" width="25"></a>
<a href="https://instagram.com/cuedmonton/" style="border:none;text-decoration:none;color:#000;font-size:14px;margin:0;padding:0" target="_blank"><img src="https://static.concordia.ab.ca/img/instagramlogo2016062701.png" alt="Concordia University of Edmonton Instagram" height="25" width="25"></a>
<a href="https://facebook.com/CUEdmonton" style="border:none;text-decoration:none;color:#000;font-size:14px;margin:0;padding:0" target="_blank"><img src="https://static.concordia.ab.ca/img/facebooklogo2016062701.png" alt="Concordia University of Edmonton Facebook" height="25" width="25"></a>
</p>
</td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
</div></div></div>