Jetty 11 is producing excessive number of threads

Equbay Kiflay equbay.kiflay at concordia.ab.ca
Thu Oct 3 19:59:48 UTC 2024 

Hi all,

I just subscribed to this mailing list after reading through this thread. I
have the same issue as David Stava on this configuration
  OS:          Debian 12.1
  Container:   Jetty 11.0.20
  JVM:    OpenJDK 17
  Idp:       Shibboleth 5.1.3

Then I changed the JVM: to Amazon-corretto 17 and there was no difference.
Then I changed the container to Jetty 12.0.13 and there was no difference.
So my current configuration is
  OS:          Debian 12.1
  Container:   Jetty 12.0.13
  JVM:    amazon-corretto 17
  Idp:       Shibboleth 5.1.3
my jetty-base configuration is downloaded from
https://git.shibboleth.net/view/?p=java-idp-jetty-base.git

My suspicion is that
java.lang.Thread.run(java.base at 17.0.12/Thread.java:840) seems
to be stuck.

Here are the output of some commands that I run trying to diagnose thread
pool leak
     *systemctl status idp*
 idp.service - Shibboleth identity provider
     Loaded: loaded (/lib/systemd/system/idp.service; enabled; preset:
enabled)
     Active: active (running) since Thu 2024-10-03 03:00:02 MDT; 10h ago
    Process: 1115415 ExecStart=/etc/init.d/idp start (code=exited,
status=0/SUCCESS)
      Tasks: 5634 (limit: 19135)
     Memory: 8.9G
        CPU: 10min 23.821s
     CGroup: /system.slice/idp.service
             └─1115417 /usr/bin/java -Xms8192m -Xmx14336m
-Didp.home=/opt/shibboleth-idp -Djetty.home=/opt/jetty/jetty-home-12.0.13
-Djetty.base=/opt/jetty/jetty-base -Djava.io.tmpd>

Tasks and Memory start at about 25 and 450M and  increase over time until
either I restart the service or Tasks reach 19135 and the system can not
create new threads and the IdP stops responding and it has to be restarted.
(I threw all that RAM to keep the service up for about two days before
restarting it as a work around)

Looking at the problematic threads
     *ps -Lu jetty | grep pool*

1115417 1115815 ?        00:00:00 pool-3-thread-1
1115417 1115823 ?        00:00:00 pool-4-thread-1
1115417 1116042 ?        00:00:00 pool-5-thread-1
1115417 1116046 ?        00:00:00 pool-6-thread-1
1115417 1116657 ?        00:00:00 pool-7-thread-1
1115417 1116661 ?        00:00:00 pool-8-thread-1
.... (thousands of lines )
1115417 1178375 ?        00:00:00 pool-5523- thread-1
1115417 1178382 ?        00:00:00 pool-5524- thread-1
1115417 1178386 ?        00:00:00 pool-5525- thread-1
1115417 1178389 ?        00:00:00 pool-5526- thread-1
1115417 1178397 ?        00:00:00 pool-5527- thread-1
1115417 1178399 ?        00:00:00 pool-5528- thread-1
1115417 1178407 ?        00:00:00 pool-5529- thread-1
1115417 1178431 ?        00:00:00 pool-5530- thread-1
1115417 1178438 ?        00:00:00 pool-5531- thread-1
1115417 1178442 ?        00:00:00 pool-5532-thread-1

New thread pools keep on creating while the old ones are left sleeping.

Looking at the pool-<num>-1 with jstack
(the oldest two thread pools   -  elapsed time over 10 hours)
"pool-3-thread-1" #105 prio=5 os_prio=0 cpu=36.14ms elapsed=38177.32s
tid=0x00007f52f819fe80 nid=0x1106a7 waiting on condition
 [0x00007f52d6ffe000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base at 17.0.12/Native Method)
        - parking to wait for  <0x00000005aeab6258> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base at 17.0.12
/LockSupport.java:341)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:506)
        at
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base at 17.0.12
/ForkJoinPool.java:3465)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base at 17.0.12
/ForkJoinPool.java:3436)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:1625)
        at java.util.concurrent.ArrayBlockingQueue.take(java.base at 17.0.12
/ArrayBlockingQueue.java:420)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base at 17.0.12
/ThreadPoolExecutor.java:1062)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base at 17.0.12
/ThreadPoolExecutor.java:1122)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base at 17.0.12
/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840)

--
"pool-4-thread-1" #109 prio=5 os_prio=0 cpu=0.62ms elapsed=38175.95s
tid=0x00007f52f81a2ff0 nid=0x1106af waiting on condition
 [0x00007f52d69f8000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base at 17.0.12/Native Method)
        - parking to wait for  <0x00000005aeab64b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base at 17.0.12
/LockSupport.java:341)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:506)
        at
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base at 17.0.12
/ForkJoinPool.java:3465)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base at 17.0.12
/ForkJoinPool.java:3436)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:1625)
        at java.util.concurrent.ArrayBlockingQueue.take(java.base at 17.0.12
/ArrayBlockingQueue.java:420)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base at 17.0.12
/ThreadPoolExecutor.java:1062)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base at 17.0.12
/ThreadPoolExecutor.java:1122)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base at 17.0.12
/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840)

(the newest two thead pools    - elapsed time < 3 sec.)
"pool-5845-thread-1" #27300 prio=5 os_prio=0 cpu=0.72ms elapsed=2.88s
tid=0x00007f52e8527d80 nid=0x120712 waiting on condition
 [0x00007f516882d000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base at 17.0.12/Native Method)
        - parking to wait for  <0x00000004af189970> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base at 17.0.12
/LockSupport.java:341)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:506)
        at
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base at 17.0.12
/ForkJoinPool.java:3465)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base at 17.0.12
/ForkJoinPool.java:3436)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:1625)
        at java.util.concurrent.ArrayBlockingQueue.take(java.base at 17.0.12
/ArrayBlockingQueue.java:420)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base at 17.0.12
/ThreadPoolExecutor.java:1062)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base at 17.0.12
/ThreadPoolExecutor.java:1122)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base at 17.0.12
/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840)

--
"pool-5846-thread-1" #27303 prio=5 os_prio=0 cpu=0.21ms elapsed=1.83s
tid=0x00007f52f45c6180 nid=0x120716 waiting on condition
 [0x00007f516852a000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base at 17.0.12/Native Method)
        - parking to wait for  <0x00000004ae4a6360> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base at 17.0.12
/LockSupport.java:341)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:506)
        at
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base at 17.0.12
/ForkJoinPool.java:3465)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base at 17.0.12
/ForkJoinPool.java:3436)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base at 17.0.12
/AbstractQueuedSynchronizer.java:1625)
        at java.util.concurrent.ArrayBlockingQueue.take(java.base at 17.0.12
/ArrayBlockingQueue.java:420)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base at 17.0.12
/ThreadPoolExecutor.java:1062)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base at 17.0.12
/ThreadPoolExecutor.java:1122)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base at 17.0.12
/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base at 17.0.12/Thread.java:840)

Thanks,
-- 

Equbay Kiflay <equbay.kiflay at concordia.ab.ca>

IT Operations Coordinator

Information Technology Services

Toll-Free: +1 866 479 5200

<https://concordia.ab.ca/>

[image: Concordia University of Edmonton] <https://concordia.ab.ca/>

concordia.ab.ca


[image: Concordia University of Edmonton Twitter]
<https://twitter.com/CUEdmonton> [image: Concordia University of Edmonton
Instagram] <https://instagram.com/cuedmonton/> [image: Concordia University
of Edmonton Facebook] <https://facebook.com/CUEdmonton>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20241003/8918fb38/attachment.htm>

More information about the users mailing list