Elevation of the number of AJP-Threads in IDP's tomcat container

[Lanthier, Stéphanie]

Dear lists' members,

I am running a Shibboleth-IdP version 2.3.6, which integrate the CAS sso product. Those are holded in a Tomcat container 6.0.35, JDK-1.6.0_31. The mod_jk connector is used between Tomcat and Apache. The Apache version is the one that comes with the CentOS 5.7 Linux.
My Shibboleth-idp ans CAS are configured so that the source of informations is our Institutional Active Directory (in a redondant way; I have two elements in ldapURL in my DataConnector definition).

For historical reasons, I monitor the number of AJP-8009 "currentThreadCount" of the Java Tomcat process supporting my IdP with the use of JMX. Throught the jconsole tool window, we'd find this number navigating from : 
 MBeans -> Catalina -> ThreadPool -> ajp-8009 -> Attributes -> currentThreadCount

Usually, in my environnement, this number of AJP-8009 currentThreadCount is 16.

Sometimes, there are kinds of problems with the Institutional Active Directory, that leads the AD administrators to reboot the Domain Controllers. As they know about my use of those Domain Controllers, I do trust that they never reboot the two at the same time.
I cannot tell about the kind of problems that happens with the AD as I am not informed about them.

Now, my point. I have seen a correlation between :
1) those reboot when the AD is in a problematic state and 
2) a drastic augmentation of my AJP-8009 currentThreadCount.

For example, yesterday, between 11h and noon, I know that the DC have been rebooted. My currentThreadCount went to 112 around noon. It reached 126 around 4pm and it still 126 since then.  

I would have thought that the Java Garbage Collection would have finally clean some of those threads. As I am waiting since almost 24 hours, I guess that it won't happen.

Usually, I do manually reboot the Tomcat server and it cleans everything. But I don't really like this solution.

I would mention that when the DC are rebooted when they are in a stable state, then the currentThreadCount number is not affected.

I cannot consult de GC log as I have not configured the Java Tomcat process with the flags that could have allow to log the GC events.

I have run a "jstack", if this could interest someone.

On another hand, this high number of AJP-8009 threadCount doesn't seem to degrade the authentication service.

My questions : 

* Do someone has already experiment this kind of augmentation of ThreadCounts against his information source?
* Is there some parameter tuning that I could safely apply to make sure to return to low currentThreadCount parameters? 

Thank you very much for your attention,

Stéphanie

______________________________
Stéphanie Lanthier
lanthier.stephanie at uqam.ca
Analyste de l'informatique
Université du Québec à Montréal
SITel - Div. Enseignement et recherche
514-987-3000 poste 3510