Sudden issues with non-responsive node (IdP + Terracotta)

[Joshua Brodie]

I have taken over running a Shibboleth IdP cluster for couple of
months now (with Terracotta).

All seemed well until about 4 days ago, when one of the IdP nodes
becomes unresponsive at roughlt the same time (in the middle of the
night when traffic is low).

Hunting around, I cannot find an obvious smoking gum, accept from the
cata logs below.

As the traffic is low, in the of the night, I am not sure why the
threads will be all busy? The terracotta logs did not show anything
obvious. Any pointers on where to spend energy? Thanks.



Mar 1, 2012 6:28:26 AM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (200) are currently busy, waiting. Increase
maxThreads (200) or check the servlet status
Mar 1, 2012 6:51:04 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-9090
Mar 1, 2012 6:51:08 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Mar 1, 2012 6:51:12 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/idp] appears to have started a thread
named [net.sf.ehcache.CacheManager at 6370fa] but has failed to stop it.
This is very likely to create a memory leak.
Mar 1, 2012 6:51:12 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/idp] appears to have started a thread
named [Timer-1] but has failed to stop it. This is very likely to
create a memory leak.