Sudden issues with non-responsive node (IdP + Terracotta)

[Chad La Joie]

Use jstack[1] and see what the threads are doing.

[1] http://docs.oracle.com/javase/6/docs/technotes/tools/share/jstack.html

On Thu, Mar 1, 2012 at 10:15, Joshua Brodie <josbrodie at gmail.com> wrote:
> I have taken over running a Shibboleth IdP cluster for couple of
> months now (with Terracotta).
>
> All seemed well until about 4 days ago, when one of the IdP nodes
> becomes unresponsive at roughlt the same time (in the middle of the
> night when traffic is low).
>
> Hunting around, I cannot find an obvious smoking gum, accept from the
> cata logs below.
>
> As the traffic is low, in the of the night, I am not sure why the
> threads will be all busy? The terracotta logs did not show anything
> obvious. Any pointers on where to spend energy? Thanks.
>
>
>
> Mar 1, 2012 6:28:26 AM org.apache.tomcat.util.threads.ThreadPool logFull
> SEVERE: All threads (200) are currently busy, waiting. Increase
> maxThreads (200) or check the servlet status
> Mar 1, 2012 6:51:04 AM org.apache.coyote.http11.Http11Protocol pause
> INFO: Pausing Coyote HTTP/1.1 on http-9090
> Mar 1, 2012 6:51:08 AM org.apache.catalina.core.StandardService stop
> INFO: Stopping service Catalina
> Mar 1, 2012 6:51:12 AM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/idp] appears to have started a thread
> named [net.sf.ehcache.CacheManager at 6370fa] but has failed to stop it.
> This is very likely to create a memory leak.
> Mar 1, 2012 6:51:12 AM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/idp] appears to have started a thread
> named [Timer-1] but has failed to stop it. This is very likely to
> create a memory leak.
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net



-- 
Chad La Joie
www.itumi.biz
trusted identities, delivered