Sudden issues with non-responsive node (IdP + Terracotta)

[Curry, Warren]

We had the garbage collecting issues.  We re-vamped our design  of our cluster and moved away from terracotta last August 2011.   The garbasge collection can bring you to nearly a dead stop if not managed well or the volume gets larger.    

-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Russell Beall
Sent: Friday, March 02, 2012 2:38 PM
To: Shib Users
Subject: Re: Sudden issues with non-responsive node (IdP + Terracotta)

Is Terracotta locking up on garbage collection?

Do you have Terracotta cleaning up its disk store?

Regards,
Russ.

On Mar 1, 2012, at 7:15 AM, Joshua Brodie wrote:

> I have taken over running a Shibboleth IdP cluster for couple of 
> months now (with Terracotta).
> 
> All seemed well until about 4 days ago, when one of the IdP nodes 
> becomes unresponsive at roughlt the same time (in the middle of the 
> night when traffic is low).
> 
> Hunting around, I cannot find an obvious smoking gum, accept from the 
> cata logs below.
> 
> As the traffic is low, in the of the night, I am not sure why the 
> threads will be all busy? The terracotta logs did not show anything 
> obvious. Any pointers on where to spend energy? Thanks.
> 
> 
> 
> Mar 1, 2012 6:28:26 AM org.apache.tomcat.util.threads.ThreadPool 
> logFull
> SEVERE: All threads (200) are currently busy, waiting. Increase 
> maxThreads (200) or check the servlet status Mar 1, 2012 6:51:04 AM 
> org.apache.coyote.http11.Http11Protocol pause
> INFO: Pausing Coyote HTTP/1.1 on http-9090 Mar 1, 2012 6:51:08 AM 
> org.apache.catalina.core.StandardService stop
> INFO: Stopping service Catalina
> Mar 1, 2012 6:51:12 AM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/idp] appears to have started a thread 
> named [net.sf.ehcache.CacheManager at 6370fa] but has failed to stop it.
> This is very likely to create a memory leak.
> Mar 1, 2012 6:51:12 AM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/idp] appears to have started a thread 
> named [Timer-1] but has failed to stop it. This is very likely to 
> create a memory leak.
> --
> To unsubscribe from this list send an email to 
> users-unsubscribe at shibboleth.net

--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net