Production - Service not available message (intermittent)

[Christopher Bongaarts]

On 5/2/2013 8:58 PM, lalithj wrote:
> The thing is currently only few users access the system, There is no big
> load at all but still people get this service not available message.....Do
> you still think apache and tomcat threads make any impact?
>
> I enabled the logs but still I can't see any timeouts but observed below
> error, not sure it is related
>
> However, how to figure it out and troubleshoot the delay is because of
> resolver (LDAP) access or accessing mySQL shib database by shibboleth,  I
> can't think of any other thing which can cause the delay

Delayed responses from external systems would be the most likely culprit 
in these situations, so anything you can do to investigate timing there 
(looking at LDAP/mysql logs if they have timing information, running a 
tcpdump-style capture of LDAP/mysql traffic, checking for high CPU/load 
on the LDAP/mysql servers when the problem occurs, etc.) might help.

Another possibility is if your JVM heap size is too small or too large, 
and it ends up taking a lot of time doing garbage collection.  If you 
look in the Productionalization section on the Shib wiki, I think there 
are hints there on turning on GC logging. We've had this enabled for a 
long time now (added when we were troubleshooting Terracotta).

Finally, if you have low load, be sure your JVM isn't getting swapped 
out of memory, perhaps due to periodic processes like backup eating up 
extra RAM or buffers.  Your OS might offer ways to protect against that.

-- 
%%  Christopher A. Bongaarts   %%  cab at umn.edu          %%
%%  OIT - Identity Management  %%  http://umn.edu/~cab  %%
%%  University of Minnesota    %%  +1 (612) 625-1809    %%