Weird IdP Hanging Issue

[Royder, Kyle D]

shib-users:

Over the past couple of days, our Shibboleth production machines have been hanging and will sometimes correct themselves and sometimes require tomcat to be restarted.  Catalina.out doesn't provide anything helpful other than SEVERE level errors when shutting Shibboleth down.

SEVERE: A web application created a ThreadLocal with key of type .....

As far as I understand it from searching the archives, this is OK as long as it is occurring when you are shutting he container down.

The idp-process.log shows nothings as well with the followings log levels set:
edu.internet2.middleware.shibboleth="DEBUG"
org.opensaml="WARN"
edu.vt.middleware.ldap="INFO"
edu.internet2.middleware.shibboleth.idp.authn="DEBUG"
PROTOCOL_MESSAGE="DEBUG"

I figure edu.internet2.middleware.shibboleth would be the important one.

When this happens, you can still, very slowly get the authentication page and when you authenticate it just hangs.  The shibboleth logs show authentication successful but do not continue past that.  The logs become really thin instead of their usual chatter they usually just show the following.

12:12:02.131 - INFO [Shibboleth-Access:74] - 20130321T171202Z|IP_ADDRESS|idp.its.utexas.edu:443|/profile/SAML2/Redirect/SSO|
12:13:09.404 - DEBUG [edu.internet2.middleware.shibboleth.idp.authn.provider.UsernamePasswordLoginServlet:134] - Redirecting to login page /login.jsp
12:13:41.868 - DEBUG [edu.internet2.middleware.shibboleth.idp.profile.IdPProfileHandlerManager:86] - shibboleth.HandlerManager: Looking up profile handler for request path: /SAM
L2/Redirect/SSO
12:13:41.869 - INFO [Shibboleth-Access:74] - 20130321T171341Z|IP_ADDRESS|idp.its.utexas.edu:443|/profile/SAML2/Redirect/SSO|
12:13:41.870 - INFO [Shibboleth-Access:74] - 20130321T171341Z|IP_ADDRESS|idp.its.utexas.edu:443|/profile/SAML2/Redirect/SSO|
12:13:41.869 - INFO [Shibboleth-Access:74] - 20130321T171341Z|IP_ADDRESS|idp.its.utexas.edu:443|/profile/SAML2/Redirect/SSO|
12:13:41.869 - INFO [Shibboleth-Access:74] - 20130321T171341Z|IP_ADDRESS|idp.its.utexas.edu:443|/profile/SAML2/Redirect/SSO|
12:13:41.869 - DEBUG [edu.internet2.middleware.shibboleth.idp.profile.IdPProfileHandlerManager:86] - shibboleth.HandlerManager: Looking up profile handler for request path: /SAM
L2/Redirect/SSO

I don't see any errors and if I restart tomcat, it starts churning again.  It's possible that if left alone, it also ends up getting going again.

Is there any place else too look to see what's causing this?

Thanks for any help.
Kyle

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20130321/8580f6b1/attachment.html