Terracotta blip causing IdP hang

[Elliot Kendall]

We're seeing a recurring condition where IdP requests never return and
all of our Tomcat threads get hung up waiting for them. The hung
requests include one for static data like javascript files, so
evidently the whole Java application is messed up, not just the parts
that do things.

The cause seems to be a Terracotta blip. For example, this happened
last night:

2013-04-25 22:01:12,347 [HealthChecker] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TCGroupManager - Socket Connect to bar.ucsf.edu:35660(callbackport:9530) taking long time. probably not reachable.
2013-04-25 22:01:22,365 [WorkerThread(group_discovery_stage, 2, 0)] WARN com.tc.net.groups.TCGroupMemberDiscoveryStatic - Node:Node{host=bar.ucsf.edu:9510} not up. Timeout of 10000ms occured connecting to 64.54.x.x:9530

Terracotta quickly reconnected and resynced, and things were back to
normal by 22:03:10. The IdP never seemed to get the message, though.
Hung processes started piling up at between 22:02 and 22:03 and by
22:10 things were completely hosed. The last message in the IdP log
before I restarted it was:

22:11:45.538 - ERROR [org.opensaml.util.storage.ExpiringObjectStorageServiceSweeper:112] - Caught unexpected error, sweeper will execute again in 600000ms
com.tc.exception.TCNotRunningException: Terracotta is not running.

Any idea what's going on here or how to fix it? We're also tracking
down the cause of the blips, but it would be nice to be a little more
resilient as well.

The two systems are RedHat Enterprise 5 x64_64 with Java 1.6.0_29,
Tomcat 6.0.35, Terracotta 3.7.0 and IdP 2.3.2.

-- 
Elliot Kendall
IAM Support Engineer - Single Sign On
Information Technology Services
University of California, San Francisco