Terracotta blip causing IdP hang

[Russell Beall]

Yes. This is the standard condition that occurs when apparent connectivity between two nodes goes down and they end up in a split brain state. 

This is probably why TC recommends not running the app server on the same servers as the TC store. This wasn't figured out soon enough, so the main recommendation has been to run them side-by-side. I still run mine that way and it works for two reasons: one, our servers and VMs have only very rarely encountered connectivity issues, and two, I wrote a simple status checker that fails when TC is not servicing the app node. Our LB will automatically remove the node in that case and Nagios will fire warnings. 

An app node that tries to recover from split-brain condition where it was temporarily hooked to the TC server that became active which was then killed and reset to passive will end up in a nonrecoverable state and will need to be restarted. 

Reconnect options as mentioned in another post are available to lengthen the time before the servers time out and decide to change state. A longer timeout can get you through events like a quick backup that hogs resources, but the cluster can end up pausing while waiting for responses if a node is misbehaving but cannot be ejected. 

Regards,
Russ.
(iPhone)


On Apr 26, 2013, at 2:22 PM, "Cantor, Scott" <cantor.2 at osu.edu> wrote:

> On 4/26/13 2:08 PM, "Christopher Bongaarts" <cab at umn.edu> wrote:
> 
>> On 4/26/2013 1:02 PM, Elliot Kendall wrote:
>>> Any idea what's going on here or how to fix it? We're also tracking
>>> down the cause of the blips, but it would be nice to be a little more
>>> resilient as well.
>>> 
>>> The two systems are RedHat Enterprise 5 x64_64 with Java 1.6.0_29,
>>> Tomcat 6.0.35, Terracotta 3.7.0 and IdP 2.3.2.
>> 
>> We saw the same thing in a similar environment; our fix was to ditch
>> Terracotta and move to a stateless IdP. :/
> 
> I wish I had a better suggestion. If the issue is something Russell at USC
> ran into, I'm sure he'll say so.
> 
> -- Scott
> 
> 
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>