Terracotta blip causing IdP hang

[Eric Goodman]

Probably more helpful to let Russ respond since he's got a script and I'm going from memory, but I'll say that what we did at my last location was just to have something parsing the logs. When we saw similar issues, the problem showed up as repeated attempts to initiate elections in the terracotta logs. It was pretty consistent that any such error that repeated more than a small handful of times meant we had the split-brain type problem. We were able to see the problem from looking at the logs on just one server; it wasn't necessary to compare logs across servers to see the problem.

We found that when the first errors showed up in the logs users were impacted very infrequently and intermittently. After about 8 hours the errors became pervasive enough to actually start registering on other monitoring tools (i.e., "website not responding") and looked more like an "outage" than an "intermittent issues". The outage behavior occurred when stalled sync attempts grew to consume all available Terracotta threads, so I'm sure the "8 hours" estimate is more related to actual login load than clock time.

--- Eric

-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Martin Haase
Sent: Thursday, May 02, 2013 1:59 AM
To: Shib Users
Subject: Re: Terracotta blip causing IdP hang

Hi Russell,

Am 29.04.2013 06:51, schrieb Russell Beall:
> Yes. This is the standard condition that occurs when apparent connectivity between two nodes goes down and they end up in a split brain state. 
>
> This is probably why TC recommends not running the app server on the same servers as the TC store. This wasn't figured out soon enough, so the main recommendation has been to run them side-by-side. I still run mine that way and it works for two reasons: one, our servers and VMs have only very rarely encountered connectivity issues, and two, I wrote a simple status checker that fails when TC is not servicing the app node. 
Would you mind sharing this status checker? The IdP's own status handler almost always responds OK even when tomcat fails due to a Terracotta hiccup.

Regards,
Martin

> Our LB will automatically remove the node in that case and Nagios will fire warnings. 
>
> An app node that tries to recover from split-brain condition where it was temporarily hooked to the TC server that became active which was then killed and reset to passive will end up in a nonrecoverable state and will need to be restarted. 
>
> Reconnect options as mentioned in another post are available to lengthen the time before the servers time out and decide to change state. A longer timeout can get you through events like a quick backup that hogs resources, but the cluster can end up pausing while waiting for responses if a node is misbehaving but cannot be ejected. 
>
> Regards,
> Russ.
> (iPhone)
>
>
> On Apr 26, 2013, at 2:22 PM, "Cantor, Scott" <cantor.2 at osu.edu> wrote:
>
>> On 4/26/13 2:08 PM, "Christopher Bongaarts" <cab at umn.edu> wrote:
>>
>>> On 4/26/2013 1:02 PM, Elliot Kendall wrote:
>>>> Any idea what's going on here or how to fix it? We're also tracking 
>>>> down the cause of the blips, but it would be nice to be a little 
>>>> more resilient as well.
>>>>
>>>> The two systems are RedHat Enterprise 5 x64_64 with Java 1.6.0_29, 
>>>> Tomcat 6.0.35, Terracotta 3.7.0 and IdP 2.3.2.
>>> We saw the same thing in a similar environment; our fix was to ditch 
>>> Terracotta and move to a stateless IdP. :/
>> I wish I had a better suggestion. If the issue is something Russell 
>> at USC ran into, I'm sure he'll say so.
>>
>> -- Scott
>>
>>
>> --
>> To unsubscribe from this list send an email to 
>> users-unsubscribe at shibboleth.net
>>
> --
> To unsubscribe from this list send an email to 
> users-unsubscribe at shibboleth.net

--
Dr. Martin Haase, Solutions Engineer

DAASI International GmbH        
Europaplatz 3                   
D-72072 Tübingen                
Germany                    

phone: +49 7071 407109-6
fax:   +49 7071 407109-9  
email: martin.haase at daasi.de
web:   www.daasi.de

Sitz der Gesellschaft: Tübingen
Registergericht: Amtsgericht Stuttgart, HRB 382175
Geschäftsleitung: Peter Gietz