Terracotta blip causing IdP hang

[Martin Haase]

Hi Russell,

Am 29.04.2013 06:51, schrieb Russell Beall:
> Yes. This is the standard condition that occurs when apparent connectivity between two nodes goes down and they end up in a split brain state. 
>
> This is probably why TC recommends not running the app server on the same servers as the TC store. This wasn't figured out soon enough, so the main recommendation has been to run them side-by-side. I still run mine that way and it works for two reasons: one, our servers and VMs have only very rarely encountered connectivity issues, and two, I wrote a simple status checker that fails when TC is not servicing the app node. 
Would you mind sharing this status checker? The IdP's own status handler
almost always responds OK even when tomcat fails due to a Terracotta hiccup.

Regards,
Martin

> Our LB will automatically remove the node in that case and Nagios will fire warnings. 
>
> An app node that tries to recover from split-brain condition where it was temporarily hooked to the TC server that became active which was then killed and reset to passive will end up in a nonrecoverable state and will need to be restarted. 
>
> Reconnect options as mentioned in another post are available to lengthen the time before the servers time out and decide to change state. A longer timeout can get you through events like a quick backup that hogs resources, but the cluster can end up pausing while waiting for responses if a node is misbehaving but cannot be ejected. 
>
> Regards,
> Russ.
> (iPhone)
>
>
> On Apr 26, 2013, at 2:22 PM, "Cantor, Scott" <cantor.2 at osu.edu> wrote:
>
>> On 4/26/13 2:08 PM, "Christopher Bongaarts" <cab at umn.edu> wrote:
>>
>>> On 4/26/2013 1:02 PM, Elliot Kendall wrote:
>>>> Any idea what's going on here or how to fix it? We're also tracking
>>>> down the cause of the blips, but it would be nice to be a little more
>>>> resilient as well.
>>>>
>>>> The two systems are RedHat Enterprise 5 x64_64 with Java 1.6.0_29,
>>>> Tomcat 6.0.35, Terracotta 3.7.0 and IdP 2.3.2.
>>> We saw the same thing in a similar environment; our fix was to ditch
>>> Terracotta and move to a stateless IdP. :/
>> I wish I had a better suggestion. If the issue is something Russell at USC
>> ran into, I'm sure he'll say so.
>>
>> -- Scott
>>
>>
>> --
>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>>
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net

-- 
Dr. Martin Haase, Solutions Engineer

DAASI International GmbH        
Europaplatz 3                   
D-72072 Tübingen                
Germany                    

phone: +49 7071 407109-6
fax:   +49 7071 407109-9  
email: martin.haase at daasi.de
web:   www.daasi.de

Sitz der Gesellschaft: Tübingen
Registergericht: Amtsgericht Stuttgart, HRB 382175
Geschäftsleitung: Peter Gietz


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2345 bytes
Desc: S/MIME Kryptografische Unterschrift
Url : http://shibboleth.net/pipermail/users/attachments/20130502/139d9223/attachment.bin