A question about load balancers and Terracotta clustering

[Wessel, Keith William]

Russ,

You got me past the first hurtle, and it was a pretty stupid one. Looks like, if the path to the TC boot jar in JAVA_OPTS points to a non-existent jar, no errors are logged, and life goes on as if the jar was never mentioned. My experiements with a different Java version and forgetting to undo my changes in JAVA_OPTS got me here. I fixed that, regenerated the jar for good measure, and now we're connecting.

Now, if I log into node 2, shutdown tomcat on that node and destroy the SP session on the SP, then refresh in the browser, I get the generic error page from node 1. I turned up logging in the IDP to debug for both edu.internet2.middleware.shibboleth and org.opensaml. It gets this far, then no more logs:


13:18:48.429 - DEBUG [edu.internet2.middleware.shibboleth.idp.profile.saml2.SSOProfileHandler:216] [session=f820cbb71ff0d31ba6b2f08493feba312e75c6c090fe80afd9ec3894b43e382f] - Redirecting user to authentication engine at https://[hostname removed]:443/idp/AuthnEngine<https://[hostname%20removed]:443/idp/AuthnEngine>

13:18:48.500 - DEBUG [edu.internet2.middleware.shibboleth.idp.authn.AuthenticationEngine:209] [session=f820cbb71ff0d31ba6b2f08493feba312e75c6c090fe80afd9ec3894b43e382f] - Processing incoming request

13:18:48.508 - DEBUG [edu.internet2.middleware.shibboleth.idp.authn.AuthenticationEngine:240] [session=f820cbb71ff0d31ba6b2f08493feba312e75c6c090fe80afd9ec3894b43e382f] - Beginning user authentication process.

And that's where it stops. Is there another log facility I might consider turning up to see what happens next?

Any other thoughts on what I might look at here?

Keith

From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Russell Beall
Sent: Thursday, June 28, 2012 5:08 PM
To: Shib Users
Subject: Re: A question about load balancers and Terracotta clustering

Theoretically, what you are trying to do should work just fine.  This type of thing in my clusters does work.  I'd be inclined to suspect that the Terracotta server nodes are not correctly sharing their session data.

Instead of shutting down both the Terracotta server and the IdP Tomcat server, try just shutting down the Tomcat server only, or simply disabling it in your loadbalancer and then hitting the other node with a login request.

If this doesn't work either, then your sessions are not being shared.

Do the Terracotta server logs recognize when a Tomcat server client is connecting as a participant in the cluster?  Are these connection logs all showing up on a single Terracotta server node?

Regards,
Russ.

On Jun 28, 2012, at 1:58 PM, Wessel, Keith William wrote:


I know quite a few of you out there have Shib clustered using Terracotta behind a load balancer. We're in the process of moving our production cluster from round robbin DNS to actual, real hardware load balancing. (much to my and many other folks' relief).

I'm doing some testing with our test cluster, and I'm noticing something odd. As recommended, we're running a Terracotta cluster node on each of our two IDP nodes. I'm logging into a test SP then restarting shibd and Apache to force reauthentication when I refresh the SP webpage in the browser. Before refreshing, I'll shut down the IDP node that I used to log in the first time. I'll then wait for Terracotta output on the other IDP to say it's taken over as the active node. In theory, if my persistent store and the Terracotta cluster is working, it'd seem my IDP session would still be valid. Session lifetime on the IDP has been left at the default 30 minutes. However, I'm getting the login screen from the other IDP node at this point.

So, if I lost anyone:
1.      Log into SP via Shib node 1
2.      Once logged in and back at the SP, shut down Tomcat and Terracotta on Shib node 1.
3.      Restart shibd and Apache on the SP.
4.      Click refresh in the browser which takes me to Shib node 2 via the load balancer
5.      Instead of having my existing session honored, I'm getting a login screen.
FYI, I don't get the login screen if I just restart shibd and Apache then hit the same IDP node again that I hit the first time, so I know the SP isn't doing forced re-auth.

Am I overlooking something here? Is this working as designed? Or does it seem I have a Terracotta misconfiguration somewhere?

Thanks for any help that anyone can offer.

Keith

--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net<mailto:users-unsubscribe at shibboleth.net>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20120629/dd5ef862/attachment-0001.html