<html><head><base href="x-msg://500/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Keith,<div><br></div><div>There are a lot of factors involved in getting a cluster functional. It seems like you know what you are doing and are part of the way there.</div><div><br></div><div>Before you get into complicated functionality testing you should be sure that the basic functions are working.</div><div><br></div><div>The simplest functionality test involves simply swapping nodes in the loadbalancer without shutting off servers. You shouldn't even have to kill the SP server if you simply access its logout link before trying to login to it again. If this works, then you have clustered sessions.</div><div><br></div><div>I have no clue as to what might be going wrong from your log snippet. My vague guess is that one of the tomcat servers is not completing its connection to the TC server and cannot access its session data. Each node should be usable for login individually, while also being connected to TC, before you should try seeing if the login handler on one node can access a session created by another node.</div><div><br></div><div>Good luck. :-)</div><div><br></div><div>Regards,</div><div>Russ.</div><div><br><div><div>On Jun 29, 2012, at 1:03 PM, Wessel, Keith William wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">One other note: if I get the login screen from, say, node 1 then stop node 1’s Tomcat instance before clicking the login button, node 2 takes the login form submission and successfully authenticates me. So, if I take a node down mid-authentication, the other picks up the session. This didn’t seem to work yesterday when node 1 wasn’t participating in the Terracotta cluster. So, I assume this is progress.<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thoughts? Suggestions?<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks,<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Keith<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span>Wessel, Keith William<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, June 29, 2012 2:00 PM<br><b>To:</b><span class="Apple-converted-space"> </span>'Shib Users'<br><b>Subject:</b><span class="Apple-converted-space"> </span>RE: A question about load balancers and Terracotta clustering<o:p></o:p></span></div></div></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Russ,<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">You got me past the first hurtle, and it was a pretty stupid one. Looks like, if the path to the TC boot jar in JAVA_OPTS points to a non-existent jar, no errors are logged, and life goes on as if the jar was never mentioned. My experiements with a different Java version and forgetting to undo my changes in JAVA_OPTS got me here. I fixed that, regenerated the jar for good measure, and now we’re connecting.<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Now, if I log into node 2, shutdown tomcat on that node and destroy the SP session on the SP, then refresh in the browser, I get the generic error page from node 1. I turned up logging in the IDP to debug for both edu.internet2.middleware.shibboleth and org.opensaml. It gets this far, then no more logs:<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">13:18:48.429 - DEBUG [edu.internet2.middleware.shibboleth.idp.profile.saml2.SSOProfileHandler:216] [session=f820cbb71ff0d31ba6b2f08493feba312e75c6c090fe80afd9ec3894b43e382f] - Redirecting user to authentication engine at<span class="Apple-converted-space"> </span><a href="https://[hostname%20removed]:443/idp/AuthnEngine" style="color: blue; text-decoration: underline; ">https://[hostname removed]:443/idp/AuthnEngine</a><o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">13:18:48.500 - DEBUG [edu.internet2.middleware.shibboleth.idp.authn.AuthenticationEngine:209] [session=f820cbb71ff0d31ba6b2f08493feba312e75c6c090fe80afd9ec3894b43e382f] - Processing incoming request<o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">13:18:48.508 - DEBUG [edu.internet2.middleware.shibboleth.idp.authn.AuthenticationEngine:240] [session=f820cbb71ff0d31ba6b2f08493feba312e75c6c090fe80afd9ec3894b43e382f] - Beginning user authentication process.<o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">And that’s where it stops. Is there another log facility I might consider turning up to see what happens next?<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Any other thoughts on what I might look at here?<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Keith<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:users-bounces@shibboleth.net">users-bounces@shibboleth.net</a> [mailto:users-bounces@shibboleth.net]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Russell Beall<br><b>Sent:</b><span class="Apple-converted-space"> </span>Thursday, June 28, 2012 5:08 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Shib Users<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: A question about load balancers and Terracotta clustering<o:p></o:p></span></div></div></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Theoretically, what you are trying to do should work just fine. This type of thing in my clusters does work. I'd be inclined to suspect that the Terracotta server nodes are not correctly sharing their session data.<o:p></o:p></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Instead of shutting down both the Terracotta server and the IdP Tomcat server, try just shutting down the Tomcat server only, or simply disabling it in your loadbalancer and then hitting the other node with a login request.<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">If this doesn't work either, then your sessions are not being shared.<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Do the Terracotta server logs recognize when a Tomcat server client is connecting as a participant in the cluster? Are these connection logs all showing up on a single Terracotta server node?<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Regards,<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Russ.<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">On Jun 28, 2012, at 1:58 PM, Wessel, Keith William wrote:<o:p></o:p></div></div><p class="MsoNormal" style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></p><div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">I know quite a few of you out there have Shib clustered using Terracotta behind a load balancer. We’re in the process of moving our production cluster from round robbin DNS to actual, real hardware load balancing. (much to my and many other folks’ relief).<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; "> <o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">I’m doing some testing with our test cluster, and I’m noticing something odd. As recommended, we’re running a Terracotta cluster node on each of our two IDP nodes. I’m logging into a test SP then restarting shibd and Apache to force reauthentication when I refresh the SP webpage in the browser. Before refreshing, I’ll shut down the IDP node that I used to log in the first time. I’ll then wait for Terracotta output on the other IDP to say it’s taken over as the active node. In theory, if my persistent store and the Terracotta cluster is working, it’d seem my IDP session would still be valid. Session lifetime on the IDP has been left at the default 30 minutes. However, I’m getting the login screen from the other IDP node at this point.<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; "> <o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">So, if I lost anyone:<o:p></o:p></span></div></div><div style="margin-left: 0.5in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">1.</span><span style="font-size: 7pt; "> <span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Log into SP via Shib node 1<o:p></o:p></span></div></div><div style="margin-left: 0.5in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">2.</span><span style="font-size: 7pt; "> <span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Once logged in and back at the SP, shut down Tomcat and Terracotta on Shib node 1.<o:p></o:p></span></div></div><div style="margin-left: 0.5in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">3.</span><span style="font-size: 7pt; "> <span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Restart shibd and Apache on the SP.<o:p></o:p></span></div></div><div style="margin-left: 0.5in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">4.</span><span style="font-size: 7pt; "> <span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Click refresh in the browser which takes me to Shib node 2 via the load balancer<o:p></o:p></span></div></div><div style="margin-left: 0.5in; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">5.</span><span style="font-size: 7pt; "> <span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Instead of having my existing session honored, I’m getting a login screen.<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">FYI, I don’t get the login screen if I just restart shibd and Apache then hit the same IDP node again that I hit the first time, so I know the SP isn’t doing forced re-auth.<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; "> <o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Am I overlooking something here? Is this working as designed? Or does it seem I have a Terracotta misconfiguration somewhere?<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; "> <o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Thanks for any help that anyone can offer.<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; "> <o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">Keith<o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; "> <o:p></o:p></span></div></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif; ">--<br>To unsubscribe from this list send an email to<span class="apple-converted-space"> </span><a href="mailto:users-unsubscribe@shibboleth.net" style="color: blue; text-decoration: underline; ">users-unsubscribe@shibboleth.net</a><o:p></o:p></span></div></div></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div></div>--<br>To unsubscribe from this list send an email to <a href="mailto:users-unsubscribe@shibboleth.net">users-unsubscribe@shibboleth.net</a></div></span></blockquote></div><br></div></body></html>