intermittent IDP failure

[Steven Carmody]

Hi,

We're currently running IDP 2.3.5. We have a pair of load-balanced IDPs 
behind an F5; we use terracotta 3.1.1.

We have a growing number of Shib-enabled SPs, and consequently we're 
seeing a growing amount of usage. This problem has been with us for 
perhaps 12+ months. It used to be reported infrequently; with the 
increased usage its now being reported somewhat regularly.

The typical situation has a user logged into 3+ Shib-enabled SPs, and 
switching between them.  At some point, the IDP presents the user with 
an error screen. We *think* the underlying error from the logs is:

No SAMLRequest or SAMLResponse query path parameter, invalid SAML 2 HTTP 
Redirect message

That message is pretty clear -- EXCEPT the user is accessing a 
production IDP, and typically is already logged into the SP.

This msg *may* be triggered more frequently if the user is accessing a 
load-balanced SP that hasn't done anything to ensure that the SAML 
session is shared across all of the machines in the cluster. Outsourced, 
cloud-based SPs that use their own SAML implementation seem particularly 
prone to triggering this problem. Recently, tho, a local load-balanced 
cluster using the Shib SP code that had done nothing to share the SAML 
session across the cluster was also seen to be triggering the problem.

We think what happens is the user logs into an SP; the next time they 
access that SP they actually get sent to a different machine in the 
cluster; that machine has NO SAML session, so it sends the user to the 
IDP. Under normal circumstances, SSO at the IDP would handle this. But, 
for unknown reasons, sometimes the user is presented with the error.

The logs don't contain the HTTP command that the SP actually sent to the 
IDP (I suppose we could sit apache in front of our IDPs, and use the 
apache logs to capture that info). But, I have no reason to believe that 
the SPs aren't sending the query parameter when they clearly send it 
most of the time.

There's strong local suspicion of terracotta as the source of the 
problem... a couple of people edited their desktop /etc/hosts file to 
point to just one of the IDPs (rather than the load balancer), and they 
ran all day without the problem.

Does anyone recognize any part of this description ? Any thoughts or 
suggestions about the cause, or about additional steps we could take to 
try to identify the problem ?

Thanks in advance!