Overloaded shibd questions

[Nate Klingenstein]

> It can't do that. They'll either be present or the request will fail, there's no path for it to get through just because the process is failing to respond. It will throw an exception and end the request.

Okay, that's fascinating.  This is in IIS.

So, we're in a situation where a large number of sessions and modules are tethered to a single shibd server with a TCPListener on a secured network for clustering due to earlier expedient deployment decisions and an unexpectedly rapidly growing client base.  Sometimes after login and ACS processing and redirect, Shibboleth attributes will be populated.  Sometimes they won't be.  The application servers are not under any appreciable load and have been extensively laterally scaled.

The CPU is getting pegged on the shibd server, but I'm looking for confirmation that that's the root cause of attributes occasionally not being set, and if the failure mode is 500 errors rather than attributes not getting set and the request being passed through, then I'm going to have to revisit assumptions.

> It can't loop because it's overloaded unless the client loops of its own accord, which doesn't seem very plausible. It gets back a 500 error, I don't see how that could end up going back to an IdP.

The application will itself invoke a loop if the variables are unpopulated because it considers it an unauthenticated request.  It's checking for presence of a specific attribute to determine authentication status, which I've verified is sent by the IdP and mapped by Shibboleth.  Logs look like:

2021-05-18 11:42:29 DEBUG XMLTooling.StorageService [17] [default]: updated record (redacted at redacted.lala) in context (NameID) with expiration (1621381349)
2021-05-18 11:42:29 INFO Shibboleth.SessionCache [17] [default]: new session created: ID (_c50f7c9e4fd9a9de53c02344a52db454) IdP (https://sts.windows.net/redacted/) Protocol(urn:oasis:names:tc:SAML:2.0:protocol) Address (redacted)
2021-05-18 11:42:29 DEBUG XMLTooling.StorageService [17] [default]: deleted record (2351f467abfd4664dec6a1ad496635097891eda32c2f888f81e1fca761d4006c) in context (RelayState)
2021-05-18 11:42:29 DEBUG Shibboleth.SSO.SAML2 [17] [default]: ACS returning via redirect to: https://redacted.service.com/path/Login.aspx?FinalRelayState=redacted
<no attributes detected by Login.aspx, so user redirected back to /Shibboleth.sso/Login>
2021-05-18 11:42:29 DEBUG Shibboleth.Listener [17] [default]: dispatching message (default/Login::run::SAML2SI)

and loop.  The looping isn't interminable; it eventually does populate the variable and things proceed as expected.  But it has happened 5+ times in a row for a single client.

But that should never be happening if 500 errors occur rather than failing unauthenticated in the case of an overloaded shibd, and I'm not sure what else would be intermittently causing attributes to not be populated.

> That kind of logging distorts performance too much to help diagnose a load related issue.

Figured as much.

> If you mean the recovery hack, just the attribute, REMOTE_USER is populated each request based on the attribute sources specified.

Yes.  I was thinking this could be used as a stopgap to reduce the load on shibd.  But if it's really just a recovery hack that is only useful for that purpose alone, that kills the idea.

I had presumed the ultimate fix would be alleviating the load on shibd by not using a single instance of it, but if presentation of the request without any Shibboleth variables added can't be caused by an overloaded shibd because it would just flat out return a 500 instead, then I need to think.