Assertion Occasionally Signed with Incorrect Cert

[Mark Valites]

Our institution runs a set of individual IdP hosts behind a load balancer, which are all identically configured with a Stateless Clustering setup & present a single entityID. It's been working well since implementation.

Recently one of our IIS SPs had two shib changes made to it, at the same time - it was updated to 2.5.2 & it's shib cert updated. There was a desire to change the end user facing / webserver cert to match the shib one as well, so a new cert/key pair for both IIS & shib ended up getting generated/signed. The metadata (it's actually InCommon metadata) was updated to include the new cert, in addition to the old one.

From what I've gathered, some arm wrestling was needed to make it work after the changes. I don't know the details, but eventually was told it was working again. Shortly after, I noticed from the IdP logs that it was configured for SAML1.1. Luckily, it's just a test SP & one that I have access to, so I updated it to speak SAML2 & went on my way.

The next day, the admin who made the original updates indicated it wasn't working. I was surprised, but confirmed it was having an issue & began to debug. I upped the log level & flipped some settings back & forth, but had no luck. Eventually, the only change in place was the higher logging level, but it was still broken. After pounding my head on the desk a bit, it suddenly it began to work - at which point it occurred to me perhaps it was hitting different backend IdP hosts through the load balancer & one was problematic.

We're able to target specific IdP hosts behind the load balancer, so I began to enumerate through hitting each, looking for a bad apple. I thought I found two (of six), but couldn't find any discrepancies in their config or metadata when I went digging through them. I went home for the night & came back the next morning to discover that one of the two I suspected of having an issue was now working correctly, but the other, plus now two previously working ones, were not. Today, five of the six backend ones cause an issue. I still cannot locate any discrepancies in the IdP configs or metadata & a co-worker who's spot checked doesn't see any either.

There is a pretty clear error on the SP when it fails: "failed to decrypt assertion: Unable to resolve any key decryption keys." When it's thrown, the SP logs / SAML tracer show the assertion the SP is receiving has been signed with the old SP cert, which is still in the metadata (on both the IdP & SP).

I've gathered the error suggests either metadata or the cert are out of alignment somewhere & seeing the assertion shows the old cert is obviously entering the picture. By rotating across the different backend IdP hosts without changing a thing on the SP, I'm pretty confident the issue doesn't lie on the SP. But not finding any discrepancies in the backend IdP hosts & having some toggle between throwing the error & not throwing it has me at a loss. How's the IdP determine which SP cert to sign the assertion with?

In viewing the IdPStatelessClustering wiki page, I see there's now a note in the Attribute Queries section about a bug "with newer SPs that enforce strict subject matching on queries" . IdP version-wise, I admit that it has the potential to affect us, but being in the Attribute Query section, it doesn't look relevant. I'd like to say I'd expect to see the issue from more SPs as well, but would bet they're not at "newer" versions, so I'm a little afraid the issue may become more prominent when (if…) they update. Can anyone provide a bug ID/more info on this?

Any other thoughts / places to look?


-Mark