Assertion Occasionally Signed with Incorrect Cert

[Mark Valites]

On Aug 16, 2013, at 11:58 AM, Cantor, Scott wrote:

> On 8/16/13 11:23 AM, "Mark Valites" <mvalites at buffalo.edu> wrote:
> 
>> Recently one of our IIS SPs had two shib changes made to it, at the same
>> time - it was updated to 2.5.2 & it's shib cert updated. There was a
>> desire to change the end user facing / webserver cert to match the shib
>> one as well, so a new cert/key pair for both IIS & shib ended up getting
>> generated/signed. The metadata (it's actually InCommon metadata) was
>> updated to include the new cert, in addition to the old one.
> 
> You can't do that unless you're very careful to follow specific steps or
> use specific key usage options. That will cause problems if the key is
> left in the metadata but removed from the SP, again unless care is taken.

I agree with that now.

> 
>>> From what I've gathered, some arm wrestling was needed to make it work
>>> after the changes.
> 
> The change was, then, almost certainly botched and left in a broken state.

I won't argue that… :)

>> There is a pretty clear error on the SP when it fails: "failed to decrypt
>> assertion: Unable to resolve any key decryption keys." When it's thrown,
>> the SP logs / SAML tracer show the assertion the SP is receiving has been
>> signed with the old SP cert, which is still in the metadata (on both the
>> IdP & SP).
> 
> As Peter said, that is a misinterpretation of how signing and encryption
> works.

Yes, Peter/you are correct, I wrote wrong - what I think I should have said was: when the issue occurred, I could see that the assertion is encrypted with a different public key than what "worked".

> 
>> I've gathered the error suggests either metadata or the cert are out of
>> alignment somewhere & seeing the assertion shows the old cert is
>> obviously entering the picture. By rotating across the different backend
>> IdP hosts without changing a thing on the SP, I'm pretty confident the
>> issue doesn't lie on the SP.
> 
> It fundamentally does on some level, unless the metadata and SP are now in
> sync and the IdP nodes are holding on to different metadata that is now
> out of sync.

I'm still very confident the IdPs were/are in sync.

>> In viewing the IdPStatelessClustering wiki page, I see there's now a note
>> in the Attribute Queries section about a bug "with newer SPs that enforce
>> strict subject matching on queries" . IdP version-wise, I admit that it
>> has the potential to affect us, but being in the Attribute Query section,
>> it doesn't look relevant.
> 
> It's not.
> 
>> I'd like to say I'd expect to see the issue from more SPs as well, but
>> would bet they're not at "newer" versions, so I'm a little afraid the
>> issue may become more prominent when (ifŠ) they update. Can anyone
>> provide a bug ID/more info on this?
> 
> The issue is a broken SP key rollover process.

Agreed.

We sign new SP certs with our internal CA all the time here, but rarely (if ever) do the SPs update their keys/submit a new CSR - after wondering about this more, I went back & read the NativeSPMultipleCredentials page. I configured the CredentialResolver to chain together the old/new cert/keys & do the key rollover properly & now it works as expected. I'm a bit surprised we haven't had an SP do this previously.

I'm now baffled to how this worked at all. I don't have a whole lot of faith in the earlier made SP changes, but after I got my hands on it, I'm positive it worked with the old new on some of the IdPs, some of the time. I'm not sure I care enough to keep poking at it though.

Thanks,
-Mark