Key rollover docs/procedure

[Cantor, Scott]

Also, if your question was really more the physical how, well, I guess I would refer you to the documentation on how to deploy the credentials, how to select different security configurations based on the relying party, and the metadata driven configuration topic.

The "how" part for me was to deploy a new keypair, define SecurityConfiguration beans with sensible names for use of the old and new key, and a third unpublished Canary keypair I used to probe for how systems actually do what they do.

I have a dev system I can use locally for my own use to access systems using different keys to test things via /etc/hosts.

Then I published the new certificate in my metadata sources and started testing and analyzing, using my logs to identify likely systems to check and document.

I started applying EntityAttribute tags via filters or inline to metadata for systems to lock them to the old key where necessary. Once I was satisfied after weeks of inventorying, I flipped my default key for untagged systems. That broke several that were so broken they consume metadata but ignore multiple keys. Finally, I unpublished the old key from InCommon.

That left months of manual work to clean up systems I admin directly, contact vendors and wait weeks for a response, deal with all manner of problems, and generally regret my life choices.

Once I wrap up, we will have a new second key published to InCommon that I don't use by default, but use to test systems for metadata compliance, and a third Canary key locally, used to check for outright trust failures. Our on boarding process will include checking all this and tagging systems that don't handle metadata so that flipping the metadata in the future should be theoretically safe.

A biggie is those broken systems lurking behind the metadata, so identifying those is important.

-- Scott