Metadata support: duplicate entityID detection; resolver lazy eval vs. preprocessing

[Brent Putman]

As alluded to in one of the Jira issues, one potential goal of v3
metadata components is to do something with detecting duplicate
entityID's in the effective metadata collection that the IdP has.

Discussion of the duplicates-across-providers case is in IDP-299.

Here I wanted to bring up the duplicates-within-a-single-provider case. 
To do detection there, the provider would obviously have to scan every
EntityDescriptor in the collection.  In the case of the large
federations' metadata, this might be very expensive.  I suppose it could
do only the first time and cache the result, but it's still more
expensive than the stop-on-first lookup algorithm that we have today.

A related issue is potential pre-processing a newly fetched metadata
document.  My initial thinking was to avoid if possible any kind of
expensive pre-processing of the whole document in favor of more lazy
evaluation as we do today.  For example, today we do not traverse the
whole document creating a comprehensive index of entityID ->
EntityDescriptor.  We wait until an entityID is requested and then we
find it and index it lazily.  This has advantages where a particular
entity is say an IdP and is only ever going to communicate with SP's -
there's no need to do all the work processing the IdP's in the metadata.

On the other hand, the first lookup for a given entityID takes longer
than subsequent ones.  This would be eliminated if we pre-processed
upfront.  The fetch and processing is in a background thread, so it
doesn't affect perceived request time for callers, but would add to the
load on the IdP, in some cases substantially where the metadata is huge.

The v3 metadata redesign could have some other potential pre-processing
work to do, such as pushing (pulling?) extensions from parent
EntitiesDescriptors down to each child EntityDescriptor, and pushing the
"annotation" or entity attribute of parent EntitiesDescriptors down to
the descendent EntityDescriptors.  As far as I can determine right now,
those operations can also be done lazily when a particular
EntityDescriptor is resolved (with appropriate bits stored to only do it
once, the first time), and avoid the cost of processing the whole
document upfront.

Mainly just throwing this out for comment, in case anyone sees any
issues with that approach.  Doing lazy eval has the advantage of being
what we do today; doing a bunch of document preprocessing on every new
document fetch will also add more code and work, so trying to be
pragmatic here.

--Brent