InCommon metadata aggregate exceeds 50MB

[Cantor, Scott]

> I don't believe it's possible to do this with the IdP alone. AFAIK, the Metadata
> Aggregator (or something like it) is needed, correct?

I was just responding to the verification questions.

> In any case, let me know if I can help test or document any of this.

I think it might be worth pulling together a page that summarizes the general behavior of all the common plugins (of which I guess there are really 4, local aggregate, remote aggregate, local dymamic, remote dynamic) when it comes to overhead on failed lookups and how responsive to changes they are.

I'm still working to make sure I understand how the new ones actually work, because some of it's not intuitive to me vs. how I would have built them. For example, I don't think remote dynamic caches failed lookups, which means it essentially has to be the final plugin in the chain to be usable. Local dymamic hits the file system on every failure too, but that's less noticeable (though it still makes me "ick")

Also, LocalDynamic has an interesting behavior...any time it decides to refresh something, it does this unconditionally and will reload that metadata, refilter, etc, and that's in-band of request handling. So you have a tension between high frequency refresh to pick up a change and overhead of needless reparsing slowing down actual requests. The desired behavior IMHO would be SP-style "check for a change, do nothing if unchanged", which is what the local aggregate one does in the background.

For now, I think LocalDynamic is great at the end of the chain for highly unchanging metadata reloaded at low frequency, but that's not the only use case for local metadata. And Remote Dynamic is only viable at the end of the chain unless it caches failed lookups.

TL;DR, some of this stuff works non-intuitively (to me) for now and probably needs to get some additional tweaking to do what I think deployers really need, speaking as a deployer.

-- Scott