SP V3 metadata types

[Brent Putman]

On 6/12/18 10:01 AM, Tom Scavo wrote:
>
>
> Can you define what you mean by "refresh" here? Do you mean query?
> (Sorry, I don't mean to be overly pedantic.)

No, not query.  Actually I think I meant to say "reload" not refresh. 
I meant when the metadata resolver service or a specific resolver is
(re)loaded, either on IdP (re)start, or explicitly through use of the
the admin endpoints.

>
> Okay, but under what conditions will the persistent cache be accessed
> by the IdP?

Well, it's only *read* when the resolver is initialized/restarted. (A
*write* happens whenever data is resolved from the source).  But
whatever is in the cache is by definition also "live" and in-memory,
since it only got into the cache by coming in through the resolver.  So
the persistent cache is just there to re-init the state of the resolver
on a restart.  If you have 150 live valid entries before restart,
you'll have 150 live valid entries after restart (generally, that's not
*exactly* correct always, but I won't go into the edge cases).


>
>> At that point it effectively is "local", in that no query is attempted for
>> those entities, and it doesn't matter whether the source server is available
>> etc.  Isn't that what you're essentially trying to do with the notion of the
>> separate batch provider accessing an aggregate of "high value" entities?
> Yes, I think so, but I'm not sure since this is new to me. I've read
> all the documentation and AFAIK there is nothing in the docs along
> these lines. What did I miss?

If an entry is live/in-memory and valid, a source query (e.g. HTTP
request) will not be issued until you start hitting the effective "next
refresh" time for that entity (which as in the batch resolvers is based
on a combination of entity @validUntil and @cacheDuration plus resolver
config).   So once the resolver is "primed" with an entity, requests
for it get serviced from the in-memory cache.  No request to the source
will be made (until it's time to refresh based on expiration and
caching info).

>
> In any case, I'm not sure I understand this well enough to document
> it. Can you describe what happens if the server goes down for, say, 20
> mins?

I think there's 3 cases:

1) If the metadata for the entity in question is still live and valid
and isn't past the point where the "next refresh" kicks in, then no
HTTP request is issued, so the IdP won't even know the metadata server
is down.

2) If the metadata *is* past the next refresh boundary, then an attempt
is made to re-fetch from the source. If it works, great, the live
metadata is updated accordingly. If it doesn't work as in your question
(e.g. HTTP client times out, etc), then nothing gets updated in the
resolver.  However, if the live metadata is still valid, then it will
continue to be used.  That's why we start trying to refresh the
metadata *before* its validity/cache expiration, via applying the
refreshDelayFactor.

3) If the server is down and the refresh attempt fails and the metadata
in-memory is expired/invalid, then no entity is returned from that
resolver.


> Also, what happens if the server returns 304 Not Modified?

Well, first off, you'd only see that if the HttpClient instance in use
was of the caching variety.  It's the HttpClient that would decide to
issue a conditional GET, not the resolver.  As discussed extensively in
another thread, the dynamic HTTP resolver code itself does not
cache/utilize Last-Modified and ETag nor does it do conditional GET
internally; it delegates all that to the HttpClient instance.

So, the dynamic HTTP resolver itself will not really see or handle any
304, it's a matter for the HttpClient instance.  If the latter is of
the caching variety and it issues a conditional GET and gets a 304,
then it's going to return the previously cached HTTP response to the
resolver.  The metadata resolver itself won't know or care whether the
HTTP response came from a "real" fetch that returned a 200 + data, vs a
cached response from a conditional GET that resulted in a 304 (vs for
the record a request that was serviced entirely from HTTP cache (no
HTTP requests at all) based on the presence of an appropriate
Cache-Control response header in a previous response).

>
>> ...
>> the other approach we talked about was a simple Spring bean that would take
>> a metadata resolver ID and a list of entityIDs and just do the
>> resolution/priming internally on IdP (re)start.
> Is this another application of <md:AffiliationDescriptor>?

Well, I've never really thought of it that way personally.  I see the
priming as more of a local config thing, and AffiliationDescriptor as
describing functional collections of entities for SAML purposes (VOs,
etc).  I guess it theoretically could be used to drive the priming.  I
think there's an implicit assumption in there though that the
AffiliationDescriptor's members are all in that particular batch of
metadata.  And I don't think MDQ is even currently defined to return
AffiliationDescriptor is it?  So then it wouldn't even be relevant to
the dynamic case, only batch.

However, since Scott built in the admin endpoint to do the priming via
simple curl script, then I doubt we'd ever actually add the bean I
described.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/dev/attachments/20180612/a6b616ad/attachment.html>