Qs after reviewing Java dynamic metadata resolvers

[Brent Putman]

On 9/15/16 6:36 AM, Rod Widdowson wrote:
> This is slightly off-topic and apologies if it has been answered and I missed it.

I don't think we've explicitly discussed, so would be good to do so
now.  Most of what you are asking was already implemented in the
abstract dynamic provider when it first shipped in 3.2 (?).

> What is the proposed mechanism for expiring the cache entries and do you intend having a way of it happening automatically ('Oh that
> cached info is out of date I'll delete the file') as part of the refresh at background/init time? 

The general mechanism for the dynamic providers for the stuff in-memory
is that there is an expiration time that is explicitly tracked for each
entity (based on configured min/max cache resolver settings + entity
validUntil and cacheDuration. Essentially same as batch).  There is a
configurable background thread which removes expired entries.  For this
new persistent cache work, that also includes removing the persistent
cache entry.  (For the record, the 'last accessed' time is also tracked
and the cleanup thread optionally removes entities that exceed a max
idle time).

On the new init stuff we're talking about, I didn't mention earlier,
but it does skip and evict cache entries which are not "valid", as
defined by the resolver isValid() method.  Currently that only looks at
the validUntil compared to the 'now' time, so it is indeed essentially
just about (SAML) expiration.

>  I can see two potential
> mechanisms, the SAML based ones (validUntil and its partners in crime) and HTTP etags (in association, I guess, with queries) 

So all of this is implemented up in the abstract dynamic resolver,
which is not transport (HTTP) specific.  So it's all SAML expiration +
resolver policy at this point.

I don't even know how we could (easily) introduce transport-specific
notions like etags into the mix.  And I'm not convinced we easily could
or should.  AFAIK an etag is only effective when you actually execute a
conditional GET, which means an "active" check (and which carries a
cost).  So the whole code design would have to change, I think.  Right
now it's just a "passive" check of data that was stored for the entry.

Also, semi-related, don't forget that for the HTTP case, one can wire
in a caching HttpClient, which then does the Last-Modified and eTag
processing transparently via conditional GET.  That's a whole separate
layer of caching, which is really about efficiency (don't fetch new
bytes if unchanged) and not expiration policy.

> The former seems more logical and naturally extends to the "load all of this directory as metadata" thing that the TestShib
> maintainers (amongst others) want.  It is slightly "off" in that it sort of break hierarchy in that you have to crack the SAML to
> check validity, 

You do have to crack the SAML, but because of the design you kind of
need to do that anyway.  You certainly can't ignore the SAML validUntil
and cacheDuration, I think.

(Also semi-related, but the whole reason I hit upon about the
load-cache-on-init idea was that for certain SAML-related reasons, it
turns out you might have to parse, unmarshall and in some fashion
"process" all the cache data anyway.  That's the main cost as Scott
pointed out.  So once you've done that, it occurred to me you may as
well just keep it in memory as live data).


> but I guess you have to do that anyway  because (I assume) you'll want to behave similarly if the signature is
> invalid - why spend the rest of your life loading a file with an invalid signature?.

Actually, I don't know if that's strictly true.  You can't tell *why* a
signature failed to validate.  If it failed because (earlier) you
didn't have the right trust material, and now after the restart (i.e.
with a changed config) you do, then I don't think you'd want to have
thrown out the entity because of the earlier failed signature.

(Note that you would never *initially* cache an entry with a failed
signature, b/c you would never *accept* it in the first place.  You
could only have a failed signature on a cache entry that you are
loading because of a subsequent change in trust material or TrustEngine
config.  I think.)

> The latter then becomes needless, albeit a useful optimization.  I do have to say that I really love the way that the SP persists
> the etag and would really like to see this come to Java. 

Well, maybe I'm wrong but I thought the only reason it does that is
because the underlying HTTP client lib doesn't natively support caching
and conditional GET.  Apache HttpClient does and it feels more natural
to me to leave conditional GET at that layer.

There is the issue that the file-based HttpClient cache doesn't persist
across restarts, by conscious intent.  We did briefly discuss possibly
changing that, maybe as a configurable option.  I'm not opposed. 
However, since we're now going to implement a "SAML cache" that
persists across restarts, I think doing that has less value now, at
least for the dynamic resolver restart case.


>  ISTR that last time I looked I came to the conclusion that it just doesn't
> fit into the batch providers in an easy way but the dynamic ones might?

Currently it doesn't fit in the dynamic ones either because all of this
is handled more abstractly for the dynamic case in general.  The HTTP
subclass just fetches and returns the XMLObject. Nothing in the current
design allows any HTTP-specific data to be passed up and influence
anything.

I suppose data could be attached via objectMetadata (like the original
source byte[] I'm thinking about).  But also not clear how it could be
used.  Any efficiency-based reason for the fetch (conditional GET) can
already be handled by a caching HttpClient.  And I think I personally
prefer that it be isolated at that layer.  I don't actually like that
the Java HTTP batch resolver manually does eTag and Last-Modified mgmt
- it's a legacy from v2 and is there solely (I think) b/c it predates
our understanding and use of caching HttpClients.

> Needless or not, having etags becomes a nice way of hurrying up revocation.  

If you mean revocation/eviction from what I will call "an HTTP
conditional GET cache", then I agree.  AFAIK the only way to eval etag
and Last-Modified is via a conditional GET.

> But at that stage I am straying to details of the MDQ
> protocol: does it specify etags, or explicitly rule them out? I just don't have that swapped in right now so I cannot make a
> meaningful contribution. 

I don't recall that it explicitly rules them out. That would seem
wrong.  Since etag and Last-Modified are standard HTTP and web server
features, I would expect that at a minimum it leaves it open as an
implementation option.  I don't know if if specifies any RFC SHOULD or
MUST around it.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/dev/attachments/20160915/56c20ffa/attachment.html>