<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 9/15/16 6:36 AM, Rod Widdowson
wrote:<br>
</div>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">This is slightly off-topic and apologies if it has been answered and I missed it.</pre>
</blockquote>
<br>
I don't think we've explicitly discussed, so would be good to do so
now. Most of what you are asking was already implemented in the
abstract dynamic provider when it first shipped in 3.2 (?).<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">
What is the proposed mechanism for expiring the cache entries and do you intend having a way of it happening automatically ('Oh that
cached info is out of date I'll delete the file') as part of the refresh at background/init time? </pre>
</blockquote>
<br>
The general mechanism for the dynamic providers for the stuff
in-memory is that there is an expiration time that is explicitly
tracked for each entity (based on configured min/max cache resolver
settings + entity validUntil and cacheDuration. Essentially same as
batch). There is a configurable background thread which removes
expired entries. For this new persistent cache work, that also
includes removing the persistent cache entry. (For the record, the
'last accessed' time is also tracked and the cleanup thread
optionally removes entities that exceed a max idle time).<br>
<br>
On the new init stuff we're talking about, I didn't mention earlier,
but it does skip and evict cache entries which are not "valid", as
defined by the resolver isValid() method. Currently that only looks
at the validUntil compared to the 'now' time, so it is indeed
essentially just about (SAML) expiration.<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap=""> I can see two potential
mechanisms, the SAML based ones (validUntil and its partners in crime) and HTTP etags (in association, I guess, with queries)
</pre>
</blockquote>
<br>
So all of this is implemented up in the abstract dynamic resolver,
which is not transport (HTTP) specific. So it's all SAML expiration
+ resolver policy at this point.<br>
<br>
I don't even know how we could (easily) introduce transport-specific
notions like etags into the mix. And I'm not convinced we easily
could or should. AFAIK an etag is only effective when you actually
execute a conditional GET, which means an "active" check (and which
carries a cost). So the whole code design would have to change, I
think. Right now it's just a "passive" check of data that was
stored for the entry.<br>
<br>
Also, semi-related, don't forget that for the HTTP case, one can
wire in a caching HttpClient, which then does the Last-Modified and
eTag processing transparently via conditional GET. That's a whole
separate layer of caching, which is really about efficiency (don't
fetch new bytes if unchanged) and not expiration policy.<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">The former seems more logical and naturally extends to the "load all of this directory as metadata" thing that the TestShib
maintainers (amongst others) want. It is slightly "off" in that it sort of break hierarchy in that you have to crack the SAML to
check validity, </pre>
</blockquote>
<br>
You do have to crack the SAML, but because of the design you kind of
need to do that anyway. You certainly can't ignore the SAML
validUntil and cacheDuration, I think.<br>
<br>
(Also semi-related, but the whole reason I hit upon about the
load-cache-on-init idea was that for certain SAML-related reasons,
it turns out you might have to parse, unmarshall and in some fashion
"process" all the cache data anyway. That's the main cost as Scott
pointed out. So once you've done that, it occurred to me you may as
well just keep it in memory as live data).<br>
<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">but I guess you have to do that anyway because (I assume) you'll want to behave similarly if the signature is
invalid - why spend the rest of your life loading a file with an invalid signature?.
</pre>
</blockquote>
<br>
Actually, I don't know if that's strictly true. You can't tell
*why* a signature failed to validate. If it failed because
(earlier) you didn't have the right trust material, and now after
the restart (i.e. with a changed config) you do, then I don't think
you'd want to have thrown out the entity because of the earlier
failed signature.<br>
<br>
(Note that you would never *initially* cache an entry with a failed
signature, b/c you would never *accept* it in the first place. You
could only have a failed signature on a cache entry that you are
loading because of a subsequent change in trust material or
TrustEngine config. I think.)<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">The latter then becomes needless, albeit a useful optimization. I do have to say that I really love the way that the SP persists
the etag and would really like to see this come to Java. </pre>
</blockquote>
<br>
Well, maybe I'm wrong but I thought the only reason it does that is
because the underlying HTTP client lib doesn't natively support
caching and conditional GET. Apache HttpClient does and it feels
more natural to me to leave conditional GET at that layer.<br>
<br>
There is the issue that the file-based HttpClient cache doesn't
persist across restarts, by conscious intent. We did briefly
discuss possibly changing that, maybe as a configurable option. I'm
not opposed. However, since we're now going to implement a "SAML
cache" that persists across restarts, I think doing that has less
value now, at least for the dynamic resolver restart case.<br>
<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap=""> ISTR that last time I looked I came to the conclusion that it just doesn't
fit into the batch providers in an easy way but the dynamic ones might?</pre>
</blockquote>
<br>
Currently it doesn't fit in the dynamic ones either because all of
this is handled more abstractly for the dynamic case in general.
The HTTP subclass just fetches and returns the XMLObject. Nothing in
the current design allows any HTTP-specific data to be passed up and
influence anything.<br>
<br>
I suppose data could be attached via objectMetadata (like the
original source byte[] I'm thinking about). But also not clear how
it could be used. Any efficiency-based reason for the fetch
(conditional GET) can already be handled by a caching HttpClient.
And I think I personally prefer that it be isolated at that layer.
I don't actually like that the Java HTTP batch resolver manually
does eTag and Last-Modified mgmt - it's a legacy from v2 and is
there solely (I think) b/c it predates our understanding and use of
caching HttpClients.<br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">
Needless or not, having etags becomes a nice way of hurrying up revocation. </pre>
</blockquote>
<br>
If you mean revocation/eviction from what I will call "an HTTP
conditional GET cache", then I agree. AFAIK the only way to eval
etag and Last-Modified is via a conditional GET. <br>
<br>
<blockquote
cite="mid:013101d20f3d$00baceb0$02306c10$@steadingsoftware.com"
type="cite">
<pre wrap="">But at that stage I am straying to details of the MDQ
protocol: does it specify etags, or explicitly rule them out? I just don't have that swapped in right now so I cannot make a
meaningful contribution.
</pre>
</blockquote>
<br>
I don't recall that it explicitly rules them out. That would seem
wrong. Since etag and Last-Modified are standard HTTP and web
server features, I would expect that at a minimum it leaves it open
as an implementation option. I don't know if if specifies any RFC
SHOULD or MUST around it.<br>
<br>
</body>
</html>