<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>So what I've come up with at this point for the caching support
in the dynamic providers turned out to be a little different than
where I was going originally. And I think maybe different than
what Scott may have had in mind (I'm just inferring). But I think
the approach I've devised both works acceptably and has the
advantage of being "simple", so smaller surface area for bugs,
etc.<br>
</p>
<p>The idea is that the caching for the dynamic case really isn't
fundamentally different than than the caching we do for things
like the file-backed HTTP resolver:</p>
<p>1) at "fetch" time, you write the data to the cache. (In the
batch ones this is once, at init time. In the dynamic ones this
is multiply, at request time, once per EntityDescriptor. But this
is not a significant difference).</p>
<p>2) at init time, you load (all) the cached data to make it
"live".</p>
<p>The #2 I thought at first a little controversial. But the more I
thought about it, this seems not only correct but probably what a
deployer might ideally want: On a restart you probably want the
state of the new resolver instance to be the same, or as close as
possible, as before the restart. You pay the loading cost upfront
at init/restart time, rather than paying it incrementally as you
get requests and delaying those requests while you load from the
cache. Basically: 1) you already had it 2) you know it was
recently used (see next paragraph) 3) a fair assumption is that it
will be used again, so 4) go ahead and preemptively load it.<br>
</p>
<p>An initial concern with the preemptive load was the memory
usage. But note that the dynamic case is really different than
the batch case. In the latter, the concern is that you are
potentially loading and keeping very large sets, of which you use
perhaps only a fraction. With dynamic, everything you've loaded
was not only actively used, but used "recently", as defined by the
configuration and the metadata validUntil, and enforced by the
cleanup sweeper thread (which evicts expired and/or idle
metadata). And the same is true with what is loaded on restart.
Essentially you're really just going to be using the same memory
that you otherwise would have been using had you not restarted.
So personally I don't see a problem there. But let's discuss more
if I'm missing something.<br>
</p>
<p>So really, I think the use case of the dynamic caching is really
totally distinct from the "local dynamic" folder o'metadata use
case. They can reuse some shared supporting code (see below), but
they don't fundamentally have anything to do with one another.</p>
<p>The load/save component interface was also written in such a way
that it is not specifically file-based. So for example you could
plug in a JDBC or JPA-based one. And so for the other type of
concrete impl of "local dynamic", if you had the latter, you could
transparently support dynamic resolution from a database also.<br>
</p>
<p>Of course let's discuss on Friday's call.</p>
<p>I've already provisionally implemented the dynamic resolver bits,
just working now on unit tests and the load/save manager impl
(below).<br>
</p>
<p>More details below.<br>
</p>
<br>
<div class="moz-cite-prefix">On 9/1/16 9:40 PM, Cantor, Scott wrote:<br>
</div>
<blockquote cite="mid:D62E0F2E-7766-4A9C-B165-5CE775830ABC@osu.edu"
type="cite">
<pre wrap="">On 9/1/16, 8:55 PM, "dev on behalf of Brent Putman" <a class="moz-txt-link-rfc2396E" href="mailto:dev-bounces@shibboleth.netonbehalfofputmanb@georgetown.edu"><dev-bounces@shibboleth.net on behalf of putmanb@georgetown.edu></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">That's basically what I had in my mind to do. The code to save and load metadata to/from
a file is not complicated.
</pre>
</blockquote>
<pre wrap="">
It's not, but synchronization is always subtle and I was looking for the best way to keep it factored out and not have to clutter all the core code with all that logic, which is something that I don't like about the C++ version of all this.</pre>
</blockquote>
<br>
Given the approach above, I don't know if there is any (additional)
synchronization to do here. The save ops are done under a write
lock over the entityID, from which the cache key is derived. Since
resolvers should not generally share a cache, there doesn't seem to
be a need to do explicit file (or database) locking at the load/save
manager impl level. So I was thinking that the standard default
file-based impl wouldn't need locking, and therefore could be more
performant. But I suppose such locking could be an impl detail and
we could do such a locking impl at some point if necessary.<br>
<br>
The same is also true of the propose concrete "local dynamic"
impls. The load op is done under the same write lock over the
entityID, and therefore transitively under the cache key.<br>
<br>
<blockquote cite="mid:D62E0F2E-7766-4A9C-B165-5CE775830ABC@osu.edu"
type="cite"><br>
<pre wrap="">
Also, I was thinking about the work involved for the deployer in having to chain things, and the overlap of having multiple resolvers managing a backing store in each one for essentially the same data. It just seemed a little ugly.
</pre>
</blockquote>
<br>
<br>
Re-reading this, I realized I didn't really understand what you
meant here, what the use case was, etc.<br>
<br>
<br>
<blockquote cite="mid:D62E0F2E-7766-4A9C-B165-5CE775830ABC@osu.edu"
type="cite"><br>
<blockquote type="cite">
<pre wrap="">If your reasoning is that you want to factor
out the file processing code so it can be reused for the separate use cases of the
backup files *and* the true file-based dynamic resolver - I think we could achieve the
same thing via either support classes, or even some sort of file load/save interface+impl
component that is plugged into both the AbstractDynamicMetadataResolver for the backing
file use, and the concrete file-based dynamic resolver.
</pre>
</blockquote>
<pre wrap="">
Yes, I am wanting to share that code.
</pre>
</blockquote>
<br>
So the shared code I have is for an interface for an XMLObject
load/save manager. It is based on the (simple) idea of loading and
storing an XMLObject indexed under a String key. The methods are
the obvious CRUD ones, including iteration of both keys and the
managed entries themselves. What I have right now is attached.<br>
<br>
I considered directly using or basing this on the StorageService
interface. But in the end decided to make this separate for a
couple of reasons. The main one was that the requirement to treat
the stored value as a String would mean that there would have to be
some impl inefficiencies, vs being able to directly load or save a
file input/output stream. Also, there aren't any impls of
StorageService for files or databases, which I think is informative
that the use cases are kinda different. However I think it would be
possible to have an impl of this load/save manager interface that
just delegates to a supplied StorageService, if that ever seems
useful.<br>
<br>
<br>
</body>
</html>