Qs after reviewing Java dynamic metadata resolvers

[Brent Putman]

On 9/14/16 4:14 PM, Cantor, Scott wrote:
>> 1) at "fetch" time, you write the data to the cache.  (In the batch ones this is once, at
>> init time.  
> In batch, it's not just init, it's every load of a changed batch. Doesn't really change anything, just noting it.
Good point, I misspoke.  I'm too focused on "initing" right now...

>
>> 2) at init time, you load (all) the cached data to make it "live".
> I'm not opposed, but a simple Predicate optionally injected to control which entities get loaded seems like it would cover all bases, but defaulting that way doesn't bother me.

That seems pretty simple to add.


>
>> An initial concern with the preemptive load was the memory usage.
> More the init time, I would say. I don't think any amount of memory usage from this would come close to what the batch loads cost.

I agree, the memory isn't close to the batch case.  Init time - yeah, I
see your point.  I still think it's probably preferable to pay that
cost up front rather than at request time.  We could instead
theoretically load from the cache in a background thread at init time,
so it doesn't delay the startup.  But there's some synchronization
complexity there, if you get requests in for metadata before or while
it's running for entityIDs that overlap, and newer ones are resolved
than the cache.  That complexity is evil, as it always is, and doesn't
seem worth it to me personally.

>> So the shared code I have is for an interface for an XMLObject load/save manager.  It is
>> based on the (simple) idea of loading and storing an XMLObject indexed under a String key.
> Slight danger there if we re-introduce the problem of writing things out as XMLObjects, and risk the serializer breaking signatures, the bug we just fixed in the SP. I think the IdP currently backs up via streaming the data it gets directly, not with an XML roundtrip.

Yeah, that is a problem, and I already noted it in my personal notes. 
Don't have an answer yet.  In the abstract dynamic resolver, the
abstract fetch method the subclass must implement is defined currently
to return an XMLObject, not a byte[] like in the batch resolvers.  So
we don't have the original data currently.  It's not really related to
this new cache work per se, rather the original dynamic design. The
proximate reason was that all this was designed around the HTTP case,
and in the HTTP subclass I'm using an HttpClient
ResponseHandler<XMLObject> to simplify the processing.  So that
percolated up to the abstract dynamic one.

I'm not actually sure though if the roundtrip does cause a problem in
the Java code. It's certainly a strong possibility, but not certain. 

If the roundtrip is a problem, I suppose we could formally deprecate
that method and have a new method used in preference that returns a
byte[].  A bit tricky to do in a non-breaking way in a minor rev,
though.  The new method might have to be non-abstract, with a default
returning null, I think.

I suppose another really crazy idea, not fully baked, is I could cache
the original byte[] from the HttpClient call on the XMLObject's
objectMetadata, and persist that to cache instead.  Hmmm....


> There certainly is one for databases. 
Of course, duh.  Brain fart.


>>  However I think it would be possible to have an impl of this load/save manager interface
>> that just delegates to a supplied StorageService, if that ever seems useful.
>     
> I would prefer that we do that than ever build anything other than the file backed one, and yes, having additonal interfaces on top is fine, just like we do now for the replay cache, artifact map, session cache, etc. The complexity should be buried in those pieces, not in either the storage plugins or the code using those interfaces.

I'm not opposed.

However, I was just looking again at the StorageService API to confirm
viability.  I think I missed something earlier and it's not currently
possible.  I don't see mechanism on the SS API for enumeration.  At a
minimum, a mechanism to list the storage keys present in a context
would be needed - which in turn could be used to iterate the entries
themselves.  I assume there was some reason for that omission? 
Probably some of the envisioned types weren't amenable to enumeration? 
I guess we could potentially address with a SS sub-interface that added
enumeration method(s).

That's unfortunately key (no pun intended) to what I proposed, since
you have to be able to iterate over all the cache entries in order to
do the load at init time.

I *think* all the other load/save API methods can be implemented in
terms of SS primitives.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/dev/attachments/20160914/00160332/attachment-0001.html>