Dynamic metadata resolver; HttpClient usage

[Brent Putman]

On 11/19/14 6:14 AM, Rod Widdowson wrote:
>> I didn't see (point me if I'm missing) that we are by default 
>> wiring in any particular HttpClient instance into components, 
>> and in particular I'm wondering about use of caching HttpClients.
>> Should we either wire one of those in 
> As  recall the mechanics of this get pretty grubby since we have tried
> to keep all configuration defaults out of the parsers.

I think this was actually one reason for Chad's HttpClient builders in
java-support.  He was probably thinking that those build in reasonable
defaults for our own component purposes (e.g. Ian's Jira issue about
timeouts).  Then in components you don't have to do any defaulting per
se.  If you want different, you wire in an explicit bean.  Note that we
do have builders for both in-memory and file caching HttpClients.

>   At one stage
> the legacy relying-party still had a lot of hardwired bean names 
> (rather than hardwired defaults), I think we've cut it down, but it's
> still a worry

I was wondering about the hardwired names also.  I copied what was
there, but I wondered whether we ought to abstract away, or at least
centralize, the bean ref names for thing like the ParserPool, TaskTimer,
etc.


>
> However, since then we have become smarter in the use of properties.  I am
> wondering whether we could end up across all our providers with defaulting
> (no other options provided) the value of client-ref to be 
>
> "%{idp.httpclient.dynamicmetadata}:shibboleth.httpclient.caching}".  

That sounds like a tenable idea.  I was thinking of this notion for
*all* things that use HttpClient, not just the dynamic metadata
resolver, so I'd probably pick a different property name though.

In general, one of my main reasons for bringing this up is that we've
been adopting the strategy that conditional GETs etc, is delegated to
the HttpClient layer, and our components stay ignorant of it.   I still
think that's the right design choice.  However, if we don't actually
configure a default, or at least have a relatively easy opt-in, for a
caching client, then the HttpClient-based components are going to
generate a lot more traffic than they would otherwise.  And the MDQ spec
for example, last I looked (and acknowledging that it's a moving target)
makes caching sort of assumed, or at least recommended (although I saw
some discussion of making it optional for the client-side to implement).


>
>
>> and/or make the parsers, etc build a caching variant by default?  
>> Having it be on by default might be nice for deployers, but would 
>> mean that we'd have to decide upon the caching strategy/store 
>> (essentially in-memory vs filesystem), so I don't know how contentious
>> that choice would be...
> I had the same internal dialog when I first thought about this and decided that, for then, I didn't want to make any call.  There was too much configuration and too much of a chance to get it wrong.  But in all the cases until now the use were very much in the advanced space, so it was reasonable to let the deployer decide.
>
> This feel more like a game changer, but the problem remains of making the right choice.  I would have thought memory since they will be few, but I have no deployment experience.  Equally there is caching above us so maybe none is correct.


Yes, in-memory is easier, but it could be argued not as appropriate for
a default for software that is deployed in heterogeneous ways and
environments (assumptions about memory, etc).  Filesystem has less
concerns I think (disk is usually plentiful and cheap), but then you
have to deal with where to put the cache dir, etc.   Btw, I think they
do have some specialized impls, like EHcache and memcache, although
those seem more esoteric for our purposes.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/dev/attachments/20141119/7ec09a7e/attachment.html