metadata refresh frequency on IDP 3.3.x

[Brent Putman]

On 9/8/17 12:09 PM, Liam Hoekenga wrote:
>
>     Can anyone shed any light on how the IDP is deciding to reload
>     the metadata?  I guess the periodicity is getting influenced
>     by refreshDelayFactor,
>

Well, in general, in the absence of an overriding validUntil or
cacheDuration in the metadata itself which dictates a *shorter*
interval, it's going to be (maxRefreshDelay * refreshDelayFactor). 
That is mentioned in the refreshDelayFactor docs in the wiki that you
referenced, although maybe it's not obvious:

"... Attempts to refresh metadata will generally begin around the
product of this number and the maximum refresh delay."

So the default is going to be 4 hours * 0.75 = 3 hours.

Note that this refresh *attempt* isn't actually necessarily going to
reload the metadata, it's merely checking to see if there is *newer*
metadata to reload, which is source-specific (for filesystem it checks
the timestamp, for HTTP it uses conditional GET).

>     but it seems like i've got several different refresh cycles (5?)
>     in play.
>
>
> Is reload-metadata.sh introducing additional refresh cycles instead
> of updating the one already in place?


Hmm. I think you have hit upon a bug, a pretty significant one in
fact.  It seems it's been there since the beginning of 2.x, but only
became relevant in 3.x when we started supporting on-demand refresh. 
The on-demand refresh 1) isn't cancelling the existing next scheduled
refresh task, and 2) is scheduling a next refresh of its own.  So
you're seeing duplicate refresh tasks at different intervals.  I think
also that once started, you will get a distinct refresh cycle/set each
time you manually invoke the refresh. The exact details would need more
research, such as how the different cycles/sets are interacting.

But yes, AFAICT, it's a major bug. I just opened an issue:

https://issues.shibboleth.net/jira/browse/OSJ-216

For now, a potential workaround might be to reload the whole metadata
resolver service, rather than invoke a refresh.  That will replace the
whole instance and so any existing duplicate refresh cycles would be
avoided.   But of course it's more overhead, so probably need to
evaluate in light of how many metadata resolvers you have configured
and what the sources and costs are.


> If so, why do they have a shorter frequency?

That I'm not sure about off-hand.  Probably some non-obvious
interaction between the distinct cycles/sets of scheduled refresh
tasks.  Also, you didn't mention whether the metadata has a root
validUntil or cacheDuration, but that also influences the refresh cycle
(um, cycles).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20170908/57a76cd8/attachment.html>