Shibboleth IdP fails to download metadata then never tries again

[Brent Putman]

> On 8/16/17 2:37 PM, Cantor, Scott wrote:
>>
>> If there are bugs crashing the reload thread, nobody has reported them or observed them in a long time, so there's really nothing much to be said. Without a logged exception, which I thought we caught now, I don't see that there's any way to learn anything. If we're not catching Throwable, then I suppose the answer is that it must have crashed and it seems to me we should be catching Throwable, at least to log it.
>
> Yes, we do catch Throwable covering the whole refresh(), and have for
> a long time now (several years).  That was to deal with some (very
> rare) events throwing java.lang.Error which is not a subtype of
> java.lang.Exception that we used to catch, and so no next refresh was
> being scheduled.

In reviewing the refresh() code just now, I noticed that there is an
extremely, extremely unlikely possibility of an unchecked runtime
exception (or an java.lang.Error) being thrown outside of the try block
(a couple of statements newing a DateTime for the current time and a
DEBUG logging call).  That would cause the next refresh to not be
scheduled.  However, I can't see it happening consistently on the
*same* provider and not others, across multiple IdP restarts (if that
was what was being implied here). 

It's extremely unlikely to be the issue here, but to be on the safe
side, I guess the OP should check their logs carefully to see whether
such an unchecked exception or error event has happened. In particular
check the container logs, as this would be thrown by the TimerTask
running in the Timer daemon thread, so it likely would not show up in
the IdP logs.

I was also going to suggest looking at thread dumps (via jstack, etc)
to look at the running Timer threads immediately after startup and then
after the failure, to see if there is a delta in the Timer threads. 
However that might be a little difficult since the threads don't have
names.  I think I'll open an issue for the relatively simple
enhancement to add names to all our Timers, so that you can tell which
threads are which in a dump, instead of just "Thread 32539".

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20170816/277121c4/attachment.html>