Default FailFast configuration leading to intermittent failures in spring wiring

[Cantor, Scott]

> Once it starts occuring on a node it keeps happening, restarting services or
> even the whole machine doesn't seem to fix the problem.

That would suggest you have a contaminated container that's caching something. If you're using Tomcat, then this is one of those reasons we recommend Jetty, and you will have to deal with that fact, by making sure you get it cleaned up if it's not cleaning itself up. If this is Jetty, I'd find it moderately surprising, but the cause I think still has to be cached workfiles.

> In short the error looks like some sort of disconnect in spring contexts. It
> complains that the bean is missing but we can clearly see that bean is in
> spring when we shutdown the war.

You also cannot restart the context, if that's something you were trying to do. Can't tell. That won't work. Never seen it work, and we don't support it.

> Since I have been able to workaround
> this issue with by just changing parts of the environment I believe that the
> problem is almost certainly caused by threading issues.

Well, it's more than that if restarting the whole OS doesn't fix it. That's cached state.

> I can see that this
> service that is failing is being "reloaded" with some shibboleth reloader
> thread my gut tells me that's where the problem is. I thought I had a fix by
> setting "idp.service.relyingparty.failFast=true" but that didn't work in all
> cases.

Fail-fast controls whether initial errors at startup are fatal, that's all.

> Has anyone else experienced this sort of error in the past? Is there any way
> to disable the "reloading" thread?

It's not one thread. You have control over the reloading of all the services in the same place you changed the fail fast settings.

FWIW, I have the RP config set to reload here and have had no issues with it. The only outstanding issues related to reloading tends to involve remote HTTP resources and issues with timeouts or hung requests. Local configs work fine.

-- Scott