Default FailFast configuration leading to intermittent failures in spring wiring

[Hendley, Sam]

We are using tomcat but this is a "single purpose" container that is used only for shibboleth. Also this is a completely fresh install, reverted down to base OS and reinstalled from scratch and this error occurs on the first attempt after boot. What could be being cached that would cause this sort of spring wiring issue? Is there possibly a race in the initial startup that then populates some sort of cache that gets loaded from then on? 

I am also not trying to restart it at any level other than the whole tomcat server.

Thanks for the quick reply.

Sam

-----Original Message-----
From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Cantor, Scott
Sent: Monday, July 25, 2016 1:26 PM
To: Shib Users
Subject: RE: Default FailFast configuration leading to intermittent failures in spring wiring

> Once it starts occuring on a node it keeps happening, restarting 
> services or even the whole machine doesn't seem to fix the problem.

That would suggest you have a contaminated container that's caching something. If you're using Tomcat, then this is one of those reasons we recommend Jetty, and you will have to deal with that fact, by making sure you get it cleaned up if it's not cleaning itself up. If this is Jetty, I'd find it moderately surprising, but the cause I think still has to be cached workfiles.

> In short the error looks like some sort of disconnect in spring 
> contexts. It complains that the bean is missing but we can clearly see 
> that bean is in spring when we shutdown the war.

You also cannot restart the context, if that's something you were trying to do. Can't tell. That won't work. Never seen it work, and we don't support it.

> Since I have been able to workaround
> this issue with by just changing parts of the environment I believe 
> that the problem is almost certainly caused by threading issues.

Well, it's more than that if restarting the whole OS doesn't fix it. That's cached state.

> I can see that this
> service that is failing is being "reloaded" with some shibboleth 
> reloader thread my gut tells me that's where the problem is. I thought 
> I had a fix by setting "idp.service.relyingparty.failFast=true" but 
> that didn't work in all cases.

Fail-fast controls whether initial errors at startup are fatal, that's all.

> Has anyone else experienced this sort of error in the past? Is there 
> any way to disable the "reloading" thread?

It's not one thread. You have control over the reloading of all the services in the same place you changed the fail fast settings.

FWIW, I have the RP config set to reload here and have had no issues with it. The only outstanding issues related to reloading tends to involve remote HTTP resources and issues with timeouts or hung requests. Local configs work fine.

-- Scott

--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net