Default FailFast configuration leading to intermittent failures in spring wiring

[Hendley, Sam]

We have been having intermittent failures with our shibboleth integration. In some cases after a deployment a node will refuse to let us login at all. The identical code deployed to another node won't have this issue in most cases. Once it starts occuring on a node it keeps happening, restarting services or even the whole machine doesn't seem to fix the problem. We discovered that in some cases just changing timestamps on the logback.xml file was enough to get it to working on some systems. This was really weird and we ignored the issue. Over the last week it started occuring on a lot more of our deployments (though still not all).

In short the error looks like some sort of disconnect in spring contexts. It complains that the bean is missing but we can clearly see that bean is in spring when we shutdown the war. Since I have been able to workaround this issue with by just changing parts of the environment I believe that the problem is almost certainly caused by threading issues. I can see that this service that is failing is being "reloaded" with some shibboleth reloader thread my gut tells me that's where the problem is. I thought I had a fix by  setting "idp.service.relyingparty.failFast=true" but that didn't work in all cases.

Has anyone else experienced this sort of error in the past? Is there any way to disable the "reloading" thread? Any other recommendations? Sorry if I am using the wrong terminology I am very new to shibboleth, the engineer who implemented our integration has moved on so we are scrambling to figure this issue out.

Thanks in advance,
Sam Hendley

Exception:

2016-07-25 09:09:31,737 - ERROR [org.springframework.beans.factory.BeanCreationException:76] -
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'EncryptToSelfParametersStrategy' defined in file [/opt/flexnet/shibboleth-idp/system/flows/saml/saml2/common-beans.xml]: Cannot create inner bean 'net.shibboleth.idp.relyingparty.impl.EncryptionCredentialsResolver#2c15699c' of type [net.shibboleth.idp.relyingparty.impl.EncryptionCredentialsResolver] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'net.shibboleth.idp.relyingparty.impl.EncryptionCredentialsResolver#2c15699c' defined in file [/opt/flexnet/shibboleth-idp/system/flows/saml/saml2/common-beans.xml]: Cannot resolve reference to bean 'shibboleth.RelyingPartyResolverService' while setting constructor argument; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'shibboleth.RelyingPartyResolverService' is defined
                at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'net.shibboleth.idp.relyingparty.impl.EncryptionCredentialsResolver#2c15699c' defined in file [/opt/flexnet/shibboleth-idp/system/flows/saml/saml2/common-beans.xml]: Cannot resolve reference to bean 'shibboleth.RelyingPartyResolverService' while setting constructor argument; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'shibboleth.RelyingPartyResolverService' is defined
                at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'shibboleth.RelyingPartyResolverService' is defined
                at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:698)
...
2016-07-25 09:10:12,209 - INFO [net.shibboleth.ext.spring.context.DelimiterAwareApplicationContext:957] - Closing WebApplicationContext for namespace 'idp-servlet': startup date [Mon Jul 25 09:05:41 EDT 2016]; root of context hierarchy
2016-07-25 09:10:12,216 - INFO [net.shibboleth.ext.spring.context.DelimiterAwareApplicationContext:957] - Closing Root WebApplicationContext: startup date [Mon Jul 25 09:05:37 EDT 2016]; root of context hierarchy
...
2016-07-25 09:10:12,268 - INFO [net.shibboleth.ext.spring.context.FilesystemGenericApplicationContext:957] - Closing ApplicationContext:shibboleth.RelyingPartyResolverService: startup date [Mon Jul 25 09:05:42 EDT 2016]; parent: Root WebApplicationContext
2016-07-25 09:10:12,271 - INFO [net.shibboleth.utilities.java.support.service.AbstractReloadableService:207] - Service 'shibboleth.RelyingPartyResolverService': Starting shutdown
2016-07-25 09:10:12,271 - INFO [net.shibboleth.utilities.java.support.service.AbstractReloadableService:216] - Service 'shibboleth.RelyingPartyResolverService': Completing shutdown


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20160725/858bc939/attachment-0001.html>