<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Morning all,</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Had a server that hosts an IdP crash last night, and kdump filled up the root partition. I’ve resolved that issue, but now I’m getting the following errors when I start up Tomcat…</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">[In catalina.out]</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">11-Oct-2017 09:31:54.219 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">11-Oct-2017 09:31:54.220 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/idp] startup failed due to previous errors</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Had to trim this a little…</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">[In localhost]</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">11-Oct-2017 09:31:54.219 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"> org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'shibboleth.MetadataResolverService' defined in file [/usr/local/shibboleth-idp/system/conf/services-system.xml]: Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Service 'shibboleth.MetadataResolverService': Unexpected error during initial load</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: net.shibboleth.utilities.java.support.component.ComponentInitializationException: Service 'shibboleth.MetadataResolverService': Unexpected error during initial load</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ShibbolethMetadata': Cannot create inner bean '(inner bean)#22897f77' of type [org.opensaml.saml.metadata.resolver.ChainingMetadataResolver] while setting bean property 'embeddedResolver'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#22897f77': Cannot create inner bean '(inner bean)#16c93e41' of type [org.opensaml.saml.metadata.resolver.impl.FileBackedHTTPMetadataResolver] while setting bean property 'resolvers' with key [23]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#16c93e41': Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Error refreshing metadata during init</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#22897f77': Cannot create inner bean '(inner bean)#16c93e41' of type [org.opensaml.saml.metadata.resolver.impl.FileBackedHTTPMetadataResolver] while setting bean property 'resolvers' with key [23]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#16c93e41': Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Error refreshing metadata during init</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#16c93e41': Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Error refreshing metadata during init</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: net.shibboleth.utilities.java.support.component.ComponentInitializationException: Error refreshing metadata during init</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: net.shibboleth.utilities.java.support.resolver.ResolverException: net.shibboleth.utilities.java.support.resolver.ResolverException: Unable to unmarshall metadata</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: net.shibboleth.utilities.java.support.resolver.ResolverException: Unable to unmarshall metadata</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: org.opensaml.core.xml.io.UnmarshallingException: net.shibboleth.utilities.java.support.xml.XMLParserException: Unable to parse inputstream, it contained invalid XML</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: net.shibboleth.utilities.java.support.xml.XMLParserException: Unable to parse inputstream, it contained invalid XML</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Caused by: org.xml.sax.SAXParseException; lineNumber: 61; columnNumber: 68; XML document structures must start and end within the same entity.</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">11-Oct-2017 09:31:54.240 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext</p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Configuration is controlled by Puppet and has not changed since Oct 4, it had been restarted since then as well. </p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);"><span style="font-size: 14px;"><br /></span></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">So far I’ve rolled back the changes that were made on the 4th with no luck. Not sure where to look next. </p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69); min-height: 14px;"><br /></p>
<p style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);">Any help would be appreciated, thanks.</p>
</div>
<div name="messageSignatureSection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
- Mike
<div><br /></div>
</div>
</body>
</html>