We had a database problem recently and so did a service restart of our Shibb IdP to reset the database connection. Upon restart (read no changes made) we suddenly have an error about unmarshalling metadata. I couldn't see any clues as to which metadata was in dispute, so proceeded to comment them out in our relyingparty file. Finally the culprit was identified when the error went away and our IdP started up.<br>
<br>I roll through the error and see a message about the "cause" which seems to be a raw amperstand in the metadata xml. This metadata previously worked and never had this problem and is static in our file - provided by vendor ServiceNow for their SP. I open the file and inspect/search for ANY amperstands, so I can replace them, but there are NO amperstands in the xml anywhere, nor can I spot any rogue xml tags floating around - everything is open, closed, attributed and namespaced correctly. What else or where else should I be searching to find the cause of this "Unable to unmarshall metadata" error? There are no other hints, other than disabling this SP's metadata, that would indicate which xml file is the root cause...<br>
<br><br><table cellspacing="0"><tbody><tr><td>2012-10-01 09:47:25,680</td>
<td>main</td>
<td>ERROR</td>
<td>org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider</td>
<td><br></td>
<td>Unable to unmarshall metadata</td>
</tr>
<tr><td colspan="6">org.opensaml.xml.io.UnmarshallingException: org.opensaml.xml.parse.XMLParserException: Invalid XML
<br> at org.opensaml.saml2.metadata.provider.AbstractMetadataProvider.unmarshallMetadata(AbstractMetadataProvider.java:471)
<br> at org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.unmarshallMetadata(AbstractReloadingMetadataProvider.java:304)
<br> at org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.processNewMetadata(AbstractReloadingMetadataProvider.java:345)
<br> at org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.refresh(AbstractReloadingMetadataProvider.java:261)
<br> at org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.doInitialization(AbstractReloadingMetadataProvider.java:236)
<br> at org.opensaml.saml2.metadata.provider.AbstractMetadataProvider.initialize(AbstractMetadataProvider.java:407)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
<br> at java.lang.reflect.Method.invoke(Method.java:616)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
<br> at java.security.AccessController.doPrivileged(Native Method)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
<br> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
<br> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
<br> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
<br> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
<br> at edu.internet2.middleware.shibboleth.common.config.BaseService.loadContext(BaseService.java:171)
<br> at edu.internet2.middleware.shibboleth.common.config.BaseReloadableService.initialize(BaseReloadableService.java:148)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
<br> at java.lang.reflect.Method.invoke(Method.java:616)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
<br> at java.security.AccessController.doPrivileged(Native Method)
<br> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
<br> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
<br> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
<br> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
<br> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
<br> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
<br> at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)
<br> at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)
<br> at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)
<br> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3972)
<br> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4467)
<br> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
<br> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
<br> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
<br> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
<br> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
<br> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
<br> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
<br> at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
<br> at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
<br> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
<br> at org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
<br> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
<br> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
<br> at org.apache.catalina.core.StandardService.start(StandardService.java:516)
<br> at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
<br> at org.apache.catalina.startup.Catalina.start(Catalina.java:593)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
<br> at java.lang.reflect.Method.invoke(Method.java:616)
<br> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
<br> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
<br>Caused by: org.opensaml.xml.parse.XMLParserException: Invalid XML
<br> at org.opensaml.xml.parse.StaticBasicParserPool.parse(StaticBasicParserPool.java:235)
<br> at org.opensaml.saml2.metadata.provider.AbstractMetadataProvider.unmarshallMetadata(AbstractMetadataProvider.java:458)
<br> ... 72 common frames omitted
<br>Caused by: org.xml.sax.SAXParseException: The entity name must immediately follow the '&' in the entity reference.
<br> at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
<br> at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
<br> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
<br> at org.opensaml.xml.parse.StaticBasicParserPool$DocumentBuilderProxy.parse(StaticBasicParserPool.java:649)
<br> at org.opensaml.xml.parse.StaticBasicParserPool.parse(StaticBasicParserPool.java:232)
<br> ... 73 common frames omitted
</td></tr></tbody></table><br>