Errors after Relying Party Service Reload

Mon Apr 2 21:02:58 BST 2012

We've configured IdP v2.3.6 on RHEL6 x64 to reload the attribute filter
engine, attribute resolver, and relying party configuration manager services
every 2 hours in service.xml. The IdP starts normally and runs fine, however
the IdP starts throwing errors after the relying party configuration manager
reloads following a manual change to relying-party.xml even though the
affected SPs were accessed successfully prior to the reload. These issues
are resolved after restarting Tomcat. The SPs involved in the error'd
transactions differ after each Tomcat restart/relying-party update. The
specific errors encountered on the IdP are pasted below, and result in a
FatalProfileException with message "Unable to encrypt assertion" on the SP.

12:02:00.516 - ERROR
[edu.internet2.middleware.shibboleth.idp.profile.saml2.AbstractSAML2ProfileH
andler:923] - [cl.ie.nt.ip] - Could not resolve a key encryption credential
for peer entity: https://some-sp.domain.edu/shibboleth

12:02:00.518 - ERROR
[edu.internet2.middleware.shibboleth.idp.profile.saml2.AbstractSAML2ProfileH
andler:289] - [cl.ie.nt.ip] - Unable to construct encrypter

org.opensaml.xml.security.SecurityException: Could not resolve key
encryption credential

        at
edu.internet2.middleware.shibboleth.idp.profile.saml2.AbstractSAML2ProfileHa
ndler.getEncrypter(AbstractSAML2ProfileHandler.java:924)
~[shibboleth-identityprovider-2.3.6.jar:na]

        at
edu.internet2.middleware.shibboleth.idp.profile.saml2.AbstractSAML2ProfileHa
ndler.buildResponse(AbstractSAML2ProfileHandler.java:286)
~[shibboleth-identityprovider-2.3.6.jar:na]

        at
edu.internet2.middleware.shibboleth.idp.profile.saml2.SSOProfileHandler.comp
leteAuthenticationRequest(SSOProfileHandler.java:310)
[shibboleth-identityprovider-2.3.6.jar:na]

        at
edu.internet2.middleware.shibboleth.idp.profile.saml2.SSOProfileHandler.proc
essRequest(SSOProfileHandler.java:177)
[shibboleth-identityprovider-2.3.6.jar:na]

        at
edu.internet2.middleware.shibboleth.idp.profile.saml2.SSOProfileHandler.proc
essRequest(SSOProfileHandler.java:88)
[shibboleth-identityprovider-2.3.6.jar:na]

        at
edu.internet2.middleware.shibboleth.common.profile.ProfileRequestDispatcherS
ervlet.service(ProfileRequestDispatcherServlet.java:84)
[shibboleth-common-1.3.5.jar:na]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[servlet-api.jar:na]

        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206) [catalina.jar:6.0.35]

        at
edu.internet2.middleware.shibboleth.idp.util.NoCacheFilter.doFilter(NoCacheF
ilter.java:50) [shibboleth-identityprovider-2.3.6.jar:na]

        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206) [catalina.jar:6.0.35]

        at
edu.internet2.middleware.shibboleth.idp.session.IdPSessionFilter.doFilter(Id
PSessionFilter.java:81) [shibboleth-identityprovider-2.3.6.jar:na]

        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206) [catalina.jar:6.0.35]

        at
edu.internet2.middleware.shibboleth.common.log.SLF4JMDCCleanupFilter.doFilte
r(SLF4JMDCCleanupFilter.java:52) [shibboleth-common-1.3.5.jar:na]

        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:219) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127
) [catalina.jar:6.0.35]

        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
) [catalina.jar:6.0.35]

        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109) [catalina.jar:6.0.35]

        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
[catalina.jar:6.0.35]

        at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
[tomcat-coyote.jar:6.0.35]

        at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
[tomcat-coyote.jar:6.0.35]

        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
[tomcat-coyote.jar:6.0.35]

        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
[tomcat-coyote.jar:6.0.35]

        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:898) [tomcat-coyote.jar:6.0.35]

        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:690) [tomcat-coyote.jar:6.0.35]

        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_03]

The reload configuration for relying party in service.xml is (polling
attributes are the same for attribute filter and resolver):

<srv:Service id="shibboleth.RelyingPartyConfigurationManager"
xsi:type="relyingParty:SAMLMDRelyingPartyConfigurationManager"
depends-on="shibboleth.SAML1AttributeAuthority
shibboleth.SAML2AttributeAuthority"
configurationResourcePollingFrequency="PT120M"
configurationResourcePollingRetryAttempts="12">

<srv:ConfigurationResource file="/opt/shibboleth-idp/conf/relying-party.xml"
xsi:type="resource:FilesystemResource"/>

</srv:Service>

The IdP Common Errors page on the wiki suggests the "Unable to construct
encrypter" error is related to the SP metadata not having an encryption
certificate. I've confirmed that each of the SPs encountering this error all
have a certificate, all of which omit the 'use' attribute. Somewhere in the
reload of the relying party file, certain certificates are not being read
from the SP metadata files, though they are read successfully during IdP
startup.

This behavior can be reproduced in my DEV and QA environments. The only
change made to relying-party.xml between automatic reloads is to `touch` the
file to update the modified timestamp. I've removed the other reloading
configurations, leaving relying party as the only service being reloaded,
and the problem persists. There's nothing in the logs (at DEBUG level) for
the reload indicating anything failed or was not successfully reloaded.

We are well aware of the warning on the wiki regarding use of the
configuration reloading in production, and we have sufficient configuration
testing in place to prevent invalid configurations from being loaded into
the production environment. The goal of using the reloading features is to
provide shorter turn-around when adding new SPs to the IdP and modifying
attributes and release policies without resetting session state for all
authenticated users by restarting Tomcat entirely. Given this issue,
however, we've delayed implementation of this feature and instead scheduled
a restart of Tomcat nightly to pick up any changes made the previous day,
which means even the smallest change doesn't take effect until the following
day.

What is preventing the IdP from using an SP's encryption certificate
following a reload of the relying party service, when the same SP works fine
prior the reload?

Server: RHEL6 x64

IdP: 2.3.6

Tomcat: 6.0.35.0

JVM: 1.7.0_03

Brett Slaughter

Information Security & Access Management | ISAM

Division of IT, University of Missouri

 <mailto:slaughter at missouri.edu> slaughter at missouri.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20120402/541d3d72/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5992 bytes
Desc: not available
Url : http://shibboleth.net/pipermail/users/attachments/20120402/541d3d72/attachment-0001.bin 