<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks for the suggestions.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Using Yourkit I was able to obtain this a capture of a deadlocked thread.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The idp-process.log does not seem to help in identifying what happens after its encoded the messages. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thoughts?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>http-443-1 <--- Frozen for at least 4m 7s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.core.AppenderBase.doAppend(Object)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(Object)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.classic.Logger.appendLoopOnAppenders(ILoggingEvent)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.classic.Logger.callAppenders(ILoggingEvent)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(String, Marker, Level, String, Object[], Throwable)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(String, Marker, Level, String, Object[], Throwable)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>ch.qos.logback.classic.Logger.info(String)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.idp.profile.saml1.AbstractSAML1ProfileHandler.writeAuditLogEntry(BaseSAMLProfileRequestContext)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.idp.profile.saml1.ShibbolethSSOProfileHandler.completeAuthenticationRequest(ShibbolethSSOLoginContext, HTTPInTransport, HTTPOutTransport)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.idp.profile.saml1.ShibbolethSSOProfileHandler.processRequest(HTTPInTransport, HTTPOutTransport)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.idp.profile.saml1.ShibbolethSSOProfileHandler.processRequest(InTransport, OutTransport)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.common.profile.ProfileRequestDispatcherServlet.service(HttpServletRequest, HttpServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.idp.util.NoCacheFilter.doFilter(ServletRequest, ServletResponse, FilterChain)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.idp.session.IdPSessionFilter.doFilter(ServletRequest, ServletResponse, FilterChain)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>edu.internet2.middleware.shibboleth.common.log.SLF4JMDCCleanupFilter.doFilter(ServletRequest, ServletResponse, FilterChain)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.StandardContextValve.invoke(Request, Response)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.StandardHostValve.invoke(Request, Response)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.coyote.http11.Http11Processor.process(Socket)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Socket)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>org.apache.tomcat.util.net.JIoEndpoint$Worker.run()<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>java.lang.Thread.run()<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> users-bounces@shibboleth.net [mailto:users-bounces@shibboleth.net] <b>On Behalf Of </b>Daniel Fisher<br><b>Sent:</b> Tuesday, May 01, 2012 11:04 AM<br><b>To:</b> Shib Users<br><b>Subject:</b> Re: IDP 2.3.6 Stops Sending SAML Responses.<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal style='margin-bottom:12.0pt'>I think you're going to need to identify the lock it's stuck on with a<br>debugger, but the other thing I would do is go trace the code and identify<br>what happens between the time it logs that message and the audit log entry<br>is done.<o:p></o:p></p></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>We use Yourkit for profiling and it's very helpful for tracking down deadlocks. If all else fails, run jstack when it locks up and try to identify the culprit.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>--Daniel Fisher<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div></div></div></body></html>