<style>
/* Changing the layout to use less space for mobiles */
@media screen and (max-device-width: 480px), screen and (-webkit-min-device-pixel-ratio: 2) {
#email-body { min-width: 30em !important; }
#email-page { padding: 8px !important; }
#email-banner { padding: 8px 8px 0 8px !important; }
#email-avatar { margin: 1px 8px 8px 0 !important; padding: 0 !important; }
#email-fields { padding: 0 8px 8px 8px !important; }
#email-gutter { width: 0 !important; }
}
</style>
<div id="email-body">
<table id="email-wrap" align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#f0f0f0;color:#000000;width:100%;">
<tr valign="top">
<td id="email-page" style="padding:16px !important;">
<table align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#ffffff;border:1px solid #bbbbbb;color:#000000;width:100%;">
<tr valign="top">
<td bgcolor="#ffffff" style="background-color:#ffffff;color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;line-height:1;"><img src="https://shibboleth.net/images/shib_240x83.png" alt="" style="vertical-align:top;" /></td>
</tr><tr valign="top">
<td id="email-banner" style="padding:32px 32px 0 32px;">
<table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="width:100%;">
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;padding:0;">
<img id="email-avatar" src="https://issues.shibboleth.net/jira/secure/useravatar?avatarId=10202" alt="" height="48" width="48" border="0" align="left" style="padding:0;margin: 0 16px 16px 0;" />
<div id="email-action" style="padding: 0 0 8px 0;font-size:12px;line-height:18px;">
<a class="user-hover" rel="takeshi@gakunin.nii.ac.jp" id="email_takeshi@gakunin.nii.ac.jp" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=takeshi%40gakunin.nii.ac.jp" style="color:#326ca6;">Takeshi Nishimura</a>
commented on <img src="https://issues.shibboleth.net/jira/images/icons/issuetypes/bug.png" height="16" width="16" border="0" align="absmiddle" alt="Bug"> <a style='color:#326ca6;text-decoration:none;' href='https://issues.shibboleth.net/jira/browse/JOST-220'>JOST-220</a>
</div>
<div id="email-summary" style="font-size:16px;line-height:20px;padding:2px 0 16px 0;">
<a style='color:#326ca6;text-decoration:none;' href='https://issues.shibboleth.net/jira/browse/JOST-220'><strong>IdP stopping metadata retrieval</strong></a>
</div>
</td>
</tr>
</table>
</td>
</tr>
<tr valign="top">
<td id="email-fields" style="padding:0 32px 32px 32px;">
<table border="0" cellpadding="0" cellspacing="0" style="padding:0;text-align:left;width:100%;" width="100%">
<tr valign="top">
<td id="email-gutter" style="width:64px;white-space:nowrap;"></td>
<td>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr valign="top">
<td colspan="2" style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 16px 0;width:100%;">
<div class="comment-block" style="background-color:#edf5ff;border:1px solid #dddddd;color:#000000;padding:12px;"><p>We have similar issues.<br/>
If you encounter similar issue, please take information of running Java threads to see the causes of our issues are the same.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ sudo jstack -l <java's PID>
</pre>
</div></div>
<p>I think the cause is a stall of Java thread "Timer-0".</p>
<p>Our case is on Shibboleth IdP 2.3.6 / Tomcat 6.0.24 / OpenJDK 1.6.0 / CentOS 6. We added remote attribute filter as below.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml"><span class="code-tag"><srv:Service id=<span class="code-quote">"shibboleth.AttributeFilterEngine"</span> xsi:type=<span class="code-quote">"attribute-afp:ShibbolethAttributeFilteringEngine"</span> configurationResourcePollingFrequency=<span class="code-quote">"PT60M"</span> configurationResourcePollingRetryAttempts=<span class="code-quote">"999"</span>></span>
<span class="code-tag"><srv:ConfigurationResource file=<span class="code-quote">"/opt/shibboleth-idp/conf/attribute-filter.xml"</span> xsi:type=<span class="code-quote">"resource:FilesystemResource"</span>/></span>
<srv:ConfigurationResource xsi:type=<span class="code-quote">"resource:FileBackedHttpResource"</span>
url=<span class="code-quote">"https://..."</span>
file=<span class="code-quote">"..."</span> />
<span class="code-tag"></srv:Service></span>
</pre>
</div></div>
<p>When the issue occured a Java thread called "Timer-0" was stalled during retrieving remote attribute filter. This caused stopping all auto-retrieving feature by the timer, so metadata has not beed refreshed too.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>$ sudo -u tomcat jstack -l 21728
2014-11-11 18:30:41
Full thread dump OpenJDK 64-Bit Server VM (20.0-b12 mixed mode):
(snip)
"Timer-0" daemon prio=10 tid=0x00007f9ee4a5c800 nid=0x5501 runnable [0x00007f9ec5c03000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:312)
at sun.security.ssl.InputRecord.read(InputRecord.java:350)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
- locked <0x00000000d7d3eb00> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
- locked <0x00000000d7d3eb60> (a sun.security.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00000000d7d47290> (a java.io.BufferedInputStream)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.opensaml.util.resource.HttpResource.getResource(HttpResource.java:177)
at org.opensaml.util.resource.FileBackedHttpResource.getInputStream(FileBackedHttpResource.java:131)
at edu.internet2.middleware.shibboleth.common.config.SpringConfigurationUtils.populateRegistry(SpringConfigurationUtils.java:76)
at edu.internet2.middleware.shibboleth.common.config.BaseService.loadContext(BaseService.java:170)
at edu.internet2.middleware.shibboleth.common.config.BaseReloadableService$ConfigurationResourceListener.onResourceUpdate(BaseReloadableService.java:195)
at org.opensaml.util.resource.ResourceChangeWatcher.signalListeners(ResourceChangeWatcher.java:195)
- locked <0x00000000d7496a30> (a java.util.ArrayList)
at org.opensaml.util.resource.ResourceChangeWatcher.run(ResourceChangeWatcher.java:156)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- <0x00000000d7433aa8> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
(snip)
</pre>
</div></div>
<p>We have such issues 3 or 4 times per year.</p>
<p>Any comments are welcome.</p></div>
<div style="color:#505050;padding:4px 0 0 0;"> </div>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td><!-- End #email-page -->
</tr>
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:10px;line-height:14px;padding: 0 16px 16px 16px;text-align:center;">
This message is automatically generated by JIRA.<br />
If you think it was sent incorrectly, please contact your JIRA administrators<br />
For more information on JIRA, see: <a style='color:#326ca6;' href='http://www.atlassian.com/software/jira'>http://www.atlassian.com/software/jira</a>
</td>
</tr>
</table><!-- End #email-wrap -->
</div><!-- End #email-body -->