<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hey everyone,</div>
<div><br>
</div>
<div>Earlier today, two of our three Shibboleth IdP servers went down in rapid succession, seemingly for the same reason. I originally suspected it was a metadata error since we've seen a few of those recently, but it appears to have been Tomcat-related.</div>
<div><br>
</div>
<div><span style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);">[</span><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);">error</span><span style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);">]
</span><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);"><span class="t"><span class="t"><span class="t">ajp</span>_<span class="t">read</span></span>_<span class="t">header</span></span>:</span><span style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);">
</span><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);"><span class="t"><span class="t">ajp</span>_<span class="t">ilink</span></span>_<span class="t">receive</span></span><span style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);">
</span><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);">failed</span></div>
<div><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);"><span style="font-variant-ligatures: normal;">[</span><span class="t" style="font-variant-ligatures: normal;">error</span><span style="font-variant-ligatures: normal;">]
(</span><span class="t" style="font-variant-ligatures: normal;">70007</span><span style="font-variant-ligatures: normal;">)</span><span class="t" style="font-variant-ligatures: normal;">The</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">timeout</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">specified</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">has</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t">expired</span>:</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t">proxy</span>:</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">read</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">response</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">failed</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">from</span><span style="font-variant-ligatures: normal;"> (</span><span class="t" style="font-variant-ligatures: normal;">null</span><span style="font-variant-ligatures: normal;">) (</span><span class="t" style="font-variant-ligatures: normal;">localhost)</span></span></div>
<div><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);"><span class="t" style="font-variant-ligatures: normal;"><span style="font-variant-ligatures: normal;">[</span><span class="t" style="font-variant-ligatures: normal;">error</span><span style="font-variant-ligatures: normal;">]
(</span><span class="t" style="font-variant-ligatures: normal;">70007</span><span style="font-variant-ligatures: normal;">)</span><span class="t" style="font-variant-ligatures: normal;">The</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">timeout</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">specified</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">has</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t">expired</span>:</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t">proxy</span>:</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">read</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">response</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">failed</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">from</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t"><span class="t"><span class="t"><span class="t">127</span>.<span class="t">0</span></span>.<span class="t">0</span></span>.<span class="t">1</span></span>:<span class="t">8009</span></span><span style="font-variant-ligatures: normal;">
(</span><span class="t" style="font-variant-ligatures: normal;">localhost)</span></span></span></div>
<div><span class="t" style="color: rgb(51, 51, 51); font-family: 'Droid Sans Mono', Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; font-variant-ligatures: normal; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgba(61, 171, 255, 0.117647);"><span class="t" style="font-variant-ligatures: normal;"><span style="font-variant-ligatures: normal;">[</span><span class="t" style="font-variant-ligatures: normal;">error</span><span style="font-variant-ligatures: normal;">]
(</span><span class="t" style="font-variant-ligatures: normal;">70007</span><span style="font-variant-ligatures: normal;">)</span><span class="t" style="font-variant-ligatures: normal;">The</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">timeout</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">specified</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">has</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t">expired</span>:</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;"><span class="t"><span class="t">ajp</span>_<span class="t">ilink</span></span>_<span class="t">receive</span></span><span style="font-variant-ligatures: normal;">()
</span><span class="t" style="font-variant-ligatures: normal;">can</span><span style="font-variant-ligatures: normal;">'</span><span class="t" style="font-variant-ligatures: normal;">t</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">receive</span><span style="font-variant-ligatures: normal;">
</span><span class="t" style="font-variant-ligatures: normal;">header</span></span></span></div>
<div><br>
</div>
<div>These aren't terribly abnormal errors, I understand. What's confusing us is why this would have happened on two separate servers within minutes of one another (it doesn't seem to be load-related). Could this be caused by a response type or a string included
in a response, maybe?</div>
<div><br>
</div>
<div>Any help would be appreciated. We've never seen it before and we hope to never see it again, but we came very close to a system outage and would love to figure out how to guarantee that it doesn't pop up again.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Ben</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</body>
</html>