<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://issues.shibboleth.net/jira">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-792dda78-db98-4421-a0d5-8fb358149e3d" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="000998514@neu.edu" id="email_000998514@neu.edu" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=000998514%40neu.edu" style="color:#0052cc;; color: #3b73af; text-decoration: none">David Mak</a> <strong>commented</strong> on <a href="https://issues.shibboleth.net/jira/browse/IDP-1483" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-53ab7859-0bbf-4cf4-a4f5-248cb25df9d5" height="16" width="16" border="0" align="absmiddle" alt="Bug"> IDP-1483</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/IDP-1483" style="color: #3b73af; text-decoration: none">Re: CAS Proxy authentication failures require IDP restart to resolve.</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> <p style="margin: 10px 0 0 0; margin-top: 0">Thank you for the suggestion on creating a separate HttpClient bean for the CAS Proxy AuthN requests. We will look into doing that for the interim. What is your opinion on changing the defaults in the services.properties configuration for the default HTTP client config, particularly for increasing the max # of connections or decreasing the timeout?</p> <p style="margin: 10px 0 0 0">The limited testing I have done to troubleshoot / isolate the TCP connection problems we're having shows that it is indeed poorly behaving TCP connections that results in our exhaustion of those default 100 connections. We fear that even having a dedicated CAS proxy http bean may not be sufficient during high load. Some tcpdumps of the http requests from the IDP to the CAS proxy SP/endpoint shows that the server is shutting down the TCP connection with a FIN-ACK, which results in the connection on the IDP going into CLOSE-WAIT state. Eventually, the CAS Proxy/SP sends a RESET, at which point the connection on the IDP is closed. We see that in most cases, that RESET comes after about 5 minutes.</p> <p style="margin: 10px 0 0 0">We suspect that with enough sessions, and a five minute window before the TCP connections are closed, we can easily exhaust the 100 default limit.</p> <p style="margin: 10px 0 0 0">Here's an example of what we see - one data packet, followed by a FIN-ACK from the CAS Proxy service on 155.33.26.54/https with an ACK response from the IDP server:</p> <p style="margin: 10px 0 0 0"> </p>
<div class="preformatted panel" style="border-width: 1px;; border: 1px solid #cccccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #cccccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="preformattedContent panelContent" style="padding: 9px 12px">
<pre style="margin: 10px 0 0 0; margin-top: 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal; white-space: pre; word-break: normal; word-wrap: break-word; word-break: break-word; white-space: pre-wrap">00:19:00.145799 00:23:e9:88:32:ca > 00:50:56:87:00:5c, ethertype IPv4 (0x0800), length 340: 155.33.26.54.443 > 155.33.26.222.54298: Flags [P.], seq 6650:6924, ack 792, win 5171, options [nop,nop,TS val 2422716678 ecr 1984379442], length 274
00:19:00.145815 00:23:e9:88:32:ca > 00:50:56:87:00:5c, ethertype IPv4 (0x0800), length 66: 155.33.26.54.443 > 155.33.26.222.54298: Flags [F.], seq 6924, ack 792, win 5171, options [nop,nop,TS val 2422716678 ecr 1984379442], length 0
00:19:00.185258 00:50:56:87:00:5c > 00:23:e9:88:32:ca, ethertype IPv4 (0x0800), length 66: 155.33.26.222.54298 > 155.33.26.54.443: Flags [.], ack 6925, win 23168, options [nop,nop,TS val 1984379666 ecr 2422716678], length 0
</pre>
</div>
</div> <p style="margin: 10px 0 0 0"> </p> <p style="margin: 10px 0 0 0">Here's the reset that comes five minutes later:</p>
<div class="preformatted panel" style="border-width: 1px;; border: 1px solid #cccccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #cccccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="preformattedContent panelContent" style="padding: 9px 12px">
<pre style="margin: 10px 0 0 0; margin-top: 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal; white-space: pre; word-break: normal; word-wrap: break-word; word-break: break-word; white-space: pre-wrap">00:24:00.663872 00:23:e9:88:32:ca > 00:50:56:87:00:5c, ethertype IPv4 (0x0800), length 60: 155.33.26.54.443 > 155.33.26.222.54298: Flags [R.], seq 6925, ack 792, win 0, length 0</pre>
</div>
</div> <p style="margin: 10px 0 0 0">This is the CLOSE_WAIT state which goes away after the RESET:</p> <p style="margin: 10px 0 0 0"> </p>
<div class="preformatted panel" style="border-width: 1px;; border: 1px solid #cccccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #cccccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="preformattedContent panelContent" style="padding: 9px 12px">
<pre style="margin: 10px 0 0 0; margin-top: 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal; white-space: pre; word-break: normal; word-wrap: break-word; word-break: break-word; white-space: pre-wrap">Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 1 0 ::ffff:155.33.26.222:54298 ::ffff:155.33.26.54:443 CLOSE_WAIT 12111/java
</pre>
</div>
</div> <p style="margin: 10px 0 0 0"> </p> <p style="margin: 10px 0 0 0"> </p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/IDP-1483#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-6e0f81a0-1988-4947-8418-5383f47034c9" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.shibboleth.net/jira/browse/IDP-1483#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v8.5.1#805001-<span title="c5b54a7317f297e17ddad5bb52a1850cf2bdb4bc" data-commit-id="c5b54a7317f297e17ddad5bb52a1850cf2bdb4bc}">sha1:c5b54a7</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="https://issues.shibboleth.net/jira/images/mail/atlassian-email-logo.png" alt="Atlassian logo" title="Atlassian logo" width="191" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>