<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Calibri Light";
panose-1:2 15 3 2 2 2 4 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:46684078;
mso-list-type:hybrid;
mso-list-template-ids:-762829510 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">First off, thank you to everyone who provided input regarding the load issues we experienced with the start of the semester. As expected, load testing has pointed pretty conclusively to the data connector being the weak link. Testing
has shown that at any realistic load we experience at any given time, the IdP doesn’t break a sweat. Over the last several weeks I’ve been focusing on our LDAP implementation and have identified some key areas that we can improve (all of which will be phased
in over the next couple of months in preparation for the start of the Spring semester).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve built some new LDAP servers that will be used exclusively for our Shib environment but I’m getting some sporadic timeouts to these new servers that I can’t explain (and don’t seem to be happening in the same fashion in our current
production). In troubleshooting the issue, I’ve been able to replicate it when connecting a single IdP to a single new LDAP (eliminating the network load balancer), then change over to current production LDAP and see the problem not happen. I’m pretty certain
it’s my newly built LDAP servers (same LDAP we use with production: Microsoft’s AD LDS server – not Active Directory) and not an IdP issue itself. What’s incredibly frustrating is the timeout issue goes away after a number of attempts (so a bunch of timeouts
for 2-3 minutes, then it starts working flawlessly) and I have no issues until I restart the IdP (and/or LDAP server). This “number of attempts” doesn’t seem to be related to the initial startup of IdP or LDAP, I’ve restarted both and let them sit for minutes
and sometimes hours before testing. It’s like I have to hammer the LDAP server for a period of time to get it to “wake up” for searches, although binding never has an issue. Although I can’t completely eliminate the firewall from my testing, the network
team assures me I’m not getting any disruption from that front. Like production, these are virtual machines but beefed up significantly from current production.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From a packet capture of the LDAP traffic to and from the IdP and LDAP, I see:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>typically just under 1 second between searchRequest from IdP and searchResDone success back from production LDAP.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>on new build LDAP with same IdP when the server is “asleep”:<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>when responseTimeout is default 3 seconds, I get the IdP waiting 3 seconds then sending an abandonRequest to LDAP. The IdP logs a “timeout connecting to LDAP” type error and I get no attributes back. This all seems as expected.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>when responseTimeout is changed to 30 seconds, I get the LDAP server only waiting 4 seconds before sending back searchResDone timeLimitExceeded to the IdP. The IdP doesn’t log any error, I just get no attributes back.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">c.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>on a new build LDAP with same IdP but the server has decided to “wake up””<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>I see the response time start to improve (and once under the 4 second timeout start to succeed) as attempts progress, eventually reaching the same just under 1 second elapsed time that I see with our production LDAP environment.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m still convinced the issue is ultimately with my new LDAP servers, but my question is about that 4 seconds… where is that coming from? The packet capture seems to clearly show (in production and newly built IdP) that the IdP is including
that value regardless of any timeout value in the IdP (see example below). I’m at a loss as to why I’m seeing that and what I can do about it (at least for my testing and troubleshooting). Any ideas?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Of course, I’ve been staring at this stuff for weeks, so hopefully I’m not just looking past something simple (highly possible).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Example searchRequest from IdP to LDAP:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif">searchRequest<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> baseObject: ou=Active,ou=People,dc=example,dc=domain,dc=com<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> scope: wholeSubtree (2)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> derefAliases: neverDerefAliases (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> sizeLimit: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> timeLimit: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> typesOnly: False<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> Filter: (uid=123456789)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> filter: equalityMatch (3)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> equalityMatch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> attributeDesc: cn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> assertionValue: 123456789<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri Light",sans-serif"> attributes: 0 items<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ryan Tapp<o:p></o:p></p>
<p class="MsoNormal">California State University Long Beach<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>