<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:#0563C1;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:#954F72;
        text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin-top:0in;
        margin-right:0in;
        margin-bottom:0in;
        margin-left:.5in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.EmailStyle17
        {mso-style-type:personal-compose;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-family:"Calibri",sans-serif;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
/* List Definitions */
@list l0
        {mso-list-id:606541809;
        mso-list-type:hybrid;
        mso-list-template-ids:1225416322 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
        {mso-level-text:"%1\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level2
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level3
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l0:level4
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level5
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level6
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l0:level7
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level8
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level9
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l1
        {mso-list-id:1985623959;
        mso-list-type:hybrid;
        mso-list-template-ids:-212948590 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
        {mso-level-text:"%1\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1:level2
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1:level3
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l1:level4
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1:level5
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1:level6
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l1:level7
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1:level8
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1:level9
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l2
        {mso-list-id:2052681259;
        mso-list-type:hybrid;
        mso-list-template-ids:172098166 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
        {mso-level-text:"%1\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l2:level2
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l2:level3
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l2:level4
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l2:level5
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l2:level6
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l2:level7
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l2:level8
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l2:level9
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
ol
        {margin-bottom:0in;}
ul
        {margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>This week, we engaged with performance testing of our PeopleSoft system and I wanted to share our key findings in the event it can help someone else.  I am summarizing the findings below, so please grant me the liberty to be slightly off in the description.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>Background:<o:p></o:p></b></p><p class=MsoNormal>Our PeopleSoft system is used by faculty, staff, and students and is configured to be accessed via web proxies.  The web proxies (2) are load balanced (round robin, cookie based) and configured as Shibboleth SPs, which defers authentication to our load balanced (round robin, cookie based) Shibboleth IDPs (2), which then perform the actual authentication via CAS (single node, for now) that uses AD.   <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>Findings:<o:p></o:p></b></p><p class=MsoNormal>During our performance testing, when we tried to authenticate 1,000 users at 5 per second and perform a series of tasks within PeopleSoft we would encounter a high error rate that our PeopleSoft weblogic and app servers could not recover from unless we restarted those services.  In digging into where the errors were being generated, we uncovered the following:<o:p></o:p></p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l2 level1 lfo1'>Our two Shibboleth IDPs were targeting the same DCs (in the same order) for attribute resolution per the attribute-resolver.xml file<o:p></o:p></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l2 level1 lfo1'>Our CAS server was also targeting the same DC for authentication<o:p></o:p></li></ol><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>In essence, the single DC was seeing traffic from three hosts, two for attribute resolution (IDPs) and one for authentication (CAS) at the same time resulting in CPU usage from 60% to 100% during our load testing.  <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>Actions taken:<o:p></o:p></b></p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l1 level1 lfo2'>We changed the order within our Shibboleth IDPs (attribute-resolver.xml) so the first DC they would hit for attribute resolution would be different  <o:p></o:p></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l1 level1 lfo2'>We changed the DC that CAS was using for authentication to not be either DC the Shibboleth IDPs were using for attribute resolution; i.e. a 3<sup>rd</sup> DC.  <o:p></o:p></li></ol><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>Results:<o:p></o:p></b></p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo3'>Each DC, when conducting the load test, now operates around 40% CPU usage<o:p></o:p></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo3'>The 3<sup>rd</sup> DC, the one that CAS is using for authentication, is operating around 15% CPU usage when conducing the load test<o:p></o:p></li></ol><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We were tipped off by this post: <a href="http://shibboleth.1660669.n2.nabble.com/Shibboleth-idp-performance-question-td7589479.html">http://shibboleth.1660669.n2.nabble.com/Shibboleth-idp-performance-question-td7589479.html</a>  that our IDPs could be a bottleneck if attribute resolution and/or authentication was taking too long <-Thanks Scott!  <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>That being said, we still have more work to do as when we perform a load test with 1,000 users at 10 per second we encounter a high error rate.  We are hoping to determine how we can configure the system to not error out, but to resolve gracefully.  <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Again, hopefully our findings can help someone else with their configuration.  <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks, Jay <o:p></o:p></p><p class=MsoNormal>________________________________<o:p></o:p></p><p class=MsoNormal>Jason Rappaport<o:p></o:p></p><p class=MsoNormal>Identity and Access Management Analyst<o:p></o:p></p><p class=MsoNormal>Office of Information Technology<o:p></o:p></p><p class=MsoNormal>Email:  <a href="mailto:jasonrap@princeton.edu"><span style='color:windowtext'>jasonrap@princeton.edu</span></a> <o:p></o:p></p><p class=MsoNormal>Office:  609-258-8464<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>