IdP capacity / load testing advice

[Jerry Shipman]

Hello,

Thanks for your help. 

In case anyone is interested, I found a couple of things so far: 
(1) telling the IdP to log at WARN instead of DEBUG increased the throughput by about 60%.
(2) IdP was CPU limited ... adding a second (virtual?) CPU to the VM (whatever it means) almost exactly doubled the throughput.

After I did those two things, it looks (just from looking at the machines' "load average") like the SP is the limiting factor. (the SP still just has the one virtual CPU so I think it makes sense.)

I think that I have got it set up such that the nameid is different for each iteration, even though the user is the same.
* can someone verify that this is how I would tell: by looking at this in the SP's transaction.log ?

2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1912]: New session (ID: _afd34...) with (applicationId: default) for principal from (IdP: https://shibidp-test.cit.cornell.edu/idp/shibboleth) at (ClientAddress: 128.253.101.144) with (NameIdentifier: 6XKYNVT6PLK5...) using (Protocol: urn:oasis:names:tc:SAML:2.0:protocol) from (AssertionID: _7eab0...)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1912]: Cached the following attributes with session (ID: _afd34...) for (applicationId: default) {
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1912]:         uid (1 values)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1912]:         eppn (1 values)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1912]: }
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1938]: New session (ID: _74483...) with (applicationId: default) for principal from (IdP: https://shibidp-test.cit.cornell.edu/idp/shibboleth) at (ClientAddress: 128.253.101.144) with (NameIdentifier: KTA3PUHTTFI...) using (Protocol: urn:oasis:names:tc:SAML:2.0:protocol) from (AssertionID: _0cbf88...)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1938]: Cached the following attributes with session (ID: _7448...) for (applicationId: default) {
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1938]:         uid (1 values)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1938]:         eppn (1 values)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1938]: }
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1990]: New session (ID: _7160f...) with (applicationId: default) for principal from (IdP: https://shibidp-test.cit.cornell.edu/idp/shibboleth) at (ClientAddress: 128.253.101.144) with (NameIdentifier: T5G5YRPVLB...) using (Protocol: urn:oasis:names:tc:SAML:2.0:protocol) from (AssertionID: _f75d5...)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1990]: Cached the following attributes with session (ID: _7160f...) for (applicationId: default) {
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1990]:         uid (1 values)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1990]:         eppn (1 values)
2015-07-10 12:11:31 INFO Shibboleth-TRANSACTION [1990]: }


> It eliminates a small amount of XML parsing. I don't know what you're testing exactly. Testing the IdP response alone without processing it at an SP will essentially mean that you can eliminate the SP from the test by doing IdP-initiated, and probably end up driving higher, though unrealistic, loads.

I am trying to figure out how to set up the IdP (or how many VMs I need, etc) in order to comfortably handle about 200 logins/second, which is (unless I screwed something up, which is always possible) the peak that I measured from our other (un-federated) login system. I guess that in the realistic case they would be coming from 100s of different users logging in against dozens of different SPs, or something. I guess I could make 200 test accounts and 10 test SPs; but it's more work, so I was trying to avoid that (as long as it's reasonable to do so and still test the IdP's characteristics). So, I am not worried about the SP's performance.

So far I have gotten up to about 40/s on the one VM, but I think the limiting factor is not the IdP. I thought it was the machine running "the grinder" to generate the load, but I ran it on a few load-generating-machines simultaneously and got the same result, so now I think it's the SP.

I think that next I will try either IdP-initiated-login-and-ignore-the-SP-entirely or just asking the systems guys to add another virtual CPU to the SP's VM. #2 is easier for me. I was just wondering how valid of an approach #1 is, as maybe it would "scale higher". I guess I will try both and see.

Anyway, thanks again for your help!

Jerry