IdP capacity / load testing advice

[Cantor, Scott]

On 7/8/15, 11:06 AM, "users on behalf of Jerry Shipman" <users-bounces at shibboleth.net on behalf of jes59 at cornell.edu> wrote:


>1. does anyone have a ballpark-idea of how many simultaneous logins per second an IdP should be expected to handle?

That's very site dependent. Using it for webmail would probably double your load for example.

>2. in my preliminary test so far, I was logging a single user in over-and-over, in a bunch of parallel threads. Are there reasons to think that this is representative, or not representative, of multiple simultaneous users? i.e. should I make 1000 test accounts and use them instead? (maybe the IdP caches the user's record, or something, and so it would be faster to log 1 guy in 1000 times, than to log 1000 guys in 1 time?)

There will be slight differences. The bigger issue is that testing that way will bottleneck the SP in many cases without making some changes there, depending on the NameID configuration in the IdP.

>3. Does anyone have a feel for whether, when there is one (shib) SP and one (shib) IdP, which one should be the limiting factor? (my preliminary test suggests it is the IdP, but I am wondering if I should set up multiple dummy SPs on different machines to test against, or something like that.)

The bottleneck of thousands of sessions for a single NameID aside, the SP is faster.

>4. I think I can remove part of the concern about the SP as a limiting factor in question (3), by doing "IdP initiated login". Does that sound plausible? e.g., does the "IdP initiated login" go through the same code path inside the IdP, such that the results would be about representative of a normal login?

It eliminates a small amount of XML parsing. I don't know what you're testing exactly. Testing the IdP response alone without processing it at an SP will essentially mean that you can eliminate the SP from the test by doing IdP-initiated, and probably end up driving higher, though unrealistic, loads.

>5. What kinds of things should I look at that would be signs of trouble under load? So far I am looking at "load average" on the machines; the response time on the various http page submissions and redirects, etc; the total number of logins per second (across all threads); failed logins (i.e. can't get the content of the protected page on the SP) or http requests; errors in the IdP log. (It seems like I should look at the garbage collection log, but I couldn't immediately find it.)

I don't know anything about load testing, but you should never get errors or something's wrong.

>6. it seems like I might have our test IdP in debug mode. I'll try disabling that to see; but, do you think that would affect the result?

Drastically.

-- Scott