IdP capacity / load testing advice

[Jerry Shipman]

Hello,

I am trying to do some load testing and get a feel for how our (shib 2.4) IdP does, for the purpose of planning a cluster. I suspect that there are a lot of subtleties involved in the load testing that I don't know about... I wonder if anybody can provide me with pointers about how best to do this?

Here are some specific questions that I thought of:

1. does anyone have a ballpark-idea of how many simultaneous logins per second an IdP should be expected to handle?

2. in my preliminary test so far, I was logging a single user in over-and-over, in a bunch of parallel threads. Are there reasons to think that this is representative, or not representative, of multiple simultaneous users? i.e. should I make 1000 test accounts and use them instead? (maybe the IdP caches the user's record, or something, and so it would be faster to log 1 guy in 1000 times, than to log 1000 guys in 1 time?)

3. Does anyone have a feel for whether, when there is one (shib) SP and one (shib) IdP, which one should be the limiting factor? (my preliminary test suggests it is the IdP, but I am wondering if I should set up multiple dummy SPs on different machines to test against, or something like that.)

4. I think I can remove part of the concern about the SP as a limiting factor in question (3), by doing "IdP initiated login". Does that sound plausible? e.g., does the "IdP initiated login" go through the same code path inside the IdP, such that the results would be about representative of a normal login?

5. What kinds of things should I look at that would be signs of trouble under load? So far I am looking at "load average" on the machines; the response time on the various http page submissions and redirects, etc; the total number of logins per second (across all threads); failed logins (i.e. can't get the content of the protected page on the SP) or http requests; errors in the IdP log. (It seems like I should look at the garbage collection log, but I couldn't immediately find it.)

6. it seems like I might have our test IdP in debug mode. I'll try disabling that to see; but, do you think that would affect the result?

7. I am currently using "the grinder" -- does anyone have reason to think it is not a good idea? So far I have run one test with "the grinder" and one with a perl script that I made, and they gave about the same result, so I figure that means it's OK.

8. It seems like would is not an uncommon thing to try to do, and I should be able to find information in the mailing list archive. But I didn't find it (sorry). If someone remembers something that could help me search (e.g. who did it) then please let me know?

Thanks for the help,
Jerry Shipman