IdP capacity / load testing advice

[Etienne Dysli-Metref]

Hi Jerry,

On 08/07/15 17:06, Jerry Shipman wrote:
> I am trying to do some load testing and get a feel for how our (shib
> 2.4) IdP does, for the purpose of planning a cluster. I suspect that
> there are a lot of subtleties involved in the load testing that I
> don't know about... I wonder if anybody can provide me with pointers
> about how best to do this?

There is a "Capacity Planning" page [1] in the wiki for IdPv3 with some
considerations about load testing. These are not specific to v3. There
is also a contributed JMeter test for IdPv3 and some people have posted
their results on the "Load Testing Contributed Results" page [2].

Since you mention clustering, it also helps to ask yourself why you want
a cluster: more capacity, more availability, or both?

> 4. I think I can remove part of the concern about the SP as a
> limiting factor in question (3), by doing "IdP initiated login". Does
> that sound plausible? e.g., does the "IdP initiated login" go through
> the same code path inside the IdP, such that the results would be
> about representative of a normal login?

Yes, you can remove the SP part with unsolicited SSO requests (easy with
v3). That's what the JMeter test on [2] does. Then you're only
exercising the IdP.

> 5. What kinds of things should I look at that would be signs of
> trouble under load? So far I am looking at "load average" on the
> machines; the response time on the various http page submissions and
> redirects, etc; the total number of logins per second (across all
> threads); failed logins (i.e. can't get the content of the protected
> page on the SP) or http requests; errors in the IdP log. (It seems
> like I should look at the garbage collection log, but I couldn't
> immediately find it.)

"Signs of trouble" depend on your definition of a reliable, available
service. For my testing, I defined: 1) zero errors 2) response time
under 5 seconds (99th percentile). If those numbers were exceeded, then
the maximum acceptable IdP load was reached. When this is enough to
handle expected peak load then fine, otherwise more capacity should be
added.

I don't think you need to look at the GC log (it's disabled by default,
by the way) unless you want to figure out the optimal heap size for your
JVM. If you want to squeeze more performance out of your IdP, then you
can dive into tuning connection pools, the JVM, etc. but it takes at lot
of time to get to know the internals in order to tune things properly.

> 7. I am currently using "the grinder" -- does anyone have reason to
> think it is not a good idea? So far I have run one test with "the
> grinder" and one with a perl script that I made, and they gave about
> the same result, so I figure that means it's OK.

As long as they report correct response times that should be OK.

Cheers,
   Etienne

[1] https://wiki.shibboleth.net/confluence/display/IDP30/Capacity+Planning
[2]
https://wiki.shibboleth.net/confluence/display/IDP30/Load+Testing+Contributed+Results

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://shibboleth.net/pipermail/users/attachments/20150709/2fa235ea/attachment.sig>