Optimizing IdP's performance

[Chad La Joie]

Manual, thanks for bring up this topic.  I've included a bunch of
information below.  However, as I note every time this conversation
comes up; before anyone goes on an optimizing binge it's important to
understand:
- You must understand what your needs are.  Spending 6 months
customizing your IdP to perform 100 simultaneous SSOs when you only
ever see 10 in your real world use is a waste of time and likely
leaves you with an IdP setup that is harder to upgrade.
- The measure that you really care about, when doing this testing, is
# of simultaneous requests which perform under a given latency
threshold.  As I mention below, I've already done quite a bit of
per-request optimization in the IdP so people almost never need to
worry about the per-request latency except whatever is added by
querying attribute sources.
- If people ever feel like they've discovered a memory leak or a
specialized request that seems to take longer than it should, I can
provide instructions for capturing the data necessary for me to
analyze the IdP and either tell you why you're not seeing what you
think your seeing or fix any issue in the IdP.

On Mon, Aug 1, 2011 at 08:11, Manuel Haim <haim at hrz.uni-marburg.de> wrote:
> Hi list,
> 1) Enable debug mode (or even trace mode) in your IdP's logging.xml and
> perform a login. Then look at the logs. Which actions do consume most of
> the time? Anything above 50ms could be worth optimizing.

I've already done pretty extensive profiling on the IdP.  On an
average request, sans waiting for user input, approximately 50% of the
time is spent doing cryptography and approximately 50% of the time is
spent waiting for attribute data sources to return information.
Optimizing the crypto is possible but would be quite complicated for
deployers.  Optimizing attribute access is possible but different
solutions will apply for different deployments.  One option, if user
data doesn't change much, is to turn on caching in the data
connectors.  Another option is to host a "shadow" copy of the data
source (or at least the part that you care about) on the IdP machine
itself.

Note though, that optimizing any of this does not necessarily increase
throughput, it just decreases latency.

> 4) Give enough RAM to Tomcat. In our case, the options -Xms256m
> -Xmx1536m gave good results, even after 150.000 logins.

The amount of memory needed is directly proportional to the amount of
metadata that you are loading in the IdP.  Unfortunately I can't give
a good formula for computing the amount of memory used based on the
file size as it changes, sometimes quite drastically, with changes in
the XML parser itself.

> 5) Spend enough CPU power for your processes. Check the load on your
> servers - how busy is the CPU when performing a load test? If running on
> a virtual server, do some load testing with different numbers of CPUs,
> and compare the results.

Oracles and IBM's JVMs don't scale very well beyond 16 CPUs, so
throwing any more at the process quickly demonstrates the concept of
diminishing returns.  I *believe* that the Oracle Java 7 CPU can now
take advantage of the AES instructions showing up on various CPUs.  As
crypto takes up ~50% of all the time for a given request, having
access to these instructions would be very nice however most
virtualization technologies today do not allow access to them.  So
there would certainly be a trade off.

> 6) Optimize your software stack. We use a light-weight http(s) load
> balancer (pound) to connect to the Tomcat instances directly (no Apache
> needed). For back-channel requests to work correctly, the SSL Client
> Cert must be forwarded (and as we use http instead of an ajp connection,
> we needed to add or implement an SSLValve to Tomcat's server.xml which
> moves the certificate from the http header into the httpRequest object).

You can also try changing out Tomcat if you really want to optimize at
this layer.  Tomcat's connection handling is pretty poor in my areas
and has a couple of high-latency sections (due to global or
semi-global locks).  It also does not tend to handle high loads very
well.

> 8) If using LDAP: Optimize your LDAP queries. Do not use
> ":caseExactmatch:" if an attribute is not indexed case-sensitive. For
> your LDAP connectors in attribute-resolver.xml, try to define connection
> pools (however, when under load, the IdP already does connection pooling
> by default). And for LDAP authentication / JAAS Configuration
> (login.config), try to implement a custom dnResolver (which statically
> builds the entryDn, for example; by default, the IdP performs two costly
> LDAP binds for each login: one by the SearchDnResolver which looks for
> the user's entryDn but does not use connection pooling whatsoever - this
> one can be optimized as described - and one for the LDAP authentication
> request which must bind by the user's entryDn and thus cannot be pooled).

A potential big win for the LDAP data connector (total impact depends
on the LDAP server) is explicitly listing the attributes you want to
get back.

I *strongly* discourage people from simply creating the DN from given
input from the user.  It's a very brittle setup and I would remove any
such suggestion added to the wiki.  If that extra search, done via
pooled connections, is such a onerous amount of work then the problem
is with your LDAP.

-- 
Chad La Joie
www.itumi.biz
trusted identities, delivered