Optimizing IdP's performance

[Manuel Haim]

Hi list,

I'd like to collect and hear some of your thoughts on optimizing the
overall performance of your IdP cluster - especially those ideas that
still seem to be missing on the Shibboleth wiki.

Here is what we got:
1) Enable debug mode (or even trace mode) in your IdP's logging.xml and
perform a login. Then look at the logs. Which actions do consume most of
the time? Anything above 50ms could be worth optimizing.
2) Disable debug mode before going productive or performing load tests
with the Grinder!
3) Have enough physical RAM available. If a node starts to use swap
space, it will become incredibly slow. Under Debian GNU/Linux, 2GB seem
to be good for a Tomcat setup without Terracotta, 3-4GB if you run
Tomcat and a Terracotta server on the same machine.
4) Give enough RAM to Tomcat. In our case, the options -Xms256m
-Xmx1536m gave good results, even after 150.000 logins.
5) Spend enough CPU power for your processes. Check the load on your
servers - how busy is the CPU when performing a load test? If running on
a virtual server, do some load testing with different numbers of CPUs,
and compare the results.
6) Optimize your software stack. We use a light-weight http(s) load
balancer (pound) to connect to the Tomcat instances directly (no Apache
needed). For back-channel requests to work correctly, the SSL Client
Cert must be forwarded (and as we use http instead of an ajp connection,
we needed to add or implement an SSLValve to Tomcat's server.xml which
moves the certificate from the http header into the httpRequest object).
7) Optimize Terracotta. What are your ideas here? Can the persistent
disk cache be turned off, so the Terracotta server runs in RAM only? The
Shibboleth wiki gives pretty less hints on this...
8) If using LDAP: Optimize your LDAP queries. Do not use
":caseExactmatch:" if an attribute is not indexed case-sensitive. For
your LDAP connectors in attribute-resolver.xml, try to define connection
pools (however, when under load, the IdP already does connection pooling
by default). And for LDAP authentication / JAAS Configuration
(login.config), try to implement a custom dnResolver (which statically
builds the entryDn, for example; by default, the IdP performs two costly
LDAP binds for each login: one by the SearchDnResolver which looks for
the user's entryDn but does not use connection pooling whatsoever - this
one can be optimized as described - and one for the LDAP authentication
request which must bind by the user's entryDn and thus cannot be pooled).

How do you optimize your IdP? (except for the points already mentioned
on the Productionalization pages in the Shibboleth wiki...)

-Manuel