Shibboleth idp performance question

[Jim Ennis]

Hello Manuel

Thanks for the many good suggestions.

Our servers are Sun T4 servers with 64GB each, no signs of swapping is occurring.

We are querying/authenticating against a VM based AD domain.

I will talk to the primary Shibboleth expert and see when we can try some of your suggestions against our environments.

Jim Ennis
Director Systems and Operations
University of Central Florida
12716 Pegasus Drive
CSB 308
Orlando, FL 32816

E-mail: jim.ennis at ucf.edu
Voice: 407-823-1701
Fax: 407-882-9017


-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Manuel Haim
Sent: Wednesday, August 21, 2013 3:48 AM
To: users at shibboleth.net
Subject: Re: Shibboleth idp performance question

Hi Jim,

> Any advice or tips would be recommended.

performance was an issue here when we started using the IdP, so here are some ideas:

- Make sure your machines have enough RAM (and do not need to use SWAP) (we use KVM virtualized machines, each with 4 CPUs @ 2.5 GHz and 3GB RAM)

- Increase Java heap space for tomcat (e.g. add option "-Xmx1536m" to JAVA_OPTS environment variable; under Debian Linux you therefor need to edit the file /etc/default/tomcat6 )

- Turn on debug mode (or trace mode) on one IdP server and perform a single login (make sure that only you access that IdP server, as otherwise the logs will be flooded with useless information). Then have a look at the logs to see which operation needs which time (in milliseconds). How long does a single login attempt run, where could you optimize?

These are our values from 2011 after optimization (we only query LDAP):
100ms LDAP Authentication
 15ms other things (don't know)
 60ms attribute resolution (getting account data from LDAP)  80ms attribute resolution (getting personal data from LDAP)  50ms other things (attribute filter, signing, SAML stuff) (i.e. 305ms total for a single login attempt)

- Do you query LDAP or SQL databases? How performant are your queries?
Could you add additional indexes somewhere?

- To see how well your IdP performs, start a performance test using "The Grinder" as described under https://wiki.shibboleth.net/confluence/display/SHIB2/IdPProdLoadTest

- When querying LDAP in the attribute resolver, the IdP usually uses connection pooling by default, but you could try to configure (and
optimize) a ConnectionPool in your DataConnector yourself, this is what we use:

  <dc:ConnectionPool
      minPoolSize="0"
      maxPoolSize="4"
      blockWhenEmpty="true"
      validateDN="o=YourOrg,c=YourCountry"
  />

(However, compared to the IdP default connection pooling, this does not seem to influence our total logins per second, but somehow decreases the latency for a single login attempt.)


FYI, we only query LDAP (and do not use a SQL database for persistentId / uApprove yet) and got about 60 logins per second (using 2 IdP machines behind a load balancer) when testing two years ago. Instead of Terracotta, we use the Memcached StorageService [1]. Even after 150,000 test logins, the system performed well (see screenshots starting from page 16 in our German presentation [2]).

Some more ideas can be found in my answer "Optimizing IdP's performance"
from 2011, see [3] (however, some information there may be obsolete by now).

-Manuel


[1]
Memcached StorageService
https://wiki.shibboleth.net/confluence/display/SHIB2/Memcached+StorageService

[2]
Shibboleth IdP clustering - Memcached vs Terracotta (performance screenshots starting at page 16) https://www.dfn.de/fileadmin/3Beratung/Betriebstagungen/bt55/forum-aai-haim.pdf

[3]
Optimizing IdP's performance
https://groups.google.com/forum/#!topic/shibboleth-users/9_D0XLW7XV8

--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net