Load Test Numbers with Different idp.properties settings

[Nate Klingenstein]

2 c3.large IdP nodes with ample everything else; same environment as described earlier.  Numbers are cumulative for whole cluster.

I couldn't get a whole lot of variance in the numbers.  I did validate that the IdP remains the bottleneck by dropping one node and observing TPS drop by more than half.  Please indicate other things you'd like me to test.

The biggest apparent performance issue is brief, sharp drops in TPS and increases in latency.  I'm guessing these are garbage collection events based on whatever GC is configured in this environment.


Uncommented delta; starting point mostly stock
peak TPS, rough average maximum TPS(eyeballed after everything has run for a few minutes, ~15000 tests in), error rate %


no change
97, 85, .1337% (no, I didn't make that up)


idp.storage.StorageService = org.opensaml.storage.impl.MemoryStorageService
idp.storage.cleanupInterval = PT10M
109, 93, .0866%


idp.storage.ClientStorageService = org.opensaml.storage.impl.ServletRequestScopedStorageService
101, 90, .0764%


idp.transientId.generator = shibboleth.CryptoTransientIdGenerator
99, 93, .1129%


idp.session.trackSPSessions = true
Startup appears to fail silently with logs at default levels.


idp.session.enabled = shibboleth.Conditions.True
idp.session.StorageService = shibboleth.ClientStorageService
100, 90, .113%


idp.session.enabled = shibboleth.Conditions.True
idp.storage.StorageService = org.opensaml.storage.impl.MemoryStorageService
idp.storage.cleanupInterval = PT10M
106, 95, .026%


idp.session.enabled = shibboleth.Conditions.True
idp.session.StorageService = shibboleth.ClientStorageService
idp.signing.config = shibboleth.SigningConfiguration.SHA256
idp.transientId.generator = shibboleth.CryptoTransientIdGenerator
#idp.nameid.saml2.legacyGenerator = shibboleth.LegacySAML2NameIDGenerator
#idp.nameid.saml1.legacyGenerator = shibboleth.LegacySAML1NameIdentifierGenerator
106, 97, .050%


idp.xml.securityManager = org.apache.xerces.util.SecurityManager
Jetty hangs on boot and must be killed (may be a local environment issue).  Looks like:
16:08:15.094 - INFO [net.shibboleth.idp.service.AbstractReloadableService:191] - Service 'shibboleth.LogbackLogging': reload time set to: 300000, starting refresh thread
16:08:15.103 - INFO [net.shibboleth.idp.service.AbstractReloadableService:236] - Service 'shibboleth.LogbackLogging': Starting shutdown
16:08:15.103 - INFO [net.shibboleth.idp.service.AbstractReloadableService:238] - Service 'shibboleth.LogbackLogging': Completing shutdown