Diagnosing LDAP connection errors.

[Tom Poage]

We recently included an SP that substantially increased IdP activity.
Compounding the issue is the start-of-quarter bump in general demand on
most systems and their infrastructure.

I started noticing random periods of LDAP connection errors:

> 06:41:03.580 - ERROR [edu.vt.middleware.ldap.pool.DefaultLdapFactory:109] - unabled to connect to the ldap
> javax.naming.CommunicationException: ldap.ucdavis.edu:389
[stack trace]
> 06:41:03.582 - ERROR [edu.internet2.middleware.shibboleth.common.attribute.resolver.provider.ShibbolethAttributeResolver:379] - Received the following error from data connector ucdLDAP, no failover data connector available
[stack trace]

This suggests the connection pool might be empty:

>         at edu.vt.middleware.ldap.pool.DefaultLdapFactory.create(DefaultLdapFactory.java:28) [vt-ldap-3.3.4.jar:na]
>         at edu.internet2.middleware.shibboleth.common.attribute.resolver.provider.dataConnector.LdapPoolEmptyStrategy.checkOut(LdapPoolEmptyStrategy.java:53) [shibboleth-common-1.3.3.jar:na]
>         at edu.internet2.middleware.shibboleth.common.attribute.resolver.provider.dataConnector.LdapDataConnector.searchLdap(LdapDataConnector.java:367) [shibboleth-common-1.3.3.jar:na]

Is there something to look for in the stack traces that would tell me
the IdP LDAP connection pool is empty (vs. downstream resource issues)?

Recommendations/experiences tuning <ConnectionPool> for a less-than-idle
(~55k auths per day across three nodes) group of systems, cf.
ResolverLDAPDataConnector wiki page?

Thanks!
Tom.