ldap queries hanging
Ted Wood
TWood at parker.edu
Thu Dec 15 15:44:39 EST 2016
Hello shibboleth users,
I’m running into an issue where authentication seems to hang with this message in the idp-process.log:
2016-12-15 11:33:40,858 - DEBUG [org.ldaptive.auth.PooledSearchDnResolver:244] - resolve user=joe_random
2016-12-15 11:33:40,859 - DEBUG [org.ldaptive.auth.PooledSearchDnResolver:310] - searching for DN using userFilter
Normally this would be followed by a successful bind and then authentication but occasionally it will just hang there and then seemingly all logins are blocked until I restart tomcat.
I have three dc’s in my ldap pool but I can’t get the connectionStrategy to change from ACTIVE_PASSIVE but I don’t know if that has any bearing on this issue. DNS is working fine and an ldapsearch from any of the servers returns the expected results immediately but I thought I should mention this. I’ve also done a tcpdump and watched as a number of users authenticate and the authentication – when it’s not hung – always goes to the first dc in the list. To rule out a particular dc, I’ve changed the order and the first one in the list is always used and the problem also always eventually happens so I can’t say that a dc isn’t timing out but all other AD functionality is working fine.
I’m running it on a cluster of three identical virtual machines on Vmware that are load balanced with lvs with persistence at five minutes.
I am also using the memcached storage service which I suspect may be playing a role in this but I can’t put my finger on it.
I’ve tried removing all but one of the idp servers from the cluster for a day or so at a time but the issue always happens eventually so I don’t think it’s related to a particular box.
Any help would be greatly appreciated.
My configuration is:
Idp version 3.1.1
Tomcat info:
Server version: Apache Tomcat/8.5.5
Server built: Aug 31 2016 19:51:16 UTC
Server number: 8.5.5.0
OS Name: Linux
OS Version: 3.10.0-327.36.3.el7.x86_64
Architecture: amd64
JVM Version: 1.8.0_111-b14
JVM Vendor: Oracle Corporation
The spymemcached jar is spymemcached-2.11.4.jar.
Ldaptive jar is ldaptive-1.0.6.jar.
Ldap.properties:
# LDAP authentication configuration, see authn/ldap-authn-config.xml
## Authenticator strategy, either anonSearchAuthenticator, bindSearchAuthenticator, directAuthenticator, adAuthenticator
idp.authn.LDAP.authenticator = bindSearchAuthenticator
## Connection properties ##
idp.authn.LDAP.ldapURL = ldaps://my.redacted.domain.controller:3269 ldaps:// my.redacted.domain.controller:3269 ldaps:// my.redacted.domain.controller:3269
idp.authn.LDAP.useStartTLS = false
idp.authn.LDAP.useSSL = true
#idp.authn.LDAP.connectTimeout = 5000
## SSL configuration, either jvmTrust, certificateTrust, or keyStoreTrust
#idp.authn.LDAP.sslConfig = certificateTrust
## If using certificateTrust above, set to the trusted certificate's path
idp.authn.LDAP.trustCertificates = %{idp.home}/credentials/redacteddomain.edu.crt
## If using keyStoreTrust above, set to the truststore path
idp.authn.LDAP.trustStore = %{idp.home}/credentials/ldap-server.truststore
## Return attributes during authentication
## NOTE: this is not used during attribute resolution; configure that directly in the
## attribute-resolver.xml configuration via a DataConnector's <dc:ReturnAttributes> element
idp.authn.LDAP.returnAttributes = mail,givenname,sn,objectguid,samaccountname
## DN resolution properties ##
# Search DN resolution, used by anonSearchAuthenticator, bindSearchAuthenticator
# for AD: CN=Users,DC=example,DC=org
idp.authn.LDAP.baseDN = dc=redacted,dc=edu
idp.authn.LDAP.subtreeSearch = true
idp.authn.LDAP.userFilter = (| (samaccountname={user}) (userprincipalname={user}) )
# bind search configuration
# for AD: idp.authn.LDAP.bindDN=adminuser at domain.com
idp.authn.LDAP.bindDN = cn=REDACTED_AD_USERNAME RO,ou=shibboleth,ou=services,dc=parkernet,dc=edu
idp.authn.LDAP.bindDNCredential = REDACTED_AD_PASSWORD
# Format DN resolution, used by directAuthenticator, adAuthenticator
# for AD use idp.authn.LDAP.dnFormat=%s at domain.com
idp.authn.LDAP.dnFormat = uid=%s,ou=people,dc=example,dc=org
# LDAP attribute configuration, see attribute-resolver.xml
idp.attribute.resolver.LDAP.ldapURL = %{idp.authn.LDAP.ldapURL}
idp.attribute.resolver.LDAP.baseDN = %{idp.authn.LDAP.baseDN}
idp.attribute.resolver.LDAP.bindDN = %{idp.authn.LDAP.bindDN}
idp.attribute.resolver.LDAP.bindDNCredential = %{idp.authn.LDAP.bindDNCredential}
idp.attribute.resolver.LDAP.useStartTLS = %{idp.authn.LDAP.useStartTLS:true}
idp.attribute.resolver.LDAP.trustCertificates = %{idp.authn.LDAP.trustCertificates}
idp.attribute.resolver.LDAP.searchFilter = (| (samaccountname=$requestContext.principalName) (userprincipalname=$requestContext.principalName) )
# LDAP pool configuration, used for both authn and DN resolution
#idp.pool.LDAP.minSize = 3
#idp.pool.LDAP.maxSize = 10
#idp.pool.LDAP.validateOnCheckout = false
#idp.pool.LDAP.validatePeriodically = true
#idp.pool.LDAP.validatePeriod = 300
#idp.pool.LDAP.prunePeriod = 300
#idp.pool.LDAP.idleTime = 600
#idp.pool.LDAP.blockWaitTime = 3000
#idp.pool.LDAP.failFastInitialize = false
memcached bean from global.xml:
<bean id="shibboleth.MemcachedStorageService"
class="org.opensaml.storage.impl.memcached.MemcachedStorageService"
c:timeout="2">
<constructor-arg name="client">
<bean class="net.spy.memcached.spring.MemcachedClientFactoryBean"
p:servers="10.13.6.39:11211,10.13.6.42:11211,10.13.6.46:11211"
p:protocol="BINARY"
p:locatorType="ARRAY_MOD"
p:failureMode="Redistribute">
<property name="hashAlg">
<util:constant static-field="net.spy.memcached.DefaultHashAlgorithm.FNV1_64_HASH" />
</property>
<property name="transcoder">
<!-- DO NOT MODIFY THIS PROPERTY -->
<bean class="org.opensaml.storage.impl.memcached.StorageRecordTranscoder" />
</property>
</bean>
</constructor-arg>
</bean>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20161215/043189a3/attachment-0001.html>
More information about the users
mailing list