LDAP authn connection pool locked up

[Daniel Fisher]

On Wed, Sep 14, 2016 at 2:05 PM, Chris Reeves <chris.reeves at york.ac.uk>
wrote:

> The first sign that we saw of issues with LDAP authentication were the
> following messages suggesting that the IdP was reconnecting:
> --------
> 2016-09-13T08:37:30.401+01:00 - WARN [org.ldaptive.AbstractOperatio
> n$ReopenOperationExceptionHandler:277] - Operation exception encountered,
> reopening connection
> 2016-09-13T08:37:30.402+01:00 - WARN [org.ldaptive.AbstractOperatio
> n$ReopenOperationExceptionHandler:277] - Operation exception encountered,
> reopening connection
> 2016-09-13T08:37:30.402+01:00 - WARN [org.ldaptive.AbstractOperatio
> n$ReopenOperationExceptionHandler:277] - Operation exception encountered,
> reopening connection
> --------
>

Were you able to confirm that the reopen was successful?



>
> Following these initial messages we only saw two authentication attempts
> complete:
> --------
> 2016-09-13T08:37:48.076+01:00 - INFO [net.shibboleth.idp.authn.impl
> .ValidateUsernamePasswordAgainstLDAP:152] - Profile Action
> ValidateUsernamePasswordAgainstLDAP: Login by 'user1' failed
> 2016-09-13T08:39:05.566+01:00 - INFO [net.shibboleth.idp.authn.impl
> .ValidateUsernamePasswordAgainstLDAP:152] - Profile Action
> ValidateUsernamePasswordAgainstLDAP: Login by 'user2' failed
> --------
>

These were probably the only two that the pool returned before the pool was
emptied by the validator.
And they likely were not in a working state either.


>
> Any ideas why the LDAP connection pool got locked up? It seems to have been
> detecting broken connections to a certain extent, as we see connections
> being
> reopened in the logs (although there are only 20 log entries to this
> effect,
> spread between 08:37:30 and 08:52:37). I would have thought it should be
> able
> to recover from a blip in connectivity to its LDAP authentication source -
> the
> blip would have been less than a second, but even if it had been as long
> as 10
> seconds that should have been recoverable, surely?
>
>
I'm guessing that the pool couldn't create new connections.
Do you have any ldaptive logs that show what was happening to the pool?
I'd be interested in the thread dump if you can share it off list.


> Are there any config options that we should be looking at tweaking, such as
> timeouts, retry counts, validation, etc. that could improve the situation
> if
> this were to happen again?
>

Setting connectTimeout and responseTimeout on the connectionConfig may
allow the connection pool to recover quicker.

--Daniel Fisher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20160914/c4e06cc4/attachment.html>