Guidance on troubleshooting org.ldaptive.pool.BlockingConnectionPool issue

[Mike Osterman]

On Wed, May 15, 2019 at 10:59 AM Daniel Fisher <dfisher at vt.edu> wrote:

> On Tue, May 14, 2019 at 7:56 PM Mike Osterman <ostermmg at whitman.edu>
> wrote:
>
>> Hello,
>>
>> We've had an issue where our IdP (v3.3.1 - I know, we need to update)
>> will intermittently become unresponsive, and that event always coincides
>> with these messages in idp-warn.log:
>>
>> WARN [org.ldaptive.pool.BlockingConnectionPool:882] -
>> org.ldaptive.pool.AbstractConnectionPool$DefaultPooledConnectionProxy at 5b65c083
>> failed validation
>>
>
> Do you notice anything else about the host when this happens? High disk
> I/O, pegged CPU? Has the JVM ever produced a core dump file?
>

Thanks, Daniel. You nailed it. CPU pegs right before and stays pegged until
we kill the process:
(Link to monitoring system graphic:
https://www.evernote.com/l/AAHThhdHvYdPXbyj1u2Q56-ZbWlmNgzltu0)

The only log event prior to the BlockingConnectionPool error is 50 minutes
prior:
*2019-05-14 12:03:54,300 - INFO
[org.opensaml.saml.metadata.resolver.impl.AbstractReloadingMetadataResolver:306]
- Metadata Resolver FileBackedHTTPMetadataResolver ICMD: Next refresh cycle
for metadata provider
'http://md.incommon.org/InCommon/InCommon-metadata.xml
<http://md.incommon.org/InCommon/InCommon-metadata.xml>' will occur on...*

I don't believe we have any JVM core dumps. I'm assuming we'd need to
generate it when in the midst of the pegged CPU condition? If so, is that
the "kill -3" technique or something else?

Thank you!
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20190515/4b78c96c/attachment.html>