Sjibb LDAP pool hangs on firewall TCP session truncation

[Rich Graves]

I recently switched authentication source to Active Directory/LDAP servers on the other side of a Palo Alto firewall. Jetty 9 proxied through Apache on CentOS Linux 7, OpenJDK 1.8.0.91 (java-1.8.0-openjdk-headless-1.8.0.91-0.b14.el7_2.x86_64).

Until we figured out how to fix the firewall damage (by customizing the "ldap" application -- it turns out that the configured TCP idle timeouts are not used for recognized L7 apps) I was seeing the following unhappy behavior:

1) Firewall closes the persistent pooled connections because they are "idle."
2) Shibb does not recognize that this has happened.
3) Shibb authentication breaks completely. It appears that a jetty service restart is necessary to restore service.

I can reproduce this on demand with, on the shibb server, iptables -I OUTPUT -d $ldap -j DROP.

Things I tried that did not fix the problem:

1) Enabled keepalives with net.ipv4.tcp_keepalive_time = 300.
2) Set idp.pool.LDAP.validateOnCheckout = true.
3) Toggled between ldaps and ldap+starttls (in hope that one end or the other might use a happier code path).
4) Dropped idp.authn.LDAP.returnAttributes (one of the goals of this exercise is to enable expiring password flow with pwdLastSet).
5) Increase and decrease idp.authn.LDAP.connectTimeout (it wasn't the connect, it was the continuation of an existing session).

Is there anything else I could do to make shibb using adAuthenticator (needed for password expiration flows and AD-specific error messages, I think) more resilient in the face of stalled persistent LDAP connections? Could I have disabled connection pooling entirely with idp.pool.LDAP.minSize=0, idp.pool.LDAP.maxSize=0?

Where it seems to get stuck is:

2016-06-13 00:24:58,818 - ERROR [org.springframework.webflow.conversation.impl.LockTimeoutException:76] -
org.springframework.webflow.conversation.impl.LockTimeoutException: Unable to acquire conversation lock after 30 seconds
        at org.springframework.webflow.conversation.impl.JdkConcurrentConversationLock.lock(JdkConcurrentConversationLock.java:44)

What more information might be helpful?