Sudden error condition this morning

[Cantor, Scott]

On 9/22/15, 1:20 PM, "users on behalf of Jeffrey Crawford" <users-bounces at shibboleth.net on behalf of jeffreyc at ucsc.edu> wrote:

>As things ramp up for the fall quarter, we started getting more traffic on our shibboleth servers. However this morning something broke loose and I'm trying to figure out what it is. These
> messages started up around 7:35

Well, mostly that says is the IdP was nonresponsive to the point that requests aren't getting handled.

>2015-09-22 07:35:04,898 - WARN [org.ldaptive.AbstractOperation$ReopenOperationExceptionHandler:277] - [xxx.xxx.xx.xx] - Operation exception encountered, reopening connection

But that seems pretty clearly to be an LDAP problem, probably leading to the rest later.

>I saw from an earlier thread that this means that the db (ldap in this case) transaction is taking a long time to being hung. However this just started this morning and we are not using a DB, just LDAP We didn't seem to have any ldap issues so I'm wondering what could cause something like this.

I see an LDAP issue there, unless that doesn't mean what it seems to.

The difference from V2 is just that there's a specific request processing step from web flow that happens to time out when these kinds of problems hit, vs. the container eventually refusing requests. I don't know if that timeout is tunable.

-- Scott