IdP 2.2.1 (with ECP) and long transactions

[Cantor, Scott]

On 7/12/12 5:48 PM, "Derek Yuen" <derek.yuen at mail.utoronto.ca> wrote:
>
>Around the time of the problems,
>we also see in the catalina logs:

I'm pretty sure that's just the outcome of having lots of traffic stacking
up waiting for responses. You can increase your child capacity, but that
won't do anything to fix the root cause.

>So I've raised the maxThreads in server.xml for Tomcat (from 200 to 500)
>and slightly increased the MaxChildren (from 255 to 350) in httpd.
>This has mostly eliminated the java.net.SocketException: Broken pipe
>errors but did not fix long transaction times.

Right. The pipe errors aren't errors, they're just telling you the clients
gave up.

>The apache server-status page indicates during the problem periods that
>most, if not all of the apache processes stuck in "W" or "Sending Reply"
>state.

Apache doesn't *do* anything in the IdP, it's just relaying traffic. It
can't be doing anything but waiting.

>We thought it might be the attribute stores but there's no evidence from
>the idp-process log files of any ldap or MySQL latency.
>  (We're fortunate to have debug logging on our production IdPs!)

Well, I'd suggest you stop. That's a ton of overhead in my experience.

>Recently, a major service has changed to use Shibboleth for
>authentication and have begun forcing students to migrate to our Live at edu
>service.
>I presume that the increase in authentication request has pushed the IdPs
>beyond what they're currently configured for.

What kind of load are you handling and with what hardware? If the machines
are Linux, and not ancient, then you'd have to be handling well in excess
of 5 figures to even make a dent, and you'd be able to tell easily, your
CPU would spike. Otherwise, it's I/O somewhere.

> 
>My guess is that there might be a timeout parameter somewhere in the
>chain (perhaps in Tomcat?) that isn't set right.

That can't fix anything, it would just cause errors to be returned.
Perhaps better, but not from the perspective of a user.

>Or maybe i'm completely wrong?

I think you're overlooking logging and the back-end pieces.

-- Scott