IdP 2.2.1 (with ECP) and long transactions

[Derek Yuen]

Thanks for your thoughts, Scott.

We have two modern Linux servers -- 2 cores, 4GB memory. We can easily add more if necessary.
Between the two IdPs, we're handling between 1-1.5 million logins per day; we expect to handle more come September.

At the time of the problems, we did not see any contention for CPU or memory resources. 
>From what we see (from sar data), cpu utilization was nearly zero and memory usage was unremarkable.

The only evidence we have are generic broken pipe entries in catalina.log.
While we have "comprehensive" log data from the IdP, we haven't been able to figure out how to effectively use it.

Investigating the back-end pieces was a second path we were about to look at -- we were hoping to buy some time in stabilizing the system first by increasing the child capacity.
At this point, could you kindly advise us as to how we should approach investigating the backend pieces from the IdP's perspective?
The strange thing is that we don't see any performance problems on the back ends (kerberos, ldap, mysql) … but I'll dig a bit deeper just in case we've missed something.



Much appreciated,
Derek





On 2012-07-12, at 7:16 PM, Cantor, Scott wrote:

> On 7/12/12 5:48 PM, "Derek Yuen" <derek.yuen at mail.utoronto.ca> wrote:
>> 
>> Around the time of the problems,
>> we also see in the catalina logs:
> 
> I'm pretty sure that's just the outcome of having lots of traffic stacking
> up waiting for responses. You can increase your child capacity, but that
> won't do anything to fix the root cause.
> 
>> So I've raised the maxThreads in server.xml for Tomcat (from 200 to 500)
>> and slightly increased the MaxChildren (from 255 to 350) in httpd.
>> This has mostly eliminated the java.net.SocketException: Broken pipe
>> errors but did not fix long transaction times.
> 
> Right. The pipe errors aren't errors, they're just telling you the clients
> gave up.
> 
>> The apache server-status page indicates during the problem periods that
>> most, if not all of the apache processes stuck in "W" or "Sending Reply"
>> state.
> 
> Apache doesn't *do* anything in the IdP, it's just relaying traffic. It
> can't be doing anything but waiting.
> 
>> We thought it might be the attribute stores but there's no evidence from
>> the idp-process log files of any ldap or MySQL latency.
>> (We're fortunate to have debug logging on our production IdPs!)
> 
> Well, I'd suggest you stop. That's a ton of overhead in my experience.
> 
>> Recently, a major service has changed to use Shibboleth for
>> authentication and have begun forcing students to migrate to our Live at edu
>> service.
>> I presume that the increase in authentication request has pushed the IdPs
>> beyond what they're currently configured for.
> 
> What kind of load are you handling and with what hardware? If the machines
> are Linux, and not ancient, then you'd have to be handling well in excess
> of 5 figures to even make a dent, and you'd be able to tell easily, your
> CPU would spike. Otherwise, it's I/O somewhere.
> 
>> 
>> My guess is that there might be a timeout parameter somewhere in the
>> chain (perhaps in Tomcat?) that isn't set right.
> 
> That can't fix anything, it would just cause errors to be returned.
> Perhaps better, but not from the perspective of a user.
> 
>> Or maybe i'm completely wrong?
> 
> I think you're overlooking logging and the back-end pieces.
> 
> -- Scott
> 
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>