Problem with tomcat hanging on Shib 2.4

David Mansfield shibboleth at dm.cobite.com
Fri Aug 29 10:47:53 EDT 2014


On 08/29/2014 10:40 AM, Cantor, Scott wrote:
> On 8/29/14, 10:29 AM, "Matthew Slowe" <m.slowe at kent.ac.uk> wrote:
>> First I'm going to refer to a thread on the JISC-SHIBBOLETH mailing list
>> last year on the subject (no signin required)
>>
>> https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind1310&L=jisc-shibboleth&F
>> =&S=&P=60
>>
>> We have three IDPs running in a very similar setup to yourselves (3 RHEL
>> VMs (each are 2cpu, 4G) on VMware) running, then, 1.7.0_25 (now _55)
>> each servicing up to 330,000 authentications per day.
> And that's Oracle's JVM. So that points the finger at Tomcat. Jetty is a
> much better choice, so that would be my next suggestion. I actually have
> never run 2.x under Tomcat in fact, only 1.3. I ran for a couple of years
> on Jetty 7.5 and now I'm on 9.1 (for about a month, so far no issues and
> classes started this week).
>
> I can't directly compare, as I'm on Red Hat 5, not 6. But we have had no
> issues under comparable loads to that, slightly lower.
>
> I'm also not on VMs, and that's another red flag for me. I don't know if
> the OP is.
I have dozens of centos6 machines running in VM and I've never had a 
single issue that was caused by the VM so far. Not to rule it out, but 
with java being java, it's more likely something higher in the software 
stack.
>
> The other thing I'd point to is whether authentication might be involved.
> The IdP can appear to hang if that's blocking.

A 'kill -QUIT'  will dump a thread trace to the log (usually) and then 
you can determine if there are hanging threads, or a deadlock on a 
resource that may be limited and obtained recursively or in a incorrect 
order etc.  For me, with tomcat, it has usually been the case that a 
combination of the logger locks and the database connection pooling 
causes these kind of deadlocks (when one thread has the potential of 
obtaining more than one db connection for example).

David


>
> -- Scott
>



More information about the users mailing list