Rare ELException

[Cameron Kerr]

I would wonder why you use the Prefork MPM with Apache 2.4 instead of say the Event MPM, which is much more scalable. I would also suggest that enabling Keepalives (with a reasonably short lifetime) would also help with scalability and cut down on port utilisation.

Cheers,
Cameron

From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Brian Moon
Sent: Friday, 19 May 2017 11:54 AM
To: Shib Users <users at shibboleth.net>
Subject: Re: Rare ELException

Hello all,

While stress testing one of our IdP servers today we were able to reproduce this error.  Below is some information about our environment and the testing parameters:

IdP Server:

  *   SLES 12 SP2
  *   IdP Version 3.3.1
  *   Oracle JDK 1.8.0_112-b15
  *   Tomcat 8.0.36-20.7
  *   JAVA_OPTS: -Xms3g -Xmx3g -XX:MaxGCPauseMillis=400 -XX:+DisableExplicitGC -XX:+UseG1GC
  *   2 Cores
  *   6 GB RAM (VM memory reservation set to reserve all guest memory)
  *   Apache 2.4.23-21.1

     *   Apache Prefork MPM
     *   ServerLimit 5120
     *   MaxClients 5120
Testing VM:

  *   Windows 7 Enterprise (64-bit)
  *   4 Cores
  *   8 GB RAM
  *   Oracle JRE 1.8.0_131-b11
  *   Apache JMeter 3.2 r1790748

     *   Test plan: https://wiki.shibboleth.net/confluence/download/attachments/17072886/shibboleth-unsolicited-sso-test.jmx?version=2&modificationDate=1435784363124&api=v2
     *   ThreadCount 1500
     *   Duration 300
     *   RampUpPeriod 10
     *   ProviderId https://sp.testshib.org/shibboleth-sp
For the Apache config on the IdP server, the ServerLimit and MaxClients were set to something ridiculous so we could see what kind of load it could take.  On the testing VM we were running out of ephemeral ports, so we added a registry key to increase the max port number that could be used by a user: https://www.baselogic.com/blog/solved-java-net-bindexception-address-use-connect-issue-windows/ (this is not needed to reproduce the error, but cuts down on some noise).  We did still have some socket errors on the testing VM, but overall it seemed to run a lot better.

Hope that helps in reproducing this and tracking down the cause.


Brian Moon
Senior System Administrator
Enterprise Systems
Santa Clara University
Office: 408.554.4830
bmoon at scu.edu<mailto:bmoon at scu.edu>

On Mon, Apr 24, 2017 at 9:26 AM, Cantor, Scott <cantor.2 at osu.edu<mailto:cantor.2 at osu.edu>> wrote:
> Yes, I'm currently waiting for my session to timeout on the login form so I can
> see what it does. I would expect it to throw a flow execution error though, so
> I don't know if that's going to do it. I also see *very* few of these in my log,
> like once a day or so, and that's with a decent volume, so that's puzzling.

A basic "Wait too long" timeout doesn't have anything to do with this error, so that's as expected. So I have no idea how one would reproduce it.

-- Scott


--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net<mailto:users-unsubscribe at shibboleth.net>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20170519/879a1322/attachment.html>