Problem with tomcat hanging on Shib 2.4

[Brian Koehmstedt]

On 8/29/2014 12:13 PM, Cantor, Scott wrote:
> On 8/29/14, 2:58 PM, "Brian Koehmstedt" <bkoehmstedt at ucmerced.edu> wrote:
>
>> I am almost positive that PermGen does NOT come out out of heap.
>> Actually, I can confirm this as I am looking at JVM stats now that are
>> showing me perm is indeed separate.
> I certainly thought so. Anyway, nutshell, that means 768M is enough to
> handle 200,000+ logins (that's not typical but we've hit it at times) with
> some caching, loading both InCommon's metadata and a file of 250+ entities
> locally, plus a couple more non-trivial sized files.
>
> So if you're getting heap errors with 1G, something else has been added to
> the JVM or Tomcat is a disaster. I am readily able to believe the latter.

I'm willing to believe there's a RedHat/Tomcat issue here, and I'm 
hoping to eventually track it down with GC stats and heap dumps.

I will say there is a small amount of "custom" Java code called as 
Script from an attribute-resolver to generate targeted IDs on a 
per-service basis.  I immediately jumped to this as a possibility, and 
I've code-reviewed it and see no evidence of caching.  i.e., I was 
thinking maybe I was caching the targetedIds in a hash table, but as I 
review this code, I do not indeed do any caching of these IDs.  
(targetedId is essentially a salted hash of primary identifier and 
service id).

But, usually the author code reviewing his own code is a bad idea, so 
John, you're welcome to look at it too.  :)

At any rate, I am going to add these JVM options to log GC and get heap 
dumps when the OutOfMemoryExceptions occur:

JAVA_OPTS="${JAVA_OPTS} -Xmx1024M -XX:MaxPermSize=512M -server -Djava.library.path=/usr/lib64 -Djavax.net.ssl.trustStore=/jdk/cacerts -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${LOG_DIR}/java_pid<pid>.hprof" -XX:ErrorFile="${LOG_DIR}/hs_err_pid<pid>.log" -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:${LOG_DIR}/java_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=7 -XX:GCLogFileSize=1M -XX:+PrintCommandLineFlags -verbose:gc"