IDP clustering with Terracotta 3.6

[Christopher Bongaarts]

On 12/15/2011 1:18 PM, Russell Beall wrote:

> I alleviated this and other issues by converting the
> EventingMapBasedStorageService to use the Terracotta recommended
> ConcurrentDistributedHashMap instead of the java built-in
> ConcurrentHashMap.  It is a simple tweak to the IdP code actually and
[...]
> I can publish the patch for this if desired.

Yes please (or send it to me privately), assuming it is more involved 
than simply changing the inheritance of the EMBSS class.

> I currently have been staging and load testing a RedHat Linux cluster
> of 2.3.3 IdP nodes clustered with 3.5.1 TC.

We have been doing load testing with RedHat Linux with 2.3.5 against TC 
3.5.3 (current production is IdP 2.1.3, TC 3.5.2).

> I encountered some problems with the settings currently published and
> had to tweak them to get it to stabilize.  In particular the setting
> which specifies the pause interval for the garbage collector became a
> real problem and I had to get rid of it: -XX:MaxGCPauseMillis=5000
>
> On Linux, this setting started being used with a vengeance (whereas
> it basically was ignored in Solaris) and my TC server would enter
> continuous Full GC cycles even though the heap was not full, trying
> to make sure that it always did its GC at 5000 ms or less.  This
> resulted in severe unworkability because really it ended up keeping
> itself in one continuous stream of GC.

We saw something like this when I added that option to the Terracotta 
JVM on Solaris in production.  After restarting Tomcat, the load on the 
machine would spike up to around 5-6 (compared to a normal load of 1-2). 
  But the system was still responding acceptably, so I let it run. 
After a day or two, the load had settled back down to 1-2.  My current 
conclusion is that it takes a while for the JVM to figure out the sweet 
spot, and if it gets behind it stays behind until the load eases up 
enough to catch up.  Also, it's an older rev of the JVM (Sun 1.6.0_13) 
so it might be already-fixed bugs.

What are your current JVM options set to in the Linux world?  Our 
Solaris Terracotta servers are set to:

-Xmx2g
-XX:+UseParallelOldGC
-XX:MaxGCPauseMillis=5000
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:-TraceClassUnloading
(plus 64-bit options added by init script)

and Tomcat:

-d64
-server
-XX:+DisableExplicitGC
-XX:+UseParallelOldGC
-Xmx2g
-XX:MaxPermSize=256m
-XX:+HeapDumpOnOutOfMemoryError
(plus terracotta boot jar/JMX options)

-- 
%%  Christopher A. Bongaarts   %%  cab at umn.edu          %%
%%  OIT - Identity Management  %%  http://umn.edu/~cab  %%
%%  University of Minnesota    %%  +1 (612) 625-1809    %%