IDP clustering with Terracotta 3.6

[Mark Cairney]

Hi Russell,

That's really interesting- it's been a while since I looked at this page and there's been quite a lot of new information added to it since.

Is there anything else to bear in mind if you change to using the ConcurrentDistributedMap instead of the ConcurrentHashMap? In particular is there a trade-off in terms of resiliency?

On 22 Dec 2011, at 22:58, Russell Beall wrote:

> Sorry for the delay on this.
> 
> I've written the instructions for improving the EventingMapBasedStorageService to use the ConcurrentDistributedMap object.  It got to be long enough that I pulled it out of this e-mail that I was writing and added it to the end of the IdPClusterIssues page:
> 
> <https://wiki.shibboleth.net/confluence/display/SHIB2/IdPClusterIssues#IdPClusterIssues-ConfiguringtheTerracottaConcurrentDistributedMap>
> 
> For those still using TC, the performance improvement is definitely worth considering.
> 
> Regards,
> Russ.
> 
> On Dec 15, 2011, at 11:59 AM, Christopher Bongaarts wrote:
> 
>> On 12/15/2011 1:18 PM, Russell Beall wrote:
>> 
>>> I alleviated this and other issues by converting the
>>> EventingMapBasedStorageService to use the Terracotta recommended
>>> ConcurrentDistributedHashMap instead of the java built-in
>>> ConcurrentHashMap.  It is a simple tweak to the IdP code actually and
>> [...]
>>> I can publish the patch for this if desired.
>> 
>> Yes please (or send it to me privately), assuming it is more involved 
>> than simply changing the inheritance of the EMBSS class.
>> 
>>> I currently have been staging and load testing a RedHat Linux cluster
>>> of 2.3.3 IdP nodes clustered with 3.5.1 TC.
>> 
>> We have been doing load testing with RedHat Linux with 2.3.5 against TC 
>> 3.5.3 (current production is IdP 2.1.3, TC 3.5.2).
>> 
>>> I encountered some problems with the settings currently published and
>>> had to tweak them to get it to stabilize.  In particular the setting
>>> which specifies the pause interval for the garbage collector became a
>>> real problem and I had to get rid of it: -XX:MaxGCPauseMillis=5000
>>> 
>>> On Linux, this setting started being used with a vengeance (whereas
>>> it basically was ignored in Solaris) and my TC server would enter
>>> continuous Full GC cycles even though the heap was not full, trying
>>> to make sure that it always did its GC at 5000 ms or less.  This
>>> resulted in severe unworkability because really it ended up keeping
>>> itself in one continuous stream of GC.
>> 
>> We saw something like this when I added that option to the Terracotta 
>> JVM on Solaris in production.  After restarting Tomcat, the load on the 
>> machine would spike up to around 5-6 (compared to a normal load of 1-2). 
>> But the system was still responding acceptably, so I let it run. 
>> After a day or two, the load had settled back down to 1-2.  My current 
>> conclusion is that it takes a while for the JVM to figure out the sweet 
>> spot, and if it gets behind it stays behind until the load eases up 
>> enough to catch up.  Also, it's an older rev of the JVM (Sun 1.6.0_13) 
>> so it might be already-fixed bugs.
>> 
>> What are your current JVM options set to in the Linux world?  Our 
>> Solaris Terracotta servers are set to:
>> 
>> -Xmx2g
>> -XX:+UseParallelOldGC
>> -XX:MaxGCPauseMillis=5000
>> -verbose:gc
>> -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps
>> -XX:-TraceClassUnloading
>> (plus 64-bit options added by init script)
>> 
>> and Tomcat:
>> 
>> -d64
>> -server
>> -XX:+DisableExplicitGC
>> -XX:+UseParallelOldGC
>> -Xmx2g
>> -XX:MaxPermSize=256m
>> -XX:+HeapDumpOnOutOfMemoryError
>> (plus terracotta boot jar/JMX options)
>> 
>> -- 
>> %%  Christopher A. Bongaarts   %%  cab at umn.edu          %%
>> %%  OIT - Identity Management  %%  http://umn.edu/~cab  %%
>> %%  University of Minnesota    %%  +1 (612) 625-1809    %%
>> --
>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
> 
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
> 

/********************************* 
Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: mark.cairney at ed.ac.uk

*********************************/


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.