IDP clustering with Terracotta 3.6

[Russell Beall]

Resiliency is determined by the backing store. The data in each node is still sent to the TC server and can be recovered upon restart or upon a user switching to a new IdP node. 

With the distributed map there is simply less traffic among IdP nodes that don't need a copy of the data -- (all passive TC servers will still get a copy).

Additionally, the garbage collection is much smoother, in particular this is probably with the lock objects that are handled better. 

Russ.
(iPhone)


On Dec 23, 2011, at 2:50 AM, Mark Cairney <mark.cairney at ed.ac.uk> wrote:

> Hi Russell,
> 
> That's really interesting- it's been a while since I looked at this page and there's been quite a lot of new information added to it since.
> 
> Is there anything else to bear in mind if you change to using the ConcurrentDistributedMap instead of the ConcurrentHashMap? In particular is there a trade-off in terms of resiliency?
> 
> On 22 Dec 2011, at 22:58, Russell Beall wrote:
> 
>> Sorry for the delay on this.
>> 
>> I've written the instructions for improving the EventingMapBasedStorageService to use the ConcurrentDistributedMap object.  It got to be long enough that I pulled it out of this e-mail that I was writing and added it to the end of the IdPClusterIssues page:
>> 
>> <https://wiki.shibboleth.net/confluence/display/SHIB2/IdPClusterIssues#IdPClusterIssues-ConfiguringtheTerracottaConcurrentDistributedMap>
>> 
>> For those still using TC, the performance improvement is definitely worth considering.
>> 
>> Regards,
>> Russ.
>> 
>> On Dec 15, 2011, at 11:59 AM, Christopher Bongaarts wrote:
>> 
>>> On 12/15/2011 1:18 PM, Russell Beall wrote:
>>> 
>>>> I alleviated this and other issues by converting the
>>>> EventingMapBasedStorageService to use the Terracotta recommended
>>>> ConcurrentDistributedHashMap instead of the java built-in
>>>> ConcurrentHashMap.  It is a simple tweak to the IdP code actually and
>>> [...]
>>>> I can publish the patch for this if desired.
>>> 
>>> Yes please (or send it to me privately), assuming it is more involved 
>>> than simply changing the inheritance of the EMBSS class.
>>> 
>>>> I currently have been staging and load testing a RedHat Linux cluster
>>>> of 2.3.3 IdP nodes clustered with 3.5.1 TC.
>>> 
>>> We have been doing load testing with RedHat Linux with 2.3.5 against TC 
>>> 3.5.3 (current production is IdP 2.1.3, TC 3.5.2).
>>> 
>>>> I encountered some problems with the settings currently published and
>>>> had to tweak them to get it to stabilize.  In particular the setting
>>>> which specifies the pause interval for the garbage collector became a
>>>> real problem and I had to get rid of it: -XX:MaxGCPauseMillis=5000
>>>> 
>>>> On Linux, this setting started being used with a vengeance (whereas
>>>> it basically was ignored in Solaris) and my TC server would enter
>>>> continuous Full GC cycles even though the heap was not full, trying
>>>> to make sure that it always did its GC at 5000 ms or less.  This
>>>> resulted in severe unworkability because really it ended up keeping
>>>> itself in one continuous stream of GC.
>>> 
>>> We saw something like this when I added that option to the Terracotta 
>>> JVM on Solaris in production.  After restarting Tomcat, the load on the 
>>> machine would spike up to around 5-6 (compared to a normal load of 1-2). 
>>> But the system was still responding acceptably, so I let it run. 
>>> After a day or two, the load had settled back down to 1-2.  My current 
>>> conclusion is that it takes a while for the JVM to figure out the sweet 
>>> spot, and if it gets behind it stays behind until the load eases up 
>>> enough to catch up.  Also, it's an older rev of the JVM (Sun 1.6.0_13) 
>>> so it might be already-fixed bugs.
>>> 
>>> What are your current JVM options set to in the Linux world?  Our 
>>> Solaris Terracotta servers are set to:
>>> 
>>> -Xmx2g
>>> -XX:+UseParallelOldGC
>>> -XX:MaxGCPauseMillis=5000
>>> -verbose:gc
>>> -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps
>>> -XX:-TraceClassUnloading
>>> (plus 64-bit options added by init script)
>>> 
>>> and Tomcat:
>>> 
>>> -d64
>>> -server
>>> -XX:+DisableExplicitGC
>>> -XX:+UseParallelOldGC
>>> -Xmx2g
>>> -XX:MaxPermSize=256m
>>> -XX:+HeapDumpOnOutOfMemoryError
>>> (plus terracotta boot jar/JMX options)
>>> 
>>> -- 
>>> %%  Christopher A. Bongaarts   %%  cab at umn.edu          %%
>>> %%  OIT - Identity Management  %%  http://umn.edu/~cab  %%
>>> %%  University of Minnesota    %%  +1 (612) 625-1809    %%
>>> --
>>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>> 
>> --
>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>> 
> 
> /********************************* 
> Mark Cairney
> ITI UNIX Section
> Information Services
> University of Edinburgh
> 
> Tel: 0131 650 6565
> Email: mark.cairney at ed.ac.uk
> 
> *********************************/
> 
> 
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> 
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net