[Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?

[Russell Beall]

The settings I mention refer to the Tomcat JVM.  Terracotta has not required any PermGen tuning.

I'm not sure what the default PermGen size is, but Chad probably suggested removing it because the default is workable now.  I saw 128M mentioned somewhere as the latest recommendation (when I looked a few months ago), so that is what I am planning to change it to.  In production, mine is still set to 1G on the 2.1.5 release where the metadata loading hadn't yet been streamlined.

Seems like you have just a system gremlin or just a simple idiosyncrasy.  Certainly if the monitoring reports missing memory when both JVMs are dead, then the monitoring is off or hitting some kind of idiosyncrasy of system resource allocation or caching.

The histogram numbers you reported indicated that the actual Tomcat JVM had not filled up even remotely, so even though the system reported missing memory, it wasn't actually a problem.  It is likely you do not have a problem with the actual JVMs.

Regards,
Russ.

On Aug 4, 2011, at 2:35 PM, Wessel, Keith William wrote:

> Hi, Russ,
> 
> Some good feedback here.
> 
> Yes, it's a VM with 2 GB of RAM. Interestingly enough, until early June, we were running with only 1 GB, and with Tomcat and Terracotta both running, we never got about 800M... or so my system manager reports. Production has 4 GB, of course, as I know that's needed.
> 
> Are you referring to the heap and maxpermsize params for the Tomcat JVM instance? Chad suggested a while back that I remove the maxpermsize 512M that I had before, but I neve replaced it with 128M. At his suggestion, I also removed Xms and left only Xmx set to 1G. For Terracotta, I'm using the defaults in start-tc-server.sh other than the Xms that I removed earlier today. That includes Xmx 512M. I didn't see a Maxpermsize parameter in there. Where are you suggesting I set the 128M perm space? Which JVM instance?
> 
> We have used, top, yes, but our system monitoring tools use free or data from /proc. I know top can't be trusted here for various reasons including the ones you outlined.
> 
> And no, we've never seen an out of memory error. In fact, we've never even hit swap. But according to our system manager, the graphs shot up from 800 M to this close to 2 G after upgrading to 2.3.0, and downgrading back to 2.1.5 didn't change it back.
> 
> I'm starting to wonder if the system monitoring was messed up before, and this whole thing has been a two-month wild goose chase. But I'll pursue those java opts you suggest before going any further with that rather out there theory.
> 
> Keith
> 
> 
> -----Original Message-----
> From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Russell Beall
> Sent: Thursday, August 04, 2011 3:46 PM
> To: Shib Users
> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
> 
> Hi Keith,
> 
> It looks like you are setting this up on a VM which has a max of 2G.  That is not enough room to run this stuff.  Terracotta ideally will have 2G all to itself when running under load.  You can get by with smaller when testing, but as the number of logins increases, the memory demands on the TC server will increase.
> 
> Additionally, when specifying the maximum memory of a server, I found that this setting only applies to the heap, and then there is additional room for the PermGen for instance which has a separate setting.  You should reduce that to its recommended size of about 128M.
> 
> What tool are you using to determine that there is memory missing when neither JVM is running?
> 
> 'top' is not a good tool for monitoring this.  I regularly have large amounts of memory reported as missing by 'top', but it resurfaces over time or if the application-level memory demands require it.
> 
> In regards to the jmap -histo numbers, the numbers you see are expected.  And I was referring to the Tomcat JVM for this.  It doesn't look like Tomcat is consuming the memory.  jmap -heap can show you the detailed heap usage for both JVM instances.
> 
> Are you ever having an actual "OutOfMemoryError" ?
> 
> Russ.
> 
> On Aug 4, 2011, at 1:05 PM, Wessel, Keith William wrote:
> 
>> Thanks, all, for the further input here.
>> 
>> We had only two scriptlets in our attribute resolver; I had forgotten that everything else that I thought used scripts was just templates. I removed the two script-based attributes and there was no difference. Nearly all of our 2 GB of RAM was used.
>> 
>> As a wild guess, I removed the -Xms 512m from start-tc-server.sh, leaving -Xmx 512m. Interestingly, this seems to have helped a bit. Now, after start-up and with only the SLB health checks, we've settled at about 1.5 G. I'm still thinking that's a bit on the high side, though, considering we've got no load here.
>> 
>> Not sure, Russ, if I'm supposed to be looking at the heap for the Tomcat instance of Java or Terracotta with jmap -histo. I assume the Terracotta instance. It claims 2077 objects in the heap. Total number of instnaces is 41301, total bytes is 53590984.
>> 
>> If you were looking for the Tomcat Java instance, it has 4238 objects, total instances are 4050098, total bytes are 277186624.
>> 
>> Does that look sane?
>> 
>> If so, my next step, since I can't explain any differences between production and test from when this started, will be to have our system managers re-install this VM from scratch. I can then restore my Shib, Tomcat, Java and Terracotta installs and see if things look happier. Something's definitely not right as, as I mentioned long ago in this thread, we still have close to 1 GB of memory unaccounted for even after shutting down Tomcat and Terracotta... 1 GB that's available at boot time.
>> 
>> Thanks for pointing me to the right location in the Rhino jar file. I thought that was what I needed to do. I'll keep that trick in my back pocket for future performance improvements, but for now, I don't think that's going to solve this situation.
>> 
>> Keith
>> 
>> 
>> -----Original Message-----
>> From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Russell Beall
>> Sent: Thursday, August 04, 2011 11:55 AM
>> To: Shib Users
>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>> 
>> The instructions for the rhino engine are on this page:
>> http://www.mozilla.org/rhino/footprint.html
>> 
>> It simply says to remove the classfiles in the optimizer package.
>> 
>> Basically this means unjar the rhino jar, delete the optimizer directory, and re-jar the remaining contents.  There is code in the library which dynamically decides whether this package is available, and if so, uses it, and if not, reverts to interpreted mode.  I think interpreted mode is more efficient in this case anyway, because the use of the scriplets causes a new construction of objects on each use rather than a re-use of once-compiled definitions -- which defeats the purpose of doing the optimization.
>> 
>> I seriously doubt this is your problem however with just the SLB check pinging the system.
>> 
>> What does "jmap -histo" say about the number and size of objects in the heap?
>> 
>> Regards,
>> Russ.
>> 
>> On Aug 4, 2011, at 8:22 AM, Wessel, Keith William wrote:
>> 
>>> Russ and others,
>>> 
>>> Thanks for the continued suggestions. We sadly can't take the stateless clustering option at this time, and we'd prefer to stick with the Shibboleth project recommended route of Terracotta rather than using Memcache.
>>> 
>>> I tried upgrading to Java 1.6.0 build 26 and the latest Terracotta 3.5.2 yesterday to see if that would help. Sadly, no. Memory went through the rough within minutes of start-up, consuming pretty much all the physical RAM with nothing more than once-a-minute hits of the healthcheck URL from our SLB. No other load.
>>> 
>>> I seriously doubt removing the Rhino optimizer will help, but I'm definitely willing to give it a shot. Russ, as you said in a previous list post that I found in the list archives, Rhino's documentation on exactly what to remove and what to update in the JVM environment is sketchy to say the least. Can you point me to more specifics on this? Consulting Dr. Google didn't seem to turn up much beyond a basic page on decreasing the Rhino footprint (that lacked specifics on how to remove the optimizer) and your previous list posts.
>>> 
>>> Thanks,
>>> Keith
>>> 
>>> 
>>> -----Original Message-----
>>> From: Russell Beall [mailto:beall at usc.edu] 
>>> Sent: Thursday, July 28, 2011 4:20 PM
>>> To: users at shibboleth.net
>>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>> 
>>> With Terracotta enabled, the Rhino javascript engine has an optimization which creates new classnames every time it is used.  If you are using a lot of scriptlets, this will eat up memory over a long period of time.
>>> 
>>> Additionally, JVM versions prior to 1.6.0_23 have a memory leak in the JMX system which is part of what Terracotta uses.  This has caused a memory consumption of 2GB in about a month's time in my installation running about 50K logins per day.
>>> 
>>> The solutions are to remove the optimizer code in the Rhino library, (just delete the class files), and update the JVM.  The Rhino library has instructions for what code to remove on their website.
>>> 
>>> These were all the leaks I found, and after this I ran load testing over extended periods of time with small heap sizes and never leaked a thing.
>>> 
>>> Regards,
>>> Russ.
>>> 
>>> On Jul 28, 2011, at 2:12 PM, Wessel, Keith William wrote:
>>> 
>>>> Chad and others,
>>>> 
>>>> This is a start.
>>>> 
>>>> With the Xms and MaxPermSize removed, we're not gobbling up memory...
>>>> 
>>>> Unless we enable Terracotta clustering. Now, instead of an steady clmb in memory usage following IDP start-up, we see an immediate (within five minutes of start-up) consumption of nearly all the system's 2GB of memory. This is with Xmx for the IDP's Tomcat instance set to 1024m.
>>>> 
>>>> If it matters, we're running Tomcat 6.0.20, Terracotta 3.2.1 and Java 6.0.18.
>>>> 
>>>> I did go ahead and upgrade the IDP to 2.3.2 since the version of the IDP we're running seems to have little impact on this problem.
>>>> 
>>>> Before upgrading the Terracotta instance, I thought I'd ask here to see if anyone had seen anything like this before or could point me in the direction of what to look at. The Java opts for the Terracotta JVM are the defaults in the TC start-up script. I haven't touched those.
>>>> 
>>>> I see, on the Wiki page on clustering issues, that it still recommends running Tomcat with both Xms and Xmx of 1G and makes a reference to MaxPermSize also of 1G. But since I just removed these options, I hesitate to add them back.
>>>> 
>>>> And if this question needs to go to the Terracotta forums, just say so.
>>>> 
>>>> Thanks for any further adive anyone can add. I'd love to get rid of Terracotta, but at this point, that's not an option for us if we want to offer artifact resolution on our cluster.
>>>> 
>>>> Thanks,
>>>> Keith
>>>> 
>>>> -----Original Message-----
>>>> From: Wessel, Keith William 
>>>> Sent: Friday, July 22, 2011 1:29 PM
>>>> To: 'users at shibboleth.net'
>>>> Subject: RE: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>> 
>>>> Thanks, Chad, good to know. I honestly hadn't checked the docs on recommended Java params in over a year to see what had changed.
>>>> 
>>>> I've removed Xms and permsize so the server will use the defaults. We're going to reboot and turn the health checks back on, and we'll see what happens.
>>>> 
>>>> Keith
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: Chad La Joie [mailto:lajoie at itumi.biz] 
>>>> Sent: Friday, July 22, 2011 10:19 AM
>>>> To: users at shibboleth.net
>>>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>> 
>>>> Get rid of -Xms setting or at at the very lest, turn it way down.  If I
>>>> recall, the behavior when the min and max are exactly the same is
>>>> undefined.  I think the old behavior was to ignore the max.  You can
>>>> also decrease the perm size now.  The memory leak that was in the IdP in
>>>> the 2.0/2.1 days is gone so the default perm size is just fine.  Pretty
>>>> sure our docs have been updated to reflect that.
>>>> 
>>>> On 7/22/11 10:53 AM, Wessel, Keith William wrote:
>>>>> Thanks, Chad, for the heap analysis, I really appreciate it.
>>>>> 
>>>>> Interestingly, our Xmx is already set to 1024m. Here's our JVM parameters:
>>>>> 
>>>>> JAVA_OPTS="-Xms1024m -Xmx1024m -XX:MaxPermSize=512m \
>>>>> -XX:+UseParallelOldGC \
>>>>> -XX:MaxGCPauseMillis=5000 \
>>>>> -XX:+PrintGCDateStamps \
>>>>> -XX:+MustCallLoadClassInternal
>>>>> -XX:-TraceClassUnloading \
>>>>> -Djava.security.auth.login.config==$IDP_HOME/conf/login.config"
>>>>> 
>>>>> I've taken your suggestion of turning the openSAML logging up to DEBUG. We'll see what that shows us.
>>>>> 
>>>>> Thing is, if it's related to Metadata reloading, I'm puzzled why turning off the SLB healthchecks caused the memory consumption issues to go away.
>>>>> 
>>>>> Fun times. If I find anything interesting, I'll report back.
>>>>> 
>>>>> Thanks again to Chad and everyone else who's chimed in on this.
>>>>> 
>>>>> Keith
>>>>> 
>>>>> -----Original Message-----
>>>>> From: Chad La Joie [mailto:lajoie at itumi.biz] 
>>>>> Sent: Friday, July 22, 2011 6:41 AM
>>>>> To: users at shibboleth.net
>>>>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>>> 
>>>>> Okay, I analyzed the memory dumps you sent me.  There is no memory leak
>>>>> displayed within these dumps.  The actual memory used remains roughly
>>>>> constant: the first dump was using 23.4MB and the second was using 22.9MB.
>>>>> 
>>>>> As I mentioned before, using things like Unix 'top' is *not* a reliable
>>>>> way to check the amount of memory a Java process is actually taking up.
>>>>> 
>>>>> In general, we see Jetty/Tomcat + IdP hover around 25-50MB depending on
>>>>> how much metadata is loaded and how complicated the relying party,
>>>>> resolver, and filter engine configurations are (i.e., the more things
>>>>> declared, the more memory used).  When reading in new metadata the IdP's
>>>>> used memory will spike (DOM is one of the worlds least efficient APIs)
>>>>> and then immediately drop back down once the read-in is complete.
>>>>> 
>>>>> Now, because all JVM garbage collection strategies contain at least one
>>>>> "stop the world" event[1], the JVM will try to minimize the number of
>>>>> times the garbage collector is called.  The easiest way to do this is to
>>>>> not run the GC at all and just keep requesting memory from the OS until
>>>>> the JVM max memory limit is reached and only then do a garbage
>>>>> collection.  Once that point is reached the GC will do one massive
>>>>> cleanup and the process starts again.
>>>>> 
>>>>> So, my theory on whats happening here is that the you probably aren't
>>>>> setting the max memory limit and so the JVM is using a large'ish number
>>>>> (I think 1-2GB on "server class" machines is pretty common for most JVMs
>>>>> now).  The IdP is probably reloading metadata more frequently (turn on
>>>>> debug logging for org.opensaml.saml2.metadata.provider to see) and
>>>>> therefore the JVM is requesting memory from the OS at a quicker clip.
>>>>> Because the OS only has 1GB, now 2GB, of memory, and a modest portion
>>>>> will be taken up or reserved by the OS and other stuff, the OS serves
>>>>> some of the JVM requested memory from swap.
>>>>> 
>>>>> So, if I'm right, the short answer would be to start the JVM with
>>>>> something like -Xmx1024M.  If the metadata file you're loading is only a
>>>>> few megs you can probably get away with -Xmx512M or less (if you get an
>>>>> out of memory exception then your choice was too low).
>>>>> 
>>>>> [1] This is technically no longer true, Azul Systems does have a pretty
>>>>> fancy garbage collector that no longer contains any stop-the-world
>>>>> event.  But you really need to run it on special hardware/software for
>>>>> it to be useful.
>>>>> 
>>>>> On 7/20/11 5:03 PM, Wessel, Keith William wrote:
>>>>>> All,
>>>>>> 
>>>>>> Yes, I'm still fighting with the memory usage on our Shib test cluster.
>>>>>> 
>>>>>> Interestingly enough, if we turn off the health checks from our SLB, the problem seems to go away. Mind you, there's very little activity otherwise on the IDP, so it's unclear whether it's the way the healthchecks are performed or just the total lack of activity that resolves the problem.
>>>>>> 
>>>>>> We did notice, once back on 2.1.5, that it took longer for the memory usage to climb when using the older /idp/Profile/Status than when using /idp/status, for what that's worth.
>>>>>> 
>>>>>> So, two questions.
>>>>>> 
>>>>>> First, can anyone offer suggestions other than /idp/Profile/Status and /idp/status for our SLB to do healthchecks against? Or is /idp/status the best way to do healthchecks from an SLB?
>>>>>> 
>>>>>> Secondly, Chad, are you still willing to look at heap dumps from before and after the problem occurs? If so, I can send a link to them to you off-list.
>>>>>> 
>>>>>> Thanks, all,
>>>>>> Keith
>>>>>> 
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Wessel, Keith William 
>>>>>> Sent: Tuesday, June 21, 2011 1:58 PM
>>>>>> To: 'shibboleth-users at internet2.edu'
>>>>>> Subject: RE: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>>>> 
>>>>>> No, only the IDP was upgraded. Everything else (Java, Tomcat, Terracotta) stayed the same. I didn't want to confuse things and change too much at once.
>>>>>> 
>>>>>> $IDP_HOME/lib/endorsed contains (and has contained) xalan-2.7.1.jar and xercesImpl-2.9.1.jar. Those look okay?
>>>>>> 
>>>>>> Thanks for the thought on that.
>>>>>> 
>>>>>> Keith
>>>>>> 
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: shibboleth-users-request at internet2.edu [mailto:shibboleth-users-request at internet2.edu] On Behalf Of Chad La Joie
>>>>>> Sent: Tuesday, June 21, 2011 1:51 PM
>>>>>> To: shibboleth-users at internet2.edu
>>>>>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>>>> 
>>>>>> When you did the upgrade on your test machine, did you upgrade *just* 
>>>>>> the IdP or Tomcat/Java as well?  If you upgraded Tomcat/Java check to 
>>>>>> make sure you endorsed the Xerces/Xalan libraries appropriately.  Xerces 
>>>>>> had a really bad memory leak a while ago and it's possible that might 
>>>>>> cause what you're seeing if you're loading a lot of metadata with a 
>>>>>> modestly frequent (once or twice a day) refresh interval.
>>>>>> 
>>>>>> On 6/21/11 2:43 PM, Wessel, Keith William wrote:
>>>>>>> I would agree. Even we have been running 2.1.5 for over a year and haven't seen any problems on our RHEL 4 servers in test or production... until now, with our test servers acting weird.
>>>>>>> 
>>>>>>> So, if this turns out to be a Redhat issue, I apologize for wasting your and the rest of the list's time with it.
>>>>>>> 
>>>>>>> Keith
>>>>>>> 
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: shibboleth-users-request at internet2.edu [mailto:shibboleth-users-request at internet2.edu] On Behalf Of Chad La Joie
>>>>>>> Sent: Tuesday, June 21, 2011 1:41 PM
>>>>>>> To: shibboleth-users at internet2.edu
>>>>>>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>>>>> 
>>>>>>> Well, I'm pretty certain there isn't an issue with 2.1.5.  There are a
>>>>>>> lot of people running that so if there was an issue there I'm pretty
>>>>>>> sure it would have surfaced by now.
>>>>>>> 
>>>>>>> On 6/21/11 2:36 PM, Wessel, Keith William wrote:
>>>>>>>> Chad,
>>>>>>>> 
>>>>>>>> Thanks for the helpful reply, and sorry for the delay. I've been doing more testing with this.
>>>>>>>> 
>>>>>>>> I'm suspecting there's something else at play on this system, but I can't put my finger on what. I rolled back to IDP 2.1.5, and the problem continued: between a few hours and a couple days after a reboot, the memory usage on the system shot up. Something's confusing the kernel, though. Even after dtopping Tomcat, the memory doesn't return to the system as being available again. It stays that way until another reboot. Removing Terracotta from the mix doesn't change anything.
>>>>>>>> 
>>>>>>>> I produced Java heap dumps if you have time and are interested in looking at them, Chad. The heap dumps were back on IDP V 2.1.5 as it doesn't seem to matter which IDP version I'm running tomake this happen. Email me off-list and let me know where to send the URLs for the dumps if you think it might help.
>>>>>>>> 
>>>>>>>> As far as I can figure out, nothing was upgraded, patched or changed on this system just prior to theIDP upgrade. So, if it was 2.3.0 that was causing the problems, downgrading back to 2.1.5 should have fixed it... but it didn't. The IDP upgrade was clearly a red herring.
>>>>>>>> 
>>>>>>>> My next step is to ask the system manager to roll back to the previous kernel, though that was upgraded a couple months ago, and this problem didn't appear until a couple weeks ago when I upgraded the IDP. Might also be interesting to see what the system does without Tomcat or Terracotta running so I can at least eliminate Shib from the mix and leave you all alone if this has nothing to do with Shib. At this point, I'm pretty sure it doesn't, but I'd still like concrete proof.
>>>>>>>> 
>>>>>>>> Keith
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: shibboleth-users-request at internet2.edu [mailto:shibboleth-users-request at internet2.edu] On Behalf Of Chad La Joie
>>>>>>>> Sent: Thursday, June 16, 2011 6:40 AM
>>>>>>>> To: shibboleth-users at internet2.edu
>>>>>>>> Subject: Re: [Shib-Users] IDP 2.3.0 memory hungry? Or something else in our setup?
>>>>>>>> 
>>>>>>>> There is nothing new in 2.3.0 that should be consuming more memory than
>>>>>>>> any other recent version (2.1.2 and below does have a known memory
>>>>>>>> leak).  And I've had no other reports of higher-than-expected memory
>>>>>>>> usage from other folks who have upgraded.  I'd note though that Top is
>>>>>>>> not a good mechanism to look at Java memory usage.  It tells you nothing
>>>>>>>> about the amount of heap that is actually in use, only the amount of
>>>>>>>> memory currently reserved by the JVM.
>>>>>>>> 
>>>>>>>> So, to start with, can you remove Terracotta from the mix and run your
>>>>>>>> test again? I just want to get the test environment down to as few
>>>>>>>> things as possible.
>>>>>>>> 
>>>>>>>> If you are still seeing memory issues, the next step would be to produce
>>>>>>>> some heap dumps for analysis.  The best would be to do this when the
>>>>>>>> system first starts and then some time later after the process's memory
>>>>>>>> has increased.  You can use the command 'jmap -dump:format=b<pid>'.
>>>>>>>> Then just put the files up on a server somewhere I can grab them and
>>>>>>>> send me the URL directly.
>>>>>>>> 
>>>>>>>> Note, the heap will contain *all* information currently in heap, this
>>>>>>>> includes things like any passwords that show up in your
>>>>>>>> attribute-resolver.xml and such.  So, if you're uncomfortable with me
>>>>>>>> seeing that you can comment out those things.
>>>>>>>> 
>>>>>>>> On 6/15/11 3:57 PM, Wessel, Keith William wrote:
>>>>>>>>> Hi, all,
>>>>>>>>> 
>>>>>>>>> I just upgraded our test IDP cluster to 2.3.0 yesterday. Since then, the system has seemed a bit more short on free memory. It's running on Tomcat 6.0.20 and JDK 1.6.0.18. We've also got Terracotta 3.2.1 in the mix. It's on a RHEL 4-based 64-bit virtual server.
>>>>>>>>> 
>>>>>>>>> We were sitting at about 800 MB of memory usage and hardly touching swap when we were running IDP 2.1.5. After the upgrade, we jumped up to close to the full 1 GB of RAM allocated to the VM and diving well into swap. We upgraded the VM to 2 GB of RAM today and, after being up and running for a little over an hour, we saw it climb to using up most of that 2 GB of RAM and again eating into swap. The growth on our performance graph is a nice exponential curve, starting out slow, then accelerating.
>>>>>>>>> 
>>>>>>>>> The funny thing is I see no activity on the test cluster during this time, and that should be the case. The only activity would have been our SLB hitting /idp/status with health checks.
>>>>>>>>> 
>>>>>>>>> I ran a Terracotta DGC manually a few minutes ago (via run-dgc-if-active.sh), and that, as I suspected, changed nothing in terms of server memory consumption.
>>>>>>>>> 
>>>>>>>>> I'll also add that we made no configuration changes between 2.1.5 and 2.3.0 other than a hand full of syntactical ones in the config files to meet new config requirements.
>>>>>>>>> 
>>>>>>>>> Looking at top output, it's definitely the Tomcat Java process, and not the Terracotta one, that seems to be running with higher physical memory usage. The total (physical + swap) look similar between 2.1.5 and 2.3.0, but the 2.3.0 seems to be keeping a lot more in physical RAM.
>>>>>>>>> 
>>>>>>>>> Can anyone think of any type of misconfiguration that would be causing this? Do we need to consider upgrading Tomcat, Java or Terracotta? Or is 2.3.0 just that much more physical memory hungry than 2.1.5?
>>>>>>>>> 
>>>>>>>>> I'm happy to provide any other details to get to the bottom of this. Thanks for any help that anyone can offer out there.
>>>>>>>>> 
>>>>>>>>> Keith
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> -- 
>>>> Chad La Joie
>>>> http://itumi.biz
>>>> trusted identities, delivered
>>>> 
>>>> -- 
>>>> To unsubscribe from this group, send email to
>>>> users+unsubscribe at shibboleth.net
>>>> 
>>>> -- 
>>>> To unsubscribe from this group, send email to
>>>> users+unsubscribe at shibboleth.net
>>> 
>>> -- 
>>> To unsubscribe from this group, send email to
>>> users+unsubscribe at shibboleth.net
>>> --
>>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>> 
>> --
>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>> --
>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
> 
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net