Weird IdP Hanging Issue

[Russell Beall]

I'm just wondering if you provided any extra memory to the tomcat process.  You might be hitting your memory limits and causing tomcat to go into regular Full GC cycles.  This is usually what I have seen cause the behavior you described.  It would be useful to add debug logging to the tomcat process which will print garbage collection details.  For instance, I use these in my JAVA_OPTS:

-verbose:gc 
-XX:+PrintGCTimeStamps 
-XX:-TraceClassUnloading 

Also, regarding the resource reloading polling frequency, I have those set at one minute, but the reload is never invoked unless there is a change.  It should be safe to maintain a short interval on that reload check.  If you remove the reload interval, then I believe it won't ever check and you would have to restart the IdP to get it to load the change.  That might be fine for the relying-party.xml file which Chad said shouldn't really be reloaded on the fly, but others that get changed more frequently and are safe should be reloadable, such as the attribute-filter.xml that was discussed already.

Regards,
Russ.

On Mar 21, 2013, at 11:11 AM, "Royder, Kyle D" <kroyder at austin.utexas.edu> wrote:

> Thanks for the help!  I'll turn up our LDAP logging and watch check the LDAP logs as well, and turn configuration reloading.  We have a pool of IdPs and like to reboot them one at a time anyways to bring the one with changes back down if there is an issue to make sure the service stays up as a safety precaution.  If this is our change policy, I don't think configuration reloading with gain us anything the way we are currently doing things.
> 
> Thanks,
> Kyle
> 
> -----Original Message-----
> From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Cantor, Scott
> Sent: Thursday, March 21, 2013 1:04 PM
> To: Shib Users
> Subject: RE: Weird IdP Hanging Issue
> 
>> I haven't had to mess with this before, but reading about it on the wiki, I'm
>> assuming you're referring to the configurationResourcePollingFrequency
>> attribute added to one of the four reloadable services in service.xml?
> 
> Yes.
> 
>> If so, it does look like this was setup to reload all four every 1 minute?
> 
> Yikes.
> 
>> I don't want this turned on so I'm going to remove the
>> configurationResourcePollingFrequency attribute from all four of these.
> 
> You may well want the filer policy reloading, but probably not every minute.
> 
>> Hopefully this will resolve this issue.  The weird thing is that none of these
>> configs have been changes in a couple of weeks and this just started over the
>> past couple of days.
> 
> Agreed, I didn't necessarily think it was the cause, but there's an explicit bug in relying-party reloading, though I think that actually hangs hard.
> 
> I would have to think your data connectors are the underlying cause here. If it's LDAP, I'd probably suggest logging more there.
> 
> -- Scott
> 
> 
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>