HTTP Status 404 - /idp/status: Out of Left Field!

[Kanuch, Andrew]

Good Morning,

We have a fully functional IDP box that suddenly stopped working at some point between 9:00 AM yesterday, and 9:00 AM today. I can pull up the tomcat service page, but not the IDP page. (Server 2008 R2 Enterprise, Shibboleth 2.3.6 )

I am unsure how best to proceed because of what I found in the tomcat logs. All the logs after restarting the service show nothing but "INFO" or "FINE" entries.   But log entries dated to around 9am yesterday morning show a slew of warnings and severe messages.

(I've trimmed out a fair amount of java addressing to condense it)

WARNING: Exception obtaining work path for
Nov 2, 2014 9:43:07 AM org.apache.catalina.startup.HostConfig checkResources
WARNING: Error delete redeploy resources from context []
java.io.IOException: Insufficient system resources exist to complete the requested service

(This is repeated three times)

Nov 2, 2014 9:43:07 AM org.apache.catalina.startup.HostConfig checkResources
INFO: Undeploying context [/manager]
Nov 2, 2014 9:43:07 AM org.apache.catalina.core.StandardContext getWorkPath
WARNING: Exception obtaining work path for /manager
Nov 2, 2014 9:43:07 AM org.apache.catalina.startup.HostConfig checkResources
WARNING: Error delete redeploy resources from context [/manager]
java.io.IOException: Insufficient system resources exist to complete the requested service

(This is repeated five times)

INFO: Undeploying context [/docs]
Nov 2, 2014 9:43:08 AM org.apache.catalina.core.StandardContext getWorkPath
WARNING: Exception obtaining work path for /docs
Nov 2, 2014 9:43:08 AM org.apache.catalina.startup.HostConfig checkResources
WARNING: Error delete redeploy resources from context [/docs]

(This is repeated three times)

INFO: Undeploying context [/idp]
Nov 2, 2014 9:43:08 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/idp] appears to have started a thread named [net.sf.ehcache.CacheManager at 1f2af1c] but has failed to stop it. This is very likely to create a memory leak.
Nov 2, 2014 9:43:08 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/idp] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Nov 2, 2014 9:43:08 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/idp] appears to have started a thread named [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This is very likely to create a memory leak.



So I would guess there already was a memory leak, and it ended up corrupting some of my tomcat files.

Is there a way I could redeploy tomcat from the windows installer, but leave Shib IDP files  as they are?  Or am I off the mark and to you think something else is the cause?

Thank you for your time.

Andy
Communications Network Analyst
University Networking and Research Computing
South Dakota State University
SOHO #200, Box 2231
Brookings, SD 57007

"It matters not how strait the gate, how charged with punishments the scroll, I am the master of my fate: I am the captain of my soul."
--William Earnest Henry

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20141103/086545b3/attachment.html