Shibboleth process randomly consuming 100% CPU

[Manolo Garcia Alvarez]

Hi,

we are dealing with CPU problems in one of our three Shibboleth servers.
All of the three servers that are part of our IdP service have exactly the
same configuration and same versions of Linux (2.6.32), Apache (2.2.15),
Tomcat (7.0.68), Shibboleth (3.1.2) and JVM (Oracle 1.7.0_51-b13). Randomly
and without any known reason, the Java process of one of the three servers
starts to consume more CPU and soon it gets to nearly 100%, and the only
solution is to restart Shibboleth. We haven't been able to identify the
reason of this behavior, it takes place apparently at any hour, with fewer
or more users requests. This is causing us huge problems and many
complaints, because the user requests get stuck and it causes service
outages.

We have reviewed the logs and the only thing that seems to be an error when
the CPU is 100% is an error in the Tomcat logs:

Feb 18, 2019 12:27:06 PM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet [idp] in context with path [/idp]
threw exception [Request processing failed; nested exception
is java.lang.IllegalStateException: getOutputStream() has already been
called for this response] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:663)
at
org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:213)
at
javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:104)
at
net.shibboleth.utilities.java.support.net.CookieBufferingFilter$CookieBufferingHttpServletResponseProxy.getWriter(CookieBu
fferingFilter.java:116)
at
javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:104)
at
org.opensaml.storage.impl.ServletRequestScopedStorageService$OutputInterceptingHttpServletResponseProxy.getWriter(ServletR
equestScopedStorageService.java:564)
at
javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:104)
at
org.opensaml.storage.impl.ServletRequestScopedStorageService$OutputInterceptingHttpServletResponseProxy.getWriter(ServletR
equestScopedStorageService.java:564)
at
org.springframework.web.servlet.view.velocity.VelocityView.mergeTemplate(VelocityView.java:517)
at
org.springframework.web.servlet.view.velocity.VelocityView.doRender(VelocityView.java:462)
at
org.springframework.web.servlet.view.velocity.VelocityView.renderMergedTemplateModel(VelocityView.java:292)
at
org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167)
at
org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
at
org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1228)
at
org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1011)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:955)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
net.shibboleth.idp.log.SLF4JMDCServletFilter.doFilter(SLF4JMDCServletFilter.java:64)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
net.shibboleth.utilities.java.support.net.RequestResponseContextFilter.doFilter(RequestResponseContextFilter.java:60)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.opensaml.storage.impl.ServletRequestScopedStorageService.doFilter(ServletRequestScopedStorageService.java:276)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.opensaml.storage.impl.ServletRequestScopedStorageService.doFilter(ServletRequestScopedStorageService.java:276)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
net.shibboleth.utilities.java.support.net.CookieBufferingFilter.doFilter(CookieBufferingFilter.java:67)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:203)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:744)

Has anyone experienced the same issue? Any idea we could use to solve this?

Thanks a lot !

------------------------------
Manolo García
Arquitectura i Sistemes
Universitat Oberta de Catalunya

93 326 (3451) | 689 88 30 93 | mgarciaal at uoc.edu
Parc Mediterrani de la Tecnologia (edifici B3)
Av. Carl Friedrich Gauss, 5.
08860 Castelldefels
[image: Universitat Oberta de Catalunya]
Aquest missatge s'adreça exclusivament a qui va destinat i pot contenir
informació privilegiada o confidencial i dades de caràcter personal, la
difusió de les quals és regulada per la Llei orgànica de protecció de dades
i la Llei de serveis de la societat de la informació. Si no sou la persona
destinatària indicada (o la responsable de lliurar-lo a qui va destinat),
no heu de copiar aquest missatge ni lliurar-lo a tercers per cap concepte.
Si heu rebut aquest missatge per error o l'heu aconseguit per altres
mitjans, us demanem que ens ho comuniqueu immediatament per aquesta mateixa
via i l'elimineu irreversiblement.

Abans d'imprimir aquest missatge electrònic penseu en el medi ambient.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20190311/8ea0fb97/attachment.html>