Shibboleth process randomly consuming 100% CPU
Manolo Garcia Alvarez
mgarciaal at uoc.edu
Thu May 23 04:43:24 EDT 2019
Hi,
just a follow-up in case there's anyone interested. Finally we got to know
what the problem was, it was caused by the JVM Garbage Collector.
We changed the default algorith to UseConcMarkSweepGC and everything's
working fine since then.
Thanks and regards.
------------------------------
Manolo García
Arquitectura i Sistemes
Universitat Oberta de Catalunya
93 326 (3451) | 689 88 30 93 | mgarciaal at uoc.edu
Parc Mediterrani de la Tecnologia (edifici B3)
Av. Carl Friedrich Gauss, 5.
08860 Castelldefels
[image: Universitat Oberta de Catalunya]
Aquest missatge s'adreça exclusivament a qui va destinat i pot contenir
informació privilegiada o confidencial i dades de caràcter personal, la
difusió de les quals és regulada per la Llei orgànica de protecció de dades
i la Llei de serveis de la societat de la informació. Si no sou la persona
destinatària indicada (o la responsable de lliurar-lo a qui va destinat),
no heu de copiar aquest missatge ni lliurar-lo a tercers per cap concepte.
Si heu rebut aquest missatge per error o l'heu aconseguit per altres
mitjans, us demanem que ens ho comuniqueu immediatament per aquesta mateixa
via i l'elimineu irreversiblement.
Abans d'imprimir aquest missatge electrònic penseu en el medi ambient.
El lun., 11 mar. 2019 a las 8:36, Manolo Garcia Alvarez (<mgarciaal at uoc.edu>)
escribió:
> Hi,
>
> we are dealing with CPU problems in one of our three Shibboleth servers.
> All of the three servers that are part of our IdP service have exactly the
> same configuration and same versions of Linux (2.6.32), Apache (2.2.15),
> Tomcat (7.0.68), Shibboleth (3.1.2) and JVM (Oracle 1.7.0_51-b13). Randomly
> and without any known reason, the Java process of one of the three servers
> starts to consume more CPU and soon it gets to nearly 100%, and the only
> solution is to restart Shibboleth. We haven't been able to identify the
> reason of this behavior, it takes place apparently at any hour, with fewer
> or more users requests. This is causing us huge problems and many
> complaints, because the user requests get stuck and it causes service
> outages.
>
> We have reviewed the logs and the only thing that seems to be an error
> when the CPU is 100% is an error in the Tomcat logs:
>
> Feb 18, 2019 12:27:06 PM org.apache.catalina.core.StandardWrapperValve
> invoke
> SEVERE: Servlet.service() for servlet [idp] in context with path [/idp]
> threw exception [Request processing failed; nested exception
> is java.lang.IllegalStateException: getOutputStream() has already been
> called for this response] with root cause
> java.lang.IllegalStateException: getOutputStream() has already been called
> for this response
> at org.apache.catalina.connector.Response.getWriter(Response.java:663)
> at
> org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:213)
> at
> javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:104)
> at
> net.shibboleth.utilities.java.support.net.CookieBufferingFilter$CookieBufferingHttpServletResponseProxy.getWriter(CookieBu
> fferingFilter.java:116)
> at
> javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:104)
> at
> org.opensaml.storage.impl.ServletRequestScopedStorageService$OutputInterceptingHttpServletResponseProxy.getWriter(ServletR
> equestScopedStorageService.java:564)
> at
> javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:104)
> at
> org.opensaml.storage.impl.ServletRequestScopedStorageService$OutputInterceptingHttpServletResponseProxy.getWriter(ServletR
> equestScopedStorageService.java:564)
> at
> org.springframework.web.servlet.view.velocity.VelocityView.mergeTemplate(VelocityView.java:517)
> at
> org.springframework.web.servlet.view.velocity.VelocityView.doRender(VelocityView.java:462)
> at
> org.springframework.web.servlet.view.velocity.VelocityView.renderMergedTemplateModel(VelocityView.java:292)
> at
> org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167)
> at
> org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
> at
> org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1228)
> at
> org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1011)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:955)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
> at
> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> net.shibboleth.idp.log.SLF4JMDCServletFilter.doFilter(SLF4JMDCServletFilter.java:64)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> net.shibboleth.utilities.java.support.net.RequestResponseContextFilter.doFilter(RequestResponseContextFilter.java:60)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.opensaml.storage.impl.ServletRequestScopedStorageService.doFilter(ServletRequestScopedStorageService.java:276)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.opensaml.storage.impl.ServletRequestScopedStorageService.doFilter(ServletRequestScopedStorageService.java:276)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> net.shibboleth.utilities.java.support.net.CookieBufferingFilter.doFilter(CookieBufferingFilter.java:67)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:203)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
> at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:744)
>
> Has anyone experienced the same issue? Any idea we could use to solve this?
>
> Thanks a lot !
>
> ------------------------------
> Manolo García
> Arquitectura i Sistemes
> Universitat Oberta de Catalunya
>
> 93 326 (3451) | 689 88 30 93 | mgarciaal at uoc.edu
> Parc Mediterrani de la Tecnologia (edifici B3)
> Av. Carl Friedrich Gauss, 5.
> 08860 Castelldefels
> [image: Universitat Oberta de Catalunya]
> Aquest missatge s'adreça exclusivament a qui va destinat i pot contenir
> informació privilegiada o confidencial i dades de caràcter personal, la
> difusió de les quals és regulada per la Llei orgànica de protecció de dades
> i la Llei de serveis de la societat de la informació. Si no sou la persona
> destinatària indicada (o la responsable de lliurar-lo a qui va destinat),
> no heu de copiar aquest missatge ni lliurar-lo a tercers per cap concepte.
> Si heu rebut aquest missatge per error o l'heu aconseguit per altres
> mitjans, us demanem que ens ho comuniqueu immediatament per aquesta mateixa
> via i l'elimineu irreversiblement.
>
> Abans d'imprimir aquest missatge electrònic penseu en el medi ambient.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20190523/286034fe/attachment.html>
More information about the users
mailing list