Tomcat error popping up on rotation servers in rapid succession

[Botsford, Benjamin T]

Hey everyone,

Earlier today, two of our three Shibboleth IdP servers went down in rapid succession, seemingly for the same reason. I originally suspected it was a metadata error since we've seen a few of those recently, but it appears to have been Tomcat-related.

[error] ajp_read_header: ajp_ilink_receive failed
[error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost)
[error] (70007)The timeout specified has expired: proxy: read response failed from 127.0.0.1:8009 (localhost)
[error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

These aren't terribly abnormal errors, I understand. What's confusing us is why this would have happened on two separate servers within minutes of one another (it doesn't seem to be load-related). Could this be caused by a response type or a string included in a response, maybe?

Any help would be appreciated. We've never seen it before and we hope to never see it again, but we came very close to a system outage and would love to figure out how to guarantee that it doesn't pop up again.

Thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20170613/e40cff5e/attachment-0001.html>