<div class="gmail_quote"><div>Specifically on the Terracotta side, and perhaps dated information:</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I would say that one primary issue is Terracotta. I don't use it, ergo my<br>
IdP runs nicely without much oversight and my standard monitors are well<br>
able to let me know in the very rare event something breaks.<br></blockquote><div><br></div><div>We haven't had many Shib outages, but almost all of the outages we have had over the last 3 years are Terracotta-related. We find that monitoring is best done at the Terracotta log level. </div>
<div><br></div><div>Specifically, we see problems after brief (1-15 seconds) network outages -- it's never been an issue for longer ones. It seems to be when an outage is just long enough for each Terracotta server to decide other servers are unavailable, start an election and elect itself (as the only available server) master, but the outage still is short enough that all the servers are back in communication when the servers go to publish the election results. </div>
<div><br></div><div>In that case, each server indefinitely asserts itself as the master and refuses to acknowledge the same messages ("no, I'm the master") from the other servers. They never re-initiate an election for master -- presumedly because they haven't finished the current one? -- and over time, the Terracotta client queries back up (since the servers need to resolve the election to returning session information).</div>
<div><br></div><div>That query backup also seemed to be inferable by watching Java thread counts on the Terracotta client.</div><div><br></div><div><div>After a longer outage, the servers recognize that "new" servers became available since the last election completed, and they cooperatively re-elect a new master, it's only the very brief ones that are the issue.</div>
</div><div><br></div><div>If it's of interest, I can see if I can find the specific information we look for in the Terracotta logs, but basically it's continual errors about finding other masters that refuse to accept election results. Http logging was no good because the base pages still render, but also, once we see the error, the entire cluster will reliably stop responding in 2-10 hours (depending on load -- as all the queries start backing up), so taking the one misbehaving server offline is only a very temporary band-aid fix.</div>
<div><br></div><div>Again though, all of the above is really about monitoring Terracotta, not monitoring Shibboleth. And to be fair, we haven't seen the error in probably over a year, so it's possible the underlying Terracotta error has been fixed and this is all dated information anyway.</div>
<div><br></div><div>--- Eric</div></div>