load-balancing tests and the shib IdP

[Eric Goodman]

Specifically on the Terracotta side, and perhaps dated information:


> I would say that one primary issue is Terracotta. I don't use it, ergo my
> IdP runs nicely without much oversight and my standard monitors are well
> able to let me know in the very rare event something breaks.
>

We haven't had many Shib outages, but almost all of the outages we have had
over the last 3 years are Terracotta-related. We find that monitoring is
best done at the Terracotta log level.

Specifically, we see problems after brief (1-15 seconds) network outages --
it's never been an issue for longer ones. It seems to be when an outage is
just long enough for each Terracotta server to decide other servers are
unavailable, start an election and elect itself (as the only available
server) master, but the outage still is short enough that all the servers
are back in communication when the servers go to publish the election
results.

In that case, each server indefinitely asserts itself as the master and
refuses to acknowledge the same messages ("no, I'm the master") from the
other servers. They never re-initiate an election for master -- presumedly
because they haven't finished the current one? -- and over time, the
Terracotta client queries back up (since the servers need to resolve the
election to returning session information).

That query backup also seemed to be inferable by watching Java thread
counts on the Terracotta client.

After a longer outage, the servers recognize that "new" servers became
available since the last election completed, and they cooperatively
re-elect a new master, it's only the very brief ones that are the issue.

If it's of interest, I can see if I can find the specific information we
look for in the Terracotta logs, but basically it's continual errors about
finding other masters that refuse to accept election results. Http logging
was no good because the base pages still render, but also, once we see the
error, the entire cluster will reliably stop responding in 2-10 hours
(depending on load -- as all the queries start backing up), so taking the
one misbehaving server offline is only a very temporary band-aid fix.

Again though, all of the above is really about monitoring Terracotta, not
monitoring Shibboleth. And to be fair, we haven't seen the error in
probably over a year, so it's possible the underlying Terracotta error has
been fixed and this is all dated information anyway.

--- Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20120906/01d6a511/attachment-0001.html