Shibboleth process randomly consuming 100% CPU

[Peter Schober]

* Manolo Garcia Alvarez <mgarciaal at uoc.edu> [2019-03-11 13:24]:
> I know 3.1.2 is old, we're on the process of upgrading to 3.4. It will take
> some time because we've got a lot of authentications and we need to be up
> 24x7, but I would try to speed it up if I'm sure the problem relies on the
> Shibboleth version.

Sorry, I can't tell you that.

But having 3 "shared nothing" servers and a load balancer should be
all anyone needs in order to make a seamless upgrade?

E.g. take a node of the loadbalancer, replace it with a new machine or
upgrade in place, adjust config and test with aacli until satisfied,
move it back into the loadbalancing pool, possibly with only a smaller
percentage of traffic going to it while making sure results are what's
expected (e.g. NameID values are the same, audit log shows same
attributes sent to the same SP for the same subject, etc.).
Then swap out all other nodes in the same manner one by one.

> The load balancer is moving the traffic to other servers but it
> takes 5 minutes more or less, so a lot of users are affected by the
> error.

Again that sounds like a problem description that already contains its
own fix. Continuing to route traffic to a nonfunctional machine for
five minutes doesn't make a lot of sense, why not take the node out of
the pool much faster? Or does it take so long for the condition to
manifest itself? Or does CPU usage increase so suddenly you don't have
time to detect this in a monitoring system and initiate the pool
change in time?

I realise this all has nothing to do with what you've asked about.
Still given the (limited) information provided some of the issues seem
avoidable.

-peter