memcached failover

[Paul B. Henson]

> From: Andrew Morgan
> Sent: Monday, August 15, 2016 3:02 PM
> 
> So...  are you guys planning to take down your authentication service in
> order to perform a Hazelcast upgrade?  Or is there something I'm missing?
> This seems like a pretty major problem...

Interesting; I don't believe I had seen that before. I don't think I actually upgraded Hazelcast that many times while I was using it, the Hazelcast upgrades were driven by the upstream Unicon build, and I assume they only updated it as necessary, not whenever a random new version was released. I'm also pretty sure I violated this recommendation at least once without noticing any issues. The CAS ticket registry and the shibboleth storage engine only use the Hazelcast distributed map functionality, which is a tiny fraction of the overall capability. It's possible that if you are only using this subset of the functionality the limitation does not apply, although perhaps it would not be wise to gamble on it.

So to answer your question though, if I wanted to follow the recommendation strictly, I would not have a complete outage of the authentication service. We have a three node cluster. What I would probably do would be pull one node out of the cluster during a maintenance window, upgrade it, temporarily configure it to be a standalone node, and bring it back online, put it back in service, and pull the other two nodes out. Upgrade the other two nodes, bring them back online, pull the first node back off-line, reconfigure it back into the cluster, and bring it back online. There would be some state loss during the maintenance window, but no actual outage. But I would only do this if trying to do the upgrade live failed in my dev environment...

--
Paul B. Henson  |  (909) 979-6361  |  http://www.cpp.edu/~henson/
Operating Systems and Network Analyst  |  henson at cpp.edu
California State Polytechnic University  |  Pomona CA 91768