memcache failover error

[Andrew Morgan]

I have 2 nodes of IDP v3.3 running behind a network load balancer.  Each 
node runs memcached as well, and the IDP is configured to balance memcache 
sessions between the nodes.  From global.xml:

     <bean id="shibboleth.MemcachedStorageService"
           class="org.opensaml.storage.impl.memcached.MemcachedStorageService"
           c:timeout="2">
         <constructor-arg name="client">
             <bean class="net.spy.memcached.spring.MemcachedClientFactoryBean"
                   p:servers="login1:11211,login2:11211"
                   p:protocol="BINARY"
                   p:locatorType="CONSISTENT"
                   p:failureMode="Redistribute">
                 <property name="hashAlg">
                     <util:constant static-field="net.spy.memcached.DefaultHashAlgorithm.FNV1_64_HASH" />
                 </property>
                 <property name="transcoder">
                     <!-- DO NOT MODIFY THIS PROPERTY -->
                     <bean class="org.opensaml.storage.impl.memcached.StorageRecordTranscoder" />
                 </property>
             </bean>
         </constructor-arg>
     </bean>

However, when I rebooted login1, myself (and I assume others) were shown 
an error after authentication.  The following appeared in the 
idp-process.log:

2017-03-09 14:25:20,773 - INFO [net.shibboleth.idp.authn.duo.impl.ValidateDuoWebResponse:202] - Profile Action ValidateDuoWebResponse: Duo authentication succeeded for 'morgan'
2017-03-09 14:25:20,801 - ERROR [net.shibboleth.idp.session.impl.StorageBackedSessionManager:527] - Exception while storing new session for principal morgan
java.io.IOException: Memcached operation error
         at org.opensaml.storage.impl.memcached.MemcachedStorageService.handleAsyncResult(MemcachedStorageService.java:665)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Cancelled
         at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177)
Caused by: java.util.concurrent.CancellationException: Cancelled
         at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177)
2017-03-09 14:25:20,822 - ERROR [net.shibboleth.idp.session.impl.UpdateSessionWithAuthenticationResult:178] - Profile Action UpdateSessionWithAuthenticationResult: Error creating session for principal morgan
net.shibboleth.idp.session.SessionException: Exception while storing new session
         at net.shibboleth.idp.session.impl.StorageBackedSessionManager.createSession(StorageBackedSessionManager.java:529)
Caused by: java.io.IOException: Memcached operation error
         at org.opensaml.storage.impl.memcached.MemcachedStorageService.handleAsyncResult(MemcachedStorageService.java:665)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Cancelled
         at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177)
Caused by: java.util.concurrent.CancellationException: Cancelled
         at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177)
2017-03-09 14:25:20,937 - ERROR [net.shibboleth.idp.cas:-2] - Uncaught runtime exception
java.lang.IllegalStateException: Cannot locate IdP session
         at net.shibboleth.idp.cas.flow.impl.AbstractCASProtocolAction.getIdPSession(AbstractCASProtocolAction.java:166)
2017-03-09 14:25:20,947 - WARN [org.opensaml.profile.action.impl.LogEvent:105] - A non-proceed event occurred while processing the request: RuntimeException
2017-03-09 14:25:20,949 - INFO [Shibboleth-Audit.SSO:241] - 20170309T222520Z||||https://www.apereo.org/cas/protocol/login||||morgan|||||


I was expecting the memcache client to failover to the remaining node 
(login2) with a loss of existing sessions, but I didn't expect there to be 
errors for users.

Do I have memcache configured correctly?  If necessary, I can do some 
testing on my test instances.

Thanks,
 	Andy