memcached

[Andrew Morgan]

On Tue, 6 Jun 2017, John C. Pfeifer wrote:

> I have memcached configured for my IdP (3.3.1) exactly as described in 
> the wiki and it is working fine in my dev environment (one instance of 
> shib, one instance of memcached).  However, in my clustered QA env (4 
> shib, 4 memcached), if the connection between shib and a given memcached 
> is lost, it never gets reestablished.  I’ve tried to work my way through 
> various documentation but nothing seems to address this issue.

John,

We are running memcache in production with 2 nodes.  Here is what we used 
in global.xml:

     <bean id="shibboleth.MemcachedStorageService"
           class="org.opensaml.storage.impl.memcached.MemcachedStorageService"
           c:timeout="2">
         <constructor-arg name="client">
             <bean class="net.spy.memcached.spring.MemcachedClientFactoryBean"
                   p:servers="login1.onid.oregonstate.edu:11211,login2.onid.oregonstate.edu:11211"
                   p:protocol="BINARY"
                   p:locatorType="CONSISTENT"
                   p:failureMode="Redistribute">
                 <property name="hashAlg">
                     <util:constant static-field="net.spy.memcached.DefaultHashAlgorithm.FNV1_64_HASH" />
                 </property>
                 <property name="transcoder">
                     <!-- DO NOT MODIFY THIS PROPERTY -->
                     <bean class="org.opensaml.storage.impl.memcached.StorageRecordTranscoder" />
                 </property>
             </bean>
         </constructor-arg>
     </bean>


This configuration came from the wiki page example.

I have had issues with failover, but not with reconnection.  When I take 1 
memcached node offline, there are log entries in Tomcat's catalina.out 
from the spy.memcached library indicating that a node has failed and it is 
trying to reconnect.  It uses an exponential backoff for reconnects, 
retrying after 2, 4, 8, 16, 32, etc seconds.

However, it doesn't seem to failover to the other node correctly for me. 
It keeps trying to use the same node as the hash algorithm generates.

Can you check your catalina logs for the reconnection messages?

I haven't been able to figure out why the failover is failing for me.

 	Andy