Package org.opensaml.storage.impl.memcached

Class MemcachedStorageService

java.lang.Object

net.shibboleth.utilities.java.support.component.AbstractInitializableComponent

net.shibboleth.utilities.java.support.component.AbstractIdentifiedInitializableComponent

net.shibboleth.utilities.java.support.component.AbstractIdentifiableInitializableComponent

org.opensaml.storage.impl.memcached.MemcachedStorageService

All Implemented Interfaces:

Component, DestructableComponent, IdentifiableComponent, IdentifiedComponent, InitializableComponent, StorageService

public class MemcachedStorageService
extends AbstractIdentifiableInitializableComponent
implements StorageService

Memcached storage service. The implementation of context names is based on the implementation of simulated namespaces discussed on the Memcached project site:

https://code.google.com/p/memcached/wiki/NewProgrammingTricks#Namespacing

This storage service supports arbitrary-length context names and keys despite the 250-byte limit on memcached keys. Keys whose length is greater than 250 bytes are hashed using the SHA-512 algorithm and hex encoded to produce a 128-character key that is stored in memcached. Collisions are avoided irrespective of hashing by using the memcached add operation on all create operations which guarantees that an entry is created if and only if a key of the same value does not already exist. Note that context names and keys are assumed to have single-byte encodings in UTF-8 (i.e. ASCII characters) such that key lengths are equal to their size in bytes. Hashed keys naturally meet this requirement.

An optional context key-tracking feature is available to support updateContextExpiration(String, Long). Key tracking is disabled by default, but can be enabled by setting the enableContextKeyTracking parameter in the MemcachedStorageService(net.spy.memcached.MemcachedClient, int, boolean) constructor. While there is modest performance impact for create and delete operations, the feature limits the number of keys per context. With the default 1M memcached slab size, in the worst case 4180 keys are permitted per context. In many if not most situations the value is easily double that. The limitation can be overcome by increasing the slab size, which decreases overall cache memory consumption efficiency. When key tracking is disabled, there is no limit on the number of keys per context other than overall cache capacity.

Limitations and requirements

1. The memcached binary protocol is strong recommended for efficiency and full versioning support. In particular, deleteWithVersion(long, String, String) and deleteWithVersion(long, Object) will throw runtime errors under the ASCII protocol.
2. Memcached server 1.4.14 or later MUST be used with binary protocol for proper handling of cache entry expiration values. See the 1.4.14 release notes for details.

Field Summary

Fields

Modifier and Type	Field	Description
protected static final String	CTX_KEY_DELETED_SUFFIX	Key suffix for entry that contains a list of deleted context keys.
private static final String	CTX_KEY_LIST_DELIMITER	Delimiter of items in the context key list.
protected static final String	CTX_KEY_LIST_SUFFIX	Key suffix for entry that contains a list of context keys.
private final org.slf4j.Logger	logger	Logger instance.
private static final int	MAX_KEY_LENGTH	Maximum length in bytes of memcached keys.
private final net.spy.memcached.MemcachedClient	memcacheClient	Memcached client instance.
private int	operationTimeout	Memcached asynchronous operation timeout in seconds.
private MemcachedStorageCapabilities	storageCapabilities	Invariant storage capabilities.
private final net.spy.memcached.transcoders.Transcoder<MemcachedStorageRecord<?>>	storageRecordTranscoder	Handles conversion of MemcachedStorageRecord to bytes and vice versa.
private final net.spy.memcached.transcoders.Transcoder<String>	stringTranscoder	Handles conversion of strings to bytes and vice versa.
private boolean	trackContextKeys	Flag that controls context key tracking.

Constructor Summary

Constructors

Constructor	Description
MemcachedStorageService(net.spy.memcached.MemcachedClient client, int timeout)	Creates a new instance.
MemcachedStorageService(net.spy.memcached.MemcachedClient client, int timeout, boolean enableContextKeyTracking)	Creates a new instance with optional context key tracking.

Method Summary

Modifier and Type	Method	Description
boolean	create(Object value)	Creates a new record in the store using an annotated object as the source.
boolean	create(String context, String key, String value, Long expiration)	Creates a new record in the store with an expiration.
<T> boolean	create(String context, String key, T value, StorageSerializer<T> serializer, Long expiration)	Creates a new record in the store with an expiration, using a custom serialization process for an arbitrary object.
protected String	createNamespace(String context)	Creates a cache-wide unique namespace for the given context name.
boolean	delete(Object value)	Deletes an existing record from the store, using an annotated object as the source.
boolean	delete(String context, String key)	Deletes an existing record from the store.
void	deleteContext(String context)	Forcibly removes all records in a given context along with any associated resources devoted to maintaining the context.
boolean	deleteWithVersion(long version, Object value)	Deletes an existing record from the store, using an annotated object as the source, if it currently has a specified version.
boolean	deleteWithVersion(long version, String context, String key)	Deletes an existing record from the store if it currently has a specified version.
protected void	doDestroy()
StorageCapabilities	getCapabilities()	Returns the capabilities of the underlying store.
private <T> T	handleAsyncResult(net.spy.memcached.internal.OperationFuture<T> result)	Handle async result.
protected String	lookupNamespace(String context)	Looks up the namespace for the given context name in the cache.
private String	memcachedKey(String... parts)	Creates a memcached key from one or more parts.
Object	read(Object value)	Returns an existing record from the store, if one exists, and uses it to update the annotated fields of a target object.
<T> StorageRecord<T>	read(String context, String key)	Returns an existing record from the store, if one exists.
<T> Pair<Long,StorageRecord<T>>	read(String context, String key, long version)	Returns an existing record from the store, along with its version.
void	reap(String context)	Manually trigger a cleanup of expired records.
void	setCapabilities(MemcachedStorageCapabilities capabilities)	Sets the storage capabilities.
boolean	update(Object value)	Updates an existing record in the store, using an annotated object as the source.
boolean	update(String context, String key, String value, Long expiration)	Updates an existing record in the store.
<T> boolean	update(String context, String key, T value, StorageSerializer<T> serializer, Long expiration)	Updates an existing record in the store using a custom serialization strategy.
void	updateContextExpiration(String context, Long expiration)	Updates the expiration time of all records in the context.
private boolean	updateContextKeyList(String suffix, String namespace, String key)	Update context key list.
boolean	updateExpiration(Object value)	Updates expiration of an existing record in the store, using an annotated object as the source.
boolean	updateExpiration(String context, String key, Long expiration)	Updates expiration of an existing record in the store.
Long	updateWithVersion(long version, Object value)	Updates an existing record in the store, if a version matches, using an annotated object as the source.
Long	updateWithVersion(long version, String context, String key, String value, Long expiration)	Updates an existing record in the store, if a version matches.
<T> Long	updateWithVersion(long version, String context, String key, T value, StorageSerializer<T> serializer, Long expiration)	Updates an existing record in the store, if a version matches, using a custom serialization strategy.

Methods inherited from class net.shibboleth.utilities.java.support.component.AbstractIdentifiableInitializableComponent

setId

Methods inherited from class net.shibboleth.utilities.java.support.component.AbstractIdentifiedInitializableComponent

doInitialize, getId

Methods inherited from class net.shibboleth.utilities.java.support.component.AbstractInitializableComponent

destroy, initialize, isDestroyed, isInitialized

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.shibboleth.utilities.java.support.component.IdentifiedComponent

getId

Field Details

CTX_KEY_LIST_SUFFIX

@Nonnull
@NotEmpty
protected static final String CTX_KEY_LIST_SUFFIX

Key suffix for entry that contains a list of context keys.

See Also:

• Constant Field Values

CTX_KEY_DELETED_SUFFIX

@Nonnull
@NotEmpty
protected static final String CTX_KEY_DELETED_SUFFIX

Key suffix for entry that contains a list of deleted context keys.

See Also:

• Constant Field Values

CTX_KEY_LIST_DELIMITER

@Nonnull
@NotEmpty
private static final String CTX_KEY_LIST_DELIMITER

Delimiter of items in the context key list.

See Also:

• Constant Field Values

MAX_KEY_LENGTH

private static final int MAX_KEY_LENGTH

Maximum length in bytes of memcached keys.

See Also:

• Constant Field Values

logger

@Nonnull
private final org.slf4j.Logger logger

Logger instance.

storageRecordTranscoder

@Nonnull
private final net.spy.memcached.transcoders.Transcoder<MemcachedStorageRecord<?>> storageRecordTranscoder

Handles conversion of MemcachedStorageRecord to bytes and vice versa.

stringTranscoder

@Nonnull
private final net.spy.memcached.transcoders.Transcoder<String> stringTranscoder

Handles conversion of strings to bytes and vice versa.

storageCapabilities

@Nonnull
private MemcachedStorageCapabilities storageCapabilities

Invariant storage capabilities.

memcacheClient

@Nonnull
private final net.spy.memcached.MemcachedClient memcacheClient

Memcached client instance.

operationTimeout

@Positive
private int operationTimeout

Memcached asynchronous operation timeout in seconds.

trackContextKeys

private boolean trackContextKeys

Flag that controls context key tracking.

Constructor Details

MemcachedStorageService

public MemcachedStorageService(@Nonnull
 net.spy.memcached.MemcachedClient client,
 @Positive
 int timeout)

Creates a new instance.

Parameters:

client - Memcached client object. The client MUST be configured to use the binary memcached protocol, i.e. BinaryConnectionFactory, in order for deleteWithVersion(long, String, String) and deleteWithVersion(long, Object) to work correctly. The binary protocol is recommended for efficiency as well.

timeout - Memcached operation timeout in seconds.

MemcachedStorageService

public MemcachedStorageService(@Nonnull
 net.spy.memcached.MemcachedClient client,
 @Positive
 int timeout,
 boolean enableContextKeyTracking)

Creates a new instance with optional context key tracking.

Parameters:

client - Memcached client object. The client MUST be configured to use the binary memcached protocol, i.e. BinaryConnectionFactory, in order for deleteWithVersion(long, String, String) and deleteWithVersion(long, Object) to work correctly. The binary protocol is recommended for efficiency as well.

timeout - Memcached operation timeout in seconds.

enableContextKeyTracking - True to enable context key tracking, false otherwise. NOTE this flag must be set to true in order for updateContextExpiration(String, Long) to work. If that capability is not needed, the flag should be set to false for better performance. The feature is disabled by default.

Method Details

getCapabilities

@Nonnull
public StorageCapabilities getCapabilities()

Returns the capabilities of the underlying store.

Specified by:

getCapabilities in interface StorageService

Returns:

interface to access the service's capabilities

setCapabilities

public void setCapabilities(@Nonnull
 MemcachedStorageCapabilities capabilities)

Sets the storage capabilities. This method should be used when the default 1M slab size is changed; the MemcachedStorageCapabilities.getValueSize() should be set equal to the chosen slab size.

Parameters:

capabilities - Memcached storage capabilities.

create

public boolean create(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nonnull @NotEmpty
 String value,
 @Nullable @Positive
 Long expiration)
               throws IOException

Creates a new record in the store with an expiration.

Specified by:

create in interface StorageService

Parameters:

context - a storage context label

key - a key unique to context

value - value to store

expiration - expiration for record, or null

Returns:

true iff record was inserted, false iff a duplicate was found

Throws:

IOException - if fatal errors occur in the insertion process

create

public <T> boolean create(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nonnull
 T value,
 @Nonnull
 StorageSerializer<T> serializer,
 @Nullable @Positive
 Long expiration)
                   throws IOException

Creates a new record in the store with an expiration, using a custom serialization process for an arbitrary object.

Specified by:

create in interface StorageService

Type Parameters:

T - type of record

Parameters:

context - a storage context label

key - a key unique to context

value - object to store

serializer - custom serializer for the object

expiration - expiration for record, or null

Returns:

true iff record was inserted, false iff a duplicate was found

Throws:

IOException - if fatal errors occur in the insertion process

create

public boolean create(@Nonnull
 Object value)
               throws IOException

Creates a new record in the store using an annotated object as the source.

The individual parameters for the creation are extracted from the object using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

create in interface StorageService

Parameters:

value - object to store

Returns:

true iff record was inserted, false iff a duplicate was found

Throws:

IOException - if fatal errors occur in the insertion process

read

public <T> StorageRecord<T> read(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key)
                          throws IOException

Returns an existing record from the store, if one exists.

Specified by:

read in interface StorageService

Type Parameters:

T - type of record

Parameters:

context - a storage context label

key - a key unique to context

Returns:

the record read back, if present, or null

Throws:

IOException - if errors occur in the read process

read

public Object read(@Nonnull
 Object value)
            throws IOException

Returns an existing record from the store, if one exists, and uses it to update the annotated fields of a target object.

The context and key to look up are obtained from the target object, and the value and expiration are written back, using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

read in interface StorageService

Parameters:

value - object to look up and populate

Returns:

the updated object passed into the method, or null if no record was found

Throws:

IOException - if errors occur in the read process

read

public <T> Pair<Long,StorageRecord<T>> read(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Positive
 long version)
                                     throws IOException

Returns an existing record from the store, along with its version.

The first member of the pair returned will contain the version of the record in the store, or will be null if no record exists. The second member will contain the record read back. If null, the record either didn't exist (if the first member was also null) or the record was the same version as that supplied by the caller.

Specified by:

read in interface StorageService

Type Parameters:

T - type of record

Parameters:

context - a storage context label

key - a key unique to context

version - only return record if newer than supplied version

Returns:

a pair consisting of the version of the record read back, if any, and the record itself

Throws:

IOException - if errors occur in the read process

update

public boolean update(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nonnull @NotEmpty
 String value,
 @Nullable @Positive
 Long expiration)
               throws IOException

Updates an existing record in the store.

Specified by:

update in interface StorageService

Parameters:

context - a storage context label

key - a key unique to context

value - updated value

expiration - expiration for record, or null

Returns:

true if the update succeeded, false if the record does not exist

Throws:

IOException - if errors occur in the update process

update

public <T> boolean update(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nonnull
 T value,
 @Nonnull
 StorageSerializer<T> serializer,
 @Nullable @Positive
 Long expiration)
                   throws IOException

Updates an existing record in the store using a custom serialization strategy.

Specified by:

update in interface StorageService

Type Parameters:

T - type of record

Parameters:

context - a storage context label

key - a key unique to context

value - updated value

serializer - custom serializer

expiration - expiration for record, or null

Returns:

true if the update succeeded, false if the record does not exist

Throws:

IOException - if errors occur in the update process

update

public boolean update(@Nonnull
 Object value)
               throws IOException

Updates an existing record in the store, using an annotated object as the source.

The individual parameters for the update are extracted from the object using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

update in interface StorageService

Parameters:

value - object to update from

Returns:

true if the update succeeded, false if the record does not exist

Throws:

IOException - if errors occur in the update process

updateWithVersion

public Long updateWithVersion(@Positive
 long version,
 @Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nonnull @NotEmpty
 String value,
 @Nullable @Positive
 Long expiration)
                       throws IOException,
VersionMismatchException

Updates an existing record in the store, if a version matches.

Specified by:

updateWithVersion in interface StorageService

Parameters:

version - only update if the current version matches this value

context - a storage context label

key - a key unique to context

value - updated value

expiration - expiration for record, or null

Returns:

the version of the record after update, null if no record exists

Throws:

IOException - if errors occur in the update process

VersionMismatchException - if the record has already been updated to a newer version

updateWithVersion

@Nullable
public <T> Long updateWithVersion(@Positive
 long version,
 @Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nonnull
 T value,
 @Nonnull
 StorageSerializer<T> serializer,
 @Nullable @Positive
 Long expiration)
                           throws IOException,
VersionMismatchException

Updates an existing record in the store, if a version matches, using a custom serialization strategy.

Specified by:

updateWithVersion in interface StorageService

Type Parameters:

T - type of record

Parameters:

version - only update if the current version matches this value

context - a storage context label

key - a key unique to context

value - updated value

serializer - custom serializer

expiration - expiration for record, or null

Returns:

the version of the record after update, null if no record exists

Throws:

IOException - if errors occur in the update process

VersionMismatchException - if the record has already been updated to a newer version

updateWithVersion

@Nullable
public Long updateWithVersion(@Positive
 long version,
 @Nonnull
 Object value)
                       throws IOException,
VersionMismatchException

Updates an existing record in the store, if a version matches, using an annotated object as the source.

The individual parameters for the update are extracted from the object using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

updateWithVersion in interface StorageService

Parameters:

version - only update if the current version matches this value

value - object to update from

Returns:

the version of the record after update, null if no record exists

Throws:

IOException - if errors occur in the update process

VersionMismatchException - if the record has already been updated to a newer version

updateExpiration

public boolean updateExpiration(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key,
 @Nullable @Positive
 Long expiration)
                         throws IOException

Updates expiration of an existing record in the store.

Specified by:

updateExpiration in interface StorageService

Parameters:

context - a storage context label

key - a key unique to context

expiration - expiration for record, or null

Returns:

true if the update succeeded, false if the record does not exist

Throws:

IOException - if errors occur in the update process

updateExpiration

public boolean updateExpiration(@Nonnull
 Object value)
                         throws IOException

Updates expiration of an existing record in the store, using an annotated object as the source.

The individual parameters for the update are extracted from the object using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

updateExpiration in interface StorageService

Parameters:

value - object to update from

Returns:

true if the update succeeded, false if the record does not exist

Throws:

IOException - if errors occur in the update process

delete

public boolean delete(@Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key)
               throws IOException

Deletes an existing record from the store.

Specified by:

delete in interface StorageService

Parameters:

context - a storage context label

key - a key unique to context

Returns:

true iff the record existed and was deleted

Throws:

IOException - if errors occur in the deletion process

delete

public boolean delete(@Nonnull
 Object value)
               throws IOException

Deletes an existing record from the store, using an annotated object as the source.

The individual parameters for the deletion are extracted from the object using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

delete in interface StorageService

Parameters:

value - object to delete

Returns:

true iff the record existed and was deleted

Throws:

IOException - if errors occur in the deletion process

deleteWithVersion

public boolean deleteWithVersion(@Positive
 long version,
 @Nonnull @NotEmpty
 String context,
 @Nonnull @NotEmpty
 String key)
                          throws IOException,
VersionMismatchException

Deletes an existing record from the store if it currently has a specified version.

Specified by:

deleteWithVersion in interface StorageService

Parameters:

version - record version to delete

context - a storage context label

key - a key unique to context

Returns:

true iff the record existed and was deleted

Throws:

IOException - if errors occur in the deletion process

VersionMismatchException - if the record has already been updated to a newer version

deleteWithVersion

public boolean deleteWithVersion(@Positive
 long version,
 @Nonnull
 Object value)
                          throws IOException,
VersionMismatchException

Deletes an existing record from the store, using an annotated object as the source, if it currently has a specified version.

The individual parameters for the deletion are extracted from the object using the annotations in the org.opensaml.storage.annotation package. If any are missing, or a field inaccessible, a runtime exception of some kind will occur.

Specified by:

deleteWithVersion in interface StorageService

Parameters:

version - record version to delete

value - object to delete

Returns:

true iff the record existed and was deleted

Throws:

IOException - if errors occur in the deletion process

VersionMismatchException - if the record has already been updated to a newer version

reap

public void reap(@Nonnull @NotEmpty
 String context)
          throws IOException

Manually trigger a cleanup of expired records. The method MAY return without guaranteeing that cleanup has already occurred.

Specified by:

reap in interface StorageService

Parameters:

context - a storage context label

Throws:

IOException - if errors occur in the cleanup process

updateContextExpiration

public void updateContextExpiration(@Nonnull @NotEmpty
 String context,
 @Nullable
 Long expiration)
                             throws IOException

Updates the expiration time of all records in the context.

Specified by:

updateContextExpiration in interface StorageService

Parameters:

context - a storage context label

expiration - a new expiration timestamp, or null

Throws:

IOException - if errors occur in the cleanup process

deleteContext

public void deleteContext(@Nonnull @NotEmpty
 String context)
                   throws IOException

Forcibly removes all records in a given context along with any associated resources devoted to maintaining the context.

Specified by:

deleteContext in interface StorageService

Parameters:

context - a storage context label

Throws:

IOException - if errors occur in the cleanup process

doDestroy

protected void doDestroy()

Overrides:

doDestroy in class AbstractInitializableComponent

lookupNamespace

protected String lookupNamespace(String context)
                          throws IOException

Looks up the namespace for the given context name in the cache.

Parameters:

context - Context name.

Returns:

Corresponding namespace for given context or null if no namespace exists for context.

Throws:

IOException - On memcached operation errors.

createNamespace

protected String createNamespace(String context)
                          throws IOException

Creates a cache-wide unique namespace for the given context name. The context-namespace mapping is stored in the cache.

Parameters:

context - Context name.

Returns:

Namespace name for given context.

Throws:

IOException - On memcached operation errors.

memcachedKey

private String memcachedKey(String... parts)

Creates a memcached key from one or more parts.

Parameters:

parts - Key parts (i.e. namespace, local name)

Returns:

Key comprised of 250 characters or less.

handleAsyncResult

private <T> T handleAsyncResult(net.spy.memcached.internal.OperationFuture<T> result)
                         throws IOException

Handle async result.

Type Parameters:

T - type of result

Parameters:

result - the result

Returns:

the result

Throws:

IOException - if an error occurs

updateContextKeyList

private boolean updateContextKeyList(String suffix,
 String namespace,
 String key)
                              throws IOException

Update context key list.

Parameters:

suffix - the suffix

namespace - the namespace

key - the storage key

Returns:

whether the update was a success

Throws:

IOException - if an error occurs