StorageService API

[Cantor, Scott]

 > The StorageService API is of interest because it will be a core API
> component needed for the CAS protocol (ticket storage). I've been
> studying the component and considering how it would be used in my case,
> and more generally to meet other needs. While I can imagine how I would
> use it in my case, it would be more awkward than I'd like.

Can you look at the replay cache implementation and try and explain why you think it's awkward there? What is it about your desired semantics that make it harder than that? Actually, the comparison would probably be to the SAML artifact map, but the Java on that isn't done. I do have one in C++ that uses this API, and it is a very simple piece of c ode.

My experience with this API is that simple things are simple and hard things are possible, which is pretty much the sweet spot for me.

> I struggled with the notion of exposing two different storage formats
> (string/text) for all operations.

Some use cases need storage of small amounts of data, and others need large amounts, and this allows the back-end to optimize for both separately. Most use cases use one or the other, and it's of no consequence, they just use the set that matter to their use case. Back-ends that don't gain anything from the difference have a trivial amount of work to forward one set of calls to the other.

> I'm fairly certain I don't understand
> the use cases where that feature would be required or even desirable,
> but for simple cases like mine it's an obstacle.

I don't understand why it's an obstacle, that isn't clear to me.
 
> If you must have both
> methods, it would be nice to expose a simple create/read/delete
> interface that delegates through internal logic, i.e.

You may be correct that in Java this makes more sense. In C, computing the length of a null-terminated string is an O(N) operation. If it's O(1) in Java (I have no idea offhand), then it would be reasonable to collapse them and do this. It is definitely not in my native tongue.

> Choosing a context name for every call seems verbose. I can imagine most
> components working with a StorageService would use one or at most a few
> context names.

More than you might think need more than one, and a session cache layered on this actually is easier to implement by using a session key as a context. A lot use one, yes.
 
> The StorageServiceView would expose the same contract as StorageService,
> but without the context name parameter. It's more or less syntactic
> sugar, but valuable in my mind.

I don't see the trade off being worth it if that's the only gain, but it's certainly easy to do. That's what I meant by just wrapping this in the existing PersistenceManager.

I think that API is fine as a wrapper. I don't think it's fine as a core storage API because it assumes magic we have no code for (object peristence).

> It may be helpful if not necessary to provide a registry for context names.

Certainly doable, but in practice it's not been a major concern because anything that's not core code ought to use package-syle or URL-style contexts. Have to think about whether it's worthwhile to add a registry.

> Going beyond the CAS use case, I had a concern about the use of String
> as the storage format in all cases. Thinking about RDBMS and LDAP there
> seems like a fair bit of impedance mismatch.

I'm not following that. Databases want strings, that's what they handle well. Doing something else means adding technology, which means more testing, more instability from broken libraries, etc. That's not worth it to me if the alternative is just some extra code that's easy to write.

Basically, I would be terrified of using Hibernate, to use one obvious example. My trust issues with that sort of code are likely to be different than some I imagine.
 
> I'm most familiar with the
> RDBMS case, but I believe the concerns apply to LDAP as well. In the
> RDBMS case it would be most natural to persist an annotated type where
> the context name (@Table), key (@Id), version (@Version), and possibly
> expiration are derived from fields on the object.

If somebody wants to write and test all that, they can, but it can't be me (I don't know any of that technology), so my solution was to port a working system. That was the goal I had.

To do an object-based store is not at all simpler for me to do, it's much, much more work. It's easier for the callers of the API, but the amount of code being saved there is not large in amount or complexity. So it was a conscious trade-off.
 
> While you could  decompose an object into those fields before calling
> StorageService#create, you would lose the expressive power of modern
> Java ORM tools. You'd also incur BLOB/CLOB storage which has been a
> substantial headache for us in the CAS project.

Yes, it assumes BLOB (or more precisely text) storage in a database. Could you elaborate about why  that is a headache? I can't see how requiring custom table schema in a database could be seen as a better option, which is the only alternative to object storage I'm aware of, but I need to understand your experience here.
 
My approach has worked with a single table layout through 5 major releases of the SP. Using custom tables with some kind of ORM mapping would have broken on every release and required scripts for upgrades. So I think I must not be following your thinking.

> In summary, the PersistenceManager interface is much more like what I
> would want from a general persistence/storage API component.

It's what I would want too, if we had the persistence code to back it up. We had an API, but not working code.
 
> Perhaps  there's room for both; for example the StorageService provides a
> cache-like interface and PersistenceManager for more general object
> persistence. I wish I had some suggestions to consolidate the two
> components into a best of both, but I simply don't see it.

Why is layering one on the other a problem? If you have a solution for turning an object into a string, then that's all you have to add to make that work.
 
There are a lot of use cases for true persistence that are not a good fit with the storage API. But we have nobody on the project working on building a generic object persistence facility. So my assumption has been that where there are such cases, they'll be coded differently. I don't know how many of those exist yet; the one that I do know about is the plugin for directed identifiers.

I think you could fairly characterize mine as a cache API, but it's not exclusively usable only for things that store data temporarily and I think it will work fine for the consent engine and some other areas. I *think* that it would work fine for the CAS tickets because SAML artifacts are not a lot different.

I'm not trying to be difficult, really, but right now we have a lot of stuff to build, and I'm really trying to avoid true object persistence being one of those things, unless we have a clear offer to build it.

Maybe the background on this is missing, but we have never had that capability in any of our releases. So this is not a trivial change to make.

-- Scott