StorageService API

[Cantor, Scott]

On 5/6/13 4:42 PM, "Marvin S. Addison" <marvin.addison at gmail.com> wrote:

>> Can you look at the replay cache implementation and try and explain
>> why you think it's awkward there?
>
>ReplayCache is a fairly different use case; there it's natural to deal
>with strings and multiple contexts.

Yes, the artifact map OTOH is an exact parallel.

>Not a problem, just more work than I'd like.

I guess you would see it as equivalent in that it's more work, but I would
not expect your CAS code proper to be calling that API, just as the SAML
artifact profile code doesn't do so. I would have a TicketCache or
whatever name is appropriate that has exactly the API you suggest, and
wraps the storage API.

My goal is not the smallest amount of code, but just getting the code in
the right place and separating concerns.

>I was unaware that use of URIs was recommended, and as long as that's
>documented clearly it should be sufficient.

Its origins in an environment where nobody else would ever extend it means
I never bothered to note that detail before

>The problems are very much a function of the database platform, but I've
>had experience with PostgreSQL, Oracle, and MySQL (in descending order).

Just for reference, mine is mainly SQL Server, Oracle, and MySQL.

> 
>On PostgreSQL we had problems with orphaned large objects; one
>transaction would delete the object and for some reason other
>transactions would not get notice and the others would error out. It's a
>recoverable error, but not a very graceful one. Generally the
>maintenance around large objects was a pain on PG.

Hmm. I should note something that may be relevant here: I do *not* use any
transaction isolation level other than SERIALIZABLE. Which I consider to
be the only real one, the others are game playing. If I wanted to do my
own locking, I'd use Berkeley.

But I have no real experience with PG other than for Jira and Confluence,
and I can't say those have been any trouble.

>The second class of issues is deadlocks, and we have only suggestive
>data that those were exacerbated by LOB storage. I would be more
>confident in saying that a mixture of OLTP workload (heavy read/write
>concurrency), foreign keys, and large objects tends to exacerbate
>deadlocks. The severity of the problems varied dramatically with
>platform; the rate of deadlocks on PG was orders of magnitude higher
>than Oracle.

That's alarming, I can't think why any properly written code would ever
deadlock unless the database driver was just broken. Oracle unfortunately
throws dirty write errors because of its default locking (or lack
thereof), but those aren't deadlocks.

This is with the CAS ticket cache, or something else? Just wondering about
the workload.

>The last issue is a system issue of capacity planning and platform
>limits. LOB data types do not declare storage limits like other database
>data types, so it's much harder to measure and estimate storage. I
>believe most platforms have tools for tracking down LOB data, but in
>many cases it's non trivial, and the application folks troubleshooting
>database issues tended not to be DBAs in my experience.

That would be a bigger concern for some of the newer use cases that I have
less experience with. The basic APIs (CAS ticket cache included) would be
self-cleaning and not generally build up much disk usage.

>Totally understand. With tools like Hibernate it's not terribly
>difficult to provide this for RDBMS backend, but it is work. I guess
>it's just a matter of the extent to which you want to support all
>possible backends with a single API.

That probably needed to be explicit: I am *totally* trying to propose a
single API (for the components that can use such an API), to the exclusion
of just about any other consideration.

> It's perfectly reasonable to say database is an edge case and the
>existing API will support it in some manner even if it isn't the most
>natural.

I wouldn't say it's an edge case, though our experience is that people
just don't want to deploy databases (and it really raises the bar for
people). I would say that LDAP is an edge case for me, but that isn't to
say I have the only vote on that.

I think there are a couple of things I can do once I have some time:

Obviously two minutes of checking and I see that String.length() is O(1),
so you're correct that there is no reason whatsoever given what's there
now to have separate methods for short/long data. That's a carryover from
C++ I can drop.

Secondly, I'll try and look at building some support for automating or
simplifying some of the simple wrapping required to store objects that
have a natural mapping of fields to keys, values, expirations and such.
Maybe just stuff in a new interface that our own objects that want to be
directly storable can implement.

-- Scott