en passant: is IndexingObjectStore safe?

[Ian Young]

On 17 Jan 2014, at 04:22, Brent Putman <putmanb at georgetown.edu> wrote:

> On 1/15/14 11:49 PM, Cantor, Scott wrote:
>> Is it optimizing the string storage, or the actual Cert/CRL object
>> storage? If that latter, any chance they implemented equals() ?
> 
> It's optimizing the string storage.  These XMLObject impls don't know
> anything about or use the Java cert and CRL classes.  The XMLObjects
> faithfully model the XML schema, so xs:base64Binary.

I know Chad put a lot of work into getting the amount of space the IdP used down at one point, prompted by observations about how much heap the UKf metadata took up. I'd observe that the UKf metadata has a lot of shared certificates, most of which (500 or so) come from one commercial multi-tenant IdP, so it's likely this optimisation is directly related to that.

I'd be interested to know what proportion of the overall improvement came from this particular optimisation, but I don't suppose there is a way to find out without actually compiling up a new version of the IdP and trying it. If it's not a large proportion, then one option is to just rip it out.

I'd be very reluctant, though, to give up a lot of the earlier gains. If that was the position, then the obvious alternative would be to build a better version of IndexingObjectStore which was more careful about storing things, and used an internally generated sequence number instead of the hashCode() for stored objects. I don't think that would be hard to do.

If we don't see a need for a generic class of this kind, and are really just interested in the IndexedObjectStore<String> case, or even the generic case where the objects stored are immutable, then this could turn into a simple object pool where the object itself (or the previously stored version) is returned instead of an index. We could retain the reference counting mechanism, or use weak references instead to simplify the API a bit.

If we were going in that direction, though, it seems to me that for the <String> case we are pretty much just replicating the functionality of String.intern(). One interesting thing about Java 7 (which we now require) is that at least the Oracle JVM changed the implementation of intern() to save things in the conventional heap rather than PermGen space, and allow it to be garbage collected. So at least in principle, we might be able to replace this whole mechanism with intern(), if we knew that all target JVMs did similar things for intern().

I found this blog post:

http://java-performance.info/string-intern-in-java-6-7-8/

He covers all of this in some detail. It seems like the main issue in the Oracle Java 7 implementation is that the default size of the intern() table is a bit small, and this might affect performance in our case. intern() is based on a fixed-sized array accessed in O(1) (from indexed by the object's hashCode() modulo the table size), but if you have multiple strings in each hash bucket you need to search that chain linearly. So at small N, intern() is O(1), but with large N it becomes O(N) which is probably not good. I don't know where exactly our use case would fall without doing a few more sums.

	-- Ian



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5943 bytes
Desc: not available
Url : http://shibboleth.net/pipermail/dev/attachments/20140117/4a91d7de/attachment.bin