SP metadata cache keeps growing

[Cantor, Scott]

On 3/3/20, 10:05 AM, "users on behalf of Jon Agland" <users-bounces at shibboleth.net on behalf of Jon.Agland at jisc.ac.uk> wrote:

> A colleague has also tested two versions of Shibboleth SP (3.0.4 and 2.5.3 ) with the metadata consumption and
> signature verification on.  The reason for this was that the adhoc reports did seem to indicate slightly older versions
> were affected.    We've observed that 3.0.4 took ~17 minutes, whilst 2.5.3 took ~35-37 minutes, this was done on the
> same machine/hardware.

That's surprising. My guess is Xerces is the difference, but short of maybe the 2.5 SP running on Xerces 2 perhaps (which would be horrendously old), I can't explain the difference.

I wouldn't have expected Xerces 3 to make much difference, but that's hard to go back and remember. Oddly, the very latest Xerces actually has a change that has caused people to bitch that it's slower, though I think that's more for high performance use on small messages.

Anyway, either result is ridiculous, and it's just officially unusable now regardless. It's surprising Java's as good as it is, but the signature code there has a much better canonicalizer and the parser doesn't allocate memory like it's being paid by the block. I do wonder if the Xerces allocator might be adjusted to stop exponentializing. If I tried a linear allocation growth curve once it reaches a certain block size, it might be interesting to test that. I don't think it would be that hard to try out.

-- Scott