SP metadata cache keeps growing

[Jon Agland]

On Thu, 2020-02-27 at 22:35 +0100, Peter Schober wrote:
> * Cantor, Scott
> <cantor.2 at osu.edu> [2020-02-27 20:49]:
> > 
> > I have not seen anything resembling that bad a result, so that means
> > something's probably really wrong. Maybe the signature step is that
> > bad now, but I didn't think so. I know the parsing isn't.
> With signature valdidation:
> 
> # time systemctl start shibd
> real    76m27.798s
> 
> 2020-02-27 19:39:10 INFO OpenSAML.MetadataProvider : applying metadata filter (Signature)
> 2020-02-27 20:55:30 INFO OpenSAML.MetadataProvider : applying metadata filter (EntityRoleWhiteList)
> 
> Without:
> 
> # time systemctl start shibd                                                                            
> real    0m6.974s
> 
> 2020-02-27 21:29:33 INFO OpenSAML.MetadataProvider : applying metadata filter (RequireValidUntil)
> 2020-02-27 21:29:33 INFO OpenSAML.MetadataProvider : applying metadata filter (EntityRoleWhiteList)
> 
Within the UK federation, we've similar seen growth in the size of our metadata aggregate publication, to about ~53.5MB.  We've had a few adhoc reports, but nothing really concrete, so we've been using our metadata publication logs to help us a little, as the consumption/traffic levels have also gone up a little too.  We've pro-actively contacted a handful of organisations whose SPs we can identify (this is fairly standard procedure for us to contact those using our metadata publication excessively).  I guess we may get a few more calls when the metadata the SPs are running with expires/hits the validUntil, or when the service needs to restart?
A colleague has also tested two versions of Shibboleth SP (3.0.4 and 2.5.3 ) with the metadata consumption and signature verification on.  The reason for this was that the adhoc reports did seem to indicate slightly older versions were affected.    We've observed that 3.0.4 took ~17 minutes, whilst 2.5.3 took ~35-37 minutes, this was done on the same machine/hardware.
In summary we think it will be more acute as your technical debt increases, older versions of the SP, older hardware CPUs/VMs with less memory, or hosts that are swap bound onto slow storage.
Our advise will be to keep up-to-date and to move MDQ where you can (e.g.for the UK federation participants read https://www.ukfederation.org.uk/content/Documents/MDQ )
Cheers,
Jon
Jon Agland
Technical Services Manager - Trust and Identity
Jisc
T 02038198207
M 07443984222
Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG

https://www.jisc.ac.uk/trust-and-identity
https://www.ukfederation.org.uk
 
Jisc is a registered charity (number 1149740) and a company limited by
g
uarantee which is registered in England under company number.
05747339,
VAT number GB 197 0632 86.  Jisc’s registered office is: 4
Portwall
Lane, Bristol, BS1 6NB. T 0203 697 5800.

Jisc Services Limited is a wholly owned Jisc subsidiary and a company
limited by guarantee which is registered in England under company
number 02881024, VAT number GB 197 0632 86. The registered office is: 4
Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800. 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20200303/159cd4cb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5812 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/users/attachments/20200303/159cd4cb/attachment.bin>