SP metadata cache keeps growing

[Peter Schober]

Hey,

Someone from our community asked about an ever-growing cache directory
filling the local disk with copies of remote metadata. Something like:

-rw-r--r-- 1 shibd shibd 52M 26. Feb 08:57 aconet-metadata.xml
-rw-r--r-- 1 shibd shibd 52M 25. Feb 11:09 aconet-metadata.xml.0194
-rw-r--r-- 1 shibd shibd 52M 25. Feb 09:45 aconet-metadata.xml.0d83
... more of the same here ...
-rw-r--r-- 1 shibd shibd 52M 25. Feb 10:43 aconet-metadata.xml.fcf0
-rw-r--r-- 1 shibd shibd 52M 25. Feb 10:50 aconet-metadata.xml.fe78
-rw-r--r-- 1 shibd shibd  44 26. Feb 10:22 aconet-metadata.xml.tag

Amassing 2GB so far and growing.

(It might well be that they restarted shibd dozens of times while
trying to get something else to work, so those timestamps are possibly
not related to automatic refresh runs.)

They also claimed that restarting shibd took too long to finish
according to some service manager timeout (I'm thinking the old
SHIBD_WAIT thing from the SysV starup scripts, but that doesn't match
the OS being CentOS 8, which means systemd as service manager) but
that stopping and starting shibd in seperate steps only took a few
seconds -- in both cases loading metadata the size of above
(containing all of eduGAIN) from the same configuration[1].

The configured MetadataProvider uses a Signature MetadataFilter but
with verifyBackup="false" set, so I can't explain the slow (re-)start
other speculating that this is related to the cache issue, e.g. the
fact that there are so many backup copies means they're not found and
hence (re-)created at start, explaining the slow startup times.  But
why that would only manifest at restarts and not start && stop I have
no idea. (Bear in mind that maybe the issue report to me was confused
and/or maybe I caused some additional loss in translation.)

I didn't find anything in the archives and nothing in Jira about
amassing metadata backup copies, using a search like:
project in (CPPOST, SSPCPP, CPPXT) AND text ~ "metadata" AND text ~ "cache"

I'll try to get a look at the logs but thought I'd ask here in case
this sounds familiar to anyone here.

Cheers,
-peter

[1] The metadata provider used should be a verbatim copy of the second
    example from our docs at
    https://wiki.univie.ac.at/display/federation/Shibboleth+SP+3