shibd unable to verify signature when metadata is cached

[Bradley Beddoes]

Hello,
I was hoping to get some assistance with an issue we've run into at the AAF
with shibd.

We've recently switched the SP exhibiting the fault described below to our
new metadata source so I am certainly of the opinion that either this new
metadata source or our SP configuration, is going to be the root cause here
but after exhausting all the diagnostic avenues I can think of I was hoping
someone else may be able to point me in a direction I've not considered.

The problem we're seeing occurs when our shibd instance is restarted. When
the upstream metadata document matches what has been previously cached
locally, signature verification fails, taking down the service. Signature
verification does not fail and the service functions correctly, when the
upstream metadata has changed and is re-downloaded after restart.

I've gathered a reasonable amount of logging/detail about this and instead
of cramming it all into an unreadable form in this email I've collated it
at https://gist.github.com/bradleybeddoes/c3b3d1cef20d60c577f3

This gist shows:

0: is simply informational, our local versions etc
1: shows the log output on an erroneous shibd restart
2: shows the log output on a successful restart where I've removed the
cached metadata and tag files from local disk prior, forcing download (or
upstream has legitimately changed - I've checked with both scenarios, same
logging).
3: our source metadata document (Full XML at
https://md.aaf.edu.au/aaf-metadata.xml)
4: the cached metadata document from /var/cache/shibboleth - I do note
there are minor differences
5: xmlsectool output, noting that for the 'same' document, based on
EntitiesDescriptor ID, we're getting a failed digest calculation from the
version recovered from /var/cache/shibboleth.

Thus far I've:

   - Been able to recreate the behavior described above at random times
   over a few days;
   - Looked extensively at our distribution endpoints for something invalid
   coming back following request for the source document,
   but I can't see any anomalies at that layer at this time;
   - Looked into our historical metadata source document cache and
   successfully verified them with xmlsectool;
   - Confirmed that a 2.4.x IdP on the same server has not exhibited any
   errors in metadata loading/verification;
   - Searched this list and the issue tracker for something similar to no
   avail.

Thanks in advance for any help.

cheers,
Bradley
-- 
*Bradley Beddoes* | Technical Lead - Innovation, Software Development and
Infrastructure
*Australian Access Federation Inc*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20160321/90e513fb/attachment.html>