outage due to not downloading incommon metadata

[Hyzer, Chris]

We had a partial outage with our SPs.

The error was "Unknown or Unusable Identity Provider"

It is supposed to periodically download the incommon idp only metadata

            <MetadataProvider type="XML" uri="http://md.incommon.org/InCommon/InCommon-metadata-idp-only.xml"
                 backingFilePath="InCommon-metadata-idp-only.xml" reloadInterval="7200">
               <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
               <MetadataFilter type="Signature" certificate="incommon.pem"/>
            </MetadataProvider>

But the files on our server s are not consistent, and some are old.  The file on the affected server was last downloaded in dec, and the 4 week interval expired and the SP stopped logging people in.  You can see in some of our nodes (which are configured the same), it is working correctly, and some need to be updated.

ls -altr /var/cache/shibboleth/InCommon-metadata-idp-only.xml
Server1: -rw-r--r--. 1 shibd shibd 20748191 Dec 21 00:59 /var/cache/shibboleth/InCommon-metadata-idp-only.xml
Server2: -rw-r--r--. 1 shibd shibd 20738297 Jan  3 15:18 /var/cache/shibboleth/InCommon-metadata-idp-only.xml
Server3 -rw-r--r--. 1 shibd shibd 20738297 Jan  3 15:17 /var/cache/shibboleth/InCommon-metadata-idp-only.xml
Server4: -rw-r--r--. 1 shibd shibd 20738297 Jan  3 15:16 /var/cache/shibboleth/InCommon-metadata-idp-only.xml
Server5: -rw-r--r--. 1 shibd shibd 20748191 Dec 21 00:59 /var/cache/shibboleth/InCommon-metadata-idp-only.xml

QUESTION 1: Any idea why some servers are downloading and some aren't?
QUESTION 2: Should I be bouncing shibd weekly?

QUESTION 3: I didn't have the /Status handler hooked up, so it took a user report to find out about this.  Would the /Status return a 500 and Nagios would tell me or do I need to look at the results for an <OK/>?

Bouncing the SP fixed this issue for now.

<Version Shibboleth="2.6.0"/>
<NonWindows sysname="Linux" machine="x86_64"/>

QUESTION 4: This was logged as a WARN, should it be an error?

2018-01-04 11:17:50 WARN Shibboleth.SessionInitiator.SAML2 [6]: unable to locate metadata for provider (https://idp.pennkey.upenn.ed
u/idp/shibboleth)
2018-01-04 11:17:53 WARN OpenSAML.MetadataProvider [7]: ignored expired metadata instance for (https://idp.pennkey.upenn.edu/idp/shi
bboleth)

[root at fastprod-small-b-01 shibboleth]# grep -i error *
[root at fastprod-small-b-01 shibboleth]# grep -i fatal *
[root at fastprod-small-b-01 shibboleth]# pwd
/var/log/shibboleth
[root at fastprod-small-b-01 shibboleth]#

https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPLogging

"WARN: Indicates something that's noteworthy, but not an actual error.
ERROR: Something's wrong. Users are likely to be affected any time one appears."

Thanks!
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20180104/fb289c9f/attachment.html>