Occasional problem with InCommon metadata refreshes

[Brian Koehmstedt]

After we recently migrated to a new machine for our Shibboleth IdP 
(2.4), we're encountering some occasional odd behavior relating to the 
InCommon metadata loading.

About every two weeks, in the early morning, the following happens:

- Looking at the idp-process log, things look normal in regards to 
periodically loading metadata:
03:27:31.623 - INFO 
[org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider:271] 
- Next refresh cycle for metadata provider 
'/local/users/shib/cvs/shibboleth/conf/incommon/InCommon-metadata.xml' 
will occur on '2014-06-26T10:32:31.623Z' 
('2014-06-26T03:32:31.623-07:00' local time)

- But then we start seeing things like this for any InCommon service:
03:30:55.488 - WARN 
[edu.internet2.middleware.shibboleth.idp.profile.AbstractSAMLProfileHandler:305] 
- No metadata for relying party https://services.box.com/sp, treating 
party as anonymous

We have other non-Incommon services that operate just fine, so clearly 
something is going wrong with the refresh of the InCommon metadata from 
disk into memory.

When this happens, Shib IdP appears to me to act like it has loaded a 
0-byte InCommon metadata file, where it doesn't find any InCommon SP 
metadata.

Also, this never corrects itself without restarting Tomcat.  In other 
words, even though we continue to see in the log file that the refresh 
is happening at a periodic interval,  IdP continuously acts like it's 
loading 0 bytes from the file, until Tomcat is restarted.

Here's the metadata provider setup:

         <metadata:MetadataProvider id="IdPMD" 
xsi:type="metadata:FilesystemMetadataProvider"
metadataFile="/local/users/shib/cvs/shibboleth/conf/incommon/InCommon-metadata.xml"
                                    maxRefreshDelay="PT15M" />

And we have a crontab that runs at 1am every night that downloads the 
metadata file to a temp location and once the download is complete, 
moves it to 
/local/users/shib/cvs/shibboleth/conf/incommon/InCommon-metadata.xml.