[JIRA] (SSPCPP-973) shibd re-downloading metadata on every restart

[peter (Jira)]

peter ( https://shibboleth.atlassian.net/secure/ViewProfile.jspa?accountId=60ad33f05dc18500704c2ef8 ) *created* an issue

Shibboleth SP - C++ ( https://shibboleth.atlassian.net/browse/SSPCPP?atlOrigin=eyJpIjoiN2Q0OWQ3ZTFlYjMwNGMwZWFmODU4OGIzN2IyMzJlZmQiLCJwIjoiaiJ9 ) / Bug ( https://shibboleth.atlassian.net/browse/SSPCPP-973?atlOrigin=eyJpIjoiN2Q0OWQ3ZTFlYjMwNGMwZWFmODU4OGIzN2IyMzJlZmQiLCJwIjoiaiJ9 ) SSPCPP-973 ( https://shibboleth.atlassian.net/browse/SSPCPP-973?atlOrigin=eyJpIjoiN2Q0OWQ3ZTFlYjMwNGMwZWFmODU4OGIzN2IyMzJlZmQiLCJwIjoiaiJ9 ) shibd re-downloading metadata on every restart ( https://shibboleth.atlassian.net/browse/SSPCPP-973?atlOrigin=eyJpIjoiN2Q0OWQ3ZTFlYjMwNGMwZWFmODU4OGIzN2IyMzJlZmQiLCJwIjoiaiJ9 )

Issue Type: Bug Affects Versions: 3.2.2, 3.4.1 Assignee: Scott Cantor ( https://shibboleth.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A5b78efc9-1379-42cc-a3f6-56c6ea3a0007 ) Created: 24/Mar/23 2:01 PM Priority: Trivial Reporter: peter ( https://shibboleth.atlassian.net/secure/ViewProfile.jspa?accountId=60ad33f05dc18500704c2ef8 )

I noticed that since we changed our metadata hosting shibd (initially v3.2.2 using libsaml 3.2.0, tested on Debian 11; then replicated with v3.4.1 from official packages using libsaml 3.2.1, tested on Rocky 9) will re-download metadata from a MetadataProvider/@type="XML" on every restart and will also re-apply a MetadataFilter/@type="Signature" every time – defeating the purpose of MetadataFilter/@verifyBackup="false".

Scott identified the missing . tag file as the reason for that and seemingly a different behaviour of different web servers or server configurations providing these documents for download.

<!-- works as expected -->
<MetadataProvider type= "XML" url= "http://md.incommon.org/InCommon/InCommon-metadata.xml" backingFilePath= "incommon-metadata.xml" >
   <MetadataFilter type= "Signature" certificate= "inc-md-cert.pem" verifyBackup= "false" />
</MetadataProvider>
<!-- fails to write .tag file -->
<MetadataProvider type= "XML" url= "http://eduid.at/md/aconet-interfed.xml" backingFilePath= "aconet-metadata.xml" >
   <MetadataFilter type= "Signature" certificate= "aconet-metadata-signing.crt" verifyBackup= "false" />
</MetadataProvider>

(Referenced certificate files can be found here https://ds.incommon.org/certs/inc-md-cert.pem ( https://ds.incommon.org/certs/inc-md-cert.pem ) and here https://eduid.at/md/aconet-metadata-signing.crt ( https://eduid.at/md/aconet-metadata-signing.crt ) respectively, not that this matters.)

$ ls /var/cache/shibboleth/
aconet-metadata.xml  incommon-metadata.xml  incommon-metadata.xml.tag

Eyeballing the respective HTTP Response Headers they seem OK but something must be off in the case of eduid.at. cURL used on the cli as below uses HTTP/2 for eduid and HTTP/1.1 for InC, for example:

$ curl -sSI http://md.incommon.org/InCommon/InCommon-metadata.xml 
HTTP/1.1 200 OK
Date: Fri, 24 Mar 2023 17:44:35 GMT
Server: Apache
Last-Modified: Thu, 23 Mar 2023 19:24:55 GMT
ETag: "5c536c2-5f7963b4d4653"
Accept-Ranges: bytes
Content-Length: 96810690
Content-Type: application/samlmetadata+xml

$ curl -sSI https://eduid.at/md/aconet-interfed.xml
HTTP/2 200 
strict-transport-security: max-age=15768000
last-modified: Fri, 24 Mar 2023 07:12:20 GMT
etag: "4b1e931-5f7a01d2efd00"
accept-ranges: bytes
content-length: 78768433
content-type: application/samlmetadata+xml
date: Fri, 24 Mar 2023 17:44:38 GMT
server: Apache

ETag response header values are stable across several requests. I’ve changed the web server’s etag values-generating algorithm (from default “MTime Size” to “Digest” ( https://httpd.apache.org/docs/2.4/en/mod/core.html#fileetag ) ) to test and while that obviously changed the values that were being generated it did not change shibd’s behaviour wrt re-downloading (so I changed it back).

All data is public and involved servers are publicly accessible, making this hopefully easy to test.

( https://shibboleth.atlassian.net/browse/SSPCPP-973#add-comment?atlOrigin=eyJpIjoiN2Q0OWQ3ZTFlYjMwNGMwZWFmODU4OGIzN2IyMzJlZmQiLCJwIjoiaiJ9 ) Add Comment ( https://shibboleth.atlassian.net/browse/SSPCPP-973#add-comment?atlOrigin=eyJpIjoiN2Q0OWQ3ZTFlYjMwNGMwZWFmODU4OGIzN2IyMzJlZmQiLCJwIjoiaiJ9 )

Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8 ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100219- sha1:ac3e918 )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/commits/attachments/20230324/57542551/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-static-comment-icon-135a87d4-f6e1-41fe-b06f-a252b7001bb0
Type: image/png
Size: 1084 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230324/57542551/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-static-trivial-f039f68d-aa4c-4258-99bb-ba3968100849
Type: image/png
Size: 446 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230324/57542551/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-avatar-bad14326-bf30-4ceb-9c6b-66d0ba2cc9a9
Type: image/png
Size: 345 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230324/57542551/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-static-footer-desktop-logo-388677fe-72e6-4321-adec-70b2f7fe5948
Type: image/png
Size: 10805 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230324/57542551/attachment-0007.png>