<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"> <base href="https://shibboleth.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429;">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0;" bgcolor="#f5f5f5"><!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="border-collapse: collapse; padding: 10px 20px;">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0;">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px;" width="32"><img id="header-avatar-image" class="image_fix" src="https://secure.gravatar.com/avatar/7327e88bffe0ff5d4de8c93cb952b56a?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FPH-0.png" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top;"></td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px;"><a class="user-hover" rel="5b58f937ffd0a32d7917a0df" style="color: #3b73af; text-decoration: none;" id="email_5b58f937ffd0a32d7917a0df" href="https://shibboleth.atlassian.net/secure/ViewProfile.jspa?accountId=5b58f937ffd0a32d7917a0df"> Paul B. Henson </a> <strong>commented</strong> on <a href="https://shibboleth.atlassian.net/browse/OSJ-379?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none;"><img src="cid:jira-generated-image-avatar-1f26d963-df88-4652-a0d7-0787be6dd769" height="16" width="16" border="0" align="absmiddle" alt="Bug"> OSJ-379</a></td>
</tr>
</tbody>
</table></td>
</tr>
<tr>
<td id="email-content-container" style="border-collapse: collapse; padding: 0 20px;">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate;">
<tbody>
<tr><!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="border-collapse: collapse; color: #ffffff; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly;" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff;" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0;">
<tbody>
<tr>
<td style="vertical-align: top; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly;" class="page-title-pattern-header-container"><span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle;"> <a href="https://shibboleth.atlassian.net/browse/OSJ-379?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none;">Re: "trustAnchors parameter must be non-empty" failure in httpClient</a> </span></td>
</tr>
</tbody>
</table></td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0;" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px;">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="border-collapse: collapse; padding: 0 0 10px;"><p style="margin: 10px 0 0; margin-top: 0;">I looked for any correlation to anything and couldn’t find one. The most recent failure started at 12:05pm on 5/11:</p> <p style="margin: 10px 0 0;"><tt>2023-05-11 12:05:32,219 - / - ERROR <span class="error">[org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver:238]</span> - Meta</tt><br> <tt>data Resolver FileBackedHTTPMetadataResolver ADFSTrust: Error retrieving metadata from <a href="https://sts.cpp.edu/fed" class="external-link" rel="nofollow noreferrer" style="color: #3b73af; text-decoration: none;">https://sts.cpp.edu/fed</a></tt><br> <tt>erationmetadata/2007-06/federationmetadata.xml: Unexpected error: java.security.InvalidAlgorithmParameterExcep</tt><br> <tt>tion: the trustAnchors parameter must be non-empty</tt></p> <p style="margin: 10px 0 0;">The most recent patch update prior to that was on 5/4:</p> <p style="margin: 10px 0 0;"><tt> 75 | --refresh -y upgrade | 2023-05-04 21:42 | Upgrade | 4 EE</tt></p> <p style="margin: 10px 0 0;">The cacerts file was last updated in 2022:</p> <p style="margin: 10px 0 0;"><tt><del>r</del><del>r</del><del>r</del>- 1 root root 156478 Sep 22 2022 /etc/pki/ca-trust/extracted/java/cacerts</tt></p> <p style="margin: 10px 0 0;">Hmm, although the symlink pointing to it was modified on 5/4; it looks like java got updated that day:</p> <p style="margin: 10px 0 0;">{{ Upgrade java-11-openjdk-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}<br> {{ Upgraded java-11-openjdk-1:11.0.18.0.10-2.el8_7.x86_64 @@System }}<br> {{ Upgrade java-11-openjdk-headless-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}<br> <tt> Upgraded java-11-openjdk-headless-1:11.0.18.0.10-2.el8_7.x86_64 @@System</tt></p> <p style="margin: 10px 0 0;">java uses the specific version home:</p> <p style="margin: 10px 0 0;">{{# java -XshowSettings:properties -version 2>&1 | grep java.home }}<br> <tt> java.home = /usr/lib/jvm/java-11-openjdk-11.0.19.0.7-1.el8_7.x86_64</tt></p> <p style="margin: 10px 0 0;">So on 5/4 about 9:45pm, the previous link</p> <p style="margin: 10px 0 0;"><tt>/etc/java/java-11-openjdk/java-11-openjdk-11.0.18.0.10-2.el8_7.x86_64/lib/security/cacerts</tt></p> <p style="margin: 10px 0 0;">would have gone away. It seems odd that it would work for a week then suddenly break? We use the <tt>needs-restarting</tt> utility to reboot if necessary or otherwise restart services as necessary after updates are installed. If jetty was updated it would definitely get restarted, but I don’t see a restart on 5/4, so it looks like it might not be smart enough to restart the jetty service if java is updated.</p> <p style="margin: 10px 0 0;">Usually one of my boxes goes wiggy, this time all 3 did within the same day or so. One of the other ones first showed the error on 5/12 at 5pm:</p> <p style="margin: 10px 0 0;"><tt>2023-05-12 17:02:36,998 - / - ERROR <span class="error">[org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver:238]</span> - Meta</tt><br> <tt>data Resolver FileBackedHTTPMetadataResolver ADFSTrust: Error retrieving metadata from <a href="https://sts.cpp.edu/fed" class="external-link" rel="nofollow noreferrer" style="color: #3b73af; text-decoration: none;">https://sts.cpp.edu/fed</a></tt><br> <tt>erationmetadata/2007-06/federationmetadata.xml: Unexpected error: java.security.InvalidAlgorithmParameterExcep</tt><br> <tt>tion: the trustAnchors parameter must be non-empty</tt></p> <p style="margin: 10px 0 0;">java on that system was updated the night before:</p> <p style="margin: 10px 0 0;"><tt>Begin time : Thu 11 May 2023 09:32:57 PM PDT</tt><br> {{ Upgrade java-11-openjdk-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}<br> {{ Upgraded java-11-openjdk-1:11.0.18.0.10-2.el8_7.x86_64 @@System }}<br> {{ Upgrade java-11-openjdk-headless-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}<br> <tt> Upgraded java-11-openjdk-headless-1:11.0.18.0.10-2.el8_7.x86_64 @@System</tt></p> <p style="margin: 10px 0 0;">The third one blew up at 5pm on 5/12 as well:</p> <p style="margin: 10px 0 0;"><tt>2023-05-12 17:02:37,968 - / - ERROR <span class="error">[org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver:238]</span> - Meta</tt><br> <tt>data Resolver FileBackedHTTPMetadataResolver ADFSTrust: Error retrieving metadata from <a href="https://sts.cpp.edu/fed" class="external-link" rel="nofollow noreferrer" style="color: #3b73af; text-decoration: none;">https://sts.cpp.edu/fed</a></tt><br> <tt>erationmetadata/2007-06/federationmetadata.xml: Unexpected error: java.security.InvalidAlgorithmParameterExcep</tt><br> <tt>tion: the trustAnchors parameter must be non-empty</tt></p> <p style="margin: 10px 0 0;">it had also been updated the night before:</p> <p style="margin: 10px 0 0;"><tt>Begin time : Thu 11 May 2023 09:48:17 PM PDT</tt><br> {{ Upgrade java-11-openjdk-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}<br> {{ Upgraded java-11-openjdk-1:11.0.18.0.10-2.el8_7.x86_64 @@System }}<br> {{ Upgrade java-11-openjdk-headless-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}<br> <tt> Upgraded java-11-openjdk-headless-1:11.0.18.0.10-2.el8_7.x86_64 @@System</tt></p> <p style="margin: 10px 0 0;">So there is a possible correlation, java was updated but jetty not restarted on each system before it broke. But on one system it took a week to break, and on the other two less than 24 hours?</p> <p style="margin: 10px 0 0;">And this is the only time I’ve ever noticed all 3 breaking within a short window. Usually it’s just 1, and the others are fine, but they all get updated within a week of each other.</p> <p style="margin: 10px 0 0;">I’ll have to look at needs-restarting and seeing why it doesn’t catch services that use java, and possibly hard code a jetty restart if java is updated into our patch script.</p> <p style="margin: 10px 0 0;">Thanks for the sleuthing assistance <img class="emoticon" src="cid:jira-generated-image-static-smile-fa37cd07-a04a-431f-836b-2297fb28a8c9" height="16" width="16" align="absmiddle" alt="" border="0">.</p></td>
</tr>
</tbody>
</table></td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff;" bgcolor="#ffffff"><script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"description": "View Issue",
"potentialAction": {
"@type": "ViewAction",
"target": "https://shibboleth.atlassian.net/browse/OSJ-379?inbox=true&focusedCommentId=34893&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-34893",
"name": "View Comment"
},
"publisher": {
"@type": "Organization",
"name": "Atlassian",
"url": "https://www.atlassian.com"
}
}
</script>
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px;">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0;">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0;">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle;"><a href="https://shibboleth.atlassian.net/browse/OSJ-379#add-comment?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none;"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-a482ca53-e9ff-4229-994e-0206672b91d5" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle;"> </a></td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px;"><a href="https://shibboleth.atlassian.net/browse/OSJ-379#add-comment?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none;">Add Comment</a></td>
</tr>
</tbody>
</table></td>
</tr>
</tbody>
</table></td>
</tr> <!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly;" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table></td>
</tr>
<tr>
<td id="footer-pattern" style="border-collapse: collapse; padding: 12px 20px;">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0;">
<tbody>
<tr>
<td id="footer-pattern-mobile-separated-links" class="mobile-resize-text" width="100%" colspan="2" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px;">Get Jira notifications on your phone! Download the Jira Cloud app for <a href="https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail" style="color: #3b73af; text-decoration: none;">Android</a> or <a href="https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8" style="color: #3b73af; text-decoration: none;">iOS</a>
<hr></td>
</tr>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px;">This message was sent by Atlassian Jira <span id="footer-build-information">(v1001.0.0-SNAPSHOT#100225-<span title="d57183e3fed6fdf57150cdd23beff8a631bedd93" data-commit-id="d57183e3fed6fdf57150cdd23beff8a631bedd93}">sha1:d57183e</span>)</span></td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top;">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0;">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px; opacity: 0.150;"><img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-7227a314-9c3e-469a-b6e7-31dff3c4e98c" alt="Atlassian logo" title="Atlassian logo" width="192" height="24" class="image_fix"></td>
</tr>
</tbody>
</table></td>
</tr>
</tbody>
</table></td>
</tr>
</tbody>
</table>
</body>
</html>