[JIRA] (OSJ-379) "trustAnchors parameter must be non-empty" failure in httpClient
Paul B. Henson (Jira)
jira at shibboleth.atlassian.net
Thu May 18 03:47:44 UTC 2023
Paul B. Henson ( https://shibboleth.atlassian.net/secure/ViewProfile.jspa?accountId=5b58f937ffd0a32d7917a0df ) *commented* on OSJ-379 ( https://shibboleth.atlassian.net/browse/OSJ-379?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9 )
Re: "trustAnchors parameter must be non-empty" failure in httpClient ( https://shibboleth.atlassian.net/browse/OSJ-379?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9 )
I looked for any correlation to anything and couldn’t find one. The most recent failure started at 12:05pm on 5/11:
2023-05-11 12:05:32,219 - / - ERROR [org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver:238] - Meta
data Resolver FileBackedHTTPMetadataResolver ADFSTrust: Error retrieving metadata from https://sts.cpp.edu/fed
erationmetadata/2007-06/federationmetadata.xml: Unexpected error: java.security.InvalidAlgorithmParameterExcep
tion: the trustAnchors parameter must be non-empty
The most recent patch update prior to that was on 5/4:
75 | --refresh -y upgrade | 2023-05-04 21:42 | Upgrade | 4 EE
The cacerts file was last updated in 2022:
r r r - 1 root root 156478 Sep 22 2022 /etc/pki/ca-trust/extracted/java/cacerts
Hmm, although the symlink pointing to it was modified on 5/4; it looks like java got updated that day:
{{ Upgrade java-11-openjdk-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}
{{ Upgraded java-11-openjdk-1:11.0.18.0.10-2.el8_7.x86_64 @@System }}
{{ Upgrade java-11-openjdk-headless-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}
Upgraded java-11-openjdk-headless-1:11.0.18.0.10-2.el8_7.x86_64 @@System
java uses the specific version home:
{{# java -XshowSettings:properties -version 2>&1 | grep java.home }}
java.home = /usr/lib/jvm/java-11-openjdk-11.0.19.0.7-1.el8_7.x86_64
So on 5/4 about 9:45pm, the previous link
/etc/java/java-11-openjdk/java-11-openjdk-11.0.18.0.10-2.el8_7.x86_64/lib/security/cacerts
would have gone away. It seems odd that it would work for a week then suddenly break? We use the needs-restarting utility to reboot if necessary or otherwise restart services as necessary after updates are installed. If jetty was updated it would definitely get restarted, but I don’t see a restart on 5/4, so it looks like it might not be smart enough to restart the jetty service if java is updated.
Usually one of my boxes goes wiggy, this time all 3 did within the same day or so. One of the other ones first showed the error on 5/12 at 5pm:
2023-05-12 17:02:36,998 - / - ERROR [org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver:238] - Meta
data Resolver FileBackedHTTPMetadataResolver ADFSTrust: Error retrieving metadata from https://sts.cpp.edu/fed
erationmetadata/2007-06/federationmetadata.xml: Unexpected error: java.security.InvalidAlgorithmParameterExcep
tion: the trustAnchors parameter must be non-empty
java on that system was updated the night before:
Begin time : Thu 11 May 2023 09:32:57 PM PDT
{{ Upgrade java-11-openjdk-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}
{{ Upgraded java-11-openjdk-1:11.0.18.0.10-2.el8_7.x86_64 @@System }}
{{ Upgrade java-11-openjdk-headless-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}
Upgraded java-11-openjdk-headless-1:11.0.18.0.10-2.el8_7.x86_64 @@System
The third one blew up at 5pm on 5/12 as well:
2023-05-12 17:02:37,968 - / - ERROR [org.opensaml.saml.metadata.resolver.impl.HTTPMetadataResolver:238] - Meta
data Resolver FileBackedHTTPMetadataResolver ADFSTrust: Error retrieving metadata from https://sts.cpp.edu/fed
erationmetadata/2007-06/federationmetadata.xml: Unexpected error: java.security.InvalidAlgorithmParameterExcep
tion: the trustAnchors parameter must be non-empty
it had also been updated the night before:
Begin time : Thu 11 May 2023 09:48:17 PM PDT
{{ Upgrade java-11-openjdk-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}
{{ Upgraded java-11-openjdk-1:11.0.18.0.10-2.el8_7.x86_64 @@System }}
{{ Upgrade java-11-openjdk-headless-1:11.0.19.0.7-1.el8_7.x86_64 @cpp-rocky8-appstream }}
Upgraded java-11-openjdk-headless-1:11.0.18.0.10-2.el8_7.x86_64 @@System
So there is a possible correlation, java was updated but jetty not restarted on each system before it broke. But on one system it took a week to break, and on the other two less than 24 hours?
And this is the only time I’ve ever noticed all 3 breaking within a short window. Usually it’s just 1, and the others are fine, but they all get updated within a week of each other.
I’ll have to look at needs-restarting and seeing why it doesn’t catch services that use java, and possibly hard code a jetty restart if java is updated into our patch script.
Thanks for the sleuthing assistance.
( https://shibboleth.atlassian.net/browse/OSJ-379#add-comment?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9 ) Add Comment ( https://shibboleth.atlassian.net/browse/OSJ-379#add-comment?atlOrigin=eyJpIjoiMmZhZTBhNzM0MTFiNDg4MTk5NzQ2YjY2Nzc2MDBmYjciLCJwIjoiaiJ9 )
Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8 ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100225- sha1:d57183e )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/commits/attachments/20230518/fb38b633/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-static-comment-icon-a482ca53-e9ff-4229-994e-0206672b91d5
Type: image/png
Size: 1084 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230518/fb38b633/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-avatar-1f26d963-df88-4652-a0d7-0787be6dd769
Type: image/png
Size: 345 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230518/fb38b633/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-static-smile-fa37cd07-a04a-431f-836b-2297fb28a8c9
Type: image/png
Size: 752 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230518/fb38b633/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jira-generated-image-static-footer-desktop-logo-7227a314-9c3e-469a-b6e7-31dff3c4e98c
Type: image/png
Size: 10805 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/commits/attachments/20230518/fb38b633/attachment-0007.png>
More information about the commits
mailing list