SEGFAULT on RHEL9 when using metadata downloaded via HTTPS

Tue Mar 14 13:00:30 UTC 2023

Hello,

we're configuring shibboleth SP on RHEL9 to integrate our
authentication environment with Italian Public Digital Identity System
(SPID).

When configuring metadata endpoint on https (any https endpoint has
the same result):

<MetadataProvider id="spid" type="XML"
url="https://demo.spid.gov.it/metadata.xml"
backingFilePath="metadata.xml" maxRefreshDelay="7200" />

Shibd crashes with SIGSEGV:

shibd[4614]: segfault at 55ebfda35ff4 ip 00007f76ee321809 sp
00007ffc37c1c968 error 4 in libcrypto.so.3.0.1[7f76ee1ce000+258000]

shibd debug log shows it stops after downloading the metadata file
from the endpoint.

2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML : using remote
resource (https://demo.spid.gov.it/metadata.xml)
2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML : backup
remote resource to (/var/cache/shibboleth/metadata.xml)
2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML : loaded
initial cache tag (If-None-Match:
W/"1444-j5l8MPF6u+/QXL9468tPJ9qnzIU")
2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML : will reload
remote resource at most every 7200 seconds
2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML [spid]: remote
metadata resource will be backed up to
(/var/cache/shibboleth/metadata.xml.3097)
2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML [spid]:
loading configuration from external resource...
2023-03-14 13:45:48 DEBUG OpenSAML.MetadataProvider.XML [spid]:
writing to backing file: /var/cache/shibboleth/metadata.xml.3097
2023-03-14 13:45:48 DEBUG XMLTooling.libcurl.InputStream [spid]:
libcurl trying to fetch https://demo.spid.gov.it/metadata.xml

If i check /var/cache/shibboleth/metadata.xml.3097 I see the file has
the right content from the endpoint asked to contact.

This is the full backtrace obtained with coredumpctl after installing
debug symbols:

(gdb) backtrace
#0 0x00007f690109492e in __GI___libc_free (mem=0xecd4fa1000000000) at
malloc.c:3251
#1 0x00007f6900ca078b in OPENSSL_sk_free (st=0x561cecd4f9f4) at
crypto/stack/stack.c:414
#2 OPENSSL_sk_free (st=0x561cecd4f9f4) at crypto/stack/stack.c:410
#3 0x00007f6900ca07f5 in OPENSSL_sk_pop_free (func=<optimized out>,
st=<optimized out>) at crypto/stack/stack.c:407
#4 0x00007f6900242af8 in SSL_CTX_free (a=0x561cecd2fda0) at ssl/ssl_lib.c:3431
#5 SSL_CTX_free (a=0x561cecd2fda0) at ssl/ssl_lib.c:3392
#6 0x00007f690050a58b in ossl_closeone.isra.0
(data=data at entry=0x561cec8eac90, conn=conn at entry=0x561cec8f2910,
connssl=<optimized out>, connssl=<optimized out>) at
../../lib/vtls/openssl.c:1417
#7 0x00007f69004f4e16 in ossl_close (data=0x561cec8eac90,
conn=0x561cec8f2910, sockindex=<optimized out>) at
../../lib/vtls/openssl.c:1428
#8 0x00007f69004eaf76 in Curl_ssl_close (sockindex=0,
conn=0x561cec8f2910, data=0x561cec8eac90) at ../../lib/vtls/vtls.c:672
#9 conn_shutdown (conn=0x561cec8f2910, data=0x561cec8eac90) at
../../lib/url.c:736
#10 Curl_disconnect (data=0x561cec8eac90, conn=0x561cec8f2910,
dead_connection=false) at ../../lib/url.c:855
#11 0x00007f69004a38f7 in Curl_conncache_close_all_connections
(connc=0x561cec8e39f0) at ../../lib/conncache.c:557
#12 0x00007f69004d286b in curl_multi_cleanup (multi=0x561cec8e38f0) at
../../lib/multi.c:2476
#13 curl_multi_cleanup (multi=0x561cec8e38f0) at ../../lib/multi.c:2438
#14 0x00007f6901768202 in
xmltooling::CurlURLInputStream::~CurlURLInputStream
(this=0x561cec7de918, __in_chrg=<optimized out>) at
util/CurlURLInputStream.cpp:234
#15 0x00007f690176829d in
xmltooling::CurlURLInputStream::~CurlURLInputStream
(this=0x561cec7de918, __in_chrg=<optimized out>) at
util/CurlURLInputStream.cpp:243
#16 0x00007f69016d248f in
xmltooling::CloneInputStream::~CloneInputStream (this=0x561cec9e8838,
__in_chrg=<optimized out>) at util/CloneInputStream.cpp:51
#17 0x00007f69016d255d in
xmltooling::CloneInputStream::~CloneInputStream (this=0x561cec9e8838,
__in_chrg=<optimized out>) at util/CloneInputStream.cpp:52
#18 0x00007f6901c5115e in xercesc_3_2::XMLReader::~XMLReader
(this=<optimized out>, this=<optimized out>) at
xercesc/internal/XMLReader.cpp:462
#19 0x00007f6901c417e2 in xercesc_3_2::ReaderMgr::reset
(this=0x561cec8906f8) at xercesc/internal/ReaderMgr.cpp:942
#20 0x00007f6901c366f2 in
xercesc_3_2::JanitorMemFunCall<xercesc_3_2::ReaderMgr>::reset (p=0x0,
this=0x7ffeda7cd280) at ./xercesc/util/Janitor.c:242
#21 xercesc_3_2::JanitorMemFunCall<xercesc_3_2::ReaderMgr>::~JanitorMemFunCall
(this=<optimized out>, this=<optimized out>) at
./xercesc/util/Janitor.c:206
#22 xercesc_3_2::IGXMLScanner::scanDocument (this=0x561cec890628,
src=...) at xercesc/internal/IGXMLScanner.cpp:304
#23 0x00007f6901c6d0df in xercesc_3_2::AbstractDOMParser::parse
(this=0x561cec7e0118, source=...) at
xercesc/parsers/AbstractDOMParser.cpp:545
#24 0x00007f6901c7a7da in xercesc_3_2::DOMLSParserImpl::parse
(this=0x561cec7e0118, source=0x7ffeda7cd4a0) at
xercesc/parsers/DOMLSParserImpl.cpp:753
#25 0x00007f69016d3da4 in xmltooling::ParserPool::parse
(this=0x561cec8124b0, domsrc=...) at util/ParserPool.cpp:141
#26 0x00007f69016e39ad in xmltooling::ReloadableXMLFile::load
(this=this at entry=0x561cec8cce60, backup=backup at entry=false,
backingFile="/var/cache/shibboleth/metadata/spid-test-metadata.xml.dbbe")
at util/ReloadableXMLFile.cpp:444
#27 0x00007f690238b546 in opensaml::saml2md::XMLMetadataProvider::load
(this=this at entry=0x561cec8cccf0, backup=backup at entry=false,
backingFile="/var/cache/shibboleth/metadata/spid-test-metadata.xml")
at saml2/metadata/impl/XMLMetadataProvider.cpp:184
#28 0x00007f690238b20a in
opensaml::saml2md::XMLMetadataProvider::background_load
(this=0x561cec8cccf0) at
saml2/metadata/impl/XMLMetadataProvider.cpp:277
#29 0x00007f690238b2c0 in opensaml::saml2md::XMLMetadataProvider::init
(this=0x561cec8cccf0) at
saml2/metadata/impl/XMLMetadataProvider.cpp:152
#30 0x00007f69023011f9 in
opensaml::saml2md::ChainingMetadataProvider::init
(this=0x561cec8e1f30) at
saml2/metadata/impl/ChainingMetadataProvider.cpp:244
#31 0x00007f69030c8541 in shibsp::XMLApplication::XMLApplication
(this=0x561cecb87580, sp=0x561cec7aeb70, pp=0x561cecbd4d00,
e=0x561cec8baef0, deprecationSupport=<optimized out>, base=0x0,
doc=0x0,
__in_chrg=<optimized out>, __vtt_parm=<optimized out>) at
impl/XMLApplication.cpp:269
#32 0x00007f69030e0366 in shibsp::XMLConfigImpl::XMLConfigImpl
(this=0x561cec89c630, e=<optimized out>, first=<optimized out>,
outer=0x561cec7aeb70, log=..., __in_chrg=<optimized out>,
__vtt_parm=<optimized out>) at impl/XMLServiceProvider.cpp:562
#33 0x00007f69030e1aaa in shibsp::XMLConfig::background_load
(this=0x561cec7aeb70) at impl/XMLServiceProvider.cpp:938
#34 0x00007f690301676a in shibsp::SPConfig::instantiate
(this=0x7f69033fc1a0 <shibsp::g_config>, config=<optimized out>,
rethrow=<optimized out>) at SPConfig.cpp:448
#35 0x0000561ceb601ee9 in main (argc=<optimized out>, argv=<optimized
out>) at shibd.cpp:415

Here the list of the involved packages with versions:

shibboleth-3.4.1-1.x86_64
openssl-libs-3.0.1-47.el9_1.x86_64
openssl-3.0.1-47.el9_1.x86_64
libcurl-7.76.1-19.el9_1.1.x86_64
libxmltooling10-3.2.3-1.x86_64
xerces-c-3.2.3-5.el9.x86_64

I can say for sure that using any HTTP endpoint works like a charm. As
a workaround, we configured httpd to permit access to the metadata
endpoint via http and we're able to continue the setup, but there is a
clear issue to address.

Luca

--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)

"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)

Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca at gmail.com>