shibd segfault on parsing assertion

[Ilkka Lindblom]

Hello.

I have a problem keeping my shibd running. The basic issue is that on
receiving an assertion from IDP, there is a segmentation fault every
time. I'll start by saying the stack trace is similar to this old
thread, even if the trigger is different:

http://shibboleth.net/pipermail/users/2014-June/015768.html

Aug  2 14:05:55 foobar kernel: shibd[12303]: segfault at 8 ip
00007f18be1000b1 sp 00007f18b61faea0 error 4 in
libcrypto.so.1.0.1e[7f18be059000+1ba000]
Aug  2 14:05:56 foobar abrt[12304]: Saved core dump of pid 12296
(/usr/sbin/shibd) to /var/spool/abrt/ccpp-2017-08-02-14:05:55-12296
(74375168 bytes)

 (gdb) where
#0  0x00007f18be1000b1 in BN_num_bits () from /usr/lib64/libcrypto.so.10
#1  0x00007f18be125e8d in RSA_size () from /usr/lib64/libcrypto.so.10
#2  0x0000003700fe424a in xmltooling::BasicX509Credential::getKeySize()
const () from /usr/lib64/libxmltooling.so.7
#3  0x0000003700feb4af in
xmltooling::CredentialCriteria::matches(xmltooling::Credential const&)
const () from /usr/lib64/libxmltooling.so.7
#4  0x00000036ff77fee7 in
opensaml::saml2md::AbstractMetadataProvider::resolve(std::vector<xmltooling::Credential
const*, std::allocator<xmltooling::Credential const*> >&,
xmltooling::CredentialCriteria const*) const () from /usr/lib64/libsaml.so.9
#5  0x0000003700fecf8b in
xmltooling::ExplicitKeyTrustEngine::validate(xmlsignature::Signature&,
xmltooling::CredentialResolver const&, xmltooling::CredentialCriteria*)
const ()
   from /usr/lib64/libxmltooling.so.7
#6  0x0000003700fe96a7 in
xmltooling::ChainingTrustEngine::validate(xmlsignature::Signature&,
xmltooling::CredentialResolver const&, xmltooling::CredentialCriteria*)
const ()
   from /usr/lib64/libxmltooling.so.7
#7  0x00000036ff6c7e5a in
opensaml::XMLSigningRule::evaluate(xmltooling::XMLObject const&,
xmltooling::GenericRequest const*, opensaml::SecurityPolicy&) const ()
from /usr/lib64/libsaml.so.9
#8  0x00000036ff6c3396 in
opensaml::SecurityPolicy::evaluate(xmltooling::XMLObject const&,
xmltooling::GenericRequest const*) () from /usr/lib64/libsaml.so.9
#9  0x00000036ff821147 in
opensaml::saml2p::SAML2POSTDecoder::decode(std::basic_string<char,
std::char_traits<char>, std::allocator<char> >&,
xmltooling::GenericRequest const&, opensaml::SecurityPolicy&) const ()
from /usr/lib64/libsaml.so.9
#10 0x00007f18beef92d2 in
shibsp::AssertionConsumerService::processMessage (this=0x24405e0,
application=..., httpRequest=..., httpResponse=...) at
handler/impl/AssertionConsumerService.cpp:185
#11 0x00007f18beefa4ed in shibsp::AssertionConsumerService::receive
(this=0x24405e0, in=<value optimized out>, out=...) at
handler/impl/AssertionConsumerService.cpp:155
#12 0x00007f18bef9d6a4 in shibsp::ListenerService::receive (this=<value
optimized out>, in=..., out=...) at remoting/impl/ListenerService.cpp:139
#13 0x00007f18befa1739 in shibsp::ServerThread::job (this=0x246ca20) at
remoting/impl/SocketListener.cpp:547
#14 0x00007f18befa22e7 in shibsp::ServerThread::run (this=0x246ca20) at
remoting/impl/SocketListener.cpp:487
#15 0x00007f18befa23f1 in server_thread_fn (arg=0x246ca20) at
remoting/impl/SocketListener.cpp:421
#16 0x00000032052079d1 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003204ae88fd in clone () from /lib64/libc.so.6

As the linked thread and other similar ones say, this seems like a
library mismatch of some sort. Only I have been unable to detect it. The
server is RHEL 6. I have installed Shibboleth SP binaries with Yum using
a RHEL 6 repo file created by the tool at
https://shibboleth.net/downloads/service-provider/latest/RPMS/. I have
tried to update all the relevant libraries to the latest ones in RHEL
updates. I used lsof to check that shibd is actually using
librypto.so.10 in /usr/lib64/ and not some other one.

We have also verified that the assertion seems valid and it anyway is
practically identical to the ones we send to many other SPs.

The issue can be reproduced even when shibd is run directly in the root
shell, this should eliminate any launcher scripts setting incorrect
settings (LD_PATH etc) as a root cause.

I understand this is an issue particular to this server rather than a
fault in the Shibboleth product, but can anyone suggest a way to find
the mismatch?

Thanks, IL