shibd crashing frequently

[Joshua LaPlace]

We are seeing shibd crash at least once daily. We are running on RHEL 
7.6 with FIPS and SELinux enabled.


Apache is 2.4.34-7.el7.1 installed from RHSCL

Shibboleth is 3.0.4-3.2 recompiled from the source RPM to work with the 
apache from RHSCL


The errors that we are seeing when the service crashes are:

Sep  9 16:10:01  shibd: *** Error in `/usr/sbin/shibd': double free or 
corruption (!prev): 0x00007f47d8006680 ***

Sep  9 16:10:01  shibd: ======= Backtrace: =========

Sep  9 16:10:01  shibd: /lib64/libc.so.6(+0x816b9)[0x7f47f0d896b9]

Sep  9 16:10:01  shibd: 
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7f47ef9c280d]

Sep  9 16:10:01  shibd: 
/lib64/libcrypto.so.10(EVP_CIPHER_CTX_cleanup+0x3e)[0x7f47efa805be]

Sep  9 16:10:01  shibd: 
/lib64/libcrypto.so.10(EVP_CipherInit_ex+0x81)[0x7f47efa80811]

Sep  9 16:10:01  shibd: 
/lib64/libxml-security-c.so.20(_ZN25OpenSSLCryptoSymmetricKey14decryptCtxInitEPKhS1_j+0x339)[0x7f47f3580619]

Sep  9 16:10:01  shibd: 
/lib64/libxml-security-c.so.20(_ZN25OpenSSLCryptoSymmetricKey7decryptEPKhPhjj+0xb9)[0x7f47f3580a49]

Sep  9 16:10:01  shibd: 
/lib64/libxml-security-c.so.20(_ZNK27XENCAlgorithmHandlerDefault24doGCMDecryptToSafeBufferEP9TXFMChainPK13XSECCryptoKeyjR10safeBuffer+0x121)[0x7f47f356bab1]

Sep  9 16:10:01  shibd: 
/lib64/libxml-security-c.so.20(_ZNK27XENCAlgorithmHandlerDefault19decryptToSafeBufferEP9TXFMChainP20XENCEncryptionMethodPK13XSECCryptoKeyPN11xercesc_3_211DOMDocumentER10safeBuffer+0x1fd)[0x7f47f356cc8d]

Sep  9 16:10:01  shibd: 
/lib64/libxmltooling.so.8(_ZNK10xmltooling10DataSealer6unwrapEPKc+0x2fa)[0x7f47f1d2c43a]

Sep  9 16:10:01  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp7SSCache7recoverERKNS_11ApplicationEPKcS5_+0x2d4)[0x7f47f3d949f4]

Sep  9 16:10:01  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp7SSCache7receiveERNS_3DDFERSo+0x1183)[0x7f47f3d98113]

Sep  9 16:10:01  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp15ListenerService7receiveERNS_3DDFERSo+0x2a4)[0x7f47f3dce594]

Sep  9 16:10:01  systemd: shibd.service: main process exited, 
code=killed, status=6/ABRT

Sep  9 16:10:01  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp12ServerThread3jobEv+0x69d)[0x7f47f3dd2e2d]

Sep  9 16:10:01  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp12ServerThread3runEv+0x1a1)[0x7f47f3dd4741]

Sep  9 16:10:01  shibd: 
/lib64/libshibsp.so.8(_Z16server_thread_fnPv+0x11)[0x7f47f3dd48e1]

Sep  9 16:10:01  shibd: /lib64/libpthread.so.0(+0x7ea5)[0x7f47f10ddea5]

Sep  9 16:10:01  shibd: /lib64/libc.so.6(clone+0x6d)[0x7f47f0e068cd]


and

Sep 12 14:27:19  shibd: *** Error in `/usr/sbin/shibd': corrupted size 
vs. prev_size: 0x00007f92ac052ab0 ***

Sep 12 14:27:19  shibd: ======= Backtrace: =========

Sep 12 14:27:19  shibd: /lib64/libc.so.6(+0x7f804)[0x7f92dd417804]

Sep 12 14:27:19  shibd: /lib64/libc.so.6(+0x818fb)[0x7f92dd4198fb]

Sep 12 14:27:19  shibd: 
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7f92dc05280d]

Sep 12 14:27:19  shibd: 
/lib64/libcrypto.so.10(EVP_CIPHER_CTX_cleanup+0x3e)[0x7f92dc1105be]

Sep 12 14:27:19  shibd: 
/lib64/libcrypto.so.10(EVP_CipherInit_ex+0x81)[0x7f92dc110811]

Sep 12 14:27:19  shibd: 
/lib64/libxml-security-c.so.20(_ZN25OpenSSLCryptoSymmetricKey14decryptCtxInitEPKhS1_j+0x339)[0x7f92dfc10619]

Sep 12 14:27:19  shibd: 
/lib64/libxml-security-c.so.20(_ZN25OpenSSLCryptoSymmetricKey7decryptEPKhPhjj+0xb9)[0x7f92dfc10a49]

Sep 12 14:27:19  shibd: 
/lib64/libxml-security-c.so.20(_ZNK27XENCAlgorithmHandlerDefault24doGCMDecryptToSafeBufferEP9TXFMChainPK13XSECCryptoKeyjR10safeBuffer+0x121)[0x7f92dfbfbab1]

Sep 12 14:27:19  shibd: 
/lib64/libxml-security-c.so.20(_ZNK27XENCAlgorithmHandlerDefault19decryptToSafeBufferEP9TXFMChainP20XENCEncryptionMethodPK13XSECCryptoKeyPN11xercesc_3_211DOMDocumentER10safeBuffer+0x1fd)[0x7f92dfbfcc8d]

Sep 12 14:27:19  shibd: 
/lib64/libxmltooling.so.8(_ZNK10xmltooling10DataSealer6unwrapEPKc+0x2fa)[0x7f92de3bc43a]

Sep 12 14:27:19  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp7SSCache7recoverERKNS_11ApplicationEPKcS5_+0x2d4)[0x7f92e04249f4]

Sep 12 14:27:19  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp7SSCache7receiveERNS_3DDFERSo+0x1183)[0x7f92e0428113]

Sep 12 14:27:19  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp15ListenerService7receiveERNS_3DDFERSo+0x2a4)[0x7f92e045e594]

Sep 12 14:27:19  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp12ServerThread3jobEv+0x69d)[0x7f92e0462e2d]

Sep 12 14:27:19  shibd: 
/lib64/libshibsp.so.8(_ZN6shibsp12ServerThread3runEv+0x1a1)[0x7f92e0464741]

Sep 12 14:27:19  shibd: 
/lib64/libshibsp.so.8(_Z16server_thread_fnPv+0x11)[0x7f92e04648e1]

Sep 12 14:27:19  shibd: /lib64/libpthread.so.0(+0x7ea5)[0x7f92dd76dea5]

Sep 12 14:27:19  shibd: /lib64/libc.so.6(clone+0x6d)[0x7f92dd4968cd]