shibd crashing frequently

[Joshua LaPlace]

We rebuilt the shibboleth rpm and the dependency rpms on a server that 
was identical to the production servers and we are still seeing the same 
crash periodically. It occurs maybe once a day.

shibd[2251]: *** Error in `/usr/sbin/shibd': double free or corruption 
(!prev): 0x00007f72fc05f8a0 ***
shibd[2251]: ======= Backtrace: =========
shibd[2251]: /lib64/libc.so.6(+0x816b9)[0x7f731d6f26b9]
shibd[2251]: /lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7f731c32b80d]
shibd[2251]: 
/lib64/libcrypto.so.10(EVP_CIPHER_CTX_cleanup+0x3e)[0x7f731c3e95be]
shibd[2251]: /lib64/libcrypto.so.10(EVP_CipherInit_ex+0x81)[0x7f731c3e9811]
shibd[2251]: 
/lib64/libxml-security-c.so.20(_ZN25OpenSSLCryptoSymmetricKey14decryptCtxInitEPKhS1_j+0x339)[0x7f731fee9659]
shibd[2251]: 
/lib64/libxml-security-c.so.20(_ZN25OpenSSLCryptoSymmetricKey7decryptEPKhPhjj+0xb9)[0x7f731fee9a89]
shibd[2251]: 
/lib64/libxml-security-c.so.20(_ZNK27XENCAlgorithmHandlerDefault24doGCMDecryptToSafeBufferEP9TXFMChainPK13XSECCryptoKeyjR10safeBuffer+0x121)[0x7f731fed4af1]
shibd[2251]: 
/lib64/libxml-security-c.so.20(_ZNK27XENCAlgorithmHandlerDefault19decryptToSafeBufferEP9TXFMChainP20XENCEncryptionMethodPK13XSECCryptoKeyPN11xercesc_3_211DOMDocumentER10safeBuffer+0x1fd)[0x7f731fed5ccd]
shibd[2251]: 
/lib64/libxmltooling.so.8(_ZNK10xmltooling10DataSealer6unwrapEPKc+0x2fa)[0x7f731e69553a]
shibd[2251]: 
/lib64/libshibsp.so.8(_ZN6shibsp7SSCache7recoverERKNS_11ApplicationEPKcS5_+0x2d4)[0x7f73206fdb34]
shibd[2251]: 
/lib64/libshibsp.so.8(_ZN6shibsp7SSCache7receiveERNS_3DDFERSo+0x1183)[0x7f7320701253]
shibd[2251]: 
/lib64/libshibsp.so.8(_ZN6shibsp15ListenerService7receiveERNS_3DDFERSo+0x2a4)[0x7f73207376d4]
shibd[2251]: 
/lib64/libshibsp.so.8(_ZN6shibsp12ServerThread3jobEv+0x69d)[0x7f732073bf6d]
shibd[2251]: 
/lib64/libshibsp.so.8(_ZN6shibsp12ServerThread3runEv+0x1a1)[0x7f732073d881]
shibd[2251]: 
/lib64/libshibsp.so.8(_Z16server_thread_fnPv+0x11)[0x7f732073da21]
shibd[2251]: /lib64/libpthread.so.0(+0x7ea5)[0x7f731da46ea5]
shibd[2251]: /lib64/libc.so.6(clone+0x6d)[0x7f731d76f8cd]

On 9/12/19 4:09 PM, Cantor, Scott wrote:
> On 9/12/19, 3:53 PM, "users on behalf of Joshua LaPlace" <users-bounces at shibboleth.net on behalf of joshua.laplace at enfoldsystems.com> wrote:
>
>> Shibboleth is 3.0.4-3.2 recompiled from the source RPM to work with the
>> apache from RHSCL
> Given that you built it yourself, and the errors seem like a library conflict somewhere in the vicinity of OpenSSL's memory management code, I would guess the build isn't right somewhere and is running with different libs than it was built against.
>   
> -- Scott
>
>