Segmentation fault starting shibd on Fedora Cloud/Workstation 37
Simon Cunningham
simon at nu-solutions.com
Tue Feb 21 22:55:24 UTC 2023
I have shibboleth setup on two servers in AWS running Fedora Cloud 37 (plus one VM running Fedora Workstation 37 on my machine for testing); it is working on one cloud server but not the other nor the VM. When attempting to start shibboleth on the failing servers it segfaults and dumps core, which appears to happen when reading the keys.
1) While focusing on the differences between the servers I found the following:
2) On the working server the file permissions are NOT what they apparently should be. Root owns the PEM files, not shibd.
3) On the failing server, shibd will start if I change ownership of the PEM files to root, but it won't work. We do get additional entries in the shibd.log file:
2023-02-21 14:19:30 INFO Shibboleth.Application : multiple CredentialResolver plugins, wrapping in a chain
2023-02-21 14:19:30 INFO Shibboleth.Application : building CredentialResolver of type Chaining...
2023-02-21 14:19:30 INFO XMLTooling.CredentialResolver.Chaining : building CredentialResolver of type File
2023-02-21 14:19:30 INFO XMLTooling.SecurityHelper : loading private key from file (/etc/shibboleth/sp-signing-key.pem)
2023-02-21 14:19:30 ERROR OpenSSL : error code: 33558541 in crypto/bio/bss_file.c, line 288
2023-02-21 14:19:30 ERROR OpenSSL : error data: fopen('/etc/shibboleth/sp-signing-key.pem','r')
2023-02-21 14:19:30 ERROR OpenSSL : error code: 537346050 in crypto/bio/bss_file.c, line 290
2023-02-21 14:19:30 ERROR XMLTooling.CredentialResolver.Chaining : caught exception processing embedded CredentialResolver element: Unable to load private key from file (/etc/shibboleth/sp-signing-key.pem).
2023-02-21 14:19:30 INFO XMLTooling.CredentialResolver.Chaining : building CredentialResolver of type File
2023-02-21 14:19:30 INFO XMLTooling.SecurityHelper : loading private key from file (/etc/shibboleth/sp-encrypt-key.pem)
2023-02-21 14:19:30 ERROR OpenSSL : error code: 33558541 in crypto/bio/bss_file.c, line 288
2023-02-21 14:19:30 ERROR OpenSSL : error data: fopen('/etc/shibboleth/sp-encrypt-key.pem','r')
2023-02-21 14:19:30 ERROR OpenSSL : error code: 537346050 in crypto/bio/bss_file.c, line 290
2023-02-21 14:19:30 ERROR XMLTooling.CredentialResolver.Chaining : caught exception processing embedded CredentialResolver element: Unable to load private key from file (/etc/shibboleth/sp-encrypt-key.pem).
2023-02-21 14:19:30 INFO Shibboleth.Listener : listener service starting
4) On the failing server, shibd fails to start when the PEM files are owned by shibd. Here is the output I captured via journalctl:
Feb 21 13:38:51 nusol-fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=shibd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 21 13:38:51 nusol-fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=shibd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 21 13:38:51 nusol-fedora systemd[1]: Stopped shibd.service - Shibboleth Service Provider Daemon.
░░ Subject: A stop job for unit shibd.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A stop job for unit shibd.service has finished.
░░
░░ The job identifier is 32677 and the job result is done.
Feb 21 13:38:51 nusol-fedora systemd[1]: Starting shibd.service - Shibboleth Service Provider Daemon...
░░ Subject: A start job for unit shibd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit shibd.service has begun execution.
░░
░░ The job identifier is 32677.
Feb 21 13:38:51 nusol-fedora audit[90996]: ANOM_ABEND auid=4294967295 uid=976 gid=976 ses=4294967295 subj=kernel pid=90996 comm="shibd" exe="/usr/sbin/shibd" sig=11 res=1
Feb 21 13:38:51 nusol-fedora kernel: shibd[90996]: segfault at 18 ip 00007f0f66ab37fc sp 00007fff0c563280 error 4 in libc.so.6[7f0f66a49000+156000] likely on CPU 11 (core 11, socket 0)
Feb 21 13:38:51 nusol-fedora kernel: Code: 0f 11 07 0f 11 47 20 8b 06 8b 56 04 89 47 30 31 c0 85 d2 0f 95 c0 89 47 1c 31 c0 c3 66 90 f3 0f 1e fa 41 54 55 53 48 89 fb 90 <8b> 57 18 64 8b 04 25 d0 02 00 00 39 c2 0f 84 01 01 00 00 83 7f 30
Feb 21 13:38:51 nusol-fedora audit: BPF prog-id=295 op=LOAD
Feb 21 13:38:51 nusol-fedora audit: BPF prog-id=296 op=LOAD
Feb 21 13:38:51 nusol-fedora audit: BPF prog-id=297 op=LOAD
Feb 21 13:38:51 nusol-fedora systemd[1]: Started systemd-coredump at 55-91001-0.service - Process Core Dump (PID 91001/UID 0).
░░ Subject: A start job for unit systemd-coredump at 55-91001-0.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit systemd-coredump at 55-91001-0.service has finished successfully.
░░
░░ The job identifier is 32869.
Feb 21 13:38:51 nusol-fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump at 55-91001-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 21 13:38:51 nusol-fedora systemd-coredump[91002]: [🡕] Process 90996 (shibd) of user 976 dumped core.
Module linux-vdso.so.1 with build-id f8b78ccca0c65d3195fe540e0845d39be387292c
Module UTF-16.so with build-id 42ab55d294988cccb4ef6ec7d5bed3acf41f72fb
Module libunistring.so.2 with build-id 0c89f0c5be4a239b6ab72dc3aa015b95fc8ede0d
Metadata for module libunistring.so.2 owned by FDO found: {
"type" : "rpm",
"name" : "libunistring",
"version" : "1.0-2.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libicudata.so.71 with build-id 47276216e74b87c4b9bd01ac85c7bf332e6f2aed
Metadata for module libicudata.so.71 owned by FDO found: {
"type" : "rpm",
"name" : "icu",
"version" : "71.1-2.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libpcre2-8.so.0 with build-id 51cf2b0dc0884111cd6107b8b84bc1dc9e896de6
Metadata for module libpcre2-8.so.0 owned by FDO found: {
"type" : "rpm",
"name" : "pcre2",
"version" : "10.40-1.fc37.1",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libssl.so.3 with build-id d3fc0855dc0632f42fda3b02b4035f4245633408
Metadata for module libssl.so.3 owned by FDO found: {
"type" : "rpm",
"name" : "openssl",
"version" : "3.0.5-3.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libidn2.so.0 with build-id 220865bcdf15c803f8dd897650798ecb5c2ef1e6
Metadata for module libidn2.so.0 owned by FDO found: {
"type" : "rpm",
"name" : "libidn2",
"version" : "2.3.4-1.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libnghttp2.so.14 with build-id 29a942922b7bbe874e74f9af187e423b826a9fff
Metadata for module libnghttp2.so.14 owned by FDO found: {
"type" : "rpm",
"name" : "nghttp2",
"version" : "1.51.0-1.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libxalanMsg.so.112 with build-id a2318ec6acc00deee1eaa2651c5a79657bd3a44b
Metadata for module libxalanMsg.so.112 owned by FDO found: {
"type" : "rpm",
"name" : "xalan-c",
"version" : "1.12.0-22.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libicui18n.so.71 with build-id f4748e4011f7e0fd328ade31e62bc9167e417b74
Metadata for module libicui18n.so.71 owned by FDO found: {
"type" : "rpm",
"name" : "icu",
"version" : "71.1-2.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libicuuc.so.71 with build-id f0a97a601f057dd0739c60482ff1b33c5e054456
Metadata for module libicuuc.so.71 owned by FDO found: {
"type" : "rpm",
"name" : "icu",
"version" : "71.1-2.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libselinux.so.1 with build-id 5d8ce032a7d7e48c206e1274ffba675e58e56df9
Metadata for module libselinux.so.1 owned by FDO found: {
"type" : "rpm",
"name" : "libselinux",
"version" : "3.4-5.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libdl.so.2 with build-id ff36f66b23c53809d27933a667b1f5d6b55e033b
Module libz.so.1 with build-id d94299d7572e23295ceaf7674110eb6f2689cd91
Metadata for module libz.so.1 owned by FDO found: {
"type" : "rpm",
"name" : "zlib",
"version" : "1.2.12-5.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libcrypto.so.1.1 with build-id fb5fdff778b993ef699bed5fee1e0cdd3e08eb8f
Metadata for module libcrypto.so.1.1 owned by FDO found: {
"type" : "rpm",
"name" : "openssl1.1",
"version" : "1.1.1q-2.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libssl.so.1.1 with build-id 36bc6af311e1b3c56da2e90bcbacd54547aa5637
Metadata for module libssl.so.1.1 owned by FDO found: {
"type" : "rpm",
"name" : "openssl1.1",
"version" : "1.1.1q-2.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libcurl.so.4 with build-id 65f51d54e07fd28dc61c57f5f7c728a6b68ec381
Metadata for module libcurl.so.4 owned by FDO found: {
"type" : "rpm",
"name" : "curl",
"version" : "7.85.0-5.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module ld-linux-x86-64.so.2 with build-id bb6fec54c7521fddc569a2f4e141dfb97bf3acbe
Module liblz4.so.1 with build-id 5c9099941d5c7af74f952367da1368a78d06347c
Metadata for module liblz4.so.1 owned by FDO found: {
"type" : "rpm",
"name" : "lz4",
"version" : "1.9.4-1.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libzstd.so.1 with build-id 2b3060a28a992246fb32e1412965c3195dc15558
Metadata for module libzstd.so.1 owned by FDO found: {
"type" : "rpm",
"name" : "zstd",
"version" : "1.5.2-3.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module liblzma.so.5 with build-id 223b6bb0a820acf0da9d0d4712a9bc417ad98228
Metadata for module liblzma.so.5 owned by FDO found: {
"type" : "rpm",
"name" : "xz",
"version" : "5.2.5-10.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libcap.so.2 with build-id f3f24bb6e411e3885602e753eda1e764fc1f5bfb
Metadata for module libcap.so.2 owned by FDO found: {
"type" : "rpm",
"name" : "libcap",
"version" : "2.48-5.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libxalan-c.so.112 with build-id a1e7a1ec8421be3764a7359d2d266561b9818745
Metadata for module libxalan-c.so.112 owned by FDO found: {
"type" : "rpm",
"name" : "xalan-c",
"version" : "1.12.0-22.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libresolv.so.2 with build-id 4a284590f0ff48e205a64837ba1e36e76a27ff68
Module libcrypto.so.3 with build-id 7470c53208efe3360d43315b439f03d64569b15b
Metadata for module libcrypto.so.3 owned by FDO found: {
"type" : "rpm",
"name" : "openssl",
"version" : "3.0.5-3.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libkeyutils.so.1 with build-id 3b45f76aa151e2b57a729356b6972140bc78399b
Metadata for module libkeyutils.so.1 owned by FDO found: {
"type" : "rpm",
"name" : "keyutils",
"version" : "1.6.1-5.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libkrb5support.so.0 with build-id 66b891f1a4f5dd9179f5ec3f8e9f78761db65138
Metadata for module libkrb5support.so.0 owned by FDO found: {
"type" : "rpm",
"name" : "krb5",
"version" : "1.19.2-13.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libcom_err.so.2 with build-id 232d3f343c2ea39e2fa83b87cabd7fab6990d3e1
Metadata for module libcom_err.so.2 owned by FDO found: {
"type" : "rpm",
"name" : "e2fsprogs",
"version" : "1.46.5-3.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libk5crypto.so.3 with build-id 3f48829a1e98dea02124fbf4274085dbe0cab3a1
Metadata for module libk5crypto.so.3 owned by FDO found: {
"type" : "rpm",
"name" : "krb5",
"version" : "1.19.2-13.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libkrb5.so.3 with build-id f008abac61bec044dc2c3b232be6829937cd7dc7
Metadata for module libkrb5.so.3 owned by FDO found: {
"type" : "rpm",
"name" : "krb5",
"version" : "1.19.2-13.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libc.so.6 with build-id 81daba31ee66dbd63efdc4252a872949d874d136
Module libgcc_s.so.1 with build-id bad96a3adc0a3a006e7ef4900ff3ae1ddcc33ed2
Module libm.so.6 with build-id 9ca49f056adbf7edc2bda92b95a74d491abe03f2
Module libstdc++.so.6 with build-id 5c63442039be42a72c8722574572171b19f18d66
Module libxmltooling.so.10 with build-id b35d35f7f1116091086c8d9e793687b7094e4c6e
Module libxerces-c-3.2.so with build-id 9c2f2561bf4adcb4d6e56d085cd04fe677643729
Metadata for module libxerces-c-3.2.so owned by FDO found: {
"type" : "rpm",
"name" : "xerces-c",
"version" : "3.2.3-7.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libsaml.so.12 with build-id f3d01734686336ff3ca8c3ac47ab770ca4517002
Module libsystemd.so.0 with build-id 35de99f90f9858bb028b57fa6eef9d8238562527
Metadata for module libsystemd.so.0 owned by FDO found: {
"type" : "rpm",
"name" : "systemd",
"version" : "251.10-588.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libpthread.so.0 with build-id 622fcaae1f5e94890852b15a1705dff85830347b
Module libxml-security-c.so.20 with build-id 90e289f44ebabefffde6b8dfa8d58baef113177d
Metadata for module libxml-security-c.so.20 owned by FDO found: {
"type" : "rpm",
"name" : "xml-security-c",
"version" : "2.0.4-1.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module liblog4shib.so.2 with build-id 957262d3f0f25fe915c9c504d0b89c26ba4ae317
Module libgssapi_krb5.so.2 with build-id cdd24c78b746f8c52aaf668a2747814cc749ab56
Metadata for module libgssapi_krb5.so.2 owned by FDO found: {
"type" : "rpm",
"name" : "krb5",
"version" : "1.19.2-13.fc37",
"architecture" : "x86_64",
"osCpe" : "cpe:/o:fedoraproject:fedora:37"
}
Module libshibsp.so.11 with build-id 735f0c3f26340b6f1ef2bb85c92a2449d8ada315
Module shibd with build-id 3ea4dbd0dfc8356d224c99df5e6036e777582ea8
Stack trace of thread 90996:
#0 0x00007f0f66ab37fc pthread_rwlock_rdlock at GLIBC_2.2.5 (libc.so.6 + 0x907fc)
#1 0x00007f0f665baded CRYPTO_THREAD_read_lock (libcrypto.so.3 + 0x1baded)
#2 0x00007f0f665a09d7 evp_pkey_get_legacy (libcrypto.so.3 + 0x1a09d7)
#3 0x00007f0f689b51b0 _ZN19OpenSSLCryptoKeyRSAC1EP11evp_pkey_st (libxml-security-c.so.20 + 0xbd1b0)
#4 0x00007f0f6721dc42 _ZN10xmltooling14SecurityHelper15loadKeyFromFileEPKcS2_S2_ (libxmltooling.so.10 + 0x21dc42)
#5 0x00007f0f6720f830 _ZN10xmltooling28FilesystemCredentialResolverC1EPKN11xercesc_3_210DOMElementEb (libxmltooling.so.10 + 0x20f830)
#6 0x00007f0f672113f9 _ZN10xmltooling35FilesystemCredentialResolverFactoryERKPKN11xercesc_3_210DOMElementEb (libxmltooling.so.10 + 0x2113f9)
#7 0x00007f0f6720330f _ZN10xmltooling26ChainingCredentialResolverC1EPKN11xercesc_3_210DOMElementEb (libxmltooling.so.10 + 0x20330f)
#8 0x00007f0f672034c9 _ZN10xmltooling33ChainingCredentialResolverFactoryERKPKN11xercesc_3_210DOMElementEb (libxmltooling.so.10 + 0x2034c9)
#9 0x00007f0f68bc50f8 _ZNK10xmltooling13PluginManagerINS_18CredentialResolverENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKN11xercesc_3_210DOMElementEE9newPluginERKS7_RKSB_b (libshibsp.so.11 + 0x1c50f8)
#10 0x00007f0f68bc0636 _ZN6shibsp14XMLApplicationC1EPKNS_15ServiceProviderEPKNS_16ProtocolProviderEPN11xercesc_3_210DOMElementEbPKS0_PNS7_11DOMDocumentE (libshibsp.so.11 + 0x1c0636)
#11 0x00007f0f68bd8366 _ZN6shibsp13XMLConfigImplC1EPKN11xercesc_3_210DOMElementEbPNS_9XMLConfigERN8log4shib8CategoryE (libshibsp.so.11 + 0x1d8366)
#12 0x00007f0f68bd9aaa _ZN6shibsp9XMLConfig15background_loadEv (libshibsp.so.11 + 0x1d9aaa)
#13 0x00007f0f68b0e76a _ZN6shibsp8SPConfig11instantiateEPKcb (libshibsp.so.11 + 0x10e76a)
#14 0x0000564aec401ee9 main (shibd + 0x1ee9)
#15 0x00007f0f66a4a510 __libc_start_call_main (libc.so.6 + 0x27510)
#16 0x00007f0f66a4a5c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x275c9)
#17 0x0000564aec40253e _start (shibd + 0x253e)
Stack trace of thread 90999:
#0 0x00007f0f66aaad16 __futex_abstimed_wait_common (libc.so.6 + 0x87d16)
#1 0x00007f0f66aad510 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8a510)
#2 0x00007f0f671d3764 _ZN10xmltooling17ReloadableXMLFile9reload_fnEPv (libxmltooling.so.10 + 0x1d3764)
#3 0x00007f0f66aae12d start_thread (libc.so.6 + 0x8b12d)
#4 0x00007f0f66b2fbc0 __clone3 (libc.so.6 + 0x10cbc0)
Stack trace of thread 91000:
#0 0x00007f0f66aaad16 __futex_abstimed_wait_common (libc.so.6 + 0x87d16)
#1 0x00007f0f66aad510 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8a510)
#2 0x00007f0f671d3764 _ZN10xmltooling17ReloadableXMLFile9reload_fnEPv (libxmltooling.so.10 + 0x1d3764)
#3 0x00007f0f66aae12d start_thread (libc.so.6 + 0x8b12d)
#4 0x00007f0f66b2fbc0 __clone3 (libc.so.6 + 0x10cbc0)
Stack trace of thread 90998:
#0 0x00007f0f66aaad16 __futex_abstimed_wait_common (libc.so.6 + 0x87d16)
#1 0x00007f0f66aad804 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8a804)
#2 0x00007f0f671deb50 _ZN10xmltooling12CondWaitImpl9timedwaitEPNS_5MutexEl (libxmltooling.so.10 + 0x1deb50)
#3 0x00007f0f671f728f _ZN10xmltooling20MemoryStorageService10cleanup_fnEPv (libxmltooling.so.10 + 0x1f728f)
#4 0x00007f0f66aae12d start_thread (libc.so.6 + 0x8b12d)
#5 0x00007f0f66b2fbc0 __clone3 (libc.so.6 + 0x10cbc0)
Stack trace of thread 90997:
#0 0x00007f0f66aaad16 __futex_abstimed_wait_common (libc.so.6 + 0x87d16)
#1 0x00007f0f66aad510 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8a510)
#2 0x00007f0f671d3764 _ZN10xmltooling17ReloadableXMLFile9reload_fnEPv (libxmltooling.so.10 + 0x1d3764)
#3 0x00007f0f66aae12d start_thread (libc.so.6 + 0x8b12d)
#4 0x00007f0f66b2fbc0 __clone3 (libc.so.6 + 0x10cbc0)
ELF object binary architecture: AMD x86-64
░░ Subject: Process 90996 (shibd) dumped core
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ Documentation: man:core(5)
░░
░░ Process 90996 (shibd) crashed and dumped core.
░░
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.
Feb 21 13:38:51 nusol-fedora systemd[1]: systemd-coredump at 55-91001-0.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit systemd-coredump at 55-91001-0.service has successfully entered the 'dead' state.
Feb 21 13:38:51 nusol-fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump at 55-91001-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 21 13:38:51 nusol-fedora systemd[1]: shibd.service: Main process exited, code=dumped, status=11/SEGV
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit shibd.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 11.
Feb 21 13:38:51 nusol-fedora systemd[1]: shibd.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit shibd.service has entered the 'failed' state with result 'core-dump'.
Feb 21 13:38:51 nusol-fedora systemd[1]: Failed to start shibd.service - Shibboleth Service Provider Daemon.
░░ Subject: A start job for unit shibd.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit shibd.service has finished with a failure.
░░
░░ The job identifier is 32677 and the job result is failed.
Feb 21 13:38:51 nusol-fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=shibd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Feb 21 13:38:51 nusol-fedora audit: BPF prog-id=0 op=UNLOAD
Feb 21 13:38:51 nusol-fedora audit: BPF prog-id=0 op=UNLOAD
Feb 21 13:38:51 nusol-fedora audit: BPF prog-id=0 op=UNLOAD
Feb 21 13:38:52 nusol-fedora abrt-server[91010]: Deleting problem directory ccpp-2023-02-21-13:38:52.187390-90996 (dup of ccpp-2023-02-17-17:41:03.357027-3476)
Feb 21 13:38:53 nusol-fedora abrt-notification[91083]: [🡕] Process 3476 (shibd) crashed in pthread_rwlock_rdlock at GLIBC_2.2.5()
░░ Subject: ABRT has detected unexpected termination: shibd
░░ Defined-By: ABRT
░░ Support: https://bugzilla.redhat.com/
░░ Documentation: man:abrt(1)
░░
░░ shibd killed by SIGSEGV
░░
░░ #1 [libc.so.6] pthread_rwlock_rdlock at GLIBC_2.2.5
░░ #2 [libcrypto.so.3] CRYPTO_THREAD_read_lock
░░ #3 [libcrypto.so.3] evp_pkey_get_legacy
░░ #4 [libxml-security-c.so.20] OpenSSLCryptoKeyRSA::OpenSSLCryptoKeyRSA(evp_pkey_st*)
░░ #5 [libxmltooling.so.10] xmltooling::SecurityHelper::loadKeyFromFile(char const*, char const*, char const*)
░░
░░ Use the abrt command-line tool for further analysis or to report
░░ the problem to the appropriate support site.
4) The failing servers have up-to-date software and are running on the 6.1 kernel. The working server is still on the 6.0 kernel, and I am holding back any further updates until I know what is causing the problem...I suspect things may break if I upgrade.
5) I came across an old thread in this list from 2013 that describes this issue almost to a T, with just minor differences in the OpenSSL errors codes, but otherwise seems identical in nature. The problem in that case was that the wrong version of Shibboleth was installed, but I have confirmed that we are using the same version (the CENTOS_8 release)...here is the contents of the repo file, which is identical across all three servers.
[shibboleth]
name=Shibboleth (CentOS_8)
# Please report any problems to https://shibboleth.atlassian.net/jira
type=rpm-md
mirrorlist=https://shibboleth.net/cgi-bin/mirrorlist.cgi/CentOS_8
gpgcheck=1
gpgkey=https://shibboleth.net/downloads/service-provider/RPMS/repomd.xml.key
https://shibboleth.net/downloads/service-provider/RPMS/cantor.repomd.xml.key
enabled=1
Please let me know if there is anything else I should provide to help diagnose the problem.
Regards,
Simon
More information about the users
mailing list