shibd.exe crash after upgrade to shibboleth-sp-3.2.3.1

[j.grootlipman at facilitor.nl]

On 6 different Windows 2016 servers we have been running Shibboleth SP version 3.1.0.2 for over a year and it has been rock stable.

A week ago we upgraded 2 of the servers to version 3.2.3.1 with shibboleth-sp-3.2.3.1-win64.msi (no other configuration changes). Initially this worked fine but on both servers shibd.exe has crashed since.

On one server the Shibboleth service crashed after about 4 days, on the other server after about 7 days. Simply starting the service again restored functionality.

I happen to have logging set to DEBUG and it seems the crash is in the automatic reloading of metadata: with all crashes the last line in shibd.log is something like
2021-11-11 17:18:17 DEBUG OpenSAML.MetadataProvider.XML : writing to backing file: C:/opt/shibboleth-sp/var/cache/shibboleth/metadata-kfsg-oud.xml.a05e
And it is *not* followed by something like ‘2021-11-11 21:32:53 DEBUG XMLTooling.util.CloneInputStream : initialized’
The crash seems to happen before the “XMLTooling.util.CloneInputStream”

I was able to capture a crash memory dump:
Unhandled exception at 0x00007FFCDF357143 (ntdll.dll) in memory.hdmp: 0xC0000374: A heap has been corrupted (parameters: 0x00007FFCDF3AD6B0).

Stack trace:
            ntdll.dll!RtlReportCriticalFailure‑()            Unknown
>          ntdll.dll!RtlpHeapHandleError‑()  Unknown
           ntdll.dll!RtlpLogHeapFailure‑()    Unknown
           ntdll.dll!RtlFreeHeap‑()   Unknown
           [Frames may be missing, no binary loaded for ucrtbase.dll]
           ucrtbase.dll!00007ffcdb8ade0b() Unknown

Does this information help to pinpoint the source of the crash?

Jos Groot Lipman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20211111/527c536d/attachment.htm>