SP Crashing

[Dan McLaughlin]

We are running Apache/2.4.37 (Win64) OpenSSL/1.1.1 mod_jk/1.2.46 on
Windows 2008 SP2 and Shibboleth 3.0.4.   Over the past few months
we've started experiencing issues with the SP crashing during peak
load.  We see it crash in the mornings and afternoons before 5PM. We
will usually peak at about 1000-1200 logged in users with concurrent
requests approaching 200 in the mornings and slightly lower in the
afternoons.

The shibd.exe application fault has occurred in conjunction with
module "unknown", "ntdll", and "libcrypto" as shown below.

Are there memory tuning settings in the SP and or Apache that we
should be looking at?  Are there any best practices for matching up
Apache tuning with SP tuning related to threads, timeouts, etc... that
would should make sure are in sync?  Is there any SP logging that we
could enable that you might think would help narrow in on the root
cause?

Faulting application name: shibd.exe, version: 3.0.4.0, time stamp: 0x5c81b93a
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000000
Faulting process id: 0x5ee4
Faulting application start time: 0x01d56cc5ddbccdd9
Faulting application path: E:\servers\shib-sp\sbin64\shibd.exe
Faulting module path: unknown
Report Id: 5b3b80ab-d95e-11e9-bb81-005056010496

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0

Faulting application name: shibd.exe, version: 3.0.4.0, time stamp: 0x5c81b93a
Faulting module name: libcrypto-1_1_1-x64.dll, version: 1.1.1.1, time
stamp: 0x5c0f2b49
Exception code: 0xc0000005
Fault offset: 0x00000000001783a5
Faulting process id: 0x5cc8
Faulting application start time: 0x01d56d6b4451897b
Faulting application path: E:\apps\shib-sp\sbin64\shibd.exe
Faulting module path: C:\Program Files\Shibboleth\SP\lib\libcrypto-1_1_1-x64.dll
Report Id: 8469cccb-d95e-11e9-bb81-005056010496

Faulting application name: shibd.exe, version: 3.0.4.0, time stamp: 0x5c81b93a
Faulting module name: ntdll.dll, version: 6.1.7601.24511, time stamp: 0x5d3fa9bd
Exception code: 0xc0000005
Fault offset: 0x000000000002a365
Faulting process id: 0x4f3c
Faulting application start time: 0x01d56d6b6b702f7b
Faulting application path: E:\apps\shib-sp\sbin64\shibd.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: eb2b6d4a-d989-11e9-bb81-005056010496


--

Thanks,

Dan