SP Crashing

[Cantor, Scott]

On 9/19/19, 1:33 PM, "dev on behalf of Dan McLaughlin" <dev-bounces at shibboleth.net on behalf of dmclaughlin at tech-consortium.com> wrote:

> The shibd.exe application fault has occurred in conjunction with
> module "unknown", "ntdll", and "libcrypto" as shown below.

I would be monitoring memory size and seeing how much it spikes but under Windows I have load tested plenty of times in the old days and pushed it very far without any problems apart from slowness. If I had to guess, you're perhaps using features that haven't been adequately tested and hitting bugs in the system that postdate a lot of that older testing and shaking out. Or there are regressions in V3.
 
> Are there memory tuning settings in the SP and or Apache that we
> should be looking at? 

Just avoiding prefork on Unix. The stack size should be much less than the default is, but it doesn't really matter as long as prefork isn't used. The thread count shouldn't ever get very large as long as the Apache thread count is reasonable. The logs I think will prefix the records with a thread ID so keeping an eye on how many there are is another data point. It would take a lot to cause problems, but with prefork you get 200+, and that will break.

Otherwise shibd doesn't need any tuning, apart from expiring sessions more often to reduce the running footprint if that's really a problem. On Windows, the size will shrink back down in concert with usage, vs. Linux where the peak memory usage will stay allocated to the process and never go down.

> Is there any SP logging that we could enable that you might think would help narrow in on the root
> cause?

The more the better, but the SP won't run well under load on DEBUG, it just grinds to a halt, or used to.

I don't really know how to debug anything post-mortem on Windows, get symbols in place, etc. On Linux the core dump under gdb frequently provides hints that tend to point at the right culprit. I ironically do all my development debugging on Windows, but all my crash research on Linux.

The fact is that crashes all over the place with no consistent stack trace really screams out of memory, that's just the one constant that can bite any code anywhere with no repeatability.

-- Scott