SP Crashing

[Dan McLaughlin]

Where is the debug version of the shibd.exe?  It use to be under a
./debug folder, but it's not there anymore.  Do you have the debug
version or symbols available for download somewhere?

--

Thanks,

Dan

On Thu, Sep 19, 2019 at 4:37 PM Dan McLaughlin
<dmclaughlin at tech-consortium.com> wrote:
>
> I have the debug symbols loaded and SP source downloaded with Windbg
> installed.  Here's what I've found out so far about the two crashes
> this morning, unfortunately we have apps that come grab all the WER
> reports daily, so I don't have access to the dumps from the other
> crashes at this time.  Is there any logging I could enable that might
> help point me in the right direction?
>
> shibd.exe Stopped working 9/19/2019 8:31:17 AM spservice 0xc0000005
> Access Violation 0x0002a365 ntdll.dll 6.1.7601.24511
> E:\servers\shib-sp\sbin64\shibd.exe 12,226
>
> shibd.exe Stopped working 9/19/2019 10:21:50 AM spservice 0xc0000374 A
> heap has been corrupted.
>  0x000bf302 StackHash_1aea 6.1.7601.24511
>
> --
>
> Thanks,
>
> Dan
>
> On Thu, Sep 19, 2019 at 12:55 PM Cantor, Scott <cantor.2 at osu.edu> wrote:
> >
> > On 9/19/19, 1:33 PM, "dev on behalf of Dan McLaughlin" <dev-bounces at shibboleth.net on behalf of dmclaughlin at tech-consortium.com> wrote:
> >
> > > The shibd.exe application fault has occurred in conjunction with
> > > module "unknown", "ntdll", and "libcrypto" as shown below.
> >
> > I would be monitoring memory size and seeing how much it spikes but under Windows I have load tested plenty of times in the old days and pushed it very far without any problems apart from slowness. If I had to guess, you're perhaps using features that haven't been adequately tested and hitting bugs in the system that postdate a lot of that older testing and shaking out. Or there are regressions in V3.
> >
> > > Are there memory tuning settings in the SP and or Apache that we
> > > should be looking at?
> >
> > Just avoiding prefork on Unix. The stack size should be much less than the default is, but it doesn't really matter as long as prefork isn't used. The thread count shouldn't ever get very large as long as the Apache thread count is reasonable. The logs I think will prefix the records with a thread ID so keeping an eye on how many there are is another data point. It would take a lot to cause problems, but with prefork you get 200+, and that will break.
> >
> > Otherwise shibd doesn't need any tuning, apart from expiring sessions more often to reduce the running footprint if that's really a problem. On Windows, the size will shrink back down in concert with usage, vs. Linux where the peak memory usage will stay allocated to the process and never go down.
> >
> > > Is there any SP logging that we could enable that you might think would help narrow in on the root
> > > cause?
> >
> > The more the better, but the SP won't run well under load on DEBUG, it just grinds to a halt, or used to.
> >
> > I don't really know how to debug anything post-mortem on Windows, get symbols in place, etc. On Linux the core dump under gdb frequently provides hints that tend to point at the right culprit. I ironically do all my development debugging on Windows, but all my crash research on Linux.
> >
> > The fact is that crashes all over the place with no consistent stack trace really screams out of memory, that's just the one constant that can bite any code anywhere with no repeatability.
> >
> > -- Scott
> >
> >
> > --
> > To unsubscribe from this list send an email to dev-unsubscribe at shibboleth.net