Shibboleth.Listener shutdown after XMLTooling.StorageService purge

[Cantor, Scott]

On 9/26/12 5:52 PM, "Janemarie Duh" <berryj at lafayette.edu> wrote:
>
>Today at around 12:30, users stopped being able to authenticate to the
>service. When I grepped the processes, I saw that shibd was no longer
>running.
> Shibd.log[0] showed that the storage service purged some
>records and that the Shibboleth listener shut down the worker thread. I
>restarted shibd. A load spike on the server occurred around this same
>time.

That might indicate the server ran out of RAM, which is possible if you're
running Apache without threads and/or aren't altering the default thread
stack size for the shibd process.

>I obviously need to prevent this from happening again since this is a
>high-profile/use system. What are the records that the storage service
>purges? Are they transaction records or session information? We are
>using the default storage service settings.

That has nothing to do with the crash (none of that log information is
relevant, it's all routine).

>Thanks for any leads or advice on what may have caused this shutdown.

The load spike being involved is a strong indicator of a memory issue
because of stack size and/or because of an excessive thread count from
using Apache without worker mode.

https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPApacheConfig

See last two paragraphs under Prepping Apache.

-- Scott