Shidb / Apache Segmentation Fault

[Aleix Dorca Josa]

Hi all,

I have this problem that is driving me nuts.

I'm using a standard Shibboleth IdP installation (v2.4) on a Debian Wheezy. I have Native SPs (via Apache's Shibd Module) on OSs like Debian Squeeze and Wheezy. The whole setup works flawlessly except for a particular application.

We have two servers using a load sharing scheme. These use Debian Wheezy, Apache, Moodle, GlusterFS to sync the data folder and the Shibd module to handle the authentication.

Everything (except Moodle, of course) has been installed from the Debian repositories.

The problem is: every Sunday at 7:00 (more or less, just after the whole cron stuff) Apache starts logging segmentation faults (11) whenever a user tries to log in.

I know you'll say that I have some weird cron job on Sundays, but no, nothing at all. Maybe a memory problem? Seems obvious, but the system has plenty to spare. I could even add more if necessary.

The problem is only solved if I reboot the machine. Some times TWO reboots are necessary and I can't find anything at all on the logs that points me to solving the problem.
Restarting the services Apache, Shibd or GlusterFS has no effect at all. This strikes me as odd too.

This would be a typical Apache error.log entry:

[Sun Nov 16 07:20:05 2014] [notice] child pid 7969 exit signal Segmentation fault (11)

And the shibboleth.log associated to the seg fault is:

2014-11-16 07:20:05 INFO Shibboleth.SessionCache [43]: new session created: ID (_whatever) IdP (https://idp.uda.ad/idp/shibboleth) Protocol(urn:oasis:names:tc:SAML:2.0:protocol)
2014-11-16 07:20:05 INFO Shibboleth.Listener [44]: detected socket closure, shutting down worker thread

Previously I have found this:

2014-11-16 06:42:49 INFO Shibboleth.Listener [33]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [36]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [38]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [37]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [39]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [20]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [24]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [40]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [34]: detected socket closure, shutting down worker thread
2014-11-16 06:42:49 INFO Shibboleth.Listener [19]: detected socket closure, shutting down worker thread

This has only been logged on Sunday and it seems to be the moment everything goes bananas.

As you can see the system is really low on load at this time of day (45 minutes for the first login to happen) so an extreme load problem is probably out of the question too. :-(

I'd really appreciate any suggestions. If you need more information please let me know.

Thanks,

Aleix.