Shidb / Apache Segmentation Fault

[Nate Klingenstein]

Aleix,

I am not a developer, but…

My hunch is that your shibd.sock socket file is experiencing “something bad on Sunday morning” due to external forces somewhere else in your environment.  Scott or one of the other developers may be able to enumerate the possible "something bad"s and figure what’s going on, but you might gather a little more data by switching from a UnixListener to a TCPListener and seeing what happens.

https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPTCPListener
https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPUnixListener

Just gathering more info,
Nate.

> On Nov 16, 2014, at 1:26 AM, Aleix Dorca Josa <adorca at uda.ad> wrote:
> 
> Hi all,
> 
> I have this problem that is driving me nuts.
> 
> I'm using a standard Shibboleth IdP installation (v2.4) on a Debian Wheezy. I have Native SPs (via Apache's Shibd Module) on OSs like Debian Squeeze and Wheezy. The whole setup works flawlessly except for a particular application.
> 
> We have two servers using a load sharing scheme. These use Debian Wheezy, Apache, Moodle, GlusterFS to sync the data folder and the Shibd module to handle the authentication.
> 
> Everything (except Moodle, of course) has been installed from the Debian repositories.
> 
> The problem is: every Sunday at 7:00 (more or less, just after the whole cron stuff) Apache starts logging segmentation faults (11) whenever a user tries to log in.
> 
> I know you'll say that I have some weird cron job on Sundays, but no, nothing at all. Maybe a memory problem? Seems obvious, but the system has plenty to spare. I could even add more if necessary.
> 
> The problem is only solved if I reboot the machine. Some times TWO reboots are necessary and I can't find anything at all on the logs that points me to solving the problem.
> Restarting the services Apache, Shibd or GlusterFS has no effect at all. This strikes me as odd too.
> 
> This would be a typical Apache error.log entry:
> 
> [Sun Nov 16 07:20:05 2014] [notice] child pid 7969 exit signal Segmentation fault (11)
> 
> And the shibboleth.log associated to the seg fault is:
> 
> 2014-11-16 07:20:05 INFO Shibboleth.SessionCache [43]: new session created: ID (_whatever) IdP (https://idp.uda.ad/idp/shibboleth) Protocol(urn:oasis:names:tc:SAML:2.0:protocol)
> 2014-11-16 07:20:05 INFO Shibboleth.Listener [44]: detected socket closure, shutting down worker thread
> 
> Previously I have found this:
> 
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [33]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [36]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [38]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [37]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [39]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [20]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [24]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [40]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [34]: detected socket closure, shutting down worker thread
> 2014-11-16 06:42:49 INFO Shibboleth.Listener [19]: detected socket closure, shutting down worker thread
> 
> This has only been logged on Sunday and it seems to be the moment everything goes bananas.
> 
> As you can see the system is really low on load at this time of day (45 minutes for the first login to happen) so an extreme load problem is probably out of the question too. :-(
> 
> I'd really appreciate any suggestions. If you need more information please let me know.
> 
> Thanks,
> 
> Aleix.
> 
> -- 
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net