Issue with _shibsession_ in Load Balanced Setup

[Special Monkey]

I spoke with our IdP admin who suggested making the ALB timeout higher than
the <Sessions lifetime="28800" (in shibboleth2.xml) or above 8 hours -- it
had been 10 mins.

This seems to be a step in the right direction. I haven't seen the
phpMyAdmin error again since upping the ALB timeout to 9 hours.

The long ALB timeout is a bit of a Band-Aid solution since it essentially
prevents the node from switching during a typical browser session, which
the app cannot seem to handle gracefully, i.e. without eventually
presenting the reported error -- even with shared PHP sessions.

The app is relying on the Shibboleth session and not the PHP session --
seemingly proved when deleting the phpMyAdmin session cookie does not cause
the error but deleting the Shibboleth session cookie does.

A Band-Aid is arguably better than no Band-Aid. I think this one is
limiting the load balancing by remaining so sticky, but is utilizing some
aspects of load balancing by directing clients to different nodes, and if a
node goes down, to the node that is hopefully still up, which I think would
cause the same app error, though much less frequently (e.g., only when a
node the client is on becomes unavailable or after 9 hours?).

I'm curious what the recommended approach to this issue is, assuming
something similar happens to other load balanced Shibboleth protected apps?
Since the app is able to regain a session on a reload of the page, from the
different node's SP setup, I wonder if the app (or some SP config change)
could make an accommodation, or check for a new session, before producing
the error, and requiring a reload of the page to continue.

The admin also referred me to the following:

https://wiki.shibboleth.net/confluence/display/SP3/Clustering?src=contextnavpagetreemode
https://wiki.shibboleth.net/confluence/display/SP3/ClusteringByProxy






On Thu, Nov 14, 2019 at 1:50 PM Special Monkey <specialmonkey at gmail.com>
wrote:

> I have a 2 node load balanced phpMyAdmin setup. The nodes are behind an
> AWS ALB.
>
> The cloud team mentioned this with regard to the ALB:
>
> 1 it utilizes cookies, so as long as clients browser settings are OK, the
> session will remain with a single node
> 2 the stickiness duration is currently set to 10 minutes, which means as
> long as requests are being sent between the client and lb within that time
> frame, the session will remain sticky so to speak
> but if they walk away, 10 minutes elapses, then they start interacting
> with the app again, it's possible they get bounced to the other node
>
> I notice the AWSALB cookie is constantly changing, I understand this is
> normal and is how stickiness works with the AWS ALB.
>
> I also see the _shibsession_xxx session cookie.
>
> Every so often, the app (phpMyAdmin) complains that it has lost its
> connection with the following error displayed in the web UI:
>
>
>
> *Error in processing requestError text: error (rejected)It seems that the
> connection to server has been lost. Please check your network connectivity
> and server status.*
>
> This happens after some idle time, and then clicking on, for example, a
> table, to see its contents (or seemingly any link within the app's web UI).
>
>
> The app is setup with protected paths in the Apache's shibd.conf file, for
> example:
>
> <Location /secure/bas>
>   AuthType shibboleth
>   ShibRequestSetting requireSession true
> <RequireAll>
>   Require shib-attr isMemberOf
> cn=bas,ou=groups,ou=db,ou=app,ou=foo,ou=Groups,o=foo.com,o=foo
>   Require shib-attr isMemberOf
> cn=everyone,ou=groups,ou=db,ou=app,ou=foo,ou=Groups,o=foo.com,o=foo
> </RequireAll>
> </Location>
>
> each of which (i.e., /secure/bas, etc.) are phpMyAdmin installs with
> auth_type = config setups -- where user & password are hard coded into the
> config.inc.php file, since auth is handled by Shibboleth.
>
> phpMyAdmin connects via TLS to a MariaDB cluster. The connect details to
> MariaDB are also within each config.inc.php.
>
> The following is in shibboleth2.xml:
>
> <Sessions lifetime="28800" timeout="3600" relayState="ss:mem"
>           checkAddress="false" consistentAddress="false" handlerSSL="true"
> cookieProps="https">
>
> I set session.gc_maxlifetime = 14400 in php.ini. The server is using Red
> Hat:
>
> - Apache 2.4.6
> - Shibd 3.04
> - PHP 7.3
>
> Initially, one issue was resolved by setting consistentAddress="false" as
> above ... since the ALB actually has 2 source IPs at any time, and when
> they changed, it seemed to correspond to the error occurring.
>
> consistentAddress="false" seemed to resolve -- but the issue still cropped
> up, albeit less frequently.
>
> I thought the issue might be related to the app not sharing sessions, so I
> tried configuring PHP to use Redis to share sessions between the 2 nodes,
> however, the issue still happens.
>
> Deleting the phpMyAdmin session cookie does not seem to cause the problem.
>
> Deleting the _shibsession_xxx cookie, DOES produce the error.
>
> Some things I notice when the error happens,
>
> - the link I click responds with a 302 to the IdP
> - the node that responded with the 302 is different from the node the
> browser had been on
> - the _shibsession_xxx session cookie that had been present disappears
> from local storage, and the app error:
>
>
>
> *Error in processing requestError text: error (rejected)It seems that the
> connection to server has been lost. Please check your network connectivity
> and server status.*
>
> is presented in the app's web UI.
>
> I see in the 302 provisional headers a _shibsession_xxx and a
> _shibsealed_xxx cookie -- the names for both are the same. I can't see the
> values.
>
> Reloading the page fixes the error, and now the browser is sticking to the
> alternate node. The _shibsession_xxx cookie is back and has a new value on
> reloading. Everything is fine until it switches again.
>
> Thanks for any info on how I might debug.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20191115/6008c361/attachment.html>