Session Recovery after a shibd restart

[Simon Leary]

Hello shib users,

I wanted to make sure that shibd can be restarted without users having to re-authenticate.
I tried out memcached, but I found that switching the storage service would require all users to clear their cookies or else they get a cryptic error message when their old "ss:mem" session ID becomes invalid.
I tried Session Recovery but it was giving me problems. Sometimes, I could see in the logs that the sessions were successfully recovered:

```
DEBUG Shibboleth.SessionCache [2] [default]: checking for revocation of session (_3e183b62b5f1ee735bcde24c1735041c)
DEBUG Shibboleth.SessionCache [2] [default]: attempting recovery of session (_3e183b62b5f1ee735bcde24c1735041c)
DEBUG Shibboleth.SessionCache [2] [default]: storing recovered session (_3e183b62b5f1ee735bcde24c1735041c)...
DEBUG XMLTooling.StorageService [2] [default]: inserted record (session) in context (_3e183b62b5f1ee735bcde24c1735041c) with expiration (1785295165)
DEBUG XMLTooling.StorageService [2] [default]: inserted record (_e53c2a7d6ab4c935add7aedcf60013d1943bcd15af) in context (NameID) with expiration (1785279350)
INFO Shibboleth.SessionCache [2] [default]: session recovered: ID (_3e183b62b5f1ee735bcde24c1735041c) IdP (https://webauth.umass.edu/idp/shibboleth) Protocol(urn:oasis:names:tc:SAML:2.0:protocol)
DEBUG XMLTooling.StorageService [2] [default]: updated expiration of valid records in context (_3e183b62b5f1ee735bcde24c1735041c) to (1785295165)
```

But other times, I would see "session is not available in the session store" in native log (syslog), and in shibd.log there was no attempt made to recover the session:

```
DEBUG Shibboleth.SessionCache [758291] shib_check_user [default]: session found locally, validating it for use
...
ERROR Shibboleth.Listener [758291] shib_check_user [default]: remoted message returned an error: Your session is not available in the session store, and you must re-authenticate.
DEBUG Shibboleth.SessionCache [758291] shib_check_user [default]: deleting local copy of session (_e03afa537125b57032d5e86b1e8bf73b)
...
WARN Shibboleth.ServiceProvider [758291] shib_check_user [default]: error during session lookup: Your session is not available in the session store, and you must re-authenticate.
```

Since I can see "Shibboleth.SessionCache" lines in both native log and shibd.log, I assume that the apache module is reading the session cache from shibd and is keeping its own local copy. Based on these logs, it looks like the apache module has a cache hit based on its local copy, but then is surprised to see that the session is actually absent from the proper session cache. I imagine that the SP could better handle this edge case. The apache module could treat it the same as a normal cache miss and fall back to Session Recovery. It could also flush its local copy if it detects a de-sync.

Here's what I have observed from various types of restarting:

* `systemctl reload apache2`:
    * works fine
* `systemctl restart shibd`:
    * no session recovery is attempted, must re-auth
* `systemctl restart shibd && systemctl reload apache2`:
    * works fine under basic testing
    * breaks with lots of ongoing requests
    * if any request arrives between the restart and reload, broken same as above
* `systemctl reload apache2 && systemctl restart shibd`:
    * broken same as above
* `systemctl reload-or-restart apache2 shibd`:
    * works fine (!)

I think that `systemctl reload-or-restart apache2 shibd` is the best I can do for reducing the window of time that a request can arrive after shibd has been restarted and before apache2 has been reloaded. I assume if I tried hard enough I could still get my session to break, but this is good enough for me.

I saw [an earlier thread called "When restart apache do I restart shibboleth"](https://groups.google.com/g/shibboleth-users/c/gS7BijkL-wU/m/cynS1R3ztDkJ) and it seems like the consensus was there is usually no need to restart apache unless you are messing with apache-specific configuration (like the native.logger config file!). It seems to me that when using Session Recovery, this is no longer true, and apache must be reloaded whenever shibd is restarted to prevent users from having to re-auth.

Anyways, I don't have any open questions, I just thought I would share what I learned (what I think I learned) and suggest that change.

Cheers,
Simon