Interaction between Shib and basic (file) authn in Apache 2.4

Fri Jul 21 10:18:42 EDT 2017

Hi,

I'm trying to understand the expected interaction between the Shib SP and
basic (file) authentication for 'Require valid-user' in Apache.

In this scenario I have mod_shib enabled (i.e. LoadModule), but not activated
(i.e. no 'AuthType shibboleth' or other Shib-related Apache config). I am
trying to perform authentication using basic (file) authentication, allowing
any valid user:
    AuthName "Test Realm"
    AuthType Basic
    AuthUserFile /var/www/test/testsuite/auth/htpasswd/htpasswd-test
    Require valid-user
    Satisfy All

On our current servers (Apache 2.2/Shib SP 2.4.3) this works as expected -
the users in the htpasswd file are able to authenticate and are granted
access.

On our Apache 2.4 test servers (both Apache 2.4.18/Shib SP 2.5.3 and Apache
2.4.25/Shib SP 2.6.0) users in the htpasswd file are unable to authenticate -
it looks like mod_shib is picking these up and denying access due to the
absence of an active shib session: 
----
[Wed Jul 12 17:13:11.527199 2017] [ssl:info] [pid 1367] [client 10.0.2.15:52422] AH01964: Connection to child 1 established (server testserver.york.ac.uk:443)
[Wed Jul 12 17:13:11.531968 2017] [mod_shib:debug] [pid 1367] mod_shib.cpp(320): [client 10.0.2.15:52422] get_request_config created per-request structure
[Wed Jul 12 17:13:11.532005 2017] [ssl:debug] [pid 1367] ssl_engine_kernel.c(354): [client 10.0.2.15:52422] AH02034: Initial (No.1) HTTPS request received for child 1 (server testserver.york.ac.uk:443)
[Wed Jul 12 17:13:11.532143 2017] [mod_shib:debug] [pid 1367] mod_shib.cpp(1584): [client 10.0.2.15:52422] shib_base_check_authz found uninitialized request object
[Wed Jul 12 17:13:11.532155 2017] [authz_core:debug] [pid 1367] mod_authz_core.c(809): [client 10.0.2.15:52422] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Wed Jul 12 17:13:11.532170 2017] [authz_core:debug] [pid 1367] mod_authz_core.c(809): [client 10.0.2.15:52422] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Wed Jul 12 17:13:11.533433 2017] [mod_shib:debug] [pid 1367] mod_shib.cpp(320): [client 10.0.2.15:52422] get_request_config created per-request structure
[Wed Jul 12 17:13:11.533457 2017] [ssl:debug] [pid 1367] ssl_engine_kernel.c(354): [client 10.0.2.15:52422] AH02034: Subsequent (No.2) HTTPS request received for child 1 (server testserver.york.ac.uk:443)
[Wed Jul 12 17:13:11.533524 2017] [mod_shib:debug] [pid 1367] mod_shib.cpp(1584): [client 10.0.2.15:52422] shib_base_check_authz found uninitialized request object
[Wed Jul 12 17:13:11.533535 2017] [authz_core:debug] [pid 1367] mod_authz_core.c(809): [client 10.0.2.15:52422] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Wed Jul 12 17:13:11.533542 2017] [authz_core:debug] [pid 1367] mod_authz_core.c(809): [client 10.0.2.15:52422] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Wed Jul 12 17:13:11.533838 2017] [mod_shib:debug] [pid 1367] mod_shib.cpp(867): [client 10.0.2.15:52422] shib_auth_checker entered in pid (1367)
[Wed Jul 12 17:13:11.533892 2017] [mod_shib:debug] [pid 1367] mod_shib.cpp(437): [client 10.0.2.15:52422] htaccess: denying shib-access/valid-user rule, no active session
[Wed Jul 12 17:13:11.533903 2017] [authz_core:debug] [pid 1367] mod_authz_core.c(809): [client 10.0.2.15:52422] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Wed Jul 12 17:13:11.533910 2017] [authz_core:debug] [pid 1367] mod_authz_core.c(809): [client 10.0.2.15:52422] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Wed Jul 12 17:13:11.533916 2017] [authz_core:error] [pid 1367] [client 10.0.2.15:52422] AH01629: authorization failure (no authenticated user): /test/testsuite/auth/htpasswd/only/
----

If we add 'ShibCompatValidUser On' to the server configuration then this works
as before:
----
[Wed Jul 12 17:12:11.607131 2017] [ssl:info] [pid 1278] [client 10.0.2.15:52420] AH01964: Connection to child 1 established (server testserver.york.ac.uk:443)
[Wed Jul 12 17:12:11.613349 2017] [ssl:debug] [pid 1278] ssl_engine_kernel.c(354): [client 10.0.2.15:52420] AH02034: Initial (No.1) HTTPS request received for child 1 (server testserver.york.ac.uk:443)
[Wed Jul 12 17:12:11.613558 2017] [authz_core:debug] [pid 1278] mod_authz_core.c(809): [client 10.0.2.15:52420] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Wed Jul 12 17:12:11.613578 2017] [authz_core:debug] [pid 1278] mod_authz_core.c(809): [client 10.0.2.15:52420] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Wed Jul 12 17:12:11.615625 2017] [ssl:debug] [pid 1278] ssl_engine_kernel.c(354): [client 10.0.2.15:52420] AH02034: Subsequent (No.2) HTTPS request received for child 1 (server testserver.york.ac.uk:443)
[Wed Jul 12 17:12:11.615777 2017] [authz_core:debug] [pid 1278] mod_authz_core.c(809): [client 10.0.2.15:52420] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Wed Jul 12 17:12:11.615797 2017] [authz_core:debug] [pid 1278] mod_authz_core.c(809): [client 10.0.2.15:52420] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Wed Jul 12 17:12:11.616975 2017] [authz_core:debug] [pid 1278] mod_authz_core.c(809): [client 10.0.2.15:52420] AH01626: authorization result of Require valid-user : granted
[Wed Jul 12 17:12:11.616998 2017] [authz_core:debug] [pid 1278] mod_authz_core.c(809): [client 10.0.2.15:52420] AH01626: authorization result of <RequireAny>: granted
----

The docs:
https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPApacheConfig
https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPhtaccess
are quite clear at the start of the first page and at the end of the second
page that authnz in Apache 2.4 is more complicated than in 2.2 and that
ShibCompatValidUser should be turned on for compatibility.

I also understand the other, more detailed, docs regarding the difference
in behaviour of valid-user with and without this option, although I had
assumed that the use of ShibCompatValidUser would only be relevant when the
module had been activated (i.e. AuthType shibboleth).

I suppose what I've found confusing are the references to 'older behaviour',
which I've interpreted as corresponding to the behaviour that I am observing
on my current servers (when mod_shib is not activated), but which is clearly
only referring to the previous behaviour *when mod_shib is activated*.

What is the expected behaviour when mod_shib is loaded, but not activated? On
my current servers it doesn't interfere with basic authentication and allows
users to log in, however on more recent servers it prevents authentication
even when it isn't activated (unless ShibCompatValidUser is set to On).

I'd be grateful for any clarification that can be provided.

Thanks,
    Chris