Question about Shibboleth IdP v5 with Jetty 12 and extra RemoteUser headers for TLS termination by F5

[Koeritz, Chris (cak0l)]

Dear Shibboleth Users List,
We are migrating from Shibboleth IdP v4 to v5 and have run into a snag.
Since both jetty 10 and jetty 11 are due to be EOL in January of next year, we decided to jump to jetty 12.
We did the important standard things first—upgraded to the latest IdP V4, removed any deprecation warnings, then upgraded to the latest IdP V5.
That is mostly working okay, after figuring out some oddities about the Jetty 12 configuration (using the Shibboleth provided configs as Jetty base), except for one important thing: our certificate-based login.
We use the F5 BigIP load balancer as the front end for the SSO logins, and it does the TLS termination.  The result of the TLS termination is that, after getting a valid certificate from the user, the F5 sets a header called X-Remote-User and passes that along to the IdP on port 8080 (so regular http instead of https after that point).  That identity from the certificate is used instead of a username and password combo.
This worked great in jetty 9, but it is not working the same way in jetty 12.
I’ve enabled the http-forwarded module in jetty 12, and that’s actually required due to the X-Forwarded-For and X-Forwarded-Proto headers that are also passed by the F5.
We’ve got the new style authn.properties file set up with these header values, so that the auto-wired RemoteUser bean should be getting the header values passed in.
However, after a lot of different configuration attempts, we are still not seeing the X-Remote-User header’s value that we need.  There are also X-Authn-Context-Class headers that we need, and I suspect those are also missing, although it’s hard to know since we’re blocked by the missing X-Remote-User header, with that important user ID from the certificate.
The result so far is always that Shibboleth IdP shows a failure when attempting the certificate login.  The web UI says “User login was not successful or could not meet the requirements of the requesting application” for certificate logins.  Username and password work fine though.  Here is some example logging when the certificate login is attempted.
====
2024-07-22 15:09:52,726 - 172.18.8.29 - DEBUG [net.shibboleth.idp.authn.impl.TransitionMultiFactorAuthentication:233] - Profile Action TransitionMultiFactorAuthentication: MFA flow transition after 'UseCert' event to 'authn/RemoteUser' flow
2024-07-22 15:09:54,676 - 172.18.8.29 - INFO [net.shibboleth.idp.authn.impl.RemoteUserAuthServlet:277] - e1s3: User identity not found in request
2024-07-22 15:09:54,697 - 172.18.8.29 - INFO [net.shibboleth.idp.authn.impl.ValidateExternalAuthentication:201] - Profile Action ValidateExternalAuthentication: External authentication failed, no user identity or error information returned
2024-07-22 15:09:54,698 - 172.18.8.29 - DEBUG [net.shibboleth.idp.authn.impl.TransitionMultiFactorAuthentication:212] - Profile Action TransitionMultiFactorAuthentication: Applying MFA transition rule to exit state 'authn/RemoteUser'
2024-07-22 15:09:54,699 - 172.18.8.29 - DEBUG [net.shibboleth.idp.authn.impl.TransitionMultiFactorAuthentication:233] - Profile Action TransitionMultiFactorAuthentication: MFA flow transition after 'ReselectFlow' event to 'authn/Password' flow
2024-07-22 15:09:54,699 - 172.18.8.29 - DEBUG [net.shibboleth.idp.authn.impl.ExtractUsernamePasswordFromBasicAuth:116] - Profile Action ExtractUsernamePasswordFromBasicAuth: No appropriate Authorization header found
====
Here’s the relevant section of authn.properties for the RemoteUser configuration:
====
idp.authn.RemoteUser.order = 2   ##(had left this at the original 1000, and it still did not work.  The “2” was just what was set in the old v4 config in web.xml--that file has been eliminated for shib idp v5 in our config.)
#idp.authn.RemoteUser.nonBrowserSupported = false
#idp.authn.RemoteUser.matchExpression =
# Unset in most cases only if using the authnMethodHeader or
# subjectAttribute settings
#idp.authn.RemoteUser.addDefaultPrincipals = false
idp.authn.RemoteUser.checkRemoteUser = false  ##(was turned off in old web.xml, and also disabled here, since we have a specific header below and are not expecting REMOTE_USER.)
# Comma-delimited lists of attributes or headers to pull from
#idp.authn.RemoteUser.checkAttributes =
idp.authn.RemoteUser.checkHeaders = X-Remote-User
# Advanced settings
#idp.authn.RemoteUser.subjectAttribute =
idp.authn.RemoteUser.authnMethodHeader = X-Authn-Context-Class
#idp.authn.RemoteUser.authnAuthorityHeader =
====
I’ve run tcpdump and verified that the headers are still present when they hit jetty 12.  Here’s a chunk of the tcpdump output showing the additional headers:
====
GET /idp/Authn/RemoteUser?conversation=e1s3 HTTP/1.1
Host: shibidp.its.virginia.edu
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Referer: https://shibidp.its.virginia.edu/idp/profile/SAML2/POST/SSO?execution=e1s2
DNT: 1
Sec-GPC: 1
Connection: keep-alive
Cookie: __Host-JSESSIONID=node07b5x83hhpv3w19a8v56vz0ycq230.node0; LastMRH_Session=8866f7df; F5_ST=1z1z1z1720212694z300
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Priority: u=1
X-Forwarded-For: 10.18.0.146
X-Forwarded-Proto: https
X-Remote-User: XXXXX  ##(anonymized, but has the correct user ID here.)
X-Authn-Context-Class: urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
X-Authn-Context-Class: urn:oasis:names:tc:SAML:2.0:ac:classes:Password
====
I have not been very successful with logging in Jetty or Shibboleth IdP to show the header processing.  If I set the main Jetty logback config to “DEBUG”, then Jetty goes into an infinite loop (seemingly) and never starts up.  That is, if I set:
  <root level="DEBUG">
    <appender-ref ref="jetty" />
  </root>
Then I get these log entries repeatedly, and they do not stop until the jetty startup times out (even with startup time set to 300 seconds):
====
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.java.javaURLContextFactory:55] - >>> new root context requested
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.NamingContext:429] - Looking up name="comp/env/opensaml.config.partitionName"
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.ContextFactory:102] - Trying thread context classloader
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.NamingContext:429] - Looking up name="env/opensaml.config.partitionName"
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.NamingContext:429] - Looking up name="opensaml.config.partitionName"
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.InitialContextFactory:74] - InitialContextFactory.getInitialContext()
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.InitialContextFactory:78] - Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot at 2efc5a8
2024-07-22 14:50:08,138 - DEBUG [org.eclipse.jetty.jndi.local.localContextRoot:273] - Looking up name="opensaml.config.partitionName\"
2024-07-22 14:50:08,139 - DEBUG [org.eclipse.jetty.jndi.InitialContextFactory:74] - InitialContextFactory.getInitialContext()
2024-07-22 14:50:08,139 - DEBUG [org.eclipse.jetty.jndi.InitialContextFactory:78] - Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot at 69affb4d
====
With the normal “INFO” level logging, Jetty starts up within 39 seconds on our development host.
I’ve set the Shib logback configs to DEBUG or even TRACE for selected cases, but have not seen the processing of the headers in the idp-process.log either.

So my question is: Does anyone have any experience with similar RemoteUser headers being passed through Jetty 12 and on to Shibboleth IdP successfully?  Are there some additional configuration items we need to set for this?
And has anyone seen similar wacky infinite loop behavior when Jetty is set at DEBUG level of logging?  That behavior appears to be a bug, since the only difference in the configuration is the log level.
At this point, we are not sure whether Jetty is dropping the X-Remote-User header before Shibboleth IdP sees it, or if Shibboleth IdP is somehow not configured properly to see the header passed in for the RemoteUser bean.
Our survey of the existing literature on the web has not provided much insight into this problem, but we’re hoping folks on this list may know more.
Everything else in the converted Shibboleth IdP v5 configuration seems to be working properly, e.g. username+password logins are great, LDAP lookups are fine, our web UI looks great, etc.
Any help would be very welcome.  I can provide more details of our configuration if needed.
Thanks,
Chris Koeritz

--
Chris Koeritz
Senior Linux and Storage Engineer
University of Virginia ITS - Backup, Storage, and Archive Services
P: 434 982 4690


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20240722/bb1bb372/attachment.htm>