Are all request headers accessible by Shib's Logback?

[John Morton]

Simon and anyone who is interested... 

Just thought I'd update you on my progress. I was able to get X-MS-Forwarded-Client-IP to Shib by adding a second valve to our server.xml. We waffled on whether to trust all IPs on injecting X-MS-Forwarded-Client-IP and decided to come up with a regex to represent Exchange Online. We got them from here: https://support.content.office.net/en-us/static/O365IPAddresses.xml, specifically the "EXO" (Exchange Online) section.  If upkeep of that regex becomes burdensome, we will probably revisit and potentially use "\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}".

There are two other useful headers MS injects, but for now they are only in the Tomcat access log (see access log valve below). We were not able to come up with any easy way of getting those into Shib-managed logs. I have an idea of writing a servlet filter that moves them into the regular User Agent field that is instrumented in Shib, but I'm not going to spend the time on it now. Ideally, that would be paired with an update of Shib's SLF4JMDCServletFilter.java class so that user agent was accessible in a Logback pattern.

        <!-- This valve rewrites the ClientIP in the request with the X-Forwarded-For header if the ClientIP is our Load Balancer -->
        <Valve className="org.apache.catalina.valves.RemoteIpValve"
               remoteIpHeader="X-Forwarded-For"
               internalProxies="XXX\.XXX\.XXX\.XXX"
               protocolHeader="X-Forwarded-Proto" />

        <!-- This valve rewrites the ClientIP in the HTTP request with the X-MS-Forwarded-Client-IP header if the ClientIP is in Microsoft's Exchange Online (EXO) subnet list -->
        <Valve className="org.apache.catalina.valves.RemoteIpValve"
               remoteIpHeader="X-MS-Forwarded-Client-IP"
               internalProxies="13\.107\.6\.15[2-3]|13\.107\.9\.15[2-3]|13\.107\.18\.1[0-1]|13\.107\.19\.1[0-1]|23\.103\.16[0-9]\.\d{1,3}|23\.103\.17[0-5]\.\d{1,3}|23\.103\.22[4-9]\.\d{1,3}|23\.103\.23[0-9]\.\d{1,3}|23\.103\.24[0-9]\.\d{1,3}|23\.103\.25[0-9]\.\d{1,3}|40\.9[6-9]\.\d{1,3}\.\d{1,3}|40\.10[0-3]\.\d{1,3}\.\d{1,3}|40\.10[4-5]\.\d{1,3}\.\d{1,3}|52\.9[6-9]\.\d{1,3}\.\d{1,3}|70\.37\.151\.12[8-9]|70\.37\.151\.1[3-9][0-9]|70\.37\.151\.2[0-9][0-9]|111\.221\.11[2-9]\.\d{1,3}|131\.253\.33\.215|132\.245\.\d{1,3}\.\d{1,3}|134\.170\.6[8-9]\.\d{1,3}|157\.56\.96\.1[6-9]|157\.56\.96\.2[0-9]|157\.56\.96\.3[0-1]|157\.56\.96\.22[4-9]|157\.56\.96\.23[0-9]|157\.56\.23[2-9]\.\d{1,3}|157\.56\.2[4-5][0-9]\.\d{1,3}|191\.232\.9[6-9]\.\d{1,3}|191\.232\.1[0-1][0-9]\.\d{1,3}|191\.232\.12[0-7]\.\d{1,3}|191\.234\.6\.152|191\.234\.14[0-3]\.\d{1,3}|191\.234\.22[4-7]\.\d{1,3}|204\.79\.197\.215|206\.191\.22[4-9]\.\d{1,3}|206\.191\.2[3-5][0-9]\.\d{1,3}" />

        <!-- Customized access log to print out MS headers for ECP traffic -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/app/logs"
               prefix="access_log" suffix=""
               pattern="%h %l %u %t "%r" %s %b %{X-MS-Client-Application}i %{X-MS-Client-User-Agent}i" resolveHosts="false" requestAttributesEnabled="true" />


-----Original Message-----
From: users [mailto:users-bounces at shibboleth.net] On Behalf Of John Morton
Sent: Tuesday, January 09, 2018 7:59 AM
To: Shib Users <users at shibboleth.net>
Subject: RE: Are all request headers accessible by Shib's Logback?

Thanks Simon, I will look into that. However, if anyone else has a solution only requiring a config change to Shib, I would be very interested! It would also benefit anyone else looking to the same in Jetty.

- John

-----Original Message-----
From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Simon Lundström
Sent: Tuesday, January 09, 2018 1:16 AM
To: Shib Users <users at shibboleth.net>
Subject: Re: Are all request headers accessible by Shib's Logback?

On Tue, 2018-01-09 at 01:00:49 +0000, John Morton wrote:
>Hello again. I am working toward going live with a new 3.3 deployment of Shibboleth, and while doing so, attempting to get our Service Desk a log that has all the relevant information they need to troubleshoot account lockouts.
>
>There is a header that Microsoft appends to ECP requests called X-MS-Forwarded-Client-IP. It contains the true client IP of the device that authenticates with them (that they pass through to us).
>
>I can get this header into my access log with Tomcat's access log valve, as it has access to all headers.
>
>However, it would be really great if I could use Shib's Logback instance to attach this to a message that it already knows about.
>
>I've read about the MDC and the AuditLogging framework, and have played with those, however I have not figured out how to get an arbitrary header.
>
>Does anyone know if this is possible or if I need to rely on Tomcat to log this data?

The standard X-Forwarded-For header is available via Logbacks MDC[1], see many threads on the mailinglist about this. Maybe you can either make Tomcat "rename" the X-MS-Forwarded-Client-IP header to X-F-F or just use the value of X-MS-F-C-IP in the X-F-F header?

Please report back if you have any success with it.

BR,
- Simon

1, <https://logback.qos.ch/manual/mdc.html#mis>
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
-- 
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net