Getting Tomcat to use the X-Forwarded-For header

[Darren Boss]

I'm not running in either GCP or AWS but on our own private OpenStack
clouds and I had to set up my own LB because we don't have the LBaaS
functionality deployed in our clouds and this is how I was able to get
client IPs in our logs. Even if you aren't using Kubernetes, I think
some of what I have to say may be useful.

First I had to turn on proxy protocol on our LB as this was the only
way I could find to pass the client information all the way through to
the containers. I've used both HAProxy and Nginx for our LB and both
support this. I'm pretty sure I've seen the proxy protocol setting in
GCP/GKE. Just note that your web server or tomcat will have to decode
the proxy protocol for the connection to work. I don't know how to do
this in Tomcat but with the Nginx ingress controller (this is
different from the Nginx LB) in Kubernetes there are some options you
put into the nginx ingress controller configmap to enable proxy
protocol support:
  use-forwarded-headers: "true"
  use-proxy-protocol: "true"

Since I have this intermediate Nginx ingress controller (just a Nginx
reverse proxy) between my LB and Tomcat, I found the settings that
both Keith and Brian mentioned works to put the client IPs into the
Tomcat logs.

    <Valve className="org.apache.catalina.valves.RemoteIpValve"
           remoteIpHeader="x-forwarded-for"
           protocolHeaderHttpsValue="https" />

is what I ended up adding to our server.xml.

Connections from client to tomcat look like this (this is still
simplified, there are still some k8s magic that goes on in those last
two -> as well):

Client Browser -> Nginx LB (L4 proxy protocol) -> Nginx Ingress
Controller (TLS termination) -> Tomcat

On Thu, Jul 2, 2020 at 11:44 PM Brian Moon <bmoon at scu.edu> wrote:
>
> Hello Keith,
>
> We are in the midst of migrating to GCP ourselves, so I had to deal with this just last week.  Below is how we've configured the RemoteIpValve:
>
> <Valve className="org.apache.catalina.valves.RemoteIpValve"
>        internalProxies="130\.211\.[0-3]\.\d{1,3}|35\.191\.\d{1,3}\.\d{1,3}|34\.9[6-9]\.\d{1,3}\.\d{1,3}"
>        remoteIpHeader="X-Forwarded-For"
>        remoteIpProxiesHeader="X-Forwarded-By"
>        protocolHeader="X-Forwarded-Proto" />
>
> All of the internal proxies listed there are GCP specific, so you will need to update that list with the internal proxies for the AWS ALB..
>
> More information about configuring that valve can be found here:
> https://tomcat.apache.org/tomcat-8.0-doc/api/org/apache/catalina/valves/RemoteIpValve.html
>
> Cheers!
>
> Brian Moon
> Senior System Administrator, Enterprise Systems
> Santa Clara University
>
>
> On Thu, Jul 2, 2020 at 7:27 PM Wessel, Keith <kwessel at illinois.edu> wrote:
>>
>> All,
>>
>> I've been operating our IdP behind load balancers that are kind enough to pass through the client's IP address as the source address instead of in the X-Forwarded-For header. As we begin a move to AWS with the Trusted Access Platform containers and an Amazon ALB, I now need to take an extra step to get Tomcat to use the X-Forwarded-For IP instead of the client IP (which is that of the ALB instead of the originating client). This is partly for logging of client IPs, but also for the access control facility.
>>
>> I've tried adding this to the <host> block of Tomcat's server.xml with no change in behavior:
>>
>>         <Valve className="org.apache.catalina.valves.RemoteIpValve" />
>>
>> The IP that appears in the IdP logs is still that of the ALB.
>>
>> Has anyone on this list found a way to get the originating client's IP instead of that of the load balancer inside the IdP when running Tomcat?
>>
>> Thanks,
>> Keith
>>
>> --
>> For Consortium Member technical support, see https://urldefense.com/v3/__https://wiki.shibboleth.net/confluence/x/coFAAg__;!!MLMg-p0Z!SZ8SFws3ujdjmL3MqgQ1I9trbhjbYA9S2T2fSgdcGqatKzPe_JL6GKyZVw5X$
>> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
>
> --
> For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net



-- 
Darren Boss
Senior Programmer/Analyst
Programmeur-analyste principal
darren.boss at computecanada.ca