Rancher: Load Balancer X-Forwarded

sjkeerthi · August 18, 2017, 8:44am

Hi

I have an apache container which is behind the Rancher LB when I access the apache log I could see the source IP address as LB’s internal IP.

How could I get the real source client IP address on apache log.

I tried various methods like adding
http-request add-header X-Forwarded-For %[src] on Custom haproxy.cfg

I tried by adding on my apache container with
LogFormat “%{X-Forwarded-For}i %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combined

Still, I could see the source IP address as my LB

localhost:80 10.42.247.56 - - [18/Aug/2017:08:23:34 +0000] “GET /backend/api-auth/login/?next=/backend/ HTTP/1.1” 200 1436 “http://xxxx.com/backend/” “Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36”

sjkeerthi · August 22, 2017, 6:00am

After I add my URL on the Rancher LB I could see on /etc/haproxy/haproxy.cfg

   backend 80_stage_test_com_backend
   acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
   acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
   http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
   http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
   mode http
   server 1d3c763bac284447354f6488cfdc777763382c97 10.42.125.61:80

On my Apache Container I added

RemoteIPHeader X-Forwarded-For
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\ " combine
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-forwarded

Apache Log

localhost:80 10.42.247.56 - - [18/Aug/2017:08:17:50 +0000] "GET /backend HTTP/1.1" 200 2533 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36"
localhost:80 10.42.247.56 - - [18/Aug/2017:08:18:09 +0000] "GET /backend HTTP/1.1" 200 2533 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36"
localhost:80 10.42.247.56 - - [18/Aug/2017:08:19:16 +0000] "GET /backend/api-auth/logout/?next=/backend/ HTTP/1.1" 302 452 "http://stage.test.com/backend" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36"
localhost:80 10.42.247.56 - - [18/Aug/2017:08:19:16 +0000] "GET /backend/ HTTP/1.1" 403 4991 "http://stage.test.com/backend" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36"
localhost:80 10.42.247.56 - - [18/Aug/2017:08:23:34 +0000] "GET /backend/api-auth/login/?next=/backend/ HTTP/1.1" 200 1436 "http://stage.test.com/backend/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36"

Still I could not able to get the source IP address on my access.log have no clue really what else I need to do get the client source IP address

davalex · August 22, 2017, 8:21pm

Hi,

Try adding

defaults
option forwardfor if-none

to the “Custom haproxy config” on the load balancer.

I had to use this config when running a ELB infront of Rancher’s load balances.

sjkeerthi · August 23, 2017, 1:17am

Hi

I tried with above mentioned option on Rancher LB, Still I get the same source IP Address as my LB’s IP Address.

My Container Apache Conf File

ServerName localhost
WSGIProcessGroup localhost
WSGIPassAuthorization On
WSGIDaemonProcess localhost python-path=/trunk/django/
WSGIScriptAlias /backend /trunk/django/wsgi.py process-group=localhost
WSGIApplicationGroup %{GLOBAL}

Alias "/uploads"                             "/trunk/django/uploads"
Alias "/backend/uploads"                     "/trunk/django/uploads"
	
RemoteIPHeader X-Forwarded-For
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" combine
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-forwarded

<Directory /trunk/django/uploads>
       Require all granted
</Directory>

<Directory /trunk/django>
     <Files wsgi.py>
      Require all granted
     </Files>
</Directory>
</VirtualHost>

etlweather · August 24, 2017, 12:02am

You need to tell Apache that you trust the proxy to tell you the X-Forwarded-For header…

For Apache HTTPd, in your VirtualHost configuration:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8

You don’t need to modify the log format, it will log the Remote IP automatically.

For Nginx:

server {
 ...
 set_real_ip_from 10.0.0.0/8;
 real_ip_header X-Forwarded-For;
...
}

There aren’t any configuration needed for HAProxy, by default, the Rancher LB HAProxy has the X-Forwarded-For header set.

sjkeerthi · August 24, 2017, 1:48pm

As you said I tried with below configuration still I get the LB IP Address

<VirtualHost *:80>

ServerName localhost
WSGIProcessGroup localhost
WSGIPassAuthorization On
WSGIDaemonProcess localhost python-path=/trunk/django/
WSGIScriptAlias /backend /trunk/django/wsgi.py process-group=localhost
WSGIApplicationGroup %{GLOBAL}

Alias "/uploads"                             "/trunk/django/uploads"
Alias "/backend/uploads"                     "/trunk/django/uploads"

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8
	
<Directory /trunk/django/uploads>
       Require all granted
</Directory>

<Directory /trunk/django>
     <Files wsgi.py>
      Require all granted
     </Files>
</Directory>
</VirtualHost>

Still the Apache Log

localhost:80 10.42.3.76 - - [24/Aug/2017:13:42:59 +0000] “GET /backend HTTP/1.1” 403 4972 “-” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36"
localhost:80 10.42.3.76 - - [24/Aug/2017:13:44:16 +0000] “GET /backend/api-auth/login/?next=/backend/ HTTP/1.1” 200 1436 “http://stage.xxxxx.com/backend” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36”

etlweather · August 24, 2017, 1:58pm

My container is FROM httpd:2.4-alpine.

I realized I also load the module remoteip

LoadModule remoteip_module modules/mod_remoteip.so

My common log format is the following (but I don’t think I changed that from the default):

<IfModule log_config_module>
    LogFormat "%v:%p %a %l %u %t \"%r\" %>s %b" common
    CustomLog /proc/self/fd/1 common
</IfModule>

sjkeerthi · August 24, 2017, 2:06pm

My container is ubuntu 16.04 with Apache 2.4 the default apache.conf I cloud see the below conf and I have enabled the remoteip module using on my DockerFile RUN a2enmod remoteip

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Topic		Replies	Views
HAProxy -> Apache X-Forwarded-For not working Rancher 1.x	1	5137	April 12, 2018
Load Balancer forward IP Rancher 1.x	1	1671	February 16, 2016
Haproxy http-request Rancher 1.x	0	906	February 16, 2016
GRPC Load Balancing With Rancher	3	2875	August 19, 2019
Loadbalancer + ELB + logging question	4	1320	August 8, 2016

Rancher: Load Balancer X-Forwarded

Still the Apache Log

Related topics