So I’m completely at a loss. I’ve literally spent the last 40 hour work week trying to fix this issue and I’m close to giving up. As a Hail Mary I’m hoping someone smarter than me can help me see my mistakes.
The issue: I have Apache 2.4 behind a Rancher HAProxy 1.6 Load balancer and can’t get Apache to recognize the real client ip in the logs; always showing the ip of HAproxy.
Here’s my current HAProxy.cfg
defaults
maxconn 4096
mode tcp
option forwardfor
option http-server-close
option redispatch
retries 3
timeout client 50000
timeout connect 5000
timeout server 50000
frontend 80
bind *:80
mode http
default_backend 80
backend 80
acl forwarded_proto hdr_cnt(X-Forwarded-Proto) eq 0
acl forwarded_port hdr_cnt(X-Forwarded-Port) eq 0
option forwardfor
http-request add-header X-Forwarded-For %[src]
http-request add-header X-Client-IP %[src]
http-request set-header X-haproxy-custom "castle9mm"
http-request add-header X-Forwarded-Port %[dst_port] if forwarded_port
http-request add-header X-Forwarded-Proto https if { ssl_fc } forwarded_proto
mode http
Here’s my apache virtualhost
RemoteIPHeader X-Forwarded-For RemoteIPInternalProxy 10.42.40.172 RemoteIPTrustedProxy 10.42.40.172
<IfModule mod_headers.c>
Header set X-apache-custom "castle9mm"
</IfModule>
ErrorLog ${APACHE_LOG_DIR}/error.log
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog
I’ve tried a number of different log formats and I don’t know which is correct. I’ve read that mod_remoteip should automatically replace the %h variable with the correct client ip, I’ve also read you need to replace the %h variable with %a, I’ve also read you need to replace %h with %{X-Forwarded-For}i. Nothing i tried worked.
When using chrome inspector, the only header I see is the “X-apache-custom”. I’m not even seeing the “X-haproxy-custom” header so I’m not sure if haproxy is even working right. I’ve also tried replacing the %[src] variable in haproxy.cfg with a hard-coded IP just to see if it gets set but that didn’t work either.
full disclosure: I have stripped out identifying container info in the configs. I don’t think that should matter but there may be a typo.
Any help or insight would be greatly appreciated as I feel like I’m taking crazy pills.