Routing problem in SLES 11 SP1

I have a load balanced outbound internet access setup on my SLES 11 SP1 server. This config wors perfectly fine on SLES 9 SP3. However in SLES11 SP1 while the load balancing works fine I have one small issue for dead gateway detection.

When all my VLAN interfaces are up, I am able to ping the internet ONLY using the VLAN interface that has the default route setup to go through it.

In detail: before the load balancing is setup my default route is through vlan4 and I am able to ping anything on the internet through VLAN4. However eventhough VLAN5 is up, any pings to internet thru VLAN5 fails. Once I change default route to use VLAN5 the problem is solved, But VLAN4 stops.

When my load balance script changes the default route to multi-path across all the VLAN interfaces, things are again fine.

But I need to be able to ping through a interface that does not have default route setup thru it to help detect dead gateways. Herein lies my problem. In SLES9 everything works. Is there some change in SLES11 SP1 that is causing this.

To explaing my NIC config: There are 4 nics (eth0 to 3). Three are bonded together (eth0 to 2). A bridge (br0) to the bonded iface connects my local network. Eth3 has four VLANS (VLAN4 to 7) connected to the net thru four independant ADSL router-modems. Each VLAN interface accesses one router-modem.

Output of IFOCNFIG is below. Output of ROUTE (before load balancing) follows this. Output of PING follows last.

================
bond0 Link encap:Ethernet HWaddr 00:14:38:B8:8E:8A
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:42656 errors:0 dropped:0 overruns:0 frame:0
TX packets:16869 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5997078 (5.7 Mb) TX bytes:11670363 (11.1 Mb)

br0 Link encap:Ethernet HWaddr 00:14:38:B8:8E:8A
inet addr:192.168.2.4 Bcast:192.168.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:41121 errors:0 dropped:0 overruns:0 frame:0
TX packets:15664 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4765499 (4.5 Mb) TX bytes:11424153 (10.8 Mb)

eth0 Link encap:Ethernet HWaddr 00:14:38:B8:8E:8A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:7963 errors:0 dropped:0 overruns:0 frame:0
TX packets:15932 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1095119 (1.0 Mb) TX bytes:11544892 (11.0 Mb)
Interrupt:48

eth1 Link encap:Ethernet HWaddr 00:14:38:B8:8E:8A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:20251 errors:0 dropped:0 overruns:0 frame:0
TX packets:418 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2226292 (2.1 Mb) TX bytes:53108 (51.8 Kb)
Interrupt:76

eth2 Link encap:Ethernet HWaddr 00:14:38:B8:8E:8A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:14442 errors:0 dropped:0 overruns:0 frame:0
TX packets:519 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2675667 (2.5 Mb) TX bytes:72363 (70.6 Kb)
Interrupt:26

eth3 Link encap:Ethernet HWaddr 00:12:79:3A:21:FC
inet addr:192.168.8.4 Bcast:192.168.8.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:428 errors:0 dropped:0 overruns:0 frame:0
TX packets:1044 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:87064 (85.0 Kb) TX bytes:75636 (73.8 Kb)
Interrupt:17

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1323 errors:0 dropped:0 overruns:0 frame:0
TX packets:1323 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:134772 (131.6 Kb) TX bytes:134772 (131.6 Kb)

vlan4 Link encap:Ethernet HWaddr 00:12:79:3A:21:FC
inet addr:192.168.4.4 Bcast:192.168.4.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:274 errors:0 dropped:0 overruns:0 frame:0
TX packets:319 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:70452 (68.8 Kb) TX bytes:25328 (24.7 Kb)

vlan5 Link encap:Ethernet HWaddr 00:12:79:3A:21:FC
inet addr:192.168.5.4 Bcast:192.168.5.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:50 errors:0 dropped:0 overruns:0 frame:0
TX packets:247 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2296 (2.2 Kb) TX bytes:10414 (10.1 Kb)

vlan6 Link encap:Ethernet HWaddr 00:12:79:3A:21:FC
inet addr:192.168.6.4 Bcast:192.168.6.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54 errors:0 dropped:0 overruns:0 frame:0
TX packets:238 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2604 (2.5 Kb) TX bytes:10036 (9.8 Kb)

vlan7 Link encap:Ethernet HWaddr 00:12:79:3A:21:FC
inet addr:192.168.7.4 Bcast:192.168.7.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:50 errors:0 dropped:0 overruns:0 frame:0
TX packets:238 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2296 (2.2 Kb) TX bytes:10036 (9.8 Kb)

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.7.0 * 255.255.255.0 U 0 0 0 vlan7
192.168.6.0 * 255.255.255.0 U 0 0 0 vlan6
192.168.5.0 * 255.255.255.0 U 0 0 0 vlan5
192.168.4.0 * 255.255.255.0 U 0 0 0 vlan4
192.168.2.0 * 255.255.255.0 U 0 0 0 br0
192.168.8.0 * 255.255.255.0 U 0 0 0 eth3
link-local * 255.255.0.0 U 0 0 0 eth3
loopback * 255.0.0.0 U 0 0 0 lo
default airtel-cli-rtr 0.0.0.0 UG 0 0 0 vlan4

ping -c1 -I vlan4 gmail.com
PING gmail.com (74.125.236.54) from 192.168.4.4 vlan4: 56(84) bytes of data.
64 bytes from maa03s04-in-f22.1e100.net (74.125.236.54): icmp_seq=1 ttl=56 time=31.3 ms

gmail.com ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 31.361/31.361/31.361/0.000 ms

Please help.

Ramesh.

ramesh,

It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.

Has your issue been resolved? If not, you might try one of the following options:

Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.suse.com/faq.php

If this is a reply to a duplicate posting, please ignore and accept our apologies
and rest assured we will issue a stern reprimand to our posting bot.

Good luck!

Your SUSE Forums Team
http://forums.suse.com

ramesh tn wrote:
[color=blue]

Eth3 has four VLANS (VLAN4 to 7) connected to the net
thru four independant ADSL router-modems.[/color]

The answer to your question is not obvious and I don’t have much
experience with VLANS. I had a similar problem with two subnets bound
to the same nic. My solution is described here:
http://forums.suse.com/showthread.php?378-SuSEfirewall2-2-subnets-on-same-interface

Let me know if it works for you.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

My problem, if you remove all the vlan, load balancing etc, is very simple.
I am unable to ping anything through vlan4, if the default route is not through vlan 4. The minute i change default route to use vlan4 things start working. If I change default route to use vlan5, that starts working, but vlan4 stops. This happens for all the interfaces. In fact, things work on all interfaces when the muti-path default route is set to use all 4 interfaces.

The problem, I am assuming, is much simpler than some complex routing configuration, but I am not able to pick it out. This problem does not exist on sles9.

I gave all the extra information in the hope that I can get quicker replies. Not sure if I should repost this message in another thread with the core issue alone.

Thanks in any case. Will check your thread too.

[QUOTE=KBOYLE;2938]ramesh tn wrote:
[color=blue]

Eth3 has four VLANS (VLAN4 to 7) connected to the net
thru four independant ADSL router-modems.[/color]

The answer to your question is not obvious and I don’t have much
experience with VLANS. I had a similar problem with two subnets bound
to the same nic. My solution is described here:
http://forums.suse.com/showthread.php?378-SuSEfirewall2-2-subnets-on-same-interface

Let me know if it works for you.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…[/QUOTE]

ramesh tn wrote:
[color=blue]

The problem, I am assuming, is much simpler than some complex routing
configuration, but I am not able to pick it out.[/color]

Perhaps it is…

You have explained the problem quite well but have not provided enough
specific information for anyone properly diagnose the problem. I
offered a suggestion that worked for me on the off chance it might
resolve your issue without further diagnostic effort.
[color=blue]

FW_FORWARD_ALWAYS_INOUT_DEV=“eth3”[/color]

If this does not work for you then you may want to do a packet trace to
find out what is going on.

And yes, this is the correct forum to ask your question.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

There has been a change in SLES 11 in how the kernel handles certain routing requests.

In short, try setting the following options in /etc/sysctl.conf :

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

Then run ‘sysctl -p’ or reboot the system to activate those options.

This normally needs to be set on interfaces using an IP on multiple interfaces that are in the same subnet (like when wanting to use multipath over iSCSI)… but it might also be the solution for your setup.

-Willem

[QUOTE=Magic31;3000]There has been a change in SLES 11 in how the kernel handles certain routing requests.

In short, try setting the following options in /etc/sysctl.conf :

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

Then run ‘sysctl -p’ or reboot the system to activate those options.

This normally needs to be set on interfaces using an IP on multiple interfaces that are in the same subnet (like when wanting to use multipath over iSCSI)… but it might also be the solution for your setup.

-Willem[/QUOTE]
Tried all your suggestions.
Now, from the local-vlan-interface (IP: 192.168.7.4) I am able to ping the next hop (IP: 192.168.7.2) (which is a direct link to the lan-interface on the adsl modem). Beyond that it does not work if I don’t set the default route. Only problem is, I am not sure whether this problem existed before.

[QUOTE=ramesh_tn;3046]Tried all your suggestions.
Now, from the local-vlan-interface (IP: 192.168.7.4) I am able to ping the next hop (IP: 192.168.7.2) (which is a direct link to the lan-interface on the adsl modem). Beyond that it does not work if I don’t set the default route. Only problem is, I am not sure whether this problem existed before.[/QUOTE]

It’s normal, IFAIK, for packets to follow the default route if not other routes are set… it won’t work to have all packets go out on any interface (out to different providers or connecting points that are not configured for this) as a stream must follow the same path or you will get all sorts of communication failures.

Never had to load balance on a SUSE server itself, but usually do this on a firewall appliance (like a SonicWall firewall) where specified network ranges are grouped and set to default to using different gateways. Then if that gateway fails, fallback is done to the other gateway.
That way, in the situation that all gateways are working, e-mail flows over one gateway, general internet use over an other, for example.

I am curious though to how this was working (or the idea was) in the old setup?

Cheers,
Willem

Hi Ramesh,

I’ve tried to catch what’s happening, but will probably need more facts to be helpful.

The first thing that catches my eyes is that it seems you’re trying to ping to the world with local addresses. Where does the NATing happen?

Secondly, on your Linux router I’d expect some firewalling to happen - how’s that set up?

Thirdly, you might want to do some tcpdumping of your traffic to get behind what’s actually happening on the wire: Maybe it’s not the ICMP echo request that’s being send to the wrong interface, but the echo reply? I don’t know how “ping” reacts to replies recieved on a different interface when running with “-I”.

Key should be to understand how the packets actually travel across the network, including where and how they’re NATed.

Regards,
Jens

Thank you Willem and Jens. Appreciate your interest.

Sorry about the delayed reply. I am just in charge of a small business and this is one of the things I do. I have attached my loadbalance script to this reply, in the hope that it helps. Please remember: I am just a advanced user, not a techy guy. Bear that in mind if my answers to your questions are way off.

My whole idea behind starting to use load balancing was to get over poor connections here AND to make maximum use of the charges the local telephone companies levies for usage. Usually it is very low upto a usage-point, then it shoots up. So having multiple providers and using their respective minuimum usage plans made economic sense. This setup only serves my companies web browsing needs. I know that the routes get cached and on days when I have large downloads, the entire traffic is thru just one interface. But 99% of the time, it is evenly distributed across all the interfaces and perfectly achieves my end.

The attached script is a mish-mash of everything I learned of the web and has been working for many years now. It continues to work without problems, except that I am now working on automatically detecting dead gateways and for that I need to be able to ping the real web through any/all my interfaces and detect which is down. Therein lies my problem. By default the script starts with 4-dsl routers load balanced assuming they are all having live connections. Later if/when one of them goes down, my ping script detects it and changes the load-balance in favour of the live connections. BUT: How do I know when it become lives again? The default route no longer uses that interface, as my script has changed it to use the other live ones. So once a interface is down, it is permanently down.

DO excuse, all the copyright info, it is undeserving and was purely to pamper my ego for achieving a working script in linux.

Jens, the nating happens on my linux box and on the dsl-routers. The firewall is also setup and uses nothing more than standard suse options. Your third question-- I do not understand.

[CODE]#! /bin/sh
######################################## FOR 4 CONNECTIONS ###########################################

Copyright (c) 2010-2012 T. N. Ramesh. All rights reserved.

T. N. Ramesh

script written on 17.2.2012

BEGIN INIT INFO

Provides: loadbal

Required-Start: $network $syslog

Required-Stop: $network $syslog

Default-Start: 3 5

Default-Stop: 0 1 2 6

Description: script for enabling loadbalancing through multiple internet routers

END INIT INFO

NOTE: During link failures must recall this function with live links

#######################################################################################################

In next version must consolidate hard-coded route entries to use variables & put them in a loop

Currently this script must be modified if default number of connections is different from 4 and

entries must be modified inside 2 functions “noparams” and “withparams”

Must also determine method to detect deadgateways and call script automatically with relevant

args and make it totally automatic.

#######################################################################################################

Read rc functions

Shell functions sourced from /etc/rc.status:

rc_check check and set local and overall rc status

rc_status check and set local and overall rc status

rc_status -v be verbose in local rc status and clear it afterwards

rc_status -v -r ditto and clear both the local and overall rc status

rc_status -s display “skipped” and exit with status 3

rc_status -u display “unused” and exit with status 3

rc_failed set local and overall rc status to failed

rc_failed set local and overall rc status to

rc_reset clear both the local and overall rc status

rc_exit exit appropriate to overall rc status

rc_active checks whether a service is activated by symlinks

. /etc/rc.status

Reset status of this service

rc_reset

#define aesthetics
statusCol=“echo -ne \\033[60G”
statusColorOK=“echo -ne \\033[1;32m”
statusColorFailed=“echo -ne \\033[1;31m”
statusColorNormal=“echo -ne \\033[0;39m”

#define script and config location and directory
LB_SCRIPT=loadbal.standalone.sh
LB_DIR=/usr/sbin/
#check it exist and has execute perms
if [ ! -x $LB_DIR$LB_SCRIPT ] ; then
echo -n “[”
${statusColorFailed}
echo -n " Load Balancing Script not installed or is not executable ! "
${statusColorNormal}
echo “]”
rc_status -s
exit 5
fi

#define sensible name for service to display in messages
SERVICE=“Load Balancing for multiple uplinks”

#define routes that load balancing should take by default
LB_ROUTERS=/etc/sysconfig/LB_routers

#read routes config file for passing default parameters.
[ -r $LB_ROUTERS ] && . $LB_ROUTERS

#define local interfaces
IF0=br0

#define gateway interfaces
#------###
IF1=vlan4
#------###
IF2=vlan5
#------###
IF3=vlan6
#------###
IF4=vlan7

#define LAN ip address of interfaces
IP0=192.168.2.4
IP1=192.168.4.4
IP2=192.168.5.4
IP3=192.168.6.4
IP4=192.168.7.4

#define WAN ip address of interfaces
P1=192.168.4.2
P2=192.168.5.2
P3=192.168.6.2
P4=192.168.7.2

#define network of interfaces
P0_NET=192.168.2.0/24
P1_NET=192.168.4.0/24
P2_NET=192.168.5.0/24
P3_NET=192.168.6.0/24
P4_NET=192.168.7.0/24

#move common route command to variable
RTCMD=“ip route add default scope global”

#retval=0

########################################################################

Route Tables check: found generally in /etc/iproute2/rt_tables

and presence of multiple tables

# define location of route tables

LB_RT_TABLES=rt_tables
LB_RT_TABLES_DIR=/etc/iproute2/
# ensure existence of rt_tables and it has read permission
if [ ! -r $LB_RT_TABLES_DIR$LB_RT_TABLES ] ; then
echo -n “[”
${statusColorFailed}
echo -n " $LB_RT_TABLES_DIR$LB_RT_TABLES does not exist or is not readable ! "
${statusColorNormal}
echo “]”
rc_status -s
exit 6
fi
# and check if additional table have been defined for load balancing
# NOTE: Tables must be name T1, T2, T3 and T4 for this function to work
# NOTE: grep searches for lines that do not begin with a # and contain t1 or
# t2 or t3 or t4
if [ “$(grep -Ec ‘^[^#].*t[1-4]’ $LB_RT_TABLES_DIR$LB_RT_TABLES)” -lt 2 ] ; then
echo -n “[”
${statusColorFailed}
echo -n “Must define atleast 2 tables in $LB_RT_TABLES for load balancing to function”
${statusColorNormal}
echo “]”
exit 6
fi

#########################################################################

Check number of routers specified in /etc/sysconfig/LB_routers

Display error if number is zero

N=1
for LBR in $LBR_AUTOSTART; do
if grep -q \; <<< “$LBR”; then
LBR_NAME[$N]=$(cut -d\; -f1 <<< “$LBR”)
else
LBR_NAME[$N]="$LBR"
fi
N=$(($N+1))
done

LBRs=${#LBR_NAME[*]}

if [ $LBRs -eq 0 ]; then
echo -n "["
${statusColorFailed} 
echo -n "Starting $SERVICE: no routers configured"
echo -n "Check $LB_ROUTERS or pass args in cmd line"
${statusColorNormal}
echo "]"
rc_status -u
fi

First reset status of this service

rc_reset

############################################################################

SCRIPT CALLED WITHOUT PARAMETERS USES THIS FUNCTION: noparams

See last section for calls to this function

cleanuprt () {
########################################################################
### Start basic route clean up ####

	ip route flush cache
	ip route del default
	echo -n "["
	${statusColorOK} 
	echo -n "Setting up Load balancing... - Ramesh" 
	${statusColorNormal}
	echo "]"
	route del default
	ip route flush cache

###               end basic route clean up                           ###
########################################################################

}

########################################################################
###  Final route command and display succesfully end of script       ###
###  This function is called last

finalrtcmd () {

# check if $RTCMD variable is appended with available routes 
# (it should be different from initialised value)
    if [ "$RTCMD" != "ip route add default scope global" ] ; then
#final route command to be executed. enclosed in backticks (ie.[shift]tilde)     
	    `$RTCMD`
	    echo -n "["
    ${statusColorOK}
	    echo -n "Finished setting up load balancing --Ramesh"
	    ${statusColorNormal}
	    echo "]" 
else
	    echo -n "["
    ${statusColorFailes}
	    echo "FAILURE: routing entry remains unchanged. NOT load balancing."
	    ${statusColorNormal}
	    echo "]" 
fi
rc_status -v
exit 0

}

########################################################################
###             Main routing setup starts here                       ###
### Must add more elif statements as more routers are added and      ###
### also add respective routes to existing commands                  ###

noparams () {
cleanuprt
for LBR in $LBR_AUTOSTART; do
if [ $LBR -eq 4 ]; then
ip route add $P1_NET dev $IF1 src $IP1 table t1
ip route add default via $P1 table t1
ip route add $P1_NET dev $IF1 src $IP1
ip rule add from $IP1 table t1
ip route add $P0_NET dev $IF0 table t1
ip route add $P2_NET dev $IF2 table t1
ip route add $P3_NET dev $IF3 table t1
ip route add $P4_NET dev $IF4 table t1
ip route add 127.0.0.0/8 dev lo table t1
RTCMD="$RTCMD nexthop via ${P1} dev ${IF1} weight 1"

    elif [[ $LBR -eq 5 ]]; then 
    	ip route add $P2_NET dev $IF2 src $IP2 table t2
    	ip route add default via $P2 table t2
    	ip route add $P2_NET dev $IF2 src $IP2
    	ip rule add from $IP2 table t2
    	ip route add $P0_NET dev $IF0 table t2
    	ip route add $P1_NET dev $IF1 table t2
    	ip route add $P3_NET dev $IF3 table t2
    	ip route add $P4_NET dev $IF4 table t2
    	ip route add 127.0.0.0/8 dev lo table t2
    	RTCMD="$RTCMD nexthop via ${P2} dev ${IF2} weight 1"
    
    elif [[ $LBR -eq 6 ]]; then 
    	ip route add $P3_NET dev $IF3 src $IP3 table t3
    	ip route add default via $P3 table t3
    	ip route add $P3_NET dev $IF3 src $IP3
    	ip rule add from $IP3 table t3
    	ip route add $P0_NET dev $IF0 table t3
    	ip route add $P1_NET dev $IF1 table t3
    	ip route add $P2_NET dev $IF2 table t3
    	ip route add $P4_NET dev $IF4 table t3
    	ip route add 127.0.0.0/8 dev lo table t3
    	RTCMD="$RTCMD nexthop via ${P3} dev ${IF3} weight 1"
    
    elif [[ $LBR -eq 7 ]]; then
    	ip route add $P4_NET dev $IF4 src $IP4 table t4
    	ip route add default via $P4 table t4
    	ip route add $P4_NET dev $IF4 src $IP4
    	ip rule add from $IP4 table t4
    	ip route add $P0_NET dev $IF0 table t4
    	ip route add $P1_NET dev $IF1 table t4
    	ip route add $P2_NET dev $IF2 table t4
    	ip route add $P3_NET dev $IF3 table t4
    	ip route add 127.0.0.0/8 dev lo table t4
    	RTCMD="$RTCMD nexthop via ${P4} dev ${IF4} weight 1"
    else
    echo -n "["
    ${statusColorFailed} 
    echo "$SERVICE: Failure"	
    ${statusColorNormal}
    echo "]"
    fi
done
finalrtcmd

}

############################################################################

SCRIPT CALLED WITH PARAMETERS USES THIS FUNCTION: withparams

See the last section for calls to this function

withparams () {
cleanuprt
while [ $# -gt 0 ]; do
case “$1” in
4)
ip route add $P1_NET dev $IF1 src $IP1 table t1
ip route add default via $P1 table t1
ip route add $P1_NET dev $IF1 src $IP1
ip rule add from $IP1 table t1
ip route add $P0_NET dev $IF0 table t1
ip route add $P2_NET dev $IF2 table t1
ip route add $P3_NET dev $IF3 table t1
ip route add $P4_NET dev $IF4 table t1
ip route add 127.0.0.0/8 dev lo table t1
RTCMD="$RTCMD nexthop via ${P1} dev ${IF1} weight 1"
;;
5)
ip route add $P2_NET dev $IF2 src $IP2 table t2
ip route add default via $P2 table t2
ip route add $P2_NET dev $IF2 src $IP2
ip rule add from $IP2 table t2
ip route add $P0_NET dev $IF0 table t2
ip route add $P1_NET dev $IF1 table t2
ip route add $P3_NET dev $IF3 table t2
ip route add $P4_NET dev $IF4 table t2
ip route add 127.0.0.0/8 dev lo table t2
RTCMD="$RTCMD nexthop via ${P2} dev ${IF2} weight 1"
;;
6)
ip route add $P3_NET dev $IF3 src $IP3 table t3
ip route add default via $P3 table t3
ip route add $P3_NET dev $IF3 src $IP3
ip rule add from $IP3 table t3
ip route add $P0_NET dev $IF0 table t3
ip route add $P1_NET dev $IF1 table t3
ip route add $P2_NET dev $IF2 table t3
ip route add $P4_NET dev $IF4 table t3
ip route add 127.0.0.0/8 dev lo table t3
RTCMD="$RTCMD nexthop via ${P3} dev ${IF3} weight 1"
;;
7)
ip route add $P4_NET dev $IF4 src $IP4 table t4
ip route add default via $P4 table t4
ip route add $P4_NET dev $IF4 src $IP4
ip rule add from $IP4 table t4
ip route add $P0_NET dev $IF0 table t4
ip route add $P1_NET dev $IF1 table t4
ip route add $P2_NET dev $IF2 table t4
ip route add $P3_NET dev $IF3 table t4
ip route add 127.0.0.0/8 dev lo table t4
RTCMD="$RTCMD nexthop via ${P4} dev ${IF4} weight 1"
;;
esac
#check next parameter
shift
done
finalrtcmd
}

##############################################################################

test for parameters in command line and display usage instructions

Jump to function “noparams” or “withparams” based on call.

this has been coded last as bash needs to read functions to be

aware about them

Define function “usage” for common help message for error conditions

usage () {
echo “If ‘start’ or no parameters are specified on command line,”
echo “script will automatically pick config from $LB_ROUTERS”
echo “Command line Usage: $0 4 5 6 7”
echo " or $0 4 5 6"
echo “enable only connections that are active”
echo “CLI’s Airtel = 4 CLIPL’s Airtel = 5”
echo -n “CLI’s Bsnl = 6 TLO’s Bsnl = 7”
${statusColorNormal}
echo “]”
}

Test for sanity and proceed accordingly

if [ $# = 0 ]; then
    echo -n "["
    ${statusColorOK}    
    usage
	    noparams

	    ## If loaded on boot, the parameter "start" will automatically be passed
    ## hence, this must be handled.
elif [ $1 = "start" ]; then
    echo -n "["
    ${statusColorOK}
    usage
    noparams
	    ## If stopped during system shutdown, the parameter "stop" will automatically be passed
    ## hence, this must be handled.
elif [ $1 = "stop" ]; then
    echo -n ""
    ${statusColorOK}
    echo "Stop specified. Nothing to do. Exiting..."
    exit 
elif [ $# > 0 ]; then
    if [ $# -lt 2 ]; then
	echo -n "["
	${statusColorFailed}    
	echo "Invalid parameters specified. Atleast 2 routes needed."
	usage
		exit 1

    elif [ $# -gt 4 ]; then
	echo -n "["
	${statusColorFailed}    
	echo "Invalid parameters specified. Maximum 4 routes supported."
	usage
		exit 1

    else
	## move number of args to counter, and check each value for correct-range
	## within loop
	## do not use $# directly, as we need its values for next stage.
	for args in "$@"
	# above can also simply be -- for args --  "in "$@"" is default 
	do
		    if [ $args -lt 4 ]; then
			echo -n "["
		${statusColorFailed}    
		echo "Invalid parameters specified. Args must be between 4 and 7."
		usage
		exit 1     
	    elif [ $args -gt 7 ]; then
		echo -n "["
		${statusColorFailed}    
		echo "Invalid parameters specified. Args must be between 4 and 7."
		usage
		exit 1
	    fi
	done
	## everything is fine, call withparams with original args intact
	withparams "$@"	    
	## the "$a" is to pass all args in the cmd line to the function
	## normally cmd line args are not passed to user-functions 		
    fi
fi

#############################################################################
exit
[/CODE]

[QUOTE=jmozdzen;3109]Hi Ramesh,

I’ve tried to catch what’s happening, but will probably need more facts to be helpful.

The first thing that catches my eyes is that it seems you’re trying to ping to the world with local addresses. Where does the NATing happen?

Secondly, on your Linux router I’d expect some firewalling to happen - how’s that set up?

Thirdly, you might want to do some tcpdumping of your traffic to get behind what’s actually happening on the wire: Maybe it’s not the ICMP echo request that’s being send to the wrong interface, but the echo reply? I don’t know how “ping” reacts to replies recieved on a different interface when running with “-I”.

Key should be to understand how the packets actually travel across the network, including where and how they’re NATed.

Regards,
Jens[/QUOTE]

ramesh tn wrote:
[color=blue]

I am now working on automatically detecting dead
gateways and for that I need to be able to ping the real web through
any/all my interfaces and detect which is down.[/color]

Would something like this work for you?

If you have four ISP’s, that should be four distinct networks. Choose a
device on each network, perhaps the first router (which could change)
or the ISP’s DNS server. Set up a route via the appropriate interface
in your server’s routing table to each of the selected devices and ping
the specific device. The default route should not affect the more
specific route you have defined.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

My third comment was focusing on what’s going on… when you have a successful “ping”, that’s because you send an “ICMP echo request” and the target system answers with an “ICMP echo response”. When you notice failing pings, then it might not be that the echo request is being sent incorrectly but that the response is not reaching you.

For instance, you might send the request via one interface, but due to some NAT rule “problem” with a sender IP address of a different (i.e. the default route) interface. The ping target system then would send the reply to that IP address, and the reply would enter your system via that other interface. As that response packet might not be matched to the sent packet (so to speak), the “ping” command would report this as a lost packet.

You might be able to track this down via “tcpdump”, checking on the wire what packet gets sent and if there are stray ICMP echo responses coming in via i.e. one of the other interfaces.

Regards,
Jens

[QUOTE=jmozdzen;3367]My third comment was focusing on what’s going on… when you have a successful “ping”, that’s because you send an “ICMP echo request” and the target system answers with an “ICMP echo response”. When you notice failing pings, then it might not be that the echo request is being sent incorrectly but that the response is not reaching you.

For instance, you might send the request via one interface, but due to some NAT rule “problem” with a sender IP address of a different (i.e. the default route) interface. The ping target system then would send the reply to that IP address, and the reply would enter your system via that other interface. As that response packet might not be matched to the sent packet (so to speak), the “ping” command would report this as a lost packet.

You might be able to track this down via “tcpdump”, checking on the wire what packet gets sent and if there are stray ICMP echo responses coming in via i.e. one of the other interfaces.

Regards,
Jens[/QUOTE]

You have tell me how to “tcpdump”.

BUT: If the problem is per your thoughts, why does “ping” work when the default route is set. Like I said before, when the multi-path default route is set through all interfaces, everything works fine. All my pings through each of the interfaces brings a response.

FURTHER: Based on some googling, I also set the rp_filter from the default “1” to “0”. System is currently set to use “0”. Still no help.

You have tell me how to “tcpdump”.

Normally I’d respond to that with “if you don’t know how to use tcpdump, then you’d have even more trouble reading the results.” - it’s a tool to dump network packets as seen on the interface. Are you familiar with network traffic, then “man tcpdump” should give you a good start. If you’re not familiar with the in-depths of network packets, then tcpdump most probably isn’t your tool of choice.

BUT: If the problem is per your thoughts, why does “ping” work when the default route is set. Like I said before, when the multi-path default route is set through all interfaces, everything works fine. All my pings through each of the interfaces brings a response.

Sorry, can’t tell you. I typically judge such situations by looking at what’s actually going on at the wire level after the system has tried to do all its “packet voodoo”. There’s a lot of complexity build into the IP stack nowadays and configuration can get confusing really fast - I like to avoid spending too much time barking up the wrong tree. Only after I know what wrong result was created, I try to find out where and who the culprit is.

Regards,
Jens

ramesh tn wrote:
[color=blue]

why does “ping” work when the
default route is set. Like I said before, when the multi-path default
route is set through all interfaces, everything works fine. All my
pings through each of the interfaces brings a response.[/color]

  1. Your VLANs all use private IP’s

  2. you have separate public networks for each ISP.

  3. Packets leaving from a specific VLAN/ISP will be assigned the public
    IP address associated with that particular ISP.

  4. Responses to your pings will be routed back through the ISP from
    which they were sent.

So, while you may not know from which interface the ping originated,
you should always see an echo reply because the response is returned to
the originating interface.

The problem as you described it is that you can’t determine which
interface will be used unless you change the default route. If you do
what I suggested in my last post and set a specific route to a specific
device on each ISP’s network, you could then determine if a specific
interface were down.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

ramesh tn wrote:
[color=blue]

FURTHER: Based on some googling, I also set the rp_filter from the
default “1” to “0”. System is currently set to use “0”. Still no help.[/color]

Perhaps you have already seen this, but I thought I would pass it along.

TID 7007649
Applying SLES 11 SP 1 Causing Communication Issues
http://www.novell.com/support/viewContent.do?externalId=7007649&sliceId=1


Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below…

Thx Kevin…this fixed my issue.

[QUOTE=ramesh_tn;2948]My problem, if you remove all the vlan, load balancing etc, is very simple.
I am unable to ping anything through vlan4, if the default route is not through vlan 4. The minute i change default route to use vlan4 things start working. If I change default route to use vlan5, that starts working, but vlan4 stops. This happens for all the interfaces. In fact, things work on all interfaces when the muti-path default route is set to use all 4 interfaces. [/QUOTE]

Hi,

I don’t know if you have solved this yet, but setting
“net.ipv4.conf.all.rp_filter = 0” in /etc/sysctl.conf solved it for me.

[QUOTE=tgm_its;7041]Hi,

I don’t know if you have solved this yet, but setting
“net.ipv4.conf.all.rp_filter = 0” in /etc/sysctl.conf solved it for me.[/QUOTE]

I also had an issue that turned out to be ICMP redirects. In addition to rp_filter setting above, I set the following to disallow redirects, which is the recommended setting if you’re not using your linux server as a router:

# /etc/sysctl.conf net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.default.secure_redirects = 0 net.ipv4.conf.default.send_redirects = 0

FYI, redirect messages were sent from my router whenever a leg of our MPLS went down. I could see the redirect stored in cache on my host 172.20.64.25. This was a redirect for a route between two specific hosts: 172.20.64.25 to 172.20.17.18.

$ ip route list cache match 172.20.17.18 172.20.17.18 via 172.20.64.11 dev eth1 src 172.20.64.25 cache <redirected> ipid 0x3f1a rtt 15ms rttvar 11ms ssthresh 61 cwnd 61

Doing ip route flush cache did not fix the issue due to some kind of bug. The cache apparently clears itself after 10 minutes, but only if you’re not attempting any communication between the two specific hosts.

I had to reboot to clear the cache. I’m hoping the sysctl settings above will work - haven’t had any network issues to be able to test it, and haven’t tried any artificial tests.