Described herein is the practice of using IPMP (IP MultiPathing) and VLAN
tagged (802.1q trunking) interfaces layered together for the purpose of
port considerations (read: consolidation) and HA (high availability).
The following points should be noted relevant to the host used in the
following examples:tagged (802.1q trunking) interfaces layered together for the purpose of
port considerations (read: consolidation) and HA (high availability).
The following points should be noted relevant to the host used in the
OS: Linux (CentOS 5.2)
Kernel: 2.6.18-92.el5
Host: hostA
shell prompt: hostA [0]
phys net ints: eth4, eth5
primary phys: eth4
logical ints: bond0, bond0.219, bond0.348
VLANs: 219 (10.0.1.0/24), 348 (10.4.3.0/24)
ipaddr: 10.0.1.23 (vlan 219), 10.4.3.89 (vlan 348)
channel mode: active-backup
The layered layout of VLAN tagged IPMP interfaces follows:
3: IP addr (network access) 10.0.1.23, 10.4.3.89
2: vlan tagged interfaces bond0.219, bond0.348
1: channel bonded interface bond0
0: ethernet interfaces eth4, eth5
Configuration of the above setup can be done with or without a reboot,
however it may be simpler with a reboot as it ensures proper ordering of
the loading of the relevant kernel modules. For the purpose of this doc,
configuration will be done under the pretense of a reboot. To begin,
/etc/modprobe.conf will need to be updated, adding the following two
lines:
alias bond0 bonding
options bond0 mode=1 miimon=100 primary=eth4
The above tells the Linux kernel to enable channel bonding on bootup and
create the logical bond interface of bond0. The second line sets options
for the logical interface to mode=1 (active-backup policy), miimon=100
(check the MII link every 100 milliseconds for availability of the
active interface), and primary=eth4 (sets the primary interface to eth4).
In order to have everything represented appropriately in services like
SNMP, such as IP addr to interface association, the bonding alias must
come before the ethernet aliases in modprobe.conf. The options would
fall, subsequently, after the ethernet aliases, as at least one interface
is being specified, thus the module for that interface must already be
loaded by the kernel. The following is a sample /etc/modprobe.conf,
in full:
hostA [0] cat /etc/modprobe.conf
alias bond0 bonding
alias eth0 e1000e
alias eth1 e1000e
alias eth2 e1000e
alias eth3 e1000e
alias eth4 forcedeth
alias eth5 forcedeth
options bond0 mode=1 miimon=100 primary=eth4
alias scsi_hostadapter 3w-9xxx
alias scsi_hostadapter1 sata_nv
alias scsi_hostadapter2 usb-storage
Each involved network interface must now be configured. As the
intention is a VLAN tagged IPMP connection with no further complexities,
both physical network interfaces will receive minimal configuration.
To retain this across reboots, the following two files will need to be
modified, respective to the interface:
/etc/sysconfig/network-scripts/ifcfg-eth4
/etc/sysconfig/network-scripts/ifcfg-eth5
The contents of each are as follows:
hostA [0] cat /etc/sysconfig/network-scripts/ifcfg-eth4
# nVidia Corporation MCP55 Ethernet
DEVICE=eth4
HWADDR=00:30:48:7E:ED:54
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
hostA [0] cat /etc/sysconfig/network-scripts/ifcfg-eth5
# nVidia Corporation MCP55 Ethernet
DEVICE=eth5
HWADDR=00:30:48:7E:ED:55
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
Of note, the interfaces are configured to be brought online at bootup,
though not configured with an IP addr or to use DHCP. This is
necessary to support our channel bonded (IPMP) logical interface.
The two parameters necessary for bonding are 'MASTER=bond0' (specifying
which logical bonded interface the physical interface supports) and
'SLAVE=yes' (to detail the role of this physical interface relative to
the bonded interface). The bonded interface also must be configured:
hostA [0] cat /etc/sysconfig/network-scripts/ifcfg-bond0
# bond0, coupling of eth4 and eth5
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
Similar to the physical interfaces, the bonded interface (bond0) is
set to online at bootup, though not configured with an IP addr or to
use DHCP. That is because VLAN tagged interfaces will be created on
top of this. Of note, a HWADDR statement is not included because bonded
interfaces utilize the HWADDR of one of the underlying supporting physical
interfaces, typically the first attached. Now that the IPMP interface
(bond0) has be configured, VLAN tagged virtual interfaces can be layered
over bond0, just as they would normally for a physical interface.
The configuration for such would appear as follows:
hostA [0] cat /etc/sysconfig/network-scripts/ifcfg-bond0.219
# vlan 219, 10.0.1.0/24
DEVICE=bond0.219
ONBOOT=yes
TYPE=Ethernet
IPADDR=10.0.1.23
VLAN=yes
NETWORK=10.0.1.0
NETMASK=255.255.255.0
BROADCAST=10.0.1.255
IPV6INIT=no
hostA [0] cat /etc/sysconfig/network-scripts/ifcfg-bond0.348
# vlan 348, 10.4.3.0/24
DEVICE=bond0.348
ONBOOT=yes
TYPE=Ethernet
IPADDR=10.4.3.89
VLAN=yes
NETWORK=10.4.3.0
NETMASK=255.255.255.0
BROADCAST=10.4.3.255
IPV6INIT=no
The above configurations relevant to VLANs 219 and 348 are no different
that any other VLAN tagged physical interface, aside from specifying
the DEVICE as a bond device.
Now that all configuration has been done, the only thing necessary to
still do is simply reboot the host and begin using the new VLAN tagged
IPMP interfaces, bond0.219 and bond0.348. Following the reboot, the
output from 'ifconfig' regarding the involved interfaces should appear
similar to:
hostA [0] ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:30:48:7E:ED:54
inet6 addr: fe80::230:48ff:fe7e:ed54/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:5802545 errors:0 dropped:0 overruns:0 frame:0
TX packets:1259 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2159497822 (2.0 GiB) TX bytes:153407 (149.8 KiB)
bond0.219 Link encap:Ethernet HWaddr 00:30:48:7E:ED:54
inet addr:10.0.1.23 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::230:48ff:fe7e:ed54/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:297071 errors:0 dropped:0 overruns:0 frame:0
TX packets:1236 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:52265326 (49.8 MiB) TX bytes:140243 (136.9 KiB)
bond0.348 Link encap:Ethernet HWaddr 00:30:48:7E:ED:54
inet addr:10.4.3.89 Bcast:10.4.3.255 Mask:255.255.255.0
inet6 addr: fe80::230:48ff:fe7e:ed54/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:297071 errors:0 dropped:0 overruns:0 frame:0
TX packets:1236 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:52265326 (49.8 MiB) TX bytes:140243 (136.9 KiB)
[snip...]
eth4 Link encap:Ethernet HWaddr 00:30:48:7E:ED:54
inet6 addr: fe80::230:48ff:fe7e:ed54/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:2894853 errors:0 dropped:0 overruns:0 frame:0
TX packets:1180 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1084599554 (1.0 GiB) TX bytes:147569 (144.1 KiB)
Interrupt:114 Base address:0x6000
eth5 Link encap:Ethernet HWaddr 00:30:48:7E:ED:54
inet6 addr: fe80::230:48ff:fe7e:ed54/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:2907701 errors:0 dropped:0 overruns:0 frame:0
TX packets:100 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1074898808 (1.0 GiB) TX bytes:10246 (10.0 KiB)
Interrupt:122 Base address:0x8000
A check of /var/log/messages after bootup should show that the kernel
had no issues with the above setup as well:
Feb 24 16:12:49 hostA kernel: Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)
Feb 24 16:12:49 hostA kernel: bonding: MII link monitoring set to 100 ms
Feb 24 16:12:49 hostA kernel: bonding: bond0: Adding slave eth4.
Feb 24 16:12:49 hostA kernel: bonding: bond0: making interface eth4 the new active one.
Feb 24 16:12:49 hostA kernel: bonding: bond0: first active interface up!
Feb 24 16:12:49 hostA kernel: bonding: bond0: enslaving eth4 as an active interface with an up link.
Feb 24 16:12:49 hostA kernel: bonding: bond0: Adding slave eth5.
Feb 24 16:12:49 hostA kernel: bonding: bond0: enslaving eth5 as a backup interface with an up link.
Feb 24 16:12:49 hostA kernel: 802.1Q VLAN Support v1.8 Ben Greear [greearb@candelatech.com]
Link transitions can also be seen within /var/log/messages. The following
logs are the result of physically removing the ethernet cable from eth4,
forcing a link transition and then reconnecting it. The same was also
done for eth5, as can be seen as well:
# removal of ethernet cable from eth4
Feb 24 16:15:26 hostA kernel: eth4: link down.
Feb 24 16:15:26 hostA kernel: bonding: bond0: link status definitely down for interface eth4, disabling it
Feb 24 16:15:26 hostA kernel: bonding: bond0: making interface eth5 the new active one.
# reconnecting of ethernet cable to eth4
Feb 24 16:15:31 hostA kernel: eth4: link up.
Feb 24 16:15:31 hostA kernel: bonding: bond0: link status definitely up for interface eth4.
Feb 24 16:15:31 hostA kernel: bonding: bond0: making interface eth4 the new active one.
# removal of ethernet cable from eth5
Feb 24 16:16:37 hostA kernel: eth5: link down.
Feb 24 16:16:37 hostA kernel: bonding: bond0: link status definitely down for interface eth5, disabling it
# reconnecting of ethernet cable to eth5
Feb 24 16:16:54 hostA kernel: eth5: link up.
Feb 24 16:16:54 hostA kernel: bonding: bond0: link status definitely up for interface eth5.
While the above described solution is great towards port consolidation
and availability, one possible drawback must be noted. Due to nature of
Spanning Tree Protocol (STP) (at the network switch) as well as due to
the usage of 802.1q trunking on the switch ports supporting the above,
it can take between 30 - 60 seconds before traffic resumes following a
link transition on the host. This is due to the potential network issues
in using 'portfast' on these particular ports. Also of note, without the
usage of VLAN tagging, portfast could otherwise be enabled allowing link
transitions to appear instantaneous. Regardless, link transitions on the
Linux host side occur immediately. Should instantaneous link transitions
be desirable for the above documented setup, a request needs to
be made of your friendly neighborhood network engineering (NetEng)
team to enable Spanning Tree's 'portfast trunk' configuration on the
particular switch ports in question. NetEng may be hesitant to do this,
though for good reason, and will only consider the request based upon
the fact of a single server connecting to the two ports in question.
Please keep this in mind during your request. Also note, it is your
responsibility to inform NetEng of physical changes regarding those
switch ports, such as the host being removed, being moved to another
rack, thus possibly new network connections, or a new host re-using the
existing network connections. This is to allow NetEng to appropriately
configure / unconfigure the switch ports configured with 'portfast trunk'
so that network issues don't occur (such as loops).
