16 December 2010

Using Logical Network Interfaces

Logical interfaces (logints, vips) are sometimes overlooked though can be
quite handy.  For instance, they can be used for IP based hosting (think
webservers), failover of clustered services, testing, providing network
connectivity for zones, etc.  Additionally, they are relatively trivial
to setup and use (see note 1).  As an aside, in the details below, the
terms logints and vips are used interchangeably.  The following details
these interfaces for each of our three hosts:
        HOSTS:          beastie, snorkle, tux
        PROMPT:         host [0]
        OS:             FreeBSD 8.1 (beastie)
                        Solaris 10  (snorkle)
                        CentOS 5.4 Linux  (tux)
For point of examples, FreeBSD (bsd) is used, however, both Solaris
(sol) and CentOS (lin) follow at the end.  Let's start with listing
our available interfaces to identify which will be configured for a vip
(step 1, sol|lin):
        beastie [0] /sbin/ifconfig -a
        em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
                ether 08:00:27:ce:11:c0
                inet 10.0.23.181 netmask 0xfffffe00 broadcast 10.0.23.255
                media: Ethernet autoselect (1000baseT <full-duplex>)
                status: active
        em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
                ether 08:00:27:d2:4e:8b
        <snip...>
From the above, I want to configure a vip on em0 with the next available
IP address (step 2, sol|lin):
        beastie [0] /sbin/ifconfig em0 alias 10.0.23.182 netmask 255.255.254.0 up
        beastie [0] /sbin/ifconfig em0
        em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
                ether 08:00:27:ce:11:c0
                inet 10.0.23.181 netmask 0xfffffe00 broadcast 10.0.23.255
                inet 10.0.23.182 netmask 0xfffffe00 broadcast 10.0.23.255
                media: Ethernet autoselect (1000baseT <full-duplex>)
                status: active
We now have em0 configured to respond to two separate IP addresses.
Now, to retain the configuration for bootup (step 3, sol|lin):

    # edit /etc/rc.conf adding either of the following (see note 2):
        ipv4_addrs_em0="10.0.23.182/23"

                or

        ifconfig_em0_alias0="inet 10.0.23.182 netmask 0xfffffe00"
    # the following is the result (using ipv4_addrs_em0):
        beastie [127] /bin/cat /etc/rc.conf

        # -- sysinstall generated deltas -- # Thu Sep 23 19:40:11 2010
        # Created: Thu Sep 23 19:40:11 2010
        # Enable network daemons for user convenience.
        # Please make all changes to this file, not to /etc/defaults/rc.conf.
        # This file now contains just the overrides from /etc/defaults/rc.conf.
        defaultrouter="127.0.0.1"
        hostname="beastie"
        ifconfig_em0="inet 10.0.23.181 netmask 255.255.254.0"
        ipv4_addrs_em0="10.0.23.182/23"
        ifconfig_em1="inet 192.168.56.35  netmask 255.255.255.0"
        inetd_enable="YES"
        sshd_enable="YES"
        sendmail_enable="NO"
        sendmail_submit_enable="NO"
At this point, the logint has been created and is available for use.
If desired, an entry could be added to /etc/hosts.  If the logint /
vip is no longer necessary, delete its alias entry from /etc/rc.conf
and execute the following (step 4, sol|lin):
        beastie [0] /sbin/ifconfig em0 -alias 10.0.23.182 netmask 255.255.254.0
        beastie [0] /sbin/ifconfig em0
        em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
                ether 08:00:27:ce:11:c0
                inet 10.0.23.181 netmask 0xfffffe00 broadcast 10.0.23.255
                media: Ethernet autoselect (1000baseT <full-duplex>)
                status: active

Solaris:

step 1 (bsd|lin):
        snorkle [0] /sbin/ifconfig -a
        lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                inet 127.0.0.1 netmask ff000000
        e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
                inet 10.0.23.191 netmask ffffff00 broadcast 10.0.23.255
                ether 8:0:27:56:c2:d
        e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
                inet 192.168.56.15 netmask ffffff00 broadcast 192.168.56.255
                ether 8:0:27:cb:f8:20
step 2 (bsd|lin), Solaris adds a visible logical interface to the physical
    interface identified by e1000g0.  Solaris also provides two separate
    command invocations, depending on your intention (note 3):
        snorkle [0] /sbin/ifconfig e1000g0 addif 10.0.23.192 netmask 255.255.254.0 broadcast + up
        Created new logical interface e1000g0:1

                or

        snorkle [0] /sbin/ifconfig e1000g0:3 plumb
        snorkle [0] /sbin/ifconfig e1000g0:3 10.0.23.194 netmask 255.255.254.0 broadcast + up

        snorkle [0] /sbin/ifconfig -a
        lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                inet 127.0.0.1 netmask ff000000
        e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
                inet 10.0.23.191 netmask ffffff00 broadcast 10.0.23.255
                ether 8:0:27:56:c2:d
        e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
                inet 10.0.23.192 netmask fffffe00 broadcast 10.0.23.255
        e1000g0:3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
                inet 10.0.23.194 netmask fffffe00 broadcast 10.0.23.255
step 3 (bsd|lin), create and configure the interface file:
        snorkle [0] /usr/bin/vi /etc/hostname.e1000g0:1
        snorkle [0] /usr/bin/cat /etc/hostname.e1000g0:1
        10.0.23.192 netmask 255.255.255.0 broadcast +
step 4 (bsd|lin), remove the logint.  Again, Solaris provide two ways (note 3):
        snorkle [0] /sbin/ifconfig e1000g0:3 unplumb

                or

        snorkle [0] /sbin/ifconfig e1000g0 removeif 10.0.23.192 
        snorkle [0] /sbin/ifconfig -a
        lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                inet 127.0.0.1 netmask ff000000
        e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
                inet 10.0.23.191 netmask ffffff00 broadcast 10.0.23.255
                ether 8:0:27:56:c2:d 
        e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
                inet 192.168.56.15 netmask ffffff00 broadcast 192.168.56.255
                ether 8:0:27:cb:f8:20 

Linux:

step 1 (bsd|sol):
        tux [0] /sbin/ifconfig -a
        eth0      Link encap:Ethernet  HWaddr 08:00:27:9E:A8:A8
                  inet addr:10.0.23.171  Bcast:10.0.23.255  Mask:255.255.254.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                  collisions:0 txqueuelen:1000 
                  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                  Interrupt:169 Base address:0xd020

        eth1      Link encap:Ethernet  HWaddr 08:00:27:53:7B:56  
                  inet addr:192.168.56.25  Bcast:192.168.56.255  Mask:255.255.255.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:869 errors:0 dropped:0 overruns:0 frame:0
        <snip...>
step 2 (bsd|sol):
        tux [0] /sbin/ifconfig eth0:1 10.0.23.172 netmask 255.255.254.0 up
        tux [0] /sbin/ifconfig -a
        eth0      Link encap:Ethernet  HWaddr 08:00:27:9E:A8:A8
                  inet addr:10.0.23.171  Bcast:10.0.23.255  Mask:255.255.254.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                  collisions:0 txqueuelen:1000
                  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                  Interrupt:169 Base address:0xd020 

        eth0:1    Link encap:Ethernet  HWaddr 08:00:27:9E:A8:A8  
                  inet addr:10.0.23.172  Bcast:10.0.23.255  Mask:255.255.254.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  Interrupt:169 Base address:0xd020 

        eth1      Link encap:Ethernet  HWaddr 08:00:27:53:7B:56  
        <snip...>
step 3 (bsd|sol), create and configure the interface file (note 4):
        tux [0] /usr/bin/cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth0:1
        tux [0] /bin/vi /etc/sysconfig/network-scripts/ifcfg-eth0:1
        tux [0] /bin/cat /etc/sysconfig/network-scripts/ifcfg-eth0:1
        # Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
        DEVICE=eth0:1
        BOOTPROTO=static
        BROADCAST=10.0.23.255
        HWADDR=08:00:27:9e:a8:a8
        IPADDR=10.0.23.172
        NETMASK=255.255.254.0
        NETWORK=10.0.22.0
        ONBOOT=yes
step 4 (bsd|sol):
        tux [0] /sbin/ip address delete 10.0.23.172/23 dev eth0:1
        tux [0] /sbin/ifconfig -a
        eth0      Link encap:Ethernet  HWaddr 08:00:27:9E:A8:A8
                  inet addr:10.0.23.171  Bcast:10.0.23.255  Mask:255.255.254.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                  TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
                  collisions:0 txqueuelen:1000
                  RX bytes:0 (0.0 b)  TX bytes:252 (252.0 b)
                  Interrupt:169 Base address:0xd020

        eth1      Link encap:Ethernet  HWaddr 08:00:27:53:7B:56
                  inet addr:192.168.56.25  Bcast:192.168.56.255  Mask:255.255.255.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:1923 errors:0 dropped:0 overruns:0 frame:0
        <snip...>


NOTES

note 1:  Any logints / vips created will have the same HWaddr (MAC addr)
    as the underlying physical interface.  Also, they will share the
    capacity of the network link.  If you have something that requires
    unique HWaddrs or can possibly saturate the link, vips may not be
    the way to go.  Also, though the above shows vips configured in the
    same network as the underlying physical interface, you can actually
    configure other networks instead.  Your network architecture must
    support this, however.

note 2:  According to the rc.conf man page, 'ipv4_addrs_int#' is favored
    over 'ifconfig_int#_alias#'.  The reason for this is if you have
    aliases 1 - 4 and 6, alias6 will never be added since the init script
    is looking for the next in line which would be alias5.  Additionally,
    it appears that there isn't a correlative for 'ipv6_addrs_int#', so
    'ifconfig_int#_alias#' would need to be used.  Providing that you
    understand that risk, either method could be used, at least for IPv4.
    The following illustrates multiple vips for em0 using each declaration
    statement:
        ipv4_addrs_em0="10.0.23.182/23 10.0.23.183/23"

                or

        ifconfig_em0_alias0="inet 10.0.23.182 netmask 0xfffffe00"
        ifconfig_em0_alias1="inet 10.0.23.183 netmask 0xfffffe00"
note 3:  Solaris provides two command syntaxes for for logints.
    The first, using 'addif' to 'ifconfig' creates a new logical interface
    at the next available logical instance number.  The second, using
    'plumb' to 'ifconfig' allows you to set the specific logical instance
    to create.  For removal, 'removeif' removes the specific logical
    interface identified by the IP address in question.  As with 'plumb',
    'unplumb' removes the specified logical instance by instance name.

note 4:  Linux has a nasty little habit of always trying to respond,
    including to things perhaps not within the same network segment.
    It will also respond to arp via the physical interface and the
    logical interfaces created rather than just picking one.  If this
    is an issue, have a look at the 'net.ipv4.conf.default.arp_announce'
    and 'net.ipv4.conf.default.arp_ignore' 'sysct.conf' variables.