06 July 2011

VLAN Tagging and Vanity Naming in Solaris 11

While awaiting Solaris 11 to be released, I've recently been poking
around in Solaris 11 Express for a preview of what to expect.  In setting
up a test box, I needed to set up VLAN tagging (802.1q trunking) on
an interface.  Due to some of the new Solaris 11 networking features,
configuring a VLAN tagged interface is slightly different from what
I recall.  Our details this time are:
        HOST:           solex
        PROMPT:         solex [0]
        OS:             Solaris 11 Express (release 201011, x86)
        INTERFACES:     e1000g0, e1000g2 (mon_net0)
        VLANs:          486 (192.0.45.0/24), 572 (10.23.154.0/24; mailnet0)
To start things off, we need to know what interfaces we have.  In Solaris
10 we would have used 'dladm show-dev', though in Solaris 11 the command
is 'dladm show-phys'.  Via 'dladm' options "show-phys", "show-link"
and "show-vlan", neither interface "e1000g0" nor any VLANs appear to be
configured yet:
        solex [0] /usr/sbin/dladm show-phys
        LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
        e1000g0      Ethernet             unknown    0      half      e1000g0
        e1000g1      Ethernet             up         1000   full      e1000g1
        e1000g2      Ethernet             unknown    0      half      e1000g2
        solex [0] /usr/sbin/dladm show-link
        LINK        CLASS     MTU    STATE    BRIDGE     OVER
        e1000g0     phys      1500   unknown  --         --
        e1000g1     phys      1500   up       --         --
        e1000g2     phys      1500   unknown  --         --
        solex [0] /usr/sbin/dladm show-vlan
        solex [0]
Below, "e1000g0" is plumbed and brought up, after which we see the
state is "up" rather than unknown and the device now registers a speed
and duplex:
        solex [0] /usr/sbin/ifconfig e1000g0 plumb
        solex [0] /usr/sbin/ifconfig e1000g0 up   
        solex [0] /usr/sbin/dladm show-phys       
        LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
        e1000g0      Ethernet             up         1000   full      e1000g0
        e1000g1      Ethernet             up         1000   full      e1000g1
        e1000g2      Ethernet             unknown    0      half      e1000g2
        solex [0] /usr/sbin/dladm show-link
        LINK        CLASS     MTU    STATE    BRIDGE     OVER
        e1000g0     phys      1500   up       --         --
        e1000g1     phys      1500   up       --         --
        e1000g2     phys      1500   unknown  --         --
My intention was to tag interface "e1000g0" with VLAN 486.  To do so
historically, we would plumb interface "e1000g486000" via 'ifconfig'.
The interface instance breaks down to:
        driver-name + VLAN_ID * 1000 + device-instance
As seen below, however, 'ifconfig' no longer instantiates new VLAN tagged
interfaces as that role now falls to 'dladm':
        solex [0] /usr/sbin/ifconfig e1000g486000 plumb
        ifconfig: cannot plumb e1000g486000: Could not open DLPI link
        solex [1] /usr/sbin/dladm create-vlan -l e1000g0 -v 486 
        solex [0] /usr/sbin/dladm show-vlan                    
        LINK            VID      OVER         FLAGS
        e1000g486000    486      e1000g0      -----
        solex [0] /usr/sbin/ifconfig e1000g486000 plumb
        solex [0] /usr/sbin/ifconfig e1000g486000 192.0.45.176 netmask 255.255.254.0 broadcast + up
Once 'dladm' created the new "e1000g486000" interfaces above, I could then
plumb and IP the interface.  Ok, so one additional step, not a big deal.
By default, 'dladm' creates the new interface following the logic above.
With Solaris 11, however, we now have the availability of using "vanity"
names for our VLANs.  I've setup "e1000g0" to be tagged with VLAN 572
below, though rather than a resulting interface of "e1000g572000" we'll
specify an interface name of "mailnet0":
        # mailnet0: vanity name must end in number
        solex [0] /usr/sbin/dladm create-vlan -l e1000g0 -v 572 mailnet0
        solex [0] /usr/sbin/ifconfig mailnet0 plumb
        solex [0] /usr/sbin/ifconfig -a
        lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                inet 127.0.0.1 netmask ff000000
        e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 8
                inet 0.0.0.0 netmask ff000000
                ether 8:0:27:6:a3:22
        e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
                inet 192.168.56.50 netmask ffffff00 broadcast 192.168.56.255
                ether 8:0:27:cd:62:a7
        e1000g486000: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 9
                inet 192.0.45.176 netmask fffffe00 broadcast 192.0.1.255
                ether 8:0:27:6:a3:22
        mailnet0: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 10
                inet 0.0.0.0 netmask 0
                ether 8:0:27:6:a3:22
        lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
                inet6 ::1/128
        solex [0] /usr/has/bin/vi /etc/hostname.e1000g0 /etc/hostname.e1000g486000 /etc/hostname.mailnet0
        solex [0] for i in e1000g0 e1000g486000 mailnet0
        > do echo "**** hostname.${i} ****" ; /bin/cat /etc/hostname.${i} ; echo
        > done
        **** hostname.e1000g0 ****
        0.0.0.0

        **** hostname.e1000g486000 ****
        192.0.45.176 netmask 255.255.254.0 broadcast + up

        **** hostname.mailnet0 ****
        10.23.154.76 netmask 255.255.255.240 broadcast + up
After creating the interface above, for review I ran 'ifconfig' followed
by updating /etc/hostname.INTERFACE for each interface.  The following
review via 'dladm' gives us the link state for the new logical interfaces
and the associated physical interfaces and VLAN IDs:
        solex [0] /usr/sbin/dladm show-link
        LINK        CLASS     MTU    STATE    BRIDGE     OVER
        e1000g0     phys      1500   up       --         --
        e1000g1     phys      1500   up       --         --
        e1000g2     phys      1500   unknown  --         --
        e1000g486000 vlan     1500   up       --         e1000g0
        mailnet0    vlan      1500   up       --         e1000g0
        solex [0] /usr/sbin/dladm show-vlan
        LINK            VID      OVER         FLAGS
        e1000g486000    486      e1000g0      -----
        mailnet0        572      e1000g0      -----
To maintain backward compatibility, we'll still see our physical devices
under /dev.  Additionally though, 'dladm' will maintain separate character
device files for both physical and logical devices under /dev/net.
From the listings below, the major number for our logical VLAN interfaces
relates to "vnic":
        solex [0] /bin/ls -lL /dev/e1000g0
        crw-rw-rw-   1 root     sys       20,  1 Jul  5 23:18 /dev/e1000g0
        solex [0] /bin/ls -lL /dev/net    
        total 0
        crw-rw-rw-   1 root     sys       20,  1 Jul  5 23:09 e1000g0
        crw-rw-rw-   1 root     sys       20,  2 Jul  5 23:09 e1000g1
        crw-rw-rw-   1 root     sys       20,  3 Jul  5 23:15 e1000g2
        crw-rw-rw-   1 root     sys       58, 1007 Jul  5 23:14 e1000g486000
        crw-rw-rw-   1 root     sys       58, 1009 Jul  5 23:14 mailnet0
        solex [0] /bin/egrep ' 20$| 58$' /etc/name_to_major
        e1000g 20 
        vnic 58
To maintain vanity name associations, physical interface correlation,
and associated VLAN information, 'dladm' also maintains its very own
configuration file:
        solex [0] /bin/grep -v ^# /etc/dladm/datalink.conf

        e1000g0 class=int,1;media=int,4;phyinst=int,1;phymaj=int,20;devname=string,e1000g0;
        e1000g1 class=int,1;media=int,4;phyinst=int,2;phymaj=int,20;devname=string,e1000g1;
        e1000g2 class=int,1;media=int,4;phyinst=int,3;phymaj=int,20;devname=string,e1000g2;
        e1000g486000    class=int,2;media=int,4;linkover=string,e1000g0;vid=int,486;
        mailnet0        class=int,2;media=int,4;linkover=string,e1000g0;vid=int,572;
The "datalink.conf" file above is read at system boot by 'dlmgmtd'
(dladm's daemon counterpart) and opened / modified by 'dladm' for changes.
As a final example of vanity names, we can use 'dladm' to rename a
physical interface.  Here we'll rename physical interface "e1000g2" to
"mon_net0", reviewing the update via 'dladm show-phys' and the entry in
"datalink.conf":
        solex [0] /usr/sbin/dladm rename-link e1000g2 mon_net0
        solex [0] /usr/sbin/dladm show-phys
        LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
        e1000g0      Ethernet             up         1000   full      e1000g0
        e1000g1      Ethernet             up         1000   full      e1000g1
        mon_net0     Ethernet             unknown    0      half      e1000g2
        solex [0] /bin/grep mon_net0 /etc/dladm/datalink.conf
        mon_net0        class=int,1;media=int,4;phyinst=int,3;phymaj=int,20;devname=string,e1000g2;
Of note, vanity names for devices (logical and physical) appear to have
several rules:
        - must be alpha-numeric 
        - must start with a letter
        - can contain underscores but not hyphens
        - must end in a number
        - may be up to 31 characters in length.
Furthermore, once a device has been renamed, commands will only function
on the new name.  Thus from our example "e1000g2" will not work, however
"mon_net0" will:
        solex [0] /usr/sbin/ifconfig e1000g2 plumb
        ifconfig: cannot plumb e1000g2: Could not open DLPI link
        solex [1] /usr/sbin/ifconfig mon_net0 plumb
        solex [0] /usr/sbin/ifconfig mon_net0 
        mon_net0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
                inet 0.0.0.0 netmask 0 
                ether 8:0:27:0:8b:2a 
        solex [0] /usr/sbin/dladm rename-link mon_net0 e1000g2
        /usr/sbin/dladm: rename operation failed: link busy
        solex [1] /usr/sbin/ifconfig mon_net0 unplumb
        solex [0] /usr/sbin/dladm rename-link mon_net0 e1000g2
Also seen above, if a device is in use further renaming operations
will fail until the device is removed from use.  To close things out,
the following unplumbs the remaining interfaces, removes the VLANs we
created, removes the respective "hostname.INTERFACE" files, and displays
the cleaned up "datalink.conf" file:
        solex [0] /usr/sbin/ifconfig mailnet0 unplumb
        solex [0] /usr/sbin/ifconfig e1000g486000 unplumb
        solex [0] /usr/sbin/dladm delete-vlan mailnet0
        solex [0] /usr/sbin/dladm delete-vlan e1000g486000
        solex [0] /usr/sbin/ifconfig e1000g0 unplumb
        solex [0] /usr/bin/rm /etc/hostname.e1000g0 /etc/hostname.e1000g486000 /etc/hostname.mailnet0
        solex [0] /bin/grep -v ^# /etc/dladm/datalink.conf

        e1000g0 class=int,1;media=int,4;phyinst=int,1;phymaj=int,20;devname=string,e1000g0;
        e1000g1 class=int,1;media=int,4;phyinst=int,2;phymaj=int,20;devname=string,e1000g1;
        e1000g2 class=int,1;media=int,4;phyinst=int,3;phymaj=int,20;devname=string,e1000g2;
        solex [0]
see also:
    VLAN Tagged Interfaces (Solaris)

No comments: