06 November 2010

Identifying DMP Subpaths in Solaris

Recently I stumbled across one of my notes detailing a quick way to
identify the underlying subpaths supporting VxDMP devices:
        frost [0] /usr/sbin/vxdisk path
        SUBPATH                     DANAME               DMNAME       GROUP        STATE
        c2t20301E67B00F1032d12s2    c2t20301E67B00F1032d12s2 app6         appdg        ENABLED
        c3t20311E67B00F1032d12s2    c2t20301E67B00F1032d12s2 app6         appdg        ENABLED
        c2t20301E67B00F1032d13s2    c2t20301E67B00F1032d13s2 bkupdg01     bkupdg       ENABLED
        c2t20301E67B00F1032d14s2    c2t20301E67B00F1032d14s2 bkupdg02     bkupdg       ENABLED
The output above is simple enough, the device in column 0 (SUBPATH)
supports the DMP device listed in column 1 (DANAME).  This got me to
thinking, how could you determine the subpaths for a Sun MPxIO device?
Well, Sun also has an easy means of showing this information, though
there's a catch.  The following is how the output of the 'stmsboot'
command should look:
        asimov [0] /usr/sbin/stmsboot -L
        non-STMS device name                    STMS device name
        ----------------------------------------------------------------
        /dev/rdsk/c8t304e83726743d120d2         /dev/rdsk/c5t60013A7947B5839F0000083B674E8E13d0
        /dev/rdsk/c8t304e83726743d120d4         /dev/rdsk/c5t60013A7947B5839F0000083C674E68B2d0
        /dev/rdsk/c9t304f83726743d120d2         /dev/rdsk/c5t60013A7947B5839F0000083B674E8E13d0
        /dev/rdsk/c9t304f83726743d120d4         /dev/rdsk/c5t60013A7947B5839F0000083C674E68B2d0
So, what's the catch?  According to Sun's documentation, "if you have run
devfsadm -C or performed a reconfiguration reboot then the old device
paths will not exist and the stmsboot -L command will fail to provide
this information."  Well that's a problem.  The first things that come
to my mind are reconfiguration boots after either patching or adding
new devices.  Another issue is if MPxIO is enabled for capable, newly
added devices by default, dev tree entries for the underlying subdisks
won't exist as only those for the MPxIO device(s) will be created.  As a
result, 'stmsboot' still won't work.  Additionally, some x86 versions
of Solaris 10 were released without 'stmsboot'.  Overall, this solution
appears to be unreliable.  There is, however, another way of determining
this information, though with a little bit of extra effort.

As mentioned before, the above 'stmsboot' output is how it should look
(another box was used for point of reference).  Unfortunately, MPxIO
is enabled by default on 'asimov', so no dev tree entries exist for
the underlying subpaths.  Also, 'asimov' is running one of those x86
Solaris 10 versions that didn't have 'stmsboot' included.  To identify
the subpaths, we first need to identify the current MPxIO devices:

        asimov [0] /usr/sbin/mpathadm list lu
                /dev/rdsk/c5t60013A7947B5839F0000083C674E68B2d0s2
                        Total Path Count: 2
                        Operational Path Count: 2
                /dev/rdsk/c5t60013A7947B5839F0000083B674E8E13d0s2
                        Total Path Count: 2
                        Operational Path Count: 2

For each device path listed by 'mpathadm', we can use 'luxadm' to display
the underlying paths:

        asimov [0] /usr/sbin/luxadm display /dev/rdsk/c5t60013A7947B5839F0000083C674E68B2d0s2
        DEVICE PROPERTIES for disk: /dev/rdsk/c5t60013A7947B5839F0000083C674E68B2d0s2
          Vendor:               STK
          Product ID:           FLEXLINE 380
          Revision:             0615
          Serial Num:           3T83746284 
          Unformatted capacity: 153600.000 MBytes
          Write Cache:          Enabled
          Read Cache:           Enabled
            Minimum prefetch:   0x100
            Maximum prefetch:   0x100
          Device Type:          Disk device
          Path(s):

          /dev/rdsk/c5t60013A7947B5839F0000083C674E68B2d0s2
          /devices/scsi_vhci/disk@g60013a7947b5839f0000083c674e68b2:c,raw
           Controller           /dev/cfg/c8
            Device Address              304e83726743d120,4
            Host controller port WWN    230000f16e752286
            Class                       secondary
            State                       ONLINE
           Controller           /dev/cfg/c9
            Device Address              304f83726743d120,4
            Host controller port WWN    230000f16e952286
            Class                       primary
            State                       STANDBY

This provides us with the controller for each path and the device address,
from which we can reconstruct the underlying logical device subpaths
even though they don't currently exist in the dev tree:

        /dev/rdsk/c8t304E83726743D120d4s2
        /dev/rdsk/c9t304F83726743D120d4s2

The controller is obvious, but what about the target and disk IDs?
In this case, the disk is SAN attached and the device address provides
us with the WWN of the presented LUN (304E83726743D120 (target)) and
the LUN number (4 (disk)), comma delimited.  (Of note, the character
attributes in the WWN need to be capitalized for a dev path.  Also,
'luxadm' presents the LUN number as a HEX value, thus it needs to be
translated to decimal for use as the disk value.)  To support the above,
let's take a look back at host 'frost'.  While 'frost' uses VxDMP instead
of MPxIO, we can see that our assertions of the device address hold true.
Using the first 'DANAME' listed earlier to pass off to 'luxadm':

        frost [0] /usr/sbin/luxadm display /dev/rdsk/c2t20301E67B00F1032d12s2
        DEVICE PROPERTIES for disk: /dev/rdsk/c2t20301E67B00F1032d12s2
          Vendor:               STK    
          Product ID:           BladeCtlr B210
          Revision:             0612
          Serial Num:           3T82719473     
          Unformatted capacity: 955423.750 MBytes
          Write Cache:          Enabled
          Read Cache:           Enabled
            Minimum prefetch:   0x100
            Maximum prefetch:   0x100
          Device Type:          Disk device
          Path(s):

          /dev/rdsk/c2t20301E67B00F1032d12s2
          /devices/pci@0,0/pci10de,5d@e/pci10df,fc22@0/fp@0,0/disk@w20301e67b00f1032,c:c,raw
           Controller           /dev/cfg/c2
            Device Address              20301e67b00f1032,c
            Host controller port WWN    10000000c9785d02
           Controller           /dev/cfg/c3
            Device Address              20311e67b00f1032,c
            Host controller port WWN    10000000c9785d01

As a result we have:

        /dev/rdsk/c2t20301E67B00F1032d12s2
        /dev/rdsk/c3t20311E67B00F1032d12s2

Remember, the LUN value is in HEX, so '0x0c' = '12' in decimal.
Since this is a VxDMP setup, the underlying devices will still exist:

        frost [0] /usr/bin/ls /dev/rdsk/c2t20301E67B00F1032d12s2 /dev/rdsk/c3t20311E67B00F1032d12s2
        /dev/rdsk/c2t20301E67B00F1032d12s2  /dev/rdsk/c3t20311E67B00F1032d12s2

Aside from simple curiousity, knowing the subpaths of a DMP device is
handy for several reasons.  You might decide to remove a device from
DMP control and want / need to know what the resulting disks and device
paths will be.  Instead there may be an issue with a particular IO path
and you need to identify the specific device to resolve the matter.
Perhaps you are just trying to identify the devices on your system.