19 October 2010

Solaris Devpath Enumeration

The purpose of this doc is show how to physically identify and enumerate
your disk devices on a Sun branded box.  The reasons to do this may
include identifying a disk for replacement, determining the bus it is
attached to, I/O balancing, etc.  Of note, this doc is only valid with Sun
hardware with Solaris running on it as Sun has been kind enough to release
details regarding such.  Sun details this information on sunsolve in:

        Doc ID:         208209
        Title:          Solaris[TM] Operating System: Matrix of Recognized Device Paths

The above sunsolve doc (MRDP) is typically updated every couple of months,
relative to new hardware platforms released.

The following details are required for device identification:

        platform:       sparc, x86, x64
        model:          Sun-Fire-V490, V240, SPARC-Enterprise-T5220, etc.
        ctds:           c0t0d0s2,
        device path:    /devices/pci@1e,600000/ide@d/dad@0,0:c

The following details are relevant to the examples in this document:

        host:           winston
        platform:       sparc
        model:          SPARC-Enterprise-T5220
        ctds:           c3t1938E70CD1642679d26s2
        controller:     3
        target:         1938E70CD1642679
        disk:           26
        device path:    /devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w1938e70cd1642679,1a:c
        shell prompt:   winston [0]

To get the platform and model, 'uname' can be used:

        winston [0] /usr/bin/uname -a
        SunOS winston 5.10 Generic_137111-07 sun4v sparc SUNW,SPARC-Enterprise-T5220

To get the device path, use 'ls':

        winston [0] /usr/bin/ls -l /dev/dsk/c3t1938E70CD1642679d26s2
        lrwxrwxrwx   1 root     root          90 Jan 13  2010 /dev/dsk/c3t1938E70CD1642679d26s2 -> ../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w1938e70cd1642679,1a:c

To simply identify where this disk is physically attached, MRDP can be
pulled up in a web browser, the appropriate model selected, and the path
referenced.  Stripping off '/devices' from the device path, MRDP shows:

        PCI-E Slot 4             /pci@0/pci@0/pci@8/pci@0/pci@2

This disk is attached at 'PCI-E Slot 4', which the MRDP further tells the
capabilities of the slot are 'x8 physical slot, x4 electrical signalling'.

Looking at the rest of the device path, starting 'SUNW,qlc@0,1', tells us
that this disk is attached via a fibre channel HBA (man qlc).  Further,
it appears that this may be a dual port HBA and the disk we are looking
at is on the second port as seen in 'qlc@0,1' as opposed to 'qlc@0'.
Since occasionally system logs will refer to a device by its device
ID, we can determine that by a quick 'grep' of the device path against

        winston [0] /usr/bin/grep 'pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w1938e70cd1642679,1a' /etc/path_to_inst
        "/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w1938e70cd1642679,1a" 124 "ssd"

The above begets us 'ssd124'.  To alleviate some of the leg work of the
above, you can use 'ioDev' to retrieve the same informtion.  (ioDev is
available on the Tools page.)  Sample output of ioDev shows:

winston [0] /usr/local/bin/ioDev
                                      -- ioDev v1.4 --
                            winston: sparc, SPARC-Enterprise-T5220
        Device                    IO Ident   Phys Device          Bus Speed
        c0t0d0                    sd2        DVD/CDROM
        c1t0d0                    sd0        Disk 0
        c2t50060E8005B15B00d6     ssd16      PCI-E Slot 4         PCI-E x8P/x4E
        c3t200500A0B8186895d23    ssd129     PCI-E Slot 4         PCI-E x8P/x4E
        c3t200500A0B8186895d24    ssd126     PCI-E Slot 4         PCI-E x8P/x4E
        c3t200500A0B8186895d25    ssd125     PCI-E Slot 4         PCI-E x8P/x4E
        c3t200500A0B8186895d26    ssd124     PCI-E Slot 4         PCI-E x8P/x4E
        c3t200500A0B8186895d27    ssd123     PCI-E Slot 4         PCI-E x8P/x4E
If we want no further information about a device, we can stop here. 
Given that this is a fibre attached device, let's see what else we can
dig up that's useful for identification / troubleshooting.  Proving the
assumption that this is the secondary port
on a dual port HBA, 'luxadm' shows:

        winston [0] /usr/sbin/luxadm -e port
        /devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0/fp@0,0:devctl    CONNECTED
        /devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0:devctl  CONNECTED
        /devices/pci@0/pci@0/pci@8/pci@0/pci@8/SUNW,qlc@0/fp@0,0:devctl    NOT CONNECTED
        /devices/pci@0/pci@0/pci@8/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0:devctl  CONNECTED
        /devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,qlc@0/fp@0,0:devctl    CONNECTED
        /devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,qlc@0,1/fp@0,0:devctl  CONNECTED

with our device path identified on the second line of the output.
The first line of output is the same bus path as our device leading up to
'SUNW,qlc@0'.  Our ctd of c3t1938E70CD1642679d26 includes the remote wwpn
as the target.  Using 'luxadm' can further verify this via a dump_map
of the device path from the 'luxadm' output above:

        winston [0] /usr/sbin/luxadm -e dump_map /devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0:devctl
        Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
        0    90000   0         18237c8728411e23 19237c8728411e23 0x0  (Disk device)
        1    90500   0         1028ec82957363d8 1032ec82957363de 0x0  (Disk device)
        2    90600   0         1938e70cd1642679 1939e70cd1642678 0x0  (Disk device)
        3    90700   0         23781ea982b98237 23771ea982b98236 0x0  (Disk device)
        4    90114   0         23781ea982b98237 23771ea982b98236 0x1f (Unknown Type,Host Bus Adapter)

In this output, 'luxadm' has our wwpn listed at position 2.  So now
you might be thinking that's great, but what in the world is actually
presenting my disk.  Well, 'luxadm' can get that for you via inq:

        winston [1] /usr/sbin/luxadm inq /dev/rdsk/c3t1938E70CD1642679d26s2

          Physical Path:
        Vendor:                     STK
        Product:                    BladeCtlr B210
        Revision:                   0612
        Serial Number               2C8DE82917
        Device type:                0x0 (Disk device)
        Removable media:            no
        Medium Changer Element:     no
        ISO version:                0
        ECMA version:               0
        ANSI version:               3 (Device complies to SCSI-3)
        Terminate task:             yes
        Response data format:       2
        Additional length:          0x33
        Cmd received on port:       a
        SIP Bits:                   0, 1, 0
        Command queueing:           no
                      VENDOR-SPECIFIC PARAMETERS
        Byte#                  Hex Value                             ASCII
        36    c3 38 12 82 1b 1b f2 47 b4 3e 13 e0 00 00 00 00      ...  ..........
              00 00 00 00                                         ....

We now know that the disk is presented from an STK B210.  If we'd
like to know a little more about the HBA we are using, we turn to
'fcinfo'.  Since we know that we are on controller 3 (by way of ctd
c3t1938E70CD1642679d26), we need to look for /dev/cfg/c3 in the 'fcinfo'

        winston [0] /usr/sbin/fcinfo hba-port
        HBA Port WWN: 18374cd872e87352
                OS Device Name: /dev/cfg/c3
                Manufacturer: QLogic Corp.
                Model: 375-3356-02
                Firmware Version: 4.04.01
                FCode/BIOS Version:  BIOS: 1.24; fcode: 1.24; EFI: 1.8;   
                Type: N-port
                State: online
                Supported Speeds: 1Gb 2Gb 4Gb
                Current Speed: 4Gb
                Node WWN: 19374cd872e87352

This has now given us the HBA wwpn, the HBA manufacturer and model,
the HBA firmware, etc.  This information can be quite useful if we
are experiencing problems due to the HBA, which can be resolved in a
firmware upgrade, HBA replacement, modification of current speed, etc.
Going back to 'luxadm' for a moment, we can determine the model of our
HBA and the release date on the FCode:

        winston [0] /usr/sbin/luxadm fcode_download -p

          Found Path to 0 FC100/S Cards
        Error: Could not get /devices path to FC100/P,ISP2200, ISP23xx Cards.
          Opening Device: /devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0/fp@0,0:devctl
          Detected FCode Version:       QLE2462 Host Adapter Driver(SPARC): 1.24  11/15/06

          Opening Device: /devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0:devctl
          Detected FCode Version:       QLE2462 Host Adapter Driver(SPARC): 1.24  11/15/06

Correlating the 'fcinfo' output to the 'luxadm' output can be done with
the following 'fcinfo' cmd against the HBA Port WWN:

        winston [0] /usr/sbin/fcinfo remote-port -p 18374cd872e87352
                Node WWN: 19237c8728411e23
        Remote Port WWN: 1938e70cd1642679
                Active FC4 Types: SCSI
                SCSI Target: yes
                Node WWN: 1939e70cd1642678
        Remote Port WWN: 1028ec82957363d8

The above output gives up the original, remote wwpn that we saw in our
ctd (c3t1938E70CD1642679d26).  Adding '-s' and the remote wwpn, we get:

        winston [0] /usr/sbin/fcinfo remote-port -p 18374cd872e87352 -s 1938e70cd1642679
        Remote Port WWN: 1938e70cd1642679
                Active FC4 Types: SCSI
                SCSI Target: yes
                Node WWN: 1939e70cd1642678
                  OS Device Name: /dev/rdsk/c3t1938E70CD1642679d25s2
                LUN: 26
                  Vendor: STK
                  Product: BladeCtlr B210
                  OS Device Name: /dev/rdsk/c3t1938E70CD1642679d26s2
                LUN: 27
                  Vendor: STK

showing us the original raw device, /dev/rdsk/c3t1938E70CD1642679d26s2.
Going back to 'luxadm', we can determine if this disk is possibly
multipathed as well as the status of those paths:

        winston [0] /usr/sbin/luxadm display /dev/rdsk/c3t1938E70CD1642679d26s2
        DEVICE PROPERTIES for disk: /dev/rdsk/c3t1938E70CD1642679d26s2
          Vendor:               STK
          Product ID:           BladeCtlr B210
          Revision:             0612
          Serial Num:           2C8DE82917
          Unformatted capacity: 835995.625 MBytes
          Write Cache:          Enabled
          Read Cache:           Enabled
            Minimum prefetch:   0x1
            Maximum prefetch:   0x1
          Device Type:          Disk device

            LUN path port WWN:          1938e70cd1642679
            Host controller port WWN:   18374cd872e87352
            Path status:                O.K.
            LUN path port WWN:          1939e70cd1642678
            Host controller port WWN:   187482c708723de6
            Path status:                O.K.

While the disk identified as c3t1938E70CD1642679d26 is on the secondary
port of the this Qlogic HBA, we can see that there is another path
to the disk at c6t1939E70CD1642678d26 on the primary port of the HBA.
This can be helpful information to know.  This 'luxadm' cmd also gives
up both the local and remote wwpn, useful for query with commands
like fcinfo.

No comments: