13 June 2011

Configuring iSCSI Initiators in Solaris

In part 1, we worked through how to set up iSCSI targets in Solaris
(the server side).  Now, for part 2, I present my notes on how to set
up the initiators in Solaris on the client side.  Our host details for
this are the same as part 1:
        HOSTS:          sunspot (target node), sunbeam (initiator node)
        PROMPT:         host [0]
        OS:             Solaris 10 10/09 (u8)
Before proceeding, you will need to have the following packages installed
on the initiator node:
        SUNWiscsir
        SUNWiscsiu
We will also need the iSCSI initiator ('iscsid') running so enable it with
'svcadm' (if it's not already running) and verify which 'svcs':
        sunbeam [0] /usr/sbin/svcadm enable svc:/network/iscsi/initiator:default
        sunbeam [0] /usr/bin/svcs -p svc:/network/iscsi/initiator:default
        STATE          STIME    FMRI
        online         00:20:52 svc:/network/iscsi/initiator:default
                       00:20:50      186 iscsid
Once 'iscsid' is running, we can use 'iscsiadm' to add the target node
(sunspot, 10.0.136.12) as a valid target host and discover what targets
it has available to us:
        sunbeam [0] /usr/sbin/iscsiadm add discovery-address 10.0.136.12:3260
        sunbeam [0] /usr/sbin/iscsiadm list discovery-address
        Discovery Address: 10.0.136.12:3260
        sunbeam [0] /usr/sbin/iscsiadm list discovery-address -v 10.0.136.12:3260
        Discovery Address: 10.0.136.12:3260
                Target name: iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1
                        Target address:     10.0.136.12:3260, 45
                Target name: iqn.1986-03.com.sun:02:7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0
                        Target address:     10.0.136.12:3260, 45
Using 'discovery-address' to 'iscsiadm add' above is one of three
methods for discovering available targets from the initiator, and
for my usage the simplest.  Of note, the default port for iSCSI is
3260, as seen specified above.  Since we now know what targets are
available, we can statically define them on the initiator node (sunbeam)
via 'iscsiadm add static-config'.  (The value to static-config is
"TARGET_NAME,TARGET_ADDRESS,TPGT".  The TPGT (TargetPortGroupTag) is
only included since we configured it in part 1 relative to the created
targets.)  We follow that with 'iscsiadm list target' to verify the
targets have been added (additional information can be obtained by adding
'-v' to the command):
        sunbeam [0] /usr/sbin/iscsiadm add static-config iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1,10.0.136.12,45
        sunbeam [0] /usr/sbin/iscsiadm add static-config iqn.1986-03.com.sun:02:7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0,10.0.136.12,45
        sunbeam [0] /usr/sbin/iscsiadm list target
        Target: iqn.1986-03.com.sun:02:7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0
                Alias: sunbeam0
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
        Target: iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1
                Alias: sunbeam1
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
        sunbeam [0] /usr/sbin/iscsiadm list target -v
        Target: iqn.1986-03.com.sun:02:7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0
                Alias: sunbeam0
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
                        CID: 0
                          IP address (Local): 10.0.136.117:32783
                          IP address (Peer): 10.0.136.12:3260
                          Discovery Method: Static
                          Login Parameters (Negotiated):
                                Data Sequence In Order: yes
                                Data PDU In Order: yes
                                Default Time To Retain: 20
                                Default Time To Wait: 2
                                Error Recovery Level: 0
                                First Burst Length: 65536
                                Immediate Data: yes
                                Initial Ready To Transfer (R2T): yes
                                Max Burst Length: 262144
                                Max Outstanding R2T: 1
                                Max Receive Data Segment Length: 8192
                                Max Connections: 1
                                Header Digest: NONE
                                Data Digest: NONE

        Target: iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1
                Alias: sunbeam1
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
                        CID: 0
                          IP address (Local): 10.0.136.117:32781
                          IP address (Peer): 10.0.136.12:3260
                          Discovery Method: Static
                          Login Parameters (Negotiated):
                                Data Sequence In Order: yes
                                Data PDU In Order: yes
                                Default Time To Retain: 20
                                Default Time To Wait: 2
                                Error Recovery Level: 0
                                First Burst Length: 65536
                                Immediate Data: yes
                                Initial Ready To Transfer (R2T): yes
                                Max Burst Length: 262144
                                Max Outstanding R2T: 1
                                Max Receive Data Segment Length: 8192
                                Max Connections: 1
                                Header Digest: NONE
                                Data Digest: NONE
If we run 'iscsiadm list target -S', we retrieve the LUNs and their
respective "ctd" instance for each target presented:
        sunbeam [0] /usr/sbin/iscsiadm list target -S
        Target: iqn.1986-03.com.sun:02:7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0
                Alias: sunbeam0
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
                LUN: 0
                     Vendor:  SUN
                     Product: SOLARIS
                     OS Device Name: /dev/rdsk/c2t14d0s2
                LUN: 1
                     Vendor:  SUN
                     Product: SOLARIS         
                     OS Device Name: /dev/rdsk/c2t13d0s2
                LUN: 2
                     Vendor:  SUN 
                     Product: SOLARIS
                     OS Device Name: /dev/rdsk/c2t12d0s2

        Target: iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1
                Alias: sunbeam1
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
                LUN: 0
                     Vendor:  SUN     
                     Product: SOLARIS          
                     OS Device Name: /dev/rdsk/c2t11d0s2

        sunbeam [0] /usr/sbin/iscsiadm list target -S iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1
        Target: iqn.1986-03.com.sun:02:6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1
                Alias: sunbeam1
                TPGT: 45
                ISID: 4000002a0000
                Connections: 1
                LUN: 0
                     Vendor:  SUN
                     Product: SOLARIS         
                     OS Device Name: /dev/rdsk/c2t11d0s2
With the presented targets configured, we reset the initiator discovery
modes below with 'iscsiadm modify discovery'.  Since we are only using
static configuration, the following disables iSNS (-i) and SendTargets
(-t (discovery-address)) modes, leaving static configuration enabled:
        sunbeam [0] /usr/sbin/iscsiadm modify discovery -i disable -t disable
        sunbeam [0] /usr/sbin/iscsiadm list discovery   
        Discovery:
                Static: enabled
                Send Targets: disabled
                iSNS: disabled
Just to be sure that Solaris cleaned up and updated the device tree,
I ran 'devfsadm' for the "iSCSI" device driver.  To validate the iSCSI
target LUNs are now available, I've run 'echo' piped to 'format':
        sunbeam [0] /usr/sbin/devfsadm -Cv -i iscsi
        sunbeam [0] echo | /usr/sbin/format 
        Searching for disks...done

        AVAILABLE DISK SELECTIONS:
               0. c1t0d0 <DEFAULT cyl 4092 alt 2 hd 128 sec 32>
                  /pci@0,0/pci1000,8000@16/sd@0,0
               1. c2t11d0 <DEFAULT cyl 97 alt 2 hd 64 sec 32>
                  /iscsi/disk@0000iqn.1986-03.com.sun%3A02%3A6abaa85a-bd22-4c4a-daa5-e568f0971673.sunbeam1002D,0
               2. c2t12d0 <DEFAULT cyl 97 alt 2 hd 64 sec 32>
                  /iscsi/disk@0000iqn.1986-03.com.sun%3A02%3A7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0002D,2
               3. c2t13d0 <DEFAULT cyl 97 alt 2 hd 64 sec 32>
                  /iscsi/disk@0000iqn.1986-03.com.sun%3A02%3A7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0002D,1
               4. c2t14d0 <DEFAULT cyl 97 alt 2 hd 64 sec 32>
                  /iscsi/disk@0000iqn.1986-03.com.sun%3A02%3A7dcf7ffc-5126-6d65-8a25-e0d52fbf97f8.sunbeam0002D,0
        Specify disk (enter its number): Specify disk (enter its number):
        sunbeam [1]
For the sake of being complete, using 'for' loops below I've configured
the newly presented LUNs with a Solaris partition spanning the whole disk
('fdisk -B'), and labeled each disk using 'echo' statements piped to
'format':
        sunbeam [1] for i in 11 12 13 14 ; do \
        > /usr/sbin/fdisk -B /dev/rdsk/c2t${i}d0p0 ; done
        sunbeam [0] for i in 11 12 13 14 ; do echo "label\nyes\nverify\nquit" | \
        > /usr/sbin/format c2t${i}d0 ; done
        selecting c2t11d0
        [disk formatted]

        FORMAT MENU:
                disk       - select a disk
                type       - select (define) a disk type
                partition  - select (define) a partition table
                current    - describe the current disk
                format     - format and analyze the disk
                fdisk      - run the fdisk program
                repair     - repair a defective sector
                label      - write label to the disk
                analyze    - surface analysis
                defect     - defect list management
                backup     - search for backup labels
                verify     - read and display labels
                save       - save new disk/partition definitions
                inquiry    - show vendor, product and revision
                volname    - set 8-character volume name
                !<cmd>     - execute <cmd>, then return
                quit
        format> Ready to label disk, continue?
        format>
        Primary label contents:

        Volume name = <        >
        ascii name  = <DEFAULT cyl 96 alt 2 hd 64 sec 32>
        pcyl        =   98
        ncyl        =   96
        acyl        =    2
        bcyl        =    0
        nhead       =   64
        nsect       =   32
        Part      Tag    Flag     Cylinders      Size            Blocks
          0 unassigned    wm       0             0         (0/0/0)       0
          1 unassigned    wm       0             0         (0/0/0)       0
          2     backup    wu       0 - 95       96.00MB    (96/0/0) 196608
          3 unassigned    wm       0             0         (0/0/0)       0
          4 unassigned    wm       0             0         (0/0/0)       0
          5 unassigned    wm       0             0         (0/0/0)       0
          6 unassigned    wm       0             0         (0/0/0)       0
          7 unassigned    wm       0             0         (0/0/0)       0
          8       boot    wu       0 -  0        1.00MB    (1/0/0)    2048
          9 unassigned    wm       0             0         (0/0/0)       0

        format> selecting c2t12d0
        [disk formatted]
        <snip...>
Our new disks are now ready for usage.  We simply need to create the
appropriate Solaris slices and filesystems on each disk.  Of note,
back on our target node (sunspot), running 'iscsitadm show stats'
at a 5 second interval allows us to see the quantity of read / write
operations performed against each target as well as the number of bytes
read / written.  The stats below detail from right before to just after
the 'fdisk' and 'format' commands run on the initiator node (sunbeam):
        sunspot [0] /usr/sbin/iscsitadm show stats -I 5
                                operations    bandwidth
        device                 read  write   read  write
        --------------------  -----  -----  -----  -----
        sunbeam0                 5K     39    49M    19K
        sunbeam1                 3K     19    32M     9K
        sunbeam0                  0      0     0K     0K
        sunbeam1                  0      0     0K     0K
        sunbeam0                  0      0     0K     0K
        sunbeam1                 13      0     7K     0K
        sunbeam0                  0      0     0K     0K
        sunbeam1                331     30   2.9M   5.8M
        sunbeam0                  0      0     0K     0K
        sunbeam1                  0      0     0K     0K
        ^Csunspot [130]
Also, reviewing the output of 'netstat -na' from either the initiator
node (sunbeam) or the target node (sunspot), we can see two established
connections (sessions) to port 3260, one for each target:
        sunbeam [0] /usr/bin/netstat -na | /bin/grep 3260
        10.0.136.117.32781   10.0.136.12.3260     64240      0 64240      0 ESTABLISHED
        10.0.136.117.32783   10.0.136.12.3260     64240      0 64240      0 ESTABLISHED
        sunbeam [0]

        ## ipv6 is last *.3260 listing
        sunspot [0] /usr/bin/netstat -na | /bin/grep 3260
              *.3260               *.*                0      0 49152      0 LISTEN
        10.0.136.12.3260     10.0.136.117.32781   64240      0 64240      0 ESTABLISHED
        10.0.136.12.3260     10.0.136.117.32783   64240      0 64240      0 ESTABLISHED
              *.3260                            *.*                             0      0 49152      0 LISTEN
The number of seesions per target can be increased on the initiator
node to enable multipathing by way of 'iscsiadm modify initiator-node
--configured-sessions'.  See manpage iscsiadm(1M) for details.  Finally
keep in mind that placing a volume presented by an iSCSI target into
vfstab for mount or 'fsck' at boot time will likely fail.  This is because
the iSCSI target(s) will not be available to the initiator (client node)
at that time since 'iscsid' wouldn't yet be started.

see also:

    Part 1: Configuring iSCSI Targets in Solaris
    Configuring Oracle Solaris iSCSI Targets and Initiators (Oracle.com)
    Configuring iSCSI Initiators in Linux
    Configuring iSCSI Initiators in Windows 2008

No comments: