18 June 2011

Configuring iSCSI Initiators in Linux

In part 1, we worked through how to set up iSCSI targets in Linux
(the server side).  Here, in part 2, are my notes on how to set up the
initiators in Linux (the client side).  Our host details for this are
the same as part 1:
        HOSTS:          humboldt (initiator node), cobblepot (target node)
        PROMPT:         host [0]
        OS:             CentOS 5.6 Linux (RedHat Clone)
        IP ADDRs:       humboldt (10.0.136.122), cobblepot (10.0.136.17)
Before proceeding, you will need to have the "iscsi-initiator-utils"
package installed and 'iscsi' and 'iscsid' enabled:
        humboldt [0] /bin/rpm -q iscsi-initiator-utils
        iscsi-initiator-utils-6.2.0.872-6.el5
        humboldt [0] /sbin/chkconfig --list | /bin/grep iscsi
        iscsi           0:off   1:off   2:off   3:on    4:on    5:on    6:off
        iscsid          0:off   1:off   2:off   3:on    4:on    5:on    6:off
If 'iscsi' and 'iscsid' are not enabled, then enable them with 'chkconfig':
        humboldt [0] /sbin/chkconfig --level 3 iscsi on
        humboldt [0] /sbin/chkconfig --level 3 iscsid on
        humboldt [0] /sbin/service iscsid status
        iscsid (pid  1312) is running...
The 'iscsi' service isn't checked above because it isn't really a service
but instead logs into known iSCSI targets when started.  We can, and
will, do this manually later.  If 'iscsid' isn't running, start it via
the 'service' command:
        humboldt [0] /sbin/service iscsid start
        Starting iSCSI daemon:                                     [  OK  ]
Alright, by default, there should be an "initiatorname.iscsi" file
which contains the initiator node name (INN) for our host as seen below.
Since this value must be unique, if you need to generate a new unique INN,
or this file doesn't exist, 'iscsi-iname' can be used.  This command
will output to STDOUT so if you need a new INN, you'll need to update
"initiatorname.iscsi" with the output returned:
        humboldt [0] /bin/cat /etc/iscsi/initiatorname.iscsi
        InitiatorName=iqn.1994-05.com.redhat:1d94b02baba8
        humboldt [0] /sbin/iscsi-iname
        iqn.1994-05.com.redhat:e9281ae17241
(Note, I did not update the INN on humboldt, as can be seen later.) With
that out of the way and 'iscsid' running, we will use 'iscsiadm' to
add the target node (cobblepot, 10.0.136.17) as a valid target host and
discover what targets it has available to us.  The first command below
discovers the targets on cobblepot via method "sendTargets" (-t st) and
updates the local target database (discoverydb).  The second 'iscsiadm'
command lists known target hosts and the third lists the method that
targets have been discovered by:
        humboldt [0] /sbin/iscsiadm -m discoverydb -t st -p 10.0.136.17 -D
        10.0.136.17:3260,1 iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
        10.0.136.17:3260,1 iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
        humboldt [0] /sbin/iscsiadm -m discovery -P 0
        10.0.136.17:3260 via sendtargets
        humboldt [0] /sbin/iscsiadm -m discovery -P 1
        SENDTARGETS:
        DiscoveryAddress: 10.0.136.17,3260
        Target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
                Portal: 10.0.136.17:3260,1
                        Iface Name: default
        Target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
                Portal: 10.0.136.17:3260,1
                        Iface Name: default
        iSNS:
        No targets found.
        STATIC:
        No targets found.
        FIRMWARE:
        No targets found.
        humboldt [0] /sbin/iscsiadm -m discoverydb -t st -p 10.0.136.17 -o show
        # BEGIN RECORD 2.0-872
        discovery.startup = manual
        discovery.type = sendtargets
        discovery.sendtargets.address = 10.0.136.17
        discovery.sendtargets.port = 3260
        discovery.sendtargets.auth.authmethod = None
        discovery.sendtargets.auth.username = <empty>
        discovery.sendtargets.auth.password = <empty>
        discovery.sendtargets.auth.username_in = <empty>
        discovery.sendtargets.auth.password_in = <empty>
        discovery.sendtargets.timeo.login_timeout = 15
        discovery.sendtargets.use_discoveryd = No
        discovery.sendtargets.discoveryd_poll_inval = 30
        discovery.sendtargets.reopen_max = 5
        discovery.sendtargets.timeo.auth_timeout = 45
        discovery.sendtargets.timeo.active_timeout = 30
        discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
        # END RECORD
The final 'iscsiadm' command above lists configuration details governing
communication with the specified target host.  With the available targets
discovered, we need to log in to them.  We could run the 'iscsi' init
script but it's useful to know how to do this manually.  Below we log
into both targets, first only to target "humboldt0", then to any targets
available:
        humboldt [0] /sbin/iscsiadm -m node -T iqn.2011-06.com.example.cobblepot:storage-net.humboldt0 -p 10.0.136.17 -l
        Logging in to [iface: default, target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt0, portal: 10.0.136.17,3260]
        Login to [iface: default, target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt0, portal: 10.0.136.17,3260] successful.
        humboldt [0] /sbin/iscsiadm -m node -l
        Logging in to [iface: default, target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt0, portal: 10.0.136.17,3260]
        Logging in to [iface: default, target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt1, portal: 10.0.136.17,3260]
        iscsiadm: Could not login to [iface: default, target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt0, portal: 10.0.136.17,3260].
        iscsiadm: initiator reported error (15 - already exists)
        Login to [iface: default, target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt1, portal: 10.0.136.17,3260] successful.
The messages above tell us that we are already logged into target
"humboldt0" via error messages (these are fine) and that we are now
also logged into "humboldt1".  Using 'iscsiadm' below, the first command
presents us with simple information detailing the target instances and
target names.  In the second 'iscsiadm' command, we get back more useful
information including the SCSI target, channel, ID, and LUN information,
as well as the local disk attachments:
        humboldt [0] /sbin/iscsiadm -m session
        tcp: [1] 10.0.136.17:3260,1 iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
        tcp: [2] 10.0.136.17:3260,1 iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
        humboldt [0] /sbin/iscsiadm -m session -P 3
        iSCSI Transport Class version 2.0-871
        version 2.0-872
        Target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
                Current Portal: 10.0.136.17:3260,1
                Persistent Portal: 10.0.136.17:3260,1
                        **********
                        Interface:
                        **********
                        Iface Name: default
                        Iface Transport: tcp
                        Iface Initiatorname: iqn.1994-05.com.redhat:1d94b02baba8
                        Iface IPaddress: 10.0.136.122
                        Iface HWaddress: <empty>
                        Iface Netdev: <empty>
                        SID: 1
                        iSCSI Connection State: LOGGED IN
                        iSCSI Session State: LOGGED_IN
                        Internal iscsid Session State: NO CHANGE
                        ************************
                        Negotiated iSCSI params:
                        ************************
                        HeaderDigest: None
                        DataDigest: None
                        MaxRecvDataSegmentLength: 262144
                        MaxXmitDataSegmentLength: 8192
                        FirstBurstLength: 65536
                        MaxBurstLength: 262144 
                        ImmediateData: Yes
                        InitialR2T: Yes
                        MaxOutstandingR2T: 1
                        ************************
                        Attached SCSI devices:
                        ************************
                        Host Number: 1  State: running
                        scsi1 Channel 00 Id 0 Lun: 0
                                Attached scsi disk sdb          State: running
                        scsi1 Channel 00 Id 0 Lun: 1
                                Attached scsi disk sdd          State: running
                        scsi1 Channel 00 Id 0 Lun: 2 
                                Attached scsi disk sde          State: running
        Target: iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
                Current Portal: 10.0.136.17:3260,1
                Persistent Portal: 10.0.136.17:3260,1
                        **********
                        Interface:
                        **********
                        Iface Name: default 
                        Iface Transport: tcp
                        Iface Initiatorname: iqn.1994-05.com.redhat:1d94b02baba8
                        Iface IPaddress: 10.0.136.122 
                        Iface HWaddress: <empty>
                        Iface Netdev: <empty>
                        SID: 2
                        iSCSI Connection State: LOGGED IN
                        iSCSI Session State: LOGGED_IN
                        Internal iscsid Session State: NO CHANGE 
                        ************************
                        Negotiated iSCSI params: 
                        ************************
                        HeaderDigest: None
                        DataDigest: None
                        MaxRecvDataSegmentLength: 262144
                        MaxXmitDataSegmentLength: 8192
                        FirstBurstLength: 65536
                        MaxBurstLength: 262144 
                        ImmediateData: Yes
                        InitialR2T: Yes
                        MaxOutstandingR2T: 1
                        ************************
                        Attached SCSI devices:
                        ************************
                        Host Number: 2  State: running
                        scsi2 Channel 00 Id 0 Lun: 0
                                Attached scsi disk sdc          State: running
We can see the same LUN / local disk correlation reviewing
"/dev/disk/by-path".  Reviewing "/dev/disk/by-id" provides us with the
unique SCSI ID for each related disk device.  Finally, the relationship
between local disk device and SCSI target, channel, ID, and LUN can be
seen in "/proc/scsi/scsi":
        humboldt [0] /bin/ls -l /dev/disk/by-path | /bin/awk '/iqn/ {print $9, $11}'
        ip-10.0.136.17:3260-iscsi-iqn.2011-06.com.example.cobblepot:storage-net.humboldt0-lun-0 ../../sdb
        ip-10.0.136.17:3260-iscsi-iqn.2011-06.com.example.cobblepot:storage-net.humboldt0-lun-1 ../../sdd
        ip-10.0.136.17:3260-iscsi-iqn.2011-06.com.example.cobblepot:storage-net.humboldt0-lun-2 ../../sde
        ip-10.0.136.17:3260-iscsi-iqn.2011-06.com.example.cobblepot:storage-net.humboldt1-lun-0 ../../sdc
        humboldt [0] /bin/ls -l /dev/disk/by-id/ | /bin/awk '!/(tot|CD-R|SATA)/ {print $9, $11}'
        scsi-1494554000000000031333038303334323831000000000000 ../../sdb
        scsi-1494554000000000031333038303334383132000000000000 ../../sdd
        scsi-1494554000000000031333038303334383135000000000000 ../../sde
        scsi-1494554000000000031333038303335333431000000000000 ../../sdc
        humboldt [0] /bin/cat /proc/scsi/scsi
        Attached devices:
        <snip...>
        Host: scsi1 Channel: 00 Id: 00 Lun: 00
          Vendor: IET      Model: VIRTUAL-DISK     Rev: 0
          Type:   Direct-Access                    ANSI SCSI revision: 04
        Host: scsi1 Channel: 00 Id: 00 Lun: 01
          Vendor: IET      Model: VIRTUAL-DISK     Rev: 0
          Type:   Direct-Access                    ANSI SCSI revision: 04
        Host: scsi2 Channel: 00 Id: 00 Lun: 00
          Vendor: IET      Model: VIRTUAL-DISK     Rev: 0
          Type:   Direct-Access                    ANSI SCSI revision: 04
        Host: scsi1 Channel: 00 Id: 00 Lun: 02
          Vendor: IET      Model: VIRTUAL-DISK     Rev: 0
          Type:   Direct-Access                    ANSI SCSI revision: 04
Great, lots of info, let's do something now.  Running 'fdisk' provides
us with our disks showing that "sdb - sde" are not yet configured:
        humboldt [0] /sbin/fdisk -l | /bin/grep ^Disk
        Disk /dev/sdb doesn't contain a valid partition table
        Disk /dev/sdc doesn't contain a valid partition table
        Disk /dev/sdd doesn't contain a valid partition table
        Disk /dev/sde doesn't contain a valid partition table
        Disk /dev/sda: 8589 MB, 8589934592 bytes
        Disk /dev/sdb: 100 MB, 100646912 bytes
        Disk /dev/sdc: 104 MB, 104857600 bytes
        Disk /dev/sdd: 100 MB, 100663296 bytes
        Disk /dev/sde: 104 MB, 104857600 bytes
We'll use "sde", which is cobblepot:/storage/targets/humboldt0/lun.2,
for a new volume.  Below, I've created a new partition at "sde1",
written the partition table, and created a new ext3 filesystem (FS) on it:
        humboldt [0] /sbin/fdisk /dev/sde
        Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
        Building a new DOS disklabel. Changes will remain in memory only,
        until you decide to write them. After that, of course, the previous
        content won't be recoverable.

        Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

        Command (m for help): p

        Disk /dev/sde: 104 MB, 104857600 bytes
        4 heads, 50 sectors/track, 1024 cylinders
        Units = cylinders of 200 * 512 = 102400 bytes

           Device Boot      Start         End      Blocks   Id  System
        
        Command (m for help): n 
        Command action
           e   extended
           p   primary partition (1-4)
        p
        Partition number (1-4): 1
        First cylinder (1-1024, default 1): 
        Using default value 1
        Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1024): 
        Using default value 1024

        Command (m for help): p

        Disk /dev/sde: 104 MB, 104857600 bytes
        4 heads, 50 sectors/track, 1024 cylinders
        Units = cylinders of 200 * 512 = 102400 bytes

           Device Boot      Start         End      Blocks   Id  System
        /dev/sde1               1        1024      102375   83  Linux

        Command (m for help): w
        The partition table has been altered!

        Calling ioctl() to re-read partition table.
        Syncing disks.
        humboldt [0] /sbin/fdisk -l /dev/sde

        Disk /dev/sde: 104 MB, 104857600 bytes
        4 heads, 50 sectors/track, 1024 cylinders  
        Units = cylinders of 200 * 512 = 102400 bytes

           Device Boot      Start         End      Blocks   Id  System
        /dev/sde1               1        1024      102375   83  Linux
        humboldt [0] /sbin/mkfs -V -t ext3 /dev/sde1
        mkfs (util-linux 2.13-pre7) 
        mkfs.ext3 /dev/sde1 
        mke2fs 1.39 (29-May-2006)
        Filesystem label=
        OS type: Linux
        Block size=1024 (log=0) 
        Fragment size=1024 (log=0)
        25688 inodes, 102372 blocks
        5118 blocks (5.00%) reserved for the super user
        First data block=1 
        Maximum filesystem blocks=67371008
        13 block groups
        8192 blocks per group, 8192 fragments per group
        1976 inodes per group
        Superblock backups stored on blocks:
                8193, 24577, 40961, 57345, 73729

        Writing inode tables: done                            
        Creating journal (4096 blocks): done
        Writing superblocks and filesystem accounting information: done

        This filesystem will be automatically checked every 33 mounts or
        180 days, whichever comes first.  Use tune2fs -c or -i to override.
Since iSCSI targets and LUNs may not be presented in the same order
the next time we initiate them, our local device naming may change.
To mitigate this, I've set a label of "cpot_t0l2s1" to "sde1" via
'tune2fs' followed by mounting the new volume:
        humboldt [0] /sbin/tune2fs -L cpot_t0l2s1 /dev/sde1
        tune2fs 1.39 (29-May-2006)
        you have mail in /var/spool/mail/root
        humboldt [0] /sbin/tune2fs -l /dev/sde1 | /bin/egrep 'name|UUID'
        Filesystem volume name:   cpot_t0l2s1
        Filesystem UUID:          bf1528d6-3b57-4688-b19c-283a5bbe3f78
        humboldt [0] /bin/mount -L cpot_t0l2s1 /a
        humboldt [0] /bin/df -h /a
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sde1              97M  5.6M   87M   7% /a
        humboldt [0] /bin/vi /etc/fstab
        humboldt [0] /bin/grep cpot /etc/fstab
        LABEL=cpot_t0l2s1       /a        ext3    _netdev         0 0
To have the volume available the next time we boot the system, I've
set the volume to mount via it's label in "/etc/fstab" above.  Of note,
the FS option is set to "_netdev", effectively preventing the system from
mounting this volume until network access is available.  Our new volume is
now ready for usage and our other LUNs ready to be configured for usage.
Below, from either humboldt (initiator node) or cobblepot (target node),
we can see two established connections (sessions) to port 3260, one for
each target:
        humboldt [0] /bin/netstat -na | /bin/grep 3260
        tcp        0      0 10.0.136.122:40103          10.0.136.17:3260            ESTABLISHED
        tcp        0      0 10.0.136.122:40102          10.0.136.17:3260            ESTABLISHED
        cobblepot [0] /bin/netstat -na | /bin/grep 3260
        tcp        0      0 0.0.0.0:3260                0.0.0.0:*                   LISTEN
        tcp        0      0 10.0.136.17:3260            10.0.136.122:40103          ESTABLISHED
        tcp        0      0 10.0.136.17:3260            10.0.136.122:40102          ESTABLISHED
Additionally, on our target host, a check of "/proc/net/iet/session"
now shows sessions for our INN for both targets:
        cobblepot [0] cat /proc/net/iet/session
        tid:2 name:iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
                sid:281475014263296 initiator:iqn.1994-05.com.redhat:1d94b02baba8
                        cid:0 ip:10.0.136.122 state:active hd:none dd:none
        tid:1 name:iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
                sid:562949974196736 initiator:iqn.1994-05.com.redhat:1d94b02baba8
                        cid:0 ip:10.0.136.122 state:active hd:none dd:none

see also:

    Part 1: Configuring iSCSI Targets in Linux
    Configuring iSCSI Initiators in Solaris
    Configuring iSCSI Initiators in Windows 2008

4 comments:

"Dollah" Bill said...

Worked like a champ! Mounted right up. Good write up, man. Another target iscsi target package to try is the, "scsi-target-utils", yum package.

- Dollah

troy said...

Dollah,

Absolutely excellent to hear. After our talk on the phone, I am kind of curious how 'scsi-target-utils' stacks up against IET. It looks like I'll be testing some more Linux targets soon.

--troy

Ravi said...

Hi,

I am new to ISCSI...I am trying to get run iscsi by using service iscsi start, i am getting error message "iscsi is stopped" and it never start.please help me on this.

Thanks in Advance ,
Ravi

troy said...

Ravi,

Sorry for the delay on response. 'service iscsi start' would simply execute a script under /etc/init.d, which you could do as well. My advice would be to run the startup script manually (not with 'service' command) and see what output is returned. Additionally, you could execute the init script as 'bash -x /etc/init.d/script start' which should give you additional debugging information.

--troy