15 June 2011

Configuring iSCSI Targets in Linux

Since I've already been looking into iSCSI on Solaris, I thought I'd look
to see what Linux has to offer.  While in comparison to Solaris I think
the Linux side could still mature a little, it still has potential and
a functional offering.  The following is part 1 of my notes detailing
the configuration of a test Linux iSCSI target node (server side).
Configuration of an iSCSI initiator node (client side) will be detailed
in part 2.  Our host details for the target notes are:
        HOSTS:          cobblepot (target node), humboldt (initiator node)
        PROMPT:         host [0]
        OS:             CentOS 5.6 Linux (RedHat Clone)
        IP ADDRs:       cobblepot (10.0.136.17), humboldt (10.0.136.122)
To start things off, Linux doesn't include a standard package for
providing iSCSI targets.  There are, however, several open source
offerings available.  For my usage, I've chosen iSCSI Enterprise Target
(IET) available via sourceforge (http://iscsitarget.sourceforge.net/).
The latest version at this time is iscsitarget-1.4.20.2.tar.gz, dated
2010-07-15.  After downloading it, we need to build it:
        cobblepot [0] pwd
        /opt
        cobblepot [0] /bin/ls
        iscsitarget-1.4.20.2.tar.gz
        cobblepot [0] /bin/gunzip -c iscsitarget-1.4.20.2.tar.gz | /bin/tar xf -
        cobblepot [0] cd iscsitarget-1.4.20.2
        cobblepot [0] /usr/bin/make
        Applying Patch compat-2.6.32.patch
        patching file kernel/conn.c
        Applying Patch compat-2.6.31.patch
        patching file kernel/file-io.c
        Applying Patch compat-2.6.30.patch
        patching file kernel/block-io.c
        Hunk #1 succeeded at 280 with fuzz 2.
        Applying Patch compat-2.6.29.patch
        patching file kernel/config.c
        Applying Patch compat-2.6.28.patch
        patching file kernel/conn.c
        Applying Patch compat-2.6.25-2.6.27.patch
        patching file kernel/block-io.c
        <snip...>
          CC [M]  /opt/iscsitarget-1.4.20.2/kernel/ua.o
          CC [M]  /opt/iscsitarget-1.4.20.2/kernel/seq_list.o
          LD [M]  /opt/iscsitarget-1.4.20.2/kernel/iscsi_trgt.o
          Building modules, stage 2.
          MODPOST
          CC      /opt/iscsitarget-1.4.20.2/kernel/iscsi_trgt.mod.o
          LD [M]  /opt/iscsitarget-1.4.20.2/kernel/iscsi_trgt.ko
        make[1]: Leaving directory `/usr/src/kernels/2.6.18-238.el5-x86_64'
        cobblepot [0] /usr/bin/make install
        `usr/ietd' -> `/usr/sbin/ietd'
        `usr/ietadm' -> `/usr/sbin/ietadm'
        `etc/initd/initd.redhat' -> `/etc/rc.d/init.d/iscsi-target'
        install: creating directory `/etc/iet'
        `etc/ietd.conf' -> `/etc/iet/ietd.conf'
        `etc/initiators.allow' -> `/etc/iet/initiators.allow'
        `etc/targets.allow' -> `/etc/iet/targets.allow'
        `doc/manpages/ietadm.8' -> `/usr/share/man/man8/ietadm.8'
        `doc/manpages/ietd.8' -> `/usr/share/man/man8/ietd.8'
        `doc/manpages/ietd.conf.5' -> `/usr/share/man/man5/ietd.conf.5'
        install: creating directory `/usr/share/doc/iscsitarget'
        `ChangeLog' -> `/usr/share/doc/iscsitarget/ChangeLog'
        `COPYING' -> `/usr/share/doc/iscsitarget/COPYING'
        `RELEASE_NOTES' -> `/usr/share/doc/iscsitarget/RELEASE_NOTES'
        `README' -> `/usr/share/doc/iscsitarget/README'
        `README.vmware' -> `/usr/share/doc/iscsitarget/README.vmware'
        `README.initiators' -> `/usr/share/doc/iscsitarget/README.initiators'
        install: creating directory `/lib/modules/2.6.18-238.el5/extra/iscsi'
        `kernel/iscsi_trgt.ko' -> `/lib/modules/2.6.18-238.el5/extra/iscsi/iscsi_trgt.ko'
        Running depmod
With the build out of the way, we need to load kernel module "iscsi_trgt".
Part of the install above included an init script (iscsi-target) placed
to /etc/init.d so we also need to enable it for runlevel 3:
        cobblepot [0] /sbin/lsmod | /bin/grep iscsi_trgt
        cobblepot [1] /sbin/modprobe iscsi_trgt
        cobblepot [0] /sbin/lsmod | /bin/grep iscsi_trgt
        iscsi_trgt            119168  0
        cobblepot [0] /sbin/chkconfig --add iscsi-target
        cobblepot [0] /sbin/chkconfig --level 3 iscsi-target on
        cobblepot [0] /sbin/chkconfig --list | /bin/grep iscsi-target
        iscsi-target    0:off   1:off   2:off   3:on    4:off   5:off   6:off
        cobblepot [0] /bin/ls /etc/init.d/iscsi-target /etc/rc3.d/*iscsi-target
        /etc/init.d/iscsi-target  /etc/rc3.d/S39iscsi-target
        cobblepot [0] /bin/cat /proc/net/iet/session /proc/net/iet/volume
Of note, I have not yet started "iscsi-target" (/usr/sbin/ietd) because
we still need to configure it.  Also, the files listed above under
/proc don't contain anything now, but will contain configured targets
and LUNs, respectively, once configured.  Below, I have set up disk
"sdb" with three partitions.  Partitions 1 and 2 are raw disk space (no
filesystem (FS)) and will be used as the backing stores for two LUNs.
Partition 3 has an FS, is mounted to /storage, and will be used for file
based backing stores:
        cobblepot [0] /sbin/fdisk -l /dev/sdb

        Disk /dev/sdb: 536 MB, 536870912 bytes
        64 heads, 32 sectors/track, 512 cylinders
        Units = cylinders of 2048 * 512 = 1048576 bytes

           Device Boot      Start         End      Blocks   Id  System
        /dev/sdb1               1          96       98288   83  Linux
        /dev/sdb2              97         192       98304   83  Linux
        /dev/sdb3             193         512      327680   83  Linux
        cobblepot [0] /bin/mkdir /storage ; /bin/mount /dev/sdb3 /storage
        cobblepot [0] /bin/df -h /storage
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sdb3             310M   11M  284M   4% /storage
        cobblepot [0] /bin/mkdir -p /storage/targets/humboldt0
        cobblepot [0] /bin/mkdir -p /storage/targets/humboldt1
        cobblepot [0] /bin/dd if=/dev/zero of=/storage/targets/humboldt0/lun.2 bs=1k count=100k
        102400+0 records in
        102400+0 records out
        104857600 bytes (105 MB) copied, 0.380762 seconds, 275 MB/s
        cobblepot [0] /bin/dd if=/dev/zero of=/storage/targets/humboldt1/lun.0 bs=1k count=100k
        102400+0 records in
        102400+0 records out
        104857600 bytes (105 MB) copied, 0.552485 seconds, 190 MB/s
        cobblepot [0] /usr/bin/find /storage/targets -type f -exec /usr/bin/du -sh {} \;
        101M    /storage/targets/humboldt1/lun.0
        101M    /storage/targets/humboldt0/lun.2
The two files created above, "humboldt0/lun.2" and "humboldt1/lun.0" are
our file based backing stores for another two LUNs.  With our backing
stores created, we can begin configuring our targets.  Below are the
contents of the 'ietd' configuration file that I've set up.  The first
"Target" section sets up a new target, assigns a local alias name, and
configures three LUNs.  Two of these LUNs are our raw partitions, the
third is the path to one of our file based backing stores.  The second
"Target" section creates another target and is otherwise similar to the
first section:
        cobblepot [0] /bin/vi /etc/iet/ietd.conf
        cobblepot [0] /bin/cat /etc/iet/ietd.conf
        Target iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
                Alias humboldt0
                Lun 0 Path=/dev/sdb1,Type=fileio,ScsiId=1308034281,ScsiSN=1308034287
                Lun 1 Path=/dev/sdb2,Type=fileio,ScsiId=1308034812,ScsiSN=1308034818
                Lun 2 Path=/storage/targets/humboldt0/lun.2,Type=fileio,ScsiId=1308034815,ScsiSN=1308034821
                MaxConnections 1
                MaxSessions 0
                NOPInterval 1
                NOPTimeout 5
                DataPDUInOrder Yes
                DataSequenceInOrder Yes

        Target iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
                Alias humboldt1
                Lun 0 Path=/storage/targets/humboldt1/lun.0,Type=fileio,ScsiId=1308035341,ScsiSN=1308035347
                MaxConnections 1
                MaxSessions 0
                NOPInterval 1
                NOPTimeout 5 
                DataPDUInOrder Yes
                DataSequenceInOrder Yes
Notes on the above, the entry following "Target" breaks down to:
        iqn.YYYY-MM.TLD.DOMAIN.HOST:UNIQUE_IDENTIFIER
LUN entries are:
        Lun LUN_NUMBER Path=/PATH/TO/BACKING/STORE,Type=[fileio|blockio],ScsiId=UNIQUE_ID,ScsiSN=UNIQUE_ID
Please see manpage ietd.conf(5) for further details.  Moving on,
we can set up restrictions to our targets.  In "targets.allow" below,
I've allowed all targets to be discoverable from network 10.0.136.0/24.
In "initiators.allow", I've associated our targets from earlier with
only host 10.0.136.122 (humboldt, initiator node):
        cobblepot [0] /bin/vi /etc/iet/targets.allow
        cobblepot [0] /bin/grep -v ^# /etc/iet/targets.allow
        ALL 10.0.136.0/24
        cobblepot [0] /bin/vi /etc/iet/initiators.allow     
        cobblepot [0] /bin/grep -v ^# /etc/iet/initiators.allow
        iqn.2011-06.com.example.cobblepot:storage-net.humboldt0 10.0.136.122
        iqn.2011-06.com.example.cobblepot:storage-net.humboldt1 10.0.136.122
(Multiple hosts for a particular target in "initiators.allow" are
specified as a comma separated list after the target.)  With all
configuration complete, we can run the "iscsi-target" init script.
After doing so, we now have relevant details populated under
/proc/net/iet in "session" and "volume":
        cobblepot [0] /sbin/service iscsi-target start
        Starting iSCSI Target:                                     [  OK  ]
        cobblepot [0] /bin/ps -ef | /bin/grep ietd
        root      6187     1  0 03:24 ?        00:00:00 /usr/sbin/ietd
        cobblepot [0] /bin/netstat -natp | /bin/grep 3260 
        tcp     0      0 0.0.0.0:3260        0.0.0.0:*        LISTEN      6391/ietd
        cobblepot [0] /bin/cat /proc/net/iet/session
        tid:2 name:iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
        tid:1 name:iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
        cobblepot [0] /bin/cat /proc/net/iet/volume
        tid:2 name:iqn.2011-06.com.example.cobblepot:storage-net.humboldt1
                lun:0 state:0 iotype:fileio iomode:wt blocks:204800 blocksize:512 path:/storage/targets/humboldt1/lun.0
        tid:1 name:iqn.2011-06.com.example.cobblepot:storage-net.humboldt0
                lun:0 state:0 iotype:fileio iomode:wt blocks:196576 blocksize:512 path:/dev/sdb1
                lun:1 state:0 iotype:fileio iomode:wt blocks:196608 blocksize:512 path:/dev/sdb2
                lun:2 state:0 iotype:fileio iomode:wt blocks:204800 blocksize:512 path:/storage/targets/humboldt0/lun.2
We can also display configured target parameters of our targets via 'ietadm':
        cobblepot [0] /usr/sbin/ietadm --op show --tid=1
        Wthreads=8
        Type=0
        QueuedCommands=32
        NOPInterval=1
        NOPTimeout=5
        cobblepot [0] /usr/sbin/ietadm --op show --tid=2
        Wthreads=8
        Type=0
        QueuedCommands=32
        NOPInterval=1
        NOPTimeout=5
The targets identified by "--tid" are those referenced in both the
"session" and "volume" files above.  Below are the configured session
parameters relevant to target "2":
        cobblepot [0] /usr/sbin/ietadm --op show --tid=2 --sid 0
        InitialR2T=Yes
        ImmediateData=Yes
        MaxConnections=1
        MaxRecvDataSegmentLength=8192
        MaxXmitDataSegmentLength=8192
        MaxBurstLength=262144
        FirstBurstLength=65536
        DefaultTime2Wait=2
        DefaultTime2Retain=20
        MaxOutstandingR2T=1
        DataPDUInOrder=Yes
        DataSequenceInOrder=Yes
        ErrorRecoveryLevel=0
        HeaderDigest=None
        DataDigest=None
        OFMarker=No
        IFMarker=No
        OFMarkInt=Reject
        IFMarkInt=Reject
At this point, our target node "cobblepot" has been configured for
client access from initiator node "humboldt".  The only thing we need
to do now is configure the initiator node, which will be part 2.

see also:

    ietd.conf(5)
    Part 2: Configuring iSCSI Initiators in Linux
    iSCSI Enterprise Target homepage
    Configuring iSCSI Targets in Solaris