23 January 2011

Creating an MD Root Mirror in Linux

For whatever reason, you originally set up your Linux host without a root
disk mirror and now you want it mirrored.  One could say "just re-install,"
but surely there is another way, and there is.  The solution isn't quick
or simple, though it isn't impossible.  Our host details are:
        HOST:           tux
        PROMPTS:        [tux[0] |[root@tux /]# |sh-3.2# ]
        OS:             CentOS 5.4 Linux
        PRIMARY ROOT:   sda1 (hd0 (disk 1))
        SECONDARY ROOT: sdb1 (hd1 (disk 2))
        MIRRORED ROOT:  md1
Due to the amount of details, there are two primary sections, QUICK and
DETAILED, with NOTES at the end.  QUICK simply lists out all commands,
in order of execution, with no output or discussion.  DETAILED provides
all commands, related output, and reasoning behind the steps.  With that
in mind, choose your adventure, QUICK or DETAILED.

QUICK
    ## for reasoning behind 'blkid', see note 1
        tux [0] /sbin/blkid
    ## following 'sfdisk' assumes size of sdb will support the capacity of the sda partitions (see note 2)
        tux [0] /sbin/sfdisk -d /dev/sda | /bin/sed -e 's/Id=8[23]/Id=fd/g' | /sbin/sfdisk /dev/sdb
        tux [0] /sbin/mdadm -C /dev/md1 --level=raid1 --raid-devices=2 missing /dev/sdb1
        tux [0] /sbin/mdadm -C /dev/md2 --level=raid1 --raid-devices=2 missing /dev/sdb2
        tux [0] /sbin/mdadm -C /dev/md3 --level=raid1 --raid-devices=2 missing /dev/sdb3
        tux [0] for i in 1 2 ; do echo "*** mkfs -V -t ext3 /dev/md${i} ***" ;
        > /sbin/mkfs -V -t ext3 /dev/md${i} ; done
        tux [0] /sbin/tune2fs -L rootfs /dev/md1
        tux [0] /sbin/tune2fs -L varfs /dev/md2
        tux [0] /sbin/mkswap -L SWAP-md3 /dev/md3
        tux [0] [ ! -d /a ] && /bin/mkdir /a
        tux [1] /bin/mount /dev/md1 /a
        tux [0] cd /a ; /sbin/dump 0uf - / | /sbin/restore rf -
        tux [0] /bin/rm restoresymtable
        tux [0] [ ! -d /a/var ] && /bin/mkdir /a/var
        tux [1] /bin/mount /dev/md2 /a/var
        tux [0] cd /a/var ; /sbin/dump 0uf - /var | /sbin/restore rf -
        tux [0] /bin/rm restoresymtable
        tux [0] cd /
        tux [0] /bin/cp /a/etc/fstab /a/etc/fstab.pre-md
    ## update '/a/etc/fstab' with new md devices in place of original sda devices
        tux [0] /bin/vi /a/etc/fstab
    ## if necessary, update '/a/boot/grub/device.map' to include hd1 (sdb)
        tux [0] /bin/vi /a/boot/grub/device.map
        tux [0] /sbin/grub-install --root-directory=/a /dev/sdb
        tux [0] /bin/mount -o bind /proc /a/proc
        tux [0] /bin/mount -o bind /dev /a/dev
        tux [0] /bin/mount -o bind /sys /a/sys
        tux [0] /usr/sbin/chroot /a
        [root@tux /]# /sbin/mkinitrd /boot/initrd-2.6.18-164.el5-raid.img 2.6.18-164.el5
        [root@tux /]# exit
        tux [0] /bin/cp /a/boot/grub/grub.conf /a/boot/grub/grub.conf.pre-md
    ## update /a/boot/grub/grub.conf to account for new RAID configuration on hd1 (sdb)
        tux [0] /bin/vi /a/boot/grub/grub.conf
        tux [0] /sbin/mdadm --detail --scan >> /a/etc/mdadm.conf
    ## update /a/etc/mdadm.conf to include DEVICE line for sda and sdb
        tux [0] /bin/vi /a/etc/mdadm.conf
        tux [0] /bin/umount /a/sys
        tux [0] /bin/umount /a/dev
        tux [0] /bin/umount /a/proc
        tux [0] /bin/umount /a/var
        tux [0] /bin/umount /a
        tux [0] /sbin/fsck -n /dev/md1
        tux [0] /sbin/fsck -n /dev/md2
        tux [0] /sbin/reboot
    ## boot to single user from disk hd1 (sdb) to verify successful md RAID setup
        sh-3.2# /bin/cat /proc/cmdline
        sh-3.2# /bin/df -h
        sh-3.2# /sbin/swapon -s
        sh-3.2# /usr/bin/reboot
    ## reboot to runlevel 3 from disk hd1 (sdb) to setup primary disk (hd0 (sda)), see note 3
        tux [0] /bin/dd if=/dev/zero of=/dev/sda1 bs=1024 count=4096
        tux [0] /bin/dd if=/dev/zero of=/dev/sda2 bs=1024 count=4096
        tux [0] /bin/dd if=/dev/zero of=/dev/sda3 bs=1024 count=4096
        tux [0] /bin/dd if=/dev/zero of=/dev/sda bs=1024 count=4096
        tux [0] /sbin/sfdisk -d /dev/sdb | /sbin/sfdisk /dev/sda
        tux [0] /sbin/mdadm --add /dev/md1 /dev/sda1
        tux [0] /sbin/mdadm --add /dev/md2 /dev/sda2
        tux [0] /sbin/mdadm --add /dev/md3 /dev/sda3
        tux [0] /sbin/mdadm --detail /dev/md1
        tux [0] /sbin/mdadm --detail /dev/md1
        tux [0] /sbin/grub-install /dev/sda
        tux [0] /bin/grep -v ^# /boot/grub/grub.conf
        tux [0] /bin/cat /etc/blkid/blkid.tab
        tux [0] > /etc/blkid/blkid.tab
        tux [0] /sbin/blkid
        tux [0] /bin/cat /etc/blkid/blkid.tab
        tux [0] /sbin/reboot
    ## reboot to runlevel 3 from disk hd0 (sda) to ensure that primary disk is fully functional
        tux [0] /bin/cat /proc/cmdline
        tux [0] /bin/df -h
        tux [0] /sbin/swapon -s
        tux [0] /sbin/mdadm --detail /dev/md1
DETAILED

The first thing we need to do is use 'blkid' to update the block device
cachefile, /etc/blkid/blkid.tab, clearing out any lingering "phantom
entries" (see note 1):
        ## update block device cachefile (/etc/blkid/blkid.tab)
        tux [0] /sbin/blkid
        /dev/sda3: LABEL="SWAP-sda3" TYPE="swap"
        /dev/sda2: LABEL="/var1" UUID="adb2c99d-4985-4504-a37a-9d3cf95467f1" TYPE="ext3"
        /dev/sda1: LABEL="/1" UUID="803a938a-2be1-4228-94cb-92097bbecf8a" TYPE="ext3"
With that out of the way, we use 'fdisk' to verify that 'sdb' has enough
capacity to account for the 'sda' partitions.  Subsequently, use 'sfdisk'
to apply to the 'sda' partition table to 'sdb', and verify with 'fdisk'
that the new 'sdb' table mirrors that of 'sda' (see note 2):
        tux [0] /sbin/fdisk -l /dev/sda /dev/sdb

        Disk /dev/sda: 11.0 GB, 11004805120 bytes
        255 heads, 63 sectors/track, 1337 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes

           Device Boot      Start         End      Blocks   Id  System
        /dev/sda1   *           1         945     7590681   83  Linux
        /dev/sda2             946        1206     2096482+  83  Linux
        /dev/sda3            1207        1337     1052257+  82  Linux swap / Solaris

        Disk /dev/sdb: 11.0 GB, 11005853696 bytes
        255 heads, 63 sectors/track, 1338 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes

        Disk /dev/sdb doesn't contain a valid partition table
        tux [0] /sbin/sfdisk -d /dev/sda | /bin/sed -e 's/Id=8[23]/Id=fd/g' | /sbin/sfdisk /dev/sdb
        Checking that no-one is using this disk right now ...
        OK

        Disk /dev/sdb: 1338 cylinders, 255 heads, 63 sectors/track

        sfdisk: ERROR: sector 0 does not have an msdos signature
         /dev/sdb: unrecognized partition table type
        Old situation:
        No partitions found
        New situation:
        Units = sectors of 512 bytes, counting from 0

           Device Boot    Start       End   #sectors  Id  System
        /dev/sdb1   *        63  15181424   15181362  fd  Linux raid autodetect
        /dev/sdb2      15181425  19374389    4192965  fd  Linux raid autodetect
        /dev/sdb3      19374390  21478904    2104515  fd  Linux raid autodetect
        /dev/sdb4             0         -          0   0  Empty
        Successfully wrote the new partition table

        Re-reading the partition table ...

        If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
        to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
        (See fdisk(8).)
        tux [0] /sbin/fdisk -l /dev/sda /dev/sdb

        Disk /dev/sda: 11.0 GB, 11004805120 bytes
        255 heads, 63 sectors/track, 1337 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes

           Device Boot      Start         End      Blocks   Id  System
        /dev/sda1   *           1         945     7590681   83  Linux
        /dev/sda2             946        1206     2096482+  83  Linux
        /dev/sda3            1207        1337     1052257+  82  Linux swap / Solaris

        Disk /dev/sdb: 11.0 GB, 11005853696 bytes
        255 heads, 63 sectors/track, 1338 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes

           Device Boot      Start         End      Blocks   Id  System
        /dev/sdb1   *           1         945     7590681   fd  Linux raid autodetect
        /dev/sdb2             946        1206     2096482+  fd  Linux raid autodetect
        /dev/sdb3            1207        1337     1052257+  fd  Linux raid autodetect
Of note, the filesystem (FS) ID for the partitions on 'sdb' reads "fd"
rather than "83" or "82".  When mirroring the partition table from
'sda', we substituted in "fd" via 'sed' so that the new partitions are
automatically scanned on bootup for RAID components.  Next, use 'mdadm'
to create each "md" device for the filesystems and swap device.  The 'sdb'
partitions are paired with "missing" which acts as a placeholder for the
'sda' partitions to be added later:
        tux [0] /sbin/mdadm -C /dev/md1 --level=raid1 --raid-devices=2 missing /dev/sdb1
        mdadm: array /dev/md1 started.
        tux [0] /sbin/mdadm -C /dev/md2 --level=raid1 --raid-devices=2 missing /dev/sdb2
        mdadm: array /dev/md2 started.
        tux [0] /sbin/mdadm -C /dev/md3 --level=raid1 --raid-devices=2 missing /dev/sdb3
        mdadm: array /dev/md3 started.
        tux [0] /sbin/mdadm --detail --scan 
        ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=5a13f19a:e513990a:dcb63c9e:8eb86163
        ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=2d89bff8:1cb58a4e:5ce76bbc:aee4d045
        ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=e06caa1e:8f51d4a3:0fec0e68:d0421884
        tux [0] /bin/cat /proc/mdstat
        Personalities : [raid1]
        md1 : active raid1 sdb1[1]
              7590592 blocks [2/1] [_U]

        md2 : active raid1 sdb2[1]
              2096384 blocks [2/1] [_U]

        md3 : active raid1 sdb3[1]
              1052160 blocks [2/1] [_U]

        unused devices: <none>
After verifying the "md" devices with 'mdadm' and a check of
'/proc/mdstat', we use 'mkfs' to create new filesystems on 'sdb[1|2]'
below:
        tux [0] for i in 1 2 ; do echo "*** mkfs -V -t ext3 /dev/md${i} ***" ;
        > /sbin/mkfs -V -t ext3 /dev/md${i} ; done
        *** mkfs -V -t ext3 /dev/md1 ***
        mkfs (util-linux 2.13-pre7)
        mkfs.ext3 /dev/md1
        mke2fs 1.39 (29-May-2006)
        Filesystem label=
        OS type: Linux
        Block size=4096 (log=2)
        Fragment size=4096 (log=2)
        950272 inodes, 1897648 blocks
        94882 blocks (5.00%) reserved for the super user
        First data block=0
        Maximum filesystem blocks=1946157056
        58 block groups
        32768 blocks per group, 32768 fragments per group
        16384 inodes per group
        Superblock backups stored on blocks:
                32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

        Writing inode tables: done
        Creating journal (32768 blocks): done
        Writing superblocks and filesystem accounting information: done

        This filesystem will be automatically checked every 28 mounts or
        180 days, whichever comes first.  Use tune2fs -c or -i to override.
        *** mkfs -V -t ext3 /dev/md2 ***
        mkfs (util-linux 2.13-pre7)
        mkfs.ext3 /dev/md2
        mke2fs 1.39 (29-May-2006)
        Filesystem label=
        OS type: Linux
        Block size=4096 (log=2)
        Fragment size=4096 (log=2)
        262144 inodes, 524096 blocks
        26204 blocks (5.00%) reserved for the super user
        First data block=0
        Maximum filesystem blocks=536870912
        16 block groups
        32768 blocks per group, 32768 fragments per group
        16384 inodes per group
        Superblock backups stored on blocks:
                32768, 98304, 163840, 229376, 294912

        Writing inode tables: done
        Creating journal (8192 blocks): done
        Writing superblocks and filesystem accounting information: done

        This filesystem will be automatically checked every 39 mounts or
        180 days, whichever comes first.  Use tune2fs -c or -i to override.
We get the current volume labels from 'sda[1|2]' with 'tune2fs' to
ensure that we don't create conflicting volume labels on 'md[1|2]' in the
following 'tune2fs' commands.  We use 'swapon' and a check of '/etc/fstab'
for the same reason for swap, before configuring 'md3' via 'mkswap':
        tux [0] ( /sbin/tune2fs -l /dev/sda1 ; /sbin/tune2fs -l /dev/sda2 ) | /bin/grep name
        Filesystem volume name:   /1
        Filesystem volume name:   /var1
        tux [0] /sbin/tune2fs -L rootfs /dev/md1
        tune2fs 1.39 (29-May-2006)
        tux [0] /sbin/tune2fs -L varfs /dev/md2
        tune2fs 1.39 (29-May-2006)
        tux [0] ( /sbin/tune2fs -l /dev/md1 ; /sbin/tune2fs -l /dev/md2 ) | /bin/grep name
        Filesystem volume name:   rootfs
        Filesystem volume name:   varfs
        tux [0] /sbin/swapon -s
        Filename                                Type            Size    Used    Priority
        /dev/sda3                               partition       1052248 0       -1
        tux [0] /bin/grep swap /etc/fstab
        LABEL=SWAP-sda3         swap                    swap    defaults        0 0
        tux [0] /sbin/mkswap -L SWAP-md3 /dev/md3
        Setting up swapspace version 1, size = 1077407 kB
        LABEL=SWAP-md3, no uuid
For the new configuration to be usable, we need to mirror the current
root disk volumes to the "md" volumes.  To do so, after mounting the 'md'
partitions, we use the 'dump' / 'restore' combination.  Don't forget to
remove 'restoresymtable':
        tux [0] [ ! -d /a ] && /bin/mkdir /a
        tux [1] /bin/mount /dev/md1 /a
        tux [0] cd /a ; /sbin/dump 0uf - / | /sbin/restore rf -
          DUMP: Date of this level 0 dump: Sun Jan 23 16:48:25 2011
          DUMP: Dumping /dev/sda1 (/) to standard output
          DUMP: Label: /1
          DUMP: Writing 10 Kilobyte records
          DUMP: mapping (Pass I) [regular files]
          DUMP: mapping (Pass II) [directories]
          DUMP: estimated 2040377 blocks. 
          DUMP: Volume 1 started with block 1 at: Sun Jan 23 16:48:29 2011
          DUMP: dumping (Pass III) [directories]
          DUMP: dumping (Pass IV) [regular files]
        /sbin/restore: ./lost+found: File exists
        ./tmp/rstdir1295819305: (inode 1276706) not found on tape
        ./tmp/rstmode1295819305: (inode 1276711) not found on tape
          DUMP: 59.87% done at 4058 kB/s, finished in 0:03
          DUMP: Volume 1 completed at: Sun Jan 23 16:58:06 2011
          DUMP: Volume 1 2372650 blocks (2317.04MB)
          DUMP: Volume 1 took 0:09:37 
          DUMP: Volume 1 transfer rate: 4112 kB/s
          DUMP: 2372650 blocks (2317.04MB)
          DUMP: finished in 576 seconds, throughput 4119 kBytes/sec
          DUMP: Date of this level 0 dump: Sun Jan 23 16:48:25 2011
          DUMP: Date this dump completed:  Sun Jan 23 16:58:06 2011
          DUMP: Average transfer rate: 4112 kB/s
          DUMP: DUMP IS DONE
        tux [0] /bin/rm restoresymtable
        tux [0] [ ! -d /a/var ] && /bin/mkdir /a/var
        tux [1] /bin/mount /dev/md2 /a/var
        tux [0] cd /a/var ; /sbin/dump 0uf - /var | /sbin/restore rf -
          DUMP: Date of this level 0 dump: Sun Jan 23 17:04:38 2011
          DUMP: Dumping /dev/sda2 (/var) to standard output
          DUMP: Label: /var1
          DUMP: Writing 10 Kilobyte records
          DUMP: mapping (Pass I) [regular files]
          DUMP: mapping (Pass II) [directories]
          DUMP: estimated 37351 blocks.
          DUMP: Volume 1 started with block 1 at: Sun Jan 23 17:04:40 2011
          DUMP: dumping (Pass III) [directories]
          DUMP: dumping (Pass IV) [regular files]
        /sbin/restore: ./lost+found: File exists
          DUMP: Volume 1 completed at: Sun Jan 23 17:04:52 2011
          DUMP: Volume 1 41310 blocks (40.34MB)
          DUMP: Volume 1 took 0:00:12
          DUMP: Volume 1 transfer rate: 3442 kB/s
          DUMP: 41310 blocks (40.34MB)
          DUMP: finished in 12 seconds, throughput 3442 kBytes/sec
          DUMP: Date of this level 0 dump: Sun Jan 23 17:04:38 2011
          DUMP: Date this dump completed:  Sun Jan 23 17:04:52 2011
          DUMP: Average transfer rate: 3442 kB/s
          DUMP: DUMP IS DONE
        tux [0] /bin/rm restoresymtable
        tux [0] cd /
Below, we back up '/a/etc/fstab', and update it from:
        tux [0] /bin/cp /a/etc/fstab /a/etc/fstab.pre-md
        tux [0] /bin/cat /a/etc/fstab
        LABEL=/1                /                       ext3    defaults        1 1
        LABEL=/var1             /var                    ext3    defaults        1 2
        tmpfs                   /dev/shm                tmpfs   defaults        0 0
        devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
        sysfs                   /sys                    sysfs   defaults        0 0
        proc                    /proc                   proc    defaults        0 0
        LABEL=SWAP-sda3         swap                    swap    defaults        0 0
to this:
        tux [0] /bin/cat /a/etc/fstab
        LABEL=rootfs            /                       ext3    defaults        1 1
        LABEL=varfs             /var                    ext3    defaults        1 2
        tmpfs                   /dev/shm                tmpfs   defaults        0 0
        devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
        sysfs                   /sys                    sysfs   defaults        0 0
        proc                    /proc                   proc    defaults        0 0
        LABEL=SWAP-md3          swap                    swap    defaults        0 0
Install the GRUB boot blocks to 'sdb' via 'grub-install':
        tux [0] /sbin/grub-install --root-directory=/a /dev/sdb
        /dev/sdb does not have any corresponding BIOS drive.
If you get the above error, update the GRUB 'device.map' from:
        tux [0] /bin/cat /a/boot/grub/device.map
        # this device map was generated by anaconda
        (hd0)     /dev/sda
to the following, and again run 'grub-install':
        tux [0] /bin/cat /a/boot/grub/device.map
        # this device map was generated by anaconda
        (hd0)     /dev/sda
        (hd1)     /dev/sdb
        tux [0] /sbin/grub-install --root-directory=/a /dev/sdb
        Installation finished. No error reported.
        This is the contents of the device map /a/boot/grub/device.map.
        Check if this is correct or not. If any of the lines is incorrect,
        fix it and re-run the script `grub-install'.

        # this device map was generated by anaconda
        (hd0)     /dev/sda
        (hd1)     /dev/sdb
Because we are moving to an MD mirrored root, we need to create a new
initialized RAM disk (initrd).  If we don't, the 'md' driver won't be
loaded before the mount of our root partition.  The root mount device
would then be '/dev/sdb1', placing the root FS outside of RAID control.
To create the new "initrd", remount [/proc|/dev|/sys] under '/a',
'chroot' to /a, and run 'mkinitrd':
        tux [0] /bin/mount -o bind /proc /a/proc
        tux [0] /bin/mount -o bind /dev /a/dev 
        tux [0] /bin/mount -o bind /sys /a/sys
        tux [0] /usr/sbin/chroot /a
        [root@tux /]# /bin/ls /boot/initrd*
        /boot/initrd-2.6.18-164.el5.img
        [root@tux /]# /sbin/mkinitrd /boot/initrd-2.6.18-164.el5-raid.img 2.6.18-164.el5
        [root@tux /]# /bin/ls /boot/initrd*
        /boot/initrd-2.6.18-164.el5.img  /boot/initrd-2.6.18-164.el5-raid.img
        [root@tux /]# exit
        exit
After exiting the "chroot", backup 'grub.conf' and update from this:
        tux [0] /bin/cp /a/boot/grub/grub.conf /a/boot/grub/grub.conf.pre-md
        tux [0] /bin/grep -v ^# /a/boot/grub/grub.conf
        default=0
        timeout=5
        title CentOS (2.6.18-164.el5)
                root (hd0,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=/1
                initrd /boot/initrd-2.6.18-164.el5.img
to this, modifying the "default" entry, updating title "0", and adding
in new options for our RAID setup:
        tux [0] /bin/grep -v ^# /a/boot/grub/grub.conf
        default=3
        timeout=5
        title CentOS d1 (2.6.18-164.el5)
                root (hd0,0) 
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=/1
                initrd /boot/initrd-2.6.18-164.el5.img 
        title CentOS d2 (2.6.18-164.el5)
                root (hd1,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5.img
        title CentOS RAID1 d1 (2.6.18-164.el5)
                root (hd0,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5-raid.img
        title CentOS RAID1 d2 (2.6.18-164.el5)
                root (hd1,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5-raid.img
Of note, I've left options for a non-RAID boot so that I can boot the
disks individually if necessary, outside of RAID control.  Now, since
it likely won't exist, populate 'mdadm.conf' using 'mdadm':
        tux [0] /sbin/mdadm --detail --scan
        ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=5a13f19a:e513990a:dcb63c9e:8eb86163
        ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=2d89bff8:1cb58a4e:5ce76bbc:aee4d045
        ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=e06caa1e:8f51d4a3:0fec0e68:d0421884
        tux [0] /sbin/mdadm --detail --scan >> /a/etc/mdadm.conf
We will need to update 'mdadm.conf', including the "DEVICE" entry since
'mdadm' only populates the "ARRAY" entries:
        tux [0] /bin/cat /a/etc/mdadm.conf
        DEVICE /dev/sda* /dev/sdb*
        ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=5a13f19a:e513990a:dcb63c9e:8eb86163
        ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=2d89bff8:1cb58a4e:5ce76bbc:aee4d045
        ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=e06caa1e:8f51d4a3:0fec0e68:d0421884
As an aside, the first step we started with was to update the block
device cachefile.  Either running 'blkid' again or simply checking
'/a/etc/blkid/blkid.tab' will show the new "md" devices:
        tux [0] /bin/cat /a/etc/blkid/blkid.tab
        <device DEVNO="0x0803" TIME="1295819305" TYPE="swap" LABEL="SWAP-sda3">/dev/sda3</device>
        <device DEVNO="0x0802" TIME="1295819305" LABEL="/var1" UUID="adb2c99d-4985-4504-a37a-9d3cf95467f1" TYPE="ext3">/dev/sda2</device>
        <device DEVNO="0x0801" TIME="1295819305" LABEL="/1" UUID="803a938a-2be1-4228-94cb-92097bbecf8a" TYPE="ext3">/dev/sda1</device>
        <device DEVNO="0x0901" TIME="1295820810" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" SEC_TYPE="ext2" TYPE="ext3">/dev/md1</device>
        <device DEVNO="0x0902" TIME="1295820810" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/md2</device>
        <device DEVNO="0x0903" TIME="1295820810" LABEL="SWAP-md3" TYPE="swap">/dev/md3</device>
To make sure that our work is sane thus far, umount the pseudo and 'md'
filesystems, and run 'fsck' on the 'md' filesystems prior to reboot:
        tux [0] /bin/umount /a/sys
        tux [0] /bin/umount /a/dev
        tux [0] /bin/umount /a/proc
        tux [0] /bin/umount /a/var
        tux [0] /bin/umount /a
        tux [0] /sbin/fsck -n /dev/md1
        fsck 1.39 (29-May-2006)
        e2fsck 1.39 (29-May-2006)
        rootfs: clean, 72366/950272 files, 572160/1897648 blocks
        tux [0] /sbin/fsck -n /dev/md2
        fsck 1.39 (29-May-2006)
        e2fsck 1.39 (29-May-2006)
        varfs: clean, 11/262144 files, 17206/524096 blocks
        tux [0] /sbin/fsck -n /dev/sdb2
        fsck 1.39 (29-May-2006)
        e2fsck 1.39 (29-May-2006)
        fsck.ext3: Device or resource busy while trying to open /dev/sdb2
        Filesystem mounted or opened exclusively by another program?
        tux [8] /sbin/reboot

        Broadcast message from root (pts/0) (Sun Jan 23 14:02:42 2011):

        The system is going down for reboot NOW!
The above 'fsck' on 'sdb2' is simply to illustrate that while under RAID
control of "md", 'fsck' is unable to check the supporting RAID components.
Once the BIOS allows, select disk 2 (hd1 (sdb)), which will present us
our freshly created GRUB menu:
         GNU GRUB  version 0.97  (639K lower / 1047488K upper memory)

        CentOS d1 (2.6.18-164.el5)
        CentOS d2 (2.6.18-164.el5)
        CentOS RAID1 d1 (2.6.18-164.el5)
        CentOS RAID1 d2 (2.6.18-164.el5)          <==================


           Use the <up> and <down> keys to select which entry is highlighted.
           Press enter to boot the selected OS, 'e' to edit the
           commands before booting, 'a' to modify the kernel arguments
           before booting, or 'c' for a command-line.

        The highlighted entry will be booted automatically in 5 seconds.
Before the 5 seconds timeout, hit 'e' so that we can update the kernel
boot parameters.  (The purpose of this step is to boot into single user
only to verify that the host is functional from the new MD RAID devices.
You could otherwise safely skip this part and simply boot from "CentOS
RAID1 d2 (2.6.18-164.el5)" straight to runlevel 3.):
         GNU GRUB  version 0.97  (639K lower / 1047488K upper memory)

        root (hd1,0)
        kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
        initrd /boot/initrd-2.6.18-164.el5-raid.img 

           Use the <up> and <down> keys to select which entry is highlighted.
           Prebb 'b' to boot, 'e' to edit the selected command in the
           boot sequence, 'c' for a command-line, 'o' to open a new line
           after ('O' for before) the selected line, 'd' to remove the
           selected line, or escape to go back to the main menu.
Update the "kernel" entry from this:
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
to this:
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs -s

         [ Minimal BASH-like line editing is supported.  For the first word, TAB
           lists possible command completions.  Anywhere else TAB lists the possible
           completions of a device/filename.  ESC at any time cancels.  Enter
           at any time accepts your changes.]

        grub edit> kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs -s
After hitting "Enter", type 'b' to boot the system to single user (note
the RAID aware initrd during the boot):
          Booting command-list

        root (hd1,0)
         Filesystem type is ext2fs, partition type 0xfd
        kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs -s
           [Linux-bzImage, setup=0x1e00, size=0x1c31b4]
        initrd /boot/initrd-2.6.18-164.el5-raid.img
           [Linux-initrd @ 0x37d71000, 0x27ee0e bytes]

        <snip...>

        Remounting root filesystem in read-write mode:             [  OK  ]
        Mounting local filesystems:                                [  OK  ]
        Enabling local filesystem quotas:                          [  OK  ]
        Enabling /etc/fstab swaps:                                 [  OK  ]
Once in single user, we simply check the boot options in '/proc/cmdline',
the mounted FS via 'df', and the swap device(s) with 'swapon'.  Follow up
with a reboot:
        sh-3.2# /bin/cat /proc/cmdline
        ro root=LABEL=rootfs -s
        sh-3.2# /bin/df -h
        Filsystem            Size  Used Avail Use% Mounted on
        /dev/md1             7.2G  2.1G  4.8G  31% /
        /dev/md2             2.0G   71M  1.8G   4% /var
        tmpfs                506M     0  506M   0% /dev/shm
        sh-3.2# /sbin/swapon -s
        Filename                               Type             Size    Used    Priority
        /dev/md3                               partition        1052152 0       -1
        sh-3.2# /usr/bin/reboot
        WARNING: could not determine runlevel - doing soft reboot
          (it's better to use shutdown instead of reboot from the command line)
        INIT: Switching to runlevel: 6
        INIT: Sending processes the TERM signal
        INIT: Sending processes the KILL signal
        <snip...>
After the host resets, again boot from "disk 2" and boot into runlevel
3 from "CentOS RAID1 d2 (2.6.18-164.el5"

Once in runlevel 3, we can prepare 'sda' for addition to our RAID
configuration.  Start by using 'dd' to destroy the 'sda' filesystems
and MBR (see note 3):
        tux [0] /bin/dd if=/dev/zero of=/dev/sda1 bs=1024 count=4096
        4096+0 records in
        4096+0 records out
        4194304 bytes (4.2 MB) copied, 0.0684634 seconds, 61.3 MB/s
        tux [0] /bin/dd if=/dev/zero of=/dev/sda2 bs=1024 count=4096
        4096+0 records in
        4096+0 records out
        4194304 bytes (4.2 MB) copied, 0.080138 seconds, 52.3 MB/s
        tux [0] /bin/dd if=/dev/zero of=/dev/sda3 bs=1024 count=4096
        4096+0 records in
        4096+0 records out
        4194304 bytes (4.2 MB) copied, 0.0810015 seconds, 51.8 MB/s
        tux [0] /bin/dd if=/dev/zero of=/dev/sda bs=1024 count=4096
        4096+0 records in
        4096+0 records out
        4194304 bytes (4.2 MB) copied, 0.249196 seconds, 16.8 MB/s
With 'sda' "prepared", use 'sfdisk' to mirror the partition table from
'sdb' to 'sda' (see note 2):
        tux [0] /sbin/sfdisk -d /dev/sdb | /sbin/sfdisk /dev/sda
        Checking that no-one is using this disk right now ...
        OK

        Disk /dev/sda: 1337 cylinders, 255 heads, 63 sectors/track

        sfdisk: ERROR: sector 0 does not have an msdos signature
         /dev/sda: unrecognized partition table type
        Old situation:
        No partitions found
        New situation:
        Units = sectors of 512 bytes, counting from 0

           Device Boot    Start       End   #sectors  Id  System
        /dev/sda1   *        63  15181424   15181362  fd  Linux raid autodetect
        /dev/sda2      15181425  19374389    4192965  fd  Linux raid autodetect
        /dev/sda3      19374390  21478904    2104515  fd  Linux raid autodetect
        /dev/sda4             0         -          0   0  Empty
        Successfully wrote the new partition table

        Re-reading the partition table ...

        If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
        to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
        (See fdisk(8).)
Use 'mdadm' to add the 'sdaX' partitions to the existing 'mdX' devices:
        tux [0] /sbin/mdadm --add /dev/md1 /dev/sda1
        mdadm: added /dev/sda1
        tux [0] /sbin/mdadm --add /dev/md2 /dev/sda2
        mdadm: added /dev/sda2
        tux [0] /sbin/mdadm --add /dev/md3 /dev/sda3
        mdadm: added /dev/sda3
The above will automatically start syncing data from 'sdbX' to 'sdaX'.
Use 'mdadm --detail' to view the status of the sync operation:
        tux [0] /sbin/mdadm --detail /dev/md1
        /dev/md1:
                Version : 0.90
          Creation Time : Sun Jan 23 16:43:32 2011 
             Raid Level : raid1
             Array Size : 7590592 (7.24 GiB 7.77 GB)
          Used Dev Size : 7590592 (7.24 GiB 7.77 GB)
           Raid Devices : 2
          Total Devices : 2
        Preferred Minor : 1
            Persistence : Superblock is persistent

            Update Time : Sun Jan 23 17:58:15 2011
                  State : clean, degraded, recovering
         Active Devices : 1
        Working Devices : 2
         Failed Devices : 0
          Spare Devices : 1

         Rebuild Status : 8% complete

                   UUID : e06caa1e:8f51d4a3:0fec0e68:d0421884 
                 Events : 0.494

            Number   Major   Minor   RaidDevice State
               2       8        1        0      spare rebuilding   /dev/sda1
               1       8       17        1      active sync   /dev/sdb1
When all sync operations have completed on each 'mdX' device, use
'grub-install' to install the GRUB boot blocks to 'sda', then update
'grub.conf':
        tux [0] /sbin/grub-install /dev/sda
        Installation finished. No error reported.
        This is the contents of the device map /boot/grub/device.map.
        Check if this is correct or not. If any of the lines is incorrect,
        fix it and re-run the script `grub-install'.

        # this device map was generated by anaconda
        (hd0)     /dev/sda
        (hd1)     /dev/sdb
        tux [0] /bin/grep -v ^# /boot/grub/grub.conf
        default=2
        timeout=5
        title CentOS d1 (2.6.18-164.el5)
                root (hd0,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5.img
        title CentOS d2 (2.6.18-164.el5)
                root (hd1,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5.img
        title CentOS RAID1 d1 (2.6.18-164.el5)
                root (hd0,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5-raid.img
        title CentOS RAID1 d2 (2.6.18-164.el5)
                root (hd1,0)
                kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
                initrd /boot/initrd-2.6.18-164.el5-raid.img
Above, we updated grub.conf, setting "default" from "3" (RAID hd1) to
"2" (RAID hd0), and changed the "kernel" entry for "CentOS d1" from
"root=LABEL=/1" to "root=LABEL=rootfs".  To account for our new block
device setup, we need to update 'blkid.tab':
        tux [0] /bin/cat /etc/blkid/blkid.tab
        <device DEVNO="0x0803" TIME="1295819305" TYPE="swap" LABEL="SWAP-sda3">/dev/sda3</device>
        <device DEVNO="0x0802" TIME="1295819305" LABEL="/var1" UUID="adb2c99d-4985-4504-a37a-9d3cf95467f1" TYPE="ext3">/dev/sda2</device>
        <device DEVNO="0x0801" TIME="1295819305" LABEL="/1" UUID="803a938a-2be1-4228-94cb-92097bbecf8a" TYPE="ext3">/dev/sda1</device>
        <device DEVNO="0x0901" TIME="1295823067" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" SEC_TYPE="ext2" TYPE="ext3">/dev/md1</device>
        <device DEVNO="0x0902" TIME="1295823068" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3" SEC_TYPE="ext2">/dev/md2</device>
        <device DEVNO="0x0903" TIME="1295823068" TYPE="swap" LABEL="SWAP-md3">/dev/md3</device>
Truncate the file, and run 'blkid' to see the updated entries (which
are automatically written to 'blkid.tab'):
        tux [0] > /etc/blkid/blkid.tab
        tux [0] /sbin/blkid
        /dev/sda1: LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3"
        /dev/sda2: LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3"
        /dev/sda3: TYPE="swap" LABEL="SWAP-md3"
        /dev/sdb1: LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3"
        /dev/sdb2: LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3"
        /dev/sdb3: TYPE="swap" LABEL="SWAP-md3"
        /dev/md1: LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3"
        /dev/md3: TYPE="swap" LABEL="SWAP-md3"
        /dev/md2: LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3"
        tux [0] /bin/cat /etc/blkid/blkid.tab
        <device DEVNO="0x0801" TIME="1295824703" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3">/dev/sda1</device>
        <device DEVNO="0x0802" TIME="1295824703" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/sda2</device>
        <device DEVNO="0x0803" TIME="1295824703" TYPE="swap" LABEL="SWAP-md3">/dev/sda3</device>
        <device DEVNO="0x0811" TIME="1295824703" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3">/dev/sdb1</device>
        <device DEVNO="0x0812" TIME="1295824703" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/sdb2</device>
        <device DEVNO="0x0813" TIME="1295824703" TYPE="swap" LABEL="SWAP-md3">/dev/sdb3</device>
        <device DEVNO="0x0901" TIME="1295824703" PRI="10" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3">/dev/md1</device>
        <device DEVNO="0x0903" TIME="1295824703" PRI="10" TYPE="swap" LABEL="SWAP-md3">/dev/md3</device>
        <device DEVNO="0x0902" TIME="1295824703" PRI="10" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/md2</device>
The finishing steps are to reboot the host, and booting from disk 1 (to
verify 'sda' (hd0) functionality).  Since we reset the GRUB default to
the RAID setup on 'hd0', once in runlevel 3, we verify our filesystems,
swap device, and mirror status reflect this:
        tux [0] /sbin/reboot

    # select disk 1, when prompted by GRUB, accept the new default option, CentOS RAID1 d1 (2.6.18-164.el5)

        tux [0] /bin/cat /proc/cmdline
        ro root=LABEL=rootfs
        tux [0] /bin/df -h
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/md1              7.2G  2.1G  4.8G  31% /
        /dev/md2              2.0G   72M  1.8G   4% /var
        tmpfs                 506M     0  506M   0% /dev/shm
        tux [0] /sbin/swapon -s
        Filename                                Type            Size    Used    Priority
        /dev/md3                                partition       1052152 0       -1
        tux [0] /sbin/mdadm --detail /dev/md1
        /dev/md1:
                Version : 0.90
          Creation Time : Sun Jan 23 16:43:32 2011
             Raid Level : raid1
             Array Size : 7590592 (7.24 GiB 7.77 GB)
          Used Dev Size : 7590592 (7.24 GiB 7.77 GB)
           Raid Devices : 2
          Total Devices : 2
        Preferred Minor : 1
            Persistence : Superblock is persistent

            Update Time : Sun Jan 23 18:29:06 2011
                  State : clean
         Active Devices : 2
        Working Devices : 2
         Failed Devices : 0
          Spare Devices : 0

                   UUID : e06caa1e:8f51d4a3:0fec0e68:d0421884
                 Events : 0.528

            Number   Major   Minor   RaidDevice State
               0       8        1        0      active sync   /dev/sda1
               1       8       17        1      active sync   /dev/sdb1


NOTES

Note 1: We run 'blkid' to clear out "phanton" block devices.  If we don't
    we can run into issues wherein no longer existent devices have UUIDs,
    device numbers, or volume labels conflicting with our RAID devices
    and components.  This can be particularly problematic during boot
    up when the conflict is with the root disk and its volumes.

Note 2: If the disks you are using to create your RAID 1 array are not
    the same size, you don't have to mirror the partition tables
    between the disks.  In the case of the initial 'sfdisk' operation,
    copying from 'sda' to 'sdb', you need to account for at least the
    currently used space on the 'sda' partitions.  Rather than mirroring
    the partition tables, you could manually configure them on 'sdb'
    accounting for said used space.  In the later case of mirroring the
    partition table from 'sdb' to 'sda', you must create at least those
    partitions on 'sdb' that are used in your RAID setup.  To ensure sane
    volumes, the partitions on 'sda' must be the same size as those
    configured on 'sdb'.

Note 3: We need to destroy the filesystems on 'sda' as well as the
    MBR.  We use 'dd' to destroy the MBR since it has reference to
    the disk partition table.  We use 'dd' to destroy the filesystems
    because otherwise, 'mdadm' will notice the prior file systems on the
    partitions and complain about them.  And yes, I deliberately went
    overkill with destroying the first 4MB of the disk and partitions.
    While zeroing out the first 512 MB of the disk would take care of the
    MBR, it didn't fully destroy the superblocks on the partitions and
    'mdadm' still saw them.  1MB probably would have sufficed.

see also:
    Disk Cloning in Linux
    Breaking and Syncing an MD Root Mirror