disk mirror and now you want it mirrored. One could say "just re-install,"
but surely there is another way, and there is. The solution isn't quick
or simple, though it isn't impossible. Our host details are:
HOST: tux PROMPTS: [tux[0] |[root@tux /]# |sh-3.2# ] OS: CentOS 5.4 Linux PRIMARY ROOT: sda1 (hd0 (disk 1)) SECONDARY ROOT: sdb1 (hd1 (disk 2)) MIRRORED ROOT: md1Due to the amount of details, there are two primary sections, QUICK and
DETAILED, with NOTES at the end. QUICK simply lists out all commands,
in order of execution, with no output or discussion. DETAILED provides
all commands, related output, and reasoning behind the steps. With that
in mind, choose your adventure, QUICK or DETAILED.
QUICK
## for reasoning behind 'blkid', see note 1 tux [0] /sbin/blkid ## following 'sfdisk' assumes size of sdb will support the capacity of the sda partitions (see note 2) tux [0] /sbin/sfdisk -d /dev/sda | /bin/sed -e 's/Id=8[23]/Id=fd/g' | /sbin/sfdisk /dev/sdb tux [0] /sbin/mdadm -C /dev/md1 --level=raid1 --raid-devices=2 missing /dev/sdb1 tux [0] /sbin/mdadm -C /dev/md2 --level=raid1 --raid-devices=2 missing /dev/sdb2 tux [0] /sbin/mdadm -C /dev/md3 --level=raid1 --raid-devices=2 missing /dev/sdb3 tux [0] for i in 1 2 ; do echo "*** mkfs -V -t ext3 /dev/md${i} ***" ; > /sbin/mkfs -V -t ext3 /dev/md${i} ; done tux [0] /sbin/tune2fs -L rootfs /dev/md1 tux [0] /sbin/tune2fs -L varfs /dev/md2 tux [0] /sbin/mkswap -L SWAP-md3 /dev/md3 tux [0] [ ! -d /a ] && /bin/mkdir /a tux [1] /bin/mount /dev/md1 /a tux [0] cd /a ; /sbin/dump 0uf - / | /sbin/restore rf - tux [0] /bin/rm restoresymtable tux [0] [ ! -d /a/var ] && /bin/mkdir /a/var tux [1] /bin/mount /dev/md2 /a/var tux [0] cd /a/var ; /sbin/dump 0uf - /var | /sbin/restore rf - tux [0] /bin/rm restoresymtable tux [0] cd / tux [0] /bin/cp /a/etc/fstab /a/etc/fstab.pre-md ## update '/a/etc/fstab' with new md devices in place of original sda devices tux [0] /bin/vi /a/etc/fstab ## if necessary, update '/a/boot/grub/device.map' to include hd1 (sdb) tux [0] /bin/vi /a/boot/grub/device.map tux [0] /sbin/grub-install --root-directory=/a /dev/sdb tux [0] /bin/mount -o bind /proc /a/proc tux [0] /bin/mount -o bind /dev /a/dev tux [0] /bin/mount -o bind /sys /a/sys tux [0] /usr/sbin/chroot /a [root@tux /]# /sbin/mkinitrd /boot/initrd-2.6.18-164.el5-raid.img 2.6.18-164.el5 [root@tux /]# exit tux [0] /bin/cp /a/boot/grub/grub.conf /a/boot/grub/grub.conf.pre-md ## update /a/boot/grub/grub.conf to account for new RAID configuration on hd1 (sdb) tux [0] /bin/vi /a/boot/grub/grub.conf tux [0] /sbin/mdadm --detail --scan >> /a/etc/mdadm.conf ## update /a/etc/mdadm.conf to include DEVICE line for sda and sdb tux [0] /bin/vi /a/etc/mdadm.conf tux [0] /bin/umount /a/sys tux [0] /bin/umount /a/dev tux [0] /bin/umount /a/proc tux [0] /bin/umount /a/var tux [0] /bin/umount /a tux [0] /sbin/fsck -n /dev/md1 tux [0] /sbin/fsck -n /dev/md2 tux [0] /sbin/reboot ## boot to single user from disk hd1 (sdb) to verify successful md RAID setup sh-3.2# /bin/cat /proc/cmdline sh-3.2# /bin/df -h sh-3.2# /sbin/swapon -s sh-3.2# /usr/bin/reboot ## reboot to runlevel 3 from disk hd1 (sdb) to setup primary disk (hd0 (sda)), see note 3 tux [0] /bin/dd if=/dev/zero of=/dev/sda1 bs=1024 count=4096 tux [0] /bin/dd if=/dev/zero of=/dev/sda2 bs=1024 count=4096 tux [0] /bin/dd if=/dev/zero of=/dev/sda3 bs=1024 count=4096 tux [0] /bin/dd if=/dev/zero of=/dev/sda bs=1024 count=4096 tux [0] /sbin/sfdisk -d /dev/sdb | /sbin/sfdisk /dev/sda tux [0] /sbin/mdadm --add /dev/md1 /dev/sda1 tux [0] /sbin/mdadm --add /dev/md2 /dev/sda2 tux [0] /sbin/mdadm --add /dev/md3 /dev/sda3 tux [0] /sbin/mdadm --detail /dev/md1 tux [0] /sbin/mdadm --detail /dev/md1 tux [0] /sbin/grub-install /dev/sda tux [0] /bin/grep -v ^# /boot/grub/grub.conf tux [0] /bin/cat /etc/blkid/blkid.tab tux [0] > /etc/blkid/blkid.tab tux [0] /sbin/blkid tux [0] /bin/cat /etc/blkid/blkid.tab tux [0] /sbin/reboot ## reboot to runlevel 3 from disk hd0 (sda) to ensure that primary disk is fully functional tux [0] /bin/cat /proc/cmdline tux [0] /bin/df -h tux [0] /sbin/swapon -s tux [0] /sbin/mdadm --detail /dev/md1DETAILED
The first thing we need to do is use 'blkid' to update the block device
cachefile, /etc/blkid/blkid.tab, clearing out any lingering "phantom
entries" (see note 1):
## update block device cachefile (/etc/blkid/blkid.tab) tux [0] /sbin/blkid /dev/sda3: LABEL="SWAP-sda3" TYPE="swap" /dev/sda2: LABEL="/var1" UUID="adb2c99d-4985-4504-a37a-9d3cf95467f1" TYPE="ext3" /dev/sda1: LABEL="/1" UUID="803a938a-2be1-4228-94cb-92097bbecf8a" TYPE="ext3"With that out of the way, we use 'fdisk' to verify that 'sdb' has enough
capacity to account for the 'sda' partitions. Subsequently, use 'sfdisk'
to apply to the 'sda' partition table to 'sdb', and verify with 'fdisk'
that the new 'sdb' table mirrors that of 'sda' (see note 2):
tux [0] /sbin/fdisk -l /dev/sda /dev/sdb Disk /dev/sda: 11.0 GB, 11004805120 bytes 255 heads, 63 sectors/track, 1337 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 945 7590681 83 Linux /dev/sda2 946 1206 2096482+ 83 Linux /dev/sda3 1207 1337 1052257+ 82 Linux swap / Solaris Disk /dev/sdb: 11.0 GB, 11005853696 bytes 255 heads, 63 sectors/track, 1338 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdb doesn't contain a valid partition table tux [0] /sbin/sfdisk -d /dev/sda | /bin/sed -e 's/Id=8[23]/Id=fd/g' | /sbin/sfdisk /dev/sdb Checking that no-one is using this disk right now ... OK Disk /dev/sdb: 1338 cylinders, 255 heads, 63 sectors/track sfdisk: ERROR: sector 0 does not have an msdos signature /dev/sdb: unrecognized partition table type Old situation: No partitions found New situation: Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sdb1 * 63 15181424 15181362 fd Linux raid autodetect /dev/sdb2 15181425 19374389 4192965 fd Linux raid autodetect /dev/sdb3 19374390 21478904 2104515 fd Linux raid autodetect /dev/sdb4 0 - 0 0 Empty Successfully wrote the new partition table Re-reading the partition table ... If you created or changed a DOS partition, /dev/foo7, say, then use dd(1) to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1 (See fdisk(8).) tux [0] /sbin/fdisk -l /dev/sda /dev/sdb Disk /dev/sda: 11.0 GB, 11004805120 bytes 255 heads, 63 sectors/track, 1337 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 945 7590681 83 Linux /dev/sda2 946 1206 2096482+ 83 Linux /dev/sda3 1207 1337 1052257+ 82 Linux swap / Solaris Disk /dev/sdb: 11.0 GB, 11005853696 bytes 255 heads, 63 sectors/track, 1338 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 945 7590681 fd Linux raid autodetect /dev/sdb2 946 1206 2096482+ fd Linux raid autodetect /dev/sdb3 1207 1337 1052257+ fd Linux raid autodetectOf note, the filesystem (FS) ID for the partitions on 'sdb' reads "fd"
rather than "83" or "82". When mirroring the partition table from
'sda', we substituted in "fd" via 'sed' so that the new partitions are
automatically scanned on bootup for RAID components. Next, use 'mdadm'
to create each "md" device for the filesystems and swap device. The 'sdb'
partitions are paired with "missing" which acts as a placeholder for the
'sda' partitions to be added later:
tux [0] /sbin/mdadm -C /dev/md1 --level=raid1 --raid-devices=2 missing /dev/sdb1 mdadm: array /dev/md1 started. tux [0] /sbin/mdadm -C /dev/md2 --level=raid1 --raid-devices=2 missing /dev/sdb2 mdadm: array /dev/md2 started. tux [0] /sbin/mdadm -C /dev/md3 --level=raid1 --raid-devices=2 missing /dev/sdb3 mdadm: array /dev/md3 started. tux [0] /sbin/mdadm --detail --scan ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=5a13f19a:e513990a:dcb63c9e:8eb86163 ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=2d89bff8:1cb58a4e:5ce76bbc:aee4d045 ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=e06caa1e:8f51d4a3:0fec0e68:d0421884 tux [0] /bin/cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb1[1] 7590592 blocks [2/1] [_U] md2 : active raid1 sdb2[1] 2096384 blocks [2/1] [_U] md3 : active raid1 sdb3[1] 1052160 blocks [2/1] [_U] unused devices: <none>After verifying the "md" devices with 'mdadm' and a check of
'/proc/mdstat', we use 'mkfs' to create new filesystems on 'sdb[1|2]'
below:
tux [0] for i in 1 2 ; do echo "*** mkfs -V -t ext3 /dev/md${i} ***" ;
> /sbin/mkfs -V -t ext3 /dev/md${i} ; done
*** mkfs -V -t ext3 /dev/md1 ***
mkfs (util-linux 2.13-pre7)
mkfs.ext3 /dev/md1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
950272 inodes, 1897648 blocks
94882 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1946157056
58 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
*** mkfs -V -t ext3 /dev/md2 ***
mkfs (util-linux 2.13-pre7)
mkfs.ext3 /dev/md2
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
262144 inodes, 524096 blocks
26204 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
We get the current volume labels from 'sda[1|2]' with 'tune2fs' toensure that we don't create conflicting volume labels on 'md[1|2]' in the
following 'tune2fs' commands. We use 'swapon' and a check of '/etc/fstab'
for the same reason for swap, before configuring 'md3' via 'mkswap':
tux [0] ( /sbin/tune2fs -l /dev/sda1 ; /sbin/tune2fs -l /dev/sda2 ) | /bin/grep name Filesystem volume name: /1 Filesystem volume name: /var1 tux [0] /sbin/tune2fs -L rootfs /dev/md1 tune2fs 1.39 (29-May-2006) tux [0] /sbin/tune2fs -L varfs /dev/md2 tune2fs 1.39 (29-May-2006) tux [0] ( /sbin/tune2fs -l /dev/md1 ; /sbin/tune2fs -l /dev/md2 ) | /bin/grep name Filesystem volume name: rootfs Filesystem volume name: varfs tux [0] /sbin/swapon -s Filename Type Size Used Priority /dev/sda3 partition 1052248 0 -1 tux [0] /bin/grep swap /etc/fstab LABEL=SWAP-sda3 swap swap defaults 0 0 tux [0] /sbin/mkswap -L SWAP-md3 /dev/md3 Setting up swapspace version 1, size = 1077407 kB LABEL=SWAP-md3, no uuidFor the new configuration to be usable, we need to mirror the current
root disk volumes to the "md" volumes. To do so, after mounting the 'md'
partitions, we use the 'dump' / 'restore' combination. Don't forget to
remove 'restoresymtable':
tux [0] [ ! -d /a ] && /bin/mkdir /a tux [1] /bin/mount /dev/md1 /a tux [0] cd /a ; /sbin/dump 0uf - / | /sbin/restore rf - DUMP: Date of this level 0 dump: Sun Jan 23 16:48:25 2011 DUMP: Dumping /dev/sda1 (/) to standard output DUMP: Label: /1 DUMP: Writing 10 Kilobyte records DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 2040377 blocks. DUMP: Volume 1 started with block 1 at: Sun Jan 23 16:48:29 2011 DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] /sbin/restore: ./lost+found: File exists ./tmp/rstdir1295819305: (inode 1276706) not found on tape ./tmp/rstmode1295819305: (inode 1276711) not found on tape DUMP: 59.87% done at 4058 kB/s, finished in 0:03 DUMP: Volume 1 completed at: Sun Jan 23 16:58:06 2011 DUMP: Volume 1 2372650 blocks (2317.04MB) DUMP: Volume 1 took 0:09:37 DUMP: Volume 1 transfer rate: 4112 kB/s DUMP: 2372650 blocks (2317.04MB) DUMP: finished in 576 seconds, throughput 4119 kBytes/sec DUMP: Date of this level 0 dump: Sun Jan 23 16:48:25 2011 DUMP: Date this dump completed: Sun Jan 23 16:58:06 2011 DUMP: Average transfer rate: 4112 kB/s DUMP: DUMP IS DONE tux [0] /bin/rm restoresymtable tux [0] [ ! -d /a/var ] && /bin/mkdir /a/var tux [1] /bin/mount /dev/md2 /a/var tux [0] cd /a/var ; /sbin/dump 0uf - /var | /sbin/restore rf - DUMP: Date of this level 0 dump: Sun Jan 23 17:04:38 2011 DUMP: Dumping /dev/sda2 (/var) to standard output DUMP: Label: /var1 DUMP: Writing 10 Kilobyte records DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 37351 blocks. DUMP: Volume 1 started with block 1 at: Sun Jan 23 17:04:40 2011 DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] /sbin/restore: ./lost+found: File exists DUMP: Volume 1 completed at: Sun Jan 23 17:04:52 2011 DUMP: Volume 1 41310 blocks (40.34MB) DUMP: Volume 1 took 0:00:12 DUMP: Volume 1 transfer rate: 3442 kB/s DUMP: 41310 blocks (40.34MB) DUMP: finished in 12 seconds, throughput 3442 kBytes/sec DUMP: Date of this level 0 dump: Sun Jan 23 17:04:38 2011 DUMP: Date this dump completed: Sun Jan 23 17:04:52 2011 DUMP: Average transfer rate: 3442 kB/s DUMP: DUMP IS DONE tux [0] /bin/rm restoresymtable tux [0] cd /Below, we back up '/a/etc/fstab', and update it from:
tux [0] /bin/cp /a/etc/fstab /a/etc/fstab.pre-md tux [0] /bin/cat /a/etc/fstab LABEL=/1 / ext3 defaults 1 1 LABEL=/var1 /var ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 LABEL=SWAP-sda3 swap swap defaults 0 0to this:
tux [0] /bin/cat /a/etc/fstab LABEL=rootfs / ext3 defaults 1 1 LABEL=varfs /var ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 LABEL=SWAP-md3 swap swap defaults 0 0Install the GRUB boot blocks to 'sdb' via 'grub-install':
tux [0] /sbin/grub-install --root-directory=/a /dev/sdb /dev/sdb does not have any corresponding BIOS drive.If you get the above error, update the GRUB 'device.map' from:
tux [0] /bin/cat /a/boot/grub/device.map # this device map was generated by anaconda (hd0) /dev/sdato the following, and again run 'grub-install':
tux [0] /bin/cat /a/boot/grub/device.map # this device map was generated by anaconda (hd0) /dev/sda (hd1) /dev/sdb tux [0] /sbin/grub-install --root-directory=/a /dev/sdb Installation finished. No error reported. This is the contents of the device map /a/boot/grub/device.map. Check if this is correct or not. If any of the lines is incorrect, fix it and re-run the script `grub-install'. # this device map was generated by anaconda (hd0) /dev/sda (hd1) /dev/sdbBecause we are moving to an MD mirrored root, we need to create a new
initialized RAM disk (initrd). If we don't, the 'md' driver won't be
loaded before the mount of our root partition. The root mount device
would then be '/dev/sdb1', placing the root FS outside of RAID control.
To create the new "initrd", remount [/proc|/dev|/sys] under '/a',
'chroot' to /a, and run 'mkinitrd':
tux [0] /bin/mount -o bind /proc /a/proc tux [0] /bin/mount -o bind /dev /a/dev tux [0] /bin/mount -o bind /sys /a/sys tux [0] /usr/sbin/chroot /a [root@tux /]# /bin/ls /boot/initrd* /boot/initrd-2.6.18-164.el5.img [root@tux /]# /sbin/mkinitrd /boot/initrd-2.6.18-164.el5-raid.img 2.6.18-164.el5 [root@tux /]# /bin/ls /boot/initrd* /boot/initrd-2.6.18-164.el5.img /boot/initrd-2.6.18-164.el5-raid.img [root@tux /]# exit exitAfter exiting the "chroot", backup 'grub.conf' and update from this:
tux [0] /bin/cp /a/boot/grub/grub.conf /a/boot/grub/grub.conf.pre-md tux [0] /bin/grep -v ^# /a/boot/grub/grub.conf default=0 timeout=5 title CentOS (2.6.18-164.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=/1 initrd /boot/initrd-2.6.18-164.el5.imgto this, modifying the "default" entry, updating title "0", and adding
in new options for our RAID setup:
tux [0] /bin/grep -v ^# /a/boot/grub/grub.conf default=3 timeout=5 title CentOS d1 (2.6.18-164.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=/1 initrd /boot/initrd-2.6.18-164.el5.img title CentOS d2 (2.6.18-164.el5) root (hd1,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5.img title CentOS RAID1 d1 (2.6.18-164.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5-raid.img title CentOS RAID1 d2 (2.6.18-164.el5) root (hd1,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5-raid.imgOf note, I've left options for a non-RAID boot so that I can boot the
disks individually if necessary, outside of RAID control. Now, since
it likely won't exist, populate 'mdadm.conf' using 'mdadm':
tux [0] /sbin/mdadm --detail --scan ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=5a13f19a:e513990a:dcb63c9e:8eb86163 ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=2d89bff8:1cb58a4e:5ce76bbc:aee4d045 ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=e06caa1e:8f51d4a3:0fec0e68:d0421884 tux [0] /sbin/mdadm --detail --scan >> /a/etc/mdadm.confWe will need to update 'mdadm.conf', including the "DEVICE" entry since
'mdadm' only populates the "ARRAY" entries:
tux [0] /bin/cat /a/etc/mdadm.conf DEVICE /dev/sda* /dev/sdb* ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=5a13f19a:e513990a:dcb63c9e:8eb86163 ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=2d89bff8:1cb58a4e:5ce76bbc:aee4d045 ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=e06caa1e:8f51d4a3:0fec0e68:d0421884As an aside, the first step we started with was to update the block
device cachefile. Either running 'blkid' again or simply checking
'/a/etc/blkid/blkid.tab' will show the new "md" devices:
tux [0] /bin/cat /a/etc/blkid/blkid.tab <device DEVNO="0x0803" TIME="1295819305" TYPE="swap" LABEL="SWAP-sda3">/dev/sda3</device> <device DEVNO="0x0802" TIME="1295819305" LABEL="/var1" UUID="adb2c99d-4985-4504-a37a-9d3cf95467f1" TYPE="ext3">/dev/sda2</device> <device DEVNO="0x0801" TIME="1295819305" LABEL="/1" UUID="803a938a-2be1-4228-94cb-92097bbecf8a" TYPE="ext3">/dev/sda1</device> <device DEVNO="0x0901" TIME="1295820810" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" SEC_TYPE="ext2" TYPE="ext3">/dev/md1</device> <device DEVNO="0x0902" TIME="1295820810" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/md2</device> <device DEVNO="0x0903" TIME="1295820810" LABEL="SWAP-md3" TYPE="swap">/dev/md3</device>To make sure that our work is sane thus far, umount the pseudo and 'md'
filesystems, and run 'fsck' on the 'md' filesystems prior to reboot:
tux [0] /bin/umount /a/sys tux [0] /bin/umount /a/dev tux [0] /bin/umount /a/proc tux [0] /bin/umount /a/var tux [0] /bin/umount /a tux [0] /sbin/fsck -n /dev/md1 fsck 1.39 (29-May-2006) e2fsck 1.39 (29-May-2006) rootfs: clean, 72366/950272 files, 572160/1897648 blocks tux [0] /sbin/fsck -n /dev/md2 fsck 1.39 (29-May-2006) e2fsck 1.39 (29-May-2006) varfs: clean, 11/262144 files, 17206/524096 blocks tux [0] /sbin/fsck -n /dev/sdb2 fsck 1.39 (29-May-2006) e2fsck 1.39 (29-May-2006) fsck.ext3: Device or resource busy while trying to open /dev/sdb2 Filesystem mounted or opened exclusively by another program? tux [8] /sbin/reboot Broadcast message from root (pts/0) (Sun Jan 23 14:02:42 2011): The system is going down for reboot NOW!The above 'fsck' on 'sdb2' is simply to illustrate that while under RAID
control of "md", 'fsck' is unable to check the supporting RAID components.
Once the BIOS allows, select disk 2 (hd1 (sdb)), which will present us
our freshly created GRUB menu:
GNU GRUB version 0.97 (639K lower / 1047488K upper memory) CentOS d1 (2.6.18-164.el5) CentOS d2 (2.6.18-164.el5) CentOS RAID1 d1 (2.6.18-164.el5) CentOS RAID1 d2 (2.6.18-164.el5) <================== Use the <up> and <down> keys to select which entry is highlighted. Press enter to boot the selected OS, 'e' to edit the commands before booting, 'a' to modify the kernel arguments before booting, or 'c' for a command-line. The highlighted entry will be booted automatically in 5 seconds.Before the 5 seconds timeout, hit 'e' so that we can update the kernel
boot parameters. (The purpose of this step is to boot into single user
only to verify that the host is functional from the new MD RAID devices.
You could otherwise safely skip this part and simply boot from "CentOS
RAID1 d2 (2.6.18-164.el5)" straight to runlevel 3.):
GNU GRUB version 0.97 (639K lower / 1047488K upper memory)
root (hd1,0)
kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs
initrd /boot/initrd-2.6.18-164.el5-raid.img
Use the <up> and <down> keys to select which entry is highlighted.
Prebb 'b' to boot, 'e' to edit the selected command in the
boot sequence, 'c' for a command-line, 'o' to open a new line
after ('O' for before) the selected line, 'd' to remove the
selected line, or escape to go back to the main menu.
Update the "kernel" entry from this:kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfsto this:
kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs -s [ Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists the possible completions of a device/filename. ESC at any time cancels. Enter at any time accepts your changes.] grub edit> kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs -sAfter hitting "Enter", type 'b' to boot the system to single user (note
the RAID aware initrd during the boot):
Booting command-list root (hd1,0) Filesystem type is ext2fs, partition type 0xfd kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs -s [Linux-bzImage, setup=0x1e00, size=0x1c31b4] initrd /boot/initrd-2.6.18-164.el5-raid.img [Linux-initrd @ 0x37d71000, 0x27ee0e bytes] <snip...> Remounting root filesystem in read-write mode: [ OK ] Mounting local filesystems: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling /etc/fstab swaps: [ OK ]Once in single user, we simply check the boot options in '/proc/cmdline',
the mounted FS via 'df', and the swap device(s) with 'swapon'. Follow up
with a reboot:
sh-3.2# /bin/cat /proc/cmdline ro root=LABEL=rootfs -s sh-3.2# /bin/df -h Filsystem Size Used Avail Use% Mounted on /dev/md1 7.2G 2.1G 4.8G 31% / /dev/md2 2.0G 71M 1.8G 4% /var tmpfs 506M 0 506M 0% /dev/shm sh-3.2# /sbin/swapon -s Filename Type Size Used Priority /dev/md3 partition 1052152 0 -1 sh-3.2# /usr/bin/reboot WARNING: could not determine runlevel - doing soft reboot (it's better to use shutdown instead of reboot from the command line) INIT: Switching to runlevel: 6 INIT: Sending processes the TERM signal INIT: Sending processes the KILL signal <snip...>After the host resets, again boot from "disk 2" and boot into runlevel
3 from "CentOS RAID1 d2 (2.6.18-164.el5"
Once in runlevel 3, we can prepare 'sda' for addition to our RAID
configuration. Start by using 'dd' to destroy the 'sda' filesystems
and MBR (see note 3):
tux [0] /bin/dd if=/dev/zero of=/dev/sda1 bs=1024 count=4096 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.0684634 seconds, 61.3 MB/s tux [0] /bin/dd if=/dev/zero of=/dev/sda2 bs=1024 count=4096 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.080138 seconds, 52.3 MB/s tux [0] /bin/dd if=/dev/zero of=/dev/sda3 bs=1024 count=4096 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.0810015 seconds, 51.8 MB/s tux [0] /bin/dd if=/dev/zero of=/dev/sda bs=1024 count=4096 4096+0 records in 4096+0 records out 4194304 bytes (4.2 MB) copied, 0.249196 seconds, 16.8 MB/sWith 'sda' "prepared", use 'sfdisk' to mirror the partition table from
'sdb' to 'sda' (see note 2):
tux [0] /sbin/sfdisk -d /dev/sdb | /sbin/sfdisk /dev/sda Checking that no-one is using this disk right now ... OK Disk /dev/sda: 1337 cylinders, 255 heads, 63 sectors/track sfdisk: ERROR: sector 0 does not have an msdos signature /dev/sda: unrecognized partition table type Old situation: No partitions found New situation: Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sda1 * 63 15181424 15181362 fd Linux raid autodetect /dev/sda2 15181425 19374389 4192965 fd Linux raid autodetect /dev/sda3 19374390 21478904 2104515 fd Linux raid autodetect /dev/sda4 0 - 0 0 Empty Successfully wrote the new partition table Re-reading the partition table ... If you created or changed a DOS partition, /dev/foo7, say, then use dd(1) to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1 (See fdisk(8).)Use 'mdadm' to add the 'sdaX' partitions to the existing 'mdX' devices:
tux [0] /sbin/mdadm --add /dev/md1 /dev/sda1 mdadm: added /dev/sda1 tux [0] /sbin/mdadm --add /dev/md2 /dev/sda2 mdadm: added /dev/sda2 tux [0] /sbin/mdadm --add /dev/md3 /dev/sda3 mdadm: added /dev/sda3The above will automatically start syncing data from 'sdbX' to 'sdaX'.
Use 'mdadm --detail' to view the status of the sync operation:
tux [0] /sbin/mdadm --detail /dev/md1 /dev/md1: Version : 0.90 Creation Time : Sun Jan 23 16:43:32 2011 Raid Level : raid1 Array Size : 7590592 (7.24 GiB 7.77 GB) Used Dev Size : 7590592 (7.24 GiB 7.77 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Sun Jan 23 17:58:15 2011 State : clean, degraded, recovering Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 Rebuild Status : 8% complete UUID : e06caa1e:8f51d4a3:0fec0e68:d0421884 Events : 0.494 Number Major Minor RaidDevice State 2 8 1 0 spare rebuilding /dev/sda1 1 8 17 1 active sync /dev/sdb1When all sync operations have completed on each 'mdX' device, use
'grub-install' to install the GRUB boot blocks to 'sda', then update
'grub.conf':
tux [0] /sbin/grub-install /dev/sda Installation finished. No error reported. This is the contents of the device map /boot/grub/device.map. Check if this is correct or not. If any of the lines is incorrect, fix it and re-run the script `grub-install'. # this device map was generated by anaconda (hd0) /dev/sda (hd1) /dev/sdb tux [0] /bin/grep -v ^# /boot/grub/grub.conf default=2 timeout=5 title CentOS d1 (2.6.18-164.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5.img title CentOS d2 (2.6.18-164.el5) root (hd1,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5.img title CentOS RAID1 d1 (2.6.18-164.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5-raid.img title CentOS RAID1 d2 (2.6.18-164.el5) root (hd1,0) kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=rootfs initrd /boot/initrd-2.6.18-164.el5-raid.imgAbove, we updated grub.conf, setting "default" from "3" (RAID hd1) to
"2" (RAID hd0), and changed the "kernel" entry for "CentOS d1" from
"root=LABEL=/1" to "root=LABEL=rootfs". To account for our new block
device setup, we need to update 'blkid.tab':
tux [0] /bin/cat /etc/blkid/blkid.tab <device DEVNO="0x0803" TIME="1295819305" TYPE="swap" LABEL="SWAP-sda3">/dev/sda3</device> <device DEVNO="0x0802" TIME="1295819305" LABEL="/var1" UUID="adb2c99d-4985-4504-a37a-9d3cf95467f1" TYPE="ext3">/dev/sda2</device> <device DEVNO="0x0801" TIME="1295819305" LABEL="/1" UUID="803a938a-2be1-4228-94cb-92097bbecf8a" TYPE="ext3">/dev/sda1</device> <device DEVNO="0x0901" TIME="1295823067" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" SEC_TYPE="ext2" TYPE="ext3">/dev/md1</device> <device DEVNO="0x0902" TIME="1295823068" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3" SEC_TYPE="ext2">/dev/md2</device> <device DEVNO="0x0903" TIME="1295823068" TYPE="swap" LABEL="SWAP-md3">/dev/md3</device>Truncate the file, and run 'blkid' to see the updated entries (which
are automatically written to 'blkid.tab'):
tux [0] > /etc/blkid/blkid.tab tux [0] /sbin/blkid /dev/sda1: LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3" /dev/sda2: LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3" /dev/sda3: TYPE="swap" LABEL="SWAP-md3" /dev/sdb1: LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3" /dev/sdb2: LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3" /dev/sdb3: TYPE="swap" LABEL="SWAP-md3" /dev/md1: LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3" /dev/md3: TYPE="swap" LABEL="SWAP-md3" /dev/md2: LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3" tux [0] /bin/cat /etc/blkid/blkid.tab <device DEVNO="0x0801" TIME="1295824703" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3">/dev/sda1</device> <device DEVNO="0x0802" TIME="1295824703" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/sda2</device> <device DEVNO="0x0803" TIME="1295824703" TYPE="swap" LABEL="SWAP-md3">/dev/sda3</device> <device DEVNO="0x0811" TIME="1295824703" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3">/dev/sdb1</device> <device DEVNO="0x0812" TIME="1295824703" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/sdb2</device> <device DEVNO="0x0813" TIME="1295824703" TYPE="swap" LABEL="SWAP-md3">/dev/sdb3</device> <device DEVNO="0x0901" TIME="1295824703" PRI="10" LABEL="rootfs" UUID="50e1eeb8-d726-43ca-a918-c7102ae9aa7e" TYPE="ext3">/dev/md1</device> <device DEVNO="0x0903" TIME="1295824703" PRI="10" TYPE="swap" LABEL="SWAP-md3">/dev/md3</device> <device DEVNO="0x0902" TIME="1295824703" PRI="10" LABEL="varfs" UUID="590fbc90-e94b-4f4b-ad2d-b7b61798ff80" TYPE="ext3">/dev/md2</device>The finishing steps are to reboot the host, and booting from disk 1 (to
verify 'sda' (hd0) functionality). Since we reset the GRUB default to
the RAID setup on 'hd0', once in runlevel 3, we verify our filesystems,
swap device, and mirror status reflect this:
tux [0] /sbin/reboot # select disk 1, when prompted by GRUB, accept the new default option, CentOS RAID1 d1 (2.6.18-164.el5) tux [0] /bin/cat /proc/cmdline ro root=LABEL=rootfs tux [0] /bin/df -h Filesystem Size Used Avail Use% Mounted on /dev/md1 7.2G 2.1G 4.8G 31% / /dev/md2 2.0G 72M 1.8G 4% /var tmpfs 506M 0 506M 0% /dev/shm tux [0] /sbin/swapon -s Filename Type Size Used Priority /dev/md3 partition 1052152 0 -1 tux [0] /sbin/mdadm --detail /dev/md1 /dev/md1: Version : 0.90 Creation Time : Sun Jan 23 16:43:32 2011 Raid Level : raid1 Array Size : 7590592 (7.24 GiB 7.77 GB) Used Dev Size : 7590592 (7.24 GiB 7.77 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Sun Jan 23 18:29:06 2011 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : e06caa1e:8f51d4a3:0fec0e68:d0421884 Events : 0.528 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1
NOTES
Note 1: We run 'blkid' to clear out "phanton" block devices. If we don't
we can run into issues wherein no longer existent devices have UUIDs,
device numbers, or volume labels conflicting with our RAID devices
and components. This can be particularly problematic during boot
up when the conflict is with the root disk and its volumes.
Note 2: If the disks you are using to create your RAID 1 array are not
the same size, you don't have to mirror the partition tables
between the disks. In the case of the initial 'sfdisk' operation,
copying from 'sda' to 'sdb', you need to account for at least the
currently used space on the 'sda' partitions. Rather than mirroring
the partition tables, you could manually configure them on 'sdb'
accounting for said used space. In the later case of mirroring the
partition table from 'sdb' to 'sda', you must create at least those
partitions on 'sdb' that are used in your RAID setup. To ensure sane
volumes, the partitions on 'sda' must be the same size as those
configured on 'sdb'.
Note 3: We need to destroy the filesystems on 'sda' as well as the
MBR. We use 'dd' to destroy the MBR since it has reference to
the disk partition table. We use 'dd' to destroy the filesystems
because otherwise, 'mdadm' will notice the prior file systems on the
partitions and complain about them. And yes, I deliberately went
overkill with destroying the first 4MB of the disk and partitions.
While zeroing out the first 512 MB of the disk would take care of the
MBR, it didn't fully destroy the superblocks on the partitions and
'mdadm' still saw them. 1MB probably would have sufficed.
see also:
Disk Cloning in Linux
Breaking and Syncing an MD Root Mirror
