28 November 2010

Replacing VxVM with SVM for the rootdisk

The purpose of this post is to illustrate how to replace Veritas Volume
Manager (VxVM) with Sun Volume Manager (SVM) for control of the root disk
while leaving application volumes under VxVM control.  I had originally
tested this against Solaris 9 sparc running VxVM 4.0, though have updated
it for Solaris 10 x86 running VxVM 5.0MP3.  The commands are still the
same, with the addition of particulars for x86 and handling of Solaris
boot archives introduced in Solaris 10.  Additional notes can be found
at the end.  Sections are labeled 'COMMANDS (detailed)', 'COMMANDS
(quick)
', and NOTES.  Given the complexity involved and multitude of
steps, please read entirely through and understand both the 'COMMANDS
(detailed)' and 'NOTES' sections before attempting to perform this on
any host.  Host details are:
        HOST:           snorkle (x86) 
        SHELL PROMPT:   snorkle [0]
        CDROM PROMPT:   sol10cdrom [0]
        OS:             Solaris 10 u8 x86
        ROOT VOLUMES:   rootvol swapvol var
                        dm        type         plex        dg
        PRIMARY DISK:   c1t0d0s2  auto:sliced  rootdg01    rootdg
        SECONDARY DISK: c1t1d0s2  auto:sliced  rootmirror  rootdg
        x86 BOOTLOADER: grub
The actual work necessary to accomplish this scenario can be see as two
separate parts, 1) removing VxVM control of the root disk, 2) setting
up SVM control of the root disk.  As a disclaimer, part 1 has been
floating around on the net for a while, though typically just a listing
of commands without expected output.  Part 2 should be mostly trivial for
anyone familiar with SVM.  What follows simply brings the two together.

COMMANDS (detailed)

Begin by rebooting the host to single user on a boot cdrom.  This will
allow us to make some necesary modifications to the primary root mirror
slice (c1t0d0s0):
        snorkle [0] /usr/sbin/reboot -- 'cdrom -s'
For x86, you will likely need to specify cdrom boot from the BIOS and
modify the grub boot option appropriately for boot to single user.
Once in single user, we need to check 'c1t0d0s0' for any issues, mount
it, and make backup copies of /etc/vfstab and /etc/system:
        sol10cdrom [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s0
        ** /dev/rdsk/c1t0d0s0
        ** Last Mounted on /
        ** Phase 1 - Check Blocks and Sizes
        ** Phase 2 - Check Pathnames
        ** Phase 3a - Check Connectivity
        ** Phase 3b - Verify Shadows/ACLs
        ** Phase 4 - Check Reference Counts
        ** Phase 5 - Check Cylinder Groups
        157286 files, 4379686 used, 3878825 free (5385 frags, 484180 blocks, 0.1% fragmentation)
        sol10cdrom [0]  [ ! -d /a ] && /usr/bin/mkdir /a
        sol10cdrom [0] /usr/sbin/mount /dev/dsk/c1t0d0s0 /a
        sol10cdrom [0] /usr/bin/cp /a/etc/system /a/etc/system.orig
        sol10cdrom [0] /usr/bin/cp /a/etc/vfstab /a/etc/vfstab.orig
The following lines in /a/etc/system need to be commented out or removed:
        rootdev:/pseudo/vxio@0:0
        set vxio:vol_rootdev_is_volume=1
We can comment these line out via:
        sol10cdrom [0] /usr/bin/sed -e 's;rootdev:/pseudo/vxio@0:0;*rootdev:/pseudo/vxio@0:0;' /a/etc/system.orig > /a/etc/system
        sol10cdrom [0] /usr/bin/cp /a/etc/system /a/etc/system.orig1
        sol10cdrom [0] /usr/bin/sed -e 's;set vxio:vol_rootdev_is_volume=1;*set vxio:vol_rootdev_is_volume=1;' /a/etc/system.orig1 > /a/etc/system
Next, we need to update the current 'vfstab' to this:
        sol10cdrom [0] /usr/bin/cat /a/etc/vfstab
        #device         device          mount           FS      fsck    mount   mount
        #to mount       to fsck         point           type    pass    at boot options
        #
        fd      -       /dev/fd fd      -       no      -
        /proc   -       /proc   proc    -       no      -
        #/dev/vx/dsk/bootdg/swapvol     -       -       swap    -       no      -
        #/dev/vx/dsk/bootdg/rootvol     /dev/vx/rdsk/bootdg/rootvol     /       ufs 1       no      -
        #/dev/vx/dsk/bootdg/var /dev/vx/rdsk/bootdg/var /var    ufs     1       no -
        /dev/dsk/c1t0d0s1       -       -       swap    -       no      -
        /dev/dsk/c1t0d0s0       /dev/rdsk/c1t0d0s0      /       ufs     1       no -
        /dev/dsk/c1t0d0s3       /dev/rdsk/c1t0d0s3      /var    ufs     1       no -
        /devices        -       /devices        devfs   -       no      -
        sharefs -       /etc/dfs/sharetab       sharefs -       no      -
        ctfs    -       /system/contract        ctfs    -       no      -
        objfs   -       /system/object  objfs   -       no      -
        swap    -       /tmp    tmpfs   -       yes     -
        #vshare -       /mnt/vshare     vboxfs  -       no      -
        #/dev/vx/dsk/mydg/myvol /dev/vx/rdsk/mydg/myvol /mnt/myvol      vxfs     1 no      -
        #NOTE: volume rootvol (/) encapsulated partition c1t0d0s0
        #NOTE: volume swapvol (swap) encapsulated partition c1t0d0s1
        #NOTE: volume var (/var) encapsulated partition c1t0d0s3
In the above, all VxVM volumes have been commented out and traditional
ctds values have been setup for the root volumes.  After this, we need
prevent VxVM from starting at boot time and attempting to resync its
root mirrors.  To do so, create file 'install-db' and remove 'root-done'
(which tells VxVM that the root disk is encapsulated).  For Solaris 10
hosts, the boot archive must also be updated since we made changes to
'/a/etc/system'.  Finally reboot to single user on the root disk (c1t0d0):
        sol10cdrom [0] /usr/bin/touch /a/etc/vx/reconfig.d/state.d/install-db
        sol10cdrom [0] /usr/bin/rm /a/etc/vx/reconfig.d/state.d/root-done
        sol10cdrom [0] /usr/sbin/bootadm update-archive -R /a
        sol10cdrom [0] /usr/sbin/reboot -- -s
After the reboot, we can see that the root disk is now configured with
ctds values rather than VxVM volumes:
        snorkle [0] /usr/sbin/df -h
        Filesystem             size   used  avail capacity  Mounted on
        /dev/dsk/c1t0d0s0      7.9G   4.2G   3.6G    54%    /
        /devices                 0K     0K     0K     0%    /devices
        ctfs                     0K     0K     0K     0%    /system/contract
        proc                     0K     0K     0K     0%    /proc
        mnttab                   0K     0K     0K     0%    /etc/mnttab
        swap                   1.8G   932K   1.8G     1%    /etc/svc/volatile
        objfs                    0K     0K     0K     0%    /system/object
        sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
        fd                       0K     0K     0K     0%    /dev/fd
        /dev/dsk/c1t0d0s3      1.1G   407M   633M    40%    /var
        swap                   1.8G     0K   1.8G     0%    /tmp
        swap                   1.8G     4K   1.8G     1%    /var/run
It will be necessary to start VxVM manually so that the root volumes can
be removed and the root (c1t0d0) and mirror (c1t1d0) disks can be freed
from VxVM control.  To do so:
        snorkle [0] /usr/bin/rm /etc/vx/reconfig.d/state.d/install-db
        snorkle [0] /usr/sbin/vxiod set 10
        snorkle [0] /usr/sbin/vxconfigd -m disable
        snorkle [0] /usr/sbin/vxdctl enable
        snorkle [0] /usr/sbin/vxdisk list
        DEVICE       TYPE            DISK         GROUP        STATUS
        c1t0d0s2     auto:sliced     rootdg01     rootdg       online
        c1t1d0s2     auto:sliced     rootmirror   rootdg       online   
        c1t2d0s2     auto:simple     mydg01       mydg         online   
        c1t3d0s2     auto:simple     mydg02       mydg         online
Now we can remove the root volumes from VxVM and remove the DG:
        snorkle [0] /usr/sbin/vxedit -g rootdg -rf rm rootvol
        snorkle [0] /usr/sbin/vxedit -g rootdg -rf rm swapvol   
        snorkle [0] /usr/sbin/vxedit -g rootdg -rf rm var
        snorkle [0] /usr/sbin/vxdg -g rootdg rmdisk rootmirror
        snorkle [0] /usr/sbin/vxdg -g rootdg rmdisk rootdg01    
        VxVM vxdg ERROR V-5-1-552 Disk rootdg01 is used by one or more subdisks.
                Use -k to remove device assignment.
        snorkle [16] /usr/sbin/vxdg destroy rootdg
        snorkle [0] /usr/sbin/vxdisk list
        DEVICE       TYPE            DISK         GROUP        STATUS
        c1t0d0s2     auto:sliced     -            -            online
        c1t1d0s2     auto:sliced     -            -            online   
        c1t2d0s2     auto:simple     mydg01       mydg         online
        c1t3d0s2     auto:simple     mydg02       mydg         online
VxVM gives up an error above as you cannot remove the last disk in a
DG without destroying the DG, which we do in the subsequent 'destroy'
command.  Our root disk and mirror freed of VxVM.  If other VxVM volumes
still exist and are to be retained, they can be started and mounted
to ensure they are still functioning.  (Remember to uncomment them in
/etc/vfstab first):
        snorkle [0] /usr/sbin/vxvol -g mydg startall
        snorkle [0] /usr/sbin/mount /mnt/myvol
        snorkle [0] /usr/bin/df -h | /usr/bin/grep vx
        /dev/vx/dsk/mydg/myvol   894M   8.1M   832M     1%    /mnt/myvol
The final thing to do before getting into SVM is have VxVM restore the
original VTOC of 'c1t0d0'.  The VTOC for 'ct10d0' is presented both
before and after the restoration:
        snorkle [0] /usr/sbin/prtvtoc -h /dev/rdsk/c1t0d0s2
               0      2    00      16065  16787925  16803989   /
               1      3    01   16884315   2233035  19117349
               2      5    00          0  21430710  21430709
               3      7    00   19117350   2313360  21430709   /var
               4     15    01   16803990     80325  16884314
               7     14    01      16065  21414645  21430709
               8      1    01          0     16065     16064
        snorkle [0] /etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/c1t0d0/vtoc /dev/rdsk/c1t0d0s2
        # THE ORIGINAL PARTITIONING IS AS FOLLOWS :   
        #SLICE     TAG  FLAGS    START     SIZE
         0         0x2  0x200    16065 16787925
         1         0x3  0x201 16884315  2233035
         2         0x5  0x200        0 21430710
         3         0x7  0x200 19117350  2313360
         4         0xf  0x201 16803990    80325
         5         0x0  0x200        0        0
         6         0x0  0x200        0        0
         7         0xe  0x201    16065 21414645
         8         0x1  0x201        0    16065
         9         0x0  0x201        0        0
        10         0x0  0x201        0        0
        11         0x0  0x201        0        0
        12         0x0  0x201        0        0
        13         0x0  0x201        0        0
        14         0x0  0x201        0        0
        15         0x0  0x201        0        0
        # THE NEW PARTITIONING WILL BE AS FOLLOWS :
        #SLICE     TAG  FLAGS    START     SIZE
         0         0x2  0x200    16065 16787925
         1         0x3  0x201 16803990  2313360
         2         0x5  0x200        0 21430710
         3         0x7  0x200 19117350  2313360
         4         0x0  0x200        0        0
         5         0x0  0x200        0        0
         6         0x0  0x200        0        0
         7         0x0  0x200        0        0
         8         0x1  0x201        0    16065
         9         0x0  0x201        0        0
        10         0x0  0x201        0        0
        11         0x0  0x201        0        0
        12         0x0  0x201        0        0
        13         0x0  0x201        0        0
        14         0x0  0x201        0        0
        15         0x0  0x201        0        0
        DO YOU WANT TO WRITE THIS TO THE DISK ? [Y/N] :y
        WRITING THE NEW VTOC TO THE DISK
        snorkle [0] /usr/sbin/prtvtoc -h /dev/rdsk/c1t0d0s2
               0      2    00      16065  16787925  16803989   /
               1      3    01   16803990   2313360  19117349
               2      5    00          0  21430710  21430709
               3      7    00   19117350   2313360  21430709   /var
               8      1    01          0     16065     16064
As it stands, VxVM control of the root disk has been completed removed.
If this was our only intention, we could stop here and reboot to
multiuser.  Since we are setting up SVM, we need to update /etc/system
so that SVM can run with only a single plex of a mirror:
        snorkle [0] echo "set md:mirrored_root_flag=1" >> /etc/system
Additionally, since this is an x86 host, we need to restore grub back
to its pre-VxVM state.  When VxVM was setup, it created a backup of
the original menu.lst so we move it back into place.  With this being
a Solaris 10 host, we also need to update the boot-archive prior to a
reboot into single user from the cdrom.  To prevent VxVM from starting,
we also create file 'install-db' again:
        snorkle [0] /usr/bin/mv /boot/grub/menu.lst /boot/grub/menu.lst.vxvm
        snorkle [0] /usr/bin/mv /boot/grub/menu.lst.pre_vxvm /boot/grub/menu.lst
        snorkle [0] /usr/sbin/bootadm update-archive -v
            changed /etc/system
        cannot find: /etc/cluster/nodeid: No such file or directory
        cannot find: /etc/devices/mdi_ib_cache: No such file or directory
        updating /platform/i86pc/boot_archive
        snorkle [0] /usr/bin/touch /etc/vx/reconfig.d/state.d/install-db
        snorkle [0] /usr/sbin/reboot  -- 'cdrom -s'
Remember, while the above 'reboot' command will suffice for sparc,
in x86 you may have to tell the BIOS to boot from cdrom and manage to
single user from there.

Now to the part that seems a little scary, using 'format' we will modify
the current VTOC of 'c1t0d0' (our root disk) to create space for SVM's
metadbs.  To do this, like VxVM before us, we will steal the space from
'swap'.  Below is the output from 'verify' in '/usr/sbin/format' on
'c1t0d0' prior to our modification:
        Part      Tag    Flag     Cylinders        Size            Blocks
          0       root    wm       1 - 1045        8.01GB    (1045/0/0) 16787925
          1       swap    wu    1046 - 1189        1.10GB    (144/0/0)   2313360
          2     backup    wm       0 - 1333       10.22GB    (1334/0/0) 21430710
          3        var    wm    1190 - 1333        1.10GB    (144/0/0)   2313360
          4 unassigned    wm       0               0         (0/0/0)           0
          5 unassigned    wm       0               0         (0/0/0)           0
          6 unassigned    wm       0               0         (0/0/0)           0
          7 unassigned    wm       0               0         (0/0/0)           0
          8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
          9 unassigned    wu       0               0         (0/0/0)           0
The following is the partition table after modification of slices 1, 4,
and 6:
        Part      Tag    Flag     Cylinders        Size            Blocks
          0       root    wm       1 - 1045        8.01GB    (1045/0/0) 16787925
          1       swap    wu    1050 - 1189        1.07GB    (140/0/0)   2249100
          2     backup    wm       0 - 1333       10.22GB    (1334/0/0) 21430710
          3        var    wm    1190 - 1333        1.10GB    (144/0/0)   2313360
          4 unassigned    wm       0               0         (0/0/0)           0
          5 unassigned    wm    1046 - 1047       15.69MB    (2/0/0)       32130
          6 unassigned    wm    1048 - 1049       15.69MB    (2/0/0)       32130
          7 unassigned    wm       0               0         (0/0/0)           0
          8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
          9 unassigned    wu       0               0         (0/0/0)           0
In the above, we simply took 4 cylinders from the beginning of swap and
used them to setup slices 5 and 6.  Special care should be taken while
doing the above as you otherwise risk serious data loss (see note 4).
After this, use 'fsck' to verify the root disk is sane and reboot back
into single user from the root disk:
        sol10cdrom [0] /usr/sbin/fsck -n /dev/rdsk/c1t0d0s0
        ** /dev/rdsk/c1t0d0s0 (NO WRITE)
        ** Last Mounted on /
        ** Phase 1 - Check Blocks and Sizes
        ** Phase 2 - Check Pathnames
        ** Phase 3a - Check Connectivity
        ** Phase 3b - Verify Shadow/ACLs 
        ** Phase 4 - Check Reference Counts
        ** Phase 5 - Check Cylinder Groups
        157420 files, 4379707 used, 3878804 free (5404 frags, 484175 blocks, 0.1% fragmentation)
        sol10cdrom [0] /usr/sbin/reboot -- -s
The output below is simply verification that Solaris didn't freak out
on our change of the VTOC after the reboot:
        snorkle [0] /usr/sbin/df -h
        Filesystem             size   used  avail capacity  Mounted on
        /dev/dsk/c1t0d0s0      7.9G   4.2G   3.6G    54%    /
        /devices                 0K     0K     0K     0%    /devices
        ctfs                     0K     0K     0K     0%    /system/contract
        proc                     0K     0K     0K     0%    /proc
        mnttab                   0K     0K     0K     0%    /etc/mnttab
        swap                   1.8G   932K   1.8G     1%    /etc/svc/volatile
        objfs                    0K     0K     0K     0%    /system/object
        sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
        fd                       0K     0K     0K     0%    /dev/fd
        /dev/dsk/c1t0d0s3      1.1G   407M   633M    40%    /var
        swap                   1.8G     0K   1.8G     0%    /tmp
        swap                   1.8G     4K   1.8G     1%    /var/run
        snorkle [0] /usr/sbin/prtvtoc -h /dev/rdsk/c1t0d0s2
               0      2    00      16065  16787925  16803989   /
               1      3    01   16868250   2249100  19117349
               2      5    00          0  21430710  21430709
               3      7    00   19117350   2313360  21430709   /var
               5      0    00   16803990     32130  16836119
               6      0    00   16836120     32130  16868249
               8      1    01          0     16065     16064
Now, update the partition table of c1t1d0 based upon that of c1t0d0,
create metadbs on both c1t0d0 and c1t1d0, and setup the initial SVM
configuration of the root disk:
        snorkle [0] /usr/sbin/prtvtoc -h /dev/rdsk/c1t0d0s2 | /usr/sbin/fmthard -s - /dev/rdsk/c1t1d0s2
        fmthard:  New volume table of contents now in place.
        snorkle [0] /usr/sbin/prtvtoc -h /dev/rdsk/c1t1d0s2
               0      2    00      16065  16787925  16803989
               1      3    01   16868250   2249100  19117349
               2      5    00          0  21430710  21430709
               3      7    00   19117350   2313360  21430709
               5      0    00   16803990     32130  16836119
               6      0    00   16836120     32130  16868249
               8      1    01          0     16065     16064
        snorkle [0] /usr/sbin/metadb -a -c3 -f c1t0d0s5 c1t0d0s6 c1t1d0s5 c1t1d0s6
        snorkle [0] /usr/sbin/metainit -f d0 1 1 c1t0d0s0
        d0: Concat/Stripe is setup
        snorkle [0] /usr/sbin/metainit d2 -m d0
        d2: Mirror is setup
        snorkle [0] /usr/sbin/metainit -f d3 1 1 c1t0d0s1
        d3: Concat/Stripe is setup
        snorkle [0] /usr/sbin/metainit d5 -m d3
        d5: Mirror is setup
        snorkle [0] /usr/sbin/metainit -f d6 1 1 c1t0d0s3
        d6: Concat/Stripe is setup
        snorkle [0] /usr/sbin/metainit d8 -m d6
        d8: Mirror is setup
        snorkle [0] /usr/sbin/metaroot d2
We need to update the 'vfstab' from this:
        snorkle [0] /usr/bin/cat /etc/vfstab
        #device         device          mount           FS      fsck    mount   mount
        #to mount       to fsck         point           type    pass    at boot options
        #
        fd      -       /dev/fd fd      -       no      -
        /proc   -       /proc   proc    -       no      -
        #/dev/vx/dsk/bootdg/swapvol     -       -       swap    -       no      -
        #/dev/vx/dsk/bootdg/rootvol     /dev/vx/rdsk/bootdg/rootvol     /       ufs     1       no      -
        #/dev/vx/dsk/bootdg/var /dev/vx/rdsk/bootdg/var /var    ufs     1       no      -
        /dev/dsk/c1t0d0s1       -       -       swap    -       no      -
        /dev/md/dsk/d2  /dev/md/rdsk/d2 /       ufs     1       no      -
        /dev/dsk/c1t0d0s3       /dev/rdsk/c1t0d0s3      /var    ufs     1       no      -
        /devices        -       /devices        devfs   -       no      -
        sharefs -       /etc/dfs/sharetab       sharefs -       no      -
        ctfs    -       /system/contract        ctfs    -       no      -
        objfs   -       /system/object  objfs   -       no      -
        swap    -       /tmp    tmpfs   -       yes     -
        #vshare -       /mnt/vshare     vboxfs  -       no      -
        /dev/vx/dsk/mydg/myvol  /dev/vx/rdsk/mydg/myvol /mnt/myvol      vxfs     1       no      -
        #NOTE: volume rootvol (/) encapsulated partition c1t0d0s0
        #NOTE: volume swapvol (swap) encapsulated partition c1t0d0s1
        #NOTE: volume var (/var) encapsulated partition c1t0d0s3
To this, removing the unnecessary VxVM root volumes and configuring the
other SVM devices:
        snorkle [0] /usr/bin/cat /etc/vfstab
        #device         device          mount           FS      fsck    mount   mount
        #to mount       to fsck         point           type    pass    at boot options
        #
        fd      -       /dev/fd fd      -       no      -
        /proc   -       /proc   proc    -       no      -
        /dev/md/dsk/d5  -       -       swap    -       no      -
        /dev/md/dsk/d2  /dev/md/rdsk/d2 /       ufs     1       no      -
        /dev/md/dsk/d8  /dev/md/rdsk/d8 /var    ufs     1       no      -
        /devices        -       /devices        devfs   -       no      -
        sharefs -       /etc/dfs/sharetab       sharefs -       no      -
        ctfs    -       /system/contract        ctfs    -       no      -
        objfs   -       /system/object  objfs   -       no      -
        swap    -       /tmp    tmpfs   -       yes     -
        #vshare -       /mnt/vshare     vboxfs  -       no      -
        /dev/vx/dsk/mydg/myvol  /dev/vx/rdsk/mydg/myvol /mnt/myvol      vxfs     1       no      -
        #NOTE: volume d2 (/) svm c1t[0|1]d0s0
        #NOTE: volume d5 (swap) svm c1t[0|1]d0s1
        #NOTE: volume d8 (/var) svm c1t[0|1]d0s3
To allow VxVM to restart after reboot, remove 'install-db'.  Again,
being this is Solaris 10, update the boot archive.  Finally reboot
into multiuser:
        snorkle [0] /usr/bin/rm /etc/vx/reconfig.d/state.d/install-db
        snorkle [0] /usr/sbin/bootadm update-archive -v
            changed /etc/system
        cannot find: /etc/cluster/nodeid: No such file or directory
        cannot find: /etc/devices/mdi_ib_cache: No such file or directory
        updating /platform/i86pc/boot_archive
        snorkle [0] /usr/sbin/reboot
After the reboot, I've done some verification work to ensure that our
metadbs are good, that we are now booted off of SVM metadevices, and
our VxVM volume is online.
        snorkle [0] /usr/sbin/metadb -i
                flags           first blk       block count
             a m  p  luo        16              8192            /dev/dsk/c1t0d0s5
             a    p  luo        8208            8192            /dev/dsk/c1t0d0s5
             a    p  luo        16400           8192            /dev/dsk/c1t0d0s5
             a    p  luo        16              8192            /dev/dsk/c1t0d0s6
             a    p  luo        8208            8192            /dev/dsk/c1t0d0s6
             a    p  luo        16400           8192            /dev/dsk/c1t0d0s6
             a    p  luo        16              8192            /dev/dsk/c1t1d0s5
             a    p  luo        8208            8192            /dev/dsk/c1t1d0s5
             a    p  luo        16400           8192            /dev/dsk/c1t1d0s5
             a    p  luo        16              8192            /dev/dsk/c1t1d0s6
             a    p  luo        8208            8192            /dev/dsk/c1t1d0s6
             a    p  luo        16400           8192            /dev/dsk/c1t1d0s6
         r - replica does not have device relocation information
         o - replica active prior to last mddb configuration change     
         u - replica is up to date
         l - locator for this replica was read successfully     
         c - replica's location was in /etc/lvm/mddb.cf 
         p - replica's location was patched in kernel
         m - replica is master, this is replica selected as input
         W - replica has device write errors
         a - replica is active, commits are occurring to this replica
         M - replica had problem with master blocks
         D - replica had problem with data blocks
         F - replica had format problems
         S - replica is too small to hold current data base
         R - replica had device read errors
        snorkle [0] /usr/sbin/df -h
        Filesystem             size   used  avail capacity  Mounted on
        /dev/md/dsk/d2         7.9G   4.2G   3.6G    54%    /
        /devices                 0K     0K     0K     0%    /devices
        ctfs                     0K     0K     0K     0%    /system/contract
        proc                     0K     0K     0K     0%    /proc
        mnttab                   0K     0K     0K     0%    /etc/mnttab
        swap                   1.6G   1.0M   1.6G     1%    /etc/svc/volatile
        objfs                    0K     0K     0K     0%    /system/object
        sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
        fd                       0K     0K     0K     0%    /dev/fd
        /dev/md/dsk/d8         1.1G   407M   633M    40%    /var
        swap                   1.6G     0K   1.6G     0%    /tmp
        swap                   1.6G    12K   1.6G     1%    /var/run
        swap                   1.6G     0K   1.6G     0%    /dev/vx/dmp
        swap                   1.6G     0K   1.6G     0%    /dev/vx/rdmp
        /dev/vx/dsk/mydg/myvol
                               894M   8.1M   832M     1%    /mnt/myvol
The only thing left to do for SVM is to add in 'c1t1d0' to the SVM
configuration for the root disk.  The following will attach c1t1d0 to
the mirror and automatically synchronizing the mirror in the background:
        snorkle [0] /usr/sbin/metainit d1 1 1 c1t1d0s0
        d1: Concat/Stripe is setup
        snorkle [0] /usr/sbin/metattach d2 d1
        d2: submirror d1 is attached
        snorkle [0] /usr/sbin/metainit d4 1 1 c1t1d0s1
        d4: Concat/Stripe is setup
        snorkle [0] /usr/sbin/metattach d5 d4
        d5: submirror d4 is attached
        snorkle [0] /usr/sbin/metainit d7 1 1 c1t1d0s3
        d7: Concat/Stripe is setup
        snorkle [0] /usr/sbin/metattach d8 d7
        d8: submirror d7 is attached
To view the current sync progress, use 'metastat':
        snorkle [0] /usr/sbin/metastat | /usr/bin/egrep '^d|State:|in progress'
        d8: Mirror
              State: Okay
              State: Resyncing
            Resync in progress: 31 % done
        d6: Submirror of d8
            State: Okay
        d7: Submirror of d8
            State: Resyncing
        d5: Mirror
              State: Okay
              State: Resyncing
            Resync in progress: 44 % done
        d3: Submirror of d5
            State: Okay
        d4: Submirror of d5
            State: Resyncing
        d2: Mirror
              State: Okay
              State: Resyncing
            Resync in progress: 7 % done
        d0: Submirror of d2
            State: Okay          
        d1: Submirror of d2
            State: Resyncing    
In effort to be thorough, we install the boot block to the mirror disk
and configure grub for the alternate boot path:
        snorkle [0] /usr/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t1d0s0
        stage1 written to partition 0 sector 0 (abs 16065)  
        stage2 written to partition 0, 272 sectors starting at 50 (abs 16115)
        snorkle [0] /usr/bin/cat << EOF >> /boot/grub/menu.lst
        > title Solaris 10 10/09 root mirror
        > root (hd1,0,a)
        > kernel /platform/i86pc/multiboot -mverbose
        > module /platform/i86pc/boot_archive
        > EOF
        snorkle [0] /usr/sbin/bootadm list-menu
        The location for the active GRUB menu is: /boot/grub/menu.lst
        default 0
        timeout 10
        0 Solaris 10 10/09 s10x_u8wos_08a X86
        1 Solaris failsafe
        2 Solaris 10 10/09 root mirror
Alternatively, for sparc:
        sparc [0] /usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0
        sparc [0] /usr/sbin/eeprom use-nvramrc\?=true
        sparc [0] /usr/bin/ls -ld /dev/rdsk/c1t0d0s0 /dev/rdsk/c1t1d0s0 | /usr/bin/awk '{print $9, $11}'
        /dev/rdsk/c1t0d0s0 ../../devices/pci@1c,600000/scsi@2/sd@0,0:a,raw
        /dev/rdsk/c1t1d0s0 ../../devices/pci@1c,600000/scsi@2/sd@1,0:a,raw
        sparc [0] /usr/sbin/eeprom "nvramrc=devalias rootdisk /pci@1c,600000/scsi@2/sd@0,0:a devalias rootmirror /pci@1c,600000/scsi@2/sd@1,0:a"
        sparc [0] /usr/sbin/eeprom "boot-device=rootdisk rootmirror"
Lastly, setup the dump device for crash dumps:
        snorkle [0] /usr/sbin/dumpadm -d /dev/md/dsk/d5
              Dump content: kernel pages
               Dump device: /dev/md/dsk/d5 (swap)
        Savecore directory: /var/crash/snorkle
          Savecore enabled: yes

COMMANDS (quick)
        snorkle [0] /usr/sbin/reboot -- 'cdrom -s'
        sol10cdrom [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s0
        sol10cdrom [0]  [ ! -d /a ] && /usr/bin/mkdir /a
        sol10cdrom [0] /usr/sbin/mount /dev/dsk/c1t0d0s0 /a
        sol10cdrom [0] /usr/bin/cp /a/etc/system /a/etc/system.orig
        sol10cdrom [0] /usr/bin/cp /a/etc/vfstab /a/etc/vfstab.orig
        sol10cdrom [0] /usr/bin/sed -e 's;rootdev:/pseudo/vxio@0:0;*rootdev:/pseudo/vxio@0:0;' /a/etc/system.orig > /a/etc/system
        sol10cdrom [0] /usr/bin/cp /a/etc/system /a/etc/system.orig1
        sol10cdrom [0] /usr/bin/sed -e 's;set vxio:vol_rootdev_is_volume=1;*set vxio:vol_rootdev_is_volume=1;' /a/etc/system.orig1 > /a/etc/system
    # update the current 'vfstab' to this:
        sol10cdrom [0] /usr/bin/cat /a/etc/vfstab
        #device         device          mount           FS      fsck    mount   mount
        #to mount       to fsck         point           type    pass    at boot options
        #
        fd      -       /dev/fd fd      -       no      -
        /proc   -       /proc   proc    -       no      -
        #/dev/vx/dsk/bootdg/swapvol     -       -       swap    -       no      -
        #/dev/vx/dsk/bootdg/rootvol     /dev/vx/rdsk/bootdg/rootvol     /       ufs 1       no      -
        #/dev/vx/dsk/bootdg/var /dev/vx/rdsk/bootdg/var /var    ufs     1       no -
        /dev/dsk/c1t0d0s1       -       -       swap    -       no      -
        /dev/dsk/c1t0d0s0       /dev/rdsk/c1t0d0s0      /       ufs     1       no -
        /dev/dsk/c1t0d0s3       /dev/rdsk/c1t0d0s3      /var    ufs     1       no -
        /devices        -       /devices        devfs   -       no      -
        sharefs -       /etc/dfs/sharetab       sharefs -       no      -
        ctfs    -       /system/contract        ctfs    -       no      -
        objfs   -       /system/object  objfs   -       no      -
        swap    -       /tmp    tmpfs   -       yes     -
        #vshare -       /mnt/vshare     vboxfs  -       no      -
        #/dev/vx/dsk/mydg/myvol /dev/vx/rdsk/mydg/myvol /mnt/myvol      vxfs     1 no      -
        #NOTE: volume rootvol (/) encapsulated partition c1t0d0s0
        #NOTE: volume swapvol (swap) encapsulated partition c1t0d0s1
        #NOTE: volume var (/var) encapsulated partition c1t0d0s3
        sol10cdrom [0] /usr/bin/touch /a/etc/vx/reconfig.d/state.d/install-db
        sol10cdrom [0] /usr/bin/rm /a/etc/vx/reconfig.d/state.d/root-done
        sol10cdrom [0] /usr/sbin/bootadm update-archive -R /a
        sol10cdrom [0] /usr/sbin/reboot -- -s
        snorkle [0] /usr/sbin/df -h
        snorkle [0] /usr/bin/rm /etc/vx/reconfig.d/state.d/install-db
        snorkle [0] /usr/sbin/vxiod set 10
        snorkle [0] /usr/sbin/vxconfigd -m disable
        snorkle [0] /usr/sbin/vxdctl enable
        snorkle [0] /usr/sbin/vxedit -g rootdg -rf rm rootvol
        snorkle [0] /usr/sbin/vxedit -g rootdg -rf rm swapvol
        snorkle [0] /usr/sbin/vxedit -g rootdg -rf rm var
        snorkle [0] /usr/sbin/vxdg -g rootdg rmdisk rootmirror
        snorkle [0] /usr/sbin/vxdg destroy rootdg
        snorkle [0] /usr/sbin/vxvol -g mydg startall
        snorkle [0] /usr/sbin/mount /mnt/myvol
        snorkle [0] /etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/c1t0d0/vtoc /dev/rdsk/c1t0d0s2
        DO YOU WANT TO WRITE THIS TO THE DISK ? [Y/N] :y
        WRITING THE NEW VTOC TO THE DISK
        snorkle [0] echo "set md:mirrored_root_flag=1" >> /etc/system
        snorkle [0] /usr/bin/mv /boot/grub/menu.lst /boot/grub/menu.lst.vxvm
        snorkle [0] /usr/bin/mv /boot/grub/menu.lst.pre_vxvm /boot/grub/menu.lst
        snorkle [0] /usr/sbin/bootadm update-archive -v
        snorkle [0] /usr/bin/touch /etc/vx/reconfig.d/state.d/install-db
        snorkle [0] /usr/sbin/reboot  -- 'cdrom -s'
    # output of 'format c1t0d0' -> verify pre modification
        Part      Tag    Flag     Cylinders        Size            Blocks
          0       root    wm       1 - 1045        8.01GB    (1045/0/0) 16787925
          1       swap    wu    1046 - 1189        1.10GB    (144/0/0)   2313360
          2     backup    wm       0 - 1333       10.22GB    (1334/0/0) 21430710
          3        var    wm    1190 - 1333        1.10GB    (144/0/0)   2313360
          4 unassigned    wm       0               0         (0/0/0)           0
          5 unassigned    wm       0               0         (0/0/0)           0
          6 unassigned    wm       0               0         (0/0/0)           0
          7 unassigned    wm       0               0         (0/0/0)           0
          8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
          9 unassigned    wu       0               0         (0/0/0)           0
    # output of 'format c1t0d0' -> verify post modification
        Part      Tag    Flag     Cylinders        Size            Blocks
          0       root    wm       1 - 1045        8.01GB    (1045/0/0) 16787925
          1       swap    wu    1050 - 1189        1.07GB    (140/0/0)   2249100
          2     backup    wm       0 - 1333       10.22GB    (1334/0/0) 21430710
          3        var    wm    1190 - 1333        1.10GB    (144/0/0)   2313360
          4 unassigned    wm       0               0         (0/0/0)           0
          5 unassigned    wm    1046 - 1047       15.69MB    (2/0/0)       32130
          6 unassigned    wm    1048 - 1049       15.69MB    (2/0/0)       32130
          7 unassigned    wm       0               0         (0/0/0)           0
          8       boot    wu       0 -    0        7.84MB    (1/0/0)       16065
          9 unassigned    wu       0               0         (0/0/0)           0
        sol10cdrom [0] /usr/sbin/fsck -n /dev/rdsk/c1t0d0s0
        sol10cdrom [0] /usr/sbin/reboot -- -s   
        snorkle [0] /usr/sbin/prtvtoc -h /dev/rdsk/c1t0d0s2 | /usr/sbin/fmthard -s - /dev/rdsk/c1t1d0s2
        snorkle [0] /usr/sbin/metadb -a -c3 -f c1t0d0s5 c1t0d0s6 c1t1d0s5 c1t1d0s6
        snorkle [0] /usr/sbin/metainit -f d0 1 1 c1t0d0s0
        snorkle [0] /usr/sbin/metainit d2 -m d0
        snorkle [0] /usr/sbin/metainit -f d3 1 1 c1t0d0s1
        snorkle [0] /usr/sbin/metainit d5 -m d3
        snorkle [0] /usr/sbin/metainit -f d6 1 1 c1t0d0s3
        snorkle [0] /usr/sbin/metainit d8 -m d6
        snorkle [0] /usr/sbin/metaroot d2
    # update vfstab to this
        snorkle [0] /usr/bin/cat /etc/vfstab
        #device         device          mount           FS      fsck    mount   mount
        #to mount       to fsck         point           type    pass    at boot options
        #
        fd      -       /dev/fd fd      -       no      -
        /proc   -       /proc   proc    -       no      -
        /dev/md/dsk/d5  -       -       swap    -       no      -
        /dev/md/dsk/d2  /dev/md/rdsk/d2 /       ufs     1       no      -
        /dev/md/dsk/d8  /dev/md/rdsk/d8 /var    ufs     1       no      -
        /devices        -       /devices        devfs   -       no      -
        sharefs -       /etc/dfs/sharetab       sharefs -       no      -
        ctfs    -       /system/contract        ctfs    -       no      -
        objfs   -       /system/object  objfs   -       no      -
        swap    -       /tmp    tmpfs   -       yes     -
        #vshare -       /mnt/vshare     vboxfs  -       no      -
        /dev/vx/dsk/mydg/myvol  /dev/vx/rdsk/mydg/myvol /mnt/myvol      vxfs     1       no      -
        #NOTE: volume d2 (/) svm c1t[0|1]d0s0
        #NOTE: volume d5 (swap) svm c1t[0|1]d0s1
        #NOTE: volume d8 (/var) svm c1t[0|1]d0s3
        snorkle [0] /usr/bin/rm /etc/vx/reconfig.d/state.d/install-db
        snorkle [0] /usr/sbin/bootadm update-archive -v
        snorkle [0] /usr/sbin/reboot
        snorkle [0] /usr/sbin/metainit d1 1 1 c1t1d0s0
        snorkle [0] /usr/sbin/metattach d2 d1
        snorkle [0] /usr/sbin/metainit d4 1 1 c1t1d0s1
        snorkle [0] /usr/sbin/metattach d5 d4
        snorkle [0] /usr/sbin/metainit d7 1 1 c1t1d0s3
        snorkle [0] /usr/sbin/metattach d8 d7
        snorkle [0] /usr/sbin/metastat | /usr/bin/egrep '^d|State:|in progress'
        snorkle [0] /usr/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t1d0s0
        snorkle [0] /usr/bin/cat << EOF >> /boot/grub/menu.lst
        > title Solaris 10 10/09 root mirror
        > root (hd1,0,a)
        > kernel /platform/i86pc/multiboot -mverbose
        > module /platform/i86pc/boot_archive
        > EOF
        snorkle [0] /usr/sbin/bootadm list-menu
    # for sparc hosts:   
        sparc [0] /usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0
        sparc [0] /usr/sbin/eeprom use-nvramrc\?=true
        sparc [0] /usr/bin/ls -ld /dev/rdsk/c1t0d0s0 /dev/rdsk/c1t1d0s0 | /usr/bin/awk '{print $9, $11}'
        /dev/rdsk/c1t0d0s0 ../../devices/pci@1c,600000/scsi@2/sd@0,0:a,raw
        /dev/rdsk/c1t1d0s0 ../../devices/pci@1c,600000/scsi@2/sd@1,0:a,raw
        sparc [0] /usr/sbin/eeprom "nvramrc=devalias rootdisk /pci@1c,600000/scsi@2/sd@0,0:a devalias rootmirror /pci@1c,600000/scsi@2/sd@1,0:a"
        sparc [0] /usr/sbin/eeprom "boot-device=rootdisk rootmirror"
    # don't forget the dump device:
        snorkle [0] /usr/sbin/dumpadm -d /dev/md/dsk/d5


NOTES

1) The above procedure will NOT work if VxVM is less than version 4.0
   and still needs to maintain other volumes.  This is due to a VxVM
   requirement (in prior versions) on the existence of a 'rootdg'.

2) It is assumed that the root disk is still encapsulated while under
   VxVM control, having not been moved to an initialized disk.  The above
   will only work with an encapsulated disk.

3) The above will definitely incur downtime as the majority of the work
   is performed from single user as well as booted single user from
   the cdrom.  The amount of downtime will vary depending upon a
   variety of factors such as size of the volumes, hardware platform
   and capabilities, etc.

4) This step requires deliberate modification of the partition table
   of the root disk, which is also the reason that it requires booting
   from an alternate device such as the cdrom.  Care must be taken in
   this step.  Detailed is the adjustment of 'swap' space to create two
   new slices.  This works because there are no FS constructs setup for
   'swap'.  To modify any other existing slice would most likely result
   in data loss and FS corruption.  Of all the steps presented this one is
   probably the most likely to cause problems if not performed correctly.
   A good understanding of disk partitions and structures would be useful.
   As an aside, you cannot realistically perform this steps if there are
   no free slices available.  This step also assumes all free space on the
   disk has already been allocated.  If free space is available, use it
   rather than mucking with 'swap'.

2 comments:

Anonymous said...

Good God Troy. You didn't actually do this to something at work did you?

troy said...

Yeah, actually I did. After reading your comment, however, I read back through the post and noticed several errors. Due to those errors and the problems they could cause, I've rewritten the post correcting them and updating for Solaris 10 and x86. Though lengthy, yes, I did use this for work without problems. In the rewrite, I ran through every command to verify function and expectation. Thanks for the prompting to read through it again.