18 October 2010

Breaking and Syncing a SVM Root Mirror (Solaris)

INTRO

The information detailed below is to assist with proactive planning for
the potential demise and recovery of a host back to the state it was
in prior to demise.  Circumstances this will likely be of benefit would
include the patching of a host, application upgrades by a vendor, trying
new and untested configurations, etc.  The value involved is that should
something go catastrophically wrong after the "updates," the issue can be
immediately resolved simply by rebooting and recovering to an "untainted"
(boot) device wherein the system appears as though no changes were ever
made.  Though specifically for handling of root disks which are mirrored
with Solaris Volume Manager (SVM, formerly Solstice Disk Suite), the
procedure below could also be tweaked and used with non-root disks.
Caveat, one should be familiar with Solaris, RAID 1, Solaris bootup, and
SVM before attempting the steps detailed below.

HOST INFO

        Update Task:            apply a patch cluster
        Patches Path:           /export/9_Recommended
        Host:                   adler
        Shell Prompt:           adler [0]
        OS:                     Solaris 9
        Kernel Revision:        Generic_118558-35
        Updated Kernel Rev:     Generic_122300-30       (after patching)
        Architecture:           SPARC
        Primary Disk:           c1t0d0s0        (set as disk0 and disk in OBP)
        Secondary Disk:         c1t1d0s0        (set as disk1 in OBP)
        Rootdev:                rootdev:/pseudo/md@0:0,2,blk
        OBP boot-device:        'disk0 disk1'
        Primary Disk Devpath:   /devices/pci@1c,600000/scsi@2/sd@0,0:a
        Secondary Disk Devpath: /devices/pci@1c,600000/scsi@2/sd@1,0:a
                Primary Disk OBP Path:          /pci@1c,600000/scsi@2/disk@0,0
                Secondary Disk OBP Path:        /pci@1c,600000/scsi@2/disk@1,0

        Mounted Filesystems (FS):
                adler [0] /usr/bin/df -h | /usr/bin/grep dsk
                /dev/md/dsk/d2         4.9G   119M   4.8G     3%    /
                /dev/md/dsk/d11        4.9G   1.6G   3.3G    33%    /usr
                /dev/md/dsk/d8         3.9G   287M   3.6G     8%    /var
                /dev/md/dsk/d14         50G   921M    48G     2%    /export

        Current Metadevices:
                adler [0] /usr/sbin/metastat -p
                d17 -m d15 d16 1
                d15 1 1 c1t2d0s0
                d16 1 1 c1t3d0s0
                d11 -m d9 d10 1
                d9 1 1 c1t0d0s6
                d10 1 1 c1t1d0s6
                d8 -m d6 d7 1
                d6 1 1 c1t0d0s5
                d7 1 1 c1t1d0s5
                d5 -m d3 d4 1
                d3 1 1 c1t0d0s1
                d4 1 1 c1t1d0s1
                d2 -m d0 d1 1
                d0 1 1 c1t0d0s0
                d1 1 1 c1t1d0s0
                d14 -m d12 d13 1
                d12 1 1 c1t0d0s7
                d13 1 1 c1t1d0s7

        Current Metadbs Devices:
                adler [0] /usr/sbin/metadb -i | /usr/bin/awk '{print $NF}'
                [snip...]
                /dev/dsk/c1t0d0s3
                /dev/dsk/c1t0d0s4
                /dev/dsk/c1t1d0s3
                /dev/dsk/c1t1d0s4
                [snip...]

        Current /etc/vfstab:
                #device         device            mount         FS      fsck    mount   mount
                #to mount       to fsck           point         type    pass    at boot options
                #
                fd              -                 /dev/fd       fd      -       no      -
                /proc           -                 /proc         proc    -       no      -
                /dev/md/dsk/d5  -                 -             swap    -       no      -
                /dev/md/dsk/d2  /dev/md/rdsk/d2   /             ufs     1       no      -
                /dev/md/dsk/d11 /dev/md/rdsk/d11  /usr          ufs     1       no      -
                /dev/md/dsk/d8  /dev/md/rdsk/d8   /var          ufs     1       no      -
                /dev/md/dsk/d14 /dev/md/rdsk/d14  /export       ufs     2       yes     -
                swap            -                 /tmp          tmpfs   -       yes     -


        ** NOTE:     The following procedure is applicable and has been
                     tested on Solaris versions 8 - 10 and has been
                     used on SPARC based systems.  Also, the mirror
                     at conclusion will be structured the same as the
                     original configuration.

        ** WARNING:  If /etc/system contains the uncommented line:

                        md:mirrored_root_flag=1

                     we can run the system with a single side of a mirror.
                     If the above line does not exist in /etc/system,
                     it needs to be added and the system rebooted before
                     proceeding!


COMMANDS (explained)

Before breaking our mirror, ensure that both root mirrors are bootable:

        adler [0] /usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0
        adler [0] /usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0

Now, flush out any FS locks on all filesystems, committing FS writes to
disk, and deliberately break the root mirror to free disk0 from SVM
control:

        adler [0] /usr/sbin/lockfs -af
        adler [0] /usr/sbin/metadetach d2 d0
        d2: submirror d0 is detached
        adler [0] /usr/sbin/metadetach d5 d3
        d5: submirror d3 is detached
        adler [0] /usr/sbin/metadetach d8 d6
        d8: submirror d6 is detached
        adler [0] /usr/sbin/metadetach d11 d9
        d11: submirror d9 is detached
        adler [0] /usr/sbin/metadetach d14 d12
        d14: submirror d12 is detached
        adler [0] /usr/sbin/metaclear d0
        d0: Concat/Stripe is cleared
        adler [0] /usr/sbin/metaclear d3
        d3: Concat/Stripe is cleared
        adler [0] /usr/sbin/metaclear d6
        d6: Concat/Stripe is cleared
        adler [0] /usr/sbin/metaclear d9
        d9: Concat/Stripe is cleared
        adler [0] /usr/sbin/metaclear d12
        d12: Concat/Stripe is cleared

Each slice on disk0 needs to be checked to verify it is usable.
Assuming all slices are clean, mount the slices to TMPDIR/FS
(/var would be mounted as /a/var):

        adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s0
        ** /dev/rdsk/c1t0d0s0
        ** Last Mounted on /
        ** Phase 1 - Check Blocks and Sizes
        ** Phase 2 - Check Pathnames
        ** Phase 3a - Check Connectivity
        ** Phase 3b - Verify Shadows/ACLs
        ** Phase 4 - Check Reference Counts
        ** Phase 5 - Check Cylinder Groups
        5761 files, 116508 used, 5044194 free (5090 frags, 629888 blocks, 0.1% fragmentation)
        adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s5
        ** /dev/rdsk/c1t0d0s5
        [snip...]
        adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s6
        ** /dev/rdsk/c1t0d0s6
        [snip...]
        adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s7
        ** /dev/rdsk/c1t0d0s7
        [snip...]
        adler [0] [ ! -d /a ] && /usr/bin/mkdir /a
        adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s0 /a
        adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s5 /a/var
        adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s6 /a/usr
        adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s7 /a/export

So that we can cleanly boot from disk0 without SVM, backup
TMPDIR/etc/vfstab to retain the original and modify TMPDIR/etc/vfstab
to use 'c.t.d.s.' values:

        adler [0] /usr/bin/cp /a/etc/vfstab /a/etc/vfstab.orig
        adler [0] /usr/bin/cat << EOF > /a/etc/vfstab
#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/dsk/c1t0d0s1  -       -       swap    -       no      -
/dev/dsk/c1t0d0s0  /dev/rdsk/c1t0d0s0 /       ufs     1       no      -
/dev/dsk/c1t0d0s6  /dev/rdsk/c1t0d0s6 /usr    ufs     1       no      -
/dev/dsk/c1t0d0s5  /dev/rdsk/c1t0d0s5 /var    ufs     1       no      -
/dev/dsk/c1t0d0s7  /dev/rdsk/c1t0d0s7 /export    ufs     2       yes      -
swap    -       /tmp    tmpfs   -       yes     -
EOF

Next, we need to copy TMPDIR/etc/system to retain the original and modify
it, commenting out the SVM rootdev:

        adler [0] /usr/bin/cp /a/etc/system /a/etc/system.orig
        adler [0] /usr/bin/sed -e 's;rootdev:/pseudo/md@0:0,2,blk;*rootdev:/pseudo/md@0:0,2,blk;' /a/etc/system.orig > /a/etc/system

Our example update is to patch the host with a Solaris patch cluster,
though could easily be any nature of update.  Move to the patch staging
directory and install the patch cluster to disk0 mounted to TMPDIR.
After patching has completed, umount the disk0 and reboot.  Since we are
modifying disk0, the system should reboot using the default device of
'disk', which disk0 is aliased to, thus giving us a root disk outside
of SVM control:

        adler [0] cd /export/9_Recommended
        adler [0] ./install_cluster -R /a

        Patch cluster install script for Solaris 9 Recommended Patch Cluster


        *WARNING* SYSTEMS WITH LIMITED DISK SPACE SHOULD *NOT* INSTALL PATCHES:
        With or without using the save option, the patch installation process
        [snip...]
        Use '/usr/bin/showrev -p' to verify installed patch-ids.
        Refer to individual patch README files for more patch detail.
        Rebooting the system is usually necessary after installation.
        adler [0] /usr/sbin/umount /a/export
        adler [0] /usr/sbin/umount /a/usr
        adler [0] /usr/sbin/umount /a/var
        adler [0] /usr/sbin/umount /a
        adler [0] /usr/sbin/reboot

** NOTE:  At this point, after the reboot, the system can be verified,
          tested, and run using the new updates.  Following the remainder
          of the steps below would allow the updates to be retained and
          mirroring reconfigured.  Should things not go as planned with
          disk0, the system can be rebooted to disk1.  This allows quick
          recovery of the volumes and data back to their original state
          prior to the updates.  If this is necessary, recovering the
          stability of the mirror is as simple as re-initializing and
          re-attaching the disk0 slices back to their original SVM
          mirror devices.

Assuming the box rebooted cleanly and we intend to keep the updates,
remove all SVM state DBs used for the rootmirror and recreate them:

        adler [0] /usr/sbin/metadb -d c1t0d0s3 c1t0d0s4 c1t1d0s3 c1t1d0s4
        adler [0] /usr/sbin/metadb -a -c3 -f c1t0d0s3 c1t0d0s4 c1t1d0s3 c1t1d0s4

Clear all SVM rootmirror objects, effectively those still associated
with disk1.  This is to finish cleaning up and removing the original
SVM configuration:

        adler [0] /usr/sbin/metaclear d2
        d2: Mirror is cleared
        adler [0] /usr/sbin/metaclear d5
        d5: Mirror is cleared
        adler [0] /usr/sbin/metaclear d8
        d8: Mirror is cleared
        adler [0] /usr/sbin/metaclear d11
        d11: Mirror is cleared
        adler [0] /usr/sbin/metaclear d14
        d14: Mirror is cleared
        adler [0] /usr/sbin/metaclear d1
        d1: Mirror is cleared
        adler [0] /usr/sbin/metaclear d4
        d4: Mirror is cleared
        adler [0] /usr/sbin/metaclear d7
        d7: Mirror is cleared
        adler [0] /usr/sbin/metaclear d10
        d10: Mirror is cleared
        adler [0] /usr/sbin/metaclear d13
        d13: Mirror is cleared

After the original rootmirror objects have been removed, we can
re-initialize SVM control on disk0 by recreating the rootmirror
SVM objects:

        adler [0] /usr/sbin/metainit -f d0 1 1 c1t0d0s0
        d0: Concat/Stripe is setup
        adler [0] /usr/sbin/metainit d2 -m d0
        d2: Mirror is setup
        adler [0] /usr/sbin/metainit -f d3 1 1 c1t0d0s1
        d3: Concat/Stripe is setup
        adler [0] /usr/sbin/metainit d5 -m d3
        d5: Mirror is setup
        adler [0] /usr/sbin/metainit -f d6 1 1 c1t0d0s5
        d6: Concat/Stripe is setup
        adler [0] /usr/sbin/metainit d8 -m d6
        d8: Mirror is setup
        adler [0] /usr/sbin/metainit -f d9 1 1 c1t0d0s6
        d9: Concat/Stripe is setup
        adler [0] /usr/sbin/metainit d11 -m d9
        d11: Mirror is setup
        adler [0] /usr/sbin/metainit -f d12 1 1 c1t0d0s7
        d12: Concat/Stripe is setup
        adler [0] /usr/sbin/metainit d14 -m d12
        d14: Mirror is setup
        adler [0] /usr/sbin/metaroot d2

The above 'metaroot' command will setup the 'rootdev' entry in /etc/system
and update the / slice to be a metadevice in /etc/vfstab.  It will
not, however, update the other slices in vfstab to be metadevices.
Since earlier we backed up /etc/vfstab to /etc/vfstab.orig and we are
recreating the original SVM configuration, we can simply copy the original
vfstab over the existing one:

        adler [0] /usr/bin/cp /etc/vfstab.orig /etc/vfstab
        adler [0] /usr/bin/cat /etc/vfstab
        #device         device            mount         FS      fsck    mount   mount
        #to mount       to fsck           point         type    pass    at boot options
        #
        fd              -                 /dev/fd       fd      -       no      -
        /proc           -                 /proc         proc    -       no      -
        /dev/md/dsk/d5  -                 -             swap    -       no      -
        /dev/md/dsk/d2  /dev/md/rdsk/d2   /             ufs     1       no      -
        /dev/md/dsk/d11 /dev/md/rdsk/d11  /usr          ufs     1       no      -
        /dev/md/dsk/d8  /dev/md/rdsk/d8   /var          ufs     1       no      -
        /dev/md/dsk/d14 /dev/md/rdsk/d14  /export       ufs     2       yes     -
        swap            -                 /tmp          tmpfs   -       yes     -

A subsequent reboot allows the system to boot cleanly to the new SVM
rootdev on disk0 and use our freshly created metadevices:

        adler [0] /usr/sbin/reboot

After the reboot, we need to add disk1 back into the rootmirror, allowing
it to sync in the background:

        adler [0] /usr/sbin/metainit d1 1 1 c1t1d0s0
        d1: Concat/Stripe is setup
        adler [0] /usr/sbin/metattach d2 d1
        d2: submirror d1 is attached
        adler [0] /usr/sbin/metainit d4 1 1 c1t1d0s1
        d4: Concat/Stripe is setup
        adler [0] /usr/sbin/metattach d5 d4
        d5: submirror d4 is attached
        adler [0] /usr/sbin/metainit d7 1 1 c1t1d0s5
        d7: Concat/Stripe is setup
        adler [0] /usr/sbin/metattach d8 d7
        d8: submirror d7 is attached
        adler [0] /usr/sbin/metainit d10 1 1 c1t1d0s6
        d10: Concat/Stripe is setup
        adler [0] /usr/sbin/metattach d11 d10
        d11: submirror d10 is attached
        adler [0] /usr/sbin/metainit d13 1 1 c1t1d0s7
        d13: Concat/Stripe is setup
        adler [0] /usr/sbin/metattach d14 d13
        d14: submirror d13 is attached

At this point, the mirrors are in the process of resyncing and our is
work complete.  To view the progress of the resync, metastat can be used:

        adler [0] metastat
        d2: Mirror
            Submirror 0: d0
              State: Okay
            Submirror 1: d1
              State: Resyncing
            Resync in progress: 0 % done
            Pass: 1
            Read option: roundrobin (default)
            Write option: parallel (default)
            Size: 10491456 blocks (5.0 GB)

        d0: Submirror of d2
            State: Okay
            Size: 10491456 blocks (5.0 GB)
            Stripe 0:
                Device     Start Block  Dbase        State Reloc Hot Spare
                c1t0d0s0          0     No            Okay   Yes


        d1: Submirror of d2
            State: Resyncing
            Size: 10491456 blocks (5.0 GB)
            Stripe 0:
                Device     Start Block  Dbase        State Reloc Hot Spare
                c1t1d0s0          0     No            Okay   Yes
        [snip...]


see also:
    Breaking and Syncing a Hardware Root Mirror -- Solaris
    Breaking and Syncing an MD Root Mirror