INTRO
The information detailed below is to assist with proactive planning for
the potential demise and recovery of a host back to the state it was
in prior to demise. Circumstances this will likely be of benefit wouldThe information detailed below is to assist with proactive planning for
the potential demise and recovery of a host back to the state it was
include the patching of a host, application upgrades by a vendor, trying
new and untested configurations, etc. The value involved is that should
something go catastrophically wrong after the "updates," the issue can be
immediately resolved simply by rebooting and recovering to an "untainted"
(boot) device wherein the system appears as though no changes were ever
made. Though specifically for handling of root disks which are mirrored
with Solaris Volume Manager (SVM, formerly Solstice Disk Suite), the
procedure below could also be tweaked and used with non-root disks.
Caveat, one should be familiar with Solaris, RAID 1, Solaris bootup, and
SVM before attempting the steps detailed below.
HOST INFO
Update Task: apply a patch cluster
Patches Path: /export/9_Recommended
Host: adler
Shell Prompt: adler [0]
OS: Solaris 9
Kernel Revision: Generic_118558-35
Updated Kernel Rev: Generic_122300-30 (after patching)
Architecture: SPARC
Primary Disk: c1t0d0s0 (set as disk0 and disk in OBP)
Secondary Disk: c1t1d0s0 (set as disk1 in OBP)
Rootdev: rootdev:/pseudo/md@0:0,2,blk
OBP boot-device: 'disk0 disk1'
Primary Disk Devpath: /devices/pci@1c,600000/scsi@2/sd@0,0:a
Secondary Disk Devpath: /devices/pci@1c,600000/scsi@2/sd@1,0:a
Primary Disk OBP Path: /pci@1c,600000/scsi@2/disk@0,0
Secondary Disk OBP Path: /pci@1c,600000/scsi@2/disk@1,0
Mounted Filesystems (FS):
adler [0] /usr/bin/df -h | /usr/bin/grep dsk
/dev/md/dsk/d2 4.9G 119M 4.8G 3% /
/dev/md/dsk/d11 4.9G 1.6G 3.3G 33% /usr
/dev/md/dsk/d8 3.9G 287M 3.6G 8% /var
/dev/md/dsk/d14 50G 921M 48G 2% /export
Current Metadevices:
adler [0] /usr/sbin/metastat -p
d17 -m d15 d16 1
d15 1 1 c1t2d0s0
d16 1 1 c1t3d0s0
d11 -m d9 d10 1
d9 1 1 c1t0d0s6
d10 1 1 c1t1d0s6
d8 -m d6 d7 1
d6 1 1 c1t0d0s5
d7 1 1 c1t1d0s5
d5 -m d3 d4 1
d3 1 1 c1t0d0s1
d4 1 1 c1t1d0s1
d2 -m d0 d1 1
d0 1 1 c1t0d0s0
d1 1 1 c1t1d0s0
d14 -m d12 d13 1
d12 1 1 c1t0d0s7
d13 1 1 c1t1d0s7
Current Metadbs Devices:
adler [0] /usr/sbin/metadb -i | /usr/bin/awk '{print $NF}'
[snip...]
/dev/dsk/c1t0d0s3
/dev/dsk/c1t0d0s4
/dev/dsk/c1t1d0s3
/dev/dsk/c1t1d0s4
[snip...]
Current /etc/vfstab:
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d5 - - swap - no -
/dev/md/dsk/d2 /dev/md/rdsk/d2 / ufs 1 no -
/dev/md/dsk/d11 /dev/md/rdsk/d11 /usr ufs 1 no -
/dev/md/dsk/d8 /dev/md/rdsk/d8 /var ufs 1 no -
/dev/md/dsk/d14 /dev/md/rdsk/d14 /export ufs 2 yes -
swap - /tmp tmpfs - yes -
** NOTE: The following procedure is applicable and has been
tested on Solaris versions 8 - 10 and has been
used on SPARC based systems. Also, the mirror
at conclusion will be structured the same as the
original configuration.
** WARNING: If /etc/system contains the uncommented line:
md:mirrored_root_flag=1
we can run the system with a single side of a mirror.
If the above line does not exist in /etc/system,
it needs to be added and the system rebooted before
proceeding!
COMMANDS (explained)
Before breaking our mirror, ensure that both root mirrors are bootable:
adler [0] /usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0
adler [0] /usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0
Now, flush out any FS locks on all filesystems, committing FS writes to
disk, and deliberately break the root mirror to free disk0 from SVM
control:
adler [0] /usr/sbin/lockfs -af
adler [0] /usr/sbin/metadetach d2 d0
d2: submirror d0 is detached
adler [0] /usr/sbin/metadetach d5 d3
d5: submirror d3 is detached
adler [0] /usr/sbin/metadetach d8 d6
d8: submirror d6 is detached
adler [0] /usr/sbin/metadetach d11 d9
d11: submirror d9 is detached
adler [0] /usr/sbin/metadetach d14 d12
d14: submirror d12 is detached
adler [0] /usr/sbin/metaclear d0
d0: Concat/Stripe is cleared
adler [0] /usr/sbin/metaclear d3
d3: Concat/Stripe is cleared
adler [0] /usr/sbin/metaclear d6
d6: Concat/Stripe is cleared
adler [0] /usr/sbin/metaclear d9
d9: Concat/Stripe is cleared
adler [0] /usr/sbin/metaclear d12
d12: Concat/Stripe is cleared
Each slice on disk0 needs to be checked to verify it is usable.
Assuming all slices are clean, mount the slices to TMPDIR/FS
(/var would be mounted as /a/var):
adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s0
** /dev/rdsk/c1t0d0s0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cylinder Groups
5761 files, 116508 used, 5044194 free (5090 frags, 629888 blocks, 0.1% fragmentation)
adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s5
** /dev/rdsk/c1t0d0s5
[snip...]
adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s6
** /dev/rdsk/c1t0d0s6
[snip...]
adler [0] /usr/sbin/fsck -y /dev/rdsk/c1t0d0s7
** /dev/rdsk/c1t0d0s7
[snip...]
adler [0] [ ! -d /a ] && /usr/bin/mkdir /a
adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s0 /a
adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s5 /a/var
adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s6 /a/usr
adler [0] /usr/sbin/mount /dev/dsk/c1t0d0s7 /a/export
So that we can cleanly boot from disk0 without SVM, backup
TMPDIR/etc/vfstab to retain the original and modify TMPDIR/etc/vfstab
to use 'c.t.d.s.' values:
adler [0] /usr/bin/cp /a/etc/vfstab /a/etc/vfstab.orig
adler [0] /usr/bin/cat << EOF > /a/etc/vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /usr ufs 1 no -
/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /var ufs 1 no -
/dev/dsk/c1t0d0s7 /dev/rdsk/c1t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -
EOF
Next, we need to copy TMPDIR/etc/system to retain the original and modify
it, commenting out the SVM rootdev:
adler [0] /usr/bin/cp /a/etc/system /a/etc/system.orig
adler [0] /usr/bin/sed -e 's;rootdev:/pseudo/md@0:0,2,blk;*rootdev:/pseudo/md@0:0,2,blk;' /a/etc/system.orig > /a/etc/system
Our example update is to patch the host with a Solaris patch cluster,
though could easily be any nature of update. Move to the patch staging
directory and install the patch cluster to disk0 mounted to TMPDIR.
After patching has completed, umount the disk0 and reboot. Since we are
modifying disk0, the system should reboot using the default device of
'disk', which disk0 is aliased to, thus giving us a root disk outside
of SVM control:
adler [0] cd /export/9_Recommended
adler [0] ./install_cluster -R /a
Patch cluster install script for Solaris 9 Recommended Patch Cluster
*WARNING* SYSTEMS WITH LIMITED DISK SPACE SHOULD *NOT* INSTALL PATCHES:
With or without using the save option, the patch installation process
[snip...]
Use '/usr/bin/showrev -p' to verify installed patch-ids.
Refer to individual patch README files for more patch detail.
Rebooting the system is usually necessary after installation.
adler [0] /usr/sbin/umount /a/export
adler [0] /usr/sbin/umount /a/usr
adler [0] /usr/sbin/umount /a/var
adler [0] /usr/sbin/umount /a
adler [0] /usr/sbin/reboot
** NOTE: At this point, after the reboot, the system can be verified,
tested, and run using the new updates. Following the remainder
of the steps below would allow the updates to be retained and
mirroring reconfigured. Should things not go as planned with
disk0, the system can be rebooted to disk1. This allows quick
recovery of the volumes and data back to their original state
prior to the updates. If this is necessary, recovering the
stability of the mirror is as simple as re-initializing and
re-attaching the disk0 slices back to their original SVM
mirror devices.
Assuming the box rebooted cleanly and we intend to keep the updates,
remove all SVM state DBs used for the rootmirror and recreate them:
adler [0] /usr/sbin/metadb -d c1t0d0s3 c1t0d0s4 c1t1d0s3 c1t1d0s4
adler [0] /usr/sbin/metadb -a -c3 -f c1t0d0s3 c1t0d0s4 c1t1d0s3 c1t1d0s4
Clear all SVM rootmirror objects, effectively those still associated
with disk1. This is to finish cleaning up and removing the original
SVM configuration:
adler [0] /usr/sbin/metaclear d2
d2: Mirror is cleared
adler [0] /usr/sbin/metaclear d5
d5: Mirror is cleared
adler [0] /usr/sbin/metaclear d8
d8: Mirror is cleared
adler [0] /usr/sbin/metaclear d11
d11: Mirror is cleared
adler [0] /usr/sbin/metaclear d14
d14: Mirror is cleared
adler [0] /usr/sbin/metaclear d1
d1: Mirror is cleared
adler [0] /usr/sbin/metaclear d4
d4: Mirror is cleared
adler [0] /usr/sbin/metaclear d7
d7: Mirror is cleared
adler [0] /usr/sbin/metaclear d10
d10: Mirror is cleared
adler [0] /usr/sbin/metaclear d13
d13: Mirror is cleared
After the original rootmirror objects have been removed, we can
re-initialize SVM control on disk0 by recreating the rootmirror
SVM objects:
adler [0] /usr/sbin/metainit -f d0 1 1 c1t0d0s0
d0: Concat/Stripe is setup
adler [0] /usr/sbin/metainit d2 -m d0
d2: Mirror is setup
adler [0] /usr/sbin/metainit -f d3 1 1 c1t0d0s1
d3: Concat/Stripe is setup
adler [0] /usr/sbin/metainit d5 -m d3
d5: Mirror is setup
adler [0] /usr/sbin/metainit -f d6 1 1 c1t0d0s5
d6: Concat/Stripe is setup
adler [0] /usr/sbin/metainit d8 -m d6
d8: Mirror is setup
adler [0] /usr/sbin/metainit -f d9 1 1 c1t0d0s6
d9: Concat/Stripe is setup
adler [0] /usr/sbin/metainit d11 -m d9
d11: Mirror is setup
adler [0] /usr/sbin/metainit -f d12 1 1 c1t0d0s7
d12: Concat/Stripe is setup
adler [0] /usr/sbin/metainit d14 -m d12
d14: Mirror is setup
adler [0] /usr/sbin/metaroot d2
The above 'metaroot' command will setup the 'rootdev' entry in /etc/system
and update the / slice to be a metadevice in /etc/vfstab. It will
not, however, update the other slices in vfstab to be metadevices.
Since earlier we backed up /etc/vfstab to /etc/vfstab.orig and we are
recreating the original SVM configuration, we can simply copy the original
vfstab over the existing one:
adler [0] /usr/bin/cp /etc/vfstab.orig /etc/vfstab
adler [0] /usr/bin/cat /etc/vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d5 - - swap - no -
/dev/md/dsk/d2 /dev/md/rdsk/d2 / ufs 1 no -
/dev/md/dsk/d11 /dev/md/rdsk/d11 /usr ufs 1 no -
/dev/md/dsk/d8 /dev/md/rdsk/d8 /var ufs 1 no -
/dev/md/dsk/d14 /dev/md/rdsk/d14 /export ufs 2 yes -
swap - /tmp tmpfs - yes -
A subsequent reboot allows the system to boot cleanly to the new SVM
rootdev on disk0 and use our freshly created metadevices:
adler [0] /usr/sbin/reboot
After the reboot, we need to add disk1 back into the rootmirror, allowing
it to sync in the background:
adler [0] /usr/sbin/metainit d1 1 1 c1t1d0s0
d1: Concat/Stripe is setup
adler [0] /usr/sbin/metattach d2 d1
d2: submirror d1 is attached
adler [0] /usr/sbin/metainit d4 1 1 c1t1d0s1
d4: Concat/Stripe is setup
adler [0] /usr/sbin/metattach d5 d4
d5: submirror d4 is attached
adler [0] /usr/sbin/metainit d7 1 1 c1t1d0s5
d7: Concat/Stripe is setup
adler [0] /usr/sbin/metattach d8 d7
d8: submirror d7 is attached
adler [0] /usr/sbin/metainit d10 1 1 c1t1d0s6
d10: Concat/Stripe is setup
adler [0] /usr/sbin/metattach d11 d10
d11: submirror d10 is attached
adler [0] /usr/sbin/metainit d13 1 1 c1t1d0s7
d13: Concat/Stripe is setup
adler [0] /usr/sbin/metattach d14 d13
d14: submirror d13 is attached
At this point, the mirrors are in the process of resyncing and our is
work complete. To view the progress of the resync, metastat can be used:
adler [0] metastat
d2: Mirror
Submirror 0: d0
State: Okay
Submirror 1: d1
State: Resyncing
Resync in progress: 0 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 10491456 blocks (5.0 GB)
d0: Submirror of d2
State: Okay
Size: 10491456 blocks (5.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes
d1: Submirror of d2
State: Resyncing
Size: 10491456 blocks (5.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes
[snip...]
see also:
Breaking and Syncing a Hardware Root Mirror -- Solaris
Breaking and Syncing an MD Root Mirror
