17 January 2013

Repartitioning Contiguous Space in Linux

As a follow up to the same under Solaris, I thought I'd detail
repartitioning in-use, contiguous space in Linux. To recap the
situational setup, a filesystem (FS) has run out of space and the data
contained on it couldn't be removed or compressed to free up space.
Since the next adjacent partition is configured as swap space, the plan
is to remove that partition and grow our fully consumed FS into space
reclaimed from the original swap partition while leaving our configured
FS intact. The host details for our scenario are:
        HOST:           europa
        PROMPTS:        multi user:     europa [0]
                        single user:    europa-cons [0]
        OS:             CentOS 6.2 (RHEL variant)
        FS TYPE:        ext4
        DISK:           sdb
        NOTE:           I've also used the following with no issues on
                        previous versions of CentOS (and RHEL) with ext3
                        filesystems as well.
Before continuing, a similar resolution to this scenario could have
been to simply extend the volume using LVM. Since our volumes on this
disk are not under LVM control, this is not an easily accessible option,
thus reclaiming the adjacent partition was chosen instead.

Starting things off, our application resides solely on disk "sdb" on
partitions "1" and "3", mounted under "/opt/myapp". There is also a swap
partition at "sdb2". Below, our 'df' output shows the FS in question on
"sdb1" (at 100% capacity), our swap volumes, and the partition table on
"sdb" as reported by 'fdisk':
        europa [0] /bin/df -h
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.5G  4.2G  27% /
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M   97M  861M  11% /var
        /dev/sdb1             510M  510M     0 100% /opt/myapp/logs
        /dev/sdb3             510M  8.9M  475M   2% /opt/myapp/bin
        europa [0] /sbin/swapon -s
        Filename                                Type            Size    Used    Priority
        /dev/sda2                               partition       1048568 0       -1
        /dev/sdb2                               partition       530136  0       -2
        europa [0] /sbin/fdisk -l /dev/sdb

        Disk /dev/sdb: 2147 MB, 2147483648 bytes
        255 heads, 63 sectors/track, 261 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disk identifier: 0x9ae7b0fc

           Device Boot      Start         End      Blocks   Id  System
        /dev/sdb1               1          66      530113+  83  Linux
        /dev/sdb2              67         132      530145   82  Linux swap / Solaris
        /dev/sdb3             133         198      530145   83  Linux
        europa [0]
Since "sdb2" is swap space, and in this case, we have additional space at
the end of the disk, we are going to repartition this disk. The result
will be that partitions "sdb1" and "sdb2" will be removed and "sdb1"
recreated (starting at the same cylinder) but this time, encompassing
the space previously held by "sdb2". Next, "sdb2" will be recreated,
this time using the available space at the end of the disk (due to the
size of this disk, "sdb2 (our swap volume) will be about 23 MB smaller
once reconfigured). To prepare for this, using your favorite editor
(vi?) update the entries for "sdb" in "/etc/fstab" from this:
        europa [0] /bin/grep sdb /etc/fstab
        /dev/sdb2               swap                    swap    defaults        0 0
        /dev/sdb1               /opt/myapp/logs         ext4    defaults        1 2
        /dev/sdb3               /opt/myapp/bin          ext4    defaults        1 2
        europa [0]
to this:
        europa [0] /bin/grep sdb /etc/fstab
        #/dev/sdb2              swap                    swap    defaults        0 0
        #/dev/sdb1              /opt/myapp/logs         ext4    defaults        1 2
        #/dev/sdb3              /opt/myapp/bin          ext4    defaults        1 2
        europa [0]
Once complete, we'll need to reboot into "single user mode". This is
to ensure that the "sdb" FS are not in use and removes "sdb2" from swap
usage. (If you can otherwise stop I/O to the "sdb" FS and remove "sdb2"
from swap usage, you could technically do this in "multi user mode".):
        europa [0] /sbin/reboot

        Broadcast message from root@europa
                (/dev/tty1) at 13:42 ...

        The system is going down for reboot NOW!
        europa [0]
(screen refresh)
        Running guests on default URI: libvirtd not installed; skipping this URI.
        Stopping atd:                                              [  OK  ]
        <snip...>
        Sending all processes the TERM signal...                   [  OK  ]
        Sending all processes the KILL signal...                   [  OK  ]
        Saving random seed:                                        [  OK  ]
        Syncing hardware clock to system time                      [  OK  ]
        Turning off swap:                                          [  OK  ]
        Turning off quotas:                                        [  OK  ]
        Unmounting file systems:                                   [  OK  ]
        init: Re-executing /sbin/init                              [  OK  ]
        Please stand by while rebooting the system...
        md: stopping all md devices.
        sd 3:0:0:0: [sdb] Synchronizing SCSI cache
        sd 2:0:0:0: [sda] Synchronizing SCSI cache
Once the host has rebooted and after the BIOS POSTs, we need to edit
our GRUB boot entry. Select the appropriate entry (on this host, there
is only one) and hit 'e' before the 5 seconds time out so that we can
update the kernel boot parameters:
         GNU GRUB  version 0.97  (639K lower / 1047488K upper memory)

        CentOS 6.2 (2.6.32-220.el6.x86_64


           Use the <up> and <down> keys to select which entry is highlighted.
           Press enter to boot the select OS, 'e' to edit the
           commands before booting, or 'c' for a command-line.


        The highlighted entry will be booted automatically in 5 seconds.
(after entry selection and hitting 'e'):
         GNU GRUB  version 0.97  (639K lower / 1047488K upper memory)

        root (hd0,0)
        kernel /boot/vmlinux-2.6.32-220.el6.x86_64 ro root=UUID=05e46d98-6e82->
        initrd /boot/initramfs-2.6.32-220.el6.x86_64.img

           Use the <up> and <down> keys to select which entry is highlighted.
           Prebb 'b' to boot, 'e' to edit the selected command in the
           boot sequence, 'c' for a command-line, 'o' to open a new line
           after ('O' for before) the selected line, 'd' to remove the
           selected line, or escape to go back to the main menu.
Navigate to the 'kernel' line, hit 'e', and update it from this:
        kernel /boot/vmlinux-2.6.32-220.el6.x86_64 <...> rd_NO_DM
to this:
        kernel /boot/vmlinux-2.6.32-220.el6.x86_64 <...> rd_NO_DM -s
(Since you may have other options set for your kernel, the intent here
is to update the entry to include "-s" at the end so that we boot into
"single user mode".) After hitting 'Enter', type 'b' to boot the system,
at which point the screen will refresh and we'll see the boot process:
        <snip...>
        Enabling /etc/fstab swaps:                                 [  OK  ]
        root@europa /]# _
Log in at the prompt above. Once logged in, I've simply reset PS1 for
claritiy of this writeup:
        root@europa /]# PS1=`/bin/hostname -s`'-cons [$?] '
        europa-cons [0]

        europa-cons [0] /usr/bin/who -r
                 run-level S  2013-01-17 13:49
        europa-cons [0] /sbin/runlevel
        N S
        europa-cons [0] /bin/df -h
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.5G  4.2G  27% /
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M   97M  861M  11% /var
        europa-cons [0] /sbin/swapon -s
        Filename                                Type            Size    Used    Priority
        /dev/sda2                               partition       1048568 0       -1
        europa-cons [0]
Above, I've also checked our current run level via 'who -r' (and via
'runlevel' for those who didn't learn on Solaris). Additionally, I've
run 'df' and 'swapon' to simply verify that "sdb" is no longer in use.
At this point, we get to repartition "sdb". It must be stressed before
we do any repartitioning, either we must evacuate the adjacent partition
of data (if it contains an FS), the adjacent partition was in use but
can be reclaimed, or subsequent contiguous space must exist to our
partition that we're concerned with (sdb1). If this is not the case,
you will likely risk data loss if you proceed any further.

Moving along, I've called 'fdisk /dev/sdb' to operate on that disk and
verified the existing partition table via "p":
        europa-cons [0] /sbin/fdisk /dev/sdb

        WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
                 switch off the mode (command 'c') and change display units to
                 sectors (command 'u').

        Command (m for help): p

        Disk /dev/sdb: 2147 MB, 2147483648 bytes
        255 heads, 63 sectors/track, 261 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disk identifier: 0x9ae7b0fc

           Device Boot      Start         End      Blocks   Id  System
        /dev/sdb1               1          66      530113+  83  Linux
        /dev/sdb2              67         132      530145   82  Linux swap / Solaris
        /dev/sdb3             133         198      530145   83  Linux
Our first order of business is to zero out partition 2 (sdb2, the
swap partition). After that, I've also removed partition 1 (sdb1)
and subsequently recreated it setting the first cylinder to be "1"
(as it was originally). I've also set partition 1 to end on cylinder
"132", just as partition 2 originally did. Finally, I've re-added
our swap space back to partition 2, setting the starting cylinder to
be next available (199) and accepting maximum value available (261),
and set the partition type to 82 (Linux swap / Solaris):
        Command (m for help): d
        Partition number (1-4): 2

        Command (m for help): d
        Partition number (1-4): 1

        Command (m for help): n
        Command action
           e   extended
           p   primary partition (1-4)
        p
        Partition number (1-4): 1
        First cylinder (1-261, default 1): 1
        Last cylinder, +cylinders or +size{K,M,G} (1-132, default 132): 132

        Command (m for help): n
        Command action
           e   extended
           p   primary partition (1-4)
        p
        Partition number (1-4): 2
        First cylinder (199-261, default 199):
        Using default value 199
        Last cylinder, +cylinders or +size{K,M,G} (199-261, default 261): +512M
        Value out of range.
        Last cylinder, +cylinders or +size{K,M,G} (199-261, default 261):
        Using default value 261

        Command (m for help): t
        Partition number (1-4): 2
        Hex code (type L to list codes): 82
        Changed system type of partition 2 to 82 (Linux swap / Solaris)
Of note, I initially tried setting the partion size for partition 2 to
"512M". Since the disk didn't have that much space available, I instead
settled for about 23 MB less and accepted the default end cylinder (261).
Below, I've confirmed our new partition table (p) and labeled the disk
(wrote the partition table to the disk):
        Command (m for help): p

        Disk /dev/sdb: 2147 MB, 2147483648 bytes
        255 heads, 63 sectors/track, 261 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disk identifier: 0x9ae7b0fc

           Device Boot      Start         End      Blocks   Id  System
        /dev/sdb1               1         132     1060258+  83  Linux
        /dev/sdb2             199         261      506047+  82  Linux swap / Solaris
        /dev/sdb3             133         198      530145   83  Linux

        Partition table entries are not in disk order

        Command (m for help): w
        The partition table has been altered!

        Calling ioctl() to re-read partition table.
        Syncing disks.
We can now uncomment our "sdb" related "fstab" entries, so using your
favorite text editor, update "fstab" from this:
        europa [0] /bin/grep sdb /etc/fstab
        #/dev/sdb2              swap                    swap    defaults        0 0
        #/dev/sdb1              /opt/myapp/logs         ext4    defaults        1 2
        #/dev/sdb3              /opt/myapp/bin          ext4    defaults        1 2
to this:
        europa [0] /bin/grep sdb /etc/fstab
        /dev/sdb2              swap                    swap    defaults        0 0
        /dev/sdb1              /opt/myapp/logs         ext4    defaults        1 2
        /dev/sdb3              /opt/myapp/bin          ext4    defaults        1 2
For Linux to use the newly recreated swap space, below, I've run 'mkswap'
against "/dev/sdb2". Also, while their shouldn't be any issue, I've
verified the sanity of our contained FS on "sdb1" using 'fsck':
        europa-cons [0] /sbin/mkswap /dev/sdb2
        Setting up swapspace version 1, size = 506040 KiB
        no label, UUID=c93960d1-aaa3-49de-98b5-eca5803c27ea
        europa-cons [0] /sbin/fsck -y /dev/sdb1
        fsck from util-linux-ng 2.17.2
        e2fsck 1.41.12 (17-May-2010)
        /dev/sdb1: clean, 16729/33200 files, 132528/132528 blocks
If our FS checks out to be clean (which it does), below we 'mount'
the "sdb1" FS to "/opt/myapp/logs", run 'df' to see the existing size
and usage, extend the EXT4 FS on "sdb1" using 'resize2fs', and finally
recheck our size and usage via 'df':
        europa-cons [0] /bin/mount /opt/myapp/logs
        europa-cons [0] /bin/df -h /opt/myapp/logs
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sdb1             510M  510M     0 100% /opt/myapp/logs
        europa-cons [0] /sbin/resize2fs -p /dev/sdb1
        resize2fs 1.41.12 (17-May-2010)
        Filesystem at /dev/sdb1 is mounted on /opt/myapp/logs; on-line resizing required
        old desc_blocks = 1, new_desc_blocks = 1
        Performing an on-line resize of /dev/sdb1 to 265064 (4k) blocks.
        The filesystem on /dev/sdb1 is now 265064 blocks long.

        europa-cons [0] /bin/df -h /opt/myapp/logs
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sdb1            1021M  510M  466M  53% /opt/myapp/logs
        europa-cons [0] /bin/ls /opt/myapp/logs
        account  crash  db     games  local  log         mail  opt    preserve  spool  yp
        cache    cvs    empty  lib    lock   lost+found  nis   other  run       tmp
(The "-p" option to 'resize2fs' will print out percentage of completion
while 'resize2fs' is working, which given the size of our FS, is
irrelevant.) Above, we see in the second 'df' that our FS size has indeed
been extended to consume the space reclaimed from "sdb1". As a follow up,
our files are still accessible as seen in the 'ls'. Below, we 'umount'
"/opt/myapp/logs" and run 'fsck' once more on "sdb1" to verify the FS
sanity as a follow up to our extending the FS with 'resize2fs' above:
        europa-cons [0] /bin/umount /opt/myapp/logs
        europa-cons [0] /sbin/fsck -y /dev/sdb1
        fsck from util-linux-ng 2.17.2
        e2fsck 1.41.12 (17-May-2010)
        /dev/sdb1: clean, 16729/59760 files, 134264/265064 blocks
        europa-cons [0] /sbin/reboot
(screen refresh)
        Sending all processes the TERM signal...                   [  OK  ]
        Sending all processes the KILL signal... init: rsS-sulogin main process (701)\
             killed by KILL signal
                                                                   [  OK  ]
        Saving random seed:                                        [  OK  ]
        Syncing hardware clock to system time                      [  OK  ]
        Turning off swap:                                          [  OK  ]
        Turning off quotas:                                        [  OK  ]
        Unmounting file systems:                                   [  OK  ]
        init: Re-executing /sbin/init                              [  OK  ]
        Please stand by while rebooting the system...
        md: stopping all md devices.
        sd 3:0:0:0: [sdb] Synchronizing SCSI cache
        sd 2:0:0:0: [sda] Synchronizing SCSI cache
Providing our FS is clean, above we reboot the host. This time, we don't
need to modify the GRUB boot entry below, so just let the menu time out
and boot up the host to the default run level (in this case, 3):
         GNU GRUB  version 0.97  (639K lower / 1047488K upper memory)

        CentOS 6.2 (2.6.32-220.el6.x86_64


           Use the <up> and <down> keys to select which entry is highlighted.
           Press enter to boot the select OS, 'e' to edit the
           commands before booting, or 'c' for a command-line.


        The highlighted entry will be booted automatically in 5 seconds.
(screen refresh)
        CentOS release 6.2 (Final)
        Kernel 2.6.32-220.el6.x86_64 on an x86_64

        europa login: _
With our host now back online, we simply verify that "sdb2" is allocated
to swap using 'swapon -s' and our "sdb" FS are mounted:
        europa [0] /sbin/swapon -s
        Filename                                Type            Size    Used    Priority
        /dev/sda2                               partition       1048568 0       -1
        /dev/sdb2                               partition       506036  0       -2
        europa [0] /bin/df -h
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.5G  4.2G  27% /
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M   97M  861M  11% /var
        /dev/sdb1            1021M  510M  466M  53% /opt/myapp/logs
        /dev/sdb3             510M  8.9M  475M   2% /opt/myapp/bin
        europa [0]
Our work is now complete. For the curious, the reason we can re-layout
the partition table in this manner is that a partition can be treated
as simply a container for the contained FS. Providing that you are not
reducing the size of the partition (container) or shifting the start/end
of the slice boundaries into space occupied by an FS, your FS is not
otherwise impacted and remains sane.



see also:
    Repartitioning Contiguous Space in Solaris

No comments: