16 August 2012

Remounting Part of an FS Elsewhere

The following details loopback file system (FS) mounts. These allow one to effectively mount a portion of an FS elsewhere, providing access to the underlying sub-tree(s) via alternate pathnames wherein a symlink may not be appropriate. Our host details are:

        HOSTS:          europa (Linux)
                        berkeley (FreeBSD)
                        sunspot (Solaris)
        OSes:           CentOS 6.2 (Red Hat Enterprise Linux Clone)
                        FreeBSD 8.2
                        Solaris 10
        MOUNTED FS:     /usr/local/prodapp
        LOOPBACK MOUNT: /var/log/prodapp
        NOTE:           Unless details differ between OSes, "europa"
                        will be used in the examples.
The short answer to how to setup a loopback mount is to simply use the appropriate mount options:
        europa [0] /bin/mount --bind /usr/local/prodapp/logs /var/log/prodapp
        berkeley [0] /sbin/mount -t nullfs /usr/local/prodapp/logs /var/log/prodapp
        sunspot [0] /usr/sbin/mount -F lofs /usr/local/prodapp/logs /var/log/prodapp
For further details on why, how, and expectations, I've setup a scenario wherein "/var" is undersized for the amount of data logged by application "prodapp". For our currently mounted file systems, we have:
        europa [0] /bin/df -h
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.6G  4.1G  27% /
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M  598M  360M  63% /var
        /dev/sdb1             2.0G   36M  1.9G   2% /usr/local/prodapp
A quick check under "/var" identifies the contents of "/var/log/prodapp" as our consumer:
        berkeley [0] /usr/bin/du -sk /var/* | /usr/bin/sort -n | /usr/bin/tail -5
        europa [0] /usr/bin/du -sk /var/* | /bin/sort -n | /usr/bin/tail -5
        112     /var/run
        116     /var/spool
        21948   /var/cache
        42060   /var/lib
        513144  /var/log
        europa [0] /usr/bin/du -sk /var/log/* | /bin/sort -n | /usr/bin/tail -5
        88      /var/log/audit
        104     /var/log/dracut.log
        108     /var/log/messages
        144     /var/log/anaconda.storage.log
        512224  /var/log/prodapp
        europa [0] /bin/ls -l /var/log/prodapp
        total 512220
        -rw-r--r--. 1 root root    218678 Aug 16 13:07 prodapp.log
        -rw-r--r--. 1 root root 524288000 Aug 16 13:19 prodapp.log.0.gz
Our application directory, "/usr/local/prodapp", is a separate FS with available space. Our application, however, expects to write its logs to a directory at "/var/log/prodapp". To keep our logs with our application and to setup for our loopback mount, we'll create a "logs" directory under "/usr/local/prodapp". After that, we'll move the contents of "/var/log/prodapp" to "/usr/local/prodapp/logs":
        europa [0] /bin/ls -Fb /usr/local/prodapp
        bin/  conf/  lost+found/  readme.html
        europa [0] /bin/mkdir /usr/local/prodapp/logs && /bin/ls /usr/local/prodapp
        bin/  conf/  logs/  lost+found/  readme.html
        europa [0] /bin/mv /var/log/prodapp/* /usr/local/prodapp/logs/.
        europa [0] /bin/ls -l /var/log/prodapp /usr/local/prodapp/logs
        /usr/local/prodapp/logs:
        total 512220
        -rw-r--r--. 1 root root    218678 Aug 16 13:07 prodapp.log
        -rw-r--r--. 1 root root 524288000 Aug 16 13:19 prodapp.log.0.gz

        /var/log/prodapp:
        total 0
        europa [0]
At this point, we can remount "/usr/local/prodapp/logs" to the "/var/log/prodapp" directory via a loopback mount. This will make the underlying directory sub-tree of "/usr/local/prodapp/logs" accessible also under "/var/log/prodapp":
        europa [0] /bin/mount --bind /usr/local/prodapp/logs /var/log/prodapp
        berkeley [0] /sbin/mount -t nullfs /usr/local/prodapp/logs /var/log/prodapp
        sunspot [0] /usr/sbin/mount -F lofs /usr/local/prodapp/logs /var/log/prodapp
        berkeley [0] /bin/df -h
        Filesystem                 Size    Used   Avail Capacity  Mounted on
        /dev/da0s1a                5.8G    1.9G    3.5G    35%    /
        devfs                      1.0K    1.0K      0B   100%    /dev
        /dev/da0s1d                990M    834K    910M     0%    /var
        /dev/da1s1a                1.9G    501M    1.3G    28%    /usr/local/prodapp
        /usr/local/prodapp/logs    1.9G    501M    1.3G    28%    /var/log/prodapp
        berkeley [0]
Of note, I've used the 'df -h' output from "berkeley" in the above since it reflects the expected output from both FreeBSD and Solaris. Unfortunately, as seen below, to get the same output under Linux, we need to use 'df -ah' to allow "dummy" or "virtual" FS to be displayed:
        europa [0] /bin/df -h
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.6G  4.1G  27% /
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M   97M  860M  11% /var
        /dev/sdb1             2.0G  537M  1.4G  29% /usr/local/prodapp
        europa [0] /bin/df -ah
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.6G  4.1G  27% /
        proc                     0     0     0   -  /proc
        sysfs                    0     0     0   -  /sys
        devpts                   0     0     0   -  /dev/pts
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M   97M  860M  11% /var
        none                     0     0     0   -  /proc/sys/fs/binfmt_misc
        /dev/sdb1             2.0G  537M  1.4G  29% /usr/local/prodapp
        /usr/local/prodapp/logs
                              2.0G  537M  1.4G  29% /var/log/prodapp
A quick check via 'mount' for all three OSes supports the output from 'df':
        europa [0] /bin/mount | /bin/grep prodapp
        /dev/sdb1 on /usr/local/prodapp type ext4 (rw)
        /usr/local/prodapp/logs on /var/log/prodapp type none (rw,bind)

        sunspot [0] /usr/sbin/mount | /bin/grep prodapp
        /usr/local/prodapp on /dev/dsk/c1t1d0s0 read/write/setuid/devices/intr/largefiles/logging/xattr/onerror=panic/dev=800080 on Thu Aug 16 12:56:49 2012
        /var/log/prodapp on /usr/local/prodapp/logs read/write/setuid/devices/dev=800080 on Thu Aug 16 13:22:06 2012

        berkeley [0] /sbin/mount | /usr/bin/grep prodapp
        /dev/da1s1a on /usr/local/prodapp (ufs, local)
        /usr/local/prodapp/logs on /var/log/prodapp (nullfs, local)
Since our loopback mount simply provides access to a portion of our original FS via an additional, new directory structure, we can see the same files are presented under each, thus also having the same inode numbers, etc:
        europa [0] /bin/ls -li /var/log/prodapp /usr/local/prodapp/logs
        /usr/local/prodapp/logs:
        total 512220
        33 -rw-r--r--. 1 root root    218678 Aug 16 13:07 prodapp.log
        34 -rw-r--r--. 1 root root 524288000 Aug 16 13:19 prodapp.log.0.gz

        /var/log/prodapp:
        total 512220
        33 -rw-r--r--. 1 root root    218678 Aug 16 13:07 prodapp.log
        34 -rw-r--r--. 1 root root 524288000 Aug 16 13:19 prodapp.log.0.gz
The only thing left is to add the loopback mount to the "fstab" so that it will be mounted at boot time. (Remember to add it so it will be mounted after the original FS is mounted):
        europa [0] /usr/bin/tail -2 /etc/fstab
        /dev/sdb1               /usr/local/prodapp      ext4    defaults        0 2
        /usr/local/prodapp/logs /var/log/prodapp        none    bind            0 0

        berkeley [0] /usr/bin/tail -2 /etc/fstab
        /dev/da1s1a     /usr/local/prodapp      ufs     rw              0       2
        /usr/local/prodapp/logs /var/log/prodapp        nullfs  rw      0       0

        sunspot [0] /usr/bin/tail -2 /etc/vfstab
        /dev/dsk/c1t1d0s0    /dev/rdsk/c1t1d0s0   /usr/local/prodapp   ufs   0  yes     -
        /usr/local/prodapp/logs -       /var/log/prodapp        lofs    -       yes     -
Alright, now some caveats. If the underlying FS structure (/usr/local/prodapp/logs) does not exist prior to attempting to mount the loopback FS (/var/log/prodapp), the mount of the loopback mount will fail. The following is after unmounting both "/var/log/prodapp" and "/usr/local/prodapp", thus "/usr/local/prodapp/logs" is not available:
        europa [0] /bin/mount --bind /usr/local/prodapp/logs /var/log/prodapp
        mount: special device /usr/local/prodapp/logs does not exist
        europa [32]

        berkeley [0] /sbin/mount -t nullfs /usr/local/prodapp/logs /var/log/prodapp
        mount_nullfs: /usr/local/prodapp/logs: No such file or directory
        berkeley [64]

        sunspot [0] /usr/sbin/mount -F lofs /usr/local/prodapp/logs /var/log/prodapp
        mount: /usr/local/prodapp/logs: No such file or directory
        sunspot [33]
Also, it should not, and is not possible to unmount the mounted FS while the loopback FS is mounted, as seen under FreeBSD and Solaris:
        berkeley [0] /sbin/umount /usr/local/prodapp
        umount: unmount of /usr/local/prodapp failed: Device busy
        berkeley [1]
Linux, curiously however, will allow you to do just that leaving the loopback FS still accessible just as it was previously:
        europa [0] /bin/umount /usr/local/prodapp
        europa [0] /bin/df -ah
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda1             6.0G  1.6G  4.2G  27% /
        proc                     0     0     0   -  /proc
        sysfs                    0     0     0   -  /sys
        devpts                   0     0     0   -  /dev/pts
        tmpfs                 467M     0  467M   0% /dev/shm
        /dev/sda3            1008M   98M  860M  11% /var
        /usr/local/prodapp/logs
                              2.0G  537M  1.4G  29% /var/log/prodapp
        none                     0     0     0   -  /proc/sys/fs/binfmt_misc
        europa [0] /bin/touch /var/log/prodapp/trash
        europa [0] /bin/ls -li /var/log/prodapp/trash
        35 -rw-r--r--. 1 root root 0 Aug 16 13:43 /var/log/prodapp/trash
        europa [0] /bin/mount /usr/local/prodapp
        europa [0] /bin/df -h /usr/local/prodapp
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sdb1             2.0G  537M  1.4G  29% /usr/local/prodapp
        europa [0] /bin/ls -li /usr/local/prodapp/logs/trash
        35 -rw-r--r--. 1 root root 0 Aug 16 13:43 /usr/local/prodapp/logs/trash
Finally, since the loopback FS is simply representing a portion of the original mounted FS, the loopback FS by default will be restricted to the mount options used for the mounted FS. While you can further limit the options on the loopback FS, the mount options will not be allowed to exceed those of the mounted FS. As an example, if the mounted FS is mounted read-write, the loopback FS can be subsequently mounted read-only. If, however, the mounted FS is mounted read-only, the loopback FS will also be read-only:
        berkeley [0] /sbin/mount -o ro /dev/da1s1a /usr/local/prodapp
        berkeley [0] /sbin/mount -t nullfs -o rw /usr/local/prodapp/logs /var/log/prodapp

        sunspot [0] /usr/sbin/mount -o ro /dev/dsk/c1t1d0s0 /usr/local/prodapp
        sunspot [0] /usr/sbin/mount -F lofs -o rw /usr/local/prodapp/logs /var/log/prodapp
        mount: /usr/local/prodapp/logs on /var/log/prodapp - WARNING ignoring option "rw"

        europa [0] /bin/mount -t ext4 -o ro /dev/sdb1 /usr/local/prodapp
        europa [0] /bin/mount -o bind,rw /usr/local/prodapp/logs /var/log/prodapp

        europa [0] /bin/touch /usr/local/prodapp/logs/trash.0
        berkeley [0] /usr/bin/touch /usr/local/prodapp/logs/trash.0
        touch: cannot touch `/usr/local/prodapp/logs/trash.0': Read-only file system
        europa [1] /bin/touch /var/log/prodapp/trash.0
        touch: cannot touch `/var/log/prodapp/trash.0': Read-only file system
        europa [1]
Reviewing our mounted FS options, we have:
        europa [1] /bin/mount | /bin/grep prodapp
        /dev/sdb1 on /usr/local/prodapp type ext4 (ro)
        /usr/local/prodapp/logs on /var/log/prodapp type none (rw,bind)
        europa [0] /bin/grep prodapp /proc/mounts
        /dev/sdb1 /usr/local/prodapp ext4 ro,seclabel,relatime,barrier=1,data=ordered 0 0
        /dev/sdb1 /var/log/prodapp ext4 ro,seclabel,relatime,barrier=1,data=ordered 0 0
        europa [0]

        berkeley [1] /sbin/mount -p | /usr/bin/grep prodapp
        /dev/da1s1a             /usr/local/prodapp      ufs     ro              0 2
        /usr/local/prodapp/logs /var/log/prodapp        nullfs  rw              0 0
        berkeley [0]

        sunspot [1] /usr/sbin/mount -p | /bin/grep prodapp
        /dev/dsk/c1t1d0s0 - /usr/local/prodapp ufs - no ro,intr,largefiles,logging,xattr,onerror=panic
        /usr/local/prodapp/logs - /var/log/prodapp lofs - no
        sunspot [0]

No comments: