12 January 2012

Fixing an Overly Eager chmod in Linux

A while ago, someone asked me how to recover from a mistyped recursive
'chmod' they performed.  Similar to the write-up on an "overeager chown",
they mistyped the path and it executed against the root FS (/).  Ideally,
one would have a backup to recover from, however that wasn't an option
in either the original situation or the one detailed herein.  Our host
details are:
        HOSTs:          europa, cobblepot
        PROMPT:         [HOST [0] |bash-4.1# ]
        OS:             CentOS 6.0
        NOTE:           The following should reasonably work on prior
                        versions of CentOS (or Red Hat based distros).
Before proceeding, this is a fairly long write-up.  The situation
presented is tedious, though not overly difficult.  Below, we see that
we are currently logged in as "root" and the intention is to reset the
permissions on files under "/usr/app1.0/etc" to read-only for only the
owner (mode 400).  The 'chmod' is meant to recursively execute against
that directory but due to a space in the path, is executed against "/"
and "usr/app1.0/etc" instead:
        europa [0] pwd
        europa [0] /usr/bin/whoami
        europa [0] /usr/bin/who am i
        root     pts/0        2012-01-07 20:13 (glados)
        europa [0] /bin/find /usr/app1.0/etc -exec /bin/ls -ld {} \;
        drwxr-xr-x. 2 root root 4096 Jan  7 20:36 /usr/app1.0/etc
        -rw-r--r--. 1 root root 1214 Jan  7 20:36 /usr/app1.0/etc/keyfile
        -rw-r--r--. 1 root root 1699 Jan  7 20:33 /usr/app1.0/etc/authfile
        europa [0] /bin/chmod -R 400 / usr/app1.0/etc
        bin/chmod: changing permissions of `/proc/sys': Operation not permitted
        /bin/chmod: changing permissions of `/proc/sys/kernel': Operation not permitted
        /bin/chmod: changing permissions of `/proc/sys/kernel/sched_child_runs_first': Operation not permitted
        /bin/chmod: changing permissions of `/proc/sys/kernel/sched_min_granularity_ns': Operation not permitted
        /bin/chmod: changing permissions of `/proc/sys/kernel/sched_latency_ns': Operation not permitted
        /bin/chmod: changing permissions of `/proc/sys/kernel/sched_wakeup_granularity_ns': Operation not permitted
        /bin/chmod: changing permissions of `/proc/1688/io': Permission denied
        /bin/chmod: cannot access `usr/app1.0/etc': No such file or directory
        europa [1]
        # 7864 '/proc/...' errors later
        europa [1] /bin/ls -l /
        /bin/ksh: /bin/ls: cannot execute [Permission denied]
        europa [126] /lib64/ld-linux-x86-64.so.2 /bin/ls -l /
        /bin/ksh: /lib64/ld-linux-x86-64.so.2: cannot execute [Permission denied]
        europa [126] /sbin/init 0
        /bin/ksh: /sbin/init: cannot execute [Permission denied]
        europa [126]
In the above, shortly into the 'chmod', the errors start (just shy
of 8000 errors, actually).  When the command finally finishes, we see
that we can no longer execute any binaries, including 'init'.  Our only
option at this point is to physically power off the host and boot it
using a CD / DVD (disc 1 of the install discs).  Once to the CD menu,
select the rescue option (denoted by "<============":
                        Welcome to CentOS 6.0!

        Install or upgrade an existing system
        Install system with basic video driver
        Rescue installed system                 <============
        Boot from local drive
        Memory test

                     Press [Tab] to edit options

                   Automatic boot in 60 seconds...
The screen clears after menu selection and system starts to boot from CD:
        Loading vmlinux................................................
        Loading initrd.img............................................r
        Probing EDD (edd=off to disable)... ok
We are then prompted by another series of menus.  Choose your 'Language',
'Keyboard type', 'Rescue image location' (Local CD/DVD), 'Networking'
(No), 'Rescue' (Continue).  The root FS for host "europa" will be
mounted at "/mnt/sysimage".  Once you opt for the shell entry below,
a shell is spawned:
        First Aid Kit quickstart menu

                  shell  Start shell            <============
                  fakd   Run diagnostic
                  reboot Reboot

                <Ok>             <Cancel>

        Starting shell...
        bash-4.1# _
A quick check of "/proc/mounts" verifies europa's root FS as read-write
and an 'ls' of "/mnt/sysimage" shows our non-pseudo FS directories with
mode "400":
        bash-4.1# /usr/bin/grep '/mnt/sysimage ' /proc/mounts
        /dev/sda1 /mnt/sysimage ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
        bash-4.1# /usr/bin/ls -l /mnt/sysimage
        dr--------. 25 root root 4096 2012-01-07 20:56 /mnt/sysimage
        total 104
        dr--------.  2 root root  4096 2012-01-08 00:51 bin
        dr--------.  4 root root  4096 2012-01-08 00:52 boot
        dr--------.  2 root root  4096 2010-11-12 00:33 cgroup
        drwxrwxrwt. 14 root root  3400 2012-01-07 20:55 dev
        dr--------. 80 root root  4096 2012-01-08 01:11 etc
        drwxr-xr-x. 13 root root     0 2012-01-07 20:53 sys
        dr--------.  3 root root  4096 2012-01-08 01:13 tmp
        dr--------. 14 root root  4096 2012-01-08 01:20 usr
        dr--------. 22 root root  4096 2012-01-08 00:51 var
Booting from the "rescue" option of the CD / DVD avails us to a version
of 'rpm' which we can use for the recovery of most of our system files.
Below, setting root directory structure to head from "/mnt/sysimage",
rather than the default of "/", allows us to query europa's rpm database.
(As a reminder, we can't 'chroot' to "/mnt/sysimage" and run 'rpm' from
there since none of the files on europa's filesystems are executable.) Of
note, once we set the root directory for 'rpm', any subsequent paths
supplied to the command are relative to the specified root directory:
        bash-4.1# /usr/bin/rpm --root=/mnt/sysimage -qf /bin/rpm
        bash-4.1# /usr/bin/rpm --root=/mnt/sysimage -q --dump rpm-4.8.0-12.el6.x86_64 |
        > /usr/bin/grep '/bin/rpm '
        /bin/rpm 20392 1289521670 88d60915477cee96d4de6fb63bd3ca3db7efb05a83325e69375c54af3933bc90 0100755 root root 0 0 0 X
        bash-4.1# /usr/bin/rpm --root=/mnt/sysimage -Vf /bin/rpm
        .M.......    /bin/rpm
        .M.......    /etc/rpm
        .M.......    /usr/rpm/rpm2cpio
        .M.......    /usr/lib/rpm
        .M.......    /usr/lib/rpm/macros
        .M.......    /usr/lib/rpm/platform
        .M.......  d /usr/share/man/ru/man8/rpm2cpio.8.gz
        .M.......  d /usr/share/man/sk/man8/rpm.8.gz
        .M.......    /var/lib/rpm
Above, we see that '/bin/rpm' should have a mode of 755, owned by uid:gid
root:root.  In addition, the files in the package that '/bin/rpm' belongs
to all have modes / permissions differing from their relevant entries in
europa's rpm database.  Below, an attempt to use '--setperms' illustrates
that not all 'rpm' options are available from the "rescue" version of
'rpm'.  Given that, we'll execute a 'for' loop to query every package
in the rpm database for any files with incorrect modes / permissions.
This returns more than 52000 files:
        bash-4.1# /usr/bin/rpm --root=/mnt/sysimage --setperms rpm-4.8.0-12.el6.x86_64
        --setperms: unknown option
        bash-4.1# for i in `/usr/bin/rpm --root=/mnt/sysimage -qa` ; do
        > /usr/bin/rpm --root=/mnt/sysimage -V ${i} |
        > /bin/awk '/^.M./ {print "'$i':"$NF}' ; done >> /tmp/mod-files
        # might take a minute or so for the above to return
        bash-4.1# /usr/bin/wc -l /tmp/mod-files
        bash-4.1# /usr/bin/head -4 /tmp/mod-files
Using the file we created above as input for another 'for' loop, we'll
again query the rpm database, this time for the appropriate modes /
permissions and use 'chmod' to reset those permissions.  (Some errors may
be seen regarding non-existent man pages.  This is due our use of ":"
as a delimitter and some perl man pages containing ":" in their names.
You can ignore these errors for the moment, we'll fix them later.  Also,
the run time below took about 2.5 hrs on the test host, grab some coffee):
        bash-4.1# for i in `/usr/bin/cat /tmp/mod-files` ; do a=`echo "$i" |
        > /usr/bin/cut -d: -f1` ; b=`echo "$i" | /usr/bin/cut -d: -f2` ;
        > c=`/usr/bin/rpm --root=/mnt/sysimage -q --dump $a | /usr/bin/grep "^$b " |
        > /bin/awk '{print $5}' | /bin/sed -e 's/^...//g'` ;
        > echo "chmod $c /mnt/sysimage/$b" ; /usr/bin/chmod $c /mnt/sysimage/$b ; done
        chmod 755 /mnt/sysimage//etc/kdump-adv-conf/kdump_initscripts
        chmod 0755 /mnt/sysimage//etc/kdump-adv-conf/kdump_initscripts/init
        chmod 0644 /mnt/sysimage//etc/kdump-adv-conf/kdump_initscripts/kdumpinit.rootfs
        chmod 755 /mnt/sysimage//etc/kdump-adv-conf/kdump_sample_manifests
        chmod 0644 /mnt/sysimage/usr/share/doc/yajl-1.0.7/TODO
Once the above completes, not inlcuding our pseudo FS, out of 60297
files on europa's FS, we still have 6014 files left to account for with
permissions set to 400.  A check on an identical host shows that there
should be 0 files (by default) with mode 400:
        bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
        > -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" \) -prune \
        > -o -print | /usr/bin/wc -l
        bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
        > -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" \) -prune \
        > -o -perm 400 -print | /usr/bin/wc -l

        cobblepot [0] /bin/find / \( -path "/dev" -o -path "/selinux" -o -path \
        > "/proc" -o -path "/sys" \) -prune -o -perm 400 -print | /usr/bin/wc -l
        cobblepot [0]
Further excluding europa's yum database and man directories, the number of
files shrinks to 514.  For both the yum and man directories, files should
all be mode 644 and directories 755, which we easily take care of below.
In addition, we also remove any non-directory files located under europa's
"/var/run", "/var/spool/postfix", and "/var/lock" since the files here
are mostly transient and relate to the run time environment prior to the
host power off event:
        bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
        > -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" -o -path \
        > "${a}/var/lib/yum/yumdb" -o -path "${a}/usr/share/man" -o -path \
        > "${a}/usr/app1.0" \) -prune -o -perm 400 -print | /usr/bin/wc -l
        bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -perm 400 -print |
        > /usr/bin/wc -l
        bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -type f -perm 400 \
        > -exec /usr/bin/chmod 644 {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -type d -perm 400 \
        > -exec /usr/bin/chmod 755 {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -perm 400 -print |
        > /usr/bin/wc -l
        bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -perm 400 -print |
        > /usr/bin/wc -l
        bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -type d -perm 400 \
        > -exec /usr/bin/chmod 755 {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -type f -perm 400 \
        > -exec /usr/bin/chmod 644 {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -perm 400 -print |
        > /usr/bin/wc -l
        bash-4.1# /usr/bin/find /mnt/sysimage/var/run ! -type d -exec /usr/bin/rm {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/var/spool/postfix ! -type d -exec \
        > /usr/bin/rm {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/var/lock ! -type d -exec /usr/bin/rm {} \;
After taking care of the man and yum directories, we now come to the
somewhat painful part of the recovery.  Before, we were able to rely
on the rpm database, now it's a matter of knowing the modes for those
system files not included as part of an rpm package.  As a brief,
general overview before getting to specific commands, the following
directories and their child directories should be set to 755 (some of
these will be further adjusted later (all directories are relative to
root "/mnt/sysimage")):
Additionally, the files found under the following directories should be
set 644:
After the above, the child directories found under the following
directories should be set 700:
Files found under the following should be set 600:
The following files should be set 600:
The following files should be set 644:
        /etc/selinux/targeted/contexts/files    (any files found under here)
Finallly, /etc/shadow- should be set 000.  Putting all of this together,
we get the following command sets:
        bash-4.1# a="/mnt/sysimage" ; for i in /boot/efi /etc/event.d \
        > /etc/kdump-adv-conf /etc/plymouth /etc/selinux /usr/lib64 /usr/libexec \
        > /usr/share /var ; do /usr/bin/find ${a}${i} -type d -perm 400 -exec \
        > /usr/bin/chmod 755 {} \; ; done
        bash-4.1# a="/mnt/sysimage" ; for i in /boot/grub /etc /lib/modules /usr/lib64 \
        > /usr/lib /var ; do /usr/bin/find ${a}${i} -type f -perm 400 -exec \
        > /usr/bin/chmod 644 {} \; ; done
        bash-4.1# a="/mnt/sysimage" ; for i in /etc/selinux/targeted/modules/active \
        > /lost+found /var/lost+found ; do /usr/bin/find ${a}${i} -type d -exec \
        > /usr/bin/chmod 700 {} \; ; done
        bash-4.1# /usr/bin/find /mnt/sysimage/etc/selinux/targeted/modules/active \
        > -type f -exec /usr/bin/chmod 600 {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/etc/ssh -type f -exec \
        > /usr/bin/chmod 600 {} \;
        bash-4.1# /usr/bin/find /mnt/sysimage/etc/ssh -type f -name "*.pub" -exec \
        > /usr/bin/chmod 600 {} \;
        bash-4.1# /usr/bin/chmod 644 /mnt/sysimage/etc/ssh/ssh_config
        bash-4.1# a="/mnt/sysimage" ; for i in /boot/grub/grub.conf /etc/group- \
        > /etc/gshadow- /etc/.pwd.lock /etc/lvm/cache/.cache /etc/sysconfig/ip*tables* \
        > /etc/sysconfig/system-config-firewall /var/cache/ldconfig/* \
        > /var/cache/rpcbind/* /var/log/anaconda* /var/log/audit/* /var/log/btmp \
        > /var/log/cron* /var/log/maillog* /var/log/messages* /var/log/secure* \
        > /var/log/spooler* /var/log/tallylog* /var/lib/postfix/master.lock \
        > /var/lib/random-seed /var/spool/mail/* /var/spool/anacron/cron* ; do
        > /usr/bin/chmod 600 ${a}${i} ; done
        bash-4.1# /usr/bin/find /mnt/sysimage/etc/selinux/targeted/contexts/files \
        > -type f -exec /usr/bin/chmod 644 {} \;
        bash-4.1# a="/mnt/sysimage/etc/selinux" ; /usr/bin/chmod 644 \
        > ${a}/targeted/modules/active/file_contexts.homdirs \
        > ${a}/targeted/modlues/active/policy.kern ${a}/config \
        > ${a}/targeted/policy/policy.24 ${a}/targeted/seusers
        bash-4.1# /usr/bin/chmod 000 /mnt/sysimage/etc/shadow-
        bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
        > -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" -o -path \
        > "${a}/usr/app1.0" \) -prune -perm 400 -print
        bash-4.1# /bin/reboot
        Running reboot...

        disabline swap...
        bash-4.1#       /dev/sda2
        unmounting filesystems...
The find command above (towards the end), excluding files found under our
pseudo FS and our original app directory, finds only one entry with mode
400, that of our app directory itself.  At this point, we can reboot
the host.  As the host is rebooting, don't forget to remove the CD /
DVD and now sit back an wait while the host starts its boot process.
If you have SELinux enabled, after the kernel is loaded and the system
starts to come back up, a message about SELinux relabeling will be seen,
after which the host will automatically reset again:
        *** Warning -- SELinux targeted policy relabel is required.
        *** Relabeling could take a very long time, depending on file
        *** system size and speed of hard drives.
Once the relabeling is complete and the host has reset, the host should
fully boot up to the default runlevel.  After logging into the host,
a subsequent verify of files with mode 400 shows only our app directory:
        europa [0] /bin/find / \( -path /dev -o -path /selinux -o -path /proc -o \
        > -path /sys -o -path /usr/app1.0 \) -prune -perm 400 -print
        europa [0]
We are now successfully finished with recovering our system and at
least our OS will function sanely again.  Of note, this doesn't account
for any files for which the "RPM" database has no information or are
"normal" system files, thus our app directory still needs to be handled.
Since this was one of those "I wonder if" scenarios for me, I'm fine
with that.  However, while it is possible to at least restore the OS files
to proper ownerships, having a recent backup image would have simplified
the recovery and possibly also accounted for our non-rpm application.

see also:
    Fixing an Overly Eager chown in Linux
    File Integrity Checks via Package DB

No comments: