'chmod' they performed. Similar to the write-up on an "overeager chown",
they mistyped the path and it executed against the root FS (/). Ideally,
one would have a backup to recover from, however that wasn't an option
in either the original situation or the one detailed herein. Our host
details are:
HOSTs: europa, cobblepot PROMPT: [HOST [0] |bash-4.1# ] OS: CentOS 6.0 NOTE: The following should reasonably work on prior versions of CentOS (or Red Hat based distros).Before proceeding, this is a fairly long write-up. The situation
presented is tedious, though not overly difficult. Below, we see that
we are currently logged in as "root" and the intention is to reset the
permissions on files under "/usr/app1.0/etc" to read-only for only the
owner (mode 400). The 'chmod' is meant to recursively execute against
that directory but due to a space in the path, is executed against "/"
and "usr/app1.0/etc" instead:
europa [0] pwd
/root
europa [0] /usr/bin/whoami
root
europa [0] /usr/bin/who am i
root pts/0 2012-01-07 20:13 (glados)
europa [0] /bin/find /usr/app1.0/etc -exec /bin/ls -ld {} \;
drwxr-xr-x. 2 root root 4096 Jan 7 20:36 /usr/app1.0/etc
-rw-r--r--. 1 root root 1214 Jan 7 20:36 /usr/app1.0/etc/keyfile
-rw-r--r--. 1 root root 1699 Jan 7 20:33 /usr/app1.0/etc/authfile
europa [0] /bin/chmod -R 400 / usr/app1.0/etc
bin/chmod: changing permissions of `/proc/sys': Operation not permitted
/bin/chmod: changing permissions of `/proc/sys/kernel': Operation not permitted
/bin/chmod: changing permissions of `/proc/sys/kernel/sched_child_runs_first': Operation not permitted
/bin/chmod: changing permissions of `/proc/sys/kernel/sched_min_granularity_ns': Operation not permitted
/bin/chmod: changing permissions of `/proc/sys/kernel/sched_latency_ns': Operation not permitted
/bin/chmod: changing permissions of `/proc/sys/kernel/sched_wakeup_granularity_ns': Operation not permitted
<snip...>
/bin/chmod: changing permissions of `/proc/1688/io': Permission denied
/bin/chmod: cannot access `usr/app1.0/etc': No such file or directory
europa [1]
# 7864 '/proc/...' errors later
europa [1] /bin/ls -l /
/bin/ksh: /bin/ls: cannot execute [Permission denied]
europa [126] /lib64/ld-linux-x86-64.so.2 /bin/ls -l /
/bin/ksh: /lib64/ld-linux-x86-64.so.2: cannot execute [Permission denied]
europa [126] /sbin/init 0
/bin/ksh: /sbin/init: cannot execute [Permission denied]
europa [126]
In the above, shortly into the 'chmod', the errors start (just shyof 8000 errors, actually). When the command finally finishes, we see
that we can no longer execute any binaries, including 'init'. Our only
option at this point is to physically power off the host and boot it
using a CD / DVD (disc 1 of the install discs). Once to the CD menu,
select the rescue option (denoted by "<============":
Welcome to CentOS 6.0! Install or upgrade an existing system Install system with basic video driver Rescue installed system <============ Boot from local drive Memory test Press [Tab] to edit options Automatic boot in 60 seconds...The screen clears after menu selection and system starts to boot from CD:
Loading vmlinux................................................ Loading initrd.img............................................r eady. Probing EDD (edd=off to disable)... ok <snip...>We are then prompted by another series of menus. Choose your 'Language',
'Keyboard type', 'Rescue image location' (Local CD/DVD), 'Networking'
(No), 'Rescue' (Continue). The root FS for host "europa" will be
mounted at "/mnt/sysimage". Once you opt for the shell entry below,
a shell is spawned:
First Aid Kit quickstart menu shell Start shell <============ fakd Run diagnostic reboot Reboot <Ok> <Cancel> Starting shell... bash-4.1# _A quick check of "/proc/mounts" verifies europa's root FS as read-write
and an 'ls' of "/mnt/sysimage" shows our non-pseudo FS directories with
mode "400":
bash-4.1# /usr/bin/grep '/mnt/sysimage ' /proc/mounts /dev/sda1 /mnt/sysimage ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0 bash-4.1# /usr/bin/ls -l /mnt/sysimage dr--------. 25 root root 4096 2012-01-07 20:56 /mnt/sysimage total 104 dr--------. 2 root root 4096 2012-01-08 00:51 bin dr--------. 4 root root 4096 2012-01-08 00:52 boot dr--------. 2 root root 4096 2010-11-12 00:33 cgroup drwxrwxrwt. 14 root root 3400 2012-01-07 20:55 dev dr--------. 80 root root 4096 2012-01-08 01:11 etc <snip...> drwxr-xr-x. 13 root root 0 2012-01-07 20:53 sys dr--------. 3 root root 4096 2012-01-08 01:13 tmp dr--------. 14 root root 4096 2012-01-08 01:20 usr dr--------. 22 root root 4096 2012-01-08 00:51 var bash-4.1#Booting from the "rescue" option of the CD / DVD avails us to a version
of 'rpm' which we can use for the recovery of most of our system files.
Below, setting root directory structure to head from "/mnt/sysimage",
rather than the default of "/", allows us to query europa's rpm database.
(As a reminder, we can't 'chroot' to "/mnt/sysimage" and run 'rpm' from
there since none of the files on europa's filesystems are executable.) Of
note, once we set the root directory for 'rpm', any subsequent paths
supplied to the command are relative to the specified root directory:
bash-4.1# /usr/bin/rpm --root=/mnt/sysimage -qf /bin/rpm rpm-4.8.0-12.el6.x86_64 bash-4.1# /usr/bin/rpm --root=/mnt/sysimage -q --dump rpm-4.8.0-12.el6.x86_64 | > /usr/bin/grep '/bin/rpm ' /bin/rpm 20392 1289521670 88d60915477cee96d4de6fb63bd3ca3db7efb05a83325e69375c54af3933bc90 0100755 root root 0 0 0 X bash-4.1# /usr/bin/rpm --root=/mnt/sysimage -Vf /bin/rpm .M....... /bin/rpm .M....... /etc/rpm .M....... /usr/rpm/rpm2cpio .M....... /usr/lib/rpm .M....... /usr/lib/rpm/macros .M....... /usr/lib/rpm/platform <snip...> .M....... d /usr/share/man/ru/man8/rpm2cpio.8.gz .M....... d /usr/share/man/sk/man8/rpm.8.gz .M....... /var/lib/rpm bash-4.1#Above, we see that '/bin/rpm' should have a mode of 755, owned by uid:gid
root:root. In addition, the files in the package that '/bin/rpm' belongs
to all have modes / permissions differing from their relevant entries in
europa's rpm database. Below, an attempt to use '--setperms' illustrates
that not all 'rpm' options are available from the "rescue" version of
'rpm'. Given that, we'll execute a 'for' loop to query every package
in the rpm database for any files with incorrect modes / permissions.
This returns more than 52000 files:
bash-4.1# /usr/bin/rpm --root=/mnt/sysimage --setperms rpm-4.8.0-12.el6.x86_64
--setperms: unknown option
bash-4.1# for i in `/usr/bin/rpm --root=/mnt/sysimage -qa` ; do
> /usr/bin/rpm --root=/mnt/sysimage -V ${i} |
> /bin/awk '/^.M./ {print "'$i':"$NF}' ; done >> /tmp/mod-files
# might take a minute or so for the above to return
bash-4.1# /usr/bin/wc -l /tmp/mod-files
52519
bash-4.1# /usr/bin/head -4 /tmp/mod-files
kexec-tools-2.0.0-145.el6.x86_64:/etc/kdump-adv-conf/kdump_initscripts
kexec-tools-2.0.0-145.el6.x86_64:/etc/kdump-adv-conf/kdump_initscripts/init
kexec-tools-2.0.0-145.el6.x86_64:/etc/kdump-adv-conf/kdump_initscripts/kdumpinit.rootfs
kexec-tools-2.0.0-145.el6.x86_64:/etc/kdump-adv-conf/kdump_sample_manifests
bash-4.1#
Using the file we created above as input for another 'for' loop, we'llagain query the rpm database, this time for the appropriate modes /
permissions and use 'chmod' to reset those permissions. (Some errors may
be seen regarding non-existent man pages. This is due our use of ":"
as a delimitter and some perl man pages containing ":" in their names.
You can ignore these errors for the moment, we'll fix them later. Also,
the run time below took about 2.5 hrs on the test host, grab some coffee):
bash-4.1# for i in `/usr/bin/cat /tmp/mod-files` ; do a=`echo "$i" |
> /usr/bin/cut -d: -f1` ; b=`echo "$i" | /usr/bin/cut -d: -f2` ;
> c=`/usr/bin/rpm --root=/mnt/sysimage -q --dump $a | /usr/bin/grep "^$b " |
> /bin/awk '{print $5}' | /bin/sed -e 's/^...//g'` ;
> echo "chmod $c /mnt/sysimage/$b" ; /usr/bin/chmod $c /mnt/sysimage/$b ; done
chmod 755 /mnt/sysimage//etc/kdump-adv-conf/kdump_initscripts
chmod 0755 /mnt/sysimage//etc/kdump-adv-conf/kdump_initscripts/init
chmod 0644 /mnt/sysimage//etc/kdump-adv-conf/kdump_initscripts/kdumpinit.rootfs
chmod 755 /mnt/sysimage//etc/kdump-adv-conf/kdump_sample_manifests
<snip...>
chmod 0644 /mnt/sysimage/usr/share/doc/yajl-1.0.7/TODO
bash-4.1#
Once the above completes, not inlcuding our pseudo FS, out of 60297files on europa's FS, we still have 6014 files left to account for with
permissions set to 400. A check on an identical host shows that there
should be 0 files (by default) with mode 400:
bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
> -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" \) -prune \
> -o -print | /usr/bin/wc -l
60297
bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
> -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" \) -prune \
> -o -perm 400 -print | /usr/bin/wc -l
6014
bash-4.1#
cobblepot [0] /bin/find / \( -path "/dev" -o -path "/selinux" -o -path \
> "/proc" -o -path "/sys" \) -prune -o -perm 400 -print | /usr/bin/wc -l
0
cobblepot [0]
Further excluding europa's yum database and man directories, the number offiles shrinks to 514. For both the yum and man directories, files should
all be mode 644 and directories 755, which we easily take care of below.
In addition, we also remove any non-directory files located under europa's
"/var/run", "/var/spool/postfix", and "/var/lock" since the files here
are mostly transient and relate to the run time environment prior to the
host power off event:
bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
> -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" -o -path \
> "${a}/var/lib/yum/yumdb" -o -path "${a}/usr/share/man" -o -path \
> "${a}/usr/app1.0" \) -prune -o -perm 400 -print | /usr/bin/wc -l
514
bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -perm 400 -print |
> /usr/bin/wc -l
871
bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -type f -perm 400 \
> -exec /usr/bin/chmod 644 {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -type d -perm 400 \
> -exec /usr/bin/chmod 755 {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/usr/share/man -perm 400 -print |
> /usr/bin/wc -l
0
bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -perm 400 -print |
> /usr/bin/wc -l
4582
bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -type d -perm 400 \
> -exec /usr/bin/chmod 755 {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -type f -perm 400 \
> -exec /usr/bin/chmod 644 {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/var/lib/yum/yumdb -perm 400 -print |
> /usr/bin/wc -l
0
bash-4.1# /usr/bin/find /mnt/sysimage/var/run ! -type d -exec /usr/bin/rm {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/var/spool/postfix ! -type d -exec \
> /usr/bin/rm {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/var/lock ! -type d -exec /usr/bin/rm {} \;
bash-4.1#
After taking care of the man and yum directories, we now come to thesomewhat painful part of the recovery. Before, we were able to rely
on the rpm database, now it's a matter of knowing the modes for those
system files not included as part of an rpm package. As a brief,
general overview before getting to specific commands, the following
directories and their child directories should be set to 755 (some of
these will be further adjusted later (all directories are relative to
root "/mnt/sysimage")):
/boot/efi /etc/event.d /etc/kdump-adv-conf /etc/plymouth /etc/selinux /usr/lib64 /usr/libexec /usr/share /varAdditionally, the files found under the following directories should be
set 644:
/boot/grub /etc /lib/modules /usr/lib64 /usr/lib /varAfter the above, the child directories found under the following
directories should be set 700:
/etc/selinux/targeted/modules/active /lost+found /var/lost+foundFiles found under the following should be set 600:
/etc/selinux/targeted/modules/active /etc/sshThe following files should be set 600:
/boot/grub/grub.conf /etc/group- /etc/gshadow- /etc/.pwd.lock /etc/lvm/cache/.cache /etc/sysconfig/ip*tables* /etc/sysconfig/system-config-firewall /var/cache/ldconfig/* /var/cache/rpcbind/* /var/log/anaconda/* /var/log/audit/* /var/log/btmp /var/log/cron* /var/log/maillog* /var/log/messages* /var/log/secure* /var/log/spooler* /var/log/tallylog* /var/lib/postfix/master.lock /var/lib/random-seed /var/spool/mail/* /var/spool/anacron/cron*The following files should be set 644:
/etc/ssh/ssh_config /etc/selinux/targeted/contexts/files (any files found under here) /etc/selinux/targeted/modules/active/file_contexts.homdirs /etc/selinux/targeted/modlues/active/policy.kern /etc/selinux/config /etc/selinux/targeted/policy/policy.24 /etc/selinux/targeted/seusersFinallly, /etc/shadow- should be set 000. Putting all of this together,
we get the following command sets:
bash-4.1# a="/mnt/sysimage" ; for i in /boot/efi /etc/event.d \
> /etc/kdump-adv-conf /etc/plymouth /etc/selinux /usr/lib64 /usr/libexec \
> /usr/share /var ; do /usr/bin/find ${a}${i} -type d -perm 400 -exec \
> /usr/bin/chmod 755 {} \; ; done
bash-4.1# a="/mnt/sysimage" ; for i in /boot/grub /etc /lib/modules /usr/lib64 \
> /usr/lib /var ; do /usr/bin/find ${a}${i} -type f -perm 400 -exec \
> /usr/bin/chmod 644 {} \; ; done
bash-4.1# a="/mnt/sysimage" ; for i in /etc/selinux/targeted/modules/active \
> /lost+found /var/lost+found ; do /usr/bin/find ${a}${i} -type d -exec \
> /usr/bin/chmod 700 {} \; ; done
bash-4.1# /usr/bin/find /mnt/sysimage/etc/selinux/targeted/modules/active \
> -type f -exec /usr/bin/chmod 600 {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/etc/ssh -type f -exec \
> /usr/bin/chmod 600 {} \;
bash-4.1# /usr/bin/find /mnt/sysimage/etc/ssh -type f -name "*.pub" -exec \
> /usr/bin/chmod 600 {} \;
bash-4.1# /usr/bin/chmod 644 /mnt/sysimage/etc/ssh/ssh_config
bash-4.1# a="/mnt/sysimage" ; for i in /boot/grub/grub.conf /etc/group- \
> /etc/gshadow- /etc/.pwd.lock /etc/lvm/cache/.cache /etc/sysconfig/ip*tables* \
> /etc/sysconfig/system-config-firewall /var/cache/ldconfig/* \
> /var/cache/rpcbind/* /var/log/anaconda* /var/log/audit/* /var/log/btmp \
> /var/log/cron* /var/log/maillog* /var/log/messages* /var/log/secure* \
> /var/log/spooler* /var/log/tallylog* /var/lib/postfix/master.lock \
> /var/lib/random-seed /var/spool/mail/* /var/spool/anacron/cron* ; do
> /usr/bin/chmod 600 ${a}${i} ; done
bash-4.1# /usr/bin/find /mnt/sysimage/etc/selinux/targeted/contexts/files \
> -type f -exec /usr/bin/chmod 644 {} \;
bash-4.1# a="/mnt/sysimage/etc/selinux" ; /usr/bin/chmod 644 \
> ${a}/targeted/modules/active/file_contexts.homdirs \
> ${a}/targeted/modlues/active/policy.kern ${a}/config \
> ${a}/targeted/policy/policy.24 ${a}/targeted/seusers
bash-4.1# /usr/bin/chmod 000 /mnt/sysimage/etc/shadow-
bash-4.1# a="/mnt/sysimage" ; /usr/bin/find ${a} \( -path "${a}/dev" -o \
> -path "${a}/selinux" -o -path "${a}/proc" -o -path "${a}/sys" -o -path \
> "${a}/usr/app1.0" \) -prune -perm 400 -print
/mnt/sysimage/usr/app1.0
bash-4.1# /bin/reboot
Running reboot...
disabline swap...
bash-4.1# /dev/sda2
unmounting filesystems...
<snip...>
The find command above (towards the end), excluding files found under ourpseudo FS and our original app directory, finds only one entry with mode
400, that of our app directory itself. At this point, we can reboot
the host. As the host is rebooting, don't forget to remove the CD /
DVD and now sit back an wait while the host starts its boot process.
If you have SELinux enabled, after the kernel is loaded and the system
starts to come back up, a message about SELinux relabeling will be seen,
after which the host will automatically reset again:
<snip...> *** Warning -- SELinux targeted policy relabel is required. *** Relabeling could take a very long time, depending on file *** system size and speed of hard drives. ************************************************************ <snip...>Once the relabeling is complete and the host has reset, the host should
fully boot up to the default runlevel. After logging into the host,
a subsequent verify of files with mode 400 shows only our app directory:
europa [0] /bin/find / \( -path /dev -o -path /selinux -o -path /proc -o \ > -path /sys -o -path /usr/app1.0 \) -prune -perm 400 -print /usr/app1.0 europa [0]We are now successfully finished with recovering our system and at
least our OS will function sanely again. Of note, this doesn't account
for any files for which the "RPM" database has no information or are
"normal" system files, thus our app directory still needs to be handled.
Since this was one of those "I wonder if" scenarios for me, I'm fine
with that. However, while it is possible to at least restore the OS files
to proper ownerships, having a recent backup image would have simplified
the recovery and possibly also accounted for our non-rpm application.
see also:
Fixing an Overly Eager chown in Linux
File Integrity Checks via Package DB
