30 October 2010

Fix a Broken chmod, or How to chmod chmod

So here's the setup.  You've logged into one of your servers to update
permissions on a file only to see this:

        server [0] /bin/chmod +x /opt/somefile
        -ksh: /bin/chmod: cannot execute [Permission denied]
        server [0] /bin/ls -ld /bin/chmod
        -rw-r--r-- 1 root root 38564 Jul 18 22:24 /bin/chmod

Ouch, if we use 'chmod' to modify permissions on files, yet 'chmod'
is no longer executable, then we're up the proverbial creek without
a paddle.  Right?

Those who know me will tell you that I'm particularly fond of this
seeming conundrum.  The typical answers that I hear to this situation
are normally "Just copy it over from another box," or "Copy it over
from the boot media."  I seem to recall once hearing "Restore it from
backup." While these recovery methods are fine, they are also unnecessary
since you can recover chmod's execute bit using only the resources on
the server.


        Hosts:                  snorkle, tux, beastie
        Shell Prompt:           host [0]
        OSes:                   Solaris 10, CentOS 5.4, FreeBSD 8.1
        CHMOD Default Permissions:
                Solaris:        0555 (root:root)
                Linux:          0755 (root:root
                FreeBSD:        0555 (root:wheel)
        Notes:                  The following is pretty generic and should
                                be usable on earlier versions of each OS.


All of the following will result in restoring the default  permissions
to 'chmod'.  To begin, since 'setfacl' can also modify permissions,
it could be used (see note 1 at end):

        snorkle [0] /usr/bin/setfacl -s u::r-x,g::r-x,o:r-x /bin/chmod
        tux [0] /usr/bin/setfacl --set u::rwx,g::r-x,o::r-x /bin/chmod
        beastie [0] /bin/setfacl -bn -m u::r-x,g::r-x,o::r-x /bin/chmod

Of note, both the Solaris and Linux versions of 'setfacl' include a
deliberate 'set' function, whereas FreeBSD only allows 'modify'.  As a
result, the first two options to FreeBSD's 'setfacl' seen above are
'-bn', which will remove all facls aside from the required user, group,
and other.  I mention this because otherwise 'chmod' would end up with
a facl due to setting of a default mask, thus not exactly identical to
previous permissions.

There's always copying out another executable binary with the same
permissions and overwriting it.  On all three OSes, 'ls' has the same
ownership / permissions as 'chmod'.  Since the process is identical on
all three boxes, 'snorkle' will be used for the example:

        snorkle [0] /bin/ls -ld /bin/ls
        -r-xr-xr-x   1 root     bin        18700 Jun 11  2008 /bin/ls
        snorkle [0] /bin/cp /bin/ls /tmp/chmod
        snorkle [0] /bin/cat /bin/chmod > /tmp/chmod
        snorkle [0] /bin/mv /tmp/chmod /bin/chmod

For Solaris and Linux, one can always run 'ld*.so.X', the runtime linker,
which will execute a dynamic executable such as 'chmod'.  The kernel
will map the file and pass off control to the appropriate interpreter
to handle, effectively allowing you to use 'chmod' to fix 'chmod':

        snorkle [0] /lib/ld.so.1 /bin/chmod +x /bin/chmod
        tux [0] /lib/ld-linux.so.2 /bin/chmod +x /bin/chmod

Anymore, while you may not find a C compiler on a production box, 'perl'
is quite often available.  This is useful because 'perl' has its very own
'chmod' function:

        snorkle [0] /usr/bin/perl -e 'chmod 0555, qw(/bin/chmod)'
        tux [0] /usr/bin/perl -e 'chmod 0755, qw(/bin/chmod)'
        beastie [0] /usr/bin/perl -e 'chmod 0555, qw(/bin/chmod)'

The last method to mention is 'install', familiar to those who compile
source packages (see note 2), and move it into place:

        snorkle [0] /usr/ucb/install -o root -g root -m 0555 /bin/chmod /tmp/chmod
        tux [0] /usr/bin/install -m 0755 /bin/chmod /tmp/chmod
        beastie [0] /usr/bin/install -m 0555 /bin/chmod /tmp/chmod
        snorkle [0] /bin/mv /tmp/chmod /bin/chmod

The result of any of the above methods is the restoration of the
executable bit on 'chmod', as seen below on snorkle:

        snorkle [0] /bin/ls -ld /bin/chmod
        -r-xr-xr-x   1 root     root       14236 Oct 29 23:12 /bin/chmod

Hopefully one wouldn't have to deal with this type of situation, though
as UNIX administrators, we know accidents can happen.  By remaining calm
and thinking creatively, it's easy to see that the situation really isn't
all that bad and is recoverable.  This also goes to illustrate the time
honored UNIX saying, "there's more than one way to do it."  If you have
an alternative means of handling a non-executable 'chmod' and are willing
to share it, drop a line in the comments.

Note 1:

On Linux and FreeBSD hosts, unless facls are enabled on the filesystem,
via either mount options or tune2fs / tunefs (respectively), attempting
to set a facl will result in the following:

        tux [0] /usr/bin/setfacl --set u::rwx,g::r-x,o::r-x /bin/chmod
        setfacl: chmod: Operation not supported
        beastie [0] /bin/setfacl -m u::r-x,g::r-x,o::r-x /bin/chmod
        setfacl: chmod: acl_get_file() failed: Operation not supported

A quick resolve is a remount of the filesystem with acl support:

        tux [0] /bin/mount -o remount,rw,acl /
        beastie [0] /sbin/mount -o rw,acls -u /

Additionally, upon using the illustrated 'setfacl' command on FreeBSD,
the following error may occur and can be safely ignored:

        beastie [0] /bin/setfacl -bn -m u::r-x,g::r-x,o::r-x /bin/chmod
        setfacl: /bin/chmod: warning: no mask entry

Note 2:

Solaris has two 'install' executables:

        sbin:   /usr/sbin/install
        ucb:    /usr/ucb/install

The first (sbin) is installed with package SUNWcsu (Core Solaris,
(Usr)).  It is also unusable in this situation as it is only a shell
script which would otherwise call the non-functioning 'chmod' binary.
The second (ucb) is installed with package SUNWscpu (Source Compatibility,
(Usr)).  This one is a binary executable and does not call the 'chmod'
executable to modify file permissions, which is why it works.

As a secondary, the additional options for the Solaris 'install' command,
specify the user:group.  This is due to 'install' placing 'chmod' into
/tmp as user:group 'root:staff', rather than 'root:root' by default.


Anonymous said...

Thank You! Thank You! You just saved my butt!

troy said...

You're very welcome and I'm glad it helped.

Anonymous said...

Hello Troy

Well I think I am doomed.
I accidently did a
"find . -type f -exec chmod 644 {} \;"
on directory /
which resulted in having execution grants removed from every command in /bin dir.

If you ever think of a solution to this nightmare, that would be really really really appreciated.

troy said...


I would wager to say that more than just files in /bin were modified if you executed chmod against / (unless you killed it on /bin). Unfortunately, there's not really an easy resolve. Most likely, you will need to bring the box down and boot from cdrom, mount up the FS read-write, and start the arduous task of reapplying the appropriate permissions. If you have backups, you may be able to restore from those. Another option might be the emergency / fix it / repair option from an install cdrom for your OS (sorry, don't recall the option name). You might be able to use an upgrade option from the cdrom and simply reapply those packages affecting the files in question. If the setup cd is a live cd, you could likely stat the files under /bin on the cd and use the output to reset the permissions on you "broken" files, using 'chmod' from the live cd. If your setup is Solaris, you may be able to use 'pkgchk' to restore the permissions from the pkg database (since you said only /bin was affected and 'pkgchk' is in /usr/sbin). For how to do so, have a look at a writeup I did called file integrity checks. Other than that, I'd have to give this a bit more thought and testing. I think I see a few test boxes getting wrecked in my near future.

If you manage to resolve your situation, let me know, I would be eager to hear what you did.