27 March 2011

Finding Open Files in Linux

Finding open files in Linux is quite similar to finding them in FreeBSD
and Solaris.  Here, we'll specifically identify open files on a particular
filesystem (FS) within a Linux environment.  Our host details this
time are:

        HOST:           cobblepot
        PROMPT:         cobblepot [0]
        OS:             CentOS 5.4 Linux
        NOTE:           The following should also be applicable on prior
                        CentOS versions or other Red Hat EL variants.
Given our intent, we'll use /var as our FS.  Below, we see that /var
is its own FS and follow up with 'fuser' to see if any processes are
holding open any files under "/var" (see note 1):
        cobblepot [0] /bin/df -h /var
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sda5             3.9G  235M  3.5G   7% /var
        cobblepot [0] /sbin/fuser -c /var
        /var:                 3791  3824  3995  4012  4063c  4117c  4826  9465 \
        27737 28502 29550 29553 29554 29555 29556 29557 29558 29560
In the above, we can see 18 running processes with opened files in "/var".
If we wanted to get more information about each of those processes,
we could pass the output of 'fuser' to 'ps' as a paramter to flag "-p":
        cobblepot [0] /bin/ps -fp `/sbin/fuser -c /var 2>/dev/null`
        UID        PID  PPID  C STIME TTY      STAT   TIME CMD
        root      3791     1  0  2010 ?        S<sl   0:08 auditd
        root      3824     1  0  2010 ?        Ss     9:55 syslogd -m 0
        root      3995     1  0  2010 ?        Ss     0:00 /usr/sbin/acpid
        root      4012     1  0  2010 ?        Sl   213:13 /usr/sbin/snmpd -Lsd -Lf /dev
        root      4063     1  0  2010 ?        Ss     0:03 crond
        root      4117     1  0  2010 ?        Ss     0:00 /usr/sbin/atd
        root      4826     1  0  2010 ?        Ss     0:04 /usr/sbin/httpd
        apache    9465  4826  0 Mar21 ?        S      0:00 /usr/sbin/httpd
        apache   27737  4826  0 Mar21 ?        S      0:00 /usr/sbin/httpd
        apache   28502  4826  0 Mar21 ?        S      0:00 /usr/sbin/httpd
        apache   29550  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29553  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29554  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29555  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29556  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29557  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29558  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
        apache   29560  4826  0 Mar20 ?        S      0:00 /usr/sbin/httpd
Since we now know that there are opened files under "/var", let's see
which particular files are opened.  We can do this by running a "for"
loop on the output of 'fuser', using the resulting variable as part of our
"/proc" path that we list out with 'ls'.  For those familiar with Solaris,
the loop below will produce output quite similar to that of the Solaris
'pfiles' command.  The output contains the PID, running process, and each
file under "/var" that the process currently has opened:
        cobblepot [0] for i in `/sbin/fuser -c /var 2>/dev/null` ; do
        > echo "${i}:  `/bin/cat /proc/${i}/cmdline`" ;
        > /bin/ls -ld /proc/${i}/fd/* | /bin/awk '/\/var/ {print "\t"$NF}' ; done
        3791:  auditd
                /var/log/audit/audit.log
        3824:  syslogd
                /var/log/messages
                /var/log/secure
                /var/log/maillog
                /var/log/cron
                /var/log/spooler
                /var/log/boot.log
        3995:  /usr/sbin/acpid
                /var/log/acpid
                /var/log/acpid
        <snip...>
Of interest, in the above output it appears that PID 3995, process
'acpid', is holding open "/var/log/acpid" twice.  That's because it is.
A subsequent 'ls' on "/proc/3995/fd/*" shows us that "/var/log/acpid"
is opened for writing both STDOUT (fd 1) and STDERR (fd 2):
        cobblepot [0] /bin/ls -ld /proc/3995/fd/* | /bin/grep '/var'
        l-wx------ 1 root root 64 Mar 23 15:35 /proc/3995/fd/1 -> /var/log/acpid
        l-wx------ 1 root root 64 Mar 23 15:35 /proc/3995/fd/2 -> /var/log/acpid

NOTES

Note 1:  The output from 'fuser -c /MOUNT_POINT' provides the mount point,
    followed by a list of PIDs, some of which have additional letter
    codes.  All 'fuser' output is sent to STDERR, except for PIDs
    which are sent to STDOUT.  The letter codes optionally following
    a PID indicate how a process is using a file.  From the "fuser(1)"
    man page, the letter codes are defined as:
        c       current directory.

        e       executable being run.

        f       open file. f is omitted in default display mode.

        F       open file for writing. F is omitted  in  default  display
                mode.

        r       root directory.

        m       mmap’ed file or shared library.
see also:
    Finding Open Files in Solaris
    Finding Open Files in FreeBSD
    Removing / Recovering an Open File (Linux)

2 comments:

Anonymous said...

what about 'lsof' ?

troy said...

Good question. Vic Abell's 'lsof' is an excellent tool for this. And while I've personally used 'lsof' quite a bit over the years and find it very useful, I also recall a time when I'd have to download and compile it if I wanted it on a box. Keeping with the UNIX mindset, my intent was to illustrate another way to list open files, especially if 'lsof' isn't available on the system.