23 March 2011

Finding Open Files in Solaris

During a recent discussion, another sysadmin asked that I write him a
tool to identify any open files on a particular file system (FS) in a
Solaris environment.  Rather than reinvent the wheel, I provided him
with the steps outlined here.  Our host details are:
        HOST:           sunspot
        PROMPT:         sunspot [0]
        OS:             Solaris 10 (also works for Solaris 8 and 9)
For point of example, we'll be looking at open files in /var.  To start,
we see that /var is its own FS and use 'fuser' to see if any processes
have any opened files under "/var" (see note 1):
        sunspot [0] /usr/bin/df -h /var
        Filesystem             size   used  avail capacity  Mounted on
        /dev/dsk/c1t0d0s4      3.9G   1.1G   2.8G    30%    /var
        sunspot [0] /usr/sbin/fuser -c /var
        /var:    14967o   12517c    6198o   23322co   23164co   23124co   21751o   28035co \
          16191o   16189o   16187o   16185o   16183o   16181o   16179o   16177o   16175o \
          16173o   16163co   16160co    9961co    4338o    4337o    3432o    3431o   2767o \
           2763o    2143o    1687o    1650c    1649co    1594o    1591o    1590o    1583o \
           1580o    1535o    1504co     607o       7o
In the above, we have 40 separate processes, a decent amount to go
through.  If we wanted to know what these processes were, we could pass
off the output from 'fuser' to 'ps' as seen below:
        sunspot [0] /usr/bin/ps -o user,pid,comm -p "$(/usr/sbin/fuser -c /var 2>/dev/null)"
            USER     PID  CMD
            root       7  /lib/svc/bin/svc.startd
            root     607  /sbin/vxesd
            root    1504  /usr/sbin/cron
            root    1535  /usr/lib/fm/fmd/fmd
            root    1580  /usr/lib/saf/sac
            root    1583  /usr/lib/inet/inetd
            root    1590  /usr/sbin/syslogd
            root    1591  /usr/lib/utmpd
            root    1594  /usr/lib/saf/ttymon
            smmsp   1649  /usr/lib/sendmail
            root    1650  /usr/lib/sendmail
            root    1687  /opt/VRTSdcli/xprtl/bin/xprtld
            root    2143  /usr/sbin/vxdclid
            root    2763  /sbin/sh
            root    2767  /sbin/sh
            root    3431  /sbin/sh
            root    3432  vxnotify
            root    4337  /sbin/sh
            root    4338  vxnotify
            root    6198  /usr/sfw/sbin/snmpd
            root    9961  /opt/VRTSvcs/bin/CmdServer
            daemon 12517  /usr/lib/nfs/statd
            daemon 14967  /usr/lib/nfs/nfsmapid
        <snip...>
Since we now know that there are processes with opened files in "/var",
let's see which files are opened.  By passing the output of 'fuser' to
'pfiles' and piping it to 'egrep', we can get a list of the process ID
(PID), the executing binary, and files opened by that process.  Of note,
while 'ps' returns output numerically ordered by PID as seen above,
'pfiles' will return output ordered in the manner it received it:
        sunspot [0] /usr/bin/pfiles `/usr/sbin/fuser -c /var 2>/dev/null` |
        > /usr/bin/egrep '^[0-9]|/var'
        14967:  /usr/lib/nfs/nfsmapid
              /var/VRTSvcs/log/engine_A.log
              /var/VRTSvcs/ldf/engine_A.ldf
              /var/VRTSvcs/lock/.hadlock
              /var/VRTSvcs/lock/.hadtrylock2
              /var/VRTSvcs/lock/.hadargs
              /var/run/nfs4_domain
              /var/VRTSat/.VRTSat/profile/VRTSatlocal.conf.lock
              /var/VRTSat/.VRTSat/profile/certstore/keystore/KeyStore.lock
              /var/run/name_service_door
              /var/VRTSvcs/lock/.hashadow
              /var/VRTSvcs/lock/.hashadow1
        12517:  /usr/lib/nfs/statd
              /var/run/name_service_door
        6198:   /usr/sfw/sbin/snmpd
              /var/log/snmpd.log
              /var/run/name_service_door
        <snip...>

NOTES

Note 1:  The output from 'fuser -c /MOUNT_POINT' provides a the mount
    point, followed by a list of PIDs and letter codes.  All 'fuser'
    output is sent to STDERR, except for PIDs which are sent to STDOUT.
    The letter codes following a PID indicate how a process is using a
    file.  From the "fuser(1M)" man page, the letter codes are defined as:
        c        Indicates that the process is using the file as its
                 current directory.

        m        Indicates that the process is using a  file  mapped
                 with mmap(2). See mmap(2) for details.

        n        Indicates  that  the  process  is  holding  a  non-
                 blocking mandatory lock on the file.

        o        Indicates that the process is using the file as  an
                 open file.

        r        Indicates that the process is using the file as its
                 root directory.

        t        Indicates that the process is using the file as its
                 text file.

        y        Indicates that the process is using the file as its
                 controlling terminal.

see also:
    Removing / Recovering an Open File (Solaris)
    Finding Open Files in FreeBSD
    Finding Open Files in Linux