24 July 2011

Process Memory Usage

Since resources on a host are finite, it can be useful to know the
impact of a process or processes on those resources.  In this instance,
we'll take a look at a process' use of memory.  Our details for this are:
        HOSTs:          cobblepot (Linux) 
                        berkeley (FreeBSD)
                        sunspot (Solaris)
        PROMPT:         host [0]
        OSes:           CentOS 5.6 (RedHat Linux clone)
                        FreeBSD 8.1
                        Solaris 10 u8 (10/09)
Determining the memory usage of a process should be trivial, just use
'prstat' (Solaris), 'top' (Linux, Freebsd), or 'ps' (all).  For our
purposes, we'll check out a shell (ksh for Solaris and Linux, csh for
FreeBSD).  Using our commands above, for Solaris:
    
        sunspot [0] /usr/bin/prstat -p 529 1 1 | /usr/bin/grep -v ^$
           PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
           529 root     1584K  972K sleep   59    0   0:00:00 0.0% ksh/1
        Total: 1 processes, 1 lwps, load averages: 0.00, 0.00, 0.00
        sunspot [0] /usr/bin/ps -p 529 -o pid,user,vsz,rss,s=STATE -o pri,nice=NICE \
        > -o time,pcpu,comm
          PID     USER  VSZ  RSS STATE PRI NICE        TIME %CPU COMMAND
          529     root 1584  972     S  59   20       00:00  0.0 /bin/ksh
For FreeBSD:
    
        berkeley [0] /usr/bin/top -d 1 | /usr/bin/egrep 'PID|1399'
          PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
         1399 root        1  44    0  4668K  2332K ttyin    0:00  0.00% csh
        berkeley [0] /bin/ps -p 1399 -o pid,user,nlwp,pri,nice=NICE \
        > -o vsz,rss,state,time,%cpu,comm
          PID USER NLWP PRI NICE   VSZ   RSS STAT      TIME %CPU COMMAND
         1399 root    1  44    0  4668  2332 Is+    0:00.01  0.0 csh
For Linux:
        cobblepot [0] /usr/bin/top -n 1 -p 5024 | /bin/egrep 'PID|5024'
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
         5024 root      15   0 64912 1492 1224 S  0.0  0.1   0:00.03 ksh
        cobblepot [0] /bin/ps -p 5024 -o pid,user,pri,nice=NICE \
        > -o vsz,rss,state,%cpu,%mem,time,cmd
          PID USER     PRI NICE    VSZ   RSS S %CPU %MEM     TIME CMD
         5024 root      24    0  64912  1492 S  0.0  0.1 00:00:00 /bin/ksh
In each example above, with either 'prstat' / 'top' or 'ps', we can see
the amount of memory the kernel has made available to a process (SIZE,
VSZ, VIRT) and the amount of process memory currently backed by physical
memory (RSS, RES).  (All memory values above are reported in kilobytes
(KB)).  Unfortunately, this doesn't consider the whole picture if we
are looking at groups of processes or all processes on the system.
To illustrate, totalling the allocated memory of all processes on
"cobblepot" compared to the total memory on the host shows almost 728 MB
more memory allocated than is actually available on the host:
        cobblepot [0] x=0
        cobblepot [0] for i in `ps h -e -o vsz,rss | /bin/awk '{print $1}'` ; do \
        > let x=x+i ; done ; echo $x
        1771332
        cobblepot [0] /bin/grep MemTotal /proc/meminfo
        MemTotal:      1026140 kB
Looking at resident memory shows almost 100 MB in use:
        cobblepot [0] x=0
        cobblepot [0] for i in `ps h -e -o vsz,rss | /bin/awk '{print $2}'` ; do \
        > let x=x+i ; done ; echo $x
        102360
Aside from the kernel overcommitting memory as seen in the first example,
the resident memory in the second example doesn't present an entirely
accurate picture, either.  This is because processes can share memory
segments of like pages of memory.  Using shell processes as an example,
each shell executable from above (ksh / csh) is linked against libc.
Rather than each shell instance mapping new segments of memory to load
libc, libc can be mapped once into memory and those segments shared for
subsequent shell instances.  To get a better understanding, we turn to
'pmap', available in the following packages:
        Solaris (pre Solaris 11):       SUNWesu
        Solaris (Solaris 11):           system/extended-system-utilities
        CentOS (Linux):                 procps-3.2.7-16.el5
        FreeBSD (see note 1):           pmap-20060622_1 (/usr/ports/sysutils/pmap)
Using our initial shell processes again and using 'pmap' on Solaris,
we get:
        sunspot [0] /usr/bin/pmap -x 529
        529:    /bin/ksh
         Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
        08045000      12      12      12       - rw---    [ stack ]
        08050000     164     164       -       - r-x--  ksh
        08089000       4       4       4       - rw---  ksh
        0808A000      32      32      32       - rw---    [ heap ]
        FEE40000      16      16       -       - r-x--  en_US.ISO8859-1.so.3
        FEE53000       8       8       -       - rwx--  en_US.ISO8859-1.so.3
        FEE60000      16      16       -       - r-x--  en_US.ISO8859-15.so.3
        FEE73000       8       8       -       - rwx--  en_US.ISO8859-15.so.3
        FEE80000    1076    1048       -       - r-x--  libc.so.1
        FEF90000      24      12       4       - rwx--    [ anon ]
        FEF9D000      32      32      20       - rw---  libc.so.1
        FEFA5000       8       8       4       - rw---  libc.so.1
        FEFC0000       4       4       -       - rwx--    [ anon ]
        FEFC3000     160     160       -       - r-x--  ld.so.1
        FEFEC000       4       4       -       - rwxs-    [ anon ]
        FEFF0000       4       4       4       - rwx--    [ anon ]
        FEFFB000       8       8       4       - rwx--  ld.so.1
        FEFFD000       4       4       4       - rwx--  ld.so.1
        -------- ------- ------- ------- -------
        total Kb    1584    1544      88       -
On FreeBSD:
        berkeley [0] /usr/local/bin/pmap 1679
        1679:   -csh
        Address   Kbytes     RSS  Shared    Priv Mode  Mapped File
        08048000     304     284     304       - r-x   /bin/csh
        08094000      12      12       -      12 rw-   /bin/csh
        08097000     420      36       -     420 rw-     [ anon ]
        28094000     188     172     188       - r-x   /libexec/ld-elf.so.1
        280C3000       8       8       -       8 rw-   /libexec/ld-elf.so.1
        280C5000      76      56       -      76 rw-     [ anon ]
        280D8000     244     244     244       - r-x   /lib/libncurses.so.8
        28115000      12      12       -      12 rw-   /lib/libncurses.so.8
        28118000      28      28      28       - r-x   /lib/libcrypt.so.5
        2811F000       4       4       -       4 rw-   /lib/libcrypt.so.5
        28120000      68       0       -      68 rw-     [ anon ]
        28131000    1012     784    1012       - r-x   /lib/libc.so.7
        2822E000      24      24       -      24 rw-   /lib/libc.so.7
        28234000      92      48       -      92 rw-     [ anon ]
        28300000    1024     516       -    1024 rw-     [ anon ]
        28400000    1024      12       -    1024 rw-     [ anon ]
        BFBE0000     128      40       -     128 rwx     [ anon ]
        -------- ------- ------- ------- -------
        Total Kb    4668    2280    1776    2892
On Linux:
        cobblepot [0] /usr/bin/pmap -x 5024
        5024:   /bin/ksh
        Address           Kbytes     RSS   Dirty Mode   Mapping
        0000000000400000    1200     740       0 r-x--  ksh93
        000000000072b000      72      68      24 rw---  ksh93
        000000000073d000      20       8       8 rw---    [ anon ]
        000000000093c000       4       4       0 rw---  ksh93
        000000000f0a2000     160     128     128 rw---    [ anon ]
        0000003942600000     112      96       0 r-x--  ld-2.5.so
        000000394281b000       4       4       4 r----  ld-2.5.so
        000000394281c000       4       4       4 rw---  ld-2.5.so
        0000003942a00000    1336     224       0 r-x--  libc-2.5.so
        0000003942b4e000    2048       0       0 -----  libc-2.5.so
        0000003942d4e000      16      16       4 r----  libc-2.5.so
        0000003942d52000       4       4       4 rw---  libc-2.5.so
        0000003942d53000      20      12      12 rw---    [ anon ]
        0000003942e00000       8       8       0 r-x--  libdl-2.5.so
        0000003942e02000    2048       0       0 -----  libdl-2.5.so
        0000003943002000       4       4       4 r----  libdl-2.5.so
        0000003943003000       4       4       4 rw---  libdl-2.5.so
        0000003943200000     520      12       0 r-x--  libm-2.5.so
        0000003943282000    2044       0       0 -----  libm-2.5.so
        0000003943481000       4       4       0 r----  libm-2.5.so
        0000003943482000       4       4       4 rw---  libm-2.5.so
        00002b1e5ee02000       8       8       8 rw---    [ anon ]
        00002b1e5ee10000       8       8       8 rw---    [ anon ]
        00002b1e5ee12000   55128      52       0 r----  locale-archive
        00002b1e623e8000      28      24       0 r--s-  gconv-modules.cache
        00007fff2333f000      84      52      52 rw---    [ stack ]
        00007fff233fc000      16       4       0 r-x--    [ anon ]
        ffffffffff600000    8192       0       0 -----    [ anon ]
        ----------------  ------  ------  ------ 
        total kB           73100    1492     268 
All three 'pmap' examples display effectively the same information:
        - Address:              virtual address of memory mapping
        - Kbytes:               total virtual memory size in KB
        - RSS:                  amount of memory segment currently backed by physical memory
        - Shared:               amount of memory segment that could be shared (FreeBSD only)
        - Anon|Priv|Dirty:      amount of memory that won't be shared with any other process
        - Locked:               amount of memory which will not be paged out to disk (Solaris only)
        - Mode:                 virtual memory permissions for each mapping
        - Mapped File|Mapping:  the descriptive name for each mapping or the file name
Using our Linux output above, our 'ksh' process has 73100 KB (71.39 MB)
of allocated memory, 1492 KB (1.46 MB) of it is resident.  Based on
RSS, for just one 'ksh' process we know that our 'ksh' process is using
1.46 MB of physical memory.  Each additional 'ksh' process would have
roughly the same values of allocated memory and RSS.  Due to shared
memory segments, however, each additional 'ksh' process would only use
around 268 KB of memory, which is the private memory to the respective
process.  The rest of the process memory is shared, having been mapped
by the original 'ksh' process.  (Shared memory segments will be unmapped
when the last referencing process deallocates them.) Assuming we were
to start five more 'ksh' processes, for a total of six 'ksh' processes,
we could estimate that we would only use about 2.77 MB of memory total.
This breaks down to:
        [Anon|Priv|Dirty] * (process_count - 1) + RSS(once) = total
        268 KB * (6 - 1) + 1492 KB = 2832 KB
        1340 KB + 1492 KB = 2832 KB
        2832 / 1024 = 2.77 MB
So while our original 'ps' output showed the RSS for our 'ksh' process
accurately at 1492 KB, additional 'ksh' processes, would actually be
using less memory.



Note 1:  In FreeBSD versions prior to 8, 'pmap' should install from
    /usr/ports without issue.  With FreeBSD 8, changes were made to
    SYSCALL_MODULE().  This won't prohibit 'pmap' installation, however
    without patching, 'pmap' will be unusable and the following can
    be expected:
        berkeley [0] /usr/local/bin/pmap 877
        pmap: pmap_helper module loaded but not found: No such file or directory
        berkeley [1] /usr/local/bin/pmap 877
        pmap: unable to load pmap_helper module: File exists
        berkeley [1]
    In a post to "freebsd-hackers", Selphie Keller, identified this issue
    and posted the following patch:
        --- /usr/ports/sysutils/pmap/work/pmap/pmap/pmap.c.orig 2011-07-19 23:39:44.000000000 -0400
        +++ /usr/ports/sysutils/pmap/work/pmap/pmap/pmap.c      2011-07-19 23:43:06.000000000 -0400
        @@ -83,18 +83,20 @@
                char    *prog_wild = NULL, *prog_name;
                regex_t preg;
                struct pargs *pa = NULL;
        +/* Begin Selphie Keller's patch posted to freebsd-hackers on 26 Oct 2010 for FreeBSD 8.x */
             struct kinfo_proc *kp;
             int        pmap_helper_syscall;

        -    if ((modid = modfind("pmap_helper")) == -1) {
        +    if ((modid = modfind("sys/pmap_helper")) == -1) {
                        /* module not found, try to load */
                        modid = kldload("pmap_helper.ko");
                        if (modid == -1)
                                err(1, "unable to load pmap_helper module");
        -               modid = modfind("pmap_helper");
        +               modid = modfind("sys/pmap_helper");
                        if (modid == -1)
                                err(1, "pmap_helper module loaded but not found");
                }
        +/* End Selphie Keller's patch posted to freebsd-hackers on 26 Oct 2010 for FreeBSD 8.x */

                stat.version = sizeof(stat);
                modstat(modid, &stat);
    The above patch is also available here.  To use the patch, place the
    contents into /usr/ports/sysutils/pmap/files/patch-pmap.c prior to
    installation of 'pmap' via /usr/ports.  After installing the patched
    'pmap' package, 'pmap' should function with no further issues.
    Selphie Keller's original post to "freebsd-hackers" can be found here.