Have you ever exhausted capacity on a volume when it still shows storage
space available? The following details such a situation where the
intention is to simply copy some man pages into a directory.
HOST INFO
host beastie
prompt beastie [0]
OS FreeBSD 8.1
FILESYSTEM /mnt/somevol
NOTE Though the example uses FreeBSD 8.1, the same
scenario and commands involved translate almost
identically to prior versions of FreeBSD, as
well as Linux, and Solaris.
DETAILS
We've dropped a tarball of man pages to /tmp/manpages.tar and need to
untar it into a directory under /mnt/somevol:
beastie [0] /bin/ls -Fa /mnt/somevol
./ ../ .snap/ app/ bin/ docs/ logs/ pkgs/
beastie [0] /bin/ls -Fa /mnt/somevol/docs
./ ../
beastie [0] /bin/df -h /mnt/somevol
Filesystem Size Used Avail Capacity Mounted on
/dev/da2s1a 496M 89M 367M 20% /mnt/somevol
beastie [0] /usr/bin/du -sh /tmp/manpages.tar
40K /tmp/manpages.tar
beastie [0] /bin/cp /tmp/manpages.tar /mnt/somevol/docs/.
beastie [0] cd /mnt/somevol/docs
beastie [0] /usr/bin/tar xf manpages.tar
/mnt/somevol: create/symlink failed, no inodes free
manpages/: Can't create 'manpages'
manpages/setfacl.1.gz: Failed to create dir 'manpages'Can't create 'manpages/setfacl.1.gz'
manpages/readelf.1.gz: Failed to create dir 'manpages'Can't create 'manpages/readelf.1.gz'
manpages/ld.so.1.gz: Failed to create dir 'manpages'Can't create 'manpages/ld.so.1.gz'
manpages/ld.1.gz: Failed to create dir 'manpages'Can't create 'manpages/ld.1.gz'
manpages/kenv.1.gz: Failed to create dir 'manpages'Can't create 'manpages/kenv.1.gz'
tar: Error exit delayed from previous errors.
beastie [1]
We started off with checking what directories were available under
/mnt/somevol, identifying 'docs' as our intended location for the
man pages. The filesystem shows plenty of space for our 40K tarball.
We copy over the tarball from /tmp, but when we try to untar it under
/mnt/somevol/docs, we get errors about being unable to extract the
tarball. Additionally, the first error we receive is about file / symlink
creation failed and no inodes free. So, let's check out our inodes:
beastie [1] /bin/df -i /mnt/somevol
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/da2s1a 507670 91350 375708 20% 65534 0 100% /mnt/somevol
Alright, 0 inodes free, how do our directories under /mnt/somevol look:
beastie [0] cd /mnt/somevol
beastie [0] /bin/ls -lFa
total 126
drwxr-xr-x 8 root wheel 512 Oct 30 23:12 ./
drwxr-xr-x 7 root wheel 512 Oct 7 01:54 ../
drwxrwxr-x 2 root operator 512 Oct 30 23:10 .snap/
drwxr-xr-x 3249 root wheel 113664 Oct 31 00:37 app/
drwxr-xr-x 2 root wheel 512 Oct 30 23:12 bin/
drwxr-xr-x 2 root wheel 512 Oct 31 16:34 docs/
drwxr-xr-x 2 root wheel 512 Oct 30 23:17 logs/
drwxr-xr-x 2 root wheel 512 Oct 30 23:12 pkgs/
3249 links under 'app', and given the rest of the directories have few
links, 'app' is likely our culprit. Let's see how 'app' looks:
beastie [0] /usr/bin/find ./app \( -type d -and \! -links 2 -exec /bin/ls -ld {} \; \)
drwxr-xr-x 3249 root wheel 113664 Oct 31 00:37 ./app
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:26 ./app/dir0
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:27 ./app/dir1
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:27 ./app/dir10
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:27 ./app/dir100
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:27 ./app/dir1000
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:27 ./app/dir1001
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:28 ./app/dir1002
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:28 ./app/dir1003
drwxr-xr-x 3249 root wheel 113664 Oct 31 16:28 ./app/dir1004
drwxr-xr-x 584 root wheel 9728 Oct 31 16:28 ./app/dir1005
drwxr-xr-x 4 root wheel 512 Oct 31 16:32 ./app/dir384
11 directories under 'app' with more than 2 links each, most with 3249
links, like 'app'. In totalling the link counts above, that's about
33078 links. These directories could be normal, so let's look further:
beastie [0] /usr/bin/find ./app \( -type d -and -links 2 \) -print | /usr/bin/wc -l
33048
A total of 33048 directories with only two links, meaning empty
directories. (The current directory (.) and parent directory (..) are
the only two links in these directories.) This means more than half of
our inodes are being used by empty directories. Let's have a further
look at the contents of 'app':
beastie [0] /bin/ls -Fa app
./ dir2166/ dir414/ file1584 file2754
../ dir2167/ dir415/ file1585 file2755
dir0/ dir2168/ dir416/ file1586 file2756
dir1/ dir2169/ dir417/ file1587 file2757
dir10/ dir217/ dir418/ file1588 file2758
dir100/ dir2170/ dir419/ file1589 file2759
dir1000/ dir2171/ dir42/ file159 file276
<snip...>
beastie [0] /bin/ls -l app/file1584
-rw-r--r-- 1 root wheel 0 Oct 30 23:20 app/file1584
Quite a few files and directories with a rather uniform name. A check of
'file1584', chosen at random, shows 0 bytes in size. I wonder how many
more empty files there are:
beastie [0] cd app
beastie [0] /usr/bin/find . \( -type f -and -size 0 \) | /usr/bin/wc -l
32470
Adding together the count of empty files (32470) and count of empty
directories (33048) gives us 65518 inodes in use for empty files, nearly
all available inodes for this filesystem. (Remember, everything in UNIX
is a file (even directories) and each file requires an inode. So why care
about inodes? Generically speaking, they contain pointers to the blocks
of data comprising a file. Without them, it would be fairly tedious to
locate, much less access the contents of a file.)
After reviewing the listing of empty files and directories and the
expected usage of this filesystem, we've determined that these files and
directories are unnecessary and shouldn't exist. So let's remove them,
starting with our empty files:
beastie [0] /usr/bin/find . \( -type f -and -size 0 \) -exec /bin/rm {} \;
beastie [0] /bin/df -i /mnt/somevol
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/da2s1a 507670 91350 375708 20% 33064 32470 50% /mnt/somevol
Half of our inodes reclaimed, let's kill out empty directories:
beastie [0] /usr/bin/find . \( -type d -and -links 2 \) -exec /bin/rmdir {} \;
find: ./dir0/dir0: No such file or directory
find: ./dir0/dir1: No such file or directory
find: ./dir0/dir10: No such file or directory
find: ./dir0/dir100: No such file or directory
find: ./dir0/dir1000: No such file or directory
find: ./dir0/dir1001: No such file or directory
find: ./dir0/dir1002: No such file or directory
<snip...>
beastie [0] /bin/df -i /mnt/somevol
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/da2s1a 507670 24244 442814 5% 10 65524 0% /mnt/somevol
beastie [0] /bin/ls -Fa
./ ../
The errors from find can be safely ignored, the directories are still
being removed. After the 'rmdir' completed, we can see almost all
of our inodes have been reclaimed. We also see that we've reclaimed
filesystem space. This is because each file requires at least a block of
storage, even if it's empty. Since we just removed all of the empties,
we also freed up storage space. The following review of our directories
under /mnt/somevol is to illustrate an additional point. After our
cleanup work, the 'app' directory is now empty, yet we see below that
'app' is 112 KB in size. This is because the inode for 'app' required
more space to retain data about each of the subsequent links under the
'app' directory. Assuming no further files are written to 'app', the
space listed should normalize over time.
beastie [0] cd /mnt/somevol
beastie [0] /bin/ls -lFa
total 126
drwxr-xr-x 8 root wheel 512 Oct 31 16:57 ./
drwxr-xr-x 7 root wheel 512 Oct 7 01:54 ../
drwxr-xr-x 2 root wheel 512 Oct 31 16:57 .snap/
drwxr-xr-x 2 root wheel 113664 Oct 31 17:08 app/
drwxr-xr-x 2 root wheel 512 Oct 30 23:12 bin/
drwxr-xr-x 2 root wheel 512 Oct 31 16:34 docs/
drwxr-xr-x 2 root wheel 512 Oct 30 23:17 logs/
drwxr-xr-x 2 root wheel 512 Oct 31 16:57 pkgs/
beastie [0] /usr/bin/du -sh ./*
112K ./app
2.0K ./bin
42K ./docs
24M ./logs
2.0K ./pkgs
So, now that we've cleaned up all of the bogus files and directories, we
can continue with our original plan, untar the contents of manpages.tar:
beastie [0] cd docs
beastie [0] ls
manpages.tar
beastie [0] /bin/ls -Fa
./ ../ manpages.tar
beastie [0] /usr/bin/tar xf manpages.tar
beastie [0] /bin/ls -Fa
./ ../ manpages/ manpages.tar
beastie [0] /usr/bin/find . -print
.
./manpages.tar
./manpages
./manpages/setfacl.1.gz
./manpages/readelf.1.gz
./manpages/ld.so.1.gz
./manpages/ld.1.gz
./manpages/kenv.1.gz
The situation above is likely old hat to an experienced UNIX sysadmin,
however, its still easy to get bit by it, especially if one is new to
UNIX administration. Without getting too far into filesystem internals,
filesystems such as UFS, ext[2|3], and FFS are all primarily bound by
two constraints. Those constraints are total available storage space on
the filesystem and the number of inodes allocated for use. While with
UFS and ext[2|3] we can grow our filesystem if we exhaust all free space
(and have contiguous space available to grow into), we cannot do the same
for inodes. The count of inodes are set during filesystem creation. The
only way to handle exhaustion of all available inodes is either remove
some files, as seen above, or create a new filesystem configured with an
appropriate number of inodes and move our data into it. As always, if
you have anything to add to this, such as from your experiences, or any
suggestions or questions, feel free to leave a comment.
