16 April 2011

Enabling Local FS Quotas in Linux

Quotas allow you to limit the amount of space a user or group can use
and / or the number of files a user or group can create on a filesystem
(FS).  The following shows how to set up these quotas and the expected
user experience.  Our host details for this are:
        HOST:           tux
        PROMPT:         user@tux [0]
        OS:             CentOS 5.4 Linux
        GROUPS:         sysvuser (1000), bsduser (1001)
SET UP
To start, check to make sure that the current kernel will support quotas
and the quota package is installed:
        root@tux [0] /bin/grep CONFIG_QUOTA /boot/config-`/bin/uname -r`   
        CONFIG_QUOTA=y 
        CONFIG_QUOTACTL=y
        root@tux [0] /bin/rpm -qa quota
        quota-3.13-1.2.5.el5
Next, we need to enable quotas on the filesystem in question.  Since
/users is only about 500 MB in size and our users can write to it,
quotas would come in handy here.  Below, we use 'df' to check the size
and usage of /users and see that quotas are not currently available in
"/etc/fstab" or "/proc/mounts":
        root@tux [0] /bin/df -h /users
        Filesystem            Size  Used Avail Use% Mounted on
        /dev/sdc1             496M   12M  460M   3% /users
        [0] grep '/users' /etc/fstab
        /dev/sdc1       /users       ext3    defaults        1 2
        root@tux [0] /bin/grep '/users' /proc/mounts
        /dev/sdc1 /users ext3 rw,data=ordered 0 0
Now, update "/etc/fstab", appending ',usrquota,grpquota' after 'defaults'
(see note 1) to the /usrs entry.  The entry after writing "/etc/fstab" can
be seen below.  After a remount of /users to enable the new FS options,
verify those options are now enabled by check "/proc/mounts":
        root@tux [0] /bin/grep '/users' /etc/fstab
        /dev/sdc1       /users       ext3    defaults,usrquota,grpquota      1 2
        root@tux [0] /bin/mount -o remount /users
        root@tux [0] /bin/grep '/users' /proc/mounts
        /dev/sdc1 /users ext3 rw,data=ordered,usrquota,grpquota 0 0
Though the /users FS has been mounted to support quotas, they are not
currently enabled.  We can see this via 'quotaon -p'.  A second run of
'quotaon' will enable quotas on /users:
        root@tux [0] /sbin/quotaon -p /users
        group quota on /users (/dev/sdc1) is off
        user quota on /users (/dev/sdc1) is off
        root@tux [0] /sbin/quotaon -v /users
        quotaon: Cannot find quota file on /users [/dev/sdc1] to turn quotas on/off.
        quotaon: Cannot find quota file on /users [/dev/sdc1] to turn quotas on/off.
In the above, 'quotaon' (-v for verbose) basically throws an error
because there are no quota files.  We can use 'quotacheck' to create them
in the top level of /users FS:
        root@tux [0] /sbin/quotacheck -cugmv /users
        quotacheck: Scanning /dev/sdc1 [/users]
        quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Cannot stat old group quota file: No such file or directory
        quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Cannot stat old group quota file: No such file or directory
        done
        quotacheck: Checked 7 directories and 2 files
        quotacheck: Old file not found.
        quotacheck: Old file not found.
        root@tux [0] /bin/ls /users
        aquota.group  aquota.user  bjoy  calif  lost+found  scottm  troy
        root@tux [0] /bin/ls -l /users/aquota*
        -rw------- 1 root root 7168 Apr 15 01:47 /users/aquota.group
        -rw------- 1 root root 7168 Apr 15 01:47 /users/aquota.user
Both aquota.group and aquota.user are binary files used to store any
configured quotas.  Of note, on system boot, 'quotacheck' and 'quotaon'
will run against any FS in "/etc/fstab" with any quota options (quota
(same as usrquota), usrquota, or grpquota) to enable the appropriate
quotas.  Now that the quota files have been created, a subsequent run
of 'quotaon' will enable quotas on /users, which we again verify with
'quotaon -p':
        root@tux [0] /sbin/quotaon -v /users
        /dev/sdc1 [/users]: group quotas turned on
        /dev/sdc1 [/users]: user quotas turned on
        root@tux [0] /sbin/quotaon -p /users
        group quota on /users (/dev/sdc1) is on
        user quota on /users (/dev/sdc1) is on
With user and group quotas enabled, we can start setting actual quotas.
There are two commands that we can use, 'edquota' which provides quota
editing via 'vi', or 'setquota' which allows quota attributes to be set
via the command line.  For the examples that follow, only group quotas
are configured (-g) though the handling is similar for setting individual
user quotas (-u):
        root@tux [0] /usr/sbin/edquota -g 1000 -f /users
        Disk quotas for group 1000 (gid 1000):
          Filesystem                   blocks       soft       hard     inodes     soft     hard
          /dev/sdc1                         4       5120          0          2        5        0
The above sets a quota for gid 1000 (group "sysvuser") limiting the
entire group to only 5 MB of space or a total of 5 files.  The breakdown
of the above shows that there are currently 4 1KB blocks in use with a
soft limit of 5120 1KB blocks (or 5 MB) with no hard limit.  There are
currently 2 files (or inodes) in use with a soft limit of 5 and no
hard limit.  The same could have been achieved with the following
'setquota' invocation:
        root@tux [0] /usr/sbin/setquota -g 1000 5120 0 5 0 /users
Of note, setting a quota does not enable quotas.  If quotas are not
enabled as verified by 'quotaon', the set quota will have no effect.
Also, quotas are set using 1KB blocks, regardless of FS block size.
For instance, the block size for the /users FS (sdc1) is 2KB blocks:
        root@tux [0] /sbin/tune2fs -l /dev/sdc1 | /bin/grep '^Block size:'
        Block size:               2048
The following review of the newly set group quotas show a grace time
of 7 days.  This means that users within a group with a quota set can
continue to exceed the quota with no limit for a period of 7 days,
after which time, they would be prevented from any further exception.
Setting a hard limit would still allow users to exceed the soft limit
for 7 days, though only up to the hard limit, at which point they would
again be prevented from any further exception regardless of grace time:
        root@tux [0] /usr/sbin/repquota -g /users
        *** Report for group quotas on device /dev/sdc1
        Block grace time: 7days; Inode grace time: 7days
                                Block limits                File limits
        Group           used    soft    hard  grace    used  soft  hard  grace
        ----------------------------------------------------------------------
        root      --   11292       0       0              4     0     0
        sysvuser  --       4    5120       0              2     5     0 
        bsduser   --       4       0       0              2     0     0
Since 7 days is a little long to wait when doing a write-up, we'll set
the grace period to 5 minutes (300 seconds) for both block and file
limits for all groups ('setquota' only accepts durations in seconds):
        root@tux [0] setquota -tg 300 300 /users 
        root@tux [0] repquota -g /users              
        *** Report for group quotas on device /dev/sdc1
        Block grace time: 00:05; Inode grace time: 00:05
                                Block limits                File limits
        Group           used    soft    hard  grace    used  soft  hard  grace
        ----------------------------------------------------------------------
        root      --   11292       0       0              4     0     0
        sysvuser  --       4    5120       0              2     5     0
        bsduser   --       4       0       0              2     0     0
USER EXPERIENCE
With our group quota set up, let's see how our users fare.  Under /users,
several accounts have writable directories, 2 users in group "bsduser"
(1001) and 2 in group "sysvuser" (1000).  Also, 1 user, "bjoy", has a
secondary group of "sysvuser":
        root@tux [0] for i in bjoy calif scottm troy ; do /usr/bin/id -a ${i} ; done
        uid=1003(bjoy) gid=1001(bsduser) groups=1001(bsduser),1000(sysvuser)
        uid=1001(calif) gid=1001(bsduser) groups=1001(bsduser)
        uid=1002(scottm) gid=1000(sysvuser) groups=1000(sysvuser)
        uid=1000(troy) gid=1000(sysvuser) groups=1000(sysvuser)
        root@tux [0] /bin/ls -l /users
        total 24
        drwx------ 2 bjoy   bsduser   2048 Apr 15 01:42 bjoy
        drwx------ 2 calif  bsduser   2048 Apr 15 01:42 calif
        drwx------ 2 root   root     16384 Apr 15 01:40 lost+found
        drwx------ 2 scottm sysvuser  2048 Apr 15 01:42 scottm
        drwx------ 2 troy   sysvuser  2048 Apr 15 01:42 troy 
User "scottm" has logged on, changes directory to "/users/scottm"
and creates 9 files.  The quota we set earlier allows users in group
"sysvuser" to only create 5 files:
        scottm@tux [0] cd /users/scottm
        scottm@tux [0] /bin/date ; /bin/touch f1 f2 f3 f4 f5 f6 f7 f8 f9 ; sleep 300 ; \
        > /bin/date ; /bin/touch f10
        Fri Apr 15 02:15:47 EDT 2011
        sdc1: warning, group file quota exceeded.
        Fri Apr 15 02:20:47 EDT 2011
        sdc1: write failed, group file quota exceeded too long.
        touch: cannot touch `f10': Disk quota exceeded
        scottm@tux [1] /bin/ls
        f1  f2  f3  f4  f5  f6  f7  f8  f9
        scottm@tux [0] 
While "scottm" received a warning about exceeding the group quota,
he was still able to create his original 9 files.  After the 5 minute
grace period expired, he was prohibited from creating another file "f10"
and received an error stating as much.  Since the original 9 files were
created prior to the end of the grace period, they are still available.
As "root", we can review the current group quota and see that group
"sysvuser" has exceeded its file limits:
        root@tux [0] /usr/sbin/repquota -g /users
        *** Report for group quotas on device /dev/sdc1
        Block grace time: 00:05; Inode grace time: 00:05
                                Block limits                File limits
        Group           used    soft    hard  grace    used  soft  hard  grace
        ----------------------------------------------------------------------
        root      --   11292       0       0              4     0     0
        sysvuser  -+       4    5120       0             11     5     0   none
        bsduser   --       4       0       0              2     0     0

        root@tux [0]
In the above 'repquota' output, we can immediately see if a group (or
user) has exceeded their quota in the column immediately following the
group (or user) name, normally represented by "--".  The first hyphen
represents the block quota and the second is the file quota.  If either
are exceeded, a plus (+) replaces the original hyphen.  Above, we see that
group "sysvuser" has exceeded the file quota but not the block quota (-+).

Returning back to "scottm", he removes his previous files and now goes
for exceeding the block quota:
        scottm@tux [0] /bin/rm f*
        scottm@tux [0] /bin/date ; for i in 1 2 3; do
        > /bin/dd if=/dev/zero of=${i}-2mb bs=1024 count=2048; done; sleep 300 ; \
        > /bin/date ; /bin/dd if=/dev/zero of=4-2mb bs=1024 count=2048
        Fri Apr 15 02:32:06 EDT 2011
        2048+0 records in
        2048+0 records out
        2097152 bytes (2.1 MB) copied, 0.0183195 seconds, 114 MB/s
        2048+0 records in
        2048+0 records out
        2097152 bytes (2.1 MB) copied, 0.0167435 seconds, 125 MB/s
        sdc1: warning, group block quota exceeded.
        2048+0 records in
        2048+0 records out
        2097152 bytes (2.1 MB) copied, 0.0664112 seconds, 31.6 MB/s
        Fri Apr 15 02:37:07 EDT 2011
        sdc1: write failed, group block quota exceeded too long.
        dd: writing `4-2mb': Disk quota exceeded
        1+0 records in
        0+0 records out
        0 bytes (0 B) copied, 0.00530102 seconds, 0.0 kB/s
        scottm@tux [1] /bin/ls
        1-2mb  2-2mb  3-2mb  4-2mb
        scottm@tux [0] /usr/bin/du -sh ./*
        2.1M    ./1-2mb
        2.1M    ./2-2mb
        2.1M    ./3-2mb
        0       ./4-2mb
        scottm@tux [0]
User "scottm" is again able to exceed the group quota, writing about 6
MB of data to 3 files.  He is again warned about it, this time regarding
the block quota.  After the grace period expires, while he can create a
4th file, he cannot write to it.  Running 'quota' would allow "scottm"
to review the quotas enabled for him and his group to see where he
currently stands:
        scottm@tux [0] /usr/bin/quota -ug
        Disk quotas for user scottm (uid 1002): none
        Disk quotas for group sysvuser (gid 1000):
             Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
              /dev/sdc1    6162*   5120       0    none       6*      5       0    none
        scottm@tux [1]
Next, user "bjoy" tries to create a file and succeeds.  Since "bjoy"
is in another group (bsduser) and quotas only apply to primary groups,
he sees no issue:
        bjoy@tux [0] /usr/bin/id -a
        uid=1003(bjoy) gid=1001(bsduser) groups=1000(sysvuser),1001(bsduser)
        bjoy@tux [0] cd /users/bjoy ; /bin/dd if=/dev/zero of=1-2mb bs=1024 count=2048
        2048+0 records in
        2048+0 records out
        2097152 bytes (2.1 MB) copied, 0.0188318 seconds, 111 MB/s
User "troy", in group "sysvuser", however, cannot create a new file since
the quota for the group has been exceeded, in this case by another user
in the group:
        troy@tux [0] cd /users/troy
        troy@tux [0] /bin/dd if=/dev/zero of=1-2mb bs=1024 count=2048
        sdc1: write failed, group file quota exceeded too long.
        dd: opening `1-2mb': Disk quota exceeded
        troy@tux [1] /bin/ls -al
        total 4
        drwx------ 2 troy sysvuser 2048 Apr 15 01:42 .
        drwxr-xr-x 7 root root     2048 Apr 15 01:47 ..
        troy@tux [0] 
Interestingly, if "root" was to check on an existing quota via 'repquota'
and the grace period hasn't yet expired, the grace period will show a
countdown timer of how much grace time is left for that quota.  Here the
timer shows 4 minutes left:
        root@tux [0] /usr/bin/du -sh /users/scottm/*
        6.1M    /users/scottm/1-6mb
        root@tux [0] /bin/ls -l /users/scottm/*
        -rw-r--r-- 1 scottm sysvuser 6291456 Apr 15 02:50 /users/scottm/1-6mb
        root@tux [0] /usr/sbin/repquota -g /users
        *** Report for group quotas on device /dev/sdc1
        Block grace time: 00:05; Inode grace time: 00:05
                                Block limits                File limits
        Group           used    soft    hard  grace    used  soft  hard  grace
        ----------------------------------------------------------------------
        root      --   11292       0       0              4     0     0
        sysvuser  +-    6162    5120       0  00:04       3     5     0
        bsduser   --    2058       0       0              3     0     0

        root@tux [0]
In the next example, one quota type (block) has been exceeded while the
other (file) hasn't, allowing "troy" to create a file but not write to it:
        troy@tux [0] /bin/dd if=/dev/zero of=1-2mb bs=1024 count=2048
        sdc1: write failed, group block quota exceeded too long.
        dd: writing `1-2mb': Disk quota exceeded
        1+0 records in  
        0+0 records out 
        0 bytes (0 B) copied, 0.00600435 seconds, 0.0 kB/s
        troy@tux [1] /bin/ls -al 
        total 4
        drwx------ 2 troy sysvuser 2048 Apr 15 03:00 .
        drwxr-xr-x 7 root root     2048 Apr 15 01:47 ..
        -rw-r--r-- 1 troy sysvuser    0 Apr 15 03:00 1-2mb
        troy@tux [0]
If so desired, "root" can disable either user or group quotas "on the
fly" via 'quotaoff' or 'quotaon -f'.  While the respective quota is still
configured and viewable, it is no longer enabled or in effect, allowing
"troy" to write his 2MB file:
        root@tux [0] /sbin/quotaoff -g /users
        root@tux [0] /sbin/quotaon -p /users
        group quota on /users (/dev/sdc1) is off
        user quota on /users (/dev/sdc1) is on
        root@tux [1] /usr/sbin/repquota -g /users
        *** Report for group quotas on device /dev/sdc1
        Block grace time: 00:05; Inode grace time: 00:05
                                Block limits                File limits
        Group           used    soft    hard  grace    used  soft  hard  grace
        ----------------------------------------------------------------------
        root      --   11292       0       0              4     0     0
        sysvuser  +-    6162    5120       0   none       4     5     0
        bsduser   --    2058       0       0              3     0     0

        root@tux [0]

        troy@tux [0] /bin/dd if=/dev/zero of=1-2mb bs=1024 count=2048
        2048+0 records in
        2048+0 records out
        2097152 bytes (2.1 MB) copied, 0.02708 seconds, 77.4 MB/s
        troy@tux [0] /bin/ls -al
        total 2058
        drwx------ 2 troy sysvuser    2048 Apr 15 03:00 .
        drwxr-xr-x 7 root root        2048 Apr 15 01:47 ..
        -rw-r--r-- 1 troy sysvuser 2097152 Apr 15 03:03 1-2mb
        troy@tux [0]



NOTES

Note 1: /etc/fstab options:
        root@tux [0] /bin/grep '/users' /etc/fstab
        /dev/sdc1       /users       ext3    defaults,usrquota,grpquota      1 2

    usrquota:   enable user quotas
    grpquota:   enable group quotas
    quota:      same as usrquota
    defaults:   same as using FS options rw, suid, dev, exec, auto, nouser, and async

No comments: