19 January 2011

Missing Linux inittab

Since I'm somewhat on a "break the boot process" kick, I've deliberately
removed '/etc/inittab' on a Linux host.  As the following illustrates,
a missing 'inittab' really isn't as bad as it seems.  Our details for
this setup are:
        HOST:           tux
        PROMPTS:        [sh-3.2# |tux [0] ]
        OS:             CentOS 5.4 Linux
We start off with booting the host normally.  As expected, we get through
the BIOS, we are presented with our GRUB options, and the default selection
(<===) is booted after the timeout:
         GNU GRUB  version 0.97  (639K lower / 1047488K upper memory)

        CentOS (2.6.18-164.el5)                  <===========================
        CentOS Mirror (2.6.18-164.el5)


           Use the <up> and <down> keys to select which entry is highlighted.
           Press enter to boot the selected OS, 'e' to edit the
           commands before booting, 'a' to modify the kernel arguments
           before booting, or 'c' for a command-line.

           The highlighted entry will be booted automatically in 5 seconds.

    GRUB menu clears from the screen, and default option is booted:

          Booting 'CentOS (2.6.18-164.el5)'

        root (hd0,0)
         Filesystem type is ext2fs, partition type 0x83
        kernel /boot/vmlinuz-2.6.18-164.el5 ro root=LABEL=/1
           [Linux-bzImage, setup=0x1d00, size=0x1c31b4]
        initrd /boot/initrd-2.6.18-164.el5.img
           [Linux-initrd @ 0x37d73000, 0x27c402 bytes]

        <snip...>

        INIT: version 2.86 booting
        INIT: No inittab file found

        Enter runlevel:
Above, we get partway through system bootup when 'init' errors, not
finding '/etc/inittab' and prompting us to select a runlevel.  Our only
real option here is "s", or single user mode:
        Enter runlevel: s
After selecting the run level, we are dropped to a shell to start
resolving the situation.  A quick check below verifies that '/etc/inittab'
doesn't exist.  A further check of our mounted filesystems shows '/'
mounted.  Time to fix '/etc/inittab':
        sh-3.2# /bin/ls -ld /etc/inittab
        /bin/ls: /etc/inittab: No such file or directory
        sh-3.2# /bin/mount
        /dev/sda1 on / type ext3 (rw)
        proc on /proc type proc (rw)
        sysfs on /sys type sysfs (rw)
        devpts on /dev/pts type devpts (rw,gid=5,mode=620)
        /dev/sda2 on /var type ext3 (rw)
        tmpfs on /dev/shm type tmpfs (rw)
        none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
        sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

        mount: warning /etc/mtab is not writable (e.g. read-only filesystem).
               It's possible that information reported by mount(8) is not
               up to date. For actual information about system mount points
               check the /proc/mounts file.

        sh-3.2# echo "id:3:initdefault:" >> /etc/inittab
        sh: /etc/inittab: Read-only file system
Unfortunately, the error from our 'echo' statement tells us we should
have paid more attention to the output from 'mount'.  With '/etc/mtab'
unwritable, the output from 'mount' is unreliable.  We are actually
seeing stale data from '/etc/mtab'.  Following as directed, we check
'/proc/mounts' and see that '/' is mounted read only.  A quick remount of
'/' resolves this issue so we can create a new 'inittab':
        sh-3.2# /bin/cat /proc/mounts
        rootfs / rootfs rw 0 0
        /dev/root / ext3 ro,data=ordered 0 0
        /dev /dev tmpfs rw 0 0
        /proc /proc proc rw 0 0
        /sys /sys sysfs rw 0 0
        sh-3.2# /bin/mount -o remount,rw /
        EXT3 FS on sda1, internal journal
Using your favorite text editor (vi?), populate '/etc/inittab' with an
appropriate configuration.  If this is a server, the following contents
are the minimum needed to get the host back up and functional:
        sh-3.2# /bin/cat /etc/inittab
        id:3:initdefault:
        si::sysinit:/etc/rc.d/rc.sysinit
        l3:3:wait:/etc/rc.d/rc 3
        l6:6:wait:/etc/rc.d/rc 6
        1:2345:respawn:/sbin/mingetty tty1
Of note, runlevel 6 is included because we will need it in a moment
to reboot the host.  Without it, we'd get back deceptive messages that
processes are being killed.  Without 'init' knowing what rc scripts to
call, not much is actually occuring to bring down the system, though
your terminal will likely hang.  Additionally, we've configured a tty
on the last line so that when the host comes back up, we can log into
the box and do any additional cleanup.  Without this line, no tty will
be spawned.  When we reboot the host below, we get various errors though
these can be safely ignored for the moment:
        sh-3.2# /usr/bin/reboot
        WARNING: could not determine runlevel - doing soft reboot
          (it's better to use shutdown instead of reboot from the command line)
        shutdown: warning: cannot open /var/run/shutdown.pid
        INIT: Switching to runlevel: 6
        INIT: Sending processes the TERM signal
        INIT: Sending processes the KILL signal
        Starting killall:                                          [  OK  ]
        Sending all processes the TERM signal...                   [  OK  ]
        touch: cannot touch `/var/lib/random-seed': No such file or directory
        chmod: cannot access `/var/lib/random-seed': No such file or directory
        Saving random seed:                                        [FAILED]
        Syncing hardware clock to system time                      [  OK  ]
        Turning off quotas:                                        [  OK  ]
        Please stand by while rebooting the system...
        md: stopping all md devices.
Once shutdown is complete, the screen clears, the system resets, and the
system boots.  While the system is coming up, you may see some messages
during the FS check like those below.  These types of messages are fine:
        <snip...>

        Setting up Logical Volume Management:                      [  OK  ]
        Checking filesystems
        /1: Superblock last mount time is in the future.  FIXED.
        /1: Superblock last write time is in the future.  FIXED.
        /1: clean, 31367/1898688 files, 591969/1897670 blocks
        /var1: Superblock last write time is in the future.  FIXED.
        /var1: clean, 972/524288 files, 34576/524120 blocks
                                                                   [  OK  ]
        Remounting root filesystem in read-write mode:             [  OK  ]
        Mounting local filesystems:                                [  OK  ]
        Enabling local filesystem quotas:                          [  OK  ]
        Enabling /etc/fstab swaps:                                 [  OK  ]
        INIT: Entering runlevel: 3

        <snip...> 

    getty starts, screen clears, a login prompt is provided

        CentOS release 5.4 (Final)
        Kernel 2.6.18-164.el5 on an i686

        tux login: _
At this point, we can log in and, should we choose to do so, finish
customizing our 'inittab' file without any further downtime of the host.
Ideally, we would at least add in the rest of the runlevels (0-2,4,5),
and possibly a few more ttys.  Since 'init' reads this file upon changing
to a new runlevel, no action is required for the runlevel changes.
To get 'init' to read the 'inittab' to spawn the newly added ttys,
run the following:
        tux [0] /bin/ps -ef | /bin/grep g[e]tty 
        root      2004     1  0 00:56 tty1     00:00:00 /sbin/mingetty tty1
        tux [0] /bin/grep getty /etc/inittab
        1:2345:respawn:/sbin/mingetty tty1

    After adding ttys to 'inittab':

        tux [0] /bin/grep getty /etc/inittab
        1:2345:respawn:/sbin/mingetty tty1
        2:2345:respawn:/sbin/mingetty tty2
        3:2345:respawn:/sbin/mingetty tty3
        4:2345:respawn:/sbin/mingetty tty4
        5:2345:respawn:/sbin/mingetty tty5
        6:2345:respawn:/sbin/mingetty tty6
        tux [0] /sbin/init q
        tux [0] /bin/ps -ef | /bin/grep g[e]tty
        root      2004     1  0 00:56 tty1     00:00:00 /sbin/mingetty tty1
        root      2174     1  0 01:03 tty2     00:00:00 /sbin/mingetty tty2
        root      2177     1  0 01:03 tty3     00:00:00 /sbin/mingetty tty3
        root      2180     1  0 01:03 tty4     00:00:00 /sbin/mingetty tty4
        root      2181     1  0 01:03 tty5     00:00:00 /sbin/mingetty tty5
        root      2182     1  0 01:03 tty6     00:00:00 /sbin/mingetty tty6