02 February 2013

Intro SmartOS Setup pt 2

In part 1, I detailed the general expectations, layout, and router
infrastructure VM (IVM) for our SmartOS sandbox environment. In part 2,
we move on to our services IVM, "serv1". To recap, "serv1" will host
a name server, web server, and time server for our SmartOS compute nodes
(CNs) and SmartOS VMs (SOSVMs). (SmartOS setup and configuration begins
in part 3). The VirtualBox (VBox) configuration for "serv1" is:
        - 1 vcpu
        - 1024 MB memory
        - 8 GB vdisk (dynamic / thin provisioned)
        - 2 configured interfaces in VBox:
                + Adapter 1:  internal network (intnet (10.0.7.10))
                + Adapter 2:  internal network (labworld (10.0.8.10))
During initial installation of CentOS 6.2, I chose a "webserver"
install and added packages for "bind" and "ntpd" to host our name and
time servers, respectively. Following the installation, we end up with
the following system configuration:
        serv1 [0] /usr/bin/awk '{print $0 ORS}' RS='' /etc/sysconfig/network-scripts/ifcfg-eth*
        DEVICE="eth0"
        NM_CONTROLLED="no"
        ONBOOT=yes
        HWADDR=08:00:27:49:3F:CA
        TYPE=Ethernet
        BOOTPROTO=none
        IPADDR=10.0.7.10
        NETMASK=255.255.255.0
        BROADCAST=10.0.7.255
        IPV6INIT=no

        DEVICE="eth1"
        NM_CONTROLLED="no"
        ONBOOT=yes
        HWADDR=08:00:27:7D:54:DC
        TYPE=Ethernet
        BOOTPROTO=none
        IPADDR=10.0.8.10
        NETMASK=255.255.255.0
        BROADCAST=10.0.8.255
        IPV6INIT=no

        serv1 [0] /sbin/ifconfig -a
        eth0      Link encap:Ethernet  HWaddr 08:00:27:49:3F:CA
                  inet addr:10.0.7.10  Bcast:10.0.7.255  Mask:255.255.255.0
                  inet6 addr: fe80::a00:27ff:fe49:3fca/64 Scope:Link
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:603 errors:0 dropped:0 overruns:0 frame:0
                  TX packets:359 errors:0 dropped:0 overruns:0 carrier:0
                  collisions:0 txqueuelen:1000
                  RX bytes:56411 (55.0 KiB)  TX bytes:49299 (48.1 KiB)

        eth1      Link encap:Ethernet  HWaddr 08:00:27:7D:54:DC
                  inet addr:10.0.8.10  Bcast:10.0.8.255  Mask:255.255.255.0
                  inet6 addr: fe80::a00:27ff:fe7d:54dc/64 Scope:Link
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:7 errors:0 dropped:0 overruns:0 frame:0
                  TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
                  collisions:0 txqueuelen:1000
                  RX bytes:466 (466.0 b)  TX bytes:1386 (1.3 KiB)
        serv1 [0] /bin/cat /etc/sysconfig/network
        NETWORKING=yes
        HOSTNAME=serv1.admin.none
        GATEWAY=10.0.7.37
        NOZEROCONF=yes
        serv1 [0] /bin/netstat -rn
        Kernel IP routing table
        Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
        10.0.7.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
        10.0.8.0        0.0.0.0         255.255.255.0   U         0 0          0 eth1
        0.0.0.0         10.0.7.37       0.0.0.0         UG        0 0          0 eth0
        serv1 [0] /bin/cat /etc/hosts
        127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
        ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
        10.0.7.10       serv1.admin.none        serv1
        serv1 [0] /bin/cat /etc/resolv.conf
        search vbox.none admin.none world.none
        nameserver 10.0.7.10
        domain admin.none
        serv1 [0]
Next, since this is a wholly self-contained sandbox environment with
no external access, we can be somewhat lax in security. As such, I've
disabled 'iptables' and 'selinux':
        serv1 [0] /sbin/chkconfig --level 2345 iptables off
        serv1 [0] /sbin/chkconfig --level 2345 ip6tables off
        serv1 [0] /sbin/iptables --flush
        serv1 [0] /bin/egrep -v '^#|^$' /etc/selinux/config
        SELINUX=disabled
        SELINUXTYPE=targeted
        serv1 [0]
On to our first infrastructure service, the time server. With the
sandbox entirely private, the following lines in "/etc/ntp.conf" need
to be uncommented:
        server  127.127.1.0     # local clock
        fudge   127.127.1.0 stratum 10
These effectively configure 'ntpd' to assume the local time on the host
(serv1) is always correct. Additionally, the following lines were added
for both the internal admin network (10.0.7.0/24) and the world network
(10.0.8.0/24):
        restrict 10.0.7.0 mask 255.255.255.0 nomodify notrap
        restrict 10.0.8.0 mask 255.255.255.0 nomodify notrap
The resulting "ntp.conf" file is:
        serv1 [0] /bin/egrep -v '^#|^$' /etc/ntp.conf
        driftfile /var/lib/ntp/drift
        restrict default kod nomodify notrap nopeer noquery
        restrict -6 default kod nomodify notrap nopeer noquery
        restrict 127.0.0.1
        restrict -6 ::1
        restrict 10.0.7.0 mask 255.255.255.0 nomodify notrap
        restrict 10.0.8.0 mask 255.255.255.0 nomodify notrap
        server  127.127.1.0     # local clock
        fudge   127.127.1.0 stratum 10
        includefile /etc/ntp/crypto/pw
        keys /etc/ntp/keys
With 'ntpd' configured, we can now enable it under runlevel 3 and
start it:
        serv1 [0] /sbin/runlevel
        N 3
        serv1 [0] /sbin/chkconfig --level 3 ntpd on
        serv1 [0] /sbin/service ntpd start
        Starting ntpd:                                             [  OK  ]
        serv1 [0]
The second service to prepare for is our web server (SmartOS image
server). For this, I downloaded the default page returned from:
        https://datasets.joyent.com/datasets/
and saved it as "~/joyent/datasets/index.html" on "glados" (my
workstation). The "index.html" contents are actually JSON objects
detailing the dsmanifest files for each image as explained here.
Since I only want a subset of those images available, I've modified the
contents of the "index.html" file to retain only those I want. Next,
since each image object details the downloadable image URL from Joyent,
I downloaded those images (that I've kept in the updated "index.html")
to their respective UUID named image directories. After creating
"/var/www/joyent/datasets" on "serv1", I simply copied over the resulting
"index.html" and dataset images to "serv1" via 'scp':
        # on serv1:
        serv1 [0] /bin/mkdir -p /var/www/joyent/datasets

        # on glados:
        troy@glados [0] /usr/bin/scp -r ~/joyent/datasets/* root@serv1:/var/www/joyent/datasets/.
        root@serv1's password:
        base64-1.8.4.zfs.bz2                          100%   78MB  15.5MB/s   00:05
        index.html                                    100%   10KB   9.9KB/s   00:00
        <snip...>
        nodejs-1.4.0.zfs.bz2                           20%   54MB  27.2MB/s   00:07 ETA
        troy@glados [0]
Back on "serv1", we can see our resulting image repository file /
directory structure:
        serv1 [0] /usr/bin/find /var/www/joyent -print
        /var/www/joyent
        /var/www/joyent/datasets
        /var/www/joyent/datasets/index.html
        /var/www/joyent/datasets/b00acc20-14ab-11e2-85ae-4f03a066e93e
        /var/www/joyent/datasets/b00acc20-14ab-11e2-85ae-4f03a066e93e/mongodb-1.4.0.zfs.bz2
        /var/www/joyent/datasets/3390ca7c-f2e7-11e1-8818-c36e0b12e58b
        /var/www/joyent/datasets/3390ca7c-f2e7-11e1-8818-c36e0b12e58b/standard-1.0.7.zfs.bz2
        /var/www/joyent/datasets/ef22b182-3d7a-11e2-a7a9-af27913943e2
        /var/www/joyent/datasets/ef22b182-3d7a-11e2-a7a9-af27913943e2/base-1.8.2.zfs.bz2
        /var/www/joyent/datasets/dc1a8b5e-043c-11e2-9d94-0f3fcb2b0c6d
        /var/www/joyent/datasets/dc1a8b5e-043c-11e2-9d94-0f3fcb2b0c6d/percona-1.6.0.zfs.bz2
        /var/www/joyent/datasets/84cb7edc-3f22-11e2-8a2a-3f2a7b148699
        /var/www/joyent/datasets/84cb7edc-3f22-11e2-8a2a-3f2a7b148699/base-1.8.4.zfs.bz2
        /var/www/joyent/datasets/fdea06b0-3f24-11e2-ac50-0b645575ce9d
        /var/www/joyent/datasets/fdea06b0-3f24-11e2-ac50-0b645575ce9d/base64-1.8.4.zfs.bz2
        /var/www/joyent/datasets/aa583f78-3d83-11e2-9188-fff9b605718d
        /var/www/joyent/datasets/aa583f78-3d83-11e2-9188-fff9b605718d/base64-1.8.2.zfs.bz2
        /var/www/joyent/datasets/1fc068b0-13b0-11e2-9f4e-2f3f6a96d9bc
        /var/www/joyent/datasets/1fc068b0-13b0-11e2-9f4e-2f3f6a96d9bc/nodejs-1.4.0.zfs.bz2
        /var/www/joyent/datasets/a0f8cf30-f2ea-11e1-8a51-5793736be67c
        /var/www/joyent/datasets/a0f8cf30-f2ea-11e1-8a51-5793736be67c/standard64-1.0.7.zfs.bz2
        serv1 [0]
Before our image repository can be of any use, we need to further modify
our "index.html" file to set the image URL to be relative to our image
server IP address (10.0.7.10) and configuration on "serv1":
        serv1 [0] /usr/bin/head -20 /var/www/joyent/datasets/index.html
        [
          {
            "name": "base64",
            "version": "1.8.4",
            "type": "zone-dataset",
            "description": "Base image with core packages preinstalled",
            "published_at": "2012-12-05T21:59:37.513Z",
            "os": "smartos",
            "files": [
              {
                "path": "base64-1.8.4.zfs.bz2",
                "sha1": "da918c38f1484b4cfc2c5e15b9b96583fae44b1c",
                "size": 81369208,
                "url": "http://10.0.7.10/datasets/fdea06b0-3f24-11e2-ac50-0b645575ce9d/base64-1.8.4.zfs.bz2"
              }
            ],
            "requirements": {
              "networks": [
                {
                  "name": "net0",
        <snip...>
(The resulting "index.html" file can be downloaded from here.) Next, I've
added an apache include file (joyent.conf) to present our image server.
Following this, I've also enabled 'httpd' so it starts after we reboot
"serv1" later:
        serv1 [0] /bin/cat /etc/httpd/conf.d/joyent.conf
        <Directory "/var/www/joyent">
                Options Indexes
                AllowOverride None
                Order allow,deny
                Allow from all
        </Directory>

        <VirtualHost 10.0.7.10:80>
                DocumentRoot /var/www/joyent
                ServerName serv1.admin.none
                ErrorLog logs/serv1-error_log
                CustomLog logs/serv1-access_log common
        </VirtualHost>
        serv1 [0] /sbin/chkconfig --list | /bin/grep http
        httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
        serv1 [0] /sbin/chkconfig --level 3 httpd on
        serv1 [0]
(Since "serv1" is its own name server, the FQDN for "ServerName" in the
"VirtualHost" configuration above is also detailed in "/etc/hosts" on
"serv1" in case 'httpd' starts before 'named'.) Our third, and final
service to configure is 'bind' (named). Below, I've set the system
configuration file for 'named' (for start up purposes) and listed out
all of our 'named' configuration and zone files:
        serv1 [0] /bin/grep -v ^# /etc/sysconfig/named
        OPTIONS="-c /opt/bind-config/named.conf"
        serv1 [0] /bin/ls -F /opt/bind-config
        named.conf  zones/
        serv1 [0] find /opt/bind-config -print
        /opt/bind-config
        /opt/bind-config/named.conf
        /opt/bind-config/zones
        /opt/bind-config/zones/10.0.8.rev
        /opt/bind-config/zones/vbox.none
        /opt/bind-config/zones/localhost
        /opt/bind-config/zones/10.0.7.rev
        /opt/bind-config/zones/192.168.56.rev
        /opt/bind-config/zones/127.0.0.rev
        /opt/bind-config/zones/admin.none
        /opt/bind-config/zones/world.none
        serv1 [0]
As seen above, there is a zone file for each domain and network that
we are using. Below, I've included the full contents of the 'named'
configuration and zone files:
        serv1 [0] /bin/cat /opt/bind-config/named.conf
        options {
                directory "/opt/bind-config";
                allow-transfer { 10.0.7.0/24; 192.168.56.0/24; };
                allow-recursion { none; };
                version "lab server 0.3.1";
        };

        # logging directives
        logging {
                channel named_log {
                        file "named.log" versions 3 size 25m;
                        severity info;
                        print-severity yes;
                        print-time yes;
                        print-category yes;
                };

                category default {
                        named_log;
                };
        };

        zone "admin.none" {
                type master;
                file "zones/admin.none";
        };

        zone "7.0.10.in-addr.arpa" {
                type master;
                file "zones/10.0.7.rev";
        };

        zone "world.none" {
                type master;
                file "zones/world.none";
        };

        zone "8.0.10.in-addr.arpa" {
                type master;
                file "zones/10.0.8.rev";
        };

        zone "vbox.none" {
                type master;
                file "zones/vbox.none";
        };

        zone "56.168.192.in-addr.arpa" {
                type master;
                file "zones/192.168.56.rev";
        };

        zone "localhost" {
                type master;
                file "zones/localhost";
                allow-update { none; };
        };

        zone "0.0.127.in-addr.arpa" {
                type master;
                file "zones/127.0.0.rev";
                allow-update { none; };
        };
        serv1 [0] for i in admin.none 10.0.7.rev world.none 10.0.8.rev \
        > vbox.none 192.168.56.rev localhost 127.0.0.rev ; do \
        > echo "==> /opt/bind-config/zones/${i} <==" ; \
        > /bin/cat /opt/bind-config/zones/${i} ; echo ; done
        ==> /opt/bind-config/zones/admin.none <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        serv1           A       10.0.7.10
        muttley         A       10.0.7.37

        cn40            A       10.0.7.40
        cn41            A       10.0.7.41
        cn42            A       10.0.7.42
        cn43            A       10.0.7.43
        cn44            A       10.0.7.44
        cn45            A       10.0.7.45
        cn46            A       10.0.7.46
        cn47            A       10.0.7.47
        cn48            A       10.0.7.48
        cn49            A       10.0.7.49
        cn50            A       10.0.7.50

        ==> /opt/bind-config/zones/10.0.7.rev <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        10              PTR     serv1.admin.none.
        37              PTR     muttley.admin.none.

        40              PTR     cn40.admin.none.
        41              PTR     cn41.admin.none.
        42              PTR     cn42.admin.none.
        43              PTR     cn43.admin.none.
        44              PTR     cn44.admin.none.
        45              PTR     cn45.admin.none.
        46              PTR     cn46.admin.none.
        47              PTR     cn47.admin.none.
        48              PTR     cn48.admin.none.
        49              PTR     cn49.admin.none.
        50              PTR     cn50.admin.none.

        ==> /opt/bind-config/zones/world.none <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        serv1           A       10.0.8.10
        muttley         A       10.0.8.37

        vm100           A       10.0.8.100
        vm101           A       10.0.8.101
        vm102           A       10.0.8.102
        vm103           A       10.0.8.103
        vm104           A       10.0.8.104
        vm105           A       10.0.8.105
        vm106           A       10.0.8.106
        vm107           A       10.0.8.107
        vm108           A       10.0.8.108
        vm109           A       10.0.8.109
        vm110           A       10.0.8.110
        vm111           A       10.0.8.111
        vm112           A       10.0.8.112
        vm113           A       10.0.8.113
        vm114           A       10.0.8.114
        vm115           A       10.0.8.115
        vm116           A       10.0.8.116
        vm117           A       10.0.8.117
        vm118           A       10.0.8.118
        vm119           A       10.0.8.119
        vm120           A       10.0.8.120
        vm121           A       10.0.8.121
        vm122           A       10.0.8.122
        vm123           A       10.0.8.123
        vm124           A       10.0.8.124
        vm125           A       10.0.8.125
        vm126           A       10.0.8.126
        vm127           A       10.0.8.127
        vm128           A       10.0.8.128
        vm129           A       10.0.8.129
        vm130           A       10.0.8.130

        ==> /opt/bind-config/zones/10.0.8.rev <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        10              PTR     serv1.world.none.
        37              PTR     muttley.world.none.

        100             PTR     vm100.world.none.
        101             PTR     vm101.world.none.
        102             PTR     vm102.world.none.
        103             PTR     vm103.world.none.
        104             PTR     vm104.world.none.
        105             PTR     vm105.world.none.
        106             PTR     vm106.world.none.
        107             PTR     vm107.world.none.
        108             PTR     vm108.world.none.
        109             PTR     vm109.world.none.
        110             PTR     vm110.world.none.
        111             PTR     vm111.world.none.
        112             PTR     vm112.world.none.
        113             PTR     vm113.world.none.
        114             PTR     vm114.world.none.
        115             PTR     vm115.world.none.
        116             PTR     vm116.world.none.
        117             PTR     vm117.world.none.
        118             PTR     vm118.world.none.
        119             PTR     vm119.world.none.
        120             PTR     vm120.world.none.
        121             PTR     vm121.world.none.
        122             PTR     vm122.world.none.
        123             PTR     vm123.world.none.
        124             PTR     vm124.world.none.
        125             PTR     vm125.world.none.
        126             PTR     vm126.world.none.
        127             PTR     vm127.world.none.
        128             PTR     vm128.world.none.
        129             PTR     vm129.world.none.
        130             PTR     vm130.world.none.

        ==> /opt/bind-config/zones/vbox.none <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        glados          A       192.168.56.1
        muttley         A       192.168.56.37

        ==> /opt/bind-config/zones/192.168.56.rev <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        1               PTR     glados.vbox.none.
        37              PTR     muttley.vbox.none.

        ==> /opt/bind-config/zones/localhost <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        @               A       127.0.0.1

        ==> /opt/bind-config/zones/127.0.0.rev <==
        $TTL 86400 ; Default TTL (1 day)
        @               SOA     serv1.admin.none. hostmaster.admin.none. (
                                2013012801      ; Serial number (YYYYMMDDNN)
                                3600            ; Refresh (1 hour)
                                300             ; Retry (5 minutes)
                                864000          ; Expire (10 days)
                                86400   )       ; Negative TTL (1 day)
                        NS      serv1.admin.none.

        1               PTR     localhost.
With 'named' now configured, 'named' is enabled for runlevel 3 and
finally, we reboot "serv1":
        serv1 [0] /sbin/chkconfig --level 3 named on
        serv1 [0] /sbin/init 6
Once "serv1" reboots, it is ready to function as our services host.
In the next section, part 3, we get to start checking out SmartOS and
our first compute node, "cn40".



see also:
    Intro SmartOS Setup pt 1
    Intro SmartOS Setup pt 3
    Intro SmartOS Setup pt 4
    SmartOS - HowImageServerWork
    index.html for the Image Server in this write up

No comments: