02 February 2013

Intro SmartOS Setup pt 3

In part 1, I detailed our general expectations and in part 2, our services
infrastructure VM (IVM) was configured. Here in part 3, we finally get to
check out our first SmartOS IVM and compute node, "cn40". The VirtualBox
(VBox) configuration for "cn40" is:
        - 1 vcpu
        - 2048 MB memory
        - 16 GB vdisk (dynamic / thin provisioned)
        - boots from virtual CDROM (smartos-20130111T010112Z.iso)
                + 'Live CD/DVD' checked
        - 2 configured interfaces in VBox:
                + Adapter 1:  internal network (intnet (10.0.7.40))
                        = Promiscuos Mode: Deny
                        = MAC Address:  0800272D5951
                + Adapter 2:  internal network (labworld (10.0.8.0/24))
                        = won't be IP'd on the network
                        = Promiscuos Mode: Allow All
                        = MAC Address:  080027911F6E
In the above, normally the network adapters are configured to "Deny"
for "Promiscuous Mode". In the case of "adapter 2", this needs to be
configured to "Allow All" so that we can access our SmartOS created VMs
(SOSVMs). Also, to keep identification easy, I've detailed the assigned
MAC Address for each adapter since we'll want to know that in the
next part during SmartOS configuration. With our VBox configuration
complete, power on the "cn40" IVM and let it boot from the CDROM
(SmartOS iso image). The next bit is simply the SmartOS boot / directed
configuration process:
         GNU GRUB  version 0.97  (639K lower / 2096064K upper memory)

        Live 64-bit (text)              <===============
        Live 64-bit (text) +kmdb
        Live 64-bit (ttya) +kmdb
        Live 64-bit (ttya)

           Use the <up> and <down> keys to select which entry is highlighted.
           Press enter to boot the selected OS, 'e' to edit the
           commands before booting, or 'c' for a command-line.

        The highlighted entry will be booted automatically in 10 seconds.

    # screen refresh

        SunOS Release 5.11 Version joyent_20130111T010112Z 64-bit
        Copyright (c) 2010-2012, Joyent Inc. All rights reserved.
        NOTICS: MPO disabled because memory is interleaved

        WARNING: kvm: no hardware support

    # screen refresh

        --------------------------------------------------------------------------------
         SmartOS Setup
         Copyright 2011, Joyent, Inc.                   http://wiki.smartos.org/install
        --------------------------------------------------------------------------------

        You must answer the following questions to configure the system.
        You will have a chance to review and correct your answers, as well as a
        chance to edit the final configuration, before it is applied.

        Would you like to continue with configuration? [Y/n]y

    # screen refresh

        --------------------------------------------------------------------------------
         SmartOS Setup
         Networking                                     http://wiki.smartos.org/install
        --------------------------------------------------------------------------------
        Number Link      MAC Address        State   Network
        1      e1000g0   8:0:27:2d:59:51    unknown -
        2      e1000g1   8:0:27:91:1f:6e    unknown -
        Enter the number of the NIC for the 'admin' interface: 1
        IP address (or 'dhcp' ): 10.0.7.40
        netmask: 255.255.255.0
Earlier when we did our VBox configuration, we detailed the MAC address
intended for our private, admin network, which in this case is "e1000g0"
above. Below, we configure the rest of our network settings, setting
"muttley's" (our BSD router IVM) IP address as our gateway, and our name
servers both configured to "serv1's" (our services IVM) IP address:
        --------------------------------------------------------------------------------
         SmartOS Setup
         Networking - Continued                         http://wiki.smartos.org/install
        --------------------------------------------------------------------------------

          The default gateway will determine which network will be used to connect to
          other networks.

        Enter the default gateway IP: 10.0.7.37
        Enter the Primary DNS server IP [8.8.8.8]: 10.0.7.10
        Enter the Secondary DNS server IP [8.8.4.4]: 10.0.7.10
        Enter the domain name: admin.none
        Default DNS search domain: admin.none

    # screen refresh

        --------------------------------------------------------------------------------
         SmartOS Setup
         Storage                                        http://wiki.smartos.org/install
        --------------------------------------------------------------------------------
        Please select disks for the storage pool, space separated

        Valid choices are c1t0d0
        c1t0d0

    # screen refresh

        --------------------------------------------------------------------------------
         SmartOS Setup
         Account Information                            http://wiki.smartos.org/install
        --------------------------------------------------------------------------------
        Enter root password: _
        Confirm password: _

    # screen refresh

        --------------------------------------------------------------------------------
         SmartOS Setup
         Verify Configuration                           http://wiki.smartos.org/install
        --------------------------------------------------------------------------------
        Verify that the following values are correct:

        MAC address: 8:0:27:2d:59:51
        IP address: 10.0.7.40
        Netmask: 255.255.255.0
        Gateway router IP address: 10.0.7.37
        DNS servers: 10.0.7.10,10.0.7.10
        Default DNS search domain: admin.none
        NTP server: pool.ntp.org
        Domain name: admin.none

        Is this correct? [y]: y

        Your configuration is about to be applied.
        Would you like to edit the final configuration file? [n]: y
In the above, we answered that "y" to "Is this correct." Had we answered
"n", we would have restarted the directed configuration prompts, not what
we want. Instead, we want to modify the final configuration file, so
we've answered "y" when asked if we'd like to edit it. This will drop us
to a 'vi' session to modify what will become "/usbkey/config" on "cn40":
        #
        # This file was auto-generated and must be source-able by bash.
        #

        # admin_nic is the nic admin_ip will be connected to for headnode zones.
        admin_nic=8:0:27:2d:59:51
        admin_ip=10.0.7.40
        admin_netmask=255.255.255.0
        admin_network=...
        admin_gateway=10.0.7.40

        headnode_default_gateway=10.0.7.37

        dns_resolvers=10.0.7.10,10.0.7.10               <============
        dns_domain=admin.none

        ntp_hosts=pool.ntp.org                          <============
        comput_node_ntp_hosts=10.0.7.40
I've updated the above identified lines to that seen below. I've also
added a NIC_TAG of "world_nic" to identify, but not further configure,
that network interface using the MAC Address we identified earlier during
our VBox configuration. (We'll deal further with the "world_nic" later.):
        dns_resolvers=10.0.7.10                         <============
        ntp_hosts=10.0.7.10                             <============
        world_nic=8:0:27:91:1f:6e
After we write+quit the 'vi' session, the screen refreshes to configure
our disks:
        Your data pool will be created with the following disks:
        c1t0d0
        ********************************************
        * This will erase *ALL DATA* on these disks*
        ********************************************
        are you sure? [n]: y
        cannot open 'zones': no such pool
        Creating pool zones...                                done
        Making dump zvol...                                   done
        Dump content: kernel pages
               Dump device: /dev/zvol/dsk/zones/dump (dedicated)
        Savecore directory: /var/crash/volatile
          Savecore enabled: no
           Save compressed: on
        done
        Initializing config dataset for zones...              done
        Creating config dataset...                            done
        Creating global cores dataset...                      done
        Creating opt dataset...                               done
        Creating var dataset...                               done
        Creating swap zvol...                                 done

    # screen refresh

        The system will now finish configuration and reboot. Please wait...
        rebooting...
Our SmartOS IVM / compute node (cn40) should now boot back up from the
live iso and be available for our usage:
                  _____
               ____   ____            .                   .
              _____   _____           | .-. .  . .-. :--. |-
              __         __           ;|   ||  |(.-' |  | |
              _____   _____       `--'  `-' `;-| `-' '  ' `-'
               ____   ____                  /  ; Joyent Live Image v0.147+
                  _____                     `-'   build: 20130111T010112Z


        08-00-27-2d-59-51 wscons login: _
Back on our workstation host (glados), we can now log into "cn40" as
"root" via 'ssh':
        troy@glados [0] ssh -l root cn40
        The authenticity of host 'cn40 (10.0.7.40)' can't be established.
        RSA key fingerprint is db:f4:23:b5:14:fc:99:ef:1a:ac:59:b2:be:09:d4:e1.
        Are you sure you want to continue connecting (yes/no)? yes
        Warning: Permanently added 'cn40,10.0.7.40' (RSA) to the list of known hosts.
        Password:
        Last login: Tue Jan 29 03:38:08 2013
        - SmartOS Live Image v0.147+ build: 20130111T010112Z
        [root@08-00-27-2d-59-51 ~]#
(I'm kind of particular with my shell prompts, so I simply reset it to
my liking, as seen in the following.) With this being our first venture
into SmartOS, I've done a quick check of our mounted file systems (FS;
only the default SmartOS FS are currently configured), checked for any
configured zones (zoneadm; there are none aside from "global"), verified
with 'vmadm', and checked for any images (imgadm; there are none):
        cn40 [0] /bin/df -h
        Filesystem             size   used  avail capacity  Mounted on
        /devices/ramdisk:a     251M   219M    32M    88%    /
        /devices                 0K     0K     0K     0%    /devices
        /dev                     0K     0K     0K     0%    /dev
        ctfs                     0K     0K     0K     0%    /system/contract
        proc                     0K     0K     0K     0%    /proc
        mnttab                   0K     0K     0K     0%    /etc/mnttab
        swap                   3.4G   880K   3.4G     1%    /etc/svc/volatile
        objfs                    0K     0K     0K     0%    /system/object
        sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
        /devices/pseudo/lofi@0:1
                               376M   355M    22M    95%    /usr
        /usr/lib/libc/libc_hwcap2.so.1
                               376M   355M    22M    95%    /lib/libc.so.1
        fd                       0K     0K     0K     0%    /dev/fd
        zones                   16G   650K    13G     1%    /zones
        zones/cores             10G    31K    10G     1%    /zones/global/cores
        zones/var               16G   1.9M    13G     1%    /var
        zones/config            16G    38K    13G     1%    /etc/zones
        zones/opt               16G    31K    13G     1%    /opt
        zones/usbkey            16G   127K    13G     1%    /usbkey
        /usbkey/shadow          13G   127K    13G     1%    /etc/shadow
        /usbkey/ssh             13G   127K    13G     1%    /etc/ssh
        swap                   3.4G    12K   3.4G     1%    /tmp
        swap                   3.4G    32K   3.4G     1%    /var/run
        cn40 [0] /usr/sbin/zoneadm list
        global
        cn40 [0] /usr/sbin/vmadm list
        UUID                                  TYPE  RAM      STATE             ALIAS
        cn40 [0] /usr/sbin/imgadm list
        cn40 [0]
Next, we see that "/usbkey/config" is the same as what we updated it
to during our intial configuration and that only our private "admin"
nic has been configured (via 'ifconfig'):
        cn40 [0] /bin/cat /usbkey/config
        #
        # This file was auto-generated and must be source-able by bash.
        #

        # admin_nic is the nic admin_ip will be connected to for headnode zones.
        admin_nic=8:0:27:2d:59:51
        admin_ip=10.0.7.40
        admin_netmask=255.255.255.0
        admin_network=...
        admin_gateway=10.0.7.40

        headnode_default_gateway=10.0.7.37

        dns_resolvers=10.0.7.10
        dns_domain=admin.none

        ntp_hosts=10.0.7.10
        compute_node_ntp_hosts=10.0.7.40

        world_nic=8:0:27:91:1f:6e
        cn40 [0] /usr/sbin/ifconfig -a
        lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                inet 127.0.0.1 netmask ff000000
        e1000g0: flags=1100943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
                inet 10.0.7.40 netmask ffffff00 broadcast 10.0.7.255
                ether 8:0:27:2d:59:51
        lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
                inet6 ::1/128
        cn40 [0]
A few checks of 'dladm' output further validates the above and SmartOS
was kind enough to link a bridge device (vmwarebr) for us to "e1000g0"
(which we won't be using):
        cn40 [0] /usr/sbin/dladm show-phys
        LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
        e1000g0      Ethernet             up         1000   full      e1000g0
        e1000g1      Ethernet             unknown    0      half      e1000g1
        cn40 [0] /usr/sbin/dladm show-phys -m
        LINK         SLOT     ADDRESS            INUSE CLIENT
        e1000g0      primary  8:0:27:2d:59:51    yes  e1000g0
        e1000g1      primary  8:0:27:91:1f:6e    no   --
        cn40 [0] /usr/sbin/dladm show-link
        LINK        CLASS     MTU    STATE    BRIDGE     OVER
        e1000g0     phys      1500   up       vmwarebr   --
        e1000g1     phys      1500   unknown  --         --
        vmwarebr0   bridge    1500   up       --         e1000g0
        cn40 [0]
A really useful command under SmartOS is 'sysinfo'. It details various
info of the host (think 'smbios', kind of), the OS, and configuration
attributes (including our NIC_TAGs):
        cn40 [0] /usr/bin/sysinfo -p
        Live_Image='20130111T010112Z'
        System_Type='SunOS'
        Boot_Time='1359436533'
        Manufacturer='innotek GmbH'
        Product='VirtualBox'
        Serial_Number='0'
        VM_Capable='true'
        CPU_Type='Unknown'
        CPU_Virtualization='none'
        CPU_Physical_Cores=0
        Nic_Tags=admin,world
        UUID='42e86f2d-1d47-49c9-b739-10f7760a0580'
        Hostname='08-00-27-2d-59-51'
        CPU_Total_Cores=1
        MiB_of_Memory=2047
        Disk_c1t0d0_size_in_GB=17
        NIC_admin='e1000g0'
        NIC_world='e1000g1'
        Network_Interface_e1000g0_MAC_Address='08:00:27:2d:59:51'
        Network_Interface_e1000g0_IPv4_Address='10.0.7.40'
        Network_Interface_e1000g0_NIC_Names='admin'
        Network_Interface_e1000g0_Link_Status='up'
        Network_Interface_e1000g1_MAC_Address='08:00:27:91:1f:6e'
        Network_Interface_e1000g1_IPv4_Address=''
        Network_Interface_e1000g1_NIC_Names='world'
        Network_Interface_e1000g1_Link_Status='unknown'
        Bootparam_console='text'
        Bootparam_root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA'
        Bootparam_smartos='true'
        cn40 [0]
The primary reason I've checked out 'sysinfo' above is I'm about to add
a new SMF to "cn40" that will automatically identify and configure our
"world_nic" interface (it adds a bridge) for use by our SOSVMs prior to
SmartOS trying to start any SOSVMs. To stage this, below I've created
2 directories (/opt/custom/[smf|methods]) and written the supporting SMF
manifest for our new "world-nic" service. (Notably, by default any SMF
manifests written to "/opt/custom/smf" will be imported by SmartOS at
boot time. The "world-nic" manifest can be downloaded from here.):
        cn40 [0] /bin/mkdir -p /opt/custom/smf
        cn40 [0] /bin/mkdir -p /opt/custom/methods
        cn40 [0] /bin/cat /opt/custom/smf/world-nic.xml
        <?xml version='1.0'?>

        <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
        <!--
          COPYRIGHT: Copyright (c) 2013 Troy Dietrich.

          CDDL HEADER START

           The contents of this file are subject to the terms of the
           Common Development and Distribution License, Version 1.0 only
           (the "License").  You may not use this file except in compliance
           with the License.

           You can obtain a copy of the license at:
                 http://www.opensolaris.org/os/licensing.
           See the License for the specific language governing permissions
           and limitations under the License.

          CDDL HEADER END
        -->

        <service_bundle type='manifest' name='world-nic'>

        <!--
          world-nic.xml is intended for testing purposes and is not intended
          for use in production environments.  It was written to setup the VM
          supporting interface for my virtualized SmartOS compute nodes (which
          are running under VirtualBox).  This exec'd script assumes 2 NICs have
          been provisioned to the SmartOS host, the "admin" NIC (setup during
          initial configuration) and a secondary NIC to support any VM vNICs.
          The secondary NIC should be identified in '/usbkey/config' as
          "NICTAG_nic", but otherwise unconfigured.

          Tested against SmartOS 20130111T010112Z.

                   TD : 1359329221
        -->

            <service
                name='network/world-nic'
                type='service'
                version='0.1a'>

                <dependency
                        name='network'
                        grouping='require_all'
                        restart_on='none'
                        type='service'>
                        <service_fmri value='svc:/milestone/network:default'/>
                </dependency>
                <dependent
                        name='vmadmd'
                        grouping='require_all'
                        restart_on='error'>
                        <service_fmri value='svc:/system/smartdc/vmadmd:default' />
                </dependent>

                <instance name='default' enabled='true'>

                <exec_method
                        type='method'
                        name='start'
                        exec='/opt/custom/methods/world-nic.ksh start'
                        timeout_seconds='20' />
                <exec_method
                        type='method'
                        name='stop'
                        exec='/opt/custom/methods/world-nic.ksh stop'
                        timeout_seconds='20' />

                <property_group name='startd' type='framework'>
                        <propval name='duration' type='astring' value='transient' />
                </property_group>

                <template>
                        <common_name>
                                <loctext xml:lang='C'>
                                        WorldNic Configuration
                                </loctext>
                        </common_name>
                        <documentation>
                                <doc_link name='world-nic.ksh' uri='file:///opt/custom/methods/world-nic.ksh'/>
                        </documentation>
                </template>

                </instance>

                <stability value='Unstable'/>
            </service>

        </service_bundle>
        cn40 [0]
(The supporting "world-nic" shell script can be downloaded from here.
Just save it to "/opt/custom/methods/world-nic.ksh" on your lab compute
node to use it with the above manifest.) Below, I've validated and
imported the "world-nic" manifest and we can see via 'svcs -l' the
service has been onlined during the import:
        cn40 [0] /bin/ls -ld /opt/custom/methods/world-nic.ksh
        -rwxr-xr-x   1 root     root        3746 Jan 28 01:24 /opt/custom/methods/world-nic.ksh
        cn40 [0] /usr/sbin/svccfg validate /opt/custom/smf/world-nic.xml
        cn40 [0] /usr/sbin/svccfg import /opt/custom/smf/world-nic.xml
        cn40 [0] /usr/bin/svcs -l world-nic
        fmri         svc:/network/world-nic:default
        name         WorldNic Configuration
        enabled      true
        state        online
        next_state   none
        state_time   January 29, 2013 04:01:22 AM UTC
        logfile      /var/svc/log/network-world-nic:default.log
        restarter    svc:/system/svc/restarter:default
        dependency   require_all/none svc:/milestone/network:default (online)
        cn40 [0]
Since the "world-nic" service simply creates a bridge device for usage by
our SOSVMs, the "world" NIC doesn't get plumbed or configured, however,
the bridge (vmworld) is created and onlined as seen below:
        cn40 [0] /usr/sbin/ifconfig -a
        lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
                inet 127.0.0.1 netmask ff000000
        e1000g0: flags=1100943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
                inet 10.0.7.40 netmask ffffff00 broadcast 10.0.7.255
                ether 8:0:27:2d:59:51
        lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
                inet6 ::1/128
        cn40 [0] /usr/sbin/dladm show-link
        LINK        CLASS     MTU    STATE    BRIDGE     OVER
        e1000g0     phys      1500   up       vmwarebr   --
        e1000g1     phys      1500   up       vmworld    --
        vmwarebr0   bridge    1500   up       --         e1000g0
        vmworld0    bridge    1500   up       --         e1000g1
        cn40 [0]
At this point, we'll close out part 3 since we've now installed and
minimally configured our compute node IVM. In part 4, we'll add in our
SmartOS image (template) repository and start creating and checking out
some SOSVMs on "cn40".



see also:
    Intro SmartOS Setup pt 1
    Intro SmartOS Setup pt 2
    Intro SmartOS Setup pt 4
    world-nic.xml SMF manifest
    world-nic.ksh script called in world-nic.xml