infrastructure VM (IVM) was configured. Here in part 3, we finally get to
check out our first SmartOS IVM and compute node, "cn40". The VirtualBox
(VBox) configuration for "cn40" is:
- 1 vcpu
- 2048 MB memory
- 16 GB vdisk (dynamic / thin provisioned)
- boots from virtual CDROM (smartos-20130111T010112Z.iso)
+ 'Live CD/DVD' checked
- 2 configured interfaces in VBox:
+ Adapter 1: internal network (intnet (10.0.7.40))
= Promiscuos Mode: Deny
= MAC Address: 0800272D5951
+ Adapter 2: internal network (labworld (10.0.8.0/24))
= won't be IP'd on the network
= Promiscuos Mode: Allow All
= MAC Address: 080027911F6E
In the above, normally the network adapters are configured to "Deny"for "Promiscuous Mode". In the case of "adapter 2", this needs to be
configured to "Allow All" so that we can access our SmartOS created VMs
(SOSVMs). Also, to keep identification easy, I've detailed the assigned
MAC Address for each adapter since we'll want to know that in the
next part during SmartOS configuration. With our VBox configuration
complete, power on the "cn40" IVM and let it boot from the CDROM
(SmartOS iso image). The next bit is simply the SmartOS boot / directed
configuration process:
GNU GRUB version 0.97 (639K lower / 2096064K upper memory)
Live 64-bit (text) <===============
Live 64-bit (text) +kmdb
Live 64-bit (ttya) +kmdb
Live 64-bit (ttya)
Use the <up> and <down> keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.
The highlighted entry will be booted automatically in 10 seconds.
# screen refresh
SunOS Release 5.11 Version joyent_20130111T010112Z 64-bit
Copyright (c) 2010-2012, Joyent Inc. All rights reserved.
NOTICS: MPO disabled because memory is interleaved
WARNING: kvm: no hardware support
# screen refresh
--------------------------------------------------------------------------------
SmartOS Setup
Copyright 2011, Joyent, Inc. http://wiki.smartos.org/install
--------------------------------------------------------------------------------
You must answer the following questions to configure the system.
You will have a chance to review and correct your answers, as well as a
chance to edit the final configuration, before it is applied.
Would you like to continue with configuration? [Y/n]y
# screen refresh
--------------------------------------------------------------------------------
SmartOS Setup
Networking http://wiki.smartos.org/install
--------------------------------------------------------------------------------
Number Link MAC Address State Network
1 e1000g0 8:0:27:2d:59:51 unknown -
2 e1000g1 8:0:27:91:1f:6e unknown -
Enter the number of the NIC for the 'admin' interface: 1
IP address (or 'dhcp' ): 10.0.7.40
netmask: 255.255.255.0
Earlier when we did our VBox configuration, we detailed the MAC addressintended for our private, admin network, which in this case is "e1000g0"
above. Below, we configure the rest of our network settings, setting
"muttley's" (our BSD router IVM) IP address as our gateway, and our name
servers both configured to "serv1's" (our services IVM) IP address:
--------------------------------------------------------------------------------
SmartOS Setup
Networking - Continued http://wiki.smartos.org/install
--------------------------------------------------------------------------------
The default gateway will determine which network will be used to connect to
other networks.
Enter the default gateway IP: 10.0.7.37
Enter the Primary DNS server IP [8.8.8.8]: 10.0.7.10
Enter the Secondary DNS server IP [8.8.4.4]: 10.0.7.10
Enter the domain name: admin.none
Default DNS search domain: admin.none
# screen refresh
--------------------------------------------------------------------------------
SmartOS Setup
Storage http://wiki.smartos.org/install
--------------------------------------------------------------------------------
Please select disks for the storage pool, space separated
Valid choices are c1t0d0
c1t0d0
# screen refresh
--------------------------------------------------------------------------------
SmartOS Setup
Account Information http://wiki.smartos.org/install
--------------------------------------------------------------------------------
Enter root password: _
Confirm password: _
# screen refresh
--------------------------------------------------------------------------------
SmartOS Setup
Verify Configuration http://wiki.smartos.org/install
--------------------------------------------------------------------------------
Verify that the following values are correct:
MAC address: 8:0:27:2d:59:51
IP address: 10.0.7.40
Netmask: 255.255.255.0
Gateway router IP address: 10.0.7.37
DNS servers: 10.0.7.10,10.0.7.10
Default DNS search domain: admin.none
NTP server: pool.ntp.org
Domain name: admin.none
Is this correct? [y]: y
Your configuration is about to be applied.
Would you like to edit the final configuration file? [n]: y
In the above, we answered that "y" to "Is this correct." Had we answered"n", we would have restarted the directed configuration prompts, not what
we want. Instead, we want to modify the final configuration file, so
we've answered "y" when asked if we'd like to edit it. This will drop us
to a 'vi' session to modify what will become "/usbkey/config" on "cn40":
#
# This file was auto-generated and must be source-able by bash.
#
# admin_nic is the nic admin_ip will be connected to for headnode zones.
admin_nic=8:0:27:2d:59:51
admin_ip=10.0.7.40
admin_netmask=255.255.255.0
admin_network=...
admin_gateway=10.0.7.40
headnode_default_gateway=10.0.7.37
dns_resolvers=10.0.7.10,10.0.7.10 <============
dns_domain=admin.none
ntp_hosts=pool.ntp.org <============
comput_node_ntp_hosts=10.0.7.40
I've updated the above identified lines to that seen below. I've alsoadded a NIC_TAG of "world_nic" to identify, but not further configure,
that network interface using the MAC Address we identified earlier during
our VBox configuration. (We'll deal further with the "world_nic" later.):
dns_resolvers=10.0.7.10 <============
ntp_hosts=10.0.7.10 <============
world_nic=8:0:27:91:1f:6e
After we write+quit the 'vi' session, the screen refreshes to configureour disks:
Your data pool will be created with the following disks:
c1t0d0
********************************************
* This will erase *ALL DATA* on these disks*
********************************************
are you sure? [n]: y
cannot open 'zones': no such pool
Creating pool zones... done
Making dump zvol... done
Dump content: kernel pages
Dump device: /dev/zvol/dsk/zones/dump (dedicated)
Savecore directory: /var/crash/volatile
Savecore enabled: no
Save compressed: on
done
Initializing config dataset for zones... done
Creating config dataset... done
Creating global cores dataset... done
Creating opt dataset... done
Creating var dataset... done
Creating swap zvol... done
# screen refresh
The system will now finish configuration and reboot. Please wait...
rebooting...
Our SmartOS IVM / compute node (cn40) should now boot back up from thelive iso and be available for our usage:
_____
____ ____ . .
_____ _____ | .-. . . .-. :--. |-
__ __ ;| || |(.-' | | |
_____ _____ `--' `-' `;-| `-' ' ' `-'
____ ____ / ; Joyent Live Image v0.147+
_____ `-' build: 20130111T010112Z
08-00-27-2d-59-51 wscons login: _
Back on our workstation host (glados), we can now log into "cn40" as"root" via 'ssh':
troy@glados [0] ssh -l root cn40
The authenticity of host 'cn40 (10.0.7.40)' can't be established.
RSA key fingerprint is db:f4:23:b5:14:fc:99:ef:1a:ac:59:b2:be:09:d4:e1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cn40,10.0.7.40' (RSA) to the list of known hosts.
Password:
Last login: Tue Jan 29 03:38:08 2013
- SmartOS Live Image v0.147+ build: 20130111T010112Z
[root@08-00-27-2d-59-51 ~]#
(I'm kind of particular with my shell prompts, so I simply reset it tomy liking, as seen in the following.) With this being our first venture
into SmartOS, I've done a quick check of our mounted file systems (FS;
only the default SmartOS FS are currently configured), checked for any
configured zones (zoneadm; there are none aside from "global"), verified
with 'vmadm', and checked for any images (imgadm; there are none):
cn40 [0] /bin/df -h
Filesystem size used avail capacity Mounted on
/devices/ramdisk:a 251M 219M 32M 88% /
/devices 0K 0K 0K 0% /devices
/dev 0K 0K 0K 0% /dev
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 3.4G 880K 3.4G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/devices/pseudo/lofi@0:1
376M 355M 22M 95% /usr
/usr/lib/libc/libc_hwcap2.so.1
376M 355M 22M 95% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
zones 16G 650K 13G 1% /zones
zones/cores 10G 31K 10G 1% /zones/global/cores
zones/var 16G 1.9M 13G 1% /var
zones/config 16G 38K 13G 1% /etc/zones
zones/opt 16G 31K 13G 1% /opt
zones/usbkey 16G 127K 13G 1% /usbkey
/usbkey/shadow 13G 127K 13G 1% /etc/shadow
/usbkey/ssh 13G 127K 13G 1% /etc/ssh
swap 3.4G 12K 3.4G 1% /tmp
swap 3.4G 32K 3.4G 1% /var/run
cn40 [0] /usr/sbin/zoneadm list
global
cn40 [0] /usr/sbin/vmadm list
UUID TYPE RAM STATE ALIAS
cn40 [0] /usr/sbin/imgadm list
cn40 [0]
Next, we see that "/usbkey/config" is the same as what we updated itto during our intial configuration and that only our private "admin"
nic has been configured (via 'ifconfig'):
cn40 [0] /bin/cat /usbkey/config
#
# This file was auto-generated and must be source-able by bash.
#
# admin_nic is the nic admin_ip will be connected to for headnode zones.
admin_nic=8:0:27:2d:59:51
admin_ip=10.0.7.40
admin_netmask=255.255.255.0
admin_network=...
admin_gateway=10.0.7.40
headnode_default_gateway=10.0.7.37
dns_resolvers=10.0.7.10
dns_domain=admin.none
ntp_hosts=10.0.7.10
compute_node_ntp_hosts=10.0.7.40
world_nic=8:0:27:91:1f:6e
cn40 [0] /usr/sbin/ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=1100943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
inet 10.0.7.40 netmask ffffff00 broadcast 10.0.7.255
ether 8:0:27:2d:59:51
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
cn40 [0]
A few checks of 'dladm' output further validates the above and SmartOSwas kind enough to link a bridge device (vmwarebr) for us to "e1000g0"
(which we won't be using):
cn40 [0] /usr/sbin/dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
e1000g0 Ethernet up 1000 full e1000g0
e1000g1 Ethernet unknown 0 half e1000g1
cn40 [0] /usr/sbin/dladm show-phys -m
LINK SLOT ADDRESS INUSE CLIENT
e1000g0 primary 8:0:27:2d:59:51 yes e1000g0
e1000g1 primary 8:0:27:91:1f:6e no --
cn40 [0] /usr/sbin/dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
e1000g0 phys 1500 up vmwarebr --
e1000g1 phys 1500 unknown -- --
vmwarebr0 bridge 1500 up -- e1000g0
cn40 [0]
A really useful command under SmartOS is 'sysinfo'. It details variousinfo of the host (think 'smbios', kind of), the OS, and configuration
attributes (including our NIC_TAGs):
cn40 [0] /usr/bin/sysinfo -p
Live_Image='20130111T010112Z'
System_Type='SunOS'
Boot_Time='1359436533'
Manufacturer='innotek GmbH'
Product='VirtualBox'
Serial_Number='0'
VM_Capable='true'
CPU_Type='Unknown'
CPU_Virtualization='none'
CPU_Physical_Cores=0
Nic_Tags=admin,world
UUID='42e86f2d-1d47-49c9-b739-10f7760a0580'
Hostname='08-00-27-2d-59-51'
CPU_Total_Cores=1
MiB_of_Memory=2047
Disk_c1t0d0_size_in_GB=17
NIC_admin='e1000g0'
NIC_world='e1000g1'
Network_Interface_e1000g0_MAC_Address='08:00:27:2d:59:51'
Network_Interface_e1000g0_IPv4_Address='10.0.7.40'
Network_Interface_e1000g0_NIC_Names='admin'
Network_Interface_e1000g0_Link_Status='up'
Network_Interface_e1000g1_MAC_Address='08:00:27:91:1f:6e'
Network_Interface_e1000g1_IPv4_Address=''
Network_Interface_e1000g1_NIC_Names='world'
Network_Interface_e1000g1_Link_Status='unknown'
Bootparam_console='text'
Bootparam_root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA'
Bootparam_smartos='true'
cn40 [0]
The primary reason I've checked out 'sysinfo' above is I'm about to adda new SMF to "cn40" that will automatically identify and configure our
"world_nic" interface (it adds a bridge) for use by our SOSVMs prior to
SmartOS trying to start any SOSVMs. To stage this, below I've created
2 directories (/opt/custom/[smf|methods]) and written the supporting SMF
manifest for our new "world-nic" service. (Notably, by default any SMF
manifests written to "/opt/custom/smf" will be imported by SmartOS at
boot time. The "world-nic" manifest can be downloaded from here.):
cn40 [0] /bin/mkdir -p /opt/custom/smf
cn40 [0] /bin/mkdir -p /opt/custom/methods
cn40 [0] /bin/cat /opt/custom/smf/world-nic.xml
<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
COPYRIGHT: Copyright (c) 2013 Troy Dietrich.
CDDL HEADER START
The contents of this file are subject to the terms of the
Common Development and Distribution License, Version 1.0 only
(the "License"). You may not use this file except in compliance
with the License.
You can obtain a copy of the license at:
http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
CDDL HEADER END
-->
<service_bundle type='manifest' name='world-nic'>
<!--
world-nic.xml is intended for testing purposes and is not intended
for use in production environments. It was written to setup the VM
supporting interface for my virtualized SmartOS compute nodes (which
are running under VirtualBox). This exec'd script assumes 2 NICs have
been provisioned to the SmartOS host, the "admin" NIC (setup during
initial configuration) and a secondary NIC to support any VM vNICs.
The secondary NIC should be identified in '/usbkey/config' as
"NICTAG_nic", but otherwise unconfigured.
Tested against SmartOS 20130111T010112Z.
TD : 1359329221
-->
<service
name='network/world-nic'
type='service'
version='0.1a'>
<dependency
name='network'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/milestone/network:default'/>
</dependency>
<dependent
name='vmadmd'
grouping='require_all'
restart_on='error'>
<service_fmri value='svc:/system/smartdc/vmadmd:default' />
</dependent>
<instance name='default' enabled='true'>
<exec_method
type='method'
name='start'
exec='/opt/custom/methods/world-nic.ksh start'
timeout_seconds='20' />
<exec_method
type='method'
name='stop'
exec='/opt/custom/methods/world-nic.ksh stop'
timeout_seconds='20' />
<property_group name='startd' type='framework'>
<propval name='duration' type='astring' value='transient' />
</property_group>
<template>
<common_name>
<loctext xml:lang='C'>
WorldNic Configuration
</loctext>
</common_name>
<documentation>
<doc_link name='world-nic.ksh' uri='file:///opt/custom/methods/world-nic.ksh'/>
</documentation>
</template>
</instance>
<stability value='Unstable'/>
</service>
</service_bundle>
cn40 [0]
(The supporting "world-nic" shell script can be downloaded from here.Just save it to "/opt/custom/methods/world-nic.ksh" on your lab compute
node to use it with the above manifest.) Below, I've validated and
imported the "world-nic" manifest and we can see via 'svcs -l' the
service has been onlined during the import:
cn40 [0] /bin/ls -ld /opt/custom/methods/world-nic.ksh
-rwxr-xr-x 1 root root 3746 Jan 28 01:24 /opt/custom/methods/world-nic.ksh
cn40 [0] /usr/sbin/svccfg validate /opt/custom/smf/world-nic.xml
cn40 [0] /usr/sbin/svccfg import /opt/custom/smf/world-nic.xml
cn40 [0] /usr/bin/svcs -l world-nic
fmri svc:/network/world-nic:default
name WorldNic Configuration
enabled true
state online
next_state none
state_time January 29, 2013 04:01:22 AM UTC
logfile /var/svc/log/network-world-nic:default.log
restarter svc:/system/svc/restarter:default
dependency require_all/none svc:/milestone/network:default (online)
cn40 [0]
Since the "world-nic" service simply creates a bridge device for usage byour SOSVMs, the "world" NIC doesn't get plumbed or configured, however,
the bridge (vmworld) is created and onlined as seen below:
cn40 [0] /usr/sbin/ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=1100943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
inet 10.0.7.40 netmask ffffff00 broadcast 10.0.7.255
ether 8:0:27:2d:59:51
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
cn40 [0] /usr/sbin/dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
e1000g0 phys 1500 up vmwarebr --
e1000g1 phys 1500 up vmworld --
vmwarebr0 bridge 1500 up -- e1000g0
vmworld0 bridge 1500 up -- e1000g1
cn40 [0]
At this point, we'll close out part 3 since we've now installed andminimally configured our compute node IVM. In part 4, we'll add in our
SmartOS image (template) repository and start creating and checking out
some SOSVMs on "cn40".
see also:
Intro SmartOS Setup pt 1
Intro SmartOS Setup pt 2
Intro SmartOS Setup pt 4
world-nic.xml SMF manifest
world-nic.ksh script called in world-nic.xml
