02 February 2013

Intro SmartOS Setup pt 1

For a while now, I've wanted to check out Joyent's SmartOS but until
recently, simply didn't have the time. This series of posts details my
initial setup and testing of SmartOS. (To jump straight to SmartOS within
VirtualBox (VBox), see part 3.) For better or worse, I really enjoy
sandbox environments which usually have limited, if any, external access
allowing me greater control over my test environments. With that in
mind, I've set up a self-contained test environment (no external access)
within VBox with 3 "infrastructure" VMs (IVMs) to somewhat present a
stable infrastructure to test SmartOS. Unfortunately, the physical host
on which I'm running VBox is using an AMD chipset, so I can still test
SmartOS handling of OS virtualization but not KVM level virtualization.

Starting things off, my 3 IVMs are:
        - muttley (FreeBSD 8.2 router)
                + simple configuration to handle routing between our
                  3 networks:
                        = vboxnet (
                        = intnet ( ("admin" network))
                        = labworld ( ("world" network))
        - serv1 (CentOS 6.2 services host)
                + initial services are name server (bind), web server
                  (httpd), and time server (ntpd)
                + sits on both "intnet" and "labworld"
        - cn40 (SmartOS compute node)
                + hosts our SmartOS created VMs
                        = VMs will only sit on "labworld"
                + sits on both "intnet" and "labworld"; only directly
                  accessible via "intnet"
Technically speaking, there is also a 4th host, the one running VBox.
The only bits that really matter for this host are:
        - hostname:  glados
        - actually does have real world access
        - VBox virtual NIC (vNIC):  vboxnet0 (host-only network)
        - IP address:
        - network routes for:
                + gateway
                + gateway
Since "glados" is my everyday workstation, rather than reconfigure
name resolution, I simply added entries to "/etc/hosts" for "muttley",
"serv1", "cn40" (and several other compute nodes), and several SmartOS
VMs (SOSVMs). Used throughout these posts are also 3 domains:
        - vbox.none
                + only contains DNS entries for "muttley" and "glados"
                + related network (
        - admin.none
                + contains DNS entries for all compute nodes and the
                  services host
                + related network (
        - world.none
                + contains DNS entires for all "world" facing hosts / SOSVMs
                + related network (
The above domains and their related network zones are maintained by
"serv1" (see part 2). For "muttley" and "serv1", I simply chose OSes
that I already had on hand and was familiar with, you can choose to use
whatever you'd like.

Getting to the IVM setups, the VBox configuration for "muttley" is:
        - 1 vcpu
        - 1024 MB memory
        - 8 GB virtual disk (dynamic / thin provisioned)
        - 3 configured network interfaces:
                + Adapter 1:  Internal network (intnet (
                + Adapter 2:  Host-only (vboxnet0 (
                + Adapter 3:  Internal network (labworld (
After installing "muttley" with a generic FreeBSD 8.2 install, very little
actually needs to be configured. In fact, here is the configuration in
its entirety:
        muttley [0] /bin/cat /etc/rc.conf
        ifconfig_em0="inet netmask"
        ifconfig_em1="inet netmask"
        ifconfig_em2="inet netmask"

        muttley [0] /usr/bin/grep -v ^# /etc/hosts
        ::1     localhost       localhost    glados.vbox.none glados   muttley.vbox.none muttley
        muttley [0] cat /etc/resolv.conf
        domain vbox.none
        search vbox.none world.none admin.none
        muttley [0]
The key part to "/etc/rc.conf" is enabling IP forwarding via:
Also, the "defaultrouter" entry actually points to a non-existent host.
Of note, while not yet configured until part 2, I've configured all host
resolution to be done against "serv1" ( At this point, the
router IVM for our sandbox is now configured and needs to be rebooted.
In part 2, I'll detail the configuration for "serv1", our services host.

see also:
    Intro SmartOS Setup pt 2
    Intro SmartOS Setup pt 3
    Intro SmartOS Setup pt 4


Paul Johnston said...

Just to say how useful I'm finding this stuff, my attempts to set up SmartOS on a variety of oldish hardware has not been very successful but have now got it up in a VirtualBox.

Paul Johnston said...

Really useful stuff, SmartOS can be a pain unless you really know what your doing and now at least I have a VirtualBox instance, even got pkgin to work :-)
Cheers Paul

troy said...


Absolutely awesome to hear and glad to see the posts were useful. I always love hearing how people are using something, especially if a write up I've done contributed in some small way, so thank you for commenting as well. As for SmartOS, at least from my viewpoint, it does become easier as you work with it so definitely hang in there. If you have any questions, feel free to ask.


muzyk10 said...

after so much years still inspiring :)