SmartOS as a Monitoring Host
For many in the UNIX world the demise of Sun Microsystems in 2009 was a frustrating experience. Just as Sun appeared to be embracing the Open Source world, they are gobbled up by the juggernaut that is Oracle who swiftly axed the OpenSolaris project.
However despite their demise Sun open sourced a number of fantastic technologies that have been picked up by new smaller companies (staffed by some of the best ex-Sun engineers) who are committed to continue developing the technologies openly.
In this post I will talk about some of the experiences I have had with SmartOS. A distribution built on the foundations of Solaris but for a cloudier world.
Joyent are the company that backs SmartOS. The distribution has a number interesting design decisions that make it quite different from other platforms. Joyent’s biggest contribution to the open source Illumos kernel has been porting KVM from Linux, this allow SmartOS (or any Illumos distro) to run Linux and Windows VMs rather than just native SmartOS zones. This in and of itself is nothing special but when you put KVM’s virtualisation capabilities on top ZFS, Crossbow, Zones you get a great platform. Joyent’s views on cloud architecture differ from other cloud providers such as Amazon principally in the area of storage. Joyent explores some of the difference here.
Our Usage
Having read about SmartOS awhile ago I was keen to find a good reason to play around with it. Our hosting systems run on top of Xen with a storage SAN, so SmartOS was not a good fit for our general use. However the retirement of our old monitoring server proved an ideal candidate. Rather than simply replace like for like we decided to consolidate a number of our virtual and physical servers onto a dedicated monitoring hypervisor running SmartOS.
This allow us to improve our security with a more heterogeneous mix of hypervisors. In addition more sensitive services were able to be isolated on a single host. Services such as our Puppet Master, Zabbix monitoring software, Sentry as well as remote log collection. We are also implementing a full packet capture IDS system.
SmartOS makes it very easy to get up and running quickly with a good number of base templates in both KVM and native flavours. We’ve used a combination of Debian/Ubuntu and native SmartOS VMs for things like PostgreSQL/MySQL where we wanted to get maximum disk performance. With SmartOS leveraging NetBSD’s pkgsrc you get access to 1000’s of packages, so getting more niche software installed on SmartOS isn’t too much of a chore.
We are still in the process of deploying the IDS functionality, but early signs are ZFS will save us a huge chunk of space with its built in compression support. While on the topic of ZFS, another great advantage of ZFS is that backup of our VMs becomes extremely simple by rolling a zfs send/recv script we can automate incremental backups easily to other locations.
A tool which we have found particularly useful to make the process of creating and configuring VMs has been FiFo. FiFo is an open-source Cloud Management & Orchestration system for SmartOS virtualisation environments, it’s still under heavy development but we have found it to be excellent, well worth a look if you fancy a break from the command line and like to see pretty DTrace graphs!