ajcrowe.io

Kubernetes GCP HTTP Load-Balancer Tips

2016-04-26T00:00:00+01:00

We are in the midst of deploying a Kubernetes setup for one of our products on GCP using GKE and so far it’s been a great experience. I thought I would share a couple of tips around how the ingress resource works with the GCP HTTP(s) Load-balancer.

Google provides some great instructions on getting ingress up and running however we had a couple of things which I thought would be worth sharing to help others.

Custom health check URL for GCP healtcheck resource
Ingress with multiple clusters in a GCP project
Firewall rules to your services
Beware the Backend Services quota

Health check & cluster-uid

Fortunately the first two of these are features which can be enabled by manually editing the GLBC replication controller, you can the cluster-uid example here.

$ kubectl get rc --namespace=kube-system
NAME                             DESIRED   CURRENT   AGE
elasticsearch-logging-v1         2         2         26m
heapster-v1.0.0                  1         1         26m
kibana-logging-v1                1         1         26m
kube-dns-v11                     1         1         26m
kubernetes-dashboard-v1.0.0      1         1         26m
l7-lb-controller-v0.6.0          1         1         26m
monitoring-influxdb-grafana-v3   1         1         26m

$ kubectl edit rc l7-lb-controller-v0.6.0 --namespace=kube-system

You can then edit the args array in the yml to include the following:

- args:
  - --default-backend-service=kube-system/default-http-backend
  - --sync-period=300s
  - --cluster-uid=uid
  - --health-check-path=/healthcheck/

Once saved delete the old pod and let the RC recreate it

$ kubectl delete pod -l name=glbc --namespace=kube-system
pod "l7-lb-controller-v0.6.0-ud9ix" deleted

The uid can be anything you like in our case this references the environment. Once this is set all the various resources in your project will have --uid apended to them allowing meaning cluster wont clash.

One gotcha about changing the healtcheck here is that this will apply to all you ingress resources, so you’ll need to make sure all your services return a 200 on this path.

Firewall

The Google guide suggests configuring your firewall by extracting the NodePort assigned by Kubernetes and then creating firewall rules for this, but once you have more than a few services we have found it far easier to statically assign within a range and create a generic firewall rule which covers these and all future services.

For example if we assign our services to use 30000-31000 as their nodePort:

apiVersion: v1
kind: Service
metadata:
  name: app-svc
  labels:
    app: app
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30000
  selector:
    app: app

We can then create a single run like this:

$ gcloud compute firewall-rules create allow-http-lb \
  --source-ranges 130.211.0.0/22 \
  --target-tags <cluster-tag> \
  --allow tcp:30000-31000

Backend Service Quota

Finally not really a tip as much as a heads up. Google by default gives you 3 Backend Services, however this is really inadequate once you start creating multiple services in your ingress resources.

You can request more, however it appears there is a cap of 30 per project without special approval.

I hope some of the manual step and limitations above will soon be replaced with automated options as Ingress moves from beta to production ready in 1.3, but even now in beta we have been impressed with how well everything fit together and works.

Kubernetes GCP HTTP Load-Balancer Tips was originally published by Alex Crowe at ajcrowe.io on April 26, 2016.

Managing Your Puppet World with r10k

2015-11-12T21:55:12+00:00

Puppet is a great tool for managing your infrastructure, most people start of with a few modules to manage various core services. However as the number of modules grows and therefore the complexity better systems are required to keep things running smoothly.

I’m going to give an overview of how we have structured things. This wont be an extensive technical how-to, more tips and pointers on how we have things setup which will hopefully help out others.

Version all the things
Config Examples
- Puppetfile
- /etc/r10k.yaml
Work-Flow & Tips

Version all the things

One of the core tenants of DevOps is to version everything. If you can’t track and trace whats changing in your infrastructure you’re storing up a world of pain as things grow in complexity.

Git makes branching really easy, some would say too easy. By using branching and tagging effectively you can still make large changes to your modules and preserve a stable production state (which can be destroyed and rebuilt identically).

Module Versioning

We used to operate with a single monolithic git repo which contained all our puppet configuration, but once we moved to multiple environments things started getting a bit out of hand. So start by making sure all your modules are in their own repo and versioned. We went through pulling each module out one at a time and versioning them as we went (using semver of course). These were then moved to our local GitLab.

Now you have all your modules source controlled in their own repo, but how do you tie them all together? Enter r10k.

Environment Management with r10k

To manage our environments we use r10k. This builds on librarian-puppet making dynamic environment much easier. r10k will create an environment for each git branch it finds in your r10k repo, you can also selectively build/update environments if needed. The Puppetfile in each branch is then used to build the modules.

The Data (Hiera)

Finally we need to source control our module data in Hiera. This can be either in YAML or JSON, we’ve increasingly found JSON better for our needs. We have a single branch for all our data as we split things based on custom facts, but you can now also have hiera data per environment which is something we are looking at moving to.

Config Examples

Our r10k structure looks like this:

.
├── dist # contains common bits
│   ├── base    # base classes
│   ├── profile # node profiles
│   └── role    # node roles 
├── manifests
│   └── site.pp # single line hiera_include('classes') for ENC
└── Puppetfile  # modules

Puppetfile

forge 'http://forge.puppetlabs.com'

# Puppet Forge Modules
mod 'ajcrowe/supervisord', '0.3.1'
mod 'puppetlabs/firewall', '0.4.2'
mod 'puppetlabs/ntp', '2.0.1'
mod 'puppetlabs/stdlib', '4.1.0'
mod 'ripienaar/concat', '0.2.0'

# Public Github Modules
mod 'my_forked_module',
  :git => 'https://github.com/ajcrowe/module.git',
  :ref => 'my_branch'

mod 'my_public_module',
  :git => 'https://github.com/pancentric/puppet-logstashforwarder',

# Private Gitlab Modules
mod 'module1',
  :git => 'git@gitlab.example.org:puppet/module1.git',
  :ref => '0.1.2'

mod 'module2',
  :git => 'git@gitlab.example.org:puppet/module2.git',
  :ref => '0.3.5'

/etc/r10k.yaml

:cachedir: '/var/cache/r10k'
:sources:
  :r10k:
    remote: 'git@gitlab:ops/r10k.git'
    basedir: '/etc/puppet/environments'

Work-Flow & Tips

So how does this work in practice?

New Environment

cd r10k
git checkout -b new_environment                # create new branch
vim Puppetfile                                 # make changes to module version
git commit -am 'updated module x to version x'
git push origin new_environment	               # push new branch
ssh puppetmaster                             
r10k deploy environment new_environment -p     # deploy new environment

You can also then update a single module with the following:

ssh puppetmaster
r10k deploy module <module> -e <environment>

You can also quite easily automate these changes using CI when things are committed to your r10k repo.

Difference Between Environments

One of the great advantages to this is visibility. Difference between testing and staging:

git checkout testing
git diff staging
diff --git a/Puppetfile b/Puppetfile
index f241eef..7dd72c5 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -12,7 +12,7 @@ mod 'maestrodev/rvm', '1.5.1'
 mod 'maestrodev/wget', '1.3.2'
 mod 'pdxcat/collectd', '1.1.0'
 mod 'puppetlabs/activemq', '0.2.0'
-mod 'puppetlabs/apache', '0.10.0'
+mod 'puppetlabs/apache', '1.0.1'

Development Environment

Another useful tip is to have a bleed edge environment which tracks the master branch of your modules and potentially the latest versions from the forge. You can then set this to automatically build and you can test against this for possible problems before bumping your versioned environment.

The flexibility you gain from r10k’s dynamic environment has certainly help increase the speed and safety of our puppet changes, hopefully the above will help you to as well.

Managing Your Puppet World with r10k was originally published by Alex Crowe at ajcrowe.io on November 12, 2015.

CoreOS XenServer Install Fix

2014-07-16T22:55:12+01:00

I’ve been playing around with the new OS on the block CoreOS. After running it locally with Vagrant I wanted to get it up and running on our XenServer test pool.

I grabbed the ISO, created a new VM and installed it without issue. However after rebooting it just looped indefinity.

After a chat in the helpful IRC channel it turns out kexec was not being disabled for Xen installs, the fix below sorts this out post install. This has been addressed and should be in the alpha channel shortly but in the mean time hopefully someone finds this useful.

After the install you’ll need to tweak the boot partition as follows:

sudo -s
mount LABEL=EFI-SYSTEM /mnt
echo "DEFAULT coreos.A" > /mnt/syslinux/default.cfg
umount /mnt

This might disappear after an update in which case booting the install os and repeating the above should work, either way this should get you up and running.

CoreOS XenServer Install Fix was originally published by Alex Crowe at ajcrowe.io on July 16, 2014.

Building Atom on Ubuntu 14.04

2014-05-06T00:00:00+01:00

Having seen news GitHub have Open Sourced their take on a text editor I thought I would give it a whirl. There are no pre-build packages available for Linux so lets build it ourselves.

Installing Node

Ubuntu 14.04 has a pretty up to date version of node bundled which should be fine, but I’ve grabbed the latest version and built it myself.

sudo apt-get install build-essential
...
wget http://nodejs.org/dist/v0.10.28/node-v0.10.28.tar.gz
...
tar -xzf node-v0.10.28.tar.gz
cd node-v0.10.28
./configure && make && sudo make install
...
node -v
0.10.28
npm -v
1.4.9

Build Atom

Now lets follow the instructions in the Atom README.

sudo apt-get install libgnome-keyring-dev
npm config set python /usr/bin/python2 -g # to ensure that gyp uses Python 2
git clone https://github.com/atom/atom
cd atom
script/build # Creates application at /tmp/atom-build/Atom
sudo script/grunt install # Installs command to /usr/local/bin/atom

However when we run the script /usr/local/bin/atom we get nothing, not very useful! However we can see there is a problem missing lib when we run ldd on the binary:

ldd /usr/local/share/atom/atom | grep "not found"
libudev.so.0 => not found

So lets grab the older libudev0 from the Ubuntu 12.04 package repos

wget http://mirrors.kernel.org/ubuntu/pool/main/u/udev/libudev0_175-0ubuntu9_amd64.deb
dpkg -i libudev0_175-0ubuntu9_amd64.deb

Atom should have all it needs to launch!

$ atom

Any questions post them bellow.

Building Atom on Ubuntu 14.04 was originally published by Alex Crowe at ajcrowe.io on May 06, 2014.

Changing GitLab Repository Access Rights

2014-02-16T00:00:00+00:00

We love using GitLab at Pancentric, if you haven’t checked it out you really should. However one minor grip we’ve had is permissions.

By default only users who are owners on a project can create milestones and open/close all the issues present. This makes a lot of sense on projects which are available externally, but when you are dealing with trusted colleagues this just gets in the way of getting things done!

The assignment of permissions is cumulative so each level up gets all the permissions of the previous with some additions. These are defined in the app/models/ability.rb file.

Simply shuffling symbols about allow you to change what each level grants the user, in our case we move the :admin_issue and :admin_milestone from the project_master_rules into the project_report_rules method.

def project_report_rules
  project_guest_rules + [
    :download_code,
    :fork_project,
    :write_project_snippet,
    # moved these from project_master_rules role
    :modify_issue,
    :admin_issue,
    :admin_milestone
  ]
end

You can shuffle these to suit your own needs, but remember if you want to upgrade you’ll need to revert the change before you can pull newer versions of GitLab.

*note this is accurate as of GitLab 6.5, YMMV with other versions

Changing GitLab Repository Access Rights was originally published by Alex Crowe at ajcrowe.io on February 16, 2014.

uWSGI Emperor behind Nginx on Debian

2014-02-15T00:00:00+00:00

uWSGI is an extremely flexible and fast application server which is rapidly growing in popularity. From its early roots in Python it has quickly adding support for additional languages such as Ruby, PHP and may others.

I’ll be looking at how we’ve deployed uWSGI behind Nginx using virtual hosting. I will also mention some of the clever features (with names like zerg and broodlord) to provide dynamic resource allocation depending on load.

Installation
Configuring the Emperor
Nginx Virtual Host
Auto-Scaling Workers

Installation

There are a few ways to install uWSGI, but I find the best approach is to use pip. This will install the latest stable version. It’s worth noting development of uWSGI moves at quite a pace so if you want even more stability you can use the LTS (long term support) release

apt-get install build-essential python-dev
...
pip install uwsgi
# Or for the LTS
pip install http://projects.unbit.it/downloads/uwsgi-lts.tar.gz
...
uwsgi --version
1.9.20

That was the easy! Remember to make sure the binary (defaults to /usr/local/bin/uwsgi) is able to be executed by the user you’ll be running your applications as.

Configuring the Emperor

uWSGI supports a mind boggling number of configuration formats. We will be using the ini format.

First task is to create the folder structure where the ini, log, pid and socket files will live.

$ mkdir -p /etc/uwsgi/apps-{available,enabled} # this allow us to switch apps on and off
$ mkdir -p /var/{log,run}/uwsgi

Create the Emperor configuration

[uwsgi]
daemonize = /var/log/uwsgi/emperor.log 
touch-logreopen = /etc/uwsgi/logrotate.trigger
emperor = /etc/uwsgi/apps-enabled
pidfile = /var/run/uwsgi/emperor.pid
die-on-term = true

All the above options are detailed in full in the extensive uWSGI docs. You can run the emperor with uwsgi --ini /etc/uwsgi/emperor.ini and check everything is working by looking at the log file and ensuring the pid has been created.

Next we need an init script mine is available here but you can use something like supervisor if that’s more your thing.

You can now place your applications ini files in /etc/uwsgi/apps-available and symbolically link them into your apps-enabled folder. You should end up with a folder structure looking like this.

# config
/etc/uwsgi
├── apps-available
│   ├── app1.ini
│   └── app2.ini
├── apps-enabled
│   ├── app1.ini -> ../apps-available/asone-cs.ini
│   └── app2.ini -> ../apps-available/asone.ini
├── emperor.ini
└── logrotate.trigger
# logs
/var/log/uwsgi
├── app1.log
├── app2.log
└── emperor.log
# pid and sockets
/var/run/uwsgi
├── app1.sock
├── app2.sock
└── emperor.pid

Nginx Virtual Host

Configuring Nginx to connect to your uWSGI instance couldn’t be easier. This is due to the native uwsgi protocol support.

The simplest configuration requires just two lines include uwsgi_params; and uwsgi_pass unix:/path/to/socket; or uwsgi_pass hostname:port

Here is an example complete vhost.

server {
	listen 80;
	server_name myapp.example.net;

	root /var/www/myapp;

	access_log /var/log/nginx/myapp.access;
	error_log /var/log/nginx/myapp.error;

	location / {
		include uwsgi_params;
		uwsgi_pass unix:/var/run/uwsgi/myapp.sock;
	}

	location /static/ {
		alias /var/www/myapp/assets/static/;
	}
}

You can also do other lovely things like caching but this will get you up and running.

Auto-Scaling Workers

*note, you must be using Linux and TCP sockets for this to work.

One of the downsides to this configuration is each application has its own processes and does not share a pool of workers. uWSGI has a solution to this which allows your instances to request addition workers processes from their Emperor called zergs

To configure your Emperor process to service these requests we need to add the following emperor-broodlord = num; where num is how many workers are available, to our existing emperor.ini.

Next we configure the application to request these zergs. We need to modify our existing config and add a zerg section. Below is an example of a before and after.

# exist config
[uwsgi]
socket = localhost:3031
master = true
module = myapp.wsgi
processes = 1
disable-logging = true

# modified config
[uwsgi]
socket = localhost:3031
master = true
module = myapp.wsgi
processes = 1
disable-logging = true
# new zerg config
vassal-sos-backlog = 10
zerg-server = /var/run/uwsgi/myapp-broodlord.sock

[zerg]
# setting to use for the zerg workers
zerg = /var/run/uwsgi/myapp-broodlord.sock
master = true
module = myapp.wsgi
processes = 1
disable-logging = true
idle = 30
die-on-idle = true

This configuration will result in the application asking for reinforcements when it has a backlog of requests greater than 10. The zergs will automatically die when idle for 30 seconds, returning to the pool for other applications to use.

I suggest you can play around with these settings and tune them to your specific workload. Set your idle time too low and you might find zergs are constantly starting and stopping, set it too high and you’ll not have resources available for other applications.

There are tons of other great features in uWSGI, I encourage you to explore the docs!

uWSGI Emperor behind Nginx on Debian was originally published by Alex Crowe at ajcrowe.io on February 15, 2014.

Puppet, Hiera and hashes

2013-10-07T00:00:00+01:00

Puppet is a rapidly evolving tool, since we started using it a couple of years ago the language and tools have improved hugely and so has the community around it.

I’ve recently found some more time to moved our data into yaml to be used by hiera (we might change some of this to JSON shortly). We have have found using the create_resources function combined with hashes from Hiera to be a really clean and simple way to get our resources configured.

For example:

---
classes: 
  - role::web::python::staging

pythonapps:
  myapp:
    appname: myapp
    gitremote: 'git@github.com:pancentric/myapp.git'

uwsgiapps:
  myapp:
    processes: 2
    enabled: true
    broodlord: true

envs_myapp:
  DJANGO_SETTINGS_MODULE: myapp.settings
  DJANGO_SECRET_KEY: 'secretkey'
  DJANGO_CONFIGURATION: StagingSettings
  POSTGRES_PASSWORD: mypassword
  # other params...

Then we can query Hiera for the hashes and pass them to the exceedingly useful create_resources function which will loop over each hash passing the nested hashes to the function specified.

class profile::web::python {
  # other classes...
  include ::uwsgi
  include ::pythonenv
  
  $pythonenvs = hiera('pythonenvs')
  $uwsgiapps  = hiera('uwsgiapps')

  create_resources(::pythonenv::app, $pythonenvs)
  create_resources(::uwsgi::app, $uwsgiapps)
}

This method also a great way of passing a larger number of key/value pairs into your code. In our instance we have a number of common environment variables for our Django app which both pythonenv::app and uwsgi::app need to use. We define these once as a hash in our yaml and then do a hiera_hash('env_myapp') lookup in the puppet code of both modules. This can then be used by the templates as needed.

If you had common and node specific environment variables by using the hiera_hash() function hiera will merge all matching results into one hash with the node specific overriding the common.

All of this allows for not only great separation of data and code, but a much more DRY configuration.

Puppet, Hiera and hashes was originally published by Alex Crowe at ajcrowe.io on October 07, 2013.

SmartOS as a Monitoring Host

2013-09-10T00:00:00+01:00

For many in the UNIX world the demise of Sun Microsystems in 2009 was a frustrating experience. Just as Sun appeared to be embracing the Open Source world, they are gobbled up by the juggernaut that is Oracle who swiftly axed the OpenSolaris project.

However despite their demise Sun open sourced a number of fantastic technologies that have been picked up by new smaller companies (staffed by some of the best ex-Sun engineers) who are committed to continue developing the technologies openly.

In this post I will talk about some of the experiences I have had with SmartOS. A distribution built on the foundations of Solaris but for a cloudier world.

Joyent are the company that backs SmartOS. The distribution has a number interesting design decisions that make it quite different from other platforms. Joyent’s biggest contribution to the open source Illumos kernel has been porting KVM from Linux, this allow SmartOS (or any Illumos distro) to run Linux and Windows VMs rather than just native SmartOS zones. This in and of itself is nothing special but when you put KVM’s virtualisation capabilities on top ZFS, Crossbow, Zones you get a great platform. Joyent’s views on cloud architecture differ from other cloud providers such as Amazon principally in the area of storage. Joyent explores some of the difference here.

Our Usage

Having read about SmartOS awhile ago I was keen to find a good reason to play around with it. Our hosting systems run on top of Xen with a storage SAN, so SmartOS was not a good fit for our general use. However the retirement of our old monitoring server proved an ideal candidate. Rather than simply replace like for like we decided to consolidate a number of our virtual and physical servers onto a dedicated monitoring hypervisor running SmartOS.

This allow us to improve our security with a more heterogeneous mix of hypervisors. In addition more sensitive services were able to be isolated on a single host. Services such as our Puppet Master, Zabbix monitoring software, Sentry as well as remote log collection. We are also implementing a full packet capture IDS system.

SmartOS makes it very easy to get up and running quickly with a good number of base templates in both KVM and native flavours. We’ve used a combination of Debian/Ubuntu and native SmartOS VMs for things like PostgreSQL/MySQL where we wanted to get maximum disk performance. With SmartOS leveraging NetBSD’s pkgsrc you get access to 1000’s of packages, so getting more niche software installed on SmartOS isn’t too much of a chore.

We are still in the process of deploying the IDS functionality, but early signs are ZFS will save us a huge chunk of space with its built in compression support. While on the topic of ZFS, another great advantage of ZFS is that backup of our VMs becomes extremely simple by rolling a zfs send/recv script we can automate incremental backups easily to other locations.

A tool which we have found particularly useful to make the process of creating and configuring VMs has been FiFo. FiFo is an open-source Cloud Management & Orchestration system for SmartOS virtualisation environments, it’s still under heavy development but we have found it to be excellent, well worth a look if you fancy a break from the command line and like to see pretty DTrace graphs!

SmartOS as a Monitoring Host was originally published by Alex Crowe at ajcrowe.io on September 10, 2013.

The Wonderful Evolving Mobile Landscape

2013-08-07T00:00:00+01:00

For many the battle for the mobile OS is a done deal. Android won and Apple has slipped into 2nd place, Not too dissimilar the battle of the early 90’s when the emerging Microsoft rose to dominate desktop computing. Android has now passed 900 million device activations, almost double iOS. So surely it’s a done deal, cue Android hegemony?

However more now than ever alternatives are springing up. I can imagine thoughts springing to your mind like “how can compete?”, “They wont have any apps!” or even simply “Why bother?”. I’ve certainly wondered the same things, but there is nagging part of my brain saying this time is different.

Not Your Fathers OS War

A key difference between now and the last time OS’s were vying for dominance is, of course you’ve guess it the Internet. This great leveller, doesn’t care if your on Windows, OSX, Linux, Android, iOS. Its open protocols and standards make it an equal opportunities platform. So this begs the question, if the web continues its march to become the platform for delivering applications, content and connected experiences does the OS even matter in the long run?

Google has taken this idea the furthest with ChromeOS, however even the most ardent fan would have a hard time convincing people it’s ready to replace your desktop. But if you, like me subscribe to the view that we are at the end of the beginning and not the beginning of the end of the changes a truly connected world will bring these weakness will slowly fade away. So are we looking at a future where operating systems are reduced to merely glorified browsers?

Maybe, the pull towards a world driven by the web is undeniable. But I expect the OS to matter for a little longer. So this trend is the first key benefit I see for alternative platforms.

People, All the People

Mobile technology is an amazing global success story. There are 5 billion mobile phones in use globally of which some 1 billion are smart phones, that’s a lot of dumb phones that’ll be turning smart over the next few years.

Mozilla for one has explicitly stated it hopes to providing an open source OS built around the web to cater for this emerging demand. But a broader point can be made that the shear numbers involved will create the opportunity for the small to survive, and even flourish.

Ubiquitous Black Slabs

Despite the strongest wishes of Apple, hardware continues to become more commoditised. With each passing year, phones get harder, better, faster, stronger

Daft Punk - Harder Better Faster Stronger

As these devices get cheaper and smart phones become simply phones new opportunities will open up in how we use these portable computers. Within a year or two large number of people will be carrying about enough power in their pocket to run the vast majority of the applications. Ubuntu is certainly buying into the converged future where the distinction between desktop, laptop, tablet, phone and everything in between starts to disappear.

Exciting Times

It’s incredible to think that only 6 years ago Apple released the first iPhone and Android was racing to catch up. This ushered in an new era of what was possible on a mobile device, opening up new markets and possibilities. I certainly hope the next 6 bring increased choice and the unexpected!

The Wonderful Evolving Mobile Landscape was originally published by Alex Crowe at ajcrowe.io on August 07, 2013.

Nexenta Build

2013-05-21T00:00:00+01:00

*please note I started this post awhile ago, so most of this refers to work from last summer

From my first experience of ZFS in FreeBSD 7 it was clear this was a game changing approach to storage. It’s mix of features, easy of use, scalability and robustness make it an ideal platform for enterprise storage, and the best part is it’s open source!

When we were looking at options for a NAS/SAN on a limited budget (when aren’t they) and offered us something we could grow with, the features of OpenSolaris made a lot of sense. It would allow us to continue to use commodity hardware of our choice (within some limits) and let us customise the system for our needs. But we also wanted support and a real company behind the product, enter Nexenta

I’m a big fan of choice, and when it comes to the world of storage the big boys still dominate. The “Big 5” primarily sell storage systems built on their own proprietary combination of hardware and software. This results in high prices, lock-in and lack of flexibility. Increasingly new OpenStorage companies have begun to emerge to try and take on the big boys and inject some more competition into the market.

Based on the solid core of OpenSolaris with Debian’s Apt package manager Nexenta has built a storage based OS distribution which offers ZFS, COMSTAR, Kernel based CIFS/NFS all wrapped in a web based GUI (don’t worry there is a CLI!).

Below Ill talk talk through how we got on using Nexenta. I hope someone will find some of the tips and information useful if they are setting out on a similar journey.

Initial Kit
Configuration
Thoughts
Tips

Initial Kit

Head-Node

HP Proliant DL360G7
Intel Xeon E5649 2.56GHz
48GB RAM
2x 146GB syspool
LSI 9200-8e
Intel CX4 10GbE NIC

JBOD

DataON DNS1600 (Dual Controllers)
8x 1TB Toshiba 6G SAS (Data)
2x ZeusRAM 8GB 6G SAS (Zil)
1x Talos C 230GB SSD (L2Arc)

Network

2x HP Procurve 2910al-24G
2x HP dual-port 10GbE CX4 al Module

Configuration

The pool configuration is 4 mirrored vdevs with mirrored Zils and a single L2Arc. We opted for mirrored vdevs rather than raidzX for performance as our use-case was for a write heavy workload. Mirrored vdevs also make rebuild times more predictable and recovery easier should the worst happen easier.

We accelerated our pool using a mirrored set of ZeusRAM 8GB drives. These ultra fast RAM drives have large super capacitors to protect the data in case of power failure and suffer no write performance degradation over time like some SSDs can. We also added a 230GB OCZ Talos as a L2Arc (read cache).

This is the pool layout.

nmc@nexenta:/$ zpool status
pool: tank
 state: ONLINE
 scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        tank                       ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            c0t50000393B8C93020d0  ONLINE       0     0     0
            c0t50000393B8C93064d0  ONLINE       0     0     0
          mirror-1                 ONLINE       0     0     0
            c0t50000393B8C930A8d0  ONLINE       0     0     0
            c0t50000393B8C930ACd0  ONLINE       0     0     0
          mirror-2                 ONLINE       0     0     0
            c0t50000393B8C930DCd0  ONLINE       0     0     0
            c0t50000393B8C930E0d0  ONLINE       0     0     0
          mirror-3                 ONLINE       0     0     0
            c0t50000393B8C93104d0  ONLINE       0     0     0
            c0t50000393E8CAF744d0  ONLINE       0     0     0
        logs
          mirror-4                 ONLINE       0     0     0
            c0t5000A72030044473d0  ONLINE       0     0     0
            c0t5000A72030044478d0  ONLINE       0     0     0
        cache
          c0t5E83A970000020F1d0    ONLINE       0     0     0

errors: No known data errors

We created some thinly provisioned zvols which we made available to our hypervisors over iSCSI with COMSTAR. Taking advanced of the MPIO support in COMSTAR to add extra resiliency we used two HP 2910al-24G switches and created two completely separate paths to the storage from the hypervisors. In addition we configured our JBOD to be active/active by connecting each port of the LSI-9200-8e to a SAS port on each controller (see the tips on checking MPxIO is working).

Thoughts

Since the system went live we’ve had a few blips, most of the solutions to those are covered in the tips sections below. The biggest piece of advice I would give anyone looking a similar build is to find a partner who has experience of deploying such a system, ours has been invaluable when little problems have occurred.

We have been very pleased with the performance of the system and look forward to having it expand over the coming years.

Tips

Few tips if you’re looking at a Nexenta build

Use The HCL!

As NexentaStor is based on an older OpenSolaris build (4.0 will be based on Illumos) it’s driver support can be patchy as Nexenta has to backport drivers, so make sure you stick to kit on the HCL, Intel & LSI are good choices here.

We’ve had issues with our Intel CX4 NICs when turning on flow control, this bug is still being investigated by Nexenta and only happens with the mode set to bi or tx not rx (It should be noted this can only be changed from expert mode so it’s probably debatable whether it’s supported)

Don’t Upgrade the Community Edition

If you make use of the Community Edition and then choose to go Enterprise make sure you reinstall from scratch, we had issues with missing features and strange performance problems on an upgraded node.

Configure your network correctly

Make sure you are using decent quality networking kit suitable for iSCSI. Enable jumbo frames, disable spanning tree and enable flow control (this is useful if you’re going to be mixing 1Gbe with 10Gbe).

Check your disks are using MPxIO

Assuming you’re using a JBOD that supports active/active controllers you’ll want to make use of MPxIO. However often drives do not get automatically picked up as being multipath capable by the scsi_vhci driver. You can check if it’s enabled by looking under the ‘Attach’ column in disks on NMC.

nmc@muninn:/$ show lun disk
LUN ID      Device    Type      Size      Volume     Mounted Attach GUID
c0t5*020d0  sd50      disk      1TB       tank       no      mpxio  50000393b8c93020
c0t5*064d0  sd59      disk      1TB       tank       no      mpxio  50000393b8c93064
c0t5*074d0  sd56      disk      1TB       tank       no      mpxio  50000393b8c93074
c0t5*0A8d0  sd64      disk      1TB       tank       no      mpxio  50000393b8c930a8

If you see mpxio, you’re set. If you see mpt_sas you’ll need append your disk make and model to the /kernel/drv/scsi_vhci.conf file. This file has a specific format, to make my Toshiba, STEC and OCZ drives work I changed the default:

scsi-vhci-failover-override =
"3PARdataVV", "f_sym",
"COMPELNTCompellent Vol", "f_sym";

becomes

scsi-vhci-failover-override =
"3PARdataVV", "f_sym",
"COMPELNTCompellent Vol", "f_sym",
"OCZ     TALOS", "f_sym",
"STEC    ZeusRAM", "f_sym",
"TOSHIBA MK1001TRKB", "f_sym";

It’s important to note if the manufacturer name is less than 8 characters you need to add spaces before the module. Once updated you need to reboot for the changes to take effect. You can find your vendor and model by looking at the output of format or iostat -E

Hosts file

We had some issues with the management interface appearing to lock up, these turned out to be due to stale information in the /etc/hosts file. Once we updated this to reflect the real primary interface IP the NMC started working again!

XenServer Active/Active MPIO

We use XenServer as our primary hypervisor and found it works really well with NexentaStor when using iSCSI and MPIO. However to get it working in active/active mode you have to tweak the /etc/multipath.conf file as follows

...
device {
        vendor                  "NEXENTA"
        product                 "(COMSTAR|NEXENTASTOR)"
        path_grouping_policy    group_by_prio
        failback                immediate
        no_path_retry           queue
}
...

Nexenta Build was originally published by Alex Crowe at ajcrowe.io on May 21, 2013.

Welcome to my Blog!

2013-05-20T00:00:00+01:00

Over time I hope to be filling this site with some decent content on all the technical things I do day to day to hopefully help others.

In the mean time here is a picture of a cat.

Welcome to my Blog! was originally published by Alex Crowe at ajcrowe.io on May 20, 2013.