Managing Your Puppet World with r10k

Puppet is a great tool for managing your infrastructure, most people start of with a few modules to manage various core services. However as the number of modules grows and therefore the complexity better systems are required to keep things running smoothly.

I’m going to give an overview of how we have structured things. This wont be an extensive technical how-to, more tips and pointers on how we have things setup which will hopefully help out others.

Version all the things
Config Examples
- Puppetfile
- /etc/r10k.yaml
Work-Flow & Tips

Version all the things

Version

One of the core tenants of DevOps is to version everything. If you can’t track and trace whats changing in your infrastructure you’re storing up a world of pain as things grow in complexity.

Git makes branching really easy, some would say too easy. By using branching and tagging effectively you can still make large changes to your modules and preserve a stable production state (which can be destroyed and rebuilt identically).

Module Versioning

We used to operate with a single monolithic git repo which contained all our puppet configuration, but once we moved to multiple environments things started getting a bit out of hand. So start by making sure all your modules are in their own repo and versioned. We went through pulling each module out one at a time and versioning them as we went (using semver of course). These were then moved to our local GitLab.

Now you have all your modules source controlled in their own repo, but how do you tie them all together? Enter r10k.

Environment Management with r10k

To manage our environments we use r10k. This builds on librarian-puppet making dynamic environment much easier. r10k will create an environment for each git branch it finds in your r10k repo, you can also selectively build/update environments if needed. The Puppetfile in each branch is then used to build the modules.

The Data (Hiera)

Finally we need to source control our module data in Hiera. This can be either in YAML or JSON, we’ve increasingly found JSON better for our needs. We have a single branch for all our data as we split things based on custom facts, but you can now also have hiera data per environment which is something we are looking at moving to.

Config Examples

Our r10k structure looks like this:

.
├── dist # contains common bits
│   ├── base    # base classes
│   ├── profile # node profiles
│   └── role    # node roles 
├── manifests
│   └── site.pp # single line hiera_include('classes') for ENC
└── Puppetfile  # modules

Puppetfile

forge 'http://forge.puppetlabs.com'

# Puppet Forge Modules
mod 'ajcrowe/supervisord', '0.3.1'
mod 'puppetlabs/firewall', '0.4.2'
mod 'puppetlabs/ntp', '2.0.1'
mod 'puppetlabs/stdlib', '4.1.0'
mod 'ripienaar/concat', '0.2.0'

# Public Github Modules
mod 'my_forked_module',
  :git => 'https://github.com/ajcrowe/module.git',
  :ref => 'my_branch'

mod 'my_public_module',
  :git => 'https://github.com/pancentric/puppet-logstashforwarder',

# Private Gitlab Modules
mod 'module1',
  :git => 'git@gitlab.example.org:puppet/module1.git',
  :ref => '0.1.2'

mod 'module2',
  :git => 'git@gitlab.example.org:puppet/module2.git',
  :ref => '0.3.5'

/etc/r10k.yaml

:cachedir: '/var/cache/r10k'
:sources:
  :r10k:
    remote: 'git@gitlab:ops/r10k.git'
    basedir: '/etc/puppet/environments'

Work-Flow & Tips

So how does this work in practice?

New Environment

cd r10k
git checkout -b new_environment                # create new branch
vim Puppetfile                                 # make changes to module version
git commit -am 'updated module x to version x'
git push origin new_environment	               # push new branch
ssh puppetmaster                             
r10k deploy environment new_environment -p     # deploy new environment

You can also then update a single module with the following:

ssh puppetmaster
r10k deploy module <module> -e <environment>

You can also quite easily automate these changes using CI when things are committed to your r10k repo.

Difference Between Environments

One of the great advantages to this is visibility. Difference between testing and staging:

git checkout testing
git diff staging
diff --git a/Puppetfile b/Puppetfile
index f241eef..7dd72c5 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -12,7 +12,7 @@ mod 'maestrodev/rvm', '1.5.1'
 mod 'maestrodev/wget', '1.3.2'
 mod 'pdxcat/collectd', '1.1.0'
 mod 'puppetlabs/activemq', '0.2.0'
-mod 'puppetlabs/apache', '0.10.0'
+mod 'puppetlabs/apache', '1.0.1'

Development Environment

Another useful tip is to have a bleed edge environment which tracks the master branch of your modules and potentially the latest versions from the forge. You can then set this to automatically build and you can test against this for possible problems before bumping your versioned environment.

The flexibility you gain from r10k’s dynamic environment has certainly help increase the speed and safety of our puppet changes, hopefully the above will help you to as well.

Alex Crowe