Configuration management and provisioning are different.

Configuration management tools are used to repeatably and consistently system and application uniformity across clusters of systems at scale. Many of these tools achieve this in three ways: an intuitive command line interface, a lightweight and easily-readable domain-specific language and a comprehensive REST-based API to lower the barrier-to-entry for integrations with other tools. While open-source configuration management tools such as Chef, Ansible, Puppet and Salt have been increasing in popularity over the years, there are also enterprise-grade and regulator-friendly offerings available from vendors such as Dell, Microsoft, HP and BMC.

Configuration management tools are great at keeping a running inventory of existing systems and applications up-to-date. These tools are so good at this, in fact, that many systems administrators and engineers grow tempted into using them to deploy swaths of new systems and configure them shortly thereafter.

I’ve seen this play out at many companies that I’ve worked at. This would usually manifest into an Ansible deployment playbook or a Chef cookbook that eventually became “the” cookbook. The result has always been the same, and if I had to sum this pattern up into a picture, it would look something like this:


Let me explain.

Complexity in simplicity.

One of the darling features of modern configuration management tools is its ability to express complex configuration states in an easily-readable way. This works well for creating an Ansible playbook to configure, say, an nginx instance, but begins to fall apart when trying to, say, provision the instances on which those nginx instances will be hosted and gets really ugly when you attempt to create relationships to deploy application servers with those web servers in trying to stage an environment.

Creating common templates for security groups or firewall templates, instances, storage and the like with configuration management tools outside of what they provide out of the box usually involves writing a lot of boilerplate beforehand. (For example, Ansible has a plugin for staging ebs volumes, but what if you want an EBS resource with specific defaults for certain web or application servers in certain regions? Prepare for a lot of if blocks.) Problems usually also crop up in passing metadata like instance IDs between resources. Because most of the actions done by configuration management tools are idempotent executions, the simple languages they use to to describe configurations don’t natively support variables. Storing metadata and using loops are usually done by breaking out of the DSL and into its underlying language. This breaks readability and makes troubleshooting more complicated.

Provisioning is hard.

Provisioning infrastructure is usually more complicated than configuring the software underlying that infrastructure in two ways.

The first complication arises from the complicated relationships between pieces of infrastructure. Expressing specific environmental nuances of a postgres installation is usually done by way of Chef cookbook attributes and flags. An example of this can be found here. Expressing the three different regions that the databases backing your web app need to be deployed to in a particular environment and the operating system images that those instances need to have will likely require separate layers of attributes: one for image mappings, another for instance sizing and yet another for region mapping. Expressing this in cookbooks gets challenging.

The second complication comes from gathering state. Unless your infrastructure is completely immutable (which is ideal), some state of your infrastructure in the status quo is required before deploying anything. Otherwise, you’ll be in for a surprise or two after you deploy the servers for that environment you thought didn’t exist. Tools like Terraform and AWS CloudFormation keep track of this state to prevent these situations from happening. Chef or Puppet, for example, do not. You can use built-in resources to capture this data and make decisions based on those results, but that puts you back into manipulating their DSLs to do things they weren’t intended to do.

Rollbacks are harder.

chef-provisioning and Ansible provisioning plugins do not support rolling back changes if something fails. This is problematic for three reasons:

  1. Inconsistent environments lead to increased overhead and (usually manual) sysadmin toil. Toil leads to technical debt, and debt leads to slower releases and grumpy teams. Nobody wants to have a grumpy team.
  2. In the cloud, nearly everything costs money. Resources that you deployed during tests that weren’t destroyed afterwards can add up to hefty surprises at the end of your billing cycle.
  3. Cookbook recipes and playbooks will need to account for these stale resources when executing their actions. This can lead to a more complicated codebase and a debt to pay back later on.

Provisioning tools such as Terraform, Cloudformation and Vagrant support rollback out of the box.

Use the right tools.

If you’re staring at a behemoth of a playbook for provisioning your stack or looking to make the move away from chef-provisioning, take a look at XebiaLabs awesome list of tools that make provisioning less complicated. CloudFormation is awesome at provisioning AWS infrastructure (unless you dislike JSON, in which case it is far from it), Vagrant is great at doing the same for physical infrastructure and Packer does a great job of building images as code.

Good luck!

About Me


I’m a DevOps consultant for ThoughtWorks, a software company striving for engineering excellence and a better world for our next generation of thinkers and leaders. I love everything DevOps, Windows, and Powershell, along with a bit of burgers, beer and plenty of travel. I’m on twitter @easiestnameever and LinkedIn at @carlosindfw.


BYOD Part 1: Computers In The Cloud

Computing is expensive. Desktops and laptops cost lots of money. Printers cost even more money. (Printers are really funny, actually; buying one or two isn’t so bad, but once you’re managing tens or hundreds or more laser printers and printing hundreds or thousands of pages per day, the cost of toner/ink and repair skyrocket like a SpaceX shuttle.) Desks cost even more money. Accessories cost even more money. The list goes on and on,infinitum ad nauseum.

Do you like saving money and hate fixing broken computers? Read on.

Now that we live in an age where downloading high-def movies takes less time than starting up your car, leveraging the cloud and having people bring in their own devices has become a highly lucrative alternative. The bring-your-own-device, or BYOD, movement has picked up a lot of steam over the years, so much so that Gartner expects for “half of the world’s companies” to enact it. Over a billion devices are expected to be using BYOD by 2018, and as more and larger companies begin to take advantage of cloud computing, this trend will only accelerate.

I’ll spend the next three posts talking about three key components of most BYOD environment:

  1. Virtual desktops,
  2. Laptops and desktops, and
  3. Mobile phones and tablets

I’ll explain who the major players involved with each component are, their importance in BYOD and some things to watch out for during considerations.

WIth that, let’s start by talking about computers in the cloud.

Computing. Computing Everywhere.

Bring your own stuff

Most bring-your-own-device setups will need a virtual desktop infrastructure, or VDI for short. Without going too deep into the details (and I’ll scratch the surface of this on this week’s Technical Thursdays post), virtual desktops give you computers in the cloud that can be used from anywhere on nearly anything, even phones and tablets.

A VDI is almost always comprised of:

  1. One or more servers on which these virtual machines will be hosted, which are also known as virtual machine hosts or hypervisors,
  2. Management software to start, stop, create, delete and report on machines, and
  3. Software installed on the virtual machines that make the experience more seamless and accessible.

This means that you’ll need the following to get started:

  1. A subscription to a Cloud service like Amazon EC2 or Microsoft Azure or
  2. Your own server(s) in house for hosting virtual machines (most computers made after 2007 should support this with no issues), and
  3. Enough disk space to host the number of machines you’d like to test (1TB is a good starting point),
  4. A virtual machine hypervisor like VMware ESX, Microsoft HyperV (comes with Windows 2008 R2 and up) or Xen, and
  5. A trial version of Citrix XenDesktop, VMware View, Proxmox (free) or SCVMM.

The Players

There are three major players in this space that offer all of the above with varying amounts of complexity:

  1. Citrix (XenDesktop + NetScaler, a load balancer that works really well with VDI),
  2. VMware (VMWare View), and
  3. Microsoft (HyperV + Systems Center Virtual Machine Manager 2012, usually called SCVMM).

Free and open-source solutions also exist, but they might need more love and attention depending on your situation. We’ll go into that a bit later on in this post.

The Upsides

VDI has a number of advantages aside from being a critical component of going full BYOD:

  1. The desktop is replaceable. Jim’s computer broke again? With VDI, you can get him up and running in minutes instead of hours since the desktop itself is a commodity and nothing of importance gets stored on it.

  2. Decreased hardware costs. Depending on your situation, virtual desktops make it possible to order $300 computers in bulk that can do what $2000+ computers can’t.

  3. Increased data security. Over 30 BILLION DOLLARS of valuable data and IP are lost every year due to stolen laptops and devices. Virtual desktops are configured by default to keep ALL of your data in your datacenter and your profits in your bank accounts.

  4. Your desktop is everywhere. Ever wished your team could move around within minutes instead of days? Ever wished to use cheap Chromebooks to access your desktop at work? Virtual desktops make this (and more) possible.

If you’re interested and like pretty charts, here’s a cost savings white paper published by Citrix and Gartner that go into these advantages in more detail. But we all know that every rose has thorns, and VDI is no exception. In fact, if done improperly, VDI can introduce more problems than it solves.

VDI Is A Pay To Play Sport'll still need to plan.

VDI is kind-of like a new car. If you find the right one for you and take care of it, you’ll likely enjoy it significantly more than getting that used Ferrari you thought was “affordable.” (Hint: they never are.)

Deploying computers in the cloud correctly can range from “free” (but expensive on time and labor) to ridiculously expensive depending on how couplex your infrastructure will be. Here is a list of factors that determine this complexity:

  1. Number of machines Much like their physical counterpart, managing VDIs gets increasingly complicated as you add more machines into the mix. However, unlike physical desktops, replacing broken machines or upgrading slow ones can be done with a few mouse clicks. Some setups even allow users to upgrade their own machines on the spot in seconds!
  2. Network bandwidth Virtual desktops are heavily dependent on the quality of the network on which they operate. The less bandwidth they have available to them, the more tweaking you’ll need to do to make people not hate you for taking away their machines.
  3. Your company’s workload Virtual machines on a host share computing resources with each other. Hosts will usually do everything possible to prevent one machine from hogging up resources from other machines (though this can be overridden), which means that the more intensive your use case is, the less likely VDI will work for you without significant tweaking. That said, virtual desktops work well for a wide set of use cases. (Some of Citrix’s clients use virtual desktops to do CAD and heavy graphics rendering, which most people would normally pass on VDI for.)
  4. Remote workers. Users on laptops that travel a lot will often have unpredictable network conditions. While the frameworks mentioned above handle this situation really nicely, it’s important to take this into account early on in your due diligence.

There are also many little hidden costs that can turn into money pits very easily if not taken into consideration early on in the process, such as:

  1. Will you engage Citrix, VMware or third-party consulting services to help you get started, or will you or one of your engineers go solo? (Here’s a hint: Citrix and VMware will always upsell their consulting services.)

  2. Does your company use or require VPN? (The answer is usually “yes,” but most of the products mentioned above support using desktops over plain Internet.)

  3. How many users will get a virtual desktop? How many of them will actually use it? How will they use it?

  4. You’ll always need more storage than you think you do.

  5. Does your company operate under regulatory requirements?

There’s a very easy way to come upon the answers to these questions, and it’s actually a lot easier than you think.

Just do it…as a test

Building a proof-of-concept VDI is pretty straightforward in most cases. You or your admin can probably set one up in an hour or two. Building this and adding users slowly will guide you towards the answers to these questions and help you understand whether VDI is right for your company or group. More importantly, it is much easier to build VDI automation when your VDI is small than when it’s already a massive behemoth that can’t be shut down at any cost. (Why is this important? Want to roll out 10,000 virtual desktops within minutes or automatically create and remove desktops based on server conditions? You’ll need automation to do this and much more.)

Here’s a tutorial on how to set this up with Citrix XenDesktop. Here’s another tutorial for View.

Have fun!

About Me

I’m the founder of, an IT engineering firm in Brooklyn that builds smarter and cost-effective IT solutions that help new and growing companies grow fast. Sign up for your free consultation to find out how.