Winning at Ansible: How to manipulate items in a list!

The Problem

Ansible is a great configuration management platform with a very, very extensible language for expressing yoru infrastructure as code. It works really well for common workflows (deploying files, adding authorized_keys, creating new EC2 instances, etc), but its limitations become readily apparent as you begin embarking in more custom and complex plays.

Here’s a quick example. Let’s say you have a playbook that uses a variable (or var in Ansible-speak) that contains a list of tables, like this:

- file_name: ssh_config
file_path: /usr/shared/ssh_keys
file_purpose: Shared SSH config for all mapped users.
- file_name: bash_profile
file_path: /usr/shared/bash_profile
file_purpose: Shared .bash_profile for all mapped users.

(You probably wouldn’t manage files in Ansible this way, as it already comes with a fleshed-out module for doing things with files; I just wanted to pick something that was easy to work with for this post.)

If you wanted to get a list of file_names from this var, you can do so pretty easily with set_fact and map:

- name: "Get file_names."
file_names: "{{ important_files | map(attribute='file_name') }}"

This should return:

[ u'/usr/shared/ssh_keys', u'/usr/shared/bash_profile' ]

However, what if you wanted to modify every file path to add some sort of identifier, like this:

[ u'/usr/shared/ssh_keys_12345', u'/usr/shared/bash_profile_12345' ]

The answer isn’t as clear. One of the top answers for this approach suggested extending upon the map Jinja2 filter to make this happen, but (a) I’m too lazy for that, and (b) I don’t want to depend on code that might not be on an actual production Ansible management host.

The solution

It turns out that the solution for this is more straightforward than it seems:

- name: "Set file suffix"
file_suffix: "12345"

- name: "Get and modify file_names."
file_names: "{{ important_files | map(attribute='file_name') | list | map('regex_replace','(.*)','\\1_{{ file_suffix }}') | list }}"

Let’s break this down and explain why (I think) this works:

  • map(attribute='file_name') selects items in the list whose key matches the attribute given.
  • list casts the generated data structure back into a list (I’ll explain this below)
  • map('regex_replace','$1','$2') replaces every string in the list with the pattern given. This is what actually does what you want.
  • list casts the results back down to a list again.

The thing that’s important to note about this (and the thing that had me hung up on this for a while) is that every call to map (or most other Jinja2 filters) returns the raw Python objects, NOT the objects that they point to!

What this means is that if you did this:

- name: "Set file suffix"
file_suffix: "12345"

- name: "Get and modify file_names."
file_names: "{{ important_files | map(attribute='file_name') | map('regex_replace','(.*)','\\1_{{ file_suffix }}') }}"

You might not get what you were expecting:

ok: [localhost] => {
    "msg": "Test - <generator object do_map at 0x7f9c15982e10>."

This is sort-of, kind-of explained in this bug post, but it’s not very well documented.


This is the first of a few blog posts on my experiences of using and failing at Ansible in real life. I hope that these save someone a few hours!

About Me

Carlos Nunez is a site reliability engineer for Namely, a modern take on human capital management, benefits and payroll. He loves bikes, brews and all things Windows DevOps and occasionally helps companies plan and execute their technology strategies.

Sleep better with two simple shortcuts.


Control + ⌘ + Shift + G, and
Home button triple-click.


Exposing ourselves to bright screens at night while checking our Facebook feed or reddit posts might not be as harmless as it seems. Tons of research, like this and this suggest that viewing things on bright screens right before bed makes our brains think that we’re in daylight longer than we actually are and, consequently, prevent us from falling asleep sooner than we should be. This combined with our early-start culture has been shown to lead to fatigue, decreased concentration and, in some folks, depression.

Additionally, other research has shown that prolonged exposure to artificial light (like those in most offices or our phones) can, over time, damage our eyes’ ability to adjust to incoming light and weaken their sensitivity to it.

I didn’t notice any of this until a Slashdot post introduced me to Flux several years ago. Before using this application, I was usually tired and sore (I rode my bike much more often back then) most of the time, but didn’t think much of it. I went out often back then, and most of the people I came across were just as or more tired than I was, so I thought I was fine.

I would have never thought that simply cutting blue light at night would have improved my sleeping patterns as much as they did. I was honestly surprised and, since then, intrigued about doing everything I could to improve my sleeping habits.

A few months after (happily) using Flux, I saw a developer on our floor who had the oddest setup I’ve seen up until then: a small, vertically-oriented monitor with a completely dark desktop with huge icons and a huge terminal font size. I didn’t ask him much about it, but given how exceptional he was at what he did, I naturally thought: I need to try this.

I wasn’t ready for what happened next. I had absolutely no idea that copying some developer’s setup would completely transform the way that I worked going forward.

Working on dark desktops like the one above (my current working setup) has helped me:

  • Focus better (white text on a dark background is much more readable),
  • Work longer without cutting into my sleep,
  • Utilize smaller real estate much more efficiently (my ideal monitor is a 19″ widescreen), and
  • Realize just how few companies actually support this (Material Design, I’m looking at you!)

Be jealous of my sweet eye-saving setup.

If you’re interested in giving this a try, here are two shortcuts you can set up easily on your Mac and iPhone that’ll make it super easy to toggle between the two:

For Your Mac

  1. Hit the ⌘ and Space key together to open Spotlight, then type in “Keyboard Shortcuts” and press Enter

  2. On the left hand side, click on “Accessibility” to bring up the Accessibility shortcuts on the right side. Find the “Invert Colors” shortcut on the right side, then click on the checkbox to enable it . Afterwards, click twice on the greyed-out key sequence then hit the Control, Shift, ⌘ and G keys together to activate it.

After enabling it, you can easily switch between light and dark mode by hitting:

Control + Shift + ⌘ + G

Note that this will also invert photos and images. If that creeps you out, hit that key sequence again to go back to normal!

For Your iPhone or iPad

You can also enable dark mode on your iPhone! To do so:

  1. Unlock your iPhone, then tap on Settings to open your iPhone’s settings.

  2. Tap on “General,” then on “Accessibility”.

  3. Find the “Invert Colors” option, then tap on the toggle switch to enable it. Afterwards, scroll all the way down to “Accessibility Shortcut,” then tap on it and then on “Invert Colors” to enable the shortcut.

After doing this, you’ll be able to turn on dark mode by triple-clicking your home button!

I hope this helps you as much as it’s helped me!

You’re a better engineer than you think.

I was quite surprised to discover that thousands of people were members of the “Imposter Syndrome” Google+ group within my first month at Google.

I always thought that getting into Google was probably the best social proof of “making it” that an engineer could receive. The interview process is hard, gruelingly technical, relatively unforgiving and riddled with rollercoasters; many incredibly talented Googlers had to go through the process two or more times before getting in for good. (I went through it twice…sort of.) The engineering talent at Google is nearly limitless; many of the world’s most formidable and accomplished computer scientists, sysadmins and software engineers work or worked at Google doing all sorts of things.

So imagine my surprise when literally tons of engineers join a group expressing how they feel as if they aren’t good enough to be at Google or working alongside people with Wikipedia articles written after them. Perhaps it was a big joke that completely went with my head, but given the many, many internal jokes made about not being good enough to be a Googler that I came across (mostly thanks to Memegen), I had my doubts.

I hate checklists.

I can’t help but feel that every other day, I come across a blog post from a programmer or engineer that I’ve never heard of telling me 15 nicely-edited reasons why I’m not worthy of my job. I’ve never used Haskell. I don’t know what git stack does or how to untangle complicated head conflicts from rogue git commit -forces. My .vimrc is really, really plain, and I still don’t know how to write an emacs plugin despite having used it intermittently for the last three years.

Hell, I think if I tell anyone at any conference that I don’t watch Star Trek, don’t play video games and actually love being a Windows engineer (or simply show them my relatively barren Github profile), I’ll be blacklisted by every professional computing community out there.

I can already feel the angry emails coming.

I really hate checklists telling me how to be a “good” engineer. What does “good” mean anyway? Who sets the benchmark? Aside from my manager and peers (who seem to like me, I think?), who’s judging my “goodness?” My gut feeling is that most engineers are much better than they think, and these are my three guiding principles as to why:

Are you learning?

Technology is all about learning new things. If I had to take a guess, I would be scared if anything less than 15 JavaScript frameworks got released last night. What’s last year’s computing messiah usually becomes passé this year (see: virtual machines vs. containers); the state of configuration management is a quintessential example of this.

Are you learning new things? Are you trying new things? If so, then awesome!

Are you challenging yourself?

Finding a groove and sticking with it is a comfortable place to be. However, I believe that sticking with a groove for too long is an easy way to miss things, or, worse, an easy way to think that you don’t need to learn anything new.

In the beginning of my career five years ago, I was really, really good at VBscript. I knew enough to write and maintain behemoth-sized code and where its (many) oddities were. I got so good at it, I thought that learning PowerShell (then Monad) was more of a pain than a benefit. Setting registry keys was (back then) so much more difficult with the StdRegProv provider than with using the Shell object and calling reg.exe.

Had I invested the time in learning Powershell early, I would have probably invested much more time helping build the language or at least collecting cred on Stack Overflow.

If you’re the smartest person in the room, you’re probably in the wrong room.

Are you keeping yourself challenged? Are you working around people who challenge you? Are you taking on increasingly more challenging work? If so, then you’re awesome!

Giving a ****.

Do you care about the quality of your work? Do you document what you’re doing to teach others? If so, then you’re awesome!

Ditch the checklists.

Merriam-Webster defines passion as a a strong feeling of enthusiasm or excitement for something or about doing something. If you’re passionate about what you’re doing and it shows through your work, in my book, even if it pales in comparison to what it should ideally look like, you’re a good engineer in my eyes. So much easier than checklists in my experience.

About Me

Carlos Nunez is a site reliability engineer for Namely, a human capital management and payroll solution made for humans. He loves bikes, brews and all things Windows DevOps and occasionally helps companies plan and execute their technology strategies.

A completely neutral post about containers.

Edit 2: I’ve made a few small changes to the way I’ve described Docker architecture. Thanks, /u/jmtd!

Edit: I mentioned below that Docker containers can only run a single process. While that is true by default, it is actually not the rule. You can use Docker Supervisor to manage multiple processes within a container. Here is how you do that.

Thanks, /u/carlivar!

Fact: there are at least 22 conversations arguing about the usefulness/uselessness of containers happening right now. For every person arguing about them, at least three blog articles will shortly appear telling you why containers are going to be bigger than the Super Bowl or how they are the worst thing to happen to computing since web scale became a thing.

This short post will not be any of those.

I believe in using the right tool for the job and I hate fads. I’ve also been learning about containers for the last few months and am becoming incredibly interested in figuring out how to bring them to Windows 2012 and below (Windows Server 2016+ will have them, but Microsoft will be lucky and probably scared if they saw anything above 20-30% uptake within its first two years), so here’s a short post describing what I’ve learned about them, how they work, and why they might or might not be useful for you and/or your business.

Here’s A Scenario

You’re a developer for an eCommerce website and are responsible for maintaining the order management system (let’s call it an “OMS”). The user-facing shopping cart and checkout processes are tied into the OMS, and these are updated more frequently than the service that handles order transactions. You’d like to be able to quickly test new shopping cart features to respond to market competition more quickly.

One way of doing this is to build virtual machines that mimic your production environment. Depending on the size of your technology team, either you or your sysadmins will set this up for you (if it isn’t already so). Once the machines are ready, either you or your fellow sysadmins will update their underlying operating systems, install and configure the sites dependencies, clone the branch in which you do your development (or somehow get the files across to your sparkling-new staging environment), and, finally, run a few incantations to get everything moving.

Reasons This Could Suck

This is pretty straightforward for some (many?) teams, but much easier said than done for larger ones for a few reasons:

  • Red tape: The bigger the organization is, the more likely there exists lots of complexity (and siloing to match) that makes things like creating new VMs a multi-hour/multi-day effort.
  • Patching is…patching: Patching operating systems can take a long time and introduce unforeseen regressions that might not be part of your spot test(s).
  • Golden images get complicated: Maintaining images gets increasingly complicated fast, especially as your site gains more moving parts.
  • Money: Most “enterprise-” grade virtualization or cloud platforms cost money, and more instances = more compute, networking and storage = more 💰💰.
  • Overhead: Instead of worrying about whether your new shopping cart feature works, you have to worry about whether things like whether /etc/networks is set up right or whether your host has the right Windows license applied to it.

What was originally an “easy” feature test has now become this monolith of an environment fraught with overhead that you’d rather not deal with. This is one of the many reasons why testing isn’t given the due diligence that it often needs.

Meet Containers

A container, also commonly known as a jail, is an isolated copy of an operating system’s user space within which processes can be executed independently from its parent. What this actually means is that you can fork off a “copy” of Linux or Windows…within Linux or Windows. These copies use all of the same hardware as the host they live on, but as far as your network or storage is concerned, they are all independent systems. As a result, containers don’t need their own bank of memory or disk space like a virtual machine would.

With containers, you can do things like:

  • Run a web server, a database and a message queue…without virtual machines,

  • Create restricted login environments for remote users that, to them, looks and feels like the actual system, and

  • Spin up thousands of “instances”…on a regular MacBook.

If you’re thinking “wow; that’s amazing,” you are not wrong.

Containers are not new.

Despite the recent press (and VC-backed funding) that Docker (and, in turn, containers) have been getting, containers are far from a new concept.

The earliest, and most common, form of containers were chroots and/or BSD jails. These first appeared in Version 7 UNIX and 4.2BSD in 1979 and 1982 by way of the chroot syscall and binary and has been used on a daily basis since to restrict the scope of access that users have on UNIX-based systems when they log in. When a user logs into a chroot jail, they will appear to have root access to the entire system (and might even be able to write to anything within the root partition) but, in actuality, will only have access to a volume or directory within the actual system it lives under. This gives the user the ability to do whatever they want on a system without actually destroying anything important within it.

Additionally, hosting providers have been providing virtual private servers since well before AWS was a thing. These VPSes were either virtual machines or chroots depending on the type system requested. Google has also been using (and are still using!) containers internally for executing, maintaining and easily failing over ad-hoc jobs and services since ~2006. They recently made parts of their implementation (called “Borg”) public; check out the research paper if you’re interested. (Googlers take advantage of Borg all of the time for nearly anything and all of their production services run on it. It’s unbelievably reliable and ridiculously high-scale.)

Okay, so why Docker?

So if containers aren’t a new concept, then why is Docker getting all of the attention and being hailed the Second Coming of computing?

What Docker excels in that lxc sort-of doesn’t is in ease-of-use. Docker’s guiding philosophy is that its containers should be used to do a single thing and a single thing only and that containers should be able to run on any host, anywhere. This allows Docker containers to be extremely lightweight and its underlying architecture comparatively simple. This manifests itself in several ways:

  • Every container is composed of a read-only base image that defines its base and additional writable images that overlay it. The base image is kind-of like a classic operating system golden image with the big difference being in it lacking an operating system. Typically, the additional images that overlay it are built using Dockerfiles, but they can also be built via the docker binary.

  • Images are stored in an image registry called the Docker Hub where they can be downloaded from anywhere and versioned to make rollbacks easy. Committing to and pulling from Docker Hub is built into the docker binary. Images can also be stored in private hubs if needed.

  • Docker also includes a RESTful API that makes it very easy to build tooling around.

  • Docker is FREE!

TL;DR: Docker makes it very easy and cheap to build up a containerized environment in very little time.

Why not Docker (or containers)?

While Docker is an amazing tool for getting containers up and running quickly, it doesn’t come without its limitations:

  • Docker containers can only run one process at a time. This means that services that are typically co-located on a traditional host will have to be on separate containers with Docker and the onus for getting them to communicate with each other falls on you.

For running a container with multiple processes at once, a Linux Container (or LXC container) or a VM is probably more appropriate.

  • Since Docker virtualizes the operating system within the operating system (yo dawg), it is theoretically easier for a process running within a container to escape out into its parent operating system. A (modern) virtual machine is much less susceptible to an attack like this since the hypervisor under which it runs runs within a different protection ring inside of the processor, which protects rogue processes within machines from being able to access real hardware on the host.

That said, most security researchers consider Docker to be pretty safe at the moment, so the likelihood of this happening is low enough to mitigate concerns.

  • For the same reason as above, you can’t run different operating systems within containers. This means that Windows containers on a Linux host is out of the question…for now. Additionally, Docker uses Linux-specific features in its core, meaning Windows and OS X hosts need to use a virtual machine and a lightweight service called Boot2Docker to take advantage of it. Windows Server 2016 will bring support for containers to the forefront, but that’s still a few months out.

It should be noted that containers are NOT replacements for virtual machines. Virtual machines are better solutions for heavy, monolithic apps or systems that are difficult to break apart or anything that requires an entire operating system stack. However, people have been finding containers to be pretty good replacements for small apps or services within apps (or enough motivation to break those big, heavy apps down into pieces using a service-oriented architecture).

Our Scenario…With Containers

With that, let’s go back to our expedient web dev that’s eager to test that new shopping cart feature. Taking the VM approach described above might be difficult and time-consuming. If she were to use containers, the process would be a little different:

  • Install Docker (and Boot2Docker if needed) onto their system.
  • Create Dockerfiles describing the containers needed to re-build a local instance of production or create images if needed
  • Define your environment using Docker Compose
  • docker-compose up

That’s mostly it! All of this can be done on localhost, and all of it is (relatively) easy.

About Me

Carlos Nunez is a site reliability engineer for Namely, a human capital management and payroll solution made for humans. He loves bikes, brews and all things Windows DevOps and occasionally helps companies plan and execute their technology strategies.

Concurrency is a terrible name.

I was discussing the power of Goroutines a few days ago with a fellow co-worker. Naturally, the topic of “doing things at the same time in fancy ways” came up. In code, this is usually expressed by the async or await keywords depending on your language of choice. I told him that I really liked how Goroutines abstracts much of the grunt work in sharing state across multiple threads. As nicely as he possibly could, he responded with:

You know nothing! Goroutines don’t fork threads!

This sounded ludicrous to me. I (mistakenly) thought that concurrency == parallelism because doing things “concurrently” usually means doing them at the same time simultaneously, i.e. what is typically described as being run in parallel.
Nobody ever says “I made a grilled cheese sandwich in parallel to waiting for x.” So I argued how concurrency is all about multithreading while he argued that concurrency is all about context switching. This small, but friendly, argument invited a few co-workers surrounding us, and much ado about event pumps were made.

After a few minutes of me being proven deeply wrong, one of our nearby coworkers mentioned this tidbit of knowledge:

Concurrency is a terrible name for this.

I couldn’t agree more, and my small post will talk about why.

In computer science, concurrency is the term used to describe the state in which multiple things are done at the same time within the same “thread” of execution. In contrast, parallelism is used to describe the state in which multiple things are done at the same time across multiple “threads” of execution.
The biggest difference between the two is being able to do multiple units of work simultaneously across multiple processors.

“What about multithreading,” you might ask. “I thought that the whole point of doing things across multiple threads was to do multiple things at once!”

Here’s the thing: today’s processors can only do things one instruction at a time. The massive amount of engineering, silicon and transistors that they have are built to execute one instruction at a time really really really quickly and accurately. What gets executed and when is up to the operating system queueing up work for the processor to do. Operating systems deal with this by giving every process (and their threads) a pre-defined amount of time with the processor called a time slice or quantum.

The processor is even processing instructions when the operating system has nothing for it to do; these instructions are called NOOPs in x86 assembly. (Fun fact: whenever you open up Task Manager or Activity Monitor and see the % of CPU being used, what you’re actually looking at is the ratio of instructions being executed to NOOPs.) Process scheduling is quite the loaded topic that I’m almost certain that I’m not doing justice to; if you’re interested in learning more about it, these slides from an operating systems course from UC Davis describe this really well.

Even though operating systems typically schedule work from processes to be done serially on one processor, the programmer
can tell it to divide the work amongst multiple or all processors on the system. So instead of work from this process being done one instruction at a time, it can be done n instructions at a time, where n is the number of processors installed on a system. What’s more is that since most operating systems typically slam the first processor for everything, processes that take advantage of this can typically get more done faster since they are not competing for as time on the main processor. This approach is called symmetric multiprocessing, or SMP, and Windows has supported it since Windows NT and Linux since 2.4. In other words, this is nothing new.

To make matters more confusing, these days, operating systems will often automatically schedule threads across multiple processors automatically if the application uses multiple threads, so for practicality’s sake, concurrent programming == parallel programming.


Concurrency and parallelism aren’t the same, except when they are. Sort of.

About Me

Carlos Nunez is a site reliability engineer for Namely, a human capital management and payroll solution made for humans. He loves bikes, brews and all things Windows DevOps and occasionally helps companies plan and execute their technology strategies.

Doing something boring? Try this one weird trick! Slackers hate it!

I love writing code and building awesome stuff, but there are times where fighting the urge to Reddit for 14 hours feels like this:

When this happens, I break out my secret weapon: The Pomodoro Technique. The basic premise behind this technique is alternating your time between spending several minutes on nothing but working towards a certain goal (let’s call it the hot period) or deliverable and a few minutes on anything that isn’t work (the cold period). 

While you’re working, you should be doing nothing else except the work unless it’s so critical that it can’t wait. Yes, that includes emails, IMs, and phone calls. This is critical, as this (a) trains you to put a completely unfettered focus into something, and (b) makes getting through that tough period a lot faster.

Your cold period, on the other hand, can be spent however way you want as long as it’s only for a few minutes. The cold period should be much, much shorter than the hot period; otherwise, you’ll run the risk of falling off and potentially wasting a lot of time.

My hot period is 30 minutes and my cold period is 10. For getting through a slump, this setup makes it just tolerable enough to get through the hill and the break just short enough to prevent falling into the deep end.

The official technique recommends a desk-side timer (I’m assuming to train your mind into eventually entering hot/cold periods automatically…or something), but I’ve found that any ol’ timer works just fine. I use my iPhone.

This is tom-foolery. There’s no way that this works.

Except it does! And for three reasons:

  1. It gives you something to look forward to after a few minutes of work, even if it’s short,
  2. It helps break down large and seemingly-unending challenges into smaller, more digestible ones, which makes it easier to see what the goal actually is, and
  3. It makes you feel accomplished, which will make you feel more encouraged to continue doing work so you can keep feeling accomplished.

Still not sure?

Try it for a week. Let me know how it goes!

About Me.

I’m the founder of, an IT engineering firm in Brooklyn that builds smarter and cost-effective IT solutions that help new and growing companies grow fast. Sign up for your free consultation to find out how.

for vs foreach vs “foreach”

Many developers and sysadmins starting out with Powershell will assume that this:

$arr = 1..10
$arr2 = @()
foreach ($num in $arr) { $arr2 += $num + 1 }
write-output $arr2

is the same as this:

$arr = 1..10
$arr2 = @()
for ($i = 0; $i -lt $arr.length; $i++) { $arr2 += $arr[$i] + $i }
write-output $arr2

or this:

$arr = 1..10
$arr2 = @()
$arr | foreach { $arr2 += $_ + 1 }

Just like those Farmers Insurance commercials demonstrate, they are not the same. It’s not as critical of an error as, say, mixing up Write-Output with Write-Host (which I’ll explain in another post), but knowing the difference between the two might help your scripts perform better and give you more flexibility in how you do certain things within them.

You’ll also get some neat street cred. You can never get enough street cred.

for is a keyword. foreach is an alias…until it’s not.

Developers coming from other languages might assume that foreach is native to the interpreter. Unfortunately, this is not the case if it’s used during the pipeline. In that case, foreach is an alias to the ForEach-Object cmdlet, a cmdlet that iterates over a collection passed into the pipeline while keeping an enumerator internally (much like how foreach works in other languages). Every PSCmdlet incurs a small performance penalty relative to interpreter keywords as does reading from the pipeline, so if script performance is critical, you might be better off with a traditional loop invariant.

To see what I mean, consider the amount of time it takes foreach and for to perform 100k loops (in milliseconds):

PS C:> $st = get-date ; 1..100000 | foreach { } ; $et = get-date ; ($et-$st).TotalMilliseconds

PS C:> $st = get-date ; for ($i = 0 ; $i -lt 100000; $i++) {} ; $et = get-date ; ($et-$st).TotalMilliseconds
PS C:> $st = get-date ; foreach ($i in (1..100000)) { } ; $et = get-date ; ($et-$st).TotalMilliseconds

for was almost 10x faster, and the foreach keyword was 2x as fast as for! Words do matter!

foreach (the alias) supports BEGIN, PROCESS, and END

If you look at the help documentation for ForEach-Object, you’ll see that it accepts -Begin, -Process and -End script blocks as anonymous parameters. These parameters give you the ability to run code at the beginning and end of pipeline input, so instead of having to manually check your start condition at the beginning of every iteration, you can run it once and be done with it.

For example, let’s say you wanted to write something to the console at the beginning and end of your loop. With a for statement, you would do it like this:

$maxNumber = 100
for ($i=0; $i -lt $maxNumber; $i++) {
if ($i -eq 0) {
write-host "We're starting!"
elseif ($i -eq $maxNumber-1) {
write-host "We're ending!"
# do stuff here

This will have the interpreter check the value of $i and compare it against $maxNumber twice before doing anything. This isn’t wrong per se but it does make your code a little less readable and is subject to bugs if the value of $i is messed with within the loop somewhere.

Now, compare that to this:

1..100 | foreach `
-Begin { write-host "We're starting now" } `
-Process { # do stuff here } `
-End { write-host "We're ending!" }

Not only is this much cleaner and easier to read (in my opinion), it also removes the risk of the initialization and termination code running prematurely since BEGIN and END always execute at the beginning or end of the pipeline.

Notice how you can’t do this with the foreach keyword:

PS C:\> foreach ($i in 1..10) -Begin {} -Process {echo $_} -End {}
At line:1 char:22
+ foreach ($i in 1..10) -Begin {} -Process {echo $_} -End {}
+ ~
Missing statement body in foreach loop.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : MissingForeachStatement

In this case, foreach has no concept of BEGIN, PROCESS or END; it’s just like the foreach you’re used to using with other languages.

About Me

I’m the founder of, an IT engineering firm in Brooklyn that builds smarter and cost-effective IT solutions that help new and growing companies grow fast. Sign up for your free consultation to find out how.

BYOD Part 1: Computers In The Cloud

Computing is expensive. Desktops and laptops cost lots of money. Printers cost even more money. (Printers are really funny, actually; buying one or two isn’t so bad, but once you’re managing tens or hundreds or more laser printers and printing hundreds or thousands of pages per day, the cost of toner/ink and repair skyrocket like a SpaceX shuttle.) Desks cost even more money. Accessories cost even more money. The list goes on and on,infinitum ad nauseum.

Do you like saving money and hate fixing broken computers? Read on.

Now that we live in an age where downloading high-def movies takes less time than starting up your car, leveraging the cloud and having people bring in their own devices has become a highly lucrative alternative. The bring-your-own-device, or BYOD, movement has picked up a lot of steam over the years, so much so that Gartner expects for “half of the world’s companies” to enact it. Over a billion devices are expected to be using BYOD by 2018, and as more and larger companies begin to take advantage of cloud computing, this trend will only accelerate.

I’ll spend the next three posts talking about three key components of most BYOD environment:

  1. Virtual desktops,
  2. Laptops and desktops, and
  3. Mobile phones and tablets

I’ll explain who the major players involved with each component are, their importance in BYOD and some things to watch out for during considerations.

WIth that, let’s start by talking about computers in the cloud.

Computing. Computing Everywhere.

Bring your own stuff

Most bring-your-own-device setups will need a virtual desktop infrastructure, or VDI for short. Without going too deep into the details (and I’ll scratch the surface of this on this week’s Technical Thursdays post), virtual desktops give you computers in the cloud that can be used from anywhere on nearly anything, even phones and tablets.

A VDI is almost always comprised of:

  1. One or more servers on which these virtual machines will be hosted, which are also known as virtual machine hosts or hypervisors,
  2. Management software to start, stop, create, delete and report on machines, and
  3. Software installed on the virtual machines that make the experience more seamless and accessible.

This means that you’ll need the following to get started:

  1. A subscription to a Cloud service like Amazon EC2 or Microsoft Azure or
  2. Your own server(s) in house for hosting virtual machines (most computers made after 2007 should support this with no issues), and
  3. Enough disk space to host the number of machines you’d like to test (1TB is a good starting point),
  4. A virtual machine hypervisor like VMware ESX, Microsoft HyperV (comes with Windows 2008 R2 and up) or Xen, and
  5. A trial version of Citrix XenDesktop, VMware View, Proxmox (free) or SCVMM.

The Players

There are three major players in this space that offer all of the above with varying amounts of complexity:

  1. Citrix (XenDesktop + NetScaler, a load balancer that works really well with VDI),
  2. VMware (VMWare View), and
  3. Microsoft (HyperV + Systems Center Virtual Machine Manager 2012, usually called SCVMM).

Free and open-source solutions also exist, but they might need more love and attention depending on your situation. We’ll go into that a bit later on in this post.

The Upsides

VDI has a number of advantages aside from being a critical component of going full BYOD:

  1. The desktop is replaceable. Jim’s computer broke again? With VDI, you can get him up and running in minutes instead of hours since the desktop itself is a commodity and nothing of importance gets stored on it.

  2. Decreased hardware costs. Depending on your situation, virtual desktops make it possible to order $300 computers in bulk that can do what $2000+ computers can’t.

  3. Increased data security. Over 30 BILLION DOLLARS of valuable data and IP are lost every year due to stolen laptops and devices. Virtual desktops are configured by default to keep ALL of your data in your datacenter and your profits in your bank accounts.

  4. Your desktop is everywhere. Ever wished your team could move around within minutes instead of days? Ever wished to use cheap Chromebooks to access your desktop at work? Virtual desktops make this (and more) possible.

If you’re interested and like pretty charts, here’s a cost savings white paper published by Citrix and Gartner that go into these advantages in more detail. But we all know that every rose has thorns, and VDI is no exception. In fact, if done improperly, VDI can introduce more problems than it solves.

VDI Is A Pay To Play Sport'll still need to plan.

VDI is kind-of like a new car. If you find the right one for you and take care of it, you’ll likely enjoy it significantly more than getting that used Ferrari you thought was “affordable.” (Hint: they never are.)

Deploying computers in the cloud correctly can range from “free” (but expensive on time and labor) to ridiculously expensive depending on how couplex your infrastructure will be. Here is a list of factors that determine this complexity:

  1. Number of machines Much like their physical counterpart, managing VDIs gets increasingly complicated as you add more machines into the mix. However, unlike physical desktops, replacing broken machines or upgrading slow ones can be done with a few mouse clicks. Some setups even allow users to upgrade their own machines on the spot in seconds!
  2. Network bandwidth Virtual desktops are heavily dependent on the quality of the network on which they operate. The less bandwidth they have available to them, the more tweaking you’ll need to do to make people not hate you for taking away their machines.
  3. Your company’s workload Virtual machines on a host share computing resources with each other. Hosts will usually do everything possible to prevent one machine from hogging up resources from other machines (though this can be overridden), which means that the more intensive your use case is, the less likely VDI will work for you without significant tweaking. That said, virtual desktops work well for a wide set of use cases. (Some of Citrix’s clients use virtual desktops to do CAD and heavy graphics rendering, which most people would normally pass on VDI for.)
  4. Remote workers. Users on laptops that travel a lot will often have unpredictable network conditions. While the frameworks mentioned above handle this situation really nicely, it’s important to take this into account early on in your due diligence.

There are also many little hidden costs that can turn into money pits very easily if not taken into consideration early on in the process, such as:

  1. Will you engage Citrix, VMware or third-party consulting services to help you get started, or will you or one of your engineers go solo? (Here’s a hint: Citrix and VMware will always upsell their consulting services.)

  2. Does your company use or require VPN? (The answer is usually “yes,” but most of the products mentioned above support using desktops over plain Internet.)

  3. How many users will get a virtual desktop? How many of them will actually use it? How will they use it?

  4. You’ll always need more storage than you think you do.

  5. Does your company operate under regulatory requirements?

There’s a very easy way to come upon the answers to these questions, and it’s actually a lot easier than you think.

Just do it…as a test

Building a proof-of-concept VDI is pretty straightforward in most cases. You or your admin can probably set one up in an hour or two. Building this and adding users slowly will guide you towards the answers to these questions and help you understand whether VDI is right for your company or group. More importantly, it is much easier to build VDI automation when your VDI is small than when it’s already a massive behemoth that can’t be shut down at any cost. (Why is this important? Want to roll out 10,000 virtual desktops within minutes or automatically create and remove desktops based on server conditions? You’ll need automation to do this and much more.)

Here’s a tutorial on how to set this up with Citrix XenDesktop. Here’s another tutorial for View.

Have fun!

About Me

I’m the founder of, an IT engineering firm in Brooklyn that builds smarter and cost-effective IT solutions that help new and growing companies grow fast. Sign up for your free consultation to find out how.

Is it actually possible to have an empty inbox? Try this and find out!

I’ve developed a system over the years that has kept my inbox mostly empty all of the time. It has worked for me even when I was getting 100+ emails/day, so I’d say it scales fairly well. It also works well even in the absence of Gmail’s additional feature set (I use Office 365 personally, but this worked when I used Gmail, Apple Mail and my own mail servers back in the day.), which is nice should you ever choose to use a desktop mail client.

This might not work for you. You might even be doing some variation of this
already. If that’s the case, feel free to tell me off!

Finally, if you don’t want to worry about any of this stuff and don’t ever see yourself having to use Outlook or ever again, try Google’s
and tell me that all of this is useless in the comments!

Without further ado, this is how I email:

  • I use folders to categorize my mail. I used to abuse folder structures by having folders for particular events, purchases, conversations, etc, but I’ve found that it hasn’t provided me with a lot of value and was really difficult to re-assemble after email migrations, so I now keep a minimal top-level directory structure instead. The folders that I use most often are:

  • Services/{Added, Removed}: For keeping track of new and deleted accounts I make (and I make a lot)

  • Career/{Accomplishments, Failures}: For keeping track of things I’ve done right and wrong
  • Events: Self-explanatory
  • Responded: Emails I’ve responded to
  • Responded/Sent To Me Directly: Emails sent just to me, see below
  • Purchases: Self-explanatory
  • Receipts: For receipt tracking
  • Team: Important emails from or about my team
  • Personal Messages: Important, yet personal, messages
  • Tasks: I’ll explain this below
  • Timing: I’ll explain this below

This is more useful in the presence of Gmail labels where you can mark
something as being in a particular folder without having to physically move it. It still works well for me without that feature, however.

  • My inbox is my to-do list. This is why I said I keep my inbox “mostly” empty. If a message is in my inbox, it’s either something I need to follow-up on or it isn’t there at all.

  • Follow-ups are flagged (starred). Any email that requires an action from me is starred. Gmail has this neat feature where you can change the color of the star when you star an item. Outlook has this as well with its different types of flags as well as its color-coded categorization system (which is really neat but is a mondo pain in the butt to reconfigure after migrations)

    This feature of my system is really important to me, as it helps me keep track of what my schedule is even in the absence of calendar entries (which I sometimes forget to create). That said, it’s been a personal goal of mine to schedule things in emails as soon as I get them so that I don’t have to worry about forgetting later.

  • I ranked my emails using the “Eisenhower” Decision Matrix. I say “ranked” because this works much better with Gmail labels than a traditional IMAP client. I learned this system in some class about time management back in college (I think) and use it (along with scoring things from 1 to 10) for measuring the priority of things. This has also helped me with managing my email. Here’s how I do it:

    • Rank 0 (Important and Urgent): Needs to be attended to right away and is extremely time-sensitive. You shouldn’t have too many of these in your inbox! If you do, reconsider their importance and urgency.

    • Rank 1 (Important, but not Urgent): Needs to be attended to “soon” but is not time-sensitive. This gives a bit of a nudge to flagged inbox items.

    • Rank 2 (Urgent, but not Important): Doesn’t need to be attended to right away but is time-sensitive. These could be meetings or messages sent
      directly to you

    • Rank 3 (Not Important or Urgent): These messages can (should) be deleted or filed away, see below

  • I mark messages sent to me directly using Gmail labels or automatic color assignment with rules. It is usually the case that messages sent just to me (i.e. messages where my email address is in the To: field, not messages sent to a group) are urgent and need to be responded to quickly. I usually use a bright color that stands out so that I can quickly identify these messages and do something about them.

  • I action every single email right away. Action doesn’t necessary mean
    ‘immediate response’ (though if I can respond immediately, I will;
    “immediately” usually means 160 characters or less). This means that I either flag it for follow-up later, rank it for visibility, move/label it for archival or delete it. This is really important to me. The bigger my inbox gets with crap emails, the harder it gets to clean up, so I’m extremely strict about this.

  • I “delete” most things. I think that this is the hardest thing that keeps people from having clean inboxes (that and not caring enough, since most people don’t really care about this like I do lol). Everyone’s afraid of deleting something and needing it in the future, but out of the 10s or 100s of thousands of emails I’ve deleted over the years, I can count the number of emails I’ve needed to recover with two hands, and even fewer than that were critical messages.

However, Gmail provides way more space for inboxes than people will ever need in their lifetimes, so the smart way of dealing with this is to archive into the “All Mail” bin instead of delete. This way, they’re out of view but still there if they ever need to be recovered. This is the default action in just about every client out there, so you don’t even need to reconfigure anything!

That’s how I email! Here are some great plugins and add-ons that might help take this further:

  • Boomerang for Gmail. Delay sending emails until a certain time. Works really well for actioning on emails right away without having to wait. Get it here
  • Checker Plus for Chrome. Get rich notifications for every email. You can do just about everything I’ve typed above with this extension. It works great! Get it here
  • Multiple Inbox for Gmail. This is a Labs extension in Gmail that allows you to see more than one folder along with your Inbox. It’s really useful, especially if you rank emails. To enable it, go into Settings, then Labs, then check “Multiple Inbox” and Save. After Gmail reloads, you can configure the filters that you want to see in Settings > Multiple Labels. Get it here

I hope this helps! Let me know what you think in the comments below!

About Me

I’m the founder of, an IT engineering firm in Brooklyn that builds smarter and cost-effective IT solutions that help new and growing companies grow fast. Sign up for your free consultation to find out how.

Technical Thursdays: DNS, or why using the Internet is kind of like going to Starbucks

This Thursday, we’ll talk about a system that has been extremely critical (and extremely taken for granted) for shaping the Internet as we know it: the domain name system, or DNS for short.

Before I explain what DNS is, I’ll talk about something I try really hard to hate but ultimately can’t: Starbucks.

I go to Starbucks at least once a day. Given that Google has more coffee machines (and baristas!) sitting idle than my handy downstairs Starbucks does on even their busiest days, this is slightly embarrassing to admit. I love their drinks, but as a recovering coffee snob, I passive-aggressively hate that I love their drinks. My relationship with that Seattle staple is kind of like how a lot of people feel about Taylor Swift: they’ll hate on her forever but will never admit to playing 1989 on repeat.

Wait, that’s just me?

Okay. I can live with that.

Anyway, what I find fascinating about Starbucks aside from their many variants of non-coffee coffee drinks (that are so good but so bad) is how baristas communicate drinks to each other. Somehow, someway, your order for a tall caramel-flavored latte with soy milk, whip cream and a double-shot of espresso is always a tall caramel whip redeye latte to every Starbucks barista on the planet, but trying that on a barista at Cafe Grumpy will usually get you banned for life.

What’s even more fascinating about this is that DNS works “exactly” the same way when you go to on your phone or computer to endlessly browse lists of cat pictures and gifs of people doing funny things.

(Don’t pretend like you don’t.)

You probably know that underneath the the lists and relationship videos, BuzzFeed is really a ton of servers doing lots of hard work to deliver this quality content, and is just one of the servers that shows them to you.

What you might not know is that the name of that server isn’t; it’s actually: That’s it’s IP address.

If you type in those four (or eight) numbers into Chrome (or whatever your browser of choice is; I use Safari for reasons that won’t be discussed here to avoid an intense holy war), it’ll take you right to BuzzFeed.

How does your computer know that these two things go to the same place? The answer is DNS.

What Is This DNS Magic That You Speak Of?

DNS is a system that maps names like or to IP addresses. It was created in the early 1980s when the Internet was much much MUCH smaller and has been iterated and improved upon significantly since then. Here’s the original RFC that describes how it works, and surprisingly, a lot of it has held up over time!

These mappings are stored in records. There are several kinds of them. The name-to-IP mapping that I described earlier is stored in an A record, but a DNS can also have records for other mappings to things like shortcuts to A records (CNAME records), mail servers on the network to which that IP address belongs (MX records) or random data (TXT records).

When your computer attempts to find the IP address for a web site, its DNS client (also called a resolver) performs a DNS query. The response it gets back is the DNS response.

So original, I know.

Dots and zones

The dots in a website URL are very important. Every word behind each dot is called a DNS domain, and every one of those words maps to something.

The last word in the URL, i.e. the .com, .org and .football, is called a top-level domain or TLD. Every single one is maintained by the Internet Assigned Numbers Authority, or the IANA. In the early days of simple Internet, this used to give you an idea of what the website was for. .coms were for commercial use or companies, .orgs were for non-profits and foundations, .net were for personal websites and country-specific TLDs like .us or .it were for government-run websites.

However, like most things from that time period, that’s gone completely out the window (do you think is in Libya?).

Records within a DNS are broken up into zones, and servers within the DNS are responsible for upholding their zone. These zones are usually HUGE text files that get stored completely within that server’s memory for really fast access. When your computer sends a DNS query, the DNS server you’re configured to use will ask for this server if it doesn’t have the record it’s looking for stored anywhere. It does this by asking for a special record called the State-Of-Authority, or SOA, which tells it where to go next in its search.

DNS is so hot right now

Almost every single web site you’ve visited within the last 20 years or so has likely taken advantage of DNS. If you’re like me, that’s probably a lot of websites! Furthermore, many of the assets on those web sites (think: images and code for all of those fancy site effects) are referred to by name and resolved by DNS.

The Internet as we know it would not function without DNS. As of yesterday, the size of the entire Internet was just over 1 BILLION unique web sites (and growing! exponentially!) and used by over 3 BILLION people.

Now imagine all of that traffic being handled by a single Dell server somewhere in this vast sea of Internet.

You can’t? Good. Me neither.


So how does DNS manage to work for all of these people for all of these web sites? When it comes to matters of scale, the answer is usually: throw a metric crap ton of servers at it.

DNS is no exception.

The Root

There are a few layers of servers involved in your typical DNS query. The first and top-most layer starts at the DNS root servers. These servers are ran by the Internic and are used to tell you which servers own what TLDs (see below).

There are 13 root servers throughout the world, {A through M} As you can imagine, they are very, very, very powerful clusters of servers.

The TLD companies

Every TLD is managed by a company. The DNS servers run by these companies contain the records for every website that uses those TLDs. In the case of, for example, the records for will live on a DNS server managed by the IANA, whereas the records for will be managed by Donuts.

Whenever you buy a domain with GoDaddy, (a) you are doing yourself a disservice and need to get on Gandi or Hover right now, and (b) your payment gives you the ability to create records that eventually land up on these servers.

The Public Servers

The next layer of servers in the query are the public DNS servers. These are usually hosted by either your ISP, Google or DNS companies like Dyn or OpenDNS, but there are MANY DNS servers available out there. These are almost always the DNS servers that you use on a daily basis.

While they usually have the same set of records that the root servers have, they’ll refer to the root servers above if they’re missing anything. Also, because they are used more frequently than the root servers above, they are often more susceptible to people doing bad things, so the good DNS servers will implement lots of security enhancements to prevent these things from happening. Finally, the really big DNS services usually have MANY more servers available than the root servers, so your query will always be responded to quickly.

Your Dinky Linksys

The third layer of servers involved in the queries most people make aren’t actually servers at all! Your home router most likely runs a small DNS server to help make responses to queries a lot faster. They don’t store a lot of records, and they are typically written pretty badly, so I often reconfigure these routers for my clients so that use Google or OpenDNS instead.

Your job probably has DNS servers of their own to improve performance and also upkeep internal and private records.

Your iPhone

The final layer of a query ends (well, starts) right at your phone or computer. Your computer’s DNS resolver will often store responses to common queries for a short period of time to avoid having to use DNS servers as often as possible.

While this is often a very good thing, this often causes problems when records change. If you’ve ever tried to go onto a website and were unable to, this is often one reason why. Fortunately, fixing this is as simple as clearing your DNS cache. In Windows, you can do this by clicking Start, then typing cmd /c ipconfig /flushdns into your search bar. Use these instructions to do this on your Mac or these instructions to do this on your iPhone or iPad.

This is starting to get long and I’m in the mood for a caramel frap now, so I’m going to stop while I’m ahead here!

Did you learn something today? Did I miss something? Let me know in the comments!