Continuing on from Part One, where I discuss the far-ranging benefits of continuous deployment, today I'll cover off another large part of the 'what do I do as a sysadmin' question: that being, config management.

Config management doesn't just mean capturing or templating config files for services. Put more broadly, the art of config management allows you to enforce a known state about your system, such as what is running, how it should run, and what it takes to make it run. Proper config management results in a suite of information that describes what your infrastructure does or how it should work.

A tool, a ball of wool

I touched on a pattern in Part One, in that many areas of systems administration start out sounding like a simple task or tool to solve one problem. Continuous Deployment might mean automating making changes to a website. Config management might mean making sure the file /etc/sudoers is identical across a cluster of servers. But as we saw in Part One, those small changes or additions as part of achieving Continuous Deployment might also mean speeding up the application development process, making a business more profitable - or even making someone's job less risky and therefore more enjoyable (and therefore making a human being more healthy). In the same way, config management is a ball of wool that inevitably leads to other benefits.

Config management can pave the way for automating the provisioning process should you build more systems, scale out existing systems, or are forced to rebuild (which means it also can aid the Disaster Recovery process).

It can enforce consistency across development, staging and production environments. Very significantly, it can enhance the quality of making changes to systems, because config management is code, and can be version controlled, peer reviewed, tested through syntax checkers and simulations, or staged out, before being applied in critical production environments. Less mistakes happen - therefore money is saved.

It can speed up the on-boarding process when a new developer joins a project (I make developers' Vagrant instances pull the same configuration from a Puppet master that the stage/prod servers do). It is common sense that you should not develop applications in PHP 5.5 if you are deploying to a production PHP 5.4 system. Developers love to roll their eyes about this, but this will make sense to business owners who aren't infatuated with the idea of a workstation somehow representing a personality: your customer should not be paying money for you to troubleshoot version-inconsistent problems that shouldn't have happened in the first place. You wouldn't pay a mechanic to try and reproduce a problem you've reported with your car's transmission, in a lawnmower. It is a question of professionalism as much as anything else. If you don't understand this, you don't deserve to be asking for money.

But Config Management takes time and costs money!

In 2008, I was working for a development agency as the 'ops guy' interfacing with the ops team of the upstream hosting department, for a major international charity site. We were in the process of scoping out the staging and production environment. A Hosting Ops guy was very keen to take advantage of this cool new tool called Puppet. The budget impact of writing Puppet manifests was deemed too much, and the idea was canned. We all remember the pain of having to scale out the infrastructure during the peak period of that charity, while *hundreds* of thousands of dollars were being lost by the hour as the site couldn't withstand traffic. Or spending hours trying to troubleshoot some 'random' problem that turned out to be a typo in a file on just one server in a cluster, thanks to the manual effort.

Suffice to say, Puppet was used on all such subsequent projects. Now I can't imagine not using it. And in 2015, many many public Puppet modules have been released for reuse by others, thus reducing some of that cost factor. The tech community tries to help itself.

To summarise

There's not much else to say here - this is just part of the answer to the open-ended question 'what exactly do you do, sysadmin?'. I spend a lot of time writing Puppet modules even for basic stuff - often even reverse-engineering what's already been done 'on the fly' back into code, in case it has to be reused later. To the uninitiated - and sometimes to those who are paying for it - it seems like a lot of overkill. To those of us who've lived through any of the above experiences, we know it pays for itself.

How has config management helped your business or job?

Coming up...

In Part Three I'll talk about security in systems administration. This will include everything from audits, hardening, intrusion detection, to helping businesses achieve PCI-DSS compliance and ISO27001 certification.

After that:

Part Four: monitoring,
Part Five: troubleshooting or 'ghosts in the machine',
Part Six: high availability and disasters (sometimes the same thing :) ),
Part Seven: communication.