Category: DevOps Toolbox

There are a lot of good reasons to use a trusted image (aka “foundation image”), including reliability, reduced time to launch, secure configuration. [We’ve discussed them previously]

In this article we are going to describe a development and quality assurance process for making trusted (aka “foundation” or “golden”) images.

First, this is the tool set to be used.

• Source Control Keeps track of all your scripts. Images will be built 100% by scripts, not by manual keying. Therefore source control is essential.
• Virtual Machines You’ll need to periodically start from blank slate, when you make changes to your approach which are incompatible with the current state of your machine. Using virtual machines enables us to rapidly reset to a known state and re-apply scripts.
• Vagrant Vagrant is a virtualization tool that makes it easy to quickly launch and re-launch virtual machines, bootstrap virtual machines with base packages and ssh configuration, and synchronize your files to the guest operating system.
• Continuous Integration (CI) Server For reliability and repeatability, CI servers run scheduled and automated jobs which fetch and merge the latest code, produce output, run tests, and apply tags and other metadata.
• Packer Provides the ability to build many images destined for different cloud environments from a single set of scripts.

Now let’s move into the three steps required for creating a trusted image:

1. Creating and testing the image on your local machine;
2. Building the image, and;
3. Validating and promoting the image.

Step 1: Create and Test Your Trusted Images Locally

Trusted images should not be built by hand, or you’ll quickly lose track of what’s on them! First, use version-controlled scripts to build up a virtual machine that works according to your specifications. When you control the target OS, you can use basic shell scripts. If you are building for multiple target operating systems, configuration management tools like Chef or Puppet will give you the ability to write a single set of platform-independent scripts. Vagrant is a very useful tool which makes virtualization and configuration management technologies work seamlessly together.

Vagrant works by reading a Vagrantfile which you create for your project. It describes the base image (operating system) you want, the software you need to install, and the way you want to access the machine.

You can then use Vagrant to launch local virtual machines right on your laptop/workstation using a virtualization tool like VirtualBox (a cross-platform virtualization tool), or you can work with remote virtual machines such as EC2.  Vagrant syncs a local folder to the VM so you can edit your scripts either locally or right on the VM, and because the scripts are right on the VM you can test them out in realtime as you edit and save. This makes the code/test cycle as rapid as possible, and it’s easy to keep both the scripts and the VM configuration (Vagrantfile) in source control.  Here’s an example of a Vagrantfile which starts with Ubuntu Precise and builds out the image using chef-solo:

Vagrant.configure("2") do |config|
  The base operating system that you’ll start from
  config.vm.box = "precise64"
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"
  If you install software using Git, you’ll want to forward your ssh key
  config.ssh.forward_agent = true
  We are using chef-solo as the “provisioner” to execute the configuration scripts.
  You can also use shell scripts, Puppet, SaltStack, etc.
  config.vm.provision :chef_solo do |chef|
  This configuration is specific to Chef. Other provisioners have their own options.
	chef.roles_path = "roles"
	chef.add_role 'myapp'
  end
end 

Your provisioner scripts (in this case, Chef) is where you will implement features on the VM like:

• Installing and configuring 3rd party software packages
• Downloading and installing your own company’s software, e.g. from Git
• Applying system-wide configuration settings such as iptables, log rotation, syslog, etc
• Installing any custom  SSL certificates that you may want the VM to trust

Once you have scripts that build the virtual machine you want, and you’ve committed your code, it’s time to build an image candidate.

Step 2: Building and Managing Images

To build an image candidate, you’ll start over with a fresh virtual machine, run your scripts, and capture an image from it.

Building an image is about the same as any other software “build” job; continuous integration tools like Jenkins are well-suited to this task. Create a Jenkins job to build your images. As you commit changes to your scripts, push to the source control repository and trigger the job which builds a new  image.  You can tag your images with build job number and other metadata.  This metadata will be very helpful as your image library grows. In particular, you’ll always need to be able to identify exactly which versions of your configuration scripts were used to build a specific image.

Vagrant isn’t quite as good for building images as it is for launching and managing VMs, so for image building we will use a dedicated tool called Packer. Packer is a tool for creating identical machine images for multiple platforms from a single source configuration file.

Packer can take your scripts and produce a variety of different images for platforms such as EC2, GCE, OpenStack, VirtualBox, Docker, Digital Ocean, and more. Packer’s JSON format has pretty much the same information as the Vagrantfile:

 {
  "variables": {
    "aws_account": "",
    "aws_region": "us-east-1",
    "ssh_username": "",
    "source_ami": "",
    "access_key_id": "{{env `AWS_ACCESS_KEY_ID`}}",
	"secret_access_key": "{{env `AWS_SECRET_ACCESS_KEY`}}"
  },
  "builders": [
	{
  	"type": "amazon-ebs",
      "ssh_timeout": "3m",
      "access_key": "{{user `access_key_id`}}",
      "secret_key": "{{user `secret_access_key`}}",
  	"region": "{{user `aws_region`}}",
      "ssh_username": "{{user `ssh_username`}}",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "m3.medium",
      "ami_name": "myapp-{{timestamp}}"
	}
  ],
 
  "provisioners": [
	{
  	"type": "chef-solo",
  	"roles_path": "roles",
      "cookbook_paths": ["vendor/cookbooks"],
      "run_list": [ "role[myapp]", "recipe[myapp::ami-cleanup]" ]
	}
}

Step 3: Image Validation and Promotion

Because your images aren’t exactly the same as the Vagrant environment that you used for developing configuration scripts, you need to perform software testing and quality assurance (QA) on each image candidate. Images that pass their tests will be promoted and released to downstream workflows (development/stage/production).

This should sound like a familiar problem! Again, it’s well-suited to automated continuous integration systems like Jenkins. Like any other build job, your image-building job has a build number, can trigger downstream jobs (for performing QA on the image), and it can run promotion scripts. Use these facilities to validate your images, keep track of which ones have been fully tested, and tag them with important metadata such as the branch/sha of the scripts that built it.

Conclusion

With this “trusted” or “foundation” image you can easily create and push validated images to multiple cloud environments. This can alleviate the wait time of building and launching individual machines and the issues which can arrive with those repeated human components. And isn’t automation what DevOps is all about?

Acknowledgements:  Kevin Gilpin, Hleb Rubanau, and Lucia Capano are coauthors on this post.