devopsy.kinsta.cloud

Tag: ansible

Agile Infrastructure: Because your Infrastructure Deserves Tests, Too
Editor’s Note: This post was updated Aug. 4, 2018, to reflect an updated slide presentation at the end of the article.

Infrastructure as Code is a truly powerful concept whose significance is best described with the words of Adam Jacob, CTO at Chef, in Web Operations: Keeping the Data On Time: “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.” And so it’s not for nothing that treating infrastructure as code is regarded a key principle for reducing cycle time and maximizing flow in Continuous Delivery and DevOps, respectively. Even if your plans are less bold as you read this article, wouldn’t you agree that consequent validation of what you automate on the infrastructure level — long before it goes into production — would be extremely powerful?

So, bear with me for a while and please don’t rush head over heels into automation by simply picking one of the established infrastructure automation tools such as Chef or Puppet, getting everyone trained in Ruby, and blindly start to “automate all the things”. If that’s how you do it, then you are most probably doing it wrong. In You’re Doing DevOps Wrong. Automation in the Enterprise., author Alan Sharp-Paul suggests that, before doing any automation, you first have to understand precisely the requirements and how to validate them to gauge success and maintain quality of your automations over time:
- Are the configuration files in place? Do they contain the right settings?
- Is this port open? Is that port closed?
Why Agile Infrastructure?

Honestly, you and I wouldn’t expect anyone to come up with reliable software merely by choosing an arbitrary programming language and a nifty IDE without having some sort of engineering process in place, right? Likewise, the (automated) provisioning of a solid production environment demands a no less methodical approach. Luckily, Infrastructure as Code enables us to treat the provisioning of infrastructure as well as automating deployments and configuration as an Agile engineering discipline. Although the concepts of Agile Infrastructure are not entirely new, testing against live infrastructure has only recently been facilitated with the consolidation of tools such as Ansible, Chef, Puppet, Docker, Vagrant and Serverspec under the hood of Test Kitchen (see more on that below). Here are some of the key implications of applying Agile software development practices to the infrastructure level:

Alignment of Devs and Ops

Agile software development teams are driven by customer demand and aim to produce outcome in short, regular intervals. Enabling IT Operations and Infrastructure departments to work in small iterations and react to changes quickly allows them to align with and effectively become driven by software development. The outcome of these joint efforts will be both working and deployable software at the end of each iteration.

Fail Fast: Feedback Loops

Executing infrastructure tests as part of your Continuous Delivery build pipeline (before an application is deployed and long-running tests are executed) establishes another important feedback loop that enables you to “fail fast” and get the right people involved. The underlying principle is simple: by expressing your intentions twice — once in the code and once in a test — you’ll know you’ve caught a bug whenever these assumptions don’t match.

Regression Safety

Believe it or not: whether you are using Ansible, Chef, Puppet or any other — you’re running on code! Their respective issue lists are full of unresolved issues of various severities with new issues coming in regularly. Having a sufficient safety net of tests for your infrastructure in place allows you to quickly identify bugs as they creep in (from either side) and protects you against unwanted regressions. After all, a bug in your infrastructure can be very hard to track down and can be much more severe than a bug in your application.

Refactoring

Code refactoring becomes a breeze when you have tests in place that immediately alert you when you broke the desired state of your infrastructure (and all you wanted was to make your code simpler and more maintainable). This is invaluable when you want to recompose your existing infrastructure from 3rd-party components offered on public hubs such as Ansible Galaxy, Chef Supermarket or Puppet Forge, and especially when you are migrating away from your existing automation stack.

The Red, Green, Refactor Cycle of Test-Driven Development

Test-Driven Development (TDD) is a popular software development process that involves writing code in short, repeating development cycles with the goal to enforce good design and enhance confidence. While it is certainly no silver bullet and has been controversially discussed in the past, the Red, Green, Refactor Cycle, which is at the heart of TDD and whose principles I want to quickly summarize here, hopefully gives you some inspiration on how to incrementally build up your infrastructure:
1. Red: Understand the requirements and the state you want to implement. Then think about how to capture this state in a test. Implement the test, then run your entire test suite and see the added test fail — and if it doesn’t, something is really wrong.
2. Green: Implement just enough functionality to make the test pass. Don’t care much about readability, simplicity and design for now — just make it work. Run your tests again and watch all tests pass. If they don’t, either fix your test or your implementation, or take a step back (revert to a point in your code that’s known to work) and start over with a smaller increment.
3. Refactor: Now that your tests pass, improve your implementation and make sure, by running your tests again, that you didn’t break anything. Once you are done refactoring, start the cycle over again.
Keep in mind that each repetition of the cycle refers to a small increment. In terms of infrastructure, such an increment could be the installation of a user, a package or the placement of a configuration setting. When you arrive in a spot where you broke something, you can easily revert to a point in your code that’s known to work. Leveraging such practices really makes the difference between Infrastructure Automation and Infrastructure as Code, as described in Infrastructure as Code – Automation Is Not Enough by Keith Morris.

Test Kitchen, Terraform, AWS and awspec

Now, how to get there? Test Kitchen is an easily extensible test harness that allows you to test your code written in Ansible, Chef, Puppet (and others) on various cloud providers, virtualization providers and operating systems with tests written in a variety of test frameworks.

In the following presentation, I give hands-on examples for you to quickly dive into the topic of using Test Kitchen with awspec as a framework to test Terraform deployments on AWS:
November 13, 2015
DevOps Leadership Series: Gov Does DevOps

This past week, I had the opportunity to catch up with some more industry thought leaders at the DevOpsDays DC event in our nation’s capital. This was the first major DevOps Days event to feature a large audience of government participants. It was an awesome event and is certainly going to be on my must-attend list for next year.

First off for the series, I had the chance to chat with Nathen Harvey, the Community Director at Chef. Nathen also did a great job leading the organizing committee for DevOps Days DC.

In this episode of the DevOps Leadership Series, Nathen illustrates some recurrent topics he noticed at DevOpsDays DC. Nathen ensures us that government is ready for DevOps, enterprises are ready for DevOps, and small businesses and web innovators are ready for DevOps. Then he highlights how we need to create high velocity organizations that are safe, scalable, and humane for our teams.

https://youtu.be/hdUl8-d1za4

Next, I had the pleasure of catching up with Greg Elin, the Executive Director at GovReady and former Chief Data Officer at the FCC. What really stood out to Greg at the event was how many people already have their DevOps “sea legs.” He notes the number of different organizations that are comfortable with DevOps, and are familiar with its language and concepts.

https://youtu.be/XoZ4PGIAZx4

Finally, I caught up with Dave Johnson, Senior Technical Director at Ansible. Dave shares that “the market really is not dominated yet by any particular player in this (DevOps) space.” Dave explains when he first entered the industry he thought the IT world was dominated by open source and commercial technologies. However, when talking to the industry at large now, he found most the enterprises aren’t yet doing anything yet for automation, and most enterprises are still tackling their problems with humans.

https://youtu.be/ky51j1-fNsU

Next up in the series, you’ll hear from John Willis (Co-Author of the upcoming DevOps Cookbook) and Leon Fayer (Vice President at OmniTI).

If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).

June 22, 2015
Ansible Best Practices : Automation, Provisioning and Configuration Management
What is Ansible?

Ansible is an IT automation engine, and a trouble-free model-driven configuration management and command execution framework. Made for multi-tier deployments, it designs your IT infrastructure by describing how all of your systems are interrelated instead of just managing one system at a time. Written in Python, Ansible does not use any agent. It makes uses the SSH server on the target, and no additional custom security infrastructure is required, making it easy to deploy.

Ansible comes up with following uses:

1). Normal configuration management
- Creates system files through the medium of templates
- Manages software installation with the help of yum, apt, gem, or the like
- Manages services or daemons such as start, stop, enable, disable
2). Orchestration tasks
- Removing server from load balancers
- Disabling monitoring or altering
- Perform deployment of your code by git
3). Continuous integration
- Deploy code to QA servers
- Run tests and promote software to production if tests pass
How is it different?

Ansible is a configuration management, deployment and an ad-hoc task execution tool, all in one. It does not require daemons or any other software to start the remote machines’ management. Since it uses the SSH, it easily passes a security audit, and can be used in places resistant to run a root-level daemon with a custom PKI infrastructure.

The Ansible modules can be written not only in Ruby or Python, but in any language efficient enough of returning JSON or key. In this way, Ansible manages to sidestep the popular Python vs Ruby language war, and is of interest to people who like both or neither of them.

Ansible Architecture:

Coming with a simple framework, Ansible connects with your nodes and pushes out small program termed as Ansible Modules, which are considered to be the workhorse of the system and are responsible for performing all heavy liftings. These programs are written to be resource models of the anticipated position of the system. Thereafter, Ansible carries out these modules upon the SSH by default and eliminate them when finished.

Without using servers, daemons and databases, your library related to modules can be inherited in any machine, thus giving the advantage of working with your favorite terminal program, a text editor, and most likely a version control system to keep a track of alteration to your content.

SSH keys your biggest companion

Though passwords are supported, the finest method to practice it is by using the SSH keys along with SSH-agent. The root logins are not required; you can login as any user. Ansible ‘authorised-key’ module helps in controlling which machines can access or which hosts. You can also use other available options like Kerberos or identity management systems.

Management of your inventory in simple text files

Ansible by default uses an INI file to represent the machines being managed. These files help in placing all of your managed machines in groups of your own choosing. In order to add up new machines, you do not require any additional SSL signing server. So by no means, there will be a problem in deciding about why a particular machine didn’t get linked up due to obscure NTP or DNS issues.

Ansible can also be plugged into another source of truth, if available in your infrastructure like drawing inventory, group, and sources suchlike EC2, Rackspace, OpenStack, etc,. As soon as the inventory hosts are listed, variables can be assigned to them in simple text files in a subdirectory called ‘group_vars/’ or ‘host_vars/’ or directly in the inventory file or, as already mentioned, you can use the dynamic inventory to pull inventory from the data sources like EC2, Rackspace, and OpenStack.

Playbooks, a simple and powerful automation language

Playbooks are Ansible’s configuration, deployment and orchestration language. They can explain a course of action you want the remote system to apply, or a set of steps in a general IT process.

At a fundamental stage, playbooks can be used to manage configurations and for deployments of the remote machines. At a more advanced level, they can sequence multi-tier rollouts involving rolling updates, and can delegate actions to other hosts, interacting with monitoring servers and load balancers along the way. Most importantly, the language remains readable, transparent and is developed in a basic text language. You don’t need to declare explicit ordering relationships or write code in a programming language. There are ample of ways to organise playbooks and the files they include.

Widen up Ansible: Modules, Plugins and API

Ansible modules can be written in any language that can return JSON, such as Ruby, Python, Bash, etc,. The inventory can also plug into any data source by writing a programme that speaks to that data source and returns JSON. There are multiple Python APIs for spreading Ansible’s connection types callbacks — SSH is not the only transport possible — and even for adding new server side behaviours.

AnsibleWorks (AWX)

AnsibleWorks, which also sponsors the Ansible community, also produces the AWX. It is a web-based solution to make Ansible simpler, and to provide a trouble-free usage for the IT companies. It is also designed to be complete all kinds of automation tasks.

AWX permits you to regulate the access and even allows sharing of SSH credentials, without someone being able to transfer those credentials. Inventory can either be graphically managed or synced with a wide variety of cloud sources. It keeps log of all your jobs and integrate them with LDAP. It also has an amazing browsable REST API. If required, the command line tools are also available for easy integration with Jenkins.

Conclusion

Ansible’s simplicity and ability to decrease the complexity of other tools has made it a reliable applicant for your environment. Its foremost concern is on security and reliability. It uses OpenSSH for transportation, and the language is designed around auditability by even those who are not familiar with the program. Ansible is suitable enough for managing both small setups with a handful of instances as well as the enterprise environments.
May 19, 2015
Netskope Draws Strength from DevOps In Support Of Scrum Development

Netskope keeps enterprises and their data secure despite the unbridled mix of approved and unapproved mobile apps in the market and on employee devices. Employees use unapproved mobile apps (BYOA) on their BYOD smartphones, tablets, and laptops when connected to networks away from the enterprise. IT professionals are keenly interested in tools that help them to have visibility into these apps and devices and to secure them.

Enterprises face many issues with mobile apps. They confront challenges where data location and access as well as security are concerned when complying with PCI DSS, HIPAA, SOX, GLBA, FISMA, FERPA, and the ISO standards. Netskope works to control app usage, protect enterprise data, and maintain compliance on various endpoints whether the device is on premise or remote / mobile, all in real-time.

“Netskope uses a distributed policy enforcement model that lets your people use these apps securely so you don’t have to block them completely,” says Abhay Kulkarni, VP of Engineering, Netskope.

Netskope’s Security Duties Dictate Its Development Path

“We do not follow Continuous Delivery,” says Kulkarni. Here are two of the many reasons for that.

First, Netskope’s customers demand that software releases are predictable. “Our customers are large enterprises that want predictable change windows so they can ensure that they make the necessary changes on their side or that they prepare for our updates,” says Kulkarni.

Second, because Netskope is a SOC-1 Type-II and SOC-2 Type-II certified company, it must comply with the requirements that comprise these certifications. Tightly-knit change management is critical to SOC certification. “As we do each change, we need to make sure that all changes meet the security, availability, integrity, and confidentiality criteria of these certifications by testing the entire product end-to-end,” says Kulkarni.

But Netskope follows other DevOps principles such as using its own adaptation of a well-accepted source control workflow model as well as an automation model, a model for bootstrapping machines, and also models for software installation and change management.

Netskope performs a balancing act between complying with SOC-1 and -2 trust services criteria and principles and maintaining rapid feature development. On the one hand, they manage to limit what individual developers can do on production machines—many times disallowing access. On the other hand, they use Sumo Logic to enable their coders to peruse anonymized logs in order to gauge how the service they instantiated is working.

Goals, Tools, & Challenges

“The key goal for Netskope’s DevOps approach is to ensure developer responsibility for running, monitoring, and scaling their services,” says Kulkarni. Netskope’s developers and operations staff work together from design through development while complying with a common framework, which helps in resolving development issues and brings consistency to data center operations, according to Kulkarni.

Netskope holds developers individually accountable for making their software ready for automated deployment. To help with that, Netskope uses familiar tools such as Ansible, which helps Netskope to automate everything in production. “Right from bootstrapping a machine to changing network configurations, we automate all commonly repeated tasks,” says Kulkarni.

Netskope deploys Sumo Logic agents using Ansible as a part of bootstrapping so that machines are configured from the get go to send logs to Sumo Logic. “Each service has standard logging methods that allow Netskope to create performance dashboards and to track errors,” says Kulkarni. Netskope uses a number of other tools that are commonplace in DevOps, such as Nagios, a monitoring tool and various external monitoring tools for tracking end user performance.

The biggest challenge for Netskope was not in tooling but in building a DevOps culture where developers lead the charge in responsibility for quality, error-free operation of their services. “In a non-SaaS world, developers ship a version of their product and celebrate. However, shipping a SaaS product is just the beginning. There is really no celebration since you are now responsible for the upkeep of the services,” explains Kulkarni.

A Trip Through Netskope’s Development Process

Netskope starts with a Scrum sprint planning meeting where in they decide what features to address for the given sprint. “We try to be realistic about our expectations in delivering code in a sprint by applying quality criteria—i.e. we expect code to be of a certain quality or else we cannot deploy it,” says Kulkarni.

Then developers bundle various modules on a single branch, which they test for functionality, security, performance, and integration. As the modules and the branch increase in stability, the developers upgrade them from one stack to the next until everything is deemed fit for a production deployment, explains Kulkarni.

Results

“Often we see proactive notifications from developers about issues they see in production. Detecting problems before our customers see them is a key benefit of the DevOps model,” says Kulkarni.

May 4, 2015
Ansible Tower in the Software Development Lifecycle
Ansible Tower is used in a variety of different ways, from traditional configuration management, to custom application deployment, to the orchestration of zero-downtime rolling updates. Companies like Gawker Media use Ansible to deploy their sites more than 10 times an hour. NASA uses Ansible to update security vulnerabilities and to patch manage nasa.gov weekly. Enterprises that make money delivering applications via the web find that Ansible Tower excels at removing IT bottlenecks, automating repetitive tasks, and accelerating the delivery of applications to market.

For IT operations, rolling updates with zero downtime is a very common orchestration pattern that has been discussed at length, but let’s take a look at how Ansible Tower fits earlier into the software development lifecycle and removes bottlenecks in operations, as well as in in development and test.

Ansible Tower offers a number of benefits in the SDLC, specifically to:
- Drive consistency between environments
- Enable self-service
- Improve automated testing
Development

A typical day in the life of an application developer may be to: check out code, make changes, test locally, check in changes.

During the course of this work, the development environment may become infected with small configuration tweaks and hand-crafted software stacks that differ dramatically from the environment that production code is ultimately deployed to.

In fact, a common problem in many organizations is that the development environment may not exactly (or even closely) match what’s in QA, Stage, and Release. Generally, the further away from production you get, the less and less the environments look the same. (Not just the obvious examples like software library versions, but more subtle items like configuration settings, authentication details, etc.).

Inconsistency between code and the software stack that supports the application is a common cause of software errors and resulting production downtime. With Ansible, a playbook captures the desired state of the stack and even the entire environment. Ansible Tower provides a push-button interface that templates how a playbook should be run — deploying consistently and with repeatability.

With Ansible Tower, a playbook is run from a job template, which specifies the parameters and environment details of how to run the playbook, with all of the parameters and options you would normally pass via the Ansible command line. There is no concern that the user will mistype an option or parameter to Ansible, potentially deploying to the wrong environment or with the wrong playbook arguments since the template is triggered via the push of a button (or API.)

Not only do these templates drive consistency between deployments — each time the button is pressed the same result is expected — but they also encourage a tremendous amount of playbook reuse: Multiple job templates can be created that launch the same identical playbook with different parameters. This enables the same playbook to be used between environments or stages of release. The job template can use the same playbook, but target different inventories (corresponding to different stages of release or different environments), pass different options, even use different credentials.

This drives consistency between releases, even across different stages, by ensuring that the same steps to automate provisioning and deployment are followed between each stage.

Screenshot of how to use different job templates with the same playbook, passing different parameters from the user to deploy into different environments

When combined with the the role-based access control capabilities of Ansible Tower, playbooks can be deployed at the push of a button, but in a controlled and easily-audited way. This allows an organization to spread the power of Ansible to any of its users. Even users unfamiliar with Ansible can run playbooks that others have written.

These features are commonly used to enable self-service provisioning of development environments by the developers themselves.

Typically, developers manually construct a development environment that starts out only loosely coupled to the production software stack and deviates further and further over time. Alternatively, developers are forced to request a new development environment from operations teams, usually incurring delays that proliferate throughout the software development lifecycle and lead to late deployments, missed deadlines, and lost business.

Tower enables developers to check out and provision their own development environment, built from scratch and 100% consistent with the software stack used in test, stage, and production — because it is built with the same playbook. Developers log into Ansible Tower and, under RBAC, see only the job templates that they have permission to access and deploy.

Developers do not require any knowledge of Ansible or the playbook. They can provide new parameters and options, specifying perhaps the release tag of the application to be deployed or fine tuning the deployment to fit within the constraints of the environment. The playbook handles provisioning the new servers (e.g. by calling out to VMware or Amazon Web Services), deploying and configuring the software stack, and the actual application under development. The developer presses a few buttons in Ansible Tower, goes and gets a cup of coffee, and then returns to a newly 100% provisioned environment. Tear down and re-provisioning is a snap, so a new environment for pristine development can be deployed at any time.

QA

A typical day for QA might look like this: Program a CI tool to automatically build the codebase periodically, deploy to a test environment, run tests, report results.

At most shops, some of this work is generally automated, usually building the latest code and the deploying the latest code artifacts into test. Deployment of the application stack may not be automated (or easily refreshed between test cycles), leading to the same consistency problem as in development,or worse, because test environments and artifacts are potentially reused between test cycles. Here again, the reuse of playbooks between environments through the Ansible Tower job template will promote consistency between stages of release.

Additionally, the same Ansible Tower self-service capability that can be provided to developers can also be provided to testers. QA may write their own playbooks for automated testing and Ansible Tower can be used to delegate responsibility to QA staff who are not Ansible experts or aren’t comfortable running Ansible on the command line.

In addition to self-service, it is very common to use Ansible Tower in conjunction with a Continuous Integration system in QA to automate not only application deployment in test, but provisioning of the application stack to refresh the test environment between tests.This way, Ansible Tower provides a complete RESTful API that powers the Ansible Tower user interface and makes it possible to programmatically invoke Ansible Tower to do any function that can be performed via point and click in the UI. In this way, a CI tool like Jenkins can be connected to Tower to launch job templates for automated testing. Jenkins is configured to call the Ansible Tower API to launch a specific job template that invokes an Ansible playbook to provision a new QA environment, deploy and configure the appropriate software stack, the correct version of the application, and the test suite. Automated testing can be performed and the environment decommissioned, all within one or more playbook runs.

Just as in development, this ensures that the test environment is consistent with all release stages — that what is being tested is the same as what was developed and what is being deployed into production. Test environments can quickly be provisioned and deployed 100% from scratch, improving your ability to test and to speed up the application to production delivery.

When used in conjunction with with a CI tool like Jenkins, the combination is more powerful than with a CI tool alone: Jenkins can kick off Ansible playbooks to provision environments and deploy the latest code as part of any Jenkins CI job, but under the framework of role-based access control, inventory management, and aggregated job output that Ansible Tower provides.

As you adopt Ansible Tower into Operations, consider also the benefits of Ansible and Ansible Tower throughout your software development lifecycle. Apply automation to eliminate bottlenecks through every stage of release and accelerate the delivery of your applications to market.

Let me know how you are using Ansible Tower in your efforts – we want your input!

For more information, check out these resources:
- http://www.ansible.com/continuous-delivery
- http://docs.ansible.com/guide_rolling_upgrade.html
About the Author/ Dave Johnson

Dave Johnson, Senior Technical Director, Ansible. Dave started his career at Red Hat prior to its IPO, ultimately building and leading the sales engineering organization that penetrated the US Oil & Gas, Electronic Design Automation, and Managed Service Provider markets.Prior to joining AnsibleWorks, Dave held executive positions at rPath and Bronto Software, leading engineering and services organizations that spearheaded the transformation of innovative custom solutions into repeatable Products.Dave holds a B.S. in Electrical Engineering and a B.S. in Computer Engineering from North Carolina State University. Contact info: @thisdavejohnson, Linkedin
April 27, 2015
DevOps with Purpose: It’s about your tools!
This is Part Three in a four-part series. In Part One, I focused on the critical first step of defining DevOps with a purpose by thinking about DevOps in the context of your organization’s applications. In Part Two, I provided four tips to fostering a DevOps culture in your organization.

By now you’ve hopefully noticed the emphasis on “your” in this series, because, at the end of the day adopting DevOps is about your business, your applications, and your culture. In this third part of the series, I’m going to discuss your tools.

In IT, we’re indundated with tools. Developers have their favorite tools and sys admins do too, so do the project office, the service support group, and the QA team. There are IT tools purchased by our organization years ago that we are using and others we aren’t, tools we’ve recently started using, and others we are considering at any given point in time. There are also general-purpose tools the company wants us to use for things like document sharing, instant messaging, and so on. It’s got to the point where a lot of IT organizations say, “we have two of everything like Noah’s ark”, yet they still want more tools.

Part of the reason is that, as IT practitioners, we like gadgets and tools. We like the fact that they can help us do things and can amplify our own abilities, but we also know they can fragment information and can blindside us when someone uses a tool to do something that others weren’t aware of.

Do we really need more tools to do DevOps?

DevOps has a close association with the Agile Software Development movement, and one of the core tenets of the Agile Manifesto is to value “Individuals and Interactions over processes and tools”. Nevertheless, it’s hard to find a DevOps talk or article that doesn’t discuss the need for tools like git, docker, jenkins, puppet, chef, and so on. The reason for this is actually pretty straight-forward: continuous integration and continuous delivery are best done through automation so we can follow a repeatable method and roll things back quickly if there are issues.

At a more basic level, however, we need to acknowledge that there are really two types of tools:
- Tools that help us amplify our abilities as individuals (e.g. I can drive a nail into a board with a hammer much more effectively than with my hand or a rock because of the strength of the hammer’s material and the principle of leverage)
- Tools that help us coordinate work across many humans, thereby, amplifying our individual abilities (e.g. I can build a house much faster and with better quality by leveraging different experts’ skills and using a shared set of blueprints for building construction, plumbing, electrical, heating/cooling, and so on)
The same distinction holds true for software development, deployment, and operations. One individual can theoretically gather requirements, build the software, test the software, deploy the software, and support the software while simultaneously project manage their personal time. Most computer science students have done this in fact at one time or another, but when we are talking about enterprise applications that support core business functions it clearly doesn’t make sense because of the amount of effort required at each step and the specialized skills necessary for a high level of competency in each area.

What this means is that we absolutely need tools that amplify our abilities, or that of our individual teams, i.e. the hammers; but we also need to invest in tools that help us coordinate work across teams, i.e the shared sets of blueprints.

Tools That Amplify Individual / Team Abilities

A lot of the tools attention in the DevOps community has been focused on: source code management systems like git or bitbucket; requirements planning tools like jira or rally; build tools like Jenkins; automation tools like puppet, chef, and ansible; and project management tools like MS project or AtTask. These tools help us amplify work in each of these respective areas just like hammers, saws, drills, and screwdrivers perform specific functions when building a house and often follow a natural sequence (first you hammer boards in place to create a wall, then hammer on the sheet rock, then saw or drill holes in the sheetrock, screw in electrical outlets and fixtures, and so on.)

Just like building a house has a natural sequence, so does the sequence of tools from code to deployment and it’s often referred to as the DevOps “tool chain”. This brief article isn’t intended to cover each area of the typical DevOps tool chain (I could have also added bug tracking, automated testing, and other categories too), the point is you need to take time to define what the DevOps tool chain should be for your organization and to do so intentionally with a purpose, taking into account the tool investments your organization already has, the needs and skillsets of your organization, and your own practical budget realities.

Do you have experience in using and self-supporting open source tools or do you have a preference for commercially provided tools?, Are you comfortable having multiple requirements tools in different teams or business units, or do you want a single standard? Only you can answer these questions for your organization. There is no prescriptive formula for these types of DevOps tools, although I’ve mentioned a number of commonly used ones.

Tools that Amplify Work Across Teams

The second area you should focus on are tools to help coordinate work and amplify our abilities across teams, i.e. bridging the inherent gaps in the DevOps tool chain, a subject that has received less attention in the DevOps community to-date. This makes sense because it’s human nature to think first about our own function and team. However, this silo thinking is one of the core problems DevOps was developed to address; the long-time focus on tools for specific IT “work stations” has actually entrenched the cultural silos of development, QA, operations, and project office in IT. For example, most IT organizations have two totally separate systems for tracking issues – in operations, it’s the service desk application and for code issues in development it’s the bug tracking application – and there is little or no relationship between them. In fact, you are lucky if an incident in the service desk can be tied to a bug tracking number.

If we are adopting DevOps in order to optimize the flow of new software releases to support business agility, then we need to look at how we will coordinate work and avoid information distortion and loss when different teams are using different, disconnected tools. In addition to your tool strategy for optimizing individual DevOps workstations, therefore, you also need to have a strategy for the tools that are going to help you effectively manage flow across those workstations.

Most DevOps transformation efforts start small with a core team of perhaps a dozen or so members, and their often located in one physical location. In this scenario, you might be able to make do with regular face-to-face meetings and general-purpose communication tools like email, instant messaging, and a SharePoint site or Wiki with MS Office documents. The scope of your cross-team collaboration tools aligns with the scope of your DevOps effort and you might be able to rely on people interactions to unify the tool chain.

But as you scale your DevOps efforts to larger, distributed teams across time zones and multiple business units and applications, your cross-team tools approach should scale as well. The point-to-point nature of instant messaging, ignored reply-all emails, and SharePoint ‘document dumps’ aren’t effective in coordinating the efforts of dozens or hundreds of developers, testers, admins, and project managers. Information gets lost, information gets overlooked, and the gaps in your tool chain expand. A feature misses a commit and doesn’t get into the build and isn’t deployed. An operational requirement is missed so the test environment isn’t configured properly and the release gets pushed. Three weeks are spent solving a performance issue through code when it could have been addressed faster via hardware. A legacy application dependency in production is overlooked and the new mobile app doesn’t work in production as it did in test.

There is an emerging class of purpose-built IT collaboration tools, such as ITinvolve, that enable the creation of cross-team workspaces where information is proactively shared as IT teams and tools get work done. This helps large, distributed cross-functional teams collaborate more effectively with each other in-context to raise questions, provide answers, and incorporate what’s happening up and down the DevOps tool chain in their individual work.

For example, the developer will have better visibility into when the next build is going to occur and can ensure the feature is committed in time or can reach out to the build engineer in the workspace to request a delay in order to ensure the feature makes it in. The operations engineer can have earlier and better visibility into operational requirements so he can ensure the test environment is configured properly and avoid unnecessary delays. The legacy application dependency in production can be reproduced in test to ensure the application works properly once moved to production. And so on. By eliminating the handoff gaps and information loss across the DevOps tool chain, you can reduce risk of communication issues, solve problems more holistically rather than individually, and better achieve the goal of improving business agility.

In the final part of this series, I will bring together the themes of the first three articles to demonstrate how you can chart a DevOps plan based on your Applications, your Culture, and your Tools to A.C.T. with greater agility.

About the Author

Matthew Selheimer is Chief Technical Evangelist and SVP of Marketing at Houston-based ITinvolve. Reach him at matthew@itinvolve.com
October 28, 2014
Using Docker & Ansible by John Minnihan @Gluecon 2014

Continuing our series of videos from the DevOps track at Gluecon this past year, our next video is “Why Docker + Ansible Make Chef and Puppet 100% Unnecessary by John Minnihan, ModernRepo” Again many thanks to CloudSoft for recording and making these videos available to us to post. Also the picture above is from the Ansible website originally.

This presentation sounds pretty controversial if you just look at the title. Actually John is not advocating throwing away your Puppet or Chef scripts. If you already have Chef or Puppet set up and you like the way it is working, there is no reason to change. However, John believes that Docker with Ansible can negate the need to use either of these tools going forward. John is primarily talking about a greenfield situation here I believe. Anyway, the video like the previous ones in this series are about 30 minutes. Enjoy!

June 4, 2014
Config Management & F***ing Shell Scripts

Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

People either missed this, or chose to ignore it:

“This is simply brilliant! I’ve dreamt about such tools.”
– artellectual

“I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
– sdegutis

It provided a perfect platform for many to voice their frustrations with current tools:

“Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
– mattjaynes

“In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
– s_kilk

“We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
– sdegutis

“Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
– sp1982

In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?

March 24, 2014