— ROELBOB
— ROELBOB
Editor’s Note: This post was updated Aug. 4, 2018, to reflect an updated slide presentation at the end of the article.
Infrastructure as Code is a truly powerful concept whose significance is best described with the words of Adam Jacob, CTO at Chef, in Web Operations: Keeping the Data On Time: “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.” And so it’s not for nothing that treating infrastructure as code is regarded a key principle for reducing cycle time and maximizing flow in Continuous Delivery and DevOps, respectively. Even if your plans are less bold as you read this article, wouldn’t you agree that consequent validation of what you automate on the infrastructure level — long before it goes into production — would be extremely powerful?
So, bear with me for a while and please don’t rush head over heels into automation by simply picking one of the established infrastructure automation tools such as Chef or Puppet, getting everyone trained in Ruby, and blindly start to “automate all the things”. If that’s how you do it, then you are most probably doing it wrong. In You’re Doing DevOps Wrong. Automation in the Enterprise., author Alan Sharp-Paul suggests that, before doing any automation, you first have to understand precisely the requirements and how to validate them to gauge success and maintain quality of your automations over time:
Honestly, you and I wouldn’t expect anyone to come up with reliable software merely by choosing an arbitrary programming language and a nifty IDE without having some sort of engineering process in place, right? Likewise, the (automated) provisioning of a solid production environment demands a no less methodical approach. Luckily, Infrastructure as Code enables us to treat the provisioning of infrastructure as well as automating deployments and configuration as an Agile engineering discipline. Although the concepts of Agile Infrastructure are not entirely new, testing against live infrastructure has only recently been facilitated with the consolidation of tools such as Ansible, Chef, Puppet, Docker, Vagrant and Serverspec under the hood of Test Kitchen (see more on that below). Here are some of the key implications of applying Agile software development practices to the infrastructure level:
Agile software development teams are driven by customer demand and aim to produce outcome in short, regular intervals. Enabling IT Operations and Infrastructure departments to work in small iterations and react to changes quickly allows them to align with and effectively become driven by software development. The outcome of these joint efforts will be both working and deployable software at the end of each iteration.
Executing infrastructure tests as part of your Continuous Delivery build pipeline (before an application is deployed and long-running tests are executed) establishes another important feedback loop that enables you to “fail fast” and get the right people involved. The underlying principle is simple: by expressing your intentions twice — once in the code and once in a test — you’ll know you’ve caught a bug whenever these assumptions don’t match.
Believe it or not: whether you are using Ansible, Chef, Puppet or any other — you’re running on code! Their respective issue lists are full of unresolved issues of various severities with new issues coming in regularly. Having a sufficient safety net of tests for your infrastructure in place allows you to quickly identify bugs as they creep in (from either side) and protects you against unwanted regressions. After all, a bug in your infrastructure can be very hard to track down and can be much more severe than a bug in your application.
Code refactoring becomes a breeze when you have tests in place that immediately alert you when you broke the desired state of your infrastructure (and all you wanted was to make your code simpler and more maintainable). This is invaluable when you want to recompose your existing infrastructure from 3rd-party components offered on public hubs such as Ansible Galaxy, Chef Supermarket or Puppet Forge, and especially when you are migrating away from your existing automation stack.
Test-Driven Development (TDD) is a popular software development process that involves writing code in short, repeating development cycles with the goal to enforce good design and enhance confidence. While it is certainly no silver bullet and has been controversially discussed in the past, the Red, Green, Refactor Cycle, which is at the heart of TDD and whose principles I want to quickly summarize here, hopefully gives you some inspiration on how to incrementally build up your infrastructure:
Keep in mind that each repetition of the cycle refers to a small increment. In terms of infrastructure, such an increment could be the installation of a user, a package or the placement of a configuration setting. When you arrive in a spot where you broke something, you can easily revert to a point in your code that’s known to work. Leveraging such practices really makes the difference between Infrastructure Automation and Infrastructure as Code, as described in Infrastructure as Code – Automation Is Not Enough by Keith Morris.
Now, how to get there? Test Kitchen is an easily extensible test harness that allows you to test your code written in Ansible, Chef, Puppet (and others) on various cloud providers, virtualization providers and operating systems with tests written in a variety of test frameworks.
In the following presentation, I give hands-on examples for you to quickly dive into the topic of using Test Kitchen with awspec as a framework to test Terraform deployments on AWS:
Infrastructure is code, and like any code, you need to version it. Notice how that works: If it’s infrastructure, then it’s a fundamental part of the system; nothing’s going to work without it, which makes it valuable. So valuable that in full-stack deployments, it is part of the application. And if it’s code, then it should be under version control.
While recently thumbing through some old comic books, I got a laugh from all the quirky advertisements. Fantastic promotions of awesome gadgets – like x-ray specs and radio wrist watches. Each designed to extract a gullible kid’s hard earned allowance. Of course they were all too good to be true and probably left many kids disappointed — and maybe like me, just a tad cynical.
I spied one ad that actually got me thinking about DevOps. It’s the famous “kick sand in your face” strip, which to those unfamiliar offered “98 pound weaklings” a sure fire program to build muscles, beat up bullies, and make lots of new friends down at the beach.
In IT we also have weakling apps we wish could be stronger. Services so brittle and fragile that any proposed change, update or enhancement is seen as risky – perhaps even career threatening.
So is it possible to beef up these veritable data center weaklings? Can we also find a way to turn the tables and make our apps stronger?
Well yes we can, but only with some new thinking. But definitely not (if like kids reading comics), we get sucked in by outrageous DevOps claims, along with a raft of gadgets and tools that never quite live up to our expectations.
Here are four things to consider as you look to toughen up your apps:
Looking at these examples it’s obvious the new IT Ops craftsman won’t linger behind the production curtain. On the contrary, they’ll be the ones working across the entire software pipeline; focused squarely on acquiring new skills and building solutions to drive better business outcomes. Those who blissfully remain in denial, preferring instead to watch monitoring consoles and build out physical infrastructure have a lot to lose.
While the old 98 pound weakling comic book advertisement is kind of hokey, many businesses are getting sand kicked in their faces by digital disruptors. So before it’s too late, start re-engineering the operational practices needed to toughen up your applications. Just remember that while rapidly developed apps now deliver your business muscle, they’ll atrophy just as quickly if you neglect quality and performance.
One of my favorite presentations at the recent DevOps Enterprise Summit 2015 was Jeffrey Snover of distinguished engineer at Microsoft talking about his 16 year quest to bring what is today called Windows Nano Server to market. Interestingly when I spoke with Gene Kim about what he thought were some of the best presentations of DOES he also brought up Snover’s presentation as well.
Windows Server without Windows was something once considered heretical from anyone at Microsoft. As Jeffrey relayed to me in the interview below and on stage, he paid a price for evangelizing for this. But today we have a version of Windows Server that has only 1/20th of the footprint of regular Windows Server. It runs on mainframes, command lines and has virtually (no pun intended) all of the functionality that you need for a server in the cloud without the GUI overhead that Windows usually brings.
As importantly for me, Jeffrey also gave us a glimpse of what it is like to work in Satya Nadella’s Microsoft. It sounds like a very different place than when Steve Ballmer was CEO. Is also sounds very refreshing as well.
Here is my interview with Snover:
:
Here is Jeffrey’s actual presentation at DOES ’15
— ROELBOB
Today’s DevOps teams need in-depth monitoring that offers insights into every aspect of the application-delivery chain. When you decide for an application performance monitoring (APM) solution, you can choose between an on-premises approach or SaaS-based solutions (“Monitoring as a Service”). Every deployment option brings its pros and cons or has advantages for special use cases. An installation on-premises for example works best if you want to monitor an intranet that you want to keep isolated.
Which APM deployment option makes most sense for your organization? There‘s no simple answer to this question. A lot depends on your environment and requirements. Let’s explore the points to consider in deciding between SaaS and on-premises monitoring:
With in-house, on-premises systems, all servers and data are physically under your control. When you use SaaS-based monitoring, performance metrics, reliability data, and detected issues are stored in the cloud. If your industry has strong regulatory and policy requirements, SaaS may not be an option for your organization—you may need to keep all such monitoring data on-premises.
To deploy on-premises solutions you need to provide your own hardware and storage in a private datacenter or cloud. You install and configure the monitoring software, databases, and other components of the APM solution. In general, SaaS offerings can be installed and set up within minutes using a self-service system. Some SaaS APM tools even auto-detect your entire server landscape within minutes following installation.
SaaS solutions are typically updated automatically on a frequent basis by the vendors who provide them to deliver support for new technologies and feature enhancements. On-premises installations require that you manage updates yourself and stay current with the release of new product updates. Updates for on-premises solutions typically come out less frequently, which means you sometimes have to wait longer for support of new technologies and bug fixes.
Maintaining your own servers for an on-premises APM solution requires that you manage the servers, data storage, and data backups yourself. SaaS solutions are maintained and backed up by their vendors.
On-premises software is usually licensed for longer periods of time. This means that the initial investment is often higher, but costs are recouped over time. SaaS-based APM often features “pay as you go” pricing, so you only pay for what you use. This model can be cheaper, but it depends heavily on your usage and your tool vendor’s pricing model. If you run extra servers during periods of peak demand, it makes a big difference if your billing is based on the maximum number of servers you use rather than your actual usage. Because they’re built on cloud infrastructure and often don’t require long-term contracts, SaaS make it much easier for you to opt out of the system and cancel any time. As you aren’t required to purchase and maintain your own hardware, they’re also typically more flexible and easier to scale.
When your monitoring solution runs on-premises it’s often harder to get vendor support because the system may not be reachable from outside your organization. With SaaS-based solutions you can grant access to your tool vendor’s support staff and work with them on resolving issues. This makes the SaaS approach faster and easier if you run into special cases that require vendor support.
With on-premises solutions you must connect to your internal network via VPN to view monitoring data on mobile devices (unless your organization exposes this service to the Internet). Conversely, you can access monitoring data supplied from a SaaS-based solution anywhere you have an Internet connection.
With an on-premises APM solution, you can expect in-depth customizations that suit the exact requirements of your organization. Customization of SaaS solutions on the other hand is limited, and dictated by rules defined by your tool vendor.
If you need to monitor applications that run in a public or hybrid cloud scenario, it’s likely that you’ll also want to run your monitoring in the cloud. SaaS-based solutions enable you to seamlessly monitor cloud-based environments because this is what they’re designed to do. You only need to install agent software on the involved servers.
On-premises solutions are vulnerable to critical problems like power outages and broken network connections, potentially leaving you with no way of receiving performance and availability problem alerts. On the other hand, SaaS monitoring in the cloud runs in a different geographical location and so remains available to send you alerts whenever problems are detected.
When your APM monitoring solution runs in your own data center, your IT department can determine appropriate service availability levels. SaaS monitoring providers often use software architectures that can be updated without downtime and reach service availability levels of up to 100%. Well-designed SaaS solutions use redundancy to further improve availability. It’s recommended that you take a close look at any SaaS-based solution vendor’s claim of providing “always on” availability.
SaaS tool vendors ensure that needed resources like disk space and CPU will scale up with your demand. This means you won’t need to budget for additional hardware purchases and typically leads to lower total cost of ownership (TCO) with cloud-based SaaS solutions.
Deploying monitoring software in house (on-premises) gives you full control over your data. “Monitoring as a Service” in the cloud brings new possibilities for many monitoring use cases. Depending on your requirements, it could even make sense to use both. Some tool vendors are now providing monitoring solutions that combine the best of both worlds.
With the points covered in this article, you should now know everything you need to consider when deciding which monitoring approach makes most sense for your organization.
There are all sorts of compelling business benefits awaiting those who adopt Continuous Delivery. It has the potential to reduce operating costs, reduce defects and redundant work, improve efficiency, and result in a much faster delivery speed. By releasing applications to market faster, you boost your chances of success and gain a competitive advantage. But how do you get from the theory to the practice?
If you really want to scale up and realize the benefits of CD, then you’re going to need money to retool, retrain, and change the existing culture. In order to secure the budget you need, all of the key players in your organization have to buy into the change. A strong business case is essential. Here are six key areas to look at when you build yours.
Production incidents
A natural place to start with building a business case for CD is with production incidents, because they’re very visible. When you perform a root cause analysis on an incident, you’ll often find that the software worked fine in the development environment. By standardizing the environment configurations you can reduce the number of incidents, free up resources from the provisioning process, and reduce wasted time.
It’s possible to calculate the cost of data center downtime for your company, and you can use root cause analysis to find out where provisioning errors were to blame. From that, you can extrapolate the potential savings made by reducing these outages with standardized environment configuration. You also have to consider the reduction in wasted time and effort, resources that can now be productively employed.
Deployment failures
Take a look at your past deployment failures and work out the average cost to the business of rolling back the deployment, diagnosing the cause of failure, and remediating. By automating deployment you can cut these costs completely.
Manual testing
To employ an API-based testing model that can be integrated into your delivery pipeline will pose an upfront cost, but that cost per build decreases with each new build. With manual testing, the cost rises with each new build. Automated testing gives you better coverage, earlier. If you can find defects earlier they tend to be quicker and easier to fix, which reduces the potential knock-on impact in terms of production incidents, and, ultimately, reduces costs. You’re also cutting manual labor costs and overhead.
Manual build process
If someone on your team is writing scripts to automate your builds, there’s a labor cost involved and they may cause a potential bottleneck. If you can switch to a centralized delivery pipeline where the build is automated and which development teams can customize for their needs, you can reduce the time and effort required. You also free up the developer who would have written the scripts to work on improving the product instead. At the same time, you’ll boost oversight, control, and reliability in the build process.
Better analytics
A lot of development time is wasted on building features that don’t add value. According to research by Forrester, only around a third of the ideas implemented result in a measurable improvement in the customer experience, a third have no obvious effect, and the final third actually have a negative impact. By building analytics into your code, and constructing a strong feedback loop, you can find out what really improves the customer experience and focus your efforts there. The earlier you can get feedback, the easier it is to prioritize and maximize the value you’re adding.
Focus on the big picture
Most business cases might focus on some of the benefits we mentioned above, and that’s because they’re easier to measure. But it’s the intangibles, such as improved competitiveness, goodwill, and revenue, which are harder to quantify, but which have the greatest impact.
You want to present the potential cost savings, but this is about more than cost cutting. The long term benefits filter through as you ramp Continuous Delivery up. Faster feedback loops and a greater understanding of your customer are laudable aims for any business. You’ll have to delve into specifics to generate the persuasive numbers you need, but it’s important that you keep those intangible benefits front and center as part of the business case you’re constructing.
About the Author/Andrew Phillips
Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
The Inaugural DevOps Days Ohio is being held on November 18th and 19th at Bluestone in Columbus Ohio (http://www.devopsdays.org/events/2015-ohio/ ). From fledgling startups to established industry, the Midwest is home to a large, vibrant technology community. Ohio in particular has been a flurry of activity in the past several years, drawing attention from around the globe.
There are over 300 people expected at the event to discuss culture and technology needed to improve the ways that organizations operate and deliver products to their customers. The keynote speakers include:
Courtney Kissler from Nordstrom, Vice President of E-Commerce and Store Technologies, who will be speaking on “Transforming to a Culture of Continuous Improvement” on Thursday morning.
Carmen DeArdo, Technology Director at Nationwide Insurance will be discussing the Columbus based company’s journey over the last several years which started with the application of Agile practices and has expanded to focus on improving their end to end delivery capability.
One common theme of both Courtney’s and Carmen’s talks will be on the application of lean practices as a key enabler of reducing delivery lead times.
Other keynote speakers include:
David Mortman, former Chief Security Architect at Dell
Malilka Iyer from Pivotal,
Fred Hatfull from Yelp,
Dustin Collins from Conjour
Kelly Looney and JP O’Donnell from CoverMyMeds.
Courtney and Carmen recently spoke at Gene Kim’s DevOps Enterprise Summits. You can see their interviews with Alan here (https://www.youtube.com/playlist?list=PLGnigCNRJjp67gfI72P1Y9O8NqIv13PLW)