If there’s any consistent criticism of the larger DevOps movement, it’s that the talking heads and industry gadflies are driving the conversation, not the people in the trenches.
All the thought leadership panels and “how-to” content (including the written and AV materials on this very site) make for great watercooler discussion, but lack the input of those dev, ops and IT management pros who are actually living the DevOps experience, some critics have said.
Well, fair enough. Talking heads are paid to talk. Thought leaders are chosen to lead the debate at conferences and on webcasts. Bloggers blog. Reporters report. This is simply the way of the world. Certain voices are heard via the channels they frequent. It is, as so many people are fond of saying these days, what it is.
So what about everyone else? What of the people reading this stunning prose and listening to these talks while rolling their eyes based on the connectivity of it all to their real-world experience? How can their voices be heard?
Independent of jumping into the blogsphere or applying to speak at said industry events – which you should, btw – there’s one significant opportunity available to anyone who cares to take the time – participating in industry research.
Over the next few business days, citizens of the DevOps world will have their last chance to contribute to the industry’s most widely read and respected research project, the State of DevOps Report.
If you haven’t done so already, take the time to fill out this year’s survey, and make sure that your voice is heard when the subsequent report is published in June. Sure, it takes a little time, about 15-20 minutes; but that’s because it goes to the trouble of exploring the real-world issues that weigh most heavily on today’s practitioners.
Why is this particular project so important, or even credible? Consider that the minds behind the survey and analysis resulting in the 2016 State of DevOps Report are those authors and talking heads that actually get the most respect.
Backed by DevOps Research and Assessments (DORA), the independent think tank operated by a partnership of Jez Humble, Gene Kim, and Nicole Forsgren – perhaps the most respected voices on the landscape, along with Nigel Kersten, the project is categorically aimed at surfacing the truth about DevOps adoption today – not merely promoting the interests of vendor sponsors.
Like previous iterations of the report, the 2016 edition will focus on topics including IT performance, technical practices, culture, lean management, and organizational performance. This year’s survey also examines several new areas, including the ROI of IT performance, security and compliance, and containers.
To further entice participation, project leaders are also offering free tickets to the DevOps Enterprise Summit, along with forthcoming books from some of the industry’s leading advocates to a lucky few who take the time to get involved.
DevOps success is fundamentally based on the notion of increased collaboration and continuous feedback among the people scattered across the SDLC. Likewise, the legitimacy of this research project is utterly dependent on the input of as many real-world practitioners as possible.
If you’re one of those individuals, and you really care whether or not everyone else involved in the advancement of DevOps understands what this all means to you, do yourself a favor and participate.
Much like an application being built using DevOps practices, this quality of this project will be directly determined by the degree to which anyone with insight is willing to get involved and build toward optimal results.
PORTLAND, OR–(Marketwired – Apr 7, 2016) – Software underpins every part of modern life and commerce. Organizations rely on it to drive growth in all aspects of their business. According to market estimates, a $200 billion software infrastructure market is emerging as a result of mass migration to the cloud; pressure on IT to meet the demands of billions of consumers and business users; and the growing requirement to continually deploy and manage software to meet business goals.
To help organizations compete effectively and thrive in a software-defined world, Puppet today announced its vision to give every organization the shortest path to better software. The company’s commitment to helping enterprises take full advantage of the movement towards constantly modern software is fulfilled by its industry-standard Puppet Enterprise platform. More than 30,000 companies around the world, including more than two-thirds of the Fortune 100, rely on Puppet Enterprise to continually deploy and manage software across their IT infrastructure, whether it’s physical, virtual, or in the cloud.
“Software powers everything around us, from the devices on our wrists and our walls to the work we do, the fun we have, and everything in between. Modern cars are powered by millions of lines of code, our financial world is entirely mediated by software to enable speed and throughput, and it’s critical to delivery of core functions like medicine, utilities, and food. Nevertheless, most businesses take weeks, months, and even years to deliver everything from simple upgrades to the latest innovations, and too much of this software is out of date, insecure, and thus a barrier to progress rather than an enabler of it,” said Luke Kanies, Puppet founder and CEO. “With Puppet, customers build highly efficient DevOps practices that automate manual processes and get control of the entire infrastructure and application stack, eliminating constraints on innovation, securing critical capabilities, and maximizing the business potential from their software investments.”
Opening the Gates to the Shortest Path to Better Software As part of the company’s strategic plan to make the shortest path to better software a reality for enterprises, Puppet also announced today new leadership, product updates, integrations, resources, and branding. All these elements support the company’s mission to help every organization significantly reduce the time it takes to provide both their internal and external customers with the most effective and modern software to achieve business goals.
Company name and brand refresh – The company is changing its name from “Puppet Labs” to simply “Puppet” to reflect the fact that the tight-knit Puppet community — tens of thousands of individual open source users and Puppet Enterprise customers — all refer to the company, the technology, and the community itself as “Puppet.” The new name, and a new brand identity to go with it, also reflect Puppet’s mission: helping every organization get on the shortest path to better software in service of its own customers and mission. For complete details, visit the new website at puppet.com.
First president and COO – Sanjay Mirchandani is the first executive to hold this position at Puppet. The former senior vice president of VMware will be responsible for Puppet’s go-to-market execution, including marketing, pre-sales engineering, sales, customer support, and services. For complete details, read our press release< (https://goo.gl/tKtnDO).
Project Blueshift and Puppet Enterprise 2016.1 – Blueshift represents Puppet’s engagement with leading-edge technologies and their communities — technologies like Docker, Mesos, and Kubernetes — and Puppet’s commitment to giving organizations the tools to build and operate constantly modern software. The new Puppet Enterprise 2016.1 gives customers direct control of — and real-time visibility into — the changes they need to push out, whether to an app running in a Kubernetes cluster or a fleet of VMs running in AWS. For complete details, (https://goo.gl/RTMS2v).
Atlassian HipChat integration – This new integration makes it possible for DevOps teams to direct change with the Puppet Orchestrator, see change as it occurs, then discuss and collaborate on changes in process — all right in HipChat. For complete details, read our press release (https://goo.gl/Xk2CrX).
Splunk integration – Proactive monitoring of infrastructure and applications is a key DevOps practice, enabling continuous improvement. The Puppet Enterprise App for Splunk now extends the Splunk platform to Puppet customers to diagnose issues and solve problems faster, so they can deploy critical changes with confidence. For complete details, read our press release (https://goo.gl/Fi9BFd).
About Puppet Puppet is driving the movement to a world of unconstrained software change. Its revolutionary platform is the industry standard for automating the delivery and operation of the software that powers everything around us. More than 30,000 companies — including more than two thirds of the Fortune 100 — use Puppet’s open source and commercial solutions to adopt DevOps practices, achieve situational awareness and drive software change with confidence. Based in Portland, Oregon, Puppet is a privately held company with more than 400 employees around the world. Learn more at puppet.com.
Over the course of the last several years we have seen the DevOps movement continue to mature and evolve. It has moved beyond the early adopter, visionary stage into the mainstream of the market. Therefore, it should be no surprise that some of the leading solutions in the DevOps space should likewise mature and evolve as well. Yesterday Puppet (formerly Puppet Labs) made several announcements as part of a broader message that they are ready, willing and able to continue to be a leader in this space for years to come.
There are several parts of the Puppet announcement yesterday, lets have a look at each of them:
Growth indicators – Somewhat mandatory, Puppet announced that they continue to enjoy rapid growth.
Not a lot to add here, the numbers speak for themselves.
Puppet Labs is now just Puppet. In my conversation with Nathan Rawlins, SVP of marketing, he said that this was a case of recognizing what the market is already doing. I don’t know how many people you have heard refer to the company as Puppet Labs. Most everyone I know says Puppet. So why push rope up hill? Henceforth the company will be known simply as Puppet.
But there is more than just the name here. Rawlins told me that along with the new name, they have also gone through a complete rebranding. New logo, new website look, new collateral look. The new brand seeks to give the impression of simplicity, stability and maturity.
New leadership – The company has hired a software/tech industry veteran leader, Sanjay Mirchandani as President, COO. Mirchandani has held senior leadership roles at VMware, EMC and Microsoft. His responsibilities will include go-to-market operations, including sales, marketing, customer success, and global expansion. From what Rawlins told me, Sanjay will be leading much of the day to day operations of the company.
This is not a case though of a professional leader coming in to replace the founder/CEO. Puppet’s Luke Kanies remains in the CEO role. Mirchandani will free up Kanies to do what he is best at. Be a visionary and chart the course for Puppet and in turn help shape the DevOps and software delivery market as well. It takes a confident founder to bring in a real professional to help run the business to allow you to do what you do best. Many are too insecure to share management with a newcomer who “just wouldn’t love their baby the way they do.” Kudos to Luke for recognizing this move for the long term good of the company.
Project Blueshift– Many people are focusing on Blueshift’s capabilities of using Puppet with container technologies like Docker, Kubernetes, Meseosphere, CoreOS, etc. While Blueshift does focus on this as of the release today, there is actually more to it than that. Blueshift is really a collaboration between the Puppet community and the company itself to ensure that Puppet will always be able to help companies adopt next-generation infrastructure technologies. While today those next-gen technologies are container related, the idea is to grow Blueshift into something capable of working with new technologies as they emerge. It is a longer range project beyond 2-3 years, instead looking at what could be on the horizon 5-10 years from now. It is an ambitious vision. It is always hard to know what is the next big thing. But the fact that Puppet is even attempting to keep its eye on this is commendable.
Upgrades to Puppet Enterprise and Puppet Orchestrator – Mixed in all of the news is that Puppet has released upgrades to its Enterprise product and the Orchestrator product that was released last year. Without going into a feature by feature recap (you can read the link to the PR if you like), Puppet continues to polish their enterprise products as part of their maturation.
Integrations with Splunk and HipChat – As DevOps matures, the market will want greater integration between their tools. Puppet has taken a nice step in that direction by releasing a Splunk App that will allow you to automate set up of Splunk servers and give greater visibility into enterprise DevOps environments.
The HipChat integration allows your Puppet Orchestration management to show up right in the HipChat GUI and allows users to manage it from within HipChat. A nice bonus is the access rights which make sure that those without permission do not have access to the Puppet Orchestration data.
All in all, this is a major step up by Puppet. To be fair, the folks at Chef have also been taking steps to further mature their product and organization. As DevOps continues to grow, I am sure we will see some of the other DevOps tools startups follow the leaders here in maturing and growing along with the DevOps market.
It seems IT practitioners haven’t realized yet the scale of digital disruption. Mired in the old IT thinking, they firmly believe that sophisticated tools are all businesses need to survive the ongoing digital disruption waves. Most of them warn, “Digital transformation is changing the way we do business, the way we deliver IT services must change, businesses need to adapt the way they work.” The problem is, there’s nothing new in what they offer to help businesses work the “right” way.
Indeed, the same old IT thinking’s practices are still applied with the illusion that deployment automation and configuration management tools on their own will make businesses agile, responsive to market opportunities, and therefore in line with the digital economy’s competitive challenges. That belief misses the point for two reasons: First, it’s an incorrect understanding of what business agility is, and second, it’s a misunderstanding of the global competitive environment’s transformation.
Turning your IT organization lean and agile is the competitive differentiator your business lines desperately have been calling for. Authentic DevOps implementation is the way to go. That’s the price to pay to keep up with the pace imposed by new leading-edge companies such as Google, Apple, Uber and the like.
The changing competitive environment forces businesses to focus on innovation, market responsiveness and timely revenue
A digital economy threatens the existence of all businesses. Yes, you read it right, their existence is at stake. And that fact is underestimated by IT vendors and consulting firms.
As illustrated, we’re shifting from business environments where established businesses (incumbents) control a large percentage of market segments to competitive contexts where, attracted by the amazing digital opportunities and the adoption of cloud which significantly reduces infrastructure expenses, a growing number of startups disrupt entire industries:
The proliferation of startups created by highly qualified unemployed seniors and creative entrepreneurs, coupled with the propagation of highly innovative digital services (e.g., Internet of Things [IoT] applications) substituting traditional services, have been disrupting the vast majority of industries.
Several examples substantiate that fact: the taxi industry shaken by the emergence of technology-driven companies such as Uber, the banking industry facing new breed of competitors including PayPal, Google and Apple; and the healthcare industry increasingly disrupted by companies such as GlowCap.
To grow and prosper in the digital era, your scale, brand equity or market capitalization aren’t enough anymore. To succeed, your business lines must be on top of the digital economy’s competitive standards: competitive innovation (the ability to continuously improve customer experience), market responsiveness (the ability to deliver added-value services to the right market segments at the right moment) and timely revenue (the ability to continuously increase customer value).
The digital economy has taken IT to an inflection point—deploying sophisticated tools isn’t enough; being lean and agile is the way to go
The obligation for business lines to focus on innovation, market responsiveness and timely revenue is unprecedented for techies; it forces them to think beyond tools and address issues as unusual as organizational and operational effectiveness.
Indeed, innovation, market responsiveness and timely revenue assume greater cross-functional collaboration, IT focus on business priorities, organizational flexibility and accelerated time to market. Make no mistake, you’ll never achieve them by only deploying tools, but by also making your IT operating model flexible through proactively eliminating the organizational and operational dysfunctions that prevent cross-functional collaboration, fast problem-solving and decision-making.
That’s what is meant by making the business agile.
The new IT thinking is summarized in that simple arithmetic: Digital Competitiveness = Lean and Agile Operating Model + Effective and Efficient Tools. Without the lean and agile operating model, innovation and timely revenue are wishful thinking. Without effective and efficient tools, innovation and market responsiveness are wishful thinking as well.
DevOps is the IT asset businesses need to succeed in the digital economy
Contrary to a now (unfortunately) established belief, DevOps isn’t a technology; Chef, Puppet, Jenkins, OpsWorks, alike aren’t DevOps, they’re only pieces of it. It’s important to be clear about it.
As I’ve been implementing it, DevOps is a system of principles, practices, roles and responsibilities, governance structure and tools purposely arranged to establish cross-functional collaboration, enable accelerated applications delivery and, in turn, make the business responsive to market opportunities:
As illustrated, DevOps as a business capability is made up of three components: the Rules of the Game, the IT Operating Model and the Continuous Delivery Pipeline (CDP). Let’s explore them.
DevOps Rules of the Game create the conditions for lean and agile IT environments
Rules of the Game are a fundamental dimension of DevOps: They enable the establishment of lean and agile environments. I refer to them as the DevOps non-technological piece.
In practical terms, they’re a set of principles, practices, roles and responsibilities and tools institutionalized to ensure accelerated applications delivery.
They serve as catalysts to specific organizational and operational effective practices. Let’s take the principle, “Facilitate with system thinking (involvement of all relevant stakeholders)”—it forces application development stakeholders to adopt a collaborative mindset, which in turn breaks down organizational silos to eliminate the impediments to operational flexibility and institutionalize cross-functional collaboration.
They’re the poor cousin of today’s DevOps practices, which ignore them in favor of more technical and technology issues. That’s a terrible mistake that reduces DevOps influence vs. the digital economy’s challenges.
The agile IT Operating Model establishes the lean and agile work environment
The IT Operating Model reflects the interactions that take place across your application development value stream. In practical terms it represents the work environment derived from the application of DevOps’ rules of the game.
I always remind my clients that DevOps’ rules of the game aren’t theories, they’re the foundations of a tangible strategic capability: the lean and agile IT environment. They demand a thorough transformation of your application development work space to put together business, application development, and operations:
The lean and agile IT work space for the digital era illustrated (Credit: O.C. Tanner)
The picture illustrates what’s meant by lean and agile IT environment, it substantiates Adam Jacob’s point that “Fundamentally DevOps is about taking the behaviors and beliefs that draw us together as people, combining them with a deep understanding of our customers’ needs, and using that knowledge to ship better products to our customers.”
The IT operating model is the poor cousin of today’s DevOps practices which ignore it in favor of more technical and technology issues. Once again, it’s a terrible mistake that reduces DevOps influence vs. the digital economy’s challenges.
Make no mistake, technology matters, the CDP helps to meet digital technologies challenges
The importance of the rules of the game and the resulting IT operating model don’t suggest at all that technology doesn’t matter. Make no mistake about it, the CDP is essential; I refer to it as the DevOps technological piece. Through deployment automation and configuration management tools, it significantly accelerates application development, testing, resources provisioning and deployment.
However, one thing to be aware of is, digital economy raises technological challenges that transcend speedy delivery issues. They include:
The importance of IoT
The pivotal role of predictive analytics
The increasing importance of APIs
As a matter of fact, IoT devices generate massive data and making use of it to generate revenue requires big data technologies to integrate, automate and analyze them.
On the other hand, this ocean of IoT data has generated an API business; the idea is, to monetize their data and generate revenue, businesses must provide their customers and partners with APIs that easily give access to their systems.
Implementing the CDP as part of PaaS cloud services can help to prosper in the digital economy; it allows businesses to benefit the cheap and on-demand computing, storage and network resources provided by the cloud.
Takeaway
In hindsight, with a business-oriented perspective, the ITaaS delivery model and DevOps follow the same goal: Provide business lines with the lean and agile IT environments they need to be flexible enough and responsive to meet the competitive challenges of digital economy:
Like DevOps, the ITaaS delivery model (…) takes advantage of an agile IT operating model to help businesses succeed in the digital era. It implements consumer-provider relationships across the IT value stream to ensure organizational flexibility and operational agility.
CIOs must realize that what matters to the business is to continuously generate revenue through improving customer experience and increasing customer value. Deploying applications at the speed of light for the sake of speed doesn’t make sense from a business perspective; as long as expected business benefits aren’t achieved, speed is a useless gadget.
Implement DevOps as what it is: a three-dimension IT capability including rules of the game, IT operating model and, of course, continuous delivery pipeline.
I recently reading a great Q&A with the Etsy CTO John Allspaw over on The New Stack, which contained some nuggets of good advice that increasingly are important as we continue the journey toward DevOps adoption. Of particular note were statements on the introduction of “new” things to solve problems. In a nutshell, Allspaw reinforced the importance of standardization as a means to reduce architectural debt.
If you aren’t familiar with the concept of architectural debt, it’s similar to technical debt, in which choices made today incur longer term investments of time and money when future changes are introduced. If you settle on a caching solution such as Redis, for example, there are technical and architectural implications. You can’t simply switch out Redis for memcached later without significant changes to both code and the infrastructure.
Even your choice of load balancer today implies architectural debt because of the growing inclusion of scalability as part of the application infrastructure itself. The result is integration with automation and orchestration systems that are not easily changed later on if you decide you want a different solution. The style of service registry you choose is an architectural decision that impacts clients, servers and systems across the microservices architecture. Switching from client-side to server-side will later be costly.
That’s architectural debt.
Allspaw noted the need to not reinvent the wheel when possible, and instead encourage the collaborative aspects of DevOps as a means to avoid reinventing the wheel and potentially increasing the architectural debt load on the organization.
This is not a new concept. There’s a reason enterprises in particular proudly proclaim themselves “Java shops” or “Microsoft shops” and primarily require development in one or perhaps two languages, but no more. Maintainability is a component of technical debt, just as middleware decisions are a component of architectural debt.
So as we continue to see wider DevOps adoption, it’s important that we carefully consider the impact of our choices in terms of the automation, orchestration and integration required to implement continuous delivery and deployment. Once you’ve invested time and energy (and probably budget, too) in Puppet or Chef, VMware or Cisco, OpenStack or SaltStack, you’re going to find an unavoidable amount of technical and architectural debt in development as well as production across infrastructure and networking teams.
Each of these systems, solutions and frameworks comes with its own components. It isn’t just a language, it’s a comprehensive system with management and repositories and agents and reporting components. All of these impose certain architectural requirements on organizations, from needing certain software running to prescriptive details on networking that must be enforced, lest the system break down.
We’re already starting to see that fragmentation in enterprises that are engaged in automation and orchestration efforts. When asked the number of automation frameworks (Cisco, VMware, Python, OpenStack, Chef, Puppet, etc…) in use, one-third today reported using three or more.
That’s not necessarily “bad,” as it could be indicative of standardization across silos (networking wants to use Cisco, infrastructure is already on VMware, and Python is being used to cobble it together by ops), but it also could be an early sign of the kind of fragmentation that causes exponential growth in technical and architectural debt.
Certainly there is room for experimentation in these early days, but it behooves us to maintain a watchful eye on the use of frameworks and systems and languages we employ for automation and orchestration to ensure we don’t end up buried in technical and architectural debt.
Standardization, even loosely enforced, can help minimize the impact of that debt before it spirals out of control.
The stock markets may be bearish, the price of oil is down the drain, but money continues to pour into the DevOps tools and solutions market. Just in the last few days 4 companies have announced significant new funding. Whether this is a last gasp before the doors slam shut, the tech industry thumbing their nose at the rest of the business world or proof that the recent market downtrend is a red herring I don’t know. But as someone who has lived through more than a few bubble bursts in the tech world it bears watching.
For now though, the good times continue to roll! Lets review the new funding news:
JFrog raises an astounding $50m! That amount of money is nothing to sneeze at! I spoke with Shlomi Ben Haim, CEO at JFrog. They have over 120 people working there in offices in Israel, HQ in Santa Clara and France. Their JFrog Artifactory and JFrog Bintray are hosting software for organizations the world over, big and small. Ben Haim and his team, as well as his blue chip investors believe the time is now as DevOps goes mainstream. They want to make sure they have the money to capitalize on the opportunity. Congrats to the JFrog team and we are excited to see what this money translates to for them.
Puppet Labs raises $22m in debt financing. While Puppet continues its phenomenal growth it took the opportunity to grab $22m in debt financing from Silicon Valley Bank. I spoke to Puppet CEO Luke Kanies about the raise and he told me that he is not even sure the company will use this money, as they still have a sizable portion of their last raise in the bank and are getting close to profitability. But at today’s low interest rates who can blame him for taking advantage of available cash? There was a time when debt financing was frowned upon. Why take on debt when you could give some equity for cash. But in today’s low interest rate/ high valuations world, why give valuable equity when you can get money for near free? In addition to the money, Puppet is also expanding their offices by about 37,000 square feet and have appointed financial veteran, Lou Lavigne to their board. This continues their preparation towards and IPO down the road. Of course with the market where it is, they may be better off staying private.
Datical raises $8m Series B round. Databases as part of the DevOps movement are going to have their day and Datical wants to be the company to make it happen. To do that they have raised a Series B led by S3 Ventures. They money will be used to continue development of their agile Database automation tools, as well as sales and marketing. I had a chance to speak with Ben Geller and Robert Reeves of Datical. One of the things they want to use the funds for is to expand beyond the North American market. That is a wise move given the economic environment and the reality of a global marketplace. There was a time when Internet companies could concentrate on the US and thrive, but the demographics just don’t support it anymore.
Finally, today marks the debut of Sendachi. Sendachi is a consulting firm dedicated to helping companies with their DevOps transformation. It was (initially anyway) formed by the merger of Seattle-based Clutch and London based Contino. Along with $30m dollars from Columbia Capital. The money will be used to expand the firms marketing and business teams. Some portion of it though will also probably be used to fund further acquisition. I had a chance to speak with former Clutch and now Sendachi CEO,Steven Anderson and former Contino founder and my friend Ben Wooten. They were both very excited by the potential of this Atlantic spanning venture. Anderson said that the $30m is just the start and if necessary more capital could be available for strategic purposes. My feeling is that we may be seeing the birth of a roll up in DevOps consulting practices in Sendachi.
Congratulations to all of the companies raising capital and buying and selling. Anyone who has had to raise money or have bought and sold a company knows that it is not an easy thing to accomplish. Who will be the next DevOps company raising money? We probably won’t have to wait long to find out.
Editor’s Note: This post was updated Aug. 4, 2018, to reflect an updated slide presentation at the end of the article.
Infrastructure as Code is a truly powerful concept whose significance is best described with the words of Adam Jacob, CTO at Chef, in Web Operations: Keeping the Data On Time: “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.” And so it’s not for nothing that treating infrastructure as code is regarded a key principle for reducing cycle time and maximizing flow in Continuous Delivery and DevOps, respectively. Even if your plans are less bold as you read this article, wouldn’t you agree that consequent validation of what you automate on the infrastructure level — long before it goes into production — would be extremely powerful?
So, bear with me for a while and please don’t rush head over heels into automation by simply picking one of the established infrastructure automation tools such as Chef or Puppet, getting everyone trained in Ruby, and blindly start to “automate all the things”. If that’s how you do it, then you are most probably doing it wrong. In You’re Doing DevOps Wrong. Automation in the Enterprise., author Alan Sharp-Paul suggests that, before doing any automation, you first have to understand precisely the requirements and how to validate them to gauge success and maintain quality of your automations over time:
Are the configuration files in place? Do they contain the right settings?
Is this port open? Is that port closed?
Why Agile Infrastructure?
Honestly, you and I wouldn’t expect anyone to come up with reliable software merely by choosing an arbitrary programming language and a nifty IDE without having some sort of engineering process in place, right? Likewise, the (automated) provisioning of a solid production environment demands a no less methodical approach. Luckily, Infrastructure as Code enables us to treat the provisioning of infrastructure as well as automating deployments and configuration as an Agile engineering discipline. Although the concepts of Agile Infrastructure are not entirely new, testing against live infrastructure has only recently been facilitated with the consolidation of tools such as Ansible, Chef, Puppet, Docker, Vagrant and Serverspec under the hood of Test Kitchen (see more on that below). Here are some of the key implications of applying Agile software development practices to the infrastructure level:
Alignment of Devs and Ops
Agile software development teams are driven by customer demand and aim to produce outcome in short, regular intervals. Enabling IT Operations and Infrastructure departments to work in small iterations and react to changes quickly allows them to align with and effectively become driven by software development. The outcome of these joint efforts will be both working and deployable software at the end of each iteration.
Fail Fast: Feedback Loops
Executing infrastructure tests as part of your Continuous Delivery build pipeline (before an application is deployed and long-running tests are executed) establishes another important feedback loop that enables you to “fail fast” and get the right people involved. The underlying principle is simple: by expressing your intentions twice — once in the code and once in a test — you’ll know you’ve caught a bug whenever these assumptions don’t match.
Regression Safety
Believe it or not: whether you are using Ansible, Chef, Puppet or any other — you’re running on code! Their respective issue lists are full of unresolved issues of various severities with new issues coming in regularly. Having a sufficient safety net of tests for your infrastructure in place allows you to quickly identify bugs as they creep in (from either side) and protects you against unwanted regressions. After all, a bug in your infrastructure can be very hard to track down and can be much more severe than a bug in your application.
Refactoring
Code refactoring becomes a breeze when you have tests in place that immediately alert you when you broke the desired state of your infrastructure (and all you wanted was to make your code simpler and more maintainable). This is invaluable when you want to recompose your existing infrastructure from 3rd-party components offered on public hubs such as Ansible Galaxy, Chef Supermarket or Puppet Forge, and especially when you are migrating away from your existing automation stack.
The Red, Green, Refactor Cycle of Test-Driven Development
Test-Driven Development (TDD) is a popular software development process that involves writing code in short, repeating development cycles with the goal to enforce good design and enhance confidence. While it is certainly no silver bullet and has been controversially discussed in the past, the Red, Green, Refactor Cycle, which is at the heart of TDD and whose principles I want to quickly summarize here, hopefully gives you some inspiration on how to incrementally build up your infrastructure:
Red: Understand the requirements and the state you want to implement. Then think about how to capture this state in a test. Implement the test, then run your entire test suite and see the added test fail — and if it doesn’t, something is really wrong.
Green: Implement just enough functionality to make the test pass. Don’t care much about readability, simplicity and design for now — just make it work. Run your tests again and watch all tests pass. If they don’t, either fix your test or your implementation, or take a step back (revert to a point in your code that’s known to work) and start over with a smaller increment.
Refactor: Now that your tests pass, improve your implementation and make sure, by running your tests again, that you didn’t break anything. Once you are done refactoring, start the cycle over again.
Keep in mind that each repetition of the cycle refers to a small increment. In terms of infrastructure, such an increment could be the installation of a user, a package or the placement of a configuration setting. When you arrive in a spot where you broke something, you can easily revert to a point in your code that’s known to work. Leveraging such practices really makes the difference between Infrastructure Automation and Infrastructure as Code, as described in Infrastructure as Code – Automation Is Not Enough by Keith Morris.
Test Kitchen, Terraform, AWS and awspec
Now, how to get there? Test Kitchen is an easily extensible test harness that allows you to test your code written in Ansible, Chef, Puppet (and others) on various cloud providers, virtualization providers and operating systems with tests written in a variety of test frameworks.
In the following presentation, I give hands-on examples for you to quickly dive into the topic of using Test Kitchen with awspec as a framework to test Terraform deployments on AWS:
A while back we ran an article by Tomer Levy of Logz.io about 9 open source DevOps Tools that we loved. Of course how can you pick just 9 among so many worthy choices? So we wanted to revisit the open source DevOps tools space and add another 9 tools to the list. In a few months we will be adding more open source DevOps Tools to the list as well. In the meantime here are 9 more must have open source DevOps Tools that we love:
1. Kubernetes is an open source orchestration system for Docker containers. It handles scheduling onto nodes in a compute cluster and actively manages workloads to ensure that their state matches the users declared intentions. Using the concepts of “labels” and “pods”, it groups the containers which make up an application into logical units for easy management and discovery.Originally developed and owned by Google,Kubernetes was offered to the Cloud Native Computing Foundation to seed other complementary technologies to enable a set of common container technologies. It is an integral part of CoreOS’s Tectonic product, in addition to being tightly coupled with Mesos and the Mesosphere DCOS.Kubernetes offers the promise of Google-like container usage for everyone. It seems no matter what the basic container technology we use, it is being used with many of the container-based PaaS stacks today
2. Chef – From the official what is: “Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications to any physical, virtual, or cloud location, no matter the size of the infrastructure. Each organization is comprised of one (or more) workstations, a single server, and every node that will be configured and maintained by the chef-client. Cookbooks (and recipes) are used to tell the chef-client how each node in your organization should be configured. The chef-client (which is installed on every node) does the actual configuration.
More than the above though, for many Chef along with Puppet Labs and Jenkins are the heart and soul of DevOps tools. Moving beyond its original configuration management mission today Chef represents the move to “everything as code” while still maintaining its open source commitment. Chef Delivery, Chef Compliance move Chef into the realm of a tool that is involved in the entire DevOps workflow.
3. Puppet – The Romulus to Chef’s Remus, Puppet Labs is one of the founding members of the DevOps movement. Built on open source and still fully supporting its open source heritage going forward, Puppet like Chef has moved beyond its original mission of configuration management and automation of single machines or instances to orchestration of your entire stack.
From Puppet’s open source doc, Why Puppet
Puppet has been developed to help the sysadmin community move to building and sharing mature tools that avoid the duplication of everyone solving the same problem. It does so in two ways:
It provides a powerful framework to simplify the majority of the technical tasks that sysadmins need to perform
The sysadmin work is written as code in Puppet’s custom language which is shareable just like any other code.
This means that your work as a sysadmin can get done much faster, because you can have Puppet handle most or all of the details, and you can download code from other sysadmins to help you get done even faster. The majority of Puppet implementations use at least one or two modules developed by someone else, and there are already hundreds of modules developed and shared by the community.
4. Maven – a Yiddish word meaning accumulator of knowledge, Maven was originally started as an attempt to simplify the build processes in the Jakarta Turbine project. There were several projects each with their own Ant build files that were all slightly different and JARs were checked into CVS. We wanted a standard way to build the projects, a clear definition of what the project consisted of, an easy way to publish project information and a way to share JARs across several projects.
The result is a tool that can now be used for building and managing any Java-based project. We hope that we have created something that will make the day-to-day work of Java developers easier and generally help with the comprehension of any Java-based project.
Maven’s primary goal is to allow a developer to comprehend the complete state of a development effort in the shortest period of time. In order to attain this goal there are several areas of concern that Maven attempts to deal with:
Making the build process easy
Providing a uniform build system
Providing quality project information
Providing guidelines for best practices development
Allowing transparent migration to new features
5. Gradle – the yin to Maven’s yang, Gradle and Maven are much like Puppet and Chef. In this case though Gradle was born out of frustration with Maven and other tools. From the Gradle, Inc. site:
Gradle was founded by Hans Dockter who was working in Enterprise software development and ran into an intractable build problem. At the time he was using Apache Maven. After three days and nights of exhaustive research, he discovered that the problem could not be fixed with the tool at all. Hans vowed to cure this problem for himself, for his team, his company, and for developers everywhere. He was joined in this mission by founder CTO Adam Murdoch. After releasing Gradle the open source project out into the wild in April of 2008, Hans and Adam then hired all of the key contributors whom they met along the way. These included world renown Gradle experts from companies like Thoughtworks, Typesafe, and Pivotal.
Many developers and shops today use one or the other of these open source tools.
6. Mattermost (& Zulip) Today is all about ChatOps. In many organizations it is replacing email as the primary communication method between teams and even outside members now. While there are plenty of commercial solutions in this space like HipChat and Slack, there are some worthy open source tools available as well. One of the biggest advantages to these open source solutions is you don’t have to use a hosted version if you are worried about control, governance or security issues. You can install these on your own server and maintain yourself if that is your preference. Here are two of the leading open source ChatOps tools:
Mattermost – is an open source Slack alternative. The company behind it is a Y Combinator funded startup with founders who come from Microsoft Office engineering. Frustrated that in using a SaaS hosted chat program they were locked out of their own data unless they paid to use the app they set out to create their own self-hosted app and so Mattermost was born. Not sure where the commercial element ill join this, perhaps open core type premium features? Support? Time will tell.
Zulip – Acquired and then released as open source under an Apache license by DropBox, Zulip is a modern take on ChatOps. Not based on IRC and XMPP, Zulip features: Threaded group conversations, One-on-one and group private conversations, Persistence, History, Full-history search, Team presence and buddy list, Inline image, video, and tweet previews, Drag-and-drop file uploads, Stream-wide announcements, Emails for important missed messages, Desktop notifications, Audible notifications, Hotkeys, Emoji, Code, Lightweight markup, Message editing, Invite-only streams, Starred messages, Integrations, API, Mobile apps, Desktop apps
7. OWASP – A developers and DevOps best security friend, OWASP has become the place for web application security. Their mission is to make software security visible, so that individuals and organizations worldwide can make informed decisions about true software security risks. OWASP is actually an umbrella for a number of different projects including tools, lists (OWASP top 10) and groups. Their website says it best:
The OWASP Foundation came online on December 1st 2001 it was established as a not-for-profit charitable organization in the United States on April 21, 2004 to ensure the ongoing availability and support for our work at OWASP. OWASP is an international organization and the OWASP Foundation supports OWASP efforts around the world. OWASP is an open community dedicated to enabling organizations to conceive, develop, acquire, operate, and maintain applications that can be trusted. All of the OWASP tools, documents, forums, and chapters are free and open to anyone interested in improving application security. We advocate approaching application security as a people, process, and technology problem because the most effective approaches to application security include improvements in all of these areas. We can be found at www.owasp.org.
OWASP is a new kind of organization. Our freedom from commercial pressures allows us to provide unbiased, practical, cost-effective information about application security. OWASP is not affiliated with any technology company, although we support the informed use of commercial security technology. Similar to many open-source software projects, OWASP produces many types of materials in a collaborative and open way. The OWASP Foundation is a not-for-profit entity that ensures the project’s long-term success.
8. Zabbix (& Observium) – OK so lots of people have grown to hate Nagios. There are still lot of people who use Nagios too. That being said, we wanted to highlight some open source options in the network and application monitoring space. Two we like are Zabbix and Observium.
Zabbix – The basic work sphere of Zabbix LLC is development of open source software for monitoring of networks and applications. Apart from that the company offers a wide range of professional services designed to fit every customer’s unique business demands including implementation, integration, custom development and consulting services as well as various training programs.
Zabbix team’s mission is to make a superior monitoring solution available and affordable for all.
The company’s flagship product is Zabbix, one of the most popular open source monitoring software in the world. It is already used by a vast number of companies, who have chosen it due to real scalability, high and robust performance, ease of use and extremely low costs of ownership.
Observium – is a low-maintenance auto-discovering network monitoring platform supporting a wide range of device types, platforms and operating systems including Cisco, Windows, Linux, HP, Juniper, Dell, FreeBSD, Brocade, Netscaler, NetApp and many more. Observium focuses on providing a beautiful and powerful yet simple and intuitive interface to the health and status of your network.
Professionally developed and maintained by a team of experienced network engineers and systems administrators, Observium is a platform designed and built by its users. Observium Community is available free to everyone and recieves updates and features twice annually. Observium Professional adds priority access to daily updates and new features for a small yearly fee.
9. Cucumber – OK raise your hand if you use Cucumber. It is one of the best kept secrets in the business. I am constantly amazed out how many developers and DevOps folks mention this open source tool. It’s actually several tools as their are different versions for different languages with each one having a slightly different feature set.
For those of you who don’t know, “Cucumber has revolutionalized the software development lifecycle by melding requirements, automated testing and documentation into a cohesive one: Plain text executable specifications that validate the software.”
So those are our latest 9 open source DevOps tools we love. We know there are more. Leave comments with worthy choices for our next article.
DevOpsQA NJ just held our September Meetup on Configuration Management with David Bowser and Jeremiah Sullivan from Puppet Labs and Michael Pereira from Syncsort as speakers. The venue was provided by ROKITT at a state-of-the-art office space with breathtaking views of NYC.
Michael gave a great introduction to Environment Automation by giving a live demo of BCMT – Bash Configuration Management Tool – which he developed to demonstrate 2 critical concepts – Idempotence and Convergence. Idempotence means that multiple applications of the same action do not have side effect on the system state while convergence means to bring system state in line with a defined polic
y. Changes are made on the system only if they need to be made which is one of the key behind the configuration management tools. My favorite was the definition for Configuration Management provided by Michael – “Whatever you need to run your application” – short, simple, straight to the point.
Next, David and Jeremiah gave a rundown of 2015 State of DevOps Report and a history of Puppet Labs. Jeremiah covered the Open Source Puppet followed by a live demo of Puppet Enterprise. Puppet Enterprise offers everything available in Open Source edition, plus support, training and a web based GUI. Puppet allows to automate provisioning for physical machines, as well as VMs, Docker containers, and AWS instances. One of the reasons Puppet is so powerful is due to the variety of the existing modules that are available from Puppet community.
At the end we had a great Q&A session ranging from Jenkins integration, automated validation of the modules, extending of the modules and future of Chocolatey.
We have assembled another top flight list of speakers for our DevOps Connect event @Infosecurity Europe. While pre-registration is sold out, if you are one of the 15,000+ attendees of Infosecurity Europe, you can still come to DevOps Connect on a first come, first serve basis. Details and schedule are below:
When: Thursday 4th June, Olympia London Time: 9.00 – 17.00
Agenda for DevOps Connect:Rugged DevOps
8:30am to 9:00am
Meet and Greet
9:00am to 9:10am
Introductions – Alan Shimel and Wai Man Yau
9:10am to 9:50am Joshua Corman, CTO, Sonatype and Rugged (Keynote)
Dr. StrangeOps: How I Came to Embrace Rugged DevOps & SW Supply Chains
9:50am to 10:30am Gareth Rushgrove, Puppet Labs
Maintaining Control While Letting Go
10:30am to 11:00am Florin Coada, Technical Specialist Graduate in Application Security at IBM
Security at Speed; Best Practices for a Secure DevOps Implementation
11:00am to 11:30am Helen Beal, Head of DevOps at Ranger4 Limited
The DevOps Toolkit
11:30am to 12:00pm Justin Arbuckle, Vice President, EMEA & Chief Enterprise Architect at Chef (formerly Opscode)
A Manifesto for high velocity compliance
12:00pm to 1:00pm
Lunch Break
1:00pm to 1:30pm Wai Man Yau, Sonatype EMEA
Real World Customer Stories from the Software Supply Chain
1:30pm to 2:00pm Stephen de Vries, CTO Continuum Security
Automating Security Tests for Continuous Integration
2:00pm to 2:30pm Elizabeth Lawler, CEO and CoFounder , Co-Founder Conjur Inc
SecDevOps 2.0- Managing your Robot Army
2:30pm to 3:00pm Marc Hornbeek, Senior Solutions Architect at Spirent Communications
Case study – DevOps Cost-Effectively Secured Quality and Time-to-Market for a Complex Hardware/Software Network Product
3:00pm to 3:30pm Jonathan Fletcher, Enterprise Architect – Technology & Platform (DevOps) Lead
DevOps: How the Industrial Revolution Got Us Here
3:30pm to 4:00pm Benjamin Wootten, Co-Founder at Contino
Transforming to Devops in the Enterprise
4:00pm to 4:10pm Thank you and closing – Alan Shimel and Wai Man Yau
If you are in London during these dates you can still register for Infosecurity Europe for free and attend DevOps Connect. Hope to see you there.
chael gave a great introduction to Environment Automation by giving a live demo of BCMT – Bash Configuration Management Tool – which he developed to demonstrate 2 critical concepts – Idempotence and Convergence. Idempotence means that multiple applications of the same action do not have side effect on the system state while convergence means to bring system state in line with a defined polic
erprise. Puppet Enterprise offers everything available in Open Source edition, plus support, training and a web based GUI. Puppet allows to automate provisioning for physical machines, as well as VMs, Docker containers, and AWS instances. One of the reasons Puppet is so powerful is due to the variety of the existing modules that are available from Puppet community.
