devopsy.kinsta.cloud

Tag: continuous integration

Assembling the Key Components of Continuous Delivery
As agile methodology was popularized at the start of last decade, engineering teams gained the opportunity to begin producing software in shorter cycles, while releasing it reliably at more frequent intervals.

This concept garnered greater attention in 2009, when Tim Fitz published his thoughts on continuous deployment. By the time Jez Humble authored his seminal book, “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation,” in 2010, the approach started to get significant mindshare. In the book, Humble advanced the three building blocks of continuous delivery: configuration management, continuous integration and continuous testing.

Thereafter, Agile Operations—the practice of using end-to-end monitoring, fueled by analytics, to increase operational agility—appeared. This complemented continuous delivery, with the goal of increasing the quality and velocity of software delivery while minimizing overhead.

Since then, and particularly over the last six years as DevOps adoption has exploded, there’s been significant discussion (and confusion) over the opportunity to embrace “continuous everything” across the software development life cycle (SDLC) to deliver functionality continuously and optimize end-user experience.

Today, having spoken to hundreds of organizations employing agile and DevOps, it’s possible to expand on these concepts to better define the key components of continuous delivery.

In this blog, I’ll detail the precepts of “CD” that CA’s customers recognize as critical in reducing the time, risk and expense of software delivery by frequently releasing smaller batches of code with extensive visibility and traceability.

These building blocks of continuous delivery are:
- Continuous development & integration,
- Continuous testing. and
- Continuous release.
Continuous Development and Integration

When it comes to delivering quality apps faster, every second counts.

That’s why it’s important to automate the planning, development, unit testing and integration of code with different inter-dependent components, all while packaging it for quality assurance. We categorize these activities under continuous development and integration, as shown in Fig 1.

This all starts with sprint planning and modeling of requirements in agile management software. Unit tests can be created automatically, based on requirements and use case stories. These unit tests then can be executed automatically based on pre-defined criterion, using automatically generated synthetic data.

Fig 1: Continuous Development and Integration

Next, virtual service interfaces can be generated automatically by simulating unavailable systems across the SDLC, allowing developers, testers, integration and performance monitoring teams to work in parallel—driving faster delivery, along with higher quality and reliability.

The integrated code can be built and tested automatically against validation criterion, and upon passing, made available for continuous testing. This later phase, after code development and before QA, is also commonly referred to as continuous integration.

Continuous Testing

Continuous testing across app delivery not only enables enterprises to shift testing left, engaging it earlier in the SDLC, but it also helps them shift right, leveraging the same processes throughout staging and production.

After the unit tests are passed and integrated builds are made available for QA, functional and performance tests can be completed—before the build is made available for staging and production. Various activities during continuous testing are shown in Fig 2.

Before the functional and performance tests are performed, the build promotion criterion and configuration also need to be validated. Testers require provisioning of QA environments to deploy the build, along with virtual services and test data. Using test data management to create a subset of production data and mask sensitive information is another continuous testing requirement.

Fig 2: Continuous Testing

It’s also important to engage application performance monitoring in the QA environment before running automated tests, providing feedback about any problems back to developers.

After the automated tests are executed and promotion criterion are met, the build is ready for deployment in staging or production environments.

Continuous Release

Continuous release matches release velocity to development velocity through controlled and scalable deployments.

The goal of continuous release (sometimes referred to as continuous deployment) is to continually and automatically deploy the build into production; similar activities also can be performed in a staging environment.

As shown in Fig 3, build configuration needs to be validated, with the production environment configuration updated before being sent to production. As with application code, configuration is also maintained in the version control system and can be automated for deployment.

Fig 3: Continuous Release

After the build is deployed into production, application monitoring should be initiated to track performance and customer experience. Feedback from the production environment can be passed back to developers to guide enhancements or sprint backlogs.

This blog aims to outline core CD activities, and in subsequent articles I’ll provide additional details on entry and exit criterion, as well as metrics useful in tracking continuous development & integration, continuous testing and continuous release.

In the meantime, check out, “DevOps for Digital Leaders: Reignite Business with a Modern DevOps-Enabled Software Factory,” a new book authored by my colleagues Aruna Ravichandran, Kieran Taylor and Pete Waterhouse.

Based on learnings gained from numerous customers who have already embraced CD and DevOps, it highlights key practices for starting this journey, related ROI metrics and other useful tips.

— Anand Akela
December 15, 2016
A Ferrari and DevOps: More Similar Than You Might Think

In a world where customers aren’t short of options, all businesses are characterized by the need to achieve customer delight. From a technological standpoint, a large part of that hinges on fast-tracking the release of high-quality software. A release every quarter is being replaced by a release every week and sometimes even sooner. Expectations have skyrocketed while the timelines have shrunk even more. It is now imperative for all the different components to sync together effectively to create a competitive advantage for the business that can be sustained over a period of time.

All the water-cooler talk around DevOps notwithstanding, it’s true identity still varies vastly based on the passionate opinions of participants with varied schools of thoughts. It also bears mentioning that while all of their varied opinions carry a sliver of truth, they also are significantly separated from the heart of the matter.

To better understand the situation, let’s jump into a story: Once upon a time there was a monk who owned a Ferrari and had no interest in selling it. Every morning when the monk left to teach his mindfulness class, there were three blind men who were captivated by the sound of its engine and the smell of burning rubber. And, yet, they couldn’t see the very object that had captured their fancy. So, one fine day they decided to touch the Ferrari and exchange notes.

The first man touched the broad wheels of the Ferrari and exclaimed that it was a large contraption made of soft rubber. The second man ran his hands across the exquisite curves of the car and said it was like an ancient Greek statue, while the third one placed his hand on the Ferrari’s windows and concluded it was a moving glass palace on wheels. Although each was right in his own way, none came even remotely close to capturing the essence of a Ferrari.

By now, you don’t need a monk to tell you that the story of DevOps is very similar. For some, DevOps is all about inculcating the right culture so that different groups can work together seamlessly to achieve a unified goal. For others, it is about establishing the right processes and practices that are in sync with an organization’s business goals. And for those with an axe to grind or proprietary tools to sell, DevOps is all about achieving automated nirvana.

However, the ground reality is that software development teams constantly are facing increased pressure from business leaders to deliver faster than ever before. The business leaders, in turn, are under the hammer of market forces that require them to accelerate or die. The need of the hour is to deliver high-quality software at the rate of knots, while untangling any that might impede the time to market.

Continuous Everything

We at Happiest Minds believe the simple approach to DevOps is one that is “continuous” and at its core has the agile mantra of “people over processes and processes over tools.” The idea of continuously evolving and adapting to changing requirements and demands has been captured in what we call the 7Cs of DevOps:

The main objective is to ensure agile delivery by embracing continuous planning, continuous development, continuous Integration, continuous deployments, continuous testing, continuous monitoring and continuous feedback.

However, the cultural component of DevOps cannot be overemphasized. A large part of the success or failure of any DevOps engagement depends upon how willingly it is embraced within an organization and the kind of executive sponsorship it receives from the powers that be. The three widely recognized cultural models that most organizations come under are pathological, bureaucratic and generative. Needless to say, organizations that are riddled with a pathological or bureaucratic culture are likely to experience the least amount of success with DevOps because such organizations tend to cover up failures and eventually pin them on the most susceptible scapegoats.

The breeding ground that DevOps needs to flourish is a generative culture where information is shared freely and openly, the leadership encourages employees to learn and experiment with new ideas and failure isn’t looked at as the end of the world but rather an opportunity to learn and grow.

Eventually, organizations will have to embrace DevOps as an integral part of their strategy and not just as a liberal, forward-thinking experiment that scores them some brownie points. But it will take some doing, both at the grass roots and in the board rooms. One has to remember that a lot of people are easily attracted to a Ferrari, but perhaps not to the grind that goes into owning one.

The idea of having a successful DevOps program is much the same—it’s not just about engaging the best vendor or choosing the right tools or paying lip service to creating the right culture; ultimately, it requires overcoming organizational inertia, resistance to change and making a long-term commitment to the philosophy that takes a larger view of how DevOps can help businesses stay one step ahead.

About the Author / Arsalaan Kashif

Arsalaan Kashif is associate director of Marketing, Product Engineering Services at Happiest Minds Technologies and has a keen interest in all emerging areas of technology and innovation. Having worked with Unisys, Oracle and HCL Technologies for over a decade, he has experience with taking to market products and services across areas as diverse as application services, cloud computing, DevOps and big data analytics to name a few. Outside of work, Arsalaan enjoys watching noir films and reading John Irving classics. Connect with him on LinkedIn.

November 28, 2016
Continuous Integration vs. Delivery vs. Deployment: What’s the Difference?

“Continuous” is one of the most popular words in the DevOps lexicon. What does it really mean, and what is the difference between continuous delivery, continuous deployment and continuous integration? Keep reading for some perspective.

Like many technological concepts, continuous integration, continuous delivery and continuous deployment are terms that DevOps teams tend to use without defining them precisely. For that reason, it’s worth taking a closer look at what each of these terms actually means.

Continuous Integration

Continuous integration can really mean one of two things, depending on the context.

In the narrower sense, continuous integration refers to the work done by a continuous integration server, such as Jenkins or Bamboo. A continuous integration server automatically tests code written by individual developers to make sure it can be merged into the main code base. By automating this process, continuous integration servers help developers to write and test small chunks of code on an ongoing basis, which minimizes the risk that a code change could cause serious problems and force developers to revert their application to an earlier state.

In a broader sense, continuous integration means the constant integration of changes to an application at all stages of the delivery chain. This includes but is not limited to the automated integration testing and code merging performed by an integration server. In this broader definition, continuous integration can include the integration of new design concepts to the delivery pipeline, for example. It also can involve other types of software testing beyond just automated integration tests.

Continuous Delivery

In most definitions, continuous delivery refers to the process of delivering software updates to users on a nearly constant basis. Continuous delivery is made possible by continuous integration and other optimizations at earlier stages of the development process. But the definition of continuous delivery gets a little cloudy when you start comparing it to continuous deployment.

Continuous Deployment

What’s the difference between continuous delivery and continuous deployment? For some DevOps teams, nothing. People tend to use these terms interchangeably. In fact, they are usually simply abbreviated as “CD,” leaving it up to the listener to decide what the speaker is actually talking about when he hears the term during a conversation about DevOps.

Some developers draw a distinction between continuous delivery and continuous deployment, however. For example, Mirco Hering writes that continuous delivery requires that the DevOps team manually release updates to users. In contrast, the continuous deployment pipeline is fully automated; users get updates as soon as they are written and tested, with no manual intervention by developers.

This distinction is a good way to think about the difference between the closely related concepts of continuous delivery and continuous deployment. It’s a distinction that will become increasingly important as technologies such as Docker make it easier to automate application deployment by allowing DevOps teams to drop new container images into production.

What Does Continuous Mean, Anyway?

It’s worth mentioning, too, that the word “continuous” is usually something of an exaggeration when used by DevOps folks. It implies that software changes roll down the pipeline in a totally constant, never-stopping fashion.

That is an exaggeration. Integration, delivery and deployment are almost never completely continuous. In practice, a “continuously” integrated application is likely to be rebuilt and delivered something like every 24 hours, not every single time a code change reaches the end of the pipe.

That rhythm is still a lot faster, and closer to continuous, than the “waterfall” delivery pipelines that were common before the introduction of DevOps. But it’s still important to understand that, even with all of the automation tools available today, software development is rarely completely continuous.

— Chris Tozzi

November 18, 2016
All Day DevOps
Over the past year, I have traveled to and delivered presentations at 18 DevOps events. I’ve also heard that over the past several years, John Willis has participated in more than 200 DevOps events. But not all of us have the time or budget to get ourselves and our teams out to these events where we can learn from other practitioners.

This is why we’ve assembled the All Day DevOps Conference for Nov. 15.

Free DevOps

All Day DevOps will deliver 57 practitioner-led sessions over 15 hours and 15 time zones. No vendor pitches are allowed. To break the travel and budget barrier for you and your teams, we are delivering every session live, online and free of charge. Every session will also be recorded. We are doing this because we want to share global DevOps expertise with everyone.

DevOps Brains

More than 3,500 people have now registered for the conference. We have an awesome speaker lineup covering three tracks: Modern Infrastructure, CI/CD and Automated Security. Here is a sample of some of the great sessions being delivered:
- CI/CD
  - Meta Infrastructure as Code: How Capital One Automates our Automation Tools with an Immutable Jenkins, by George Parris (Master Software Engineer at Capital One)
  - Speaking for the Dead: Critical Positive Lessons from Waterfall and Monolithic Architectures, by Michael DeHann (Former CTO & Founder of Ansible)
  - I, For One, Welcome Our New Robot Overlords, by Mykel Alvis (DevOps Computational Demonologist at Cotiviti)
- AUTOMATED SECURITY
  - Stop Changing Your Password, by Chris Swan (CTO Global Infrastructure at CSC)
  - System Hardening using Ansible: Application Deployment + Configuration Management + Continuous Delivery, by Akash Mahajan (Lead at OWASP Bangalore)
  - Building Codes for Building Code, by Josh Corman (Director of Cyber Statecraft at Atlantic Council)
- MODERN INFRASTRUCTURE
  - Terraforming Your Infrastructure: From Beginner to Semi-Pro in 30 Minutes, Britt Treece (Production Engineer at PhishMe)
  - DevOps Friendly Doc Publishing for APIs & Microservices, by Mandy Whaley (Director of Developer Experience at Cisco)
  - Monitoring, Hold the Infrastructure – Getting the Most out of Lambda, Michael (DevOps Evangelist at DataDog)
Don’t Just Listen, Talk DevOps

One of the big pieces of input we have received from the community is that everyone wants an opportunity to interact. This is why we are building a dedicated team room in Slack for you. Every attendee will have the chance to interact with the speakers in the Slack room. Q&A will be live, online and freely accessible. We will also have Birds-of-a-Feather channels set up for those with questions to be helped by those with answers.

Register Now

If you would like to join us, can you register for free. A big thanks goes out to Alan and his team at devopsy.kinsta.cloud for helping us spread word of the event here. We are really excited to have teams from North America, Europe, Asia, and Australia joining us for the event!

www.AllDaydevopsy.kinsta.cloud

— Derek E. Weeks
October 26, 2016
An IT Practitioner’s Guide To DevOps
Two technology innovations of the past decade have fundamentally changed how IT works and given rise to the “DevOps” phenomenon. Before we dive into what DevOps is and how IT organizations should adopt this culture, let’s review what gave rise to it in the first place.

Pull from Agile Development

The last decade has seen many organizations abandon the traditional “waterfall” software development process in favor of “agile” development. Agile is all about small, specific user-centric stories, described by a product owner and delivered in short sprints of two to four weeks by small, focused development teams.

Agile development led to the emergence of more frequent code drops from software engineers, which needed to be deployed more quickly. No longer was it okay to take days or weeks to deploy a new software version; developers working in two-week sprint cycles want deployment to work within minutes with a high degree of predictability.

Enabled by the Cloud

Before the cloud came along, access to infrastructure required human interaction and manual operations. Processes typically could not be automated effectively. This meant that operations were slow and error-prone and created a huge point of friction in the software development life cycle.

Around the same time that developers were adopting agile development, the cloud became real, and infrastructure could be orchestrated effectively using APIs to provision everything from machines and networks to storage, routing policies and firewalls.

DevOps: Closed Loop Automation Between Development and IT

Agile development and cloud computing created the perfect storm in which DevOps was born: this new way of doing IT is all about ensuring closed loops, with 100 percent automation.

Figure 1: DevOps is about ensuring a closed-loop, fully automated workflow between software development and deployment.

In the simplest terms, this meant that each of the steps outlined in Figure 1 need to be fully automated, and integrated with preceding and following steps. Doing this ensures software can be developed faster, deployed quickly, and iterated upon more rapidly.

Benefits

Done right, DevOps leads to a virtuous cycle of greater productivity across development and IT, greater agility in bringing new experiences to market and greater impact on the business.

Adopting DevOps: You Can Phase It In

Organizations looking to adopt DevOps should do so in phases and define metrics for each phase. Here’s a typical phased approach that you can use as a baseline and customize to your organization’s unique needs.

Phase 1: Adopt Agile Development and Cloud Computing

Agile development and cloud computing are profound technology shifts and create the underlying culture in which DevOps can thrive. So, the first step is to ensure organizational buy-in for both of these trends:
1. Agile development can be adopted in stages. The most important is to agree that software will be built and shipped in short, two-to-four-week sprint cycles, and with a focus on user stories.
2. Cloud computing can be adopted no matter what your IT infrastructure looks like today. Use hybrid cloud platforms such as OpenStack to provide programmatic access to a VMware or Linux environment, or use a public cloud platform such as Amazon Web Services.
Phase 2: Commit to a CI/CD Workflow

Continuous integration (CI) systems such as Jenkins or TeamCity provide a way to pipeline and integrate the various steps in a DevOps workflow. The choice of system doesn’t matter as much as ensuring that all steps of the workflow will be integrated (eventually) into this workflow.

Phase 3: Repeatable, Automated Software Deployment

Many organizations tend to have some manual steps in the packaging and deployment of their software, especially for getting it into production. This often reflects underlying assumptions about manual oversight and intervention, and it breaks the closed-loop philosophy of DevOps.

To succeed with DevOps, you need to fully automate the software packaging and deployment process. In addition, ensure that production builds are deployed in the exact same manner as developer builds, using identical environments.

Cloud technologies such as orchestration (OpenStack HEAT or AWS CloudFormations) and configuration management tools such as Ansible, Puppet or Chef can be used to implement this phase.

Phase 4: Automate Software Testing (Continuous Delivery)

Once software deployment has been automated, the next phase is to automate the validation process for new software releases. If you’re like most organizations, testing involves several rounds of back and forth with QA teams and product managers, and it often takes weeks to resolve the last few bugs in each release. To reach the next level of DevOps automation, this process has to be completely lights-out and automated.

This requires a fully automated software test suite, across the back-end and front-end, so that product managers have confidence in the quality of the software being released. Once implemented, you finally have a closed loop between the critical elements of the DevOps life cycle. While release to production may still require someone to approve it or “push a button,” this is a very respectable state in the DevOps journey.

Phase 5: Continuous Deployment

To get to the most advanced DevOps phase, you need to automate the “go live” process for software releases. This in turn will allow you to take even smaller, more frequent code drops and push them into production.

The most advanced software teams in the world now operate this way, and the organizational implications can be transformative: Developers come into work feeling that they can implement a feature and have it go live immediately, and product managers can make commitments to the business with the confidence they can deliver, because they have this incredibly efficient software delivery machine backing them. Most importantly for you, IT is no longer seen as a cost center; it’s seen as the fundamental enabler for business productivity and time-to-value.

Closing Thoughts

Getting to Phase 5 represents the state-of-the-art in the industry today. However, another significant breakthrough promises to raise the bar even higher.

In each of the prior phases, the DevOps workflow is integrating software development with IT operations; but there is nothing influencing software architecture to take operations into account.

In other words, we’re still bolting IT operations automation onto software development.

This is about to change. We’re on the cusp of a new era in software: instead of bolting operations onto the software development process as an afterthought, a new framework promises to bring development and operations together by design. With declarative, infrastructure-agnostic constructs to describe how applications are composed, interact and managed, Kubernetes promises to increase the operability of modern software systems by an order of magnitude.

Everyone interested in the DevOps journey should be learning about Kubernetes and evaluating how it will improve your software development and operations workflows.

About the Author / Sirish Raghuram

Sirish Raghuram is co-founder & CEO of Platform9. As CEO, Sirish quarterbacks the team at Platform9 in realizing the company’s vision, which, on a given day, can range from strategy and road map to shipping code and running tests. Before co-founding Platform9, Sirish was an early engineer at VMware, where he held technical and management leadership roles and helped ship multiple vSphere products. Sirish’s work was reflected in several patents, VMworld talks and internal product innovations within VMware Engineering. Sirish has a B.E. in Computer Science from the University of Pune, India. Connect with him on LinkedIn.
October 25, 2016
Continuous Integration vs. Continuous Delivery: There’s an Important Difference

If you read DevOps blogs today, it can be easy to assume that Continuous Integration (CI) and Continuous Delivery (CD) are synonymous terms. People often use them that way. But that’s a mistake.

As an example of conflation of the terms CI and CD, take this page, which purports to be a “history of CI/CD tools.” The author does provide a truly excellent history of CI platforms.

But CI tools are the only subject that the page discusses. It’s not a history of CI and CD. It’s just a history of CI platforms. To work the history of CD into the picture would require discussion of how software testing and delivery practices have changed over time, not just an overview of the history of CI software over the past several years.

I don’t mean to knock the page. It’s a great read, and the mistake the author makes in treating CI and CD as synonymous is hardly rare. He or she is just repeating a common confusion. You see the same lack of distinction between CI and CD on Docker’s discussion of CI/CD, for instance.

The Difference between CI and CD

It’s unclear how this conflation of the two terms originated. But it’s easy enough to clear up.

Simply put, CI is the process of integrating code into a mainline code base. In almost all scenarios today, CI is done using platforms designed specifically for the purpose. Implementing CI is, therefore, as simple as using the right tools.

CD is more complicated. CD is about the processes that have to happen after code is integrated for app changes to be delivered to users. Those processes involving testing, staging and deploying code. There is no single type of tool that does all these things. These processes take different forms, depending on the culture of the team and the type of app it is creating.

In actuality, then, CI and CD share little in common. Besides starting with the same word (“continuous”) and being different stages in modern software delivery pipelines, they don’t have much to do with one another.

Why the Difference Matters

Why is it important to steer clear of this confusion? The main reason is because, by conflating CI with CD, organizations can end up doing only the former but thinking they have implemented the latter.

Setting up a CI server such as Jenkins or Bamboo to integrate code changes continuously does not mean you are doing CD. It just means you are using a CI server.

To get to CD, you also have to organize your software testing, staging and deployment processes in a way that automates them as much as possible and makes them continuous. Doing so requires more than simply turning on a certain type of tool.

How you actually do CD is fodder for a separate, much longer article (or you could just read the book about it). But recognizing that CI is not CD, and avoiding confusing formulations including CI/CD, is a start.

— Chris Tozzi

October 20, 2016
3 AWS Design Patterns to Maximize DevOps Value
Many business leaders are planning to make DevOps the cornerstone of their IT transformation strategy. But, as surprising as it might sound, skepticism abounds regarding how major vendors address the DevOps transformation challenges—they confuse DevOps with continuous delivery (CD) infrastructure.

“We wanted DevOps and we got after a nine-month-long implementation a continuous delivery platform which speeds up IT operations without significant business value. How differently can you help?” the senior business executive of a major financial institution in New York sarcastically asked me during a conversation three months ago.

The business executive is right: DevOps isn’t a technology as I’ve been implementing it, it’s primarily an accelerated application delivery approach implemented in the form of a technology-enabled application delivery operational model. Adam Jacob, CTO of Chef, substantiates that fact in “The Secret of DevOps: It’s Always Been About People, not Technology,” when he says, “DevOps is about bringing together all the people you need to build and run your business effectively, and empowering them to move as quickly as possible towards their goals.”

Agile Admin’s Ernest Mueller in, “DevOps: It’s not Chef and Puppet,” confirms Adam’s point: “Chef and Puppet are individual tools that you can use to implement specific parts of an overall DevOps strategy if you want to use them—but that’s it. They are fine tools, but do not solve DevOps for you.”

How can we implement DevOps in line with business leader expectations? Three Amazon Web Services (AWS) Design Patterns I developed can help to get the best of DevOps on AWS cloud: Software Organization Factory, Digital Organization Factory and CD Infrastructure Factory. This article discusses the Software Organization Factory.

The Foundation of DevOps Value: Consequences to IT Service Delivery

Before delving into these AWS design patterns, a quick reminder of what DevOps is, its value propositions and, more importantly, what changes are needed in how IT services are delivered.

The Ignored DevOps Business Logic We All Must Consider

The combined effect of cloud adoption and the rising digital economy is changing competitive environments; it forces business leaders to focus on innovation and market responsiveness as the competitive differentiators.

The fact of the matter is, by facilitating access to technology, cloud accelerates entrance of several thousands of tech startups that take advantage of the recent technology innovations to substitute traditional products and services into digital. This creates a context where new entrants increasingly compete with established brands, forcing business leaders to seek value paradigms other than IT efficiency.

The business executive argues, “In these new markets, our concern is to increase customer value by delivering the right services to the right segments at the right moment. You guys need to change the IT paradigm; IT alone does not create that kind of value.”

Then, what should be changed in today’s IT approach?

The Two Dimensions of Modern IT: The Operational Model and the Continuous Delivery Infrastructure

The new IT paradigm advocated in my recent book, “Rethink Your IT with DevOps: The Complete Guide to Aligning Your IT to DevOps,” is the idea that business value results from two dimensions: organizational agility and technology efficiency. In other words, it defines modern IT as a two-dimension asset including the operational model and the continuous delivery infrastructure.

The timely delivery of innovation transcends IT issues; it not only cuts across your sales, marketing, post sales, application development and IT operations, but also demands concrete answers as to how to accelerate business prioritization, decision-making and problem solving. The operational model is where business and IT people, processes, practices and tools are combined to create agile and cross-functional environments. The following exhibit shows a typical DevOps operational model:

Typical DevOps Operational Model

Organizational flexibility for the sake of flexibility doesn’t make sense unless technology is leveraged to speed processes. This is where CD infrastructures come into play; throughout the delivery process, they build on IDEs to accelerate coding and testing, on versioning tools to convert software version into releases, on continuous integration (CI) tools to build releases and on delivery pipelines to ensure continuous delivery.

So, what is DevOps?

DevOps is a Technology-Enabled Application Delivery Model

DevOps is a combination of value, principles, processes and practices shared not only by developers and IT operations but also by the business. The goal is to help the organization deliver the right service to the right market segments at the right moment.

DevOps from the Business Perspective

As I’ve been implementing it, it’s a cross-functional application delivery model involving the business and IT and enabled by continuous delivery infrastructures.

By enabling and automating agile practices, the continuous delivery infrastructure makes the operational model agile and provides the business with the flexibility and velocity needed to meet competitive challenges.

Design Patterns as DevOps Implementation Engineering Framework

As suggested by the business executive, today’s DevOps implementations struggle to deliver the promised business benefits. The reason is, the operational model dimension is ignored in favor of tools and infrastructure.

Another reason is, DevOps is young and is still at the early stage of the technology maturity model; the expertise and engineering methodologies to implement it as what it is—a technology-enabled application delivery model—are lacking. That’s where DevOps transformation design patterns help.

What are DevOps Transformation Design Patterns?

In this article, the term DevOps transformation design patterns refers to general reusable architectures addressing specific DevOps strategy. They’re not necessarily implementable solutions, their primary goal is twofold:
- Offer a neutral language for designing and implementing technology-enabled operational models
- Offer an engineering approach that reduces design and implementation duration and effort
They’re formalized architecture design and implementation best practices for DevOps and platform-as-a-service (PaaS) services.

Design Pattern as DevOps Implementation Engineering Methodology

To meet its goals—technology-enabled application delivery model—the design pattern is structured around four items including purpose, principles, general solution and implementation techniques.

Purpose: Clarifies the purpose of the design pattern along with the DevOps strategy to address.

Principles: Are best practices formulated as rules and beliefs supposed to guarantee expected DevOps strategy benefits? They’re the determinants of the solutions to implement the DevOps strategy.

General solution: Are blueprints resulting from the application of principles? They summarize the functional and technical architecture of the DevOps platform to implement.

Implementation techniques: What are recommended technical approaches, solutions and tools to implement the general solutions?

Why AWS? Why would it be the catalyst for DevOps value?

AWS-ome Design Patterns That Will Change DevOps Implementation

AWS offers such a rich, varied and easy-to-implement services portfolio that it would be unfortunate not to take advantage of it to modernize IT. The rationale underpinning the value businesses can derive from their DevOps transformation experience with AWS is summarized in the following figure:

Rationale of the DevOps Design Pattern

It’s the belief that:
- Certain principles determine the AWS services needed to deliver the application delivery model (ADM) architecture that guarantees value.
- The implementation of the ADM using these AWS services results in AWS-based DevOps platforms that tangibly deliver expected benefits.
The Four DevOps Principles that Yield AWS Value

Let’s explore the principles I’ve been using to make CIOs happy with their AWS cloud:
1. Think holistic, not only IT: Reminds that transformation isn’t only about IT but the entire organization’s value stream from marketing and sales to development and IT operations.
2. Think cross-functional, not only IT: Reminds that the bottom line is to set up a cross-functional application development environments to accelerate business prioritization, problem-solving and decision-making, and meet time-to-value objectives. They’ll be the core value propositions of the DevOps platform.
3. Make it agile, flexible, and speedy: Reminds that agility, flexibility and speed are the catalysts for competitiveness and business value. They’ll be part of the DevOps platform.
4. Think PaaS: Reminds that AWS provides varied services for implementing state-of-the-art CD infrastructure.
Let’s illustrate three AWSome design patterns that’ll change DevOps experience with AWS!

Implement a Startup Mindset with the Software Organization Factory

The purpose is to implement DevOps strategies focused on time-to-market and time-to-value. It’s adapted for tech startups.

To implement the startup mindset, the DevOps four principles for AWS are implemented as follows:
- Think holistic and cross-functional, not only IT: Relevant business and IT resources are put together in a single and dedicated service development team supported by appropriate Scrum and XP agile practices.
- Make it agile, flexible and speedy: PaaS services building on agile practices and AWS infrastructure-as-code are provided to make application delivery processes agile, flexible and speedy.
- Think PaaS: AWS infrastructure as code and DevOps tools are combined to implement a CD infrastructure in the form of a PaaS platform.
The DevOps platform resulting from the principles application is illustrated below:

Software Organization Factory

The “Startup Mindset” platform is structured around three components including the cloud-enabled operational model, PaaS and the CD infrastructure.

The operational model builds on a three-stage application delivery life cycle:
- Plan & measure: It mobilizes business users, development, QA and operations staff on business prioritization issues using agile practices such as Scrum product backlog planning or XP’s user stories.
- Develop & Test: It mobilizes business users, development, QA and operations staff across the development life cycle using agile techniques such as Scrum iterations, XP’s test-driven development (TDD) and behavior-driven development (BDD).
- Release & Deploy: It leverages continuous integration practices to build and test releases prior to deploying to production.
- Monitor & Improve: It’s outsourced to keep the service development team on its core business: developing software.
The CD infrastructure combines the followings services:
- AWS CodePipeline and CodeDeploy to set up the delivery pipeline.
- AWS infrastructure as code including CloudFormation, OpsWorks and Elastic Beanstalk to generate the CD infrastructure.
Summing It Up

The DevOps business benefits are real, yet, it’s young and is still at the early stage of the technology maturity model. The expertise and engineering methodologies needed to implement it as what it is—a technology-enabled application delivery environment—are lacking.

DevOps design patterns such as the software organization factory fill the gaps of today’s transformation practices:
- They provide the architect team and the executive team a neutral language to discuss the business, technological and technical challenges and solutions.
- By pointing out the agile practices to leverage, they simplify and accelerate the design and deployment of the operational model.
- By pointing out the AWS services and features to leverage they facilitate the design and implementation of the CD infrastructure.
The DevOps implementation process described in this article is fully documented in my recent book, “Reinventing Your IT with DevOps: The Complete Guide to Aligning Your IT to DevOps.”

In my next article I’ll tackle the Digital Organization and the CD infrastructure design patterns.

— Philippe Abdoulaye
October 11, 2016
Creating a Continuous Integration Delivery Pipeline
This article will provide an example delivery pipeline used through continuous integration, as well as the DevOps tools involved in the automation of building, testing and deploying code through your software development life cycle (SDLC).

Background

These days, it’s almost impossible to go anywhere without hearing about DevOps. In short, DevOps is the effort to improve communication and synergies between development organizations and operations organizations—hence the term, “DevOps.” The main focus is to improve the efficiency, quality and speed to market within the software development world.

Today, you also may hear and see a lot about continuous integration and continuous delivery/deployment. There are differences between the two for sure, but for simplicity’s sake, this blog focuses on breaking down the different stages of the process. This way you can get the basics and hit the ground running with a delivery pipeline in your organization. The delivery pipeline can be broken down into a few major buckets of work, or stages, as mentioned below.

In my opinion—and, again, trying to keep things simple for now—the stages can be broken down as follows:
1. Source Code Control (Management)
2. Build Automation
3. Unit Test Automation (could also include Integration Testing here as well)
4. Deployment Automation
5. Monitoring – not included in this discussion, and can be added at any time
I have outlined the progression of these stages in Figure 1 below. Let’s review each of the stages in a little more detail.

Figure 1 – Continuous Integration Delivery Pipeline Sample

Source Code Control (Management)

Background

Source code management, or source code control, certainly is not a new topic. This has been around for decades and has evolved over time. In a nutshell, an organization stores its code in a source code control system or repository so it can be tracked, maintained, versioned and audited. Developers should not store the code on their laptops or virtual machines and trust that will suffice for managing the code.

Possible Tools
1. Git is probably the most widely used SCM system out there. It is an open-source system.
2. Subversion (SVN) has been around for quite some time. It is also an open-source system and is heavily used in many organizations, but there has been more of a push toward Git.
Git and Subversion comparison can be found here.

Build Automation

Background

Once a source code management system is in place and actively being used by a development team, the team will need to be able to compile and build their code. This is probably the first step in the whole chain of continuous integration events—it’s what gets the ball rolling. The code needs to build cleanly before an organization can even think about deploying out to its environments for testing and production.

Possible Tools
1. Gradle is an open-source build automation system used by top companies including Netflix, Google and LinkedIn.
2. Maven is another open-source build automation system.
Gradle and Maven comparison can be found here and here.

Unit Test Automation

Background

Developers unit test their code to ensure the functionality they are building works as expected. In an ideal world, the development team should be saving these unit tests so they can be reused and put into a regression test bed.

Possible Tools

There are multiple tools out there for helping developers unit test their code. Many of these tools are open-source and can be used freely.
1. JUnit is an open-source unit test framework pretty widely used.
2. CA DevTest allows for the automation of unit testing, as well as a few other bells and whistles, such as service virtualization.
Deployment Automation

Background

For the last stage in the process, delivery teams need to deploy their code/applications out to various test environments and, of course, production. To reduce errors and overhead in the deployment process, while increasing speed to market, this step can be automated through a variety of tools and methods.

Possible Tools
1. IBM Urbancode uDeploy allows users to model a process and orchestrate their deployment. This process then can be repeated across all an organization’s environments, and tweaked for each environment as needed.
2. Ansible is an open-source IT automation tool. It can be used for everything from configuration management to product installation to application deployments.This tool is rapidly gaining acceptance and momentum in the DevOps community.
Conclusion

This was simply a quick overview for companies to get started down the path of continuous integration and the DevOps world. Figure 2 is an example of what the whole flow looks like from committing the code to repo to deploying the code to an environment, to help put things into perspective. It’s easy to see how the tools interact with each other on the journey to production.

Figure 2 – Continuous Integration Process Flow

About the Author / Seth Gagnon

Seth Gagnon is a Middleware / Infrastructure / DevOps architect with more than 12 years’ experience in the field. Please visit sethgagnon.com for additional related articles. Connect with him on LinkedIn and Twitter.
October 6, 2016
Empower Developers to Build Security into DevOps

Over the past 10 years, as more organizations have made the transition to DevOps, there has been a cultural shift in which security and development teams agree quality is a priority. Yet, even with a shared quality focus, Development, Security and Operation teams still find themselves at odds on the road to continuous delivery of secure software.

Some people claim developers don’t care about security. The reality is they want to write great code that’s secure code, but often they don’t have access to tools that fit with the way they work.

There’s also tension between different sets of expectations and performance metrics for different teams. Developers are graded on their ability to deliver features and functionality to market as quickly as possible, while security engineers focus on prevention and detection. Operations teams, meanwhile, focus on reliability, uptime and stability.

Development managers feel pressure from senior management to avoid project delays. At the same time, security managers aren’t rewarded for reducing risk and may be asked to turn a blind eye and give the application a temporary pass so the project can move forward.

Prioritizing delivery to customers over security creates real dangers for the business. Veracode has found that 63 percent of internally developed applications are out of compliance with the Open Web Application Security Project (OWASP)’s Top 10 standards when initially assessed for security. And, attackers increasingly target the application layer, with web application attacks now the most frequent pattern in confirmed breaches, according to the 2016 Verizon Data Breach Investigations Report.

Empowerment = Security

Regardless of where your team is in the DevOps journey, you’re likely aware that detecting and fixing quality issues as early in the software development life cycle (SDLC) as possible increases efficiency and reduces costs. According to the Puppet 2016 State of DevOps Report, “high-performing development teams spend 50 percet less time remediating security issues” when they address security throughout the SDLC, instead of “retrofitting security at the end.”

While you want to find and fix bugs early in the SDLC, early detection of coding flaws has the unintended consequence of distorting an application’s risk posture. Development is once again in the hot seat, despite trying to do the right thing and testing new versions of an application early in the life cycle.

There is a middle path that can allow development to deliver more secure code at DevOps speed, but it requires automating security into the SDLC and into a continuous integration or continuous deployment (CI/CD) pipeline. Integrating security into the pipeline ensures that security testing happens with every release, and avoids the problem of bolting on application security late in the process.

This method has the ability to scale easily as more applications are developed. The release engineer adds a step in the CI/CD build workflow for an application security scan from the master branch. The assessment can take place in parallel to the CI/CD workflow process and any security issues can be loaded automatically into the defect tracking system that development teams already are utilizing for functional testing. Security issues are prioritized and resourced just as any quality issue.

The best way to catch software defects quickly is to analyze code as close to the time that it was written as possible. Security assessment, such as functional testing, “follows the code” through the application life cycle. During the development phase, developers analyze code in the IDE and develop unit tests that will find issues relative to their individual code changes. At this phase, developers receive immediate feedback while coding a feature, allowing developers to remediate basic issues when they are least expensive to fix.

During the build and integration phases, when a development or feature branch is merged into the master branch, compilation and integration tests are executed automatically by the CI/CD workflow. Issues detected at this stage are visible to the larger development team. No developer wants to be the one who “breaks the build” or has a basic defect that should have been detected during development phase. The integration testing is critical because functional and security tests are able to detect issues resulting when data and control flows between components. Coverage of the application expands and developers still have the time to fix before code goes moves to the next phase.

Enter Infrastructure as Code

“Infrastructure as code” (IAC)—the concept of managing, provisioning and configuration of development, QA, pre-production and production environments in a repeatable automated methodology—provides consistency that previously was challenging when each system was provisioned and configured independently. Product teams using virtual machines (VMs) or containers are able to advantage of this model more easily.

Even for those product teams that cannot take advantage of IAC, teams can still automate the deployment and configuration of the application. This becomes particularly important during systems/pre-production testing, where finding issues takes longer to remediate and the complexity increases as the application integrates with other systems. At this point in the life cycle, system and pre-production tests are automated when possible, and QA engineers and user acceptance testers may run end-to-end tests. Running security assessments at this stage is just as important as functional tests. True, fixing issues at this stage will be more costly to fix than during the development phase. But by this point the developers have been incorporating quality throughout the process and the number of high-priority issues will be significantly reduced.

Empowering developers to find and fix quality issues earlier in the life cycle provides them with the knowledge and skill set to more efficiently fix defects where it is less expensive and disruptive to their coding.

About the Author / Janet Worthington

Janet Worthington is senior product manager at Veracode, working on solutions to help developers and development teams smoothly incorporate security into the application development life cycle. Janet joined Veracode in 2012 as senior program manager, delivering Veracode’s secure development solutions to Fortune 100 companies. Prior to joining Veracode, she led software quality assurance test teams at a number of startup technology companies. Janet has over 19 years of experience in software product development and services. Connect with her on LinkedIn and Twitter.

September 26, 2016
Demystifying Scrum for the Ops Crowd
Scrum is, by far, the most popular way development teams are organized and work. In fact, well over 70 percent of Agile teams around the world use Scrum. But for those unfamiliar with the structure, Scrum may look like a strange collection of rituals and oddly named designations, such as the “Scrum Master.”

However, behind that collection of ceremonies, roles and rituals, Scrum is a huge opportunity for Operations teams to streamline development and delivery, improve productivity and create a strong sense of team within the group. Not only does Scrum encourage DevOps, but because it is empirical, risk and visibility are first-class citizens.

Scrum Basics, and More

For those wondering how to get started, here are eight things that every operations professional needs to know about Scrum:
1. Scrum is not focused solely on developing software; it’s about delivering software. The Agile Manifesto describes it as “working software” and Scrum describes it as being “Done,” but ultimately, Agile encourages teams to deliver working software to production frequently. With that in mind, it becomes clear that Scrum is not just about software development, but includes everyone involved in delivering working software to customers.
2. Definition of “Done” is a DevOps artifact. “Done” describes what it means to be finished with a product backlog item (PBI). Ideally, it describes all the steps necessary to release the PBI to production, or highlights the gap between what the Scrum team thinks Done is and Operations’ idea of what it means to be complete with a piece of work. Operations professionals need to get involved in the formation of the definition of Done (DoD) and work with the team to both improve its reach and validate its use.
3. The increment should be integrated and deployable. In Scum-speak, the increment is the working software the team produces during a Sprint (a set time frame, from start to Done). If multiple Scrum teams are working on the same product, then it is crucial that the increment be integrated across those Scrum teams. And, when I say integrated, I really mean integrated—as in, tested with processes and practices that are consistent with delivery to production. Thus, the increment is “ready to go.” By regularly integrating the increment, many of the problems that historically have plagued Operations are removed, as teams are, by design, forced into falling in sync much earlier in the life cycle.
4. The Sprint Review is not a phase gate to production. When moving from waterfall to Scrum, many organizations confuse the Sprint Review with a phase gate to move to production. Rather than being a phase gate for production—or the next phase of the process—the Sprint Review is a mechanism for the team and stakeholders to inspect and adapt at the boundary of the sprint. Ideally, the Sprint Review is executed on software that is being used by real users, in production. This increases the value of the review, as the team and stakeholders get to inspect how real people are using the software.
5. Continuous integration and continuous delivery automate Done. At the heart of Scrum is the idea that teams deliver working software in small batches to better understand the risks, requirements and solution. Adding delivery to continuous integration (CI) ensures they not only are integrating the software, but also formalizing the deployment processes to ensure there are no surprises. This is a great place for Operations to get involved and help ensure that aspect of Done is consistent with how it views delivering software. This might mean additional steps—or at the very least—a better understanding of operational versus development systems.
6. Scrum is an empirical process which helps reduce risk. The idea of delivering small chunks of functionality to support learning is at the heart of Scrum. Ultimately, Operations cares about risk. By delivering software in small batches, it is possible to surface risks early and mitigate them. By frequently delivering software, Scrum teams demonstrate they understand the processes necessary to deliver software, thus removing friction from traditional release processes. However, all of these benefits are undermined if the team does not use “production ready” deployment processes or include risk as part of their prioritization of what to work on next.
7. The Scrum Masters are Operations’ friends. The role of the Scrum Master is to both protect the team from outside distractions and help them to identify and remove impediments. To deliver Done, the team needs to work effectively with Operations and thus, the Scrum Master often will be the person who helps build those relationships. By working closely with the Scrum Master, Operations professionals will gain better insight into what the Scrum team will need from them and how they can deliver Done software.
8. Helping the Product Owner understand risk. The Product Owner (PO) manages the backlog, which is the place where all work exists. This means the PO is prioritizing the backlog and deciding on what should be done next. Normally, this is accomplished by looking at business value and negotiating with the Development team to determine what can be most effectively completed next. Operations has a great understanding of risk and, by working with the product owner, can add that context to the backlog, allowing the PO to make informed decisions and about value and risk.
Ultimately, Scrum provides a great framework to build bridges between Operations and Development and can act as a catalyst to drive fundamental improvements to the way teams collaborate and deliver business value. However, both sides need to stop hiding behind the “mystery of agile” to avoid each other and embrace the desire to break down things into small chunks, incrementally manage risk and get to repeatable Done software as fast as possible.

About the Author / Dave West

Dave West is the product owner at scrum.org. He is a frequent keynote speaker and is a widely published author of articles, along with his acclaimed book, “Head First Object-Oriented Analysis and Design.” He led the development of the Rational Unified Process (RUP) and then worked with Ivar Jacobson running the North American business for IJI. Then managed the software delivery practice at Forrester research where he was VP and research director. Prior to joining Scrum.org he was Chief Product Officer at Tasktop where he was responsible for product management, engineering and architecture. Follow him on Twitter.

— Dave West
September 23, 2016