I always come across a standard question from testers in any DevOps related presentation. “Will DevOps remove the need for Testers?” A simple answer is “No”. DevOps is not going to replace Testing as a discipline. Firms implementing DevOps have a vision for better product quality workable by brilliant Testing.
DevOps is an enabler for the following.
Reduction in IT Budgets
Efficient end-to-end delivery
Lean delivery framework
Working end-to-end automation framework
Successful businesses today are design thinking driven. Businesses have a strong sense of urgency, aided by innovative software engineering practices. Businesses today need to have a working continuous delivery practice in place. Continuous delivery also mandates better product quality and zero touch end-to-end delivery. Automaton is key to success across DevOps processes. Delivery by co-located, integrated teams using innovative tools prevents collaboration and communication issues.
Manual errors, Stove-piped communication issues get sorted with a good DevOps Governance model.
DevOps mandates “Fail Fast, Fail Often” driven by “Test First” concept. This means that testing is a continuous process and more extensive in nature. Testing allows for cohesiveness between various teams, automation of everything possible. Testing also allows for continuous improvement in test process delivered via iterations. Intelligent test automation, improved regression, continuous test augmentation are some benefits delivered by DevOps.
Key benefit delivered byDevOps is the cost reduction in end-to-end delivery.Businesses achieve cost reduction by Automation, manual error reduction and reduction of duplicate effort.
Continuous delivery enables streamlined governance and reduces time-to-market.
With fail-first, fail-fast, fail-often philosophy, shift-left testing is a reality. With early defect identification, product quality increases. With increased automation, continuous testing is a feasibility.
With co-location, productivity of DevOps teams (design, test, operations) improves.
Implementing DevOps could be tricky in organizations having monolithic systems and processes. Culture change is critical for successful initiation of DevOps and Continuous Delivery.
From a testing standpoint, DevOps addresses the following key testing challenges.
Test Environment – Virtualization, Automated Deployment, Automated build and configuration management, data refresh and maintenance becomes simple
Continuous Integration, Automation and Verification – Test First centric test driven development, behavior driven development concepts allow test processes to improve in a DevOps Model
As per a report published in 2002 by NIST – Cost of Poor Quality (in US) was $59.5 Billion per year. Cost of poor quality is higher in large organizations running legacy infrastructure. By some industry guesstimates average cost of a production bug is about $7000. Customer experience and agility for quick delivery takes a hit if the organizations aren’t quicker. DevOps becomes a positive catalyst here.
Some skeptics may say “DevOps looks easy on paper but difficult to put in place”. Any innovative change is difficult to implement at first. It takes discipline and drive to run the operations watertight to achieve the goals.
Some key factors to implement for Successful DevOps are
Cohesive, One Team mentality between teams
Implementation and Use of Tools that take away manual effort
Use of tools is key to success of a DevOps program. Plethora of tools are available to support automation. Tools are available for Environment Build, System Configuration, Application Build, Database Setup, Artifact/Document Generation, Testing, Requirement Management, Collaboration, Production Monitoring, and Reporting across DevOps activities.
Like Agile Delivery Programs DevOps needs a strong Team Cohesion to be successful. A DevOps team need to have a Product Life-cycle Management mindset as well. This also means that team needs to unlearn standard SDLC processes. This means, DevOps team should have a clear vision. Vision to Ideate, Implement, and improvise and retire the product. An integrated DevOps team with clear vision will make DevOps projects successful.
Conclusion:
DevOps testing needs to address the need of Business and Technology needs. Automation plays a major role to achieve this aim. DevOps in principle has a goal for better product quality albeit the need for faster delivery. Terms such as “Shift-Left”, “Fail-First, Fail-Fast, Fail-Often” will be regular vocabulary in DevOps. DevOps Test team comprises of Cross Functional Specialists and Testing Purists. A Performing Testing Practice for DevOps will take some time. Testing as a practice has matured over the past two decades and is well equipped to support DevOps. Use of automation and productivity enhancement tools helps testing move faster.
Successful DevOps implementation in large firms with legacy processes and technology is a challenge. A big culture change is the only way to achieve success in such firms. It will be a hybrid DevOps Model as purist implementation will be a challenge.
As wider adoption of DevOps methodologies continues to foster demand for new tooling to help streamline evolving workflows, at least one vendor is attempting to address the growing need for related monitoring capabilities.
Librato, a San Francisco-based provider of Amazon AWS applications performance monitoring (APM) services, claims that it is finding favor among the rapidly expanding DevOps community driven by its ability to inject critical visibility into the engineering lifecycle.
Acquired by Austin, Texas-based SolarWinds in January for a lofty $40 million and renamed Librato SolarWinds Cloud, the company is touting its recent adoption by well-known providers such as Slack for monitoring of DevOps-centric applications performance and health metrics.
“At its core, DevOps is destroying conventional wisdom around ritualized processes and the need to perform them, mostly in the domain of scale,” said Dave Josephsen, Developer Evangelist at Librato. “By attempting to include monitoring as a key element of the development lifecycle, this has created a whole new range of challenges, and opportunities.”
By providing a centralized Web interface and open API to help create visibility into critical applications engineering requirements, Librato maintains that it is creating the tooling necessary to empower integrated development and deployment processes, such as continuous change control.
“There are a lot of disparate tools for monitoring individual engineering tasks, but there hasn’t been any practical way to bubble up that data for efficient monitoring,” Josephsen said. “Within the context of DevOps, change control is a ritual that we’ve abandoned as we’ve embraced test driven development and automated away the human element, so there’s a distinct need for monitoring of feedback and metrics.”
For example, cloud-based productivity messaging specialists Slack are using the Librato “StatsD” lightweight daemon process to stream business and app-health metrics regarding PHP applications into the monitoring platform. Slack also leverages Librato to chart data regarding connect, query and fetch latencies for its server clusters, and to monitor internal metrics about overall system health.
“Having those graphs side-by-side lets us clearly see and compare the effects of network latency on one of our new services, and that’s very cool,” Richard Crowley, director of operations at Slack, wrote in a recent testimonial posted to the Librato Solarwinds blog.
Industry experts agree that broader adoption of DevOps strategies will continue to drive demand for relevant monitoring tools, with solutions such as Librato seeking to deliver “modern stack architecture for looking at the modern stack.”
“I think that APM is always interesting because it’s always new,” said James Governor, industry analyst with RedMonk. “The pace with which we’re building new architectures to deliver services to end users is increasing all the time, and with every new wave of technology, we tend to get new and associated models of APM tooling.”
Librato is taking advantage of the need for a system that brings together the elements necessary for cloud native monitoring, and in an agentless manner different from other competing tools such as New Relic, the analyst noted.
However, whether or not the platform is able to break down traditional engineering siloes to enable more efficient DevOps management, as Librato officials claim, remains to be seen, he said.
“Each wave of technology has promised to do this, to remove the inherent finger pointing, and each has always made the same claim to break down organizational firewalls,” Governor said. “DevOps hasn’t boiled the ocean in this manner yet, but there are some unique opportunities; unlike previous waves, where automation was intended to replace humans, this is now about augmenting people, and we are seeing that teams are choosing new tools from which developers can truly derive value.”
People want to get going with DevOps or Continuous Delivery, but need a place to start. Others are already on their way, but need some validation of their choices. In April, I published the first volume of DevOps and Continuous Delivery reference architectures which has now been viewed over 37,000 times on SlideShare (it’s free to download…no registration required). Three things helped people in the deck: (1) the reference architectures, (2) links to the sources for each architecture attached to each slide, and (3) no marketing fluff.
I am honored to have helped so many people on their DevOps and Continuous Delivery journeys. So this month, I compiled Volume 2 of the reference architectures. You can now view the deck on SlideShare, download it (again…no registration required), and share it with others. Have a look and let me know what you think.
As I did with Volume 1, if you have a reference architecture that you would like to share, I would be happy to add it into Volume 2 with full attribution to you. Just ping me in the comments section or reach out to me @weekstweets.
Part 2: Continuous delivery tool chain efficiency > Outsourcing savings
(Check out Part 1 of the series Will DevOps Kill IT Outsourcing?)
Experts on digital disruption and DevOps practices are vocal in their belief that DevOps and traditional outsourcing models don’t mix well.
“Adopting DevOps requires organizations to erase the barriers between development and operations teams,” says Dan Cornell, CTO of Denim Group. “This is hard enough to do within a single organization and nearly impossible to do across organization boundaries.”
If push came to shove and an organization needed to justify an either-or decision for DevOps over outsourcing, the argument would likely revolve largely around business value. Continuous delivery makes it possible to move to market more quickly, develop products better aligned to customer sentiment and kick the pants out of competitors who don’t do the same. That was the impetus behind the shift at GE Capital. Last year the firm’s CIO Jim Fowler said he was looking to significantly reduce IT outsourcing and shift toward more Agile teams. Earlier this summer Matt Merchant, former global managing director of DevOps for GE Capital gave devopsy.kinsta.cloud a snapshot into how this is driving innovation. (Ed. Note: Merchant suddenly passed on this summer. Our interview was just prior to this tragic event.) He explained how the company was able shift customer-facing mobile app development from a six- to eight-week release cycle to twice weekly.
But the decision to dump outsourcing in favor of supporting DevOps practices within a more centralized IT group doesn’t just have topline revenue and innovation benefits. The upside presents itself in the bottom line as well.
In many ways, we can chalk that up to the fact that the automation and cloud technology that facilitates continuous delivery is light years ahead of where it was when belt-tightening outsourcing deals crescendoed following the 2008 financial crisis. As things stand, the cloud has enabled IT to decouple infrastructure outsourcing from operational and process outsourcing.
“Companies today realize they can achieve many of the cost efficiencies that outsourcing was originally intended to achieve by leveraging cloud technologies,” says Thomas Enochs, vice president of customer success at Chef . “There is a general trend towards leveraging cloud services rather than cheap labor to achieve the scale, speed, and flexibility they desire.”
The fact is that many an outsourcing deal was born in a day when the tools of development were slow to deploy and expensive.
“DevOps practices and modern software tools have drastically reduced the time and cost required to develop custom applications,” says Bernard Sanders, CTO of CloudBolt Software. “In the early 2000s, the step of just getting the servers and infrastructure set up for a new app could be a multi-week or -month chore.”
Virtualization vendors, cloud providers and self-service automation have all made it possible for developers to get easy and affordable access to the infrastructure they need when they need it, Sanders says. And then there’s the advances in deployment pipeline automation that are seriously streamlining hand-offs over the course of the software development lifecycle. It’s all adding up to added efficiency and reduced cost of development and deployment.
“In the future, the overhead required to build custom apps will be driven to near zero, and enterprises will be able to in-source this development, retaining this function that has become key to enterprises’ brand identity, product offerings, and customer experience,” Sanders says.
Even now, though, the DevOps toolchain is completely blowing up a lot of the financial models that initially drove IT departments to aggressively outsource. While many outsourcing deals continue to be renewed without reviewing old assumptions, this is starting to change.
“We have seen cases where outsourced functions like automating test or other processes has been brought back in house because equipped with the right approach and tooling, the internal team could achieve better results, but also because the automation work was a key building-block to a longer-term DevOps plan,” says Alex Henthorn-Iwane of QualiSystems.
To build the system to automate the build and deployment (continuous deployment) process there are few main areas which need to be covered. Following are our prerequisites.
To describe above here are the main ingredients – Configuration Management – continuous integration/ Automated Build – Automated Deployment – Automated testing
Configuration management Very fist step to align your development practice is to select the right source control tool. Once you have the tool then you need to think of branching policy to make sure the starting point of the workflow is well defined and accepted by developers during their development phase. There could be lot of process around merging, branching code review etc which needs to be address and policy driven. Some key points could be – Developments and release branch should be separate. – Each change goes qualifies for next stage only if reviewed. – Code quality should not go down.
Following are a few options Source control tool – SVN (Free) – CVS (Free) – Git (Free) -Perforce (Paid) -Clearcase (Paid)
Build system Team needs to have robust and automated build system which can take care of all our needs (build, analysis, deployment etc). – Need to have automated build system which can keep building the source code from various branches to make sure we catch the broken piece fast. – Build can run the unit test and provide report – Build can check the health of the code by running findbugs and checkstyle (if any).
Above checks can verify the heath and quality of the code (static analysis). Nothing should go beyond this point if any of the above is not matching with our criteria.
Till this point we have reached to qualify the minimum criteria for continuous Integration. Here are few tools which comes under Build system Build Tools – Ant – Maven – Gradle – Make CI tool – Jenkins – Cruise control – Luntbuild -Bamboo Repository management – Nexus – Artifactory (Above are not open source but they are free to use with limited features). Code quality check – Findbugs – jacoco – cobertura
**Deployments **
Deployments are the next big bottleneck for continuous deployment. We need to have a deployment tool to move ahead, – configuration management tool like puppet or chef – Private cloud management (Openstack) – Third party tool like xl deploy, Udeploy, Opsworks etc..
Deployment strategy can be chosen on the basis of our need and the tool we are using, most of the plugins are available in jenkins (or bamboo) to make tight integration of build and deployment.
(At this stage I am not considering the dynamic provisioning of hardware in this section).
Automated testing
Once deployment phase is over QA regression suite can be triggered via jenkins (or any other tool) and entire reports can be published to central QA metrics board. This this point we can even use jacoco/cobertura to provide functional code coverage(for Java code) to demonstrate how much % of code QA regression/automation is covering and what is the level of automation. Similar way job can be setup to raise jira/bug ticket to file issues found during any of the above step.
Finally we can have last step which can perform the final release build and push to staging /production in similar fashion.
Resource Provisioning
Developers/QA can just get the hardware either from infra team or if auto provisioning is enabled they can setup the environment and use the jenkins to perform build+deployment+QA If we talk about automatic scaling of environment, we can use AWS for same. Jenkins has AWS plugin to launch instance, This can be setup to launch instance dynamically and run the application and perform the QA Automation. Once need is over aws instance can be terminated automatically.
In my previous post, “When Good Code Goes Bad“, I shared new research showing the average large development organization consumes over 15,000 known vulnerable and defective components annually. While we can’t stop software from going bad, there are practices from traditional manufacturers that we can use to improve our ability to recall and fix the “bad” software components.
The Software BOM
A Bill of Materials (BOM) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product, a “software bill of materials“ (BOM) is an inventory of the third party and open source components used to build an application.
A sample software bill of materials lists all open source and third party components, including their version, age, license type, and any known security vulnerabilities.
As noted in Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.
“A software BOM is useful both to the development organization (manufacturer) and the buyer (customer) of a software product. Builders often leverage available open source and third-party software components to create a product; a software BOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities. Buyers can use a software BOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Understanding the supply chain of software, obtaining a software BOM, and using it to analyze known vulnerabilities are crucial in managing risk.”
A software bill of materials not only inventories what is used, but in some cases it also syncs with real-time component defect data to indicate which components have known vulnerabilities or license risks. Various software supply chain automation tools expand the bill of software much further, alerting stakeholders automatically when a defect alert occurs. In advanced tools, the entire software lifecycle is automated to ensure that defective components are avoided and continuous monitoring instantly identifies newly announced vulnerabilities as soon as they are discovered.
Breaking Bad
As mentioned in my previous post, When Good Code Goes Bad, there is no way to prevent software from “going bad”. No one is immune from these issues. Therefore, we need to focus on our responses to these issues.
To paraphrase Vince Lombardi, “It’s not how you get knocked down, it’s how you get up”. Following the OpenSSL vulnerability announcement in 2014 (or pick any other new security vulnerability with or without an assigned logo), every development and IT operations team on the planet was asking two questions:
Did we ever use that open source component?
if yes, where is it?
For those organizations tracking their use of components with a software bill of materials, the answer was easy to find. For other organizations, weeks and often months of research were required. To get a sense of the lack of maturity around traceability practices, take a look at this image from Sonatype research showing announcements of OpenSSL vulnerabilities in products following the initial vulnerability disclosure in April 2014.
Automating Traceability Across the Software Supply Chain
Software supply chain complexities, coupled with the volume and velocity of open source and third-party components flowing through them, are forcing us to think differently about our current practices across development and operations. We are not the first to face these challenges and we can learn a great deal from the mature practices of other high-velocity manufacturing organizations and industries.
One thing is certainly clear. Manual practices of monitoring or occasional checkpoints cannot keep pace with volume and velocity of components being consumed. And if you extend these considerations beyond just open source components and into containers, microservices, or other images used in development and operations, the problem is further out of reach for manual processes to keep pace. The volume and velocity within our software supply chains will not diminish—and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.
But the answer is here today: automation. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality. Solutions that facilitate comprehensive software supply chain automation are poised to usher in the next wave in development productivity—with gains on par or even greater than possible with agile, Lean, and DevOps.
More information on the current state of software supply chain practices as well as examples of best practices being employed by software development teams, can be found in the 2015 State of the Software Supply Chain Report.
Milk spoils. Iron rusts. And in software, good code goes bad. Yet the difference is, with the first two, you know the change has occurred. With software, those changes are not always obvious.
Your 5,100 Binaries Went Bad
There is no way to prevent software from “going bad”. As with all products, bugs and defects,are bound to happen at some point. No one and no code is immune from these issues. But who’s looking for the 5,100 software components in your organization that went bad last year (meaning new security vulnerabilities were discovered in them)? It’s all too likely, no one.
Earlier this year, I took a deep dive into the analysis of software supply chains that fuel high velocity development practices and IT operations. The analysis revealed that some of the largest development organizations were consuming an average of 240,000 open source components to expedite development, accelerate innovation, and meet time to release deadlines. While reaping huge benefits from this consumption, these same organizations were unintentionally growing their technical and security debt. How you ask?
Here is a summary of the their average debt:
Source: 2015 State of the Software Supply Chain Report
The analysis showed that over 15,000 (7.5%) of the open source components being consumed by these organizations in 2014 had known security vulnerabilities. Of those 15,000 components, an average of 66% (9,900) had known vulnerabilities dated 2013 or older. That means, they were known vulnerable components (“bad”) before they were downloaded.
The remaining 34% (approx. 5,100) might have actually been “good” components at the time they were downloaded by development teams from public open source repositories, but at some time during 2014 a new security vulnerability was discovered and a CVE identifier was assigned.
Even with the best controls in place to protect developers from downloading open source components with known vulnerabilities, the fact that new vulnerabilities are continuously discovered — about 50 new CVEs are announced daily — means that continuous integration and delivery practices also need to be complemented by continuous governance and visibility practices.
We’re Not Alone
Software development is not the only industry facing such issues of things good things going bad. I bet all of us have received notification from our car’s manufacturer that a certain part is being recalled. While the part was known to be safe when the manufacturer sourced it from suppliers, placed it on the car during assembly and shipped it, discoveries of defects are often found years after the vehicle has been on the road. The same scenario has been played over and over in the food, pharmaceutical, defense, toy, electronics, and other industries.
The difference is those other industries are known to have more mature supply chain practices where they can track and trace every part or ingredient ever used in every product they have shipped. And when a part goes bad, they can recall and repair that defective component.
In software, track and trace practices are still in the earliest stages of maturity. In a 2014 survey of 3,300 development professionals, 63% stated that their organizations have incomplete or no records of the components used in their applications. This lack of traceability means that even when good components go bad, development and IT operations teams have no way to quickly find the defective component and replace it with a better, safer version — dramatically extending mean times to repair, or in the case of no action, growing their technical and security debt. And when applications with those defective components are in the hands of customers, the ability to “recall” the impacted software or distribute newer versions of the application are hindered.
Did he recommend a “Bomb”?
A Bill of Materials (BOM; pronounced “bomb”) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product. According to Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.”
In Part 2 of this blog, “The Software BOM Squad“, I will introduce the importance of automating the creation of a “software bill of materials“ (Software BOM) to take inventory of the third party and open source components used to build an application. When software components age more like milk than wine, a software bill of materials can help you quickly identify where your 5,100 “bad” component were used.
Continuous delivery (CD) of applications is rapidly becoming a differentiator in this application economy. CloudBees, the enterprise Jenkins company, recently announced the availability of the CloudBees Jenkins Platform to help organizations adopt CD. This new enterprise offering from CloudBees brings CD to the masses with the new Team Edition, while enabling enterprise CD deployments powered by Jenkins at scale with the Enterprise Edition. At the same time, the Jenkins open source community, in conjunction with CloudBees, released six new plugins to support the use of Docker containers within Jenkins-powered CD and CI processes. Docker is powerful new technology that allows teams focused on delivering software to standardize their development and production application environments for reduced risk and accelerated application delivery. Leveraging Docker containers in a CD process managed by Jenkins is the ultimate way to deliver better software faster.
Join Tracy from CloudBees as she talks about Cloudbees’ new enterprise offering and the changes happening in the Jenkins community to provide first class support for Docker in the creation of CD pipelines. In this webinar, Tracy will introduce features in the latest release of the CloudBees Jenkins Platform to
help manage Jenkins’ across the organization and walk through six common delivery patterns with Jenkins and Docker. Tracy will then tie it all together with a demo of an end-to-end delivery pipeline with Docker.
Continuous delivery with Jenkins and application containerization with Docker are rapidly becoming two of the foundations of the DevOps movement. Check out this webinar to learn how you can adopt these new tools and techniques in your own DevOps transformation.
Alan Shimel, Editor-in-Chief devopsy.kinsta.cloud, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.
About the Panelists
Tracy Kennedy is an associate product manager for CloudBees and is based in Richmond, Virginia. Prior to her current role, Tracy has worked at CloudBees as a solutions architect and a developer success engineer. In her free time, she enjoys rock climbing, grand strategy games, and reading. Read more about Tracy in her Meet the Bees blog post and follow her on Twitter.
Udaypal Aarkoti is the Director of Solution Architecture at CloudBees. He believes in complete end to end automation and that continuous delivery is the future of application delivery. Prior to CloudBees he lead the middleware solution architecture team at Red Hat.
When you start to implement a new methodology, it’s very easy to get bogged down by the specifics and lose sight of the overall goal. With Continuous Delivery, we see endless debate about whether to use Puppet, Chef, or Salt for deployment, or discussion about how to build a CD pipeline with containers. These are just technical challenges to overcome. They’re implementation details that have little to do with why CD is such a powerful choice for businesses.
What’s the most important metric for your software? It’s not the speed of delivery, the number of features, the performance under load, or the defect count. It’s end user satisfaction. The question you need to ask and keep asking is – How well does your software cater for the people using it?
The CD mindset keeps you laser-focused on that metric in two very important ways.
Real representation for the end user
Too many organizations are still just paying lip service to this concept. The developers and testers working on a piece of software often have no contact with end users, and that can mean that they have very little idea about their needs. If you really want to create a great piece of software for someone, then you have to understand how they work. What are their requirements and constraints?
Traditionally, software development has been siloed. Individual departments have focused on their own tasks first. Developers may be delighted with their clean repository layout. QA may be satisfied with how complete the test coverage is. But neither of these things tells you about the end user experience. The system may be completely unusable.
Personal contact is a great idea. If your team actually talks directly to end users, asks questions, and learns to empathize, then you’ll see the benefit in the product.
CD also encourages greater visibility. Not just oversight for management, but for the whole team. Everyone, from development, through QA and operations, to the actual end users should get a look at how the overall service is being put together and delivered. Have channels in place so that anyone can contribute to service improvements, regardless of their area or expertise. Greater visibility over the process, and an ability to influence it, makes for a really cohesive, motivated team.
Collecting, analyzing, and acting on the right data
You can empower everyone on the team to make suggestions and you can say that end user satisfaction is paramount, but without collecting the right data you aren’t going to be able to realize the benefits of this approach. You need help understanding where the opportunities for improvement are, and it’s vital to measure the impact of what you’re doing.
In the big data age, it’s common for companies to collect all sorts of statistics. It’s easy to automate the capture of technical operational details such as CPU load or memory usage, but how much do they really tell you? Though it may be more challenging to find out whether your end users are satisfied, it’s also a lot more important.
Eliminate the risks of educated guesses and gut feelings. Apply a solid methodology to improve your service based on user feedback, just as you hone technical performance based on technical data. You don’t need to spend hours brainstorming new features. You can analyze end user interaction, find a pain point, create a hypothesis about the cause, tweak the system to address it, deploy the change, and measure how it really performs.
Adopting this process and repeating it over and over will improve your service. Because you’re rolling out small changes, it can be done quickly and efficiently, with easy rollback if required. As long as you’re careful to measure your hypothesis and the effectiveness of your solution each time, there’s no downside.
The ideas here are nothing new. Your marketing department most likely already conducts A/B testing and measures user interaction. But how many companies are really taking the holistic approach, applying these concepts also to software development and delivery?
To make the end user the focus, and really leverage the potential of Continuous Delivery, the whole team has to pull in the same direction. Agile and CD methodologies help you cut down barriers, set shared goals, and keep the focus firmly on the customer to revolutionize the way software is delivered for the people who need it.
About the Author/Andrew Phillips
Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
In the application economy, you have to deliver software as if your business depends on it … because it does! It’s a bold statement, but true. Think of nearly every interaction a person might have today—be it for work, commerce or play. Most have a digital dimension, which relies on digital technology and platforms.
In order to survive and grow, every company needs to become a technology company and every business needs to become a digital business. Successful transformation requires Continuous Delivery, the new business imperative that enables you to rapidly develop and deliver applications that drive superior user experiences and engage your customers and staff.
But the traditional “software factory” or process for transforming an idea into a customer experience is throttled by a number of bottlenecks in the delivery pipeline. That means delivering innovative, high-quality applications, faster and more frequently can be a chaotic and complex process, particularly if your application delivery systems and processes were designed to only push out one or two releases a year.
Join moderator Alan Shimel, devopsy.kinsta.cloud; and experts Georg von Sperling, DevOps and Technology Evangelist at CA Technologies and Julie Craig, Research Director at Enterprise Management Associates (EMA) on July 21, 2015 to discuss ways to overcome today’s application economy challenges with solutions designed to transform your software delivery lifecycle into a continuous delivery pipeline of digital innovation.
Recorded: Tuesday, July 21, 2015
Recording
Q&A
1. Based on your experience tell me : continuous delivery versus traditional development. The continuous delivery model is perfect for every business ?
GvS: As a goal, yes, absolutely. Because continuous delivery practices enhance what is already being done and simply promote good behavior over creation of waste. Removing bottlenecks to increase quality, frequency and velocity simply makes sense, regardless of the business you are in. Now, that said, continuous delivery is not necessarily perfect for every business if you define it as continuously delivering to production. But what development organization would shun away from getting early and amplified feedback in User Acceptance Test (UAT)? To accomplish this, you still require every one of the concepts in continuous delivery.
ANBU: If you think of achieving Continuous delivery as a means to achieve modularity & lose-coupling in order to achieve higher workforce productivity, every business will be motivated to do it. As a concept, it is perfect for every business, but the objectives of the process and their order of their importance will vary industry to industry. As long as you make these adjustments, you will benefit a lot.
2. One of our struggles is the impact of Continuous Delivery on our Environment Strategy and the Mechanics used to maintain them. For example, applications dependent on other applications / components will require those other applications / components in the environment infrastructure in which testing is occurring. How do you see Compnay Environment Strategies changing to support Continuous Delivery? When does it become necessary to change an application’s architecture to reduce or remove these dependencies? Or are there testing methodologies that can accurately test changes without the need for large QA Environments?
GvS: The path with the least resistance is seemingly learned from the airplane manufacturing industry. When there is a new wing design that is required to undergo quality validation, the dependent systems and datasets are so complex, they could not have ever released those designs if they threw up planes into the air with different passenger or luggage loads (data) in different weather conditions (environment) attached to a full airplane body (dependencies). Likely, this would have resulted in the same adverse effects that we experience when pushing a digital asset from our IDE to the first environment – often catastrophic. Virtualizing dependent services with actual behavior – just like in a wind tunnel – is a good approach. This also allows discovery of the dependencies in consumable, shareable format which, should you find additional needs to revisit the architecture, actually aid in the process of redesign.
ANBU: I would recommend a model-based approach in your situation. In your stated example, if you model ‘infra-as-code’ and try to achieve modularity at that level, so that developers and testers call it out during their plan/design/development process to make their ‘availability and use’ defined, the problem can be solved. It requires an assessment on all the functional components (infra included) in order to model it. There are tools out there that would help you achieve such a modeling.
3. How do you measure the success of DevOps other than faster & quality releases? are there any metrics that can be collected periodically?
GvS: In the context of Continuous Delivery, there are many metrics that can and should be captured. Many more in the context of DevOps. We could likely devote an entire session on this. Let me focus, however, on continuous delivery metrics. On top of focusing on business-oriented outcomes, much like in DevOps, some operational metrics could be captured. Examples are:
– Lead time to production. Measuring cycle time from the work starts to the time the team’s definition of ‘done’.
– Defect resolution time. Really successful teams can fix defects as quickly as they are discovered. The more left-shifted the defect discovery is, the more quickly they are resolved.
– Production defect leakage. Measure both: new defects found and defects previously found and still made it to production.
– Defects. It is bad enough to ship buggy apps. Continuously doing so is even more terrible. Measure at each stage in the cycle, and notice a left-shift if CD is going well.
– Broken build time. The longer a build sits around broken, the commitment to quality is just not there.
– Production downtime during deployment. If you achieve zero-downtime, you can stop measuring here.
– Regression test duration. Teams with manual practices will measure in months or weeks. CD teams will likely measure in minutes or hours. Also include lead-time to regression test.
– Performance test outcomes. If you find tons of functional bugs in performance testing, there is likely a bottleneck not addressed early in the cycle.
– Touches. Number of tickets, emails or other communication required to do a deployment in each stage. If it isn’t “zero-touch”, there is improvement potential.
This is not a complete list, but certainly one to start with
ANBU: GVS’s response is complete.
4. How do you get legacy code/systems into the CD model? Or is CD only for new code/projects/systems?
GvS: The practices advocated with CD aren’t specific to the tenure of the project or the age of the technology. They are arguably all good practices and make perfect sense. Now, if you are an enterprise on the journey to adopt CD practices, the recommendation is certainly to begin with projects that have the highest business impact. A good heatmap analysis does go a long way to begin. If it turns out that it is your legacy code/system, so be it. Parallelize work streams, accelerate quality and automate & orchestrate the delivery pipe.
ANBU: Legacy code/systems are generally monolithic/tightly coupled. Hence the problem. Unfortunately, they are also part of the corporate string of applications, so we can’t stay away from them. We have seen examples where these systems stay as one ‘component/workstream’ and plug into the larger CD stream, creating a hybrid-model. Yes it is not perfect, but will help you implement CD. Additionally, the maintenance/re-engineering of these legacy systems are opportunities to change the code to adopt to the larger CD strategy that you put in place. If they are truly legacy, there won’t be major ‘patch’ work so they stay what they are.
5. How do you reconcile product management with CD, i.e., the definition of a story to encapsulate a deployable or atomic element that can be delivered? Often times stories are part of a larger epic that needs to be delivered as a whole and not on their own. so the issue is deploying atomic units/stories creates the fear of delivering partial functionality. How does a business deal with this and overcome this
GvS: If you pardon me, but it seems a misinterpretation of CD itself. Deployment and delivery are two, if not three, different things. I believe you may have latched onto the “live” deployment of an atomic change to production. Given the right strategy in modeling the pipeline itself, all CD elements are still valid, the atomic element may go “live” with many other atomic elements. BUT, unlike in the past, the fear of failure in that deployment is minimized as quality drastically improved and, the same, already tested, processes are utilized to set the atomic unit “live” as have been validated in lower environments. It is a journey.
ANBU: one of the benefits of CD is achieving the readiness of production deployment of (ALL or SELECT atomic units) on a push of a button. It does not force you (but you have the choice) to deploy every atomic unit into production If what you said “larger epic as a whole or none” is important, then your definition of the ‘select’ atomic units should correlate to it. Configuration mgt and release mgt should have proper ‘gates’ to make sure that they roll-out properly.
6. My question on metrics is because in legacy orgs it takes a lot more time & energy to get to CD, so measurement through metrics is critical.
GvS: Absolutely, metrics and measurement go hand-in-hand. Looking at CALMS (Culture, Automation, Lean, Measurement and Sharing) for guidance, influencing the organizational culture through measurement and sharing which both are derived from automation is the nirvana. Along with the will of the IT organization, there also has to be executive buy-in to the strategy. Yes, grass-roots is fine, but a lot can get accomplished if the company stands behind the strategy. Having defined positive business outcome metrics as part of that journey helps reinforce that postion.
ANBU: Agree with GVS.
7. What is the Quality as a result of continuous delivery? What is the tradeoff with respect to the Business impact to defect leakages in production?
GvS: Quality metrics should show drastic improvement if CD tenants are implemented. There was a question on metrics already. This is not about continuously deploying bad code, which really just makes things worse, but about continuously delivering, where quality is certainly a determining factor.
ANBU: You need to baseline ‘quality’ before and after CD. There is an absolutely no way, these measures should negatively trend, if you properly implement CD. Defect leakage in any model will have the same implications, just that in CD – the mean time to resolve (MTTR) these issues should be faster.
8. For a business to adopt a CD model, doesn’t all disciplines/functions have to change to suit the atomic delivery concepts.
GvS: Once nirvana is achieved, likely there was a change in all disciplines and functions. It is a journey that must begin. The imperative is there for the business. But, much discussed in the context of DevOps as well, CD is a journey of transformation – not something you do in a day.
ANBU: YES. this is a life-cycle process, hence the impact is for everybody. But you can start from one unit/organization and spread it out to evolve it. if you have the luxury, start a new project/unit with CD as the foundation.
9. How do you see ITIL and DevOps interacting to maximize the benefits of continues delivery?
GvS: There is a great amount of synergy between the two. And actually, how much easier would it be to have “Standard Changes” in the change management process, enrich a CMDB with actual data that isn’t stale, have service models up to date and have focused problem management.
ANBU: A well defined devops strategy can help you implement ITIL more effectively. Configuration management and release management are two particular areas where the ITIL can provide the maximum benefits to the CD strategy.
10. Jez Humble and David Farley suggest not using feature branching, because this incourages back room channel development, and to only use the master branch to do CD, but other companies in this space says feature branching is ideal because git-flow encourages this, is there a prefered strategy in your branching strategy to do CD. Is there a need to better define what is meant by ‘continuous delivery’? I am concerned with people stating that CD means code gets in untested (which is from my understanding an anti definition) and fully tested code is a part of CD.
GvS: I believe Jez’ argument, and I agree, is why it is essential to have a deployment pipeline that is triggered off trunk to ensure all necessary validations, including functional testing. A deployment pipeline is a pattern that organizations adopting CD use. The key is that the validations should be performed against a build off mainline. I also advocate a number of good approaches that help in the validation effort, such as automating the generation of functional tests and having the right data sets available for relevant validations.
ANBU: No comments.
11. Does continue delivery means continuous Deployment to production? For Horses, frequency of deployment is not too frequent but they would still need to have Continue delivery to achieve benefits of Test Automation, Agile Methodology etc.
GvS: I would argue no. Continuous delivery does not equate continuous deployment to production. If I may be so bold, I would state continuous delivery is a practice to allow deployment to production at any time with a high degree of quality and a low amount of risk with zero downtime and zero touch. A business leader recently said to me that whereas, in the past, “deployment to production was an IT decision, it is now a business decision at a push of a button”. But, CD certainly played a part in this paradigm shift, as the entire delivery process, including testing, is visible and predictable.
ANBU: Continuous Production deployment is not the ultimate goal of CD. It is to let you manage multiple workstreams in parallel as atomic units, accelerate quality through high degree of automation and provide an ability to learn/continuously improve the processes. By means of this, you make your changes ready to be deployed faster than you normally do. You should aim to achieve the frictionless automation across the life-cycle, not just in QA activities.
Your Host
Alan Shimel, Editor-in-Chief devopsy.kinsta.cloud, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.
About the Panelists
Julie Craig, Research Director at Enterprise Management Associates (EMA)
At EMA, Julie’s focus areas are Application Management, public and hybrid Cloud, Integration Technologies, DevOps/Continuous Delivery, and Application Performance (APM). Julie has over 20 years of deep and broad experience in software engineering, IT infrastructure and integration engineering, and enterprise management. Her experience in commercial software companies included development of communications interfaces and management of programming teams. Georg von Sperling, DevOps and Technology Evangelist at CA Technologies
Helping organizations adopt Continuous Delivery without creating more technical debt, Georg is a practitioner, an evangelist and a general geek on software development methodologies, technology, patterns, systems and delivery practices. He advises, helps customers visualize, plan and implement transformational approaches to delivering digital assets from idea to production for medium to large enterprise organizations.
Anbu G. Muppidathi, Vice President at Cognizant Technology Solutions
Anbu has over 25 years of experience in the IT industry and is with Cognizant for the past 20 years. Anbu’s customer management responsibilities include advising the C-suite on breakthrough thinking, competitiveness, strategy, operations and transformation. His business management responsibilities include strategizing on QE&A business unit’s growth, managing the customer relationship, delivery and business operations for the customer engagements in Americas. Prior to QE&A, Anbu was with Cognizant’s Banking & Financial Services vertical for more than a decade managing a large portfolio of North American customers.
Alan Shimel, Editor-in-Chief devopsy.kinsta.cloud, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.
Tracy Kennedy is an associate product manager for CloudBees and is based in Richmond, Virginia. Prior to her current role, Tracy has worked at CloudBees as a solutions architect and a developer success engineer. In her free time, she enjoys rock climbing, grand strategy games, and reading. Read more about Tracy in her 
Udaypal Aarkoti is the Director of Solution Architecture at CloudBees. He believes in complete end to end automation and that continuous delivery is the future of application delivery. Prior to CloudBees he lead the middleware solution architecture team at Red Hat.
Andrew Phillips heads up product management at 
Georg von Sperling, DevOps and Technology Evangelist at CA Technologies
Anbu G. Muppidathi, Vice President at Cognizant Technology Solutions