devopsy.kinsta.cloud

Tag: change management

Integrating ITIL Change Management and DevOps
A DevOps solution to the DevOps problem

Intel IT’s Enterprise landscape is highly complex and diverse. The Enterprise Change Management team has the responsibility of managing the changes in this landscape, which is of the order of hundreds of changes every week. On one hand, we had to make sure we keep the change management processes stringent so that defects/major incidents did not happen in the production environment, leading to productivity loss, shipment delays or cost impacts. On the other, there is a need to assist the capability teams to land changes into production faster than ever to meet their dynamic business demands.

A challenge that many global IT organizations face is, How do we adhere to ITIL processes while teams transform into agile/DevOps way of working? In other words, how do we manage the two seemingly conflicting goals from two different IT organizations, one focusing on the agility of the development and measuring the success of the projects by their throughput time and the other measuring the success by the quality?

In the IT enterprise, the quality is measured by the number of change induced incidents in production—incidents that are caused due to change(s) landing in production. Our team had a challenging goal of keeping it to zero! The year 2017 started with this million-dollar (or is it worth more than that?) question: How do we meet this goal while not impacting the speed of the agile teams? We came up with an agile solution. A DevOps solution for the DevOps problem sounds logical, right?

We formed a scrum team to address this problem with the theme, “Simplify and automate, keeping the focus on quality, velocity and productivity.” The scrum team had change management process owners and the representatives from all the development technologies. The chart below summarizes the steps followed:

Simplify and Automate, Keeping the Focus on Quality, Velocity and Productivity

The first sprint was all about getting the alignment. As much as the number of teams that follow agile, as many definitions of agile exist! The focus was to demystify the myths about agile as well as ITIL change management processes. At a glance, processes look incompatible with Agile, but the goals are the same. Both aim to add business value incrementally in a continuous manner without disrupting the operations. They are not contradictory but complimentary.

Subsequent to this alignment, the second sprint went to the details of the enterprise change management process that’s in place. The process was broken down into the lowest level of details based on the 5W and 2H’s (who, why, what, when, where, how, how much). For example, there are process requirements from legal standpoint that are non-negotiable. There are certain processes required to ensure architectural compliance. We also acknowledged and aligned on some of the challenges and the time taken for process adherence. Multiple discussions happened in this space and it helped all of the teams to come to a common understanding on why these process requirements exist. Process swim-lane categorizing all of these aspects was internalized as part of this sprint.

Change Management Process Swim Lane

The third sprint got things into action mode. Each of those process steps was identified for potential simplification and automation. They were further classified as short-term and long-term based on the automation efforts on continuous integration/continuous delivery (CI/CD) that were underway. The development teams were shown the flexibility and empowered to meet the process need by way of automation rather than completing a manual checklist for every change. This way of policy and rule-based automation helps in a longer way in integrating ITIL Change Management and DevOps.

All along these three sprints, process improvements were rolled out. Several automation efforts were launched. Objective risk assessment enabled us to identify changes that required advance communication to all segments. Multiple gate approvals were reduced into only two—one when the development starts and later before it goes to production. Test cases and results from the CI/CD tools were used effectively to replace documentation.

Process refresher and engagement sessions were held across the organization to share the improvements, gather feedback and incorporate them into further sprints. This helped to gain the confidence of various teams and made the journey further better.

Now the focus is on continuous improvement, which is to implement all those action items identified and also keep them updated as other factors change in the internal and external environments. Engagement between theAgile/DevOps teams and the change management teams are getting refined based on the scrum development models. The success of ITIL Change Management and DevOps integration relies on calling out the agile process followed for the project and having the right engagements at the right points in time. The below picture depicts the engagement in a simplistic manner:

Scrum and Change Management Intersection

The efforts yielded fruits and the organization met the goal. The number of change induced major incidents was ZERO for the Enterprise Segment for the whole of 2017. We need to remember that these process improvements were done without impacting anything underway; it was akin to servicing a flying aircraft in the middle of the sky. The feather in the cap was the positive feedback from the ERP vendor: “The change control process at Intel has become very agile in recent years, with transports to production now occurring nearly every day of the year. … In general, we do not consider such frequent changes to Production to be desirable due to the risks involved; however, Intel appears to be managing the process well as the quality of transports is closely monitored.”

References:
1. ITIL® Service Management v3
2. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations by Gene Kim,‎ Jez Humble,‎ Patrick Debois &‎ John Willis
This article was co-authored by Ramani Subramanian

— Srinivasan Radhakrishnan
March 28, 2018
Tips to Optimize the Cloud for Scaling

When I first started migrating companies to the cloud (about seven years ago), companies would ask me, “How come the app isn’t scaling? How come users are still complaining about the slowness?” I would explain that the cloud enables auto-scaling, but you still have to make sure your app can scale in the cloud. In people’s minds, auto-scaling and the cloud was synonyms.

I don’t get asked that question anymore (maybe people realized the cloud is not as magical as it seems). But now, people think the cloud is synonymous with speed, agility and cost-effectiveness. Sure, cloud enables all that. But you still need to make sure you optimize the cloud to facilitate that. I see many companies in the opposite situation: their accounts full of resources with names such as test1, test2 and finaltest2, or they put together a SWAT team to reduce an AWS bill, or security is an afterthought. So many engineers are making changes, it’s hard to know who is doing what.

Before you run into this situation (maybe you’re already there), you have to create tools and processes to optimize your cloud for scale.

Establish Strong Policies

The magnitude of changes in cloud infrastructure is high. Unless you establish strong policies, managing all the resources across multiple cloud accounts get out of control pretty quickly. The most effective way we have seen is using tags for resources. For example, use tags that identify whether the resource is from a certain department are added to each resource. Typically, it works very well when the tagging process is embedded into the automation. If there are new resources that don’t comply, automation should terminate these resources. You should enforce policies like these sooner than later.

And just because these resources were provisioned with proper tagging, it doesn’t mean they are provisioned for its full capacity. You should build automation to find underutilized instances and establish a process for downsizing. To ensure everyone complies with these policies, you should also establish change management process.

Change Management and CMDB

The objective of the change management process is to ensure all the infrastructure changes cause minimum disruption and meet internal guidelines. Many say that ITIL change management process is not compatible for fast environments such as cloud. I agree that the traditional way of doing change management by tracking changes manually just doesn’t work, due to the magnitude of changes. But in the cloud, you can leverage automation to streamline the change management process; for example, you can use automation to filter normal changes versus manual changes. Once all the AWS changes are reviewed and approved, in addition to bringing visibility to all the changes for security and reliability change management can simultaneously help reduce waste.

Most companies heavily leverage auto-scaling, where new resources can be launched based on specified metrics. However, this behavior makes approving changes difficult. If new resources are appearing and disappearing, how do we track what truly changed in the infrastructure? To gain a true picture, you need an infrastructure delta, which shows resource updates for the still-running resources for a given time period. You can also use automation to build a dynamic change management database, and use auto-discovery to keep CMDB in sync.

Shared Responsibility

DevOps calls for breaking down the walls between operation and development team to deliver code faster. If you release software fast, you can rapidly add value to your customer. The cloud enables this culture because developers no longer have to wait for IT to provision resources. What doesn’t work well, though, is when no one takes responsibility for optimizing cost and maintaining security. If engineers can launch resources, they are also responsible for cost and security—it’s a shared responsibility. You should provide visibility to the cost for each department, based on tagging we talked about earlier, so they know the cloud is not free 🙂 Anyone can launch resources as long as they understand the security best practices for each service they are using.

Conclusion

Lean thinking calls for reducing waste. What’s waste? Anything that doesn’t add value to the customer. You should work fearlessly to reduce waste. You move to the cloud because you can innovate fast and it’s cost-effective, so you can quickly add value to your customer. You should continue to optimize your cloud infrastructure to ensure it’s meeting its original intent.

About the Author / JT Giri

JT is the creator of nOps – AWS cloud security tool. He spent last 10 years helping companies migrate to cloud and build automated infrastructures. nOps is a next generation security and collaboration solution for the cloud. nOps applies change management practices to deliver integrated security and reduce cloud waste.

July 3, 2017
Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 2

As we covered in part one, DevOps’ best asset and biggest risk factor may be one and the same: swift and precious change. In part two, we continue our discussion of the challenges and solutions with Deena Coffman, managing director of BDO Consulting’s Technology Advisory Services practice.

David Geer: What specific tools and technologies can help DevOps teams to address change management in these ways?

Coffman: There are a variety of tools and technologies to address some of the inefficiencies in change management, allowing DevOps teams to automate most of the process. For example, teams can leverage change management software to build pre-authorization processes and workflows to speed up change management approvals. On the security side, infrastructure monitoring tools will alert an organization when a discrepancy is discovered on the network, providing application visibility to the users and monitoring for unauthorized changes. DevOps teams should also consider using a collaboration platform like Slack to improve communication via virtual meeting places.

Geer: What will these tools accomplish and how should teams be using them?

Coffman: These tools are only useful if used properly. Deployments that require change management approval can go through the automated workflows. If a change requires approval, developers can use Slack or any other collaboration platform to contact the required party and explain the reasoning behind the modification. DevOps teams must also take the time to identify and automate as many “standard” changes as possible to reduce the reliance on change control meetings to speed up their deployments.

Additionally, DevOps teams need to consider using an infrastructure monitoring tool to ensure all changes to the production environment are known and authorized. If a change finds its way to production without getting the appropriate authorization, the monitoring tools can detect it and report it to the team so they can rectify the issue.

Geer: What kinds of things do DevOps teams need to do to address security to the satisfaction of the business?

Coffman: The development team and security do not need to be at odds. By working in unison, both sides can reach their objectives.

The InfoSec team would help the development team by making efforts to reduce the latency of security projects. The development team would help not only the InfoSec team—but potentially the entire organization—by preventing an expensive and embarrassing security incident by diligently supporting tools that automate security checks.

Additionally, the build process needs to run unit testing, integration testing, functional testing, regression testing and information security testing before the code reaches QA. Security testing is often insufficient or not performed at all. DevOps teams need to understand that security testing performs an important function of catching issues early so as to avoid expensive, embarrassing issues downstream.

Geer: What specific tools and technologies can help DevOps teams to address security in these ways?

Coffman: Several tools are available to help DevOps teams address security in many different capacities, including setting up continuous integration/continuous delivery processes, automating server configuration, automating UI testing and functional security tests, testing the security requirements of applications and automating the vulnerability scan of applications and infrastructure.

Geer: What will these tools accomplish and how should teams be using them?

Coffman: DevOps teams should integrate these tools as part of their continuous delivery process. As soon as a developer checks his code in, the continuous delivery system should run the functional security tests, non-functional security tests and the security scan to ensure that the developer did not introduce a security vulnerability to the application and the infrastructure. These tools are only as powerful as the test scenarios they run. Therefore, teams should write exhaustive test cases to ensure the security of the application.

Geer: How can teams do all these things and still maintain all the benefits and expectations that people have of DevOps itself?

Coffman: By deliberately working toward improved collaboration and communication, teams can take advantage of the integration and efficiency benefits of a DevOps structure without sacrificing security or quality. Technical knowledge is important, without a doubt. To be exceptional, though, teams need equally strong teamwork and communication skills to actively support all important functions in the organization.

— David Geer

September 16, 2016
Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 1

DevOps’s best asset is swift and valuable change, which also may be its biggest risk factor. DevOps certainly challenges change management and security. We recently discussed these challenges and the solutions with Deena Coffman, managing director at BDO Consulting’s Technology Advisory Services practice. Here is part one of that exchange.

David Geer: What kinds of challenges does DevOps create for change management? Can you illustrate with an example?

Coffman: By combining the sysadmin and developer functions, DevOps can lend to “shortcutting” the change management process in the interest of expediency. It may also undercut the value of having a critical eye from both a distinct system administrator and a distinct development reviewer. DevOps is designed to move quickly, and while the pace of change is accelerating, speed is often an ingredient for a security incident. Much like having the same individual who approves invoices also issue a check, DevOps makes for a faster, but not always secure, process.

When the system of checks and balances can be purposefully circumvented, that privilege tends to be invoked more often than necessary. As a result, developers may bypass change management altogether far too frequently. The collaborative benefits from DevOps integrating typically siloed functions such as QA, software development, security and information technology operations are important, so long as they aren’t offset by compromises in quality and security that result from the reduction in the review and approval process.

The expedient nature of DevOps can also lead to error, given the absence of human experience involved in the evaluation process. Before DevOps, if a developer needed to change a setting on the web server, they would submit a change request and wait for approval. Under a DevOps structure, though, the developer simply scripts the change and commits it to the repository—automatically kicking off the build process. The process runs unit testing, integration testing, functional testing and regression testing against the new code, and then pushes the new code to the staging environment and finally, to production.

Geer: What kinds of challenges does DevOps create for security? Can you illustrate with an anecdote?

Coffman: The biggest security challenge in a DevOps environment is that speed becomes paramount, which reduces critical evaluation on security risks. Developers incentivized to quickly deploy new capabilities are naturally pushed toward productivity and efficiency, which often run counter to security. DevOps can skew that balance so that security is only lightly or sometimes insufficiently considered. Automated testing will test for known exposures, but a trained, experienced human may be able to anticipate a risk before it materializes.

Geer: How can flaws in change management in DevOps approaches lead to security issues?

Coffman: Flaws or inefficiencies in the change management process reinforce the views of those who are prone to bypassing what they perceive as tedious process requirements. If a change management process is too burdensome or seen as lacking value, people will avoid it at any opportunity—not just when an exception permits a bypass. Since technology professionals tend to have a bias against waste and inefficiency, introducing a process requirement that consumes time and effort without returning value has a high probability of triggering a wholesale negative reaction.

Geer: What kinds of things do DevOps shops and teams need to be doing to address change management so that they continue to meet the expectations of the business?

Coffman: When it comes to ensuring that change management meets business expectations, it really comes down to several key ingredients: flexibility, speed, efficiency and risk management.

DevOps teams should design and manage the change management system to be efficient, responsive, and flexible enough to accommodate cursory reviews for low risk scenarios and careful, in-depth reviews for other changes. Leadership needs to also set a tone of support for the change control process so it is not dismissed or marginalized. Additionally, engaging outside expertise and/or including additional quality and security measures for higher risk projects are recommended.

A risk review that identifies projects with security and privacy risk and distinguishing types of changes supports the automation decisions and builds a DevOps program that balances its benefits with risk management.

Stay tuned for part two.

September 13, 2016
Continuous Delivery and Deployment: A Tale of Two Models

EnerNOC (Energy Network Operations Center) is an energy intelligence software provider with more than 1,300 employees in its Boston headquarters and international locations. The public firm helps global customers to efficiently consume energy. “Our mission is to change how the world uses energy, helping customers to improve profitability and visibility for better decision-making and sustainability by focusing on how they buy energy and when and how they use it,” says James Nichols, principal engineer and cloud architect at EnerNOC.

A regular user of Splunk (the company uses other DevOps tools and platforms as well, including AWS and Oracle), EnerNOC has fully automated monitoring in its perpetual drive to move past continuous delivery and on to continuous deployment.

Embedded in EnerNOC’s story of why it uses continuous development and continuous deployment models is a chapter on its move to fully automated monitoring.

EnerNOC DevOps Models

EnerNOC’s DevOps model is in constant transition toward full automation. Currently, it has fully automated deployment, fully automated monitoring and semi-automated quality assurance (QA), which pull the company’s continuous development approaches.

A change to “builds on commit” of smaller applications within the legacy software monolith at EnerNOC came hand in hand with further automation, targeting what the company calls “trustworthy continuous deployment,” Nichols says. That was the crux of EnerNOC extending itself into deployment automation.

Wherever the model is not fully automated and continuous in deployment, it remains continuous delivery. “Most of the variance between the approaches is due to aspects of the delivery pipeline that still have manual processes; these parts are usually comprised of the remaining manual testing procedures,” says Nichols.

Change Control Necessitates Two Models

EnerNOC develops and maintains mission-critical software that must pass through a change management approval process. This process includes communications within and across the company and customers, as well as user acceptance testing. This is where the manual testing comes in. Only at the end of this approval pipeline can EnerNOC pass the software baton into the hands of production.

Due to EnerNOC’s tightly yoked systems and the manual testing, change control remains, along with a mix of continuous development and semi-automation with fully automated continuous deployment. “DevOps is a cultural movement as well as a technical one, and so the existing model could evolve with the organization’s quality and maturity evolutions,” says Nichols. If the manual testing and the relationships between systems give way to more fully automated testing and more amenable relationships, QA could more closely embrace full automation.

Monitoring Challenges/Achievements

EnerNOC’s application diversity includes hybrid software with a foothold in the Amazon cloud, in EnerNOC’s data centers and on-premises. “We use three tiers on top of Oracle in the data center. In AWS, we use a combination of big data, cloud-friendly and serverless technologies,” says Nichols. Business transaction data traverses many machines that belong to service groups’ stakeholders. “The distributed, decentralized architecture makes troubleshooting, performance management, security operations and production operations challenging,” notes Nichols. Better monitoring was the answer.

EnerNOC wanted to offer nontechnical users easy access to systems-level monitoring data with transparency into the many systems through a single pane-of-glass dashboard with readily available system notices and troubleshooting abilities. “We would use Splunk to do this and then feedback everything we learned from using Splunk into driving our DevOps and engineering processes into the future,” he says.

The EnerNOC DevOps performance team built the dashboard that accompanied the Splunk service, ensuring it would register the performance metrics that are of value to the organization. The performance, product management and development teams maintain an awareness of the metrics and receive the related alerts. “The development team incorporates the ongoing metrics early in the development life cycle to improve application performance,” says Nichols.

EnerNOC first applied Splunk to web logs for operational capacity planning, then used Splunk to give the Ops team transparency into the function of its new real-time data product that was working in production. As new products came on board, the organization added them to Splunk to provide Ops teams with dashboards, at the urgent request of Ops team members.

Current Status: All Systems Good

EnerNOC applies Splunk to test and production systems for views into consistency and acceleration. “Splunk is our primary alerting infrastructure for low-level system metrics like CPU utilization as well as for complex business transactions that span multiple systems,” says Nichols. Splunk has eyes on application health, cluster state across machines, expected systems tasks and suspect or known-bad system behavior, as well as the presence or lack of any significant event.

May 20, 2016
The War with Human Nature

What is the difference between “leading” edge and “bleeding” edge? Being the first human being to advocate change.

It is a virtual war with human nature to accept change. For all the innovation introduced into our society, most of it is a product of invention over evolution. Don’t get me wrong: Once we find something that works, we inherently work tirelessly to make it better, thus evolving it over time. But the genesis of the “new” is easier to accept as creation—a new idea, a new innovation of our own—than a change in direction, a forced acceptance of the ideas or creations that were “not invented here.”

In the workplace, most employees are looking for personal security, for a feeling derived from consistency and predictability in the day-to-day work activities. Anything that threatens an employee’s ability to continue to secure compensation becomes something that inherently meets our resistance. The No. 1 culprit: you guessed it … “change.” As long as what I do remains the same, over time I will get better and better at it. Practice makes perfect. But when something “new” is introduced in the actions I am supposed to perform, there is a risk that my value, my expertise, will be reset to zero and perhaps I will not be valued as I once was. This threat to my compensation is real; therefore, my resistance to it is equally real.

Programmers Found Guilty

You would think that engineers and those working in the software industry would be exempt from this phenomenon. But most are not. Finding an ex-Cobol, ex-Clipper, ex-Windows, current mobile Apple iOS programmer is like finding a unicorn in Central Park. Most successful Cobol programmers are still Cobol programmers, looking for jobs in mainframe shops, content to see the new generation of programmers not competing for their declining job markets. The early generation of programmers minted after the adoption of the PC computer tend to know less about Cobol and RPG-3 and more about PC-based programming languages such as GW-Basic and Pascal (at least the older ones). Programmers minted after the Windows graphical adoption know more about that generation of coding languages, and so on, and so on.

The basics of coding have remained the same since the advent of this discipline. We create a list of instructions, in sequential order, that the computer must execute to produce a desired result. There are nuances to this, and an explosion of capability in this, but the basic principles remain much the same. Yet, programmers well-versed in a particular language are shy about learning a new one, unless the platform is similar, the features are similar (albeit improved) and the belief is high that they will be successful in the new language quickly. There is no reason why a Cobol programmer could not be successful creating mobile apps. Or, for that matter, a mobile app developer could try his hand at Cobol. But it seems neither do, at least not very often.

What happens instead is that each programmer begins to establish his/her own ecosystem of support: simulation models that imitate the platform responses, highly evolved text entry tools that auto-format code and check for omissions such as missing parenthesis and scripted tools that perform the mundane tasks we need to build (or construct) new versions of code or deploy a build into a new environment. This same ecosystem of support will exist for database, network and middleware engineers as well. The tools will be decidedly different, but the reasoning for them tends to be identical: to make things easier to do.

Looking for a Magic Bullet

There is no cure for breaking down the resistance to change, except constant exposure to it. However, forcing employees to face perceived threats to their value—and their compensation—will, at a minimum, produce exorbitant stress at work—or, worse, an employee turnover rate that would rival donuts next to a police station. The goal is to introduce change that will ultimately increase the value of an employee, as well as potentially increase the compensation they worry about. This is where the DevOps value proposition must be created for each impacted audience. A generic value statement is not enough, nor is a mandate by top company executives. You cannot “force” people to adopt something new, anything new.

Creating the right value proposition must be specific to the audience you seek to adopt the “new” system, tooling or process. DevOps is a continuum of all three (people, process, technology). So for the database team, for example, changing out the ecosystem it has evolved into a new set of tools and processes, may seem like a hard sell at first. The advocate of DevOps (whether the exec teams or the service teams delivering it) must be able to articulate the value proposition clearly to the database teams.

“Why” should they move to a newer ecosystem? A stubborn engineer can always point out the holes in your tool and the beauty of his own. He/she will likely have evolved his/her own support ecosystem to a state of near utopia … but only for himself. Usually the processes and tooling engineers use to make their own lives better do not scale to others in the department. And God forbid some other engineer updates “their” scripts or makes changes to “their” process. While this is wonderful for the engineer who creates it, it is miserable for anyone else attempting to use it.

Constructing a Value Proposition

The value statement must trade prima donna utopia for scale. Using common DevOps tooling, it is possible to support many more task requests than most individually constructed ecosystems can handle in a given day. And believe me, those task request increases in volume are coming. For example, an Ops engineer may be able to handle provisioning five systems a day with his own tooling, but when requests come in to provision 50 or 100, he is dead in the water. Having the Ops engineer help manage the deployment template or framework allows users to self-serve on provisioning requests and keeps him from having to do everything for everybody. His job is different but every bit as important.

The value statement must trade current skills proficiency for next-generation, market-valued skills. In effect, a database guy is valuable, but a DevOps database guy is considerably more valuable, or a “big data” database guy, or a IoT database guy … pick your poison. Whatever you are today, whatever skill set you have, the jobs that pay more money are the jobs that fewer people know how to do. At some point, a lack of DevOps proficiency will relegate an engineer to the pages of history. On the other hand, adopting and embracing DevOps will allow engineers to get the training they so desperately need to keep their career “fresh.”

Finally, the value statement must trade today’s earnings for tomorrow’s increase in earnings. Companies that embrace DevOps are going to either save extreme amounts of overhead by staff reduction remaining at the same level of innovation productivity. Or, they are going to increase innovation productivity levels radically, making them leaders in their respective space by meeting customer demand much faster with higher quality (the path most taken). Either option should translate into higher compensation for employees who embrace DevOps. If companies do nothing to increase employee comp, choosing only to increase profits, competing firms will be happy to poach them and pay them more instead. This forces more costs in recruiting, training and employee turnover when it happens again.

A Human Strategy that Works

Your strategy for training employees to embrace change should be particular to the kind of work they do, and reflect the things that will matter to them. If you need help crafting a strategy let me know, I am looking at the moment 🙂 . Not every team will be as critical to the overall success of DevOps as perhaps your engineers are, but the value proposition should adjust for this fact. Keep in mind, if the change is less, then the resistance will be less as well. Your accounting, HR and legal teams, for example, may not feel the impact of DevOps as rapidly or as severely (in terms of change); thus, their resistance may be less, but it is likely more than zero.

DevOps is about people, process and technology. It is in that order, with a heavy emphasis on people and a relatively light impact from tooling. And it will succeed or fail based on the embrace of people, process and technology. Process and technology do not care about change. People do. Since people are the largest priority or component of DevOps, it is their fears that must be addressed and their value that must be enumerated to succeed.

To continue the conversation, feel free to contact me.

May 20, 2016
7 Causes of Continuous Delirium

When we think of CD, we typically think of continuous deployment as a result of continuous delivery, which is a goal of DevOps.

In this article, I coin the phrase, “Continuous Delirium,” to mean unending confusion or bewilderment in development environments. There are probably several examples that could fit the bill here, but for now I offer seven real-world examples. I bet you’ve seen or heard about at least one of these.

Now that I have your attention, here are seven causes of Continuous Delirium, each with cures rooted in DevOps:

No. 1: Insufficient Configuration and Build Management

Insufficient configuration and build management in software builds leads to the same or similar software impurities rearing their ugly heads again and again. You build, test, discover the defect, fix it—or so you think—build again, retest and boom: The same coding error or software malfunction recurs.

“You can cure this malady with a rational approach to the build and configuration management process, checking software in and out with established procedures for each software module using a tracking sheet that identifies the revision of the sub-modules for the entire software product,” notes Jon M. Quigley, founder of Value Transformation, a product development consultancy and training firm based in Lexington, North Carolina.

No. 2: What’s Going On? Who’s Doing What?

When there is no reasonable understanding of the work, who is doing what work and who is responsible for what—and there is no consistent measure of what is good work or bad work—you have a breeding ground for anarchy.

“You cure this ill by establishing what constitutes the appropriate quality of work as well as by periodically reviewing the state of the objective with the receiving or depending entity,” says Quigley. Make sure everyone is on the same page with the same definition of what constitutes quality work and make sure the objectives have not changed. Also, ensure these goals are also clearly understood by all.

No. 3: Confusing Clean Daily Compiles with Testing

When you do not test the software as early as possible and throughout the development process, you can’t really say it has no bugs. Error-free compiling is not error-free software. Software compilation does not have the same investigative effect as using the software in the customer environment.

“Test throughout development using complimentary testing approaches such as testing to requirements, combinations of stimulus testing and exploratory testing, for examples,” says Quigley.

No. 4: Planning Too Far Out to Control

The further you try to plan things out into the future, the less control you have as you follow the project through to conclusion. “When we treat a plan as though we know everything associated with the project for the next 12 to 18 months, we are fooling ourselves,” says Quigley. By breaking projects into bite-sized increments, you can better assure successful completion of sub-projects and of the project as a whole.

No. 5: A Lack of Customer Involvement

Customer involvement is critical to customer satisfaction. “By getting the product or a prototype into the customers’ hands for exploration, you can retrieve feedback for developers,” says Quigley. This may expose the project to risk in the form of the potential for a great deal of change at the customers’ request, but the other side of the coin is the risk that nothing changes and the customer doesn’t use the product at all.

“Rather than try to deliver one incarnation of the product at some future date, break down the delivery into separate iterations that meet prioritized customer expectations one by one as you refine the product and add features as you go along,” says Quigley.

No. 6: Isolating Change Management from Customer Involvement

Customer involvement means there will be change. Change requires change management. There is a given—and correct—assumption that customer involvement will trigger change management at some point. You cannot isolate the one from the other.

Further, just as you can better address customer requests for changes by breaking the project into smaller subsections, so also can you manage budget through change and change management. Project subsections enable you to see the need for change a little at a time so you can create change in a more targeted and cost-effective manner.

No. 7: Forget About Collaboration—at Your Own Peril

It is necessary for DevOps team members to collaborate at all levels, except those where the work on software is so minor that it wouldn’t benefit from shared thought. To maintain development as an expedient process that benefits from all stakeholders, open communications and collaboration must reach and pull from all team members. Otherwise, the project could suffer from a lack of critical input.

April 29, 2016
The Nirvana of Gated Progression

A highly functioning enterprise-class DevOps set of services has a fundamental goal of full automation. To be honest, having the right DevOps strategist (or visionary) leading this effort is as important to achieving this goal as the combination of all the challenges and topics that must be addressed. So for the sake of argument, will we assume you have one (if not, feel free to contact me—I am looking at the moment 🙂 ).

With key leadership in place, the first target is always the automation of the build and deploy process. This could easily be considered the core of DevOps or its heart. But once this automation has been achieved, the trickier topic of release orchestration begins to emerge. Strategy is to release orchestration what taste is to food: Having the right one does make all the difference.

To progress our code to production, the inherent assumption is that all testing has been completed successfully. Sounds like common sense. But then, how much of that testing occurs with human hands on it? The user acceptance testing (UAT) discipline inherently requires a human at the wheel. But the reality is, for every time a human must interact with the progression of code from the development-class environment through all the testing environments (integration, performance, user acceptance and QA) on its way to production, the momentum of innovation is slowed down—all the way down to human levels.

The Complexity of Testing Types

The first automated testing begins to occur in development or (DEV) class environments, where the programmer is under strict instructions to implement test-driven development. This requires new feature/functional testing as part of the build or deployment automation into DEV. Either the build or the deployment cannot be said to be successful “if” the accompanying tests are not “passed.” This also represents our first “gate.” Sophisticated release orchestration (RO) software can examine “flags” set during the build or deployment process to see if the testing has been completed successfully. If so, the code can progress to the next testing level environment within the software development life cycle (SDLC).

Another useful type of testing completed early in the SDLC (most often in DEV) is the examination of code coverage. Development organizations can begin to examine how much of their code is put to the test, and set arbitrary percentages as a minimum standard before the target application is able to progress. The addition of alternate types of testing occurring in DEV class environments does add to the quality of the innovation effort. However, it brings with it an increased labor effort to automate both progression—and audit—of the version under test.

For instance, feature/function testing may use a single repository to store the results for a given version of the code. I go to my testing repository and I can look up the results from version 1.4 of my application, or version 1.5 of my app, as many as I desire. Should a regulatory official or internal auditor ever ask me to prove the quality of version 1.5 of my app, I need only consult my testing repository and show them how many tests were executed, and how many passed or failed before this version of my app was ever deployed into production.

When you add code coverage testing to the process, I now need to consult two different testing repositories, producing two separate reports to assess the quality of this same version of my app. There is nothing wrong with this, but it does create more work or effort to produce than it would if a single testing repository was used. The problem only increases as we keep increasing the types of tools we use to prove quality. Security testing, for example, (ensuring code does not have inherent security holes) tends to use different tools, or, worse, processes that require human peer observation, which again can add effort to create the automation and the traceability for audit. These are critical types of testing (and/or processes) so they must be done, but they also come with an inherent cost in automation effort.

Small shops may not worry about these kinds of concerns, but large organizations will, particularly if they are still in the process of transitioning from legacy waterfall methods (where controls were high) to new agile methods (where it feels to auditors like the wild, wild west). Compiling a test log that matches the version of code under scrutiny such that it can be reproduced automatically will restore the faith of both internal auditors and regulatory officials who come asking. But the addition of each new type of testing tool will require more effort to maintain the ability to compile those test logs.

The Complexity of Testing Responses

Next, consider the impact of non-binary responses to automation efforts for gated code progression. Does my feature work (yes/no)? This is a binary evaluation, a true/false or yes/no that is easy to set my progression flag for testing response. It is another matter to evaluate the results of my code coverage and ensure this app is above 70 percent or between 70 percent and 90 percent. These kinds of questions are not binary (in other words, there is a real result to compare to, not just a pass/fail), and require further effort to set a binary flag for the gated pass/fail. When the evaluation question does not intend a binary answer at all, such as a grading of 1 to 5 where 1 equals horrible and 5 equals outstanding, I am now most often forcing a human to make an intelligent decision as to whether this code should progress.

Multiple testing repositories require additional automation effort. Multiple kinds of responses require additional automation effort, and the standards by which the gates pass or fail also may change over time, requiring tracking of the goals at the time the tests were taken. Creating independent test automation that can be called from the build process or the deployment process is wise. Associating this test automation (whether scripts or simulations, etc.) with the version of the application code is wise. And working to reduce the number of repositories to examine is wise. Having de-coupled test automation strictly from a build allows it to be re-executed in more advanced testing environments when the need arises.

Charting the Course to Nirvana

RO benefits from as full a level of automation as possible. I will discuss the RO topic more fully in another article, but the foundation of automated change progression relies on a gated history of proving quality. As long as the code under review continues to pass testing, setting progression flags to green or yes or pass, the RO software can continue to progress it along its way to production. Failing code must be sent back to the DEV environments where development teams can address the issues before making its way back through the automated process once again from beginning to end.

If the organization has made the intentional decision to keep humans substantially involved in the testing and evaluation cycle, it must realize a few impacts of that decision. Daily movement to production becomes less likely—most often it takes a week or two for changes to undergo significant human scrutiny (depending on the level of testing automation). The results of human intelligent decisions also must be chronicled in the testing repository to preserve the history for audit. This can lead to a “blame game” kind of thinking when something goes wrong, as it is immediately attached to “who” approved it, instead of to “why” they did so. And obviously the cost of labor is always the highest cost element in any innovation cycle, so buyer beware in this scenario.

On the other hand, “Nirvana” is the state when all testing (except UAT) is fully automated and gated, and provides the foundation for RO software to move code into production as often as it can prove quality. This could mean fully automated releases instead of release weekends. This could mean multiple releases per day into production. This could mean that releases themselves are no longer considered “events,” but by default are now “non-events.” Nirvana does not endanger the lives of change and release managers—it actually gives them one. For a “change” (pardon the pun), they get to discover something they vaguely remember … a life outside of work.

To continue the conversation, feel free to contact me.

April 26, 2016
Culture Change: Metrics, Tools Make a Difference
Change is in the wind.

This time of the year, as winter weather gives way to springtime renewal, this concept isn’t a cliché but rather a concrete, visceral experience. You go outside, smell the earth defrosting and know for certain that change is happening all around you. You can feel it.

Such is the case on the current DevOps landscape. Seemingly every conversation, with everyone from end users to industry analysts, reaffirms that people are making significant progress toward greater adoption. For an increasing number of organizations, DevOps is maturing from a set of budding goals into a tangible reality.

At the center of this massive shift within the software factory is the issue of cultural change; as we know in all things DevOps, culture trumps all else. So how are organizations affecting, empowering and measuring all this seismic transition, in particular the human elements?

During the recent DevOps Virtual Summit, leading practitioners representing industries ranging from banking to travel and technology shared their views on the progress of cultural reinvention.

From management tips, to tools and even metrics useful for driving transformation, here are some of their observations:
1. What’s the most critical cultural aspect of planning?
  Kazi Whitfield, director, Quality Engineering, MUFG Union Bank, N.A.: Scalability on this is the key; if you’re doing this at a new organization, you need to understand how to scale it, and starting at some smaller level helps you get there. It also helps each individual and those components that are changing to truly understand what’s changing, and why it’s changing; I think that’s a huge part of the conversation that has to happen.
1. What’s the biggest obstacle to cultural change?
  Eric Warheit, senior manager, Environment Management, Southwest Airlines: The challenges that we’re working through are organizational in nature. It goes back to the question of localization, and people maintaining a mindset that they’ve previously had, and what we do to create the conversation around what’s possible and the key impacts that we want to have. Ultimately, it’s about being productive with quality, and creating those common values.
1. How can tools enable cultural change?
  Eugene “Bear” Lehenbauer, group manager, Information Technology, Citrix: If I have to pick one that links them all, a ring to rule them all, that’s when you have an orchestration platform that can take what individual teams are already using and plug right in, or you can take what needs to be automated and automate it; that orchestration harness can now become that common thing that links them.
1. How can you ensure cross-culture collaboration?
  Warheit: Creating that concrete example, that proof case tying analytics to it, so that we have real measurable ways to make good business decisions, and being able to foster that perspective so that we can look across the organization, as opposed to in a localized way, is really critical to be able to foment that collaboration that’s needed.
1. How do you measure cultural change?
  Daniele Romano, product owner, Continuous Delivery Pipeline, ING Bank: The most important thing as an organization is to able to apply an analytical approach, meaning that for each team or department you should look at the current situation and have in mind the target situation, and then have different means to guide everyone toward that target.
1. What’s a key cultural change metric?
  Lehenbauer: We’ve a developed an agility rating scale that allows people to earn scores for something as basic as change management. Based on how those key metrics behave, and there are several of them, but they are fact-based and they’re user impact driven for the most part, they reward the right behaviors. Those types of things can draw out some of DevOps principles results, but also have hard data behind them.
1. How do you recognize advocates of cultural change?
  Whitfield: Being able to phase reward and recognition into the process is key. Organizations are doing a lot of great things and making millions of dollars, and as the individuals that are coding or testing are driving it, you have to understand how you get to those people’s hearts, to make them passionate about what they’re doing. That’s something that becomes key and it’s something that’s different per organization.
1. What’s the best way to gauge successful cultural transformation?
  Romano: There is a really only one critical human metric, which is happiness. So, if engineers are happy and they can deliver fast, at a high quality and with intense focus—that’s a very unique but important metric.
So there you have it, some of today’s most critical elements in driving the cultural transformation necessary to succeed in the DevOps world.

If you missed the DevOps Virtual Summit you can still view recordings of all the sessions including this customer panel, and stay tuned to this space for more excerpts and highlights.
March 30, 2016
Change Management & Continuous Delivery: Clash or Harmony?
Introducing changes into running software systems is risky. There’s always a chance of failure, and that can have a negative impact on your business. It’s important to mitigate that risk, but there’s more than one way to do so.

Change Management is all about ensuring that changes to existing systems and software in production are done in a controlled way. Change managers review the changes that are to be implemented, assess their impact, and perform due diligence, before giving or withholding their approval.

Continuous Delivery (CD) is about improving the process of delivering software and increasing efficiency. It aims to cut down the time between ideation and realization. It’s about reducing the scope for error and increasing the speed to market to generate a competitive advantage.

Is there an inevitable clash between CM and CD? Are they really compatible? At heart, the CD pipeline is intended to achieve the same thing as CM by enabling the smooth, error-free roll-out of changes, but today they are usually coming from very different angles.

If CM is to have a future in the CD revolution, then it needs to change. Before we discuss how that might be achieved, let’s take a look at the current state of play

Is Change Management delivering?

Ask yourself what impact change management is having on your organization. If change management was working perfectly then there wouldn’t be any production problems, and yet issues still crop up. There are a few things to consider:
1. How many entries in the CM checklist could be automated? Is the change manager doing anything that really requires human input? Fact collection and verification could be a part of your CD pipeline.
2. How much value do check boxes deliver in the first place? What is change management really bringing to the table?
3. Is the process fit for purpose? Does it actually prevent problems, or is the change manager really just a fall guy, there to sign off on the release and take the blame if something goes wrong?
Clashing with Continuous Delivery

CM is intimately linked with the idea and practice of big complicated transformations, but with CD we’re not talking about enormous, disruptive changes. CD is about changing small parts of a system and doing it often. The risk to reward ratio is different when you think and work iteratively. Smaller change sets are less likely to cause problems and easier to rollback when they do.

Does the process really require human input? Heavyweight validation will slow it down and speed is one of the key benefits Continuous Delivery offers. Streamlining and automating the checklist process makes perfect sense in many business contexts. The aims of CM and CD are in sync, but CD is better placed to deliver.

A possible advantage of traditional CM is the presence of a human eye, offering intuition that a machine or a piece of software may not be capable of. By moving beyond the checklists and reimagining the CM role we can retain this advantage and push CD forward.

From Gatekeepers to Masters of Continuous Delivery

If the bulk of CM is really an administrative exercise in checking boxes then it could be an unnecessary overhead, but there is a way to retain that human oversight for organizations wary of full automation. Why not ask the former gatekeeper to own the CD process? For a CD pipeline to be truly effective someone has to guide the process, engage with all the teams involved and understand exactly why and how the process generates business value.

A change manager, paired with a build or release engineer to handle the automation, could work to help organizations move towards CD. They are uniquely well placed to identify dependencies, map processes, and help construct the CD pipeline with minimal disruption. They can help sketch out what the pipeline should verify, what data and dependencies need to be taken into account, what automated tests are required, and how it fits with other components to automate that release procedure. They also have experience mediating between business and IT.

The traditional CM approach of committee meetings and spreadsheet checklists is dated, and for many organizations it no longer makes sense. You can find much greater business value in focusing on an efficient rollback procedure that kicks in immediately and automatically when changes fail.

Take a look at your CM and ask yourself – is it really adding anything to the process, or is it just another bottleneck? It may be time for a change.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
January 21, 2015