October 2019 will mark the 10th anniversary of the first DevOpsDays, where the term “DevOps” began to catch on. In the world of information technology, 10 years is a long time and it is quite evident based on the amount of change that has transpired since then. To name a few, we have seen the maturing of CI/CD automation and many innovations such as application release automation, containers, cloud infrastructure orchestration, DevSecOps, DevOps database, DevOps governance and micro-services.
These innovations have become part of “mainstream DevOps.” This evolution is continuing with AIOps, machine learning, value stream management and others. Each of these innovations has made DevOps not only more powerful but also more complex.
As a consultant working on DevOps projects through their life cycle, I have learned that complexity can plague the success of DevOps within an organization. Complexity cannot be avoided, as DevOps is complex and will likely continue to be. However, the key to avoiding failure through your DevOps journey is to engage the complexity by using DevOps tenets to implement DevOps. Do not try to boil the ocean. Instead, at each leg in the journey, take inventory of where you are in terms of current goals, state and best practices. Fine-tune your direction and build your solution using proven continuous delivery methods.
The 27-Factor Assessment Model
A comprehensive yet efficient assessment of your current state is key for ensuring complexity has not gotten out of hand and the organization is focused on success criteria that meet the needs of the business.
I use a 27-factor assessment model. The factors are the cross-product of current best practices for three dimensions (people, process and technology) with nine pillars.
A Scope of the 9 Pillars
Leadership: Gain insight and understanding of your leadership practices and how leadership can improve DevOps adoption and performance.
Culture: Identify opportunities to improve cross-functional collaboration and communication across your organization.
Application Development(or Design for DevOps): A practical analysis of application development practices identifies improvements for your DevOps performance.
Continuous Integration: Analysis of branching and software integration workflows and tools identifies bottlenecks and improvements for builds and packaging processes.
Continuous Testing: Analysis of end-to-end software testing, verification workflows and tools identifies bottlenecks and improvements for accelerating testing while improving the coverage and quality of tests.
Infrastructure on Demand (orElastic Infrastructure): Analysis of systems and services for orchestrating and hosting applications and DevOps workloads provides insights to improve performance and utilization of resources.
Continuous Monitoring: Analysis of end-to-end monitoring workflows and tools for applications, releases metrics and infrastructure provides improved visibility for all business and technical stakeholders.
Continuous Security: Analysis of security practices and tools identifies opportunities to integrate security practices into the continuous delivery pipeline.
Continuous Delivery/Deployment: Analysis of delivery and deployment practices identifies improvements for safer, faster and more efficient releases.
Each combination of pillar and dimension has associated best practices used to score relevance and current state relative a DevOps goal agreed by the organization. Here is an example of the continuous security pillar.
The 27-factor assessment for an application is an important input to a future state value-stream map workshop. The value-stream map is then transformed into a road map and implementation plan.
This assessment approach is thorough and yields results that are usually accepted by stakeholders because they are involved in the process at each step. The result is aligned to their priorities and goals and the solution road map is directly traceable to stakeholder inputs. The approach is flexible enough to evolve over time and can be customized to suit the priorities of an organization. For those that argue that 27 factors are too many and too complex, I must say the complexity of DevOps itself requires a comprehensive assessment approach.
Conclusion
In this article, I described a comprehensive, efficient and evolvable assessment approach that uses 27 critical success factors for assessing DevOps applications. While DevOps offers immense value for software development, delivery and deployment, a rigorous and thorough assessment with up-to-date best practices are essential to accomplish DevOps goals. Each organization is different and has different DevOps goals and requirements. The changes to the three dimensions vary according to the results of each DevOps assessment.
Facing challenges is an inevitable part of life, but how we deal with them can make a big difference. When problems show up, it’s important to focus on the things that unite us—it’s better to have a common enemy than fight internally. Experience in complex software development has taught me that it’s crucial to stop, think and act at regular intervals.
The feedback loop is the concept on which the agile mindset is built. Instead of having a project that is one year long, we divide the project into smaller mini-projects of two to four weeks. As the saying goes: There is only one way to eat an elephant–one bite at a time.
To inspect and adapt quickly, I recommend a time frame of just two weeks. During these two-week sprints, the team must work hard to achieve the goal. At the end of each two-week period, the team meets to inspect progress and creates a plan for improvements during the next sprint.
The purpose of the retrospective is to:
Inspect how the last sprint went with regards to people, relationships, process and tools.
Identify and order the major items that went well and potential improvements.
Create a plan for implementing improvements to the way the team performs its work.
These retrospectives can be general or they can be focused on one issue that occurred during a sprint. Sometimes a retrospective is about de-escalating or resolving conflict and tension. If the sprint didn’t go as planned, the facilitator needs to be cautious about how they run the retrospective. They need to keep in mind how they say certain things, because team members might already have low spirits coming into the meeting.
I stay away from using harsh judgments, blaming specific members or showing my frustration. The goal is to create positive energy for the retrospective. Facilitators should motivate the team to have fun while they participate in the retrospective. I like to have high energy, be positive and try to get the others on the same boat.
As a scrum master myself, I was shocked to find out that only 81% of scrum masters hold retrospective meetings. The retrospective is an essential part of any given sprint and provides valuable information on how to improve.
Preparation
Before the retrospective meeting, I think about the format and tools. If you work in a distributed team, like me, a digital whiteboard is recommended. Using the digital whiteboard, we can have a retrospective no matter where we are in the world.
Here is my process for a zombie retrospective meeting. It’s divided into three steps: check-in, input and output.
Figure 1: Zombie Retrospective board.
Step 1: The Check-In (15 Minutes)
The check-in stage is kind of an ice-breaker. It should be fast, like the starter before the main dish—which in this case would be the deeper and more constructive discussion. Check-ins are treated as an exercise at the beginning of the retrospective to gather information on the just-completed sprint. One nice starter for more in-depth discussion is the happiness index: a quick way to check-in with the team and to see how everybody experienced the last sprint.
I ask the question: “On a scale of 1 to 5, specify how happy are you with the last sprint? No.5 is super-happy, and 1 is totally unhappy. Put the sticky note with one keyword next to the number on the whiteboard.”
Figure 2: Icon made by Freepik from www.flaticon.com
The Scale Is:
5 = Super-happy! Don’t want to change anything!
4 = Pretty happy, but there are some things that need to be fixed.
3 = I can live with this, but there are many things that need to be fixed.
2 = Not feeling so good about this right now.
1 = Totally unhappy, I want out.
I give the team three minutes to five minutes to gather the input, then we discuss the issues. It’s a perfect moment to turn the problem into opportunity. You can use solution-focused questions, such as:
What needs to be done to raise your happiness index?”
Let’s imagine we have an ideal world, what would it look like?
How would you like to have it?
Example:
The team members identified that they are moderately happy because they spend a huge amount of time preparing for the release. The work is boring and repetitive. I then turn the problem upside down and ask how we would like to have it. The team then starts with something they want.
Goal: As a team we want to have automatic regression tests, so that we can have faster releases and use our time for value-added work.
Next, we define action items that will help us achieve the goal—we plan to invest time and implement the automated tests. The product owner will be responsible for managing the product backlog in a way that we would achieve the goal within six months.
The scrum master needs to keep in mind which team members are more open to sharing their opinion and those who are more reserved. Shy employees may have lots of great input, but if they don’t feel comfortable sharing it, they will simply sit back and listen. It’s up to the scrum master to understand the team well and know which team members may need an extra push to be more vocal, to collect the input from the colleagues that are more reserved. The first step is to create an inclusive workplace with a team culture of trust, openness and transparency. You want to reach a point where everyone feels like they’re really a part of the team, and know how their efforts contribute to the team’s success. It’s important to remember the importance of tone of voice and body language and the positive influence this can generate.
Step 2:Team Input (45 Minutes)
There are plenty of techniques to collect input from the team during the retrospective. When I prepare the whiteboard for a retrospective, I look into the different techniques and choose the one that I believe is most fitting for the current context. When we face prevalent problems and challenges, I particularly like the zombie retrospective.
Figure 3: “The Zombie Retrospective” by Agata Matynia. Image representing the zombie retrospective used to support the team input stage. This tool helps teams to visualize problems as part of the journey to aid creative solutions.
With this poster, I explain the following to the team: When a virus turns the human race into bloodthirsty mutants, we need to save ourselves from the zombie attack. We need to think clearly, so our discussion would be split into three groups.
The zombies in the illustration represent the problems that we have.
The blocked chair represents the actions we can take to stop the problems affecting us.
The last place is our hiding place. The TNT explosives, gun and hand grenade represent the team strengths.
Once the team has finished contributing their input, we then go through every idea on a case-by-case basis and discuss it. This stage should take a maximum of 45 minutes.
Following discussion, we then move on to the final stage of the retrospective. This step takes all the input and turns them into concrete, measurable goals with action items, responsible people and clear timelines.
Step 3: Team Output (15 Minutes)
The output stage outlines the goals the team would like to achieve the actions they will take, who will do it and when. The tasks are handed out to the team, and the scrum master needs to delegate people to specific areas with deadlines to ensure that things get done. Otherwise, everyone will agree that something needs to be done, but nothing will be changed.
I use the following format to achieve this:
Goal: The end result the team wants to achieve.
Actions: The action points that need to be done to achieve the goal.
Who: The owner of the goal.
When: The specific date.
Example:
Goal: As a developer team member, I want to gather analytics on how the mobile app is used so that it can focus on things that matter.
Action point: Deciding what information to gather and who will do it.
Who: Anna is the owner of this topic.
When: In the next sprint.
The retrospective at the end of a sprint should never be left out. It allows for the continuous improvement of people, relationships, process and tools. It makes the team great.
Optional Step: Inspect and Adapt the Retrospective in the Plus/Delta Format (10 Minutes)
From time to time, it is a good idea to evaluate the retrospective meeting. The goal is to be able to learn from the current retrospective to do it better next time. For this, I recommend using a quick, simple technique such as Plus/Delta to make improvements.
In this final step, the team identifies the strengths of the retrospective and what changes can be made for the next one. I ask the team to use the sticky notes to share their thoughts.
Plus: What can the team change or add to bring more value to the retrospective? How we can repeat it?
Delta: Is there anything that we should change in terms of how we run the retrospective as a team?
The answers to these questions can give scrum masters valuable feedback on how to bring the team together, solve problems more efficiently and make the best use of the time spent in the retrospective. Up until this point, team members have only been discussing the previous sprint, so giving them some time to reflect on the meeting itself sometimes helps them unwind.
More Ideas to Spice Up Your Next Retrospective
The retrospective meeting is a great opportunity to inspect and adapt. If, like me, you would like to run it every two weeks, you will have about 26 of these meetings in a year. The zombie retrospective is great, but you should avoid using the same approach every time. Get as creative as you like—there are many variations and possibilities worth exploring.
For check-in, I also like to utilize pain scales, a common communication tool used in a variety of medical contexts and settings. Trust me, it also works perfectly in business. Try the LEGO pain assessment tool as a starter.
Enterprises are embracing DevOps underpinned by automation to ensure the quality of software across all digital touchpoints. In fact, automation has become the crux of continuous integration and delivery, the vital DevOps testing goals. Without automation, DevOps would become a distant dream, and businesses would struggle to keep pace with seamless delivery and integration. However, DevOps entails more than the implementation of the latest automation tool and demands a more holistic approach. It does so by bringing together processes, people and technologies under a shared vision. As defined by Microsoft, DevOps is the coalition of products, people and processes to facilitate the seamless delivery of value to the end customers.
DevOps is considered to be the latest advancement of the agile movement. It includes a greater number of stakeholders in the process of delivering value by linking the silo-based teams. It may not be the best practice, but it’s a discipline of people-centric management. Continuous delivery is the technical foundation of DevOps automation, which extends to the agile foundation as well. The close collaboration between the operations and development teams is known as DevOps, which is used to automate the delivery pipeline.
Role of Automation in DevOps to Increase Productivity
Since digital transformation has taken over the current business model, more and more companies are embracing the DevOps practice to drive the automated deployment of software. We are shifting toward an organizational culture in which development and testing take place simultaneously, quickly and more often. This helps maintain consistently high ratings on the quality of software with faster turnaround time. The main agenda of DevOps testing is to derive quality outcomes all along the software development life cycle (SDLC) and beyond. So, with the help of DevOps culture, a new code of development, testing and delivery is introduced in the overall SDLC.
The automation in DevOps empowers companies to:
Enhance the reliability, re-usability and quality of all components.
Save costs of upgrades and maintenance.
Improve the rate of success for standardized processes to ensure the rapid delivery of applications.
Automate Everything — Is It Even Possible?
With the help of the DevOps approach, developers automate and accommodate operational activities, from the source code commit to operational productivity. The primary goal of DevOps automation is to remove bottlenecks in the pipeline by implementing automation in a single level of development. However, DevOps is not a fully automated process and is not operational without human intervention. It is the DevOps testing specialists who are responsible for the process to run flawlessly. Governance at every stage of DevOps testing is still indispensable in various processes undergoing automation. Importantly, human administration is still crucial in handling exceptions to complete the tasks of automation.
The Strategic Role of Humans in DevOps to Ensure a Continuous Pipeline
Remove the Boundaries of Roles
The primary feature of DevOps automation culture is about an enhanced collaboration between the operations and development teams. It has led to a cultural transformation at the organization’s level and inside the teams as well, to encourage continuous delivery. Since shared collaboration is an important aspect of the DevOps culture, it strengthens the collaboration within teams. Before the DevOps approach, the perceived lack of interest by the development team regarding maintenance and operations used to affect DevOps testing and business goals. However, this scenario has since changed due to an organizational shift toward automation and DevOps testing.
The gap between the silos of operations and development no longer exists with the arrival of DevOps. It has removed the boundaries of several organizational roles. Teams are required to focus on the shared goals of DevOps to enhance the end user experience, so the team members need to rethink every investment and decision around this shared goal, which requires human intervention. Automation cannot substitute the decision-making process of humans, for the latter has become vital across the organizational ecosystem as, and when, needed.
Omit One-Size-Fits-All Approach in DevOps Testing
Automation cannot be viewed as the eliminator of roles. The organization still requires expert knowledge and skill set inside your team. The focus should be on developing agile and cross-functional teams that are recognized for removing long-established stumbling blocks between the operations and development teams, and for facilitating collaboration. Developing efficient and functional teams would ensure the delivery of superior quality products.
Continuous Monitoring, Testing and Improvement
One of the key agendas of DevOps testing is to carry out a deeper level of monitoring. Since rigorous monitoring is needed at every layer of the testing process, from integration to infrastructure and transactions to components, it has become a crucial part of the road map of the DevOps approach. In this digitally connected world, monitoring and testing are no longer distinguishable. Continuous monitoring by testers offers a positive response to the user experience while eradicating pre- and post-production obstacles.
Conclusion
With the impetus in the growth of DevOps and agile practices in the past few years, testing has taken a new structure and form. It has evolved into a crucial discipline, and it’s not restricted to a phase of the development cycle anymore. It has become an essential part of the operational activities of production. So, is the role of a tester transformed radically from only a checkpoint to a crucial active participant in the DevOps testing? Teams need to cooperate based on the shared vision of enhancing augmented user experience. We need to understand the fact that automation is not an elixir to deliver on DevOps goals, but merely a part of the DevOps strategy.
How modern product decisions platforms marry the key metrics of features changes to organizational KPIs
In 1967, Joni Mitchell released one of her most iconic songs, “Both Sides Now.” The song was inspired by the book, “Henderson the Rain King,” in which author Saul Bellow writes about flying to Africa and looking out the plane window, “Look at those clouds down there; it’s very strange to look at the clouds from up above them. I remember as a boy having dreamed up at clouds a lot and having seen the cloud from both sides now, I suppose I shouldn’t really be amazed by anything.”
Although not her intention and slightly before its time of popularity, the song speaks quite accurately to the tensions between idealized DevOps and the reality on the ground that many organizations are struggling through. On paper, the core tenant of DevOps is about cooperation, seeing problems and processes from multiple team members’ perspectives from each side, but this is easier said than done.
“But now old friends they’re acting strange They shake their heads, they say I’ve changed Well something’s lost, but something’s gained In living every day.”
The Unicorns and the Rest of Us
Some of today’s most successful and disruptive companies—including Google, Netflix and Facebook—have taken their market-leading positions precisely because they have developed strategy and culture around DevOps. It has been part of the company DNA since the beginning and every employee is bought into it. This means fast, agile development and experimentation is part of the culture.
While many companies would like to capture some of the benefits that unicorns have realized from implementing these development methodologies, for many, this simply is not an option. There is legacy infrastructure, processes and people—who are sometimes the hardest to change. Often DevOps is embraced in small pockets of a company, by teams with newer members working on new projects. The mindset shift that needs to happen to have DevOps—and everything that comes along with it—become part of a company’s culture takes time, commitment and perseverance. However, a good way to start is by demonstrating the fresh, diverse perspectives that can be gained through experimentation and progressive exposure to encourage wider adoption.
Leveling Up
One of the ways modern app delivery, operations and business management teams at organizations all along the adoption spectrum are achieving governance, operational awareness and control at speed is by wrapping new code deployments in feature flags and using a centralized platform to control progressive exposure to users.
By themselves, feature flags are not new. What is new is the centralized and externalized control plane as a service and, newer still, the idea of observing system and business KPIs at the user session or request granularity level for feature flag-controlled decisions.
When something is rolled out to 5% of the users, it is not likely to light up global ops or business metrics dashboards, even if it is a bit broken—and that makes people cranky. If instead, tools are in place to monitor the KPIs within that 5% population and compare it to the other 95% looking for divergence, you have a much more powerful early warning system.
These new centralized platforms also extend observability to stakeholders beyond the few wizards driving orchestration and ops teams watching dashboards. An anti-pattern to the local control autonomy spiel modern DevOps patterns call for, this wider visibility protects against local optimization traps in which one team improves its KPI while degrading other KPIs it doesn’t know or care about.
It can be beneficial to have the idea of key metrics tied to a feature and organizational metrics that look after larger organizational concerns. This inspiration can be traced back to what LinkedIn does with its XLNT experimentation platform, where it calls the organizational metrics “guardrail KPIs.” By always tracking a set of metrics that are critical to the business as a whole, organizations gain a clearer view of the impact that both the granular updates and the larger rollouts are making to the business.
As Joni said in all her wisdom:
“I’ve looked at life from both sides now From up and down, and still somehow It’s life’s illusions I recall I really don’t know life at all.”
No team or organization can know everything all the time, but we can build in visibility to the process of rolling out functionality to our users and create a shared vision of our intentions and the results we achieve. I would argue that if you do look at DevOps from both sides, up and down, you will know the benefits.
Many organizations use a combination of open so Many organizations use a combination of open source and commercial tools and Azure DevOps (formerly called VSTS) and an application lifecycle management (ALM) tool that includes features such as Azure pipelines. This article talks about how to achieve an end-to-end DevOps setup in Azure
Pre-requisites:
Subscription
Project created with version control selected
Process template selected (Agile/Scrum)
Code repo–one on VSTS and one on GitHub/cloud GitLab
The objective of a pre-commit job is to reduce the risk of complexities, duplications and potential bugs in the code to get deployed directly into production.
In usual workflows, the build will be generated on regular intervals and released for testing. However, there is a chance of several code quality issues and building deployment issues.
With the help of pre-commit job, we can configure triggers to generate builds for every pull request made by developers. Also, this enables the development team to create pull request releases that deploy your pull request code or pull request builds, which help to detect deployment/release issues before the code changes are merged.
Pre-commit deployments also help catch deployment/release issues early in the cycle, maintain better code quality and release with higher confidence.
Configuring Agile Board in Azure DevOps for Project Management, and Tracking Steps to Do It
Steps:
With Azure Boards we can plan, track and discuss work across different teams. With Azure Boards, we can swiftly and simply start tracking tasks, features and bugs of our project.
For every project we have to create on sprint, go to Boards and select Work Items to create a new user story and provide the acceptance criteria. Select Boards and create New Work Items, then under Epic select User Story and click on Task and provide the task description.
Setting Up Branches, Version Control and Branch Policies
Steps:
Import the code from VCS.
Under Repos, create branches such as dev or test.
Set up the branches policies to improve code quality.
Improve Code Quality With Branch Policies
Branch policies help teams protect their important branches of development. Branch policies enforce a team’s change management standards and code quality.
Setting up Branch Policies for Review Code
Code reviews are good practice for the most successful software development projects. We have to configure the minimum number of reviewers to review the code and complete the pull request.
The code review done in a pull request is simply to find obvious bugs.
Code reviews help protect the team from bad commits or merges and broken builds that hinder team’s productivity.
The review will help to catch these kind of problems before the merge, protecting our important branches, such as the release branch, from undesirable changes.
Pre-Commit, Enabling Pull Request
Steps:
Workflow to complete the pull request:
Create a pull request
Complete a pull request
Benefits of pull request:
Peer review
Enough testing and better stability
Reducing conflicts
Continuous delivery
Clearer responsibility
Create a pull request:
Pull requests combine the review and merge the code into a single collaborative process.
For every bug fix or new feature in a branch, we have to create a new pull request.
Add the members of the team to the pull request once the development branch build is completed without issues. Then members can review and vote on your changes.
Link work items to your pull request:
Select the Overview tab in Azure DevOps. Select the add link button “+” in the Work Items to relate your current pull request.
Complete Pull Request:
The basic policy requires that some reviewers approve the code with no rejections.
If Allow users to approve their own changes is not selected, the creator of the pull request can still vote Approve on their pull request, but their vote won’t count toward the Require, least reviewers policy.
If any reviewer rejects the changes, the pull request cannot be completed unless the allow users selection is completed, even if some reviewers select Waiting or Reject.
To reset code reviewer votes when new changes are pushed to the source branch, check Reset code reviewer votes when there are new changes.
Nightly CI Build
Steps:
Build Triggers:
We can configure the build triggers to generate war package for every code commit made to the branch.
Under Build select Triggers to enable CI and include master branch.
One can enable the build trigger for every commit made to the branch.
Configure two builds—one for dev branch and one for master branch.
If the developer executes the code, the dev branch build is triggered automatically with help of dev branch polices.
Once the dev branch build is completed, reviewers approve the pull request and the master branch build is triggered. Once the build is completed, it moves to the release part.
The code quality analysis and unit test/Junit test is configured in the dev branch. The code quality gateways such as code review and find bug have been enabled.
Continuous Deployment (Release) through Azure Kubernetes Services (AKS)
Pre-requisites:
Any source code management tool such as Git/SVN or Azure Repo
Microsoft MSDN account
Azure account creation and VSTS/Azure DevOps account creation
Dockerfile to be created and placed along with source code in Git
YAML file to be created and placed along with source code in Git
Kubernetes cluster to be created in Azure portal
Docker hub account or Azure Container Registry
Build definition–Docker build image
Docker File Creation for Application to Deploy
Create the Service Principle to provide the access to the application/Docker image to deploy on Azure Kubernetes services. Select the Deploy to Kubernetes task and make the following configurations:
Create new Kubernetes service connection
Select the command option for deployment
Provide the configuration file location i.e, Deployment.yaml
Select the container registry type
Select the subscription
Select the Replace Tokens task to append the tag number for an image automatically while pushing it to container registry
Perform the deployment and view the logs
Test plans:
Load testing:
Pipeline History
Azure Pipelines is used to automatically build and test your code project and make it available to other users. Azure Pipelines integrates CI and CD to continuously test and build your project and ship it to any target server. It provides the full-stack end-to-end flow of CI and CD through various tools integrated as tasks in Azure DevOps.
Conclusion
With Azure DevOps, you can set up end-to-end DevOps with tracebility, pre-commit, CI and deployment to Kubernetes.
This article was co-authored bySrinivasa Rao Guntupalli, lead engineer at HCL Technologies Ltd. He works on various DevOps technologies such as Git, maven, Jenkins and cloud technologies including AWS, Azure, Azure DevOps/VSTS.
DevOps and the adoption of MSP-based IT operations are both surging trends in the IT industry today. They are also trends that contradict each other in some key respects. Here’s a look at what MSPs mean for DevOps, and vice versa.
DevOps and MSPs: A Primer
MSPs, or managed service providers, are companies that provide IT services on a contract basis. They’re a solution for organizations that want to outsource some or all of their IT needs. In turn, organizations can avoid maintaining IT staff in-house or reduce the size of their IT staff.
DevOps, of course, is an approach to software delivery that emphasizes collaboration between developers and IT operations.
The markets surrounding MSPs and DevOps may both be growing. But that’s not because they feed each other. On the contrary—in one key respect, MSPs and DevOps are at odds.
How? It’s simple. The more your company outsources IT operations to MSPs, the less it can embrace DevOps. That’s because, when IT operations are outsourced, companies lose control over the culture and process surrounding IT. Customers don’t typically dictate to their MSPs which culture, tools or processes to use. As a result, it’s not possible for companies with a heavy reliance on MSPs to “do DevOps.”
To put it another way: You can’t do DevOps if you have no IT Ops department because you outsource IT operations to an MSP. Likewise, if you maintain some IT staff but rely on MSPs to provide IT services that your employees can’t deliver, you can only embrace DevOps insofar as your in-house IT staff can support it.
DevOps-Friendly MSPs
The fact that outsourcing IT operations to MSPs means surrendering control over your IT culture doesn’t mean that you can’t use MSPs and embrace DevOps at the same time. There are plenty of DevOps-friendly MSPs out there. If you’re into DevOps as well as the MSP model, you simply need to find an MSP company that works within the DevOps culture.
Granted, DevOps is more complicated when some or all of your IT operations are run by a third-party provider. It’s more difficult to achieve the level of seamless communication and continuous feedback that DevOps demands when not all of your IT employees are in-house.
But it’s not impossible, especially if you find an MSP who is committed to transparent communication, the sharing of data and collaboration in IT planning—as opposed to one that delivers managed IT services inside a sort of “black box,” without involving the client organization in any of its operations.
Providing DevOps-friendly managed IT services requires more work on the part of MSPs. It also demands more initiative from their clients. But it can be done. And it’s an approach that more and more companies will need to embrace if the influence of both continues to rise within the IT industry.
DevOps is one of those things—like Marxism or baking macarons—that is easy to understand in theory, but much messier to implement in practice. Real-world DevOps rarely aligns with textbook DevOps or the DevOps practices that organizations should follow in theory.
That is true for a number of reasons. Let’s walk through some of the biggest below.
Your Company Is Not Organized in the Way DevOps Theory Assumes
According to DevOps theory, every company has an IT department and a development department. Doing DevOps boils down to making those departments work well together.
The reality, of course, is that not all companies organize their staff or technical operations in this way. Some don’t silo developers and IT engineers by default. Others (especially smaller companies) don’t have developers or IT staff on hand at all; they outsource that work.
These companies can still benefit from DevOps principles such as collaboration and automation. But they should avoid fixating on DevOps teachings regarding the organization of technical work.
Real DevOps Skills are Hard to Find (And Expensive to Obtain)
“DevOps engineer” is now a trending job title. But few technical training programs have caught up with the demand for DevOps.
Most universities still silo majors and courses in software development away from those that focus on IT. Other types of technical training programs function similarly.
What this means is that it is more difficult to hire a DevOps engineer, or other technical staff with a broad set of development and IT skills, than it may first appear. DevOps engineers exist, but they are usually experienced staff who learned their real-world DevOps skills on the job. They’re more expensive to hire and difficult to poach from their current gigs.
Full Automation Is a Pipe Dream
The DevOps mantra tells us that we should automate everything, from software testing and builds to deployment, production-environment monitoring and incident response.
Automation is great in theory. But the reality is that you can never automate everything. Human engineers will still need to respond to problems occasionally, or to unanticipated issues that arise during testing, deployment and production management.
An important point to emphasize here is that even if you successfully automate virtually all of your processes, the mere fact that human engineers must be available to respond when something does go wrong undercuts the idea that you can rely on automation alone. Humans still have to watch the automated processes, even if they rarely need to intervene. The simple act of watching and being available to respond when necessary is a significant burden.
Agility Conflicts With Simplicity and Reproducibility
In some respects, DevOps is a contradiction. On the one hand, it prioritizes agility and scalability. On the other, it emphasizes simplicity and reproducibility.
What this means in practice is that DevOps teams adopt microservices and containers in the interest of maximizing agility. But microservices and containers are much more complex to manage than monoliths and virtual machines.
I suspect the reality is that most DevOps organizations end up erring toward more complex technologies and architectures. They ultimately find greater value in them than they do in keeping things simple.
Still, the idea that you can be agile and be simple at the same time doesn’t apply well in the real world. (And sure, automation, infrastructure as code and the like can help in simplifying the management of complex agile technologies, but they only go so far.)
Most Organizations Still Aren’t Doing DevOps
If you spend your time surrounded by other people who have adopted DevOps and implemented it successfully, it’s easy to assume that the rest of the world is like you.
The reality, however, is that we’re now a decade into the age of DevOps, and yet a majority of organizations have made only halting steps toward implementing basic DevOps processes. If DevOps were half as easy to implement as it appears to be when you read the latest book about how to do DevOps, many more companies would be doing CI/CD by now.
Conclusion
DevOps delivers real benefits, and it’s important to continue preaching the DevOps gospel to organizations that can benefit from DevOps philosophy and practices.
However, it’s important to balance the conversation about DevOps with the reality: Real-world DevOps is much messier than it often seems in theory. Not many organizations have the configuration or resources to do textbook DevOps.
Enterprises don’t own the cloud, just the experience. If there’s an outage or even a hiccup in the performance or network service, it’s on the network and application teams (not the cloud provider) to fix it. But often with problems in the cloud comes finger-pointing, as network and application teams look to determine liability and responsibility when it comes down to crisis situations.
There is no question that applications are growing more complex, as is the infrastructure to power those applications. According to the 2019 “Stack Overflow Developers Survey,” comparing year to year the amount of time it takes for software engineers to become effective on teams is increasing. This trend could be, in part, due to the amount of tooling and rise of an operate-what-you-build model (e.g., the dawn of the full lifecycle developer). As our applications become more distributed—and to cope with scaling and availability demands—skill sets that traditionally were infrastructure and development-based have begun to blur.
Networking Shifting Left
Networking-centric capabilities such as service meshes (e.g., Istio and Linkerd) and container networking interfaces (e.g., Calico and WeaveNet) are gaining a lot of traction and visibility in the cloud-native application development world. Though SDN infrastructure has been shifting left toward development pipelines, application development teams are learning networking concepts. However, the core networking infrastructure is still maintained by the networking teams.
This paradigm of software-defined infrastructure is creating a learning curve for multiple teams. With the dawn of NetOps, software development lifecycle (SDLC) principles are being introduced to the networking teams. Traditionally, ticket- or service-driven networking teams can participate in iterations and they are now building the next generation of cloud-agnostic networking capabilities. With NetOps, there is more interaction and innovation. But change is not without resistance; the dynamic nature of modern infrastructure and the short-lived nature of pre-emptible/ephemeral infrastructure do not bode well for static networking rules.
Networking Team Pressures Today
With infrastructure headed to the trifecta of clustering, replication and load balancing, networking teams have to deal with many more dynamic addresses than before. Static ports and addresses that used to be loaded into networking tickets are now being replaced with wide ranges and service addresses powering the networking stack of our cloud-native and containerized workloads. Adding to the complexity, the networking team is leveraging public cloud infrastructure. The infrastructure that the networking teams are responsible for is leaving the data center and distributed web infrastructure technologies, such as distributed domain name systems (DNS) and content delivery networks (CDNs), are common for the networking teams to be involved with. Like the application development teams, the amount of new technology hitting the networking teams is increasing.
One of the great advantages of using public cloud infrastructure is the ability to quickly expand or contract available resources. With this ability, we can spin up and down test environments to try out changes and even the infrastructure level that was not even possible. With this, we have the ability to integrate more. The application development team, which has been iterating for some time, now has common ground with the networking team to try out iterations.
All in a CIDR
To illustrate the needed collaboration between networking and application teams, let’s look at an example:
In one of the biggest incidents of my career, my company was migrating pieces of our platform to a public cloud vendor. Because the organization was early on in this transformation, a lot of minutiae was yet to be flushed out. We were packaging classless inter-domain routing (CIDR) calculations into our first-time VPC configurations—a task the development team was responsible for at the onset of our cloud migration. It was my job to enter this for my service. Instead of a /8 CIDR, I implemented a /16 CIDR. A seemingly small miscalculation on several configs turned into an incident, blocking a majority of traffic hitting the service. The mistake was quickly corrected, but the incident response framework was well on its way to being executed.
So, who was responsible? I was the one who created the configuration and pushed to production, so this was my fault. But in retrospect, the best solution would have been to involve the networking team immediately in these types of changes. Because the application development teams were vanguarding into the public cloud, however, we cut out the networking team. This was an organizational problem that needed to be addressed. If we were going to deploy into a data center successfully, the networking teams needed to look over my requests and catch my CIDR error.
Silver Lining, Right?
During this CIDR incident, I was trying to look at the silver lining: Since we can’t access our application, that means the instances/app must have a lot of extra capacity, right? Untimely, yes, but the analogy did make sense with the silver lining. We were getting billed by a public cloud vendor and because there was a drop in usage, our bill technically should have been lower. But it still impacted the customer. The money saved in our incident was outweighed by external measures (revenue and brand damage). A cloud outage and a subsequent dip in digital experience can have a massive impact on revenue, averaging $38,855 per hour of outage. During high impact or busier times, this number can be much more.
DevNetOps Nirvana
The adage of bringing everyone to the table early proves to be valuable. Each team in an organization can have a strategic impact and improve architecture in multiple parts of the stack. Different teams have different sets of skills and can have mutual respect for the vertical skills of each of the teams. In the end, a transparent and positive relationship between network and application teams will help with network remediation and outage prevention. And, the closer the teams can work together and the more they can communicate, the more their investments and efforts can contribute to the business bottom line and quicker remediation long term.
How Red Hat’s Open Innovation Labs have helped accelerate Lockheed’s software development for the F-22 Raptor
As the key government contractor charged with helping the U.S. Air Force keep its F-22 Raptor jets in fighting trim, Lockheed Martin Aeronautics increasingly values software delivery speed as much as engine speed. So much of the continued evolution of these fighter jets depends upon continual upgrades of the software code powering all of the physical systems built into them, from tip to tail.
Until a year ago, these jets, which first rolled out of the manufacturing plant in the 1990s, were supported by the waterfall development methods of that same era. Lockheed executives and Air Force brass recognized that the F-22 needed a DevSecOps transformation, and fast.
“When you have a world-renowned platform like the F-22 Raptor, adversaries are constantly looking for ways to counter it,” said Michael Cawood, vice president of F-16/F-22 product development at Lockheed Martin. “This means we must constantly add capabilities and improve the F-22. And we must do it faster than ever before. We needed to transform our own organization and how we did things.”
To accelerate that process, Lockheed last year sent a small contingent of engineers through a residency program at the Red Hat Open Innovation Labs to fast-track the process of implementing an open source architecture onboard the F-22 while adopting agile development methodologies and DevSecOps practices. Since bringing back their learning to the organization, these engineers helped to catalyze a faster development process that has helped Lockheed introduce software features for the F-22 years ahead of schedule.
How Open Innovation Labs Works
Now in their third year of operations, the Open Innovation Labs were developed by Red Hat as immersive training grounds to help its enterprise customers figure out how to deliver better software, faster. A standard engagement has companies such as Lockheed pay for residency for anywhere between three to six developers and operations staff to get paired one to one with Red Hat employees—engineers, consultants and others—with deep knowledge in DevOps and agile development. They work with them in a time-boxed four- to 12-week engagement to build out application prototypes using DevSecOps practices along the way. The residents are also teamed with a designer and an engagement lead who doubles as an agile coach, typically working with the customer’s product owner for the prototype.
Obviously, Red Hat has a vested interest in training these people in technology such as its OpenShift Container Platform, getting them up to speed on how to orchestrate and manage containers and microservices. But the labs approach is in a different league than the standard vendor training program. It more closely follows the DevOps dojo model that enterprises like Target and Capital One are running internally to scale up cultural and process transformations across their IT organizations. Like those dojos, the working model is to build real software through a guided learning process that helps participants get hands-on experience in how to change the way they deliver software.
Red Hat runs the labs either out of a handful of dedicated spaces it runs around the world or within pop-up labs of the customer’s choosing. Besides Lockheed, other notable alumni include the likes of Volkswagen, UNICEF and Heritage Bank. According to a recent analysis by Forrester Consulting, participants in the Open Innovation Labs program are accelerating application development by more than 60%.
Lockheed’s Accelerated DevSecOps Transformation
In Lockheed’s engagement with the Open Innovation Labs, the organization sent out a cross-functional team of five developers, two operators and a product owner. They were tasked with building out a new application for the F-22 on OpenShift, using that process as a model for future development workflows. As a part of the process, Lockheed also worked with the labs to help it build out an internal dojo that could facilitate the scale-out of the knowledge gained by its small team across the entire F-22 development team. Six months after the initial residency, Lockheed managed to scale its OpenShift deployment and DevSecOps transformation across its entire 100-person team.
“By working with the Red Hat Open Innovation Labs team, we changed everything—our toolchain, our process and, most importantly, our culture,” explained Cawood.
As a result, the F-22 development organization has been able improve its ability to forecast for future sprints by 40%. More importantly, it has cut what was once an onerous five- to seven-year process of rolling out new software capabilities for the aircraft. Lockheed is on track to deliver new communications capabilities for the F-22 this summer a full three years ahead of schedule.
As Gene Kim, one of the leading thinkers in DevOps, once said, “DevOps is not about what you do, but what your outcomes are.” What he’s telling us is that DevOps is about enabling a rapid flow of work that transitions from development to testing to operations, but the bottom line is that it’s about achieving results. Unfortunately, this too often gets lost in modern-day DevOps practices.
In this article, I will discuss five common pitfalls that occur in DevOps if you don’t pay attention to the endgame and how your organization can avoid falling victim to them.
DevOps and Automation: Measuring the Wrong Outcomes
Some people believe that, at its core, DevOps is about trying to automate everything. But ask yourself this: Once you automate, are you really delivering a better outcome?
The answer is not always yes. While metrics such as deployment frequency or mean time to repair (MTTR) are interesting DevOps indicators, they are no more interesting to determining if your team is delivering actual value than measuring the number of lines of code submitted or counting the number of keystrokes by the development team. You measure outcomes by determining a baseline from where you start and compare that with where you end.
We need to understand that DevOps is about much more than automation; it’s about the entire pipeline and how your team moves from ideation to delivery to production. Even large companies with world-class DevOps teams don’t always clearly see the problem and can struggle managing the value stream and putting together multiple pipelines. An anonymous but real-life example of this is an organization I know that had been doing DevOps for 18 months, yet hadn’t delivered anything into production in that span. Are they really doing DevOps? DevOps is a process, but if it produces zero value, it’s not delivering on its promise.
The key is focusing on delivering value and measuring value delivery into production. However, before you can begin measuring value, you have to define what it means to each individual deliverable. Common metrics include the number of story points, actual revenue from product comparison and customer satisfaction. With your clearly defined metrics in place, you can measure value as it gets delivered to the customer.
DevOps and Fragile Environments
One Achilles’ heel in the DevOps process is the pre-production environments where code gets deployed and tested. Automating a delivery pipeline relies completely on code being built and deployed to various environments. Fragile environments mean that the developers and testers are wasting time dealing with problems originating from the environment and not the code itself.
While we wish all environments could be on-demand; the fact is, most are not. Another DevOps expert, Randy Bias, has an analogy about pets versus cattle in describing servers, and the same analogy can be used for environments. If environments are not standardized and repeatable, you have to be willing to move on from it like a farmer moves on from livestock; you can’t hold onto sentimental attachments like we have with our pets. What’s most important is that in no case can an environment be singular in nature.
One element of the solution is to create code and automate, but the key is building up environments that are complete. Every environment needs proper builds, proper data and proper tests validating the environment. Additionally, the automation must have the stability to perform successfully every time. If you have piece of automation that fails even just 5% of the time, that’s unacceptable.
The only way automation works in an environment is if you can trust the environment. When the environment is unstable, it’s like playing Russian roulette: No one knows exactly where the danger lies. It could be in the environment, the code, the data, the middleware … a number of places. When a problem arises, developers are often the last line of defense, but dealing with an issue such as this pulls them away from getting real work done.
Disconnected Teams
On this point, I’d like to share a warning: Disconnected toolchains lead to disconnected teams. The planning, development and testing teams can lack cohesion, and the release and operations teams often struggle to be on the same page. Each of these groups have their own tools and management, but they need to be able to communicate with each other effectively to make things work smoothly. In an ideal world, planning initiatives would flow into features for the development and testing team, which would flow into release pipelines and operational changes. Every team still can speak its own language, but the tools need to be interconnected and unified across the teams.
Mismanaging Hybrid
The term “hybrid” has many different uses in the software development space. Hybrid methodology refers to combining waterfall, agile and DevOps practices. Hybrid environments means your environment is housed in multiple locations such as the cloud, on-premises, virtual or code. There’s also hybrid geography, hybrid architecture, hybrid style—the list goes on.
The key to transitioning to DevOps is properly managing the different types of hybrid. If you can’t manage hybrid delivery, you can’t manage your entire delivery and you can’t improve your delivery holistically. Organizations must learn to manage in a world of hybrid delivery and be able to handle cloud and on-prem, as well as teams all over the planet.
Fixing Non-Constraints
The Theory of Constraints is about pinpointing the bottlenecks in your process and fixing them. Any work that’s not focused on eliminating the constraint will not result in improvement and could be considered a waste of time.
It’s important to remember that if you haven’t fixed the bottleneck, you haven’t improved anything. At the same time, if you’re automating the non-constraints, you’re wasting efforts by trying to improve what doesn’t need improving.
As we’ve pointed out, DevOps does not equal automation. If an organization has messy processes and tries to automate them, they’ll be left with an automated mess. Therefore, organizations will see the best results from automating their constraints and just as important, not automating the non-constraints.
