As many industries are going through a wave of digital transformation, a number of key technologies and principles are being deployed to business advantage. One such principle is DevOps.
This collaborative principle is proving to be extremely helpful for businesses in their pursuit of customer satisfaction and beating their competition.
The DevOps trend is growing, as cloud providers including AWS, Azure and Google Cloud are moving faster in upgrading their DevOps services and tools to address business needs.
According to a Google Cloud and Harvard Business Review Analytic Services report, “two-third of companies that use DevOps have seen benefits impacting their bottom line, including productivity (67 percent), increased speed to market (70 percent), innovation (66 percent), customer relevance (67 percent) and product/service quality (64 percent).”
While most companies are already reaping the benefits of DevOps practices, some organizations are still doing their homework. Here are seven tips to help you understand how your organization can use DevOps to its advantage:
Begin with a Small Project
It’s always best to start with a small project before you start engaging DevOps practices on a large scale. A pilot project will enable you to understand the core capabilities of the DevOps process and how you can map them to your business objectives.
Utilizing Open Source Tools
Using open source tools will keep you in touch with current trends and practices. Open source tools are easily available and easy to integrate. You can use these to enhance your DevOps experimentation.
Introducing Security
The DevOps process faces a threat in terms of security, which can be disruptive to an organization’s reputation. Hence, it is advisable for organizations to introduce security at the early stages of the DevOps process. Additionally, it saves a lot of time and resources, which are very crucial aspects of the DevOps process. The concept of DevSecOps is a wiser approach as it includes security all through the software development lifecycle (SDLC) process.
Make the Most of DevOps Best Practices
Looking for a process that drastically improves productivity and reduces time to market? DevOps best practices are designed to increase the pace of software development. With practices spanning site reliability engineering (SRE), end-to-end automation, continuous integration, testing and delivery, proactive monitoring and more, your team will be able to utilize their time effectively and devote their attention to other important business-driving processes.
Enhance Team Skills
The key to bringing out effective production results is having a skilled team that is equipped with the necessary training. Therefore, it is essential for organizations to provide their employees with a proper understanding of the organization goals in general and the processes employees are part of. According to the report, three-quarters of the best DevOps teams said they provide hands-on learning experiences with code labs and projects exercises.
Avoid the Blame Game
DevOps aims to bring about an increased pace of operations and fast learning, among employees. But most of the time for any new process to fall in place, individuals involved have to experience failure. And with the failure of the first project often comes the blame game. However, organizations need to understand that failure is the initial step to achieving future success. Moreover, firms should adopt the policy of “fail and adjust sooner.” The experience of learning from mistakes will take you farther along in your DevOps journey.
A DevOps-Friendly Environment
A DevOps-supportive culture is incomplete without the above exercises. Hence, it is essential that organizations bring about an environment that provides room for shared goals, improved transparency and making growth-driven decisions.
So, industry change-makers: Are you ready for your DevOps experience?
I’ll admit it, this article is a little strange. It is an experiment. But DevOps is all about experimentation, right? This article explores how information and behaviors in fact “flow” through an organization. It is based upon my observations with literally hundreds of DevOps teams. I have used the Three Ways of DevOps to show how and why many organizations struggle with this flow of information and behaviors, and what you can do to identify and prevent behavioral “defects” in your organization.
Simple, Really: Mindsets and Behaviors
IT organizations embarking upon Agile or DevOps ways of working are basically attempting to realize behavior change—simple. Consistent, new “desirable behaviors” that deliver value and reduce risks, wasted effort and costs—simple. This sustainable behavior change demands a change in mindsets and beliefs; it also means unlearning old ways of behaving—not so simple. This requires feedback and correction.
Start with the ‘Why’
Changing mindsets relies on the right messaging. Understanding the “why” and the “purpose” is seen as a critical factor in changing mindsets and ultimately behaviors. The right messaging is not just in terms of words, but also in terms of behaviors, such as people “walking the talk” and “leading by example.”
Desirable behaviors are about people doing the right things. Doing the right thing implies making the right decisions; for example, prioritization of work. To be able to make the right decisions requires having the right information, such as mentioned above—the why. Knowing and understanding the why helps underpin decision-making and prioritization. However, we see many teams that do not fully understand the why behind new ways of working. Somehow the message did not arrive. Somewhere in the flow of information the “why as a justification” ended up as the “why as a question.”
This made me realize that information and behavior flows through an organization. I decided to look at many of the challenges organizations face when adopting new ways of working, by using the Three Ways of DevOps.
The First Way: Flow
The First Way of DevOps is all about the flow from left to right, understanding the flow of work, seeking to increase the flow and never passing a defect downstream.
Understanding the Flow
The messaging I mentioned above flows through the organization traditionally top-down and bottom-up; with the shift to Agile ways of working, this flow is through the end-to-end value stream. This flow is upstream and downstream: Downstream is the flow and hand-offs; upstream is feedback.
I mentioned above that desirable behavior is related to making decisions. If we consider the traditional decision-making authorities in an organization, we have from left to right: strategic, tactical and operational decision-making. This can be seen, for example, as the CIO and business management team (strategic decisions), e.g., product portfolio and key goals; product owners and IT delivery managers (tactical decisions), e.g., product backlog, IT changes; teams and employees (operational decisions), e.g., sprint planning, incident prioritization.
The importance of decision-making was also confirmed in a recent Forbes article, “The Netflix Decision Making Model Is Why They Are So Successful.” Netflix’s desirable behaviors are driven by a set of corporate values. The first value is judgment, which “outlines the company’s decision-making ways and accompanying expected behaviors. Notably, Netflix encourages its employees to ‘make wise decisions despite ambiguity’.”
Every so often, it is decided to embark upon an Agile transformation at a strategic level. This decision is communicated throughout the decision-making levels as an intent—usually the why is vague and left open to interpretation. A set of values or principles are also communicated, such as customer-centric, end-to-end responsibilities and continuous improvement. However, these usually are not translated into desired behaviors. An implied set of behaviors are linked to these values and principles, and can end up being inconsistent with the original intention. This miscommunication of the why and the desired behaviors downstream is a form of a defect—often causing assumptions, driving poor decisions and resulting in undesired behaviors.
For example, as a frustrated senior level manager complains that teams won’t take ownership and frustrated teams protest about lack of empowerment, desired behaviors were not defined and agreed. As a result, they were not displayed, waste occurred, frustration, delays, lack of return-on-value from the effort and behaviors spent.
Another concept of flow is constraints—constraints stopping or slowing down the flow. In terms of behaviors, a constraint could be attitudes and beliefs (not understanding or believing in the new behavior), lack of skills or ability to perform the behavior and allocation of time to learn and experiment or improve behaviors. So, not only must we ensure that we do not pass defects downstream, we also must ensure feedback can identify constraints, impediments and barriers that are stopping the right behaviors from flowing smoothly through the end-to-end stream.
Other types of constraints could be ignorance, indifference, confusion, fear, uncertainty, doubt—or, as Mike Orzen said in a discussion, focus, attention and trust.
So, on one side we have:
Intent, implied, assumptions and inconsistent behaviors flowing from left to right (cause), meeting….
Indifference, confusion, fear, uncertainty, doubt, lack of trust etc. (effect)—you get the picture.
A management solution to this is then to tell people to collaborate, or to tell people they are now empowered—again, without defining the associated behaviors.
“… Application delivery is a set of very human-oriented tasks, a key factor that determines the speed to value is trust,” wrote Carmen DeArdo in his latest book, “A Leader’s guide to Digital Transformation,” co-authored by Jack Maher. “As teams lose trust in the validity of the work as it flows through the lifecycle, a large amount of rework and waste is introduced.” he wrote. Trust is probably one of the biggest enablers or barriers to the smooth flow of behaviors through the system.
The Second Way: Feedback
The Second Way of DevOps talks about shortening and amplifying feedback right to left so that corrections can be made. One form of this is giving feedback when poor information is passed downstream or undesirable behaviors are displayed, enabling us to correct the messaging and the behaviors. Inconsistent messaging, in the form of managers giving conflicting signals or failing to “walk the talk,” are forms of defects. A commonly chosen ABC card (Attitude, Behavior, Culture) in global workshops is, “Don’t do as I do … do as I say.” However, if there is a culture of blame or a lack of safety associated with giving feedback to correct defects in behavior, then feedback isn’t given. Undesirable behaviors and inconsistent messaging goes unchecked.
Failing to stimulate and act on feedback is another form of defect. This means undesirable behaviors continue to occur. All these behavioral defects create cultural and behavioral debt. As this debt accumulates, the ability to deliver value diminishes. In response to this, managers often revert to old management and control styles of behavior (cause), creating more resistance and reducing the willingness to give feedback or experiment (effect). Again, examples of passing defects downstream and creating more barriers and impediments to the flow of desired behaviors.
As suggested in DeArdo’s book, “Feedback should be treated as though it were the most precious of our resources, because it is.”
The Third Way: Experimentation and Learning
The Third Way of DevOps is about continuous experimentation and learning—recognizing that repetition and practice are the prerequisites to mastery.
Agile ways of working are still relatively new. DevOps is still evolving. Old management styles and models are no longer fit for purpose in this new world. Leadership in this new world is also characterized by experimentation. As one CIO stated, “Leaders are expected to walk the talk, but we don’t know what that looks like in this new model.” Leaders often fall back to an S1 leadership style—which is in conflict with the culture associated with Agile ways of working. Poor leadership styles and behaviors, as mentioned above, are also a form of defect—more behavioral debt.
Feedback becomes even more important to validate experimental approaches and to learn and improve. However, the Third Way also stresses the need to allocate time for the improvement of daily work. We sometimes see impediments on team backlogs, but we don’t see behavioral or cultural debt as a backlog item. We don’t see this on the backlog of management teams’ behavioral debt, either. Teams often are frustrated with a lack of time reserved for technical debt, impediments or improvement work—let alone behavioral debt. As Orzen stated, “Even more important than daily work is the improvement of daily work.” Continual learning and improvement should be core skills and behaviors.
As stated in the Third Way, repetition and practice are prerequisites to mastery. There is nothing new about this. It isn’t that the DevOps movement came up with this “new” philosophy: Aristotle, around 300 B.C., stated, “We are what we repeatedly do. Excellence, then, is not an act but a habit.”
We need to practice and repeat behaviors until they become habits, or culture—like “the way we do things here.” Practice also requires feedback to validate improvement gains as a result of practice.
Top athletes practice with the guidance and feedback from coaches. Scrum and Agile coaches are often assigned to Agile and DevOps teams as they practice and grow. These coaches are skilled in training on Agile and DevOps tools, such as standups, retrospectives, value stream mapping, Kanban and Kaizan. But what we see globally is that not all coaches have behavioral management skills.
Another form of feedback from practice is measurement. In terms of DevOps, teams measure metrics such as number of releases—successful releases. We don’t often measure the behavioral change. We can measure the frequency and decisions from standups, but can we measure giving feedback, confronting undesirable behavior? One small example I use is measuring the amount of “yehbuts”—yes, but!—in a dialogue, or worse still, “yehbuts” before somebody has even finished talking. This is an example of a communication defect. Another example is assuming that we understand what somebody has told us without confirming if what we believe we heard was the actual intention.
Communication defects are a major barrier to successful flow of information. Another measurement needs to be related to the impact of behavior change, such as improved customer and employee satisfaction and less rework because of assumptions. Although measuring the actual impact is difficult, having the dialogue itself is valuable: How is what we are doing adding or reducing value to the team and to the organization?
Conclusion
If we want desirable, consistent behaviors focused on creating high-performing teams, we need to look at how messaging and behaviors flow through the system and recognize defects we are passing downstream. Our vague intentions need to be quantified in terms of desirable behaviors. We need to build in feedback mechanisms to signal defects and undesirable behaviors. We need to identify the constraints and impediments that may be causing undesirable behaviors such as fear, uncertainty, doubt—behaviors that may be slowing down or stopping our transformation to new ways of working. We need to recognize that experimentation and practice is needed—knowing that we will struggle and fail, but ensuring we learn and improve as an end-to-end capability. Leadership skills, styles and behaviors need to be aligned to facilitating the three ways of behavior flow. Finally, remember there is really nothing new about this.
Case
In a Phoenix Project simulation, after recognizing the above defects, a team posted the strategic portfolio and goals on the visual management board. The CFO explained these goals and why each backlog item was prioritized against value creation (benefits and value to be realized)—an example of clarity of the why. At the same time, the CFO displayed the right behaviors—being at the standup to explain to the team and ask for direct feedback and understanding.
At the same standup, the CISO and IT team explained the potential “value leakage” backlog items in terms of debts, risks and defects, so that together—business and IT—could make the right decisions on prioritization.
“This made me realize,” said the product owner, “I was now able to make a better-informed decision, because of the team’s input on value leakage—giving feedback from the usage of the applications and the negative impact. It also gave me more trust in the team as they demonstrated an understanding of the ‘why’ and had a clear understanding of value. If, as a team, we did not fully know or understand the value, this created a great dialogue to explore it and ensure we didn’t pass a ‘defect’ (wrong decision on prioritization downstream).”
The team in the simulation learned that as well as visualization of impediments on their backlog, they also needed to visualize the behavioral defects that needed work on and allocated time.
Implementing DevOps in telco IT systems, such as business support systems/operations support systems (BSS/OSS), is not easy. This is especially true with telcos that overly rely on third-party commercial off-the-shelf (COTS) vendors or systems integrators (SIs) to implement new capabilities, and to run operations. To effectively control the vendors, telcos extensively use traditional levers found in SLAs, KPIs and RFx (requests for whatever). They also reinforce the organizational and mental boundaries between the telco and vendor staff. In a DevOps setup, such a traditional approach is not likely to produce the desired results.
How, then, can telcos ensure DevOps success in their organization? Here are five considerations.
Know and Understand DevOps Across the Organization
Time and again, the understanding of DevOps varies from the top-down and across the telco organization. Some top managers believe DevOps is just a “faster version” of the traditional delivery and operations, wherein the vendor executes almost everything. It takes time for the management to fully recognize that the telco will have to take up much more execution responsibility than what it is used to. Having a correct and similar understanding about DevOps is criticat to accept the fact that the nature of the work and the extent of the telco’s involvement will change significantly. The sooner this is done, the better.
Identify Which Applications Fit into the DevOps Model
As part of the DevOps rollout plan, list the applications that can offer the organization significant business benefits through the DevOps model. While some applications, such as eCare or mobile-apps, will fit the bill, some others may not. Highly configurable COTS applications that do not undergo frequent code changes, such as IN or core billing, may not be brought under the DevOps model. For such applications, the traditional approach with traditional SLA parameters could fetch a better cost benefit.
Reiterate that the Mindset Must Change
The telco will find itself in a position where it has to work more closely with vendors than it used to do. Most likely vendor staff will be part of telco’s Agile and Ops team. The telco will have to foster a one-team thinking. For this, the first few steps toward building mutual trust should come from the telco. Do not expect the first few iterative releases to go smoothly. When that happens, don’t get frustrated—see them as opportunities to learn, adapt and fine-tune the DevOps approach within the organization. These are also avenues to reiterate that success/failure are shared responsibility, and that there will be no finger-pointing, regardless of whether a person is an employee of a vendor. In this process, an environment of acceptance naturally develops—i.e. the product owner agrees to product deviations, while the agile team that consists of vendors agrees to fuzziness in the requirements. It is highly recommended that key roles in DevOps be taken up by the telco staff, and they should involve themselves heavily in the scrum workflows.
Obviously, classic-delivery SLA/KPI parameters and values will not fit in such a setup. Those would require a major revamp. An example: Give reward points to the team for on-time delivery and release. The number of points awarded will be fewer if not all the release objectives are met, but the team will get some points. However, if the team doesn’t deliver on time, it will not be awarded any points. Payment and commercials could be tied to the reward points. Similarly, if the telco decides to manage operations through a managed services contract, then it has to look again at the target operating model and SLAs.
Be Prepared to Invest More
Adopting DevOps doesn’t necesarily mean immediate cost reduction. The benefit analysis should cover aspects such as duration of a release and frequency of releases, not just cost reduction. To realize DevOps, telcos will have to incur cost for talent acquisition. Expect an increase in full-time equivalent (FTE) costs, especially for onsite FTEs. Telcos will have to spend on training and reskilling their staff on new tools and technology. Job shadowing from experts could be required until the team gets a good grasp on DevOps. There will be costs in creating new agile teams and creating new reporting structure.
Enabling New Technologies and Tools
It is a given that microservice-based platforms enables DevOps. This is because they truly enable agile teams to work separately with minimum dependencies and undertake continuous integration and deployment, which requires tools. Telcos will have to decide and finalize the platform, tools and DevOps framework. Some aspects to consider while finalizing the platform and tools include their market maturity, ease of maintenance and skills availability.
Conclusion
Habits run very deep. Changing those are not easy. This why most DevOps advocates emphasize that the mindset or cultural shift is the most crucial aspect of implementing DevOps in organizations with a history of traditional delivery and operations. This is especially true with telcos, which run on a large set of legacy systems, both COTS and bespoke.
A survey of 5,558 IT professionals published today by Sonatype in collaboration with CloudBees, Carnegie Mellon’s Software Engineering Institute, Signal Sciences, 9th Bit and Twistlock finds 27 percent of organizations have mature DevOps practices in place, while another 48 percent are still working on improving them.
Despite the DevOps works that remains to be accomplished, however, the survey also noted that 47 percent of respondents still manage to deploy code multiple times a week.
The survey, announced at the RSA Conference, also finds that 25 percent of respondents now consider security to be a fundamental element of quality assurance process. But responses among the 23 percent of respondents who identified themselves as developers suggest implementing best DevSecOps practices remains a significant challenge. For the third year in a row, about half of developers reported they don’t have enough time to address cybersecurity issue.
The survey, however, also noted that organizations that do implement best DevSecOps practices see substantial benefits:
Elite DevSecOps practices are 700 percent more likely to have fully integrated and automated security practices across the DevOps pipeline.
Almost two-thirds (62 percent) of respondents with elite programs have an open source governance policy in place that is improved via automation.
Organizations with elite DevSecOps practices are three times more likely to provide application security training to developers.
A full 81 percent of those with elite practices have a cybersecurity response plan in place.
More than half (51 percent) of respondents with elite practices leverage automated security products to identify vulnerabilities in containers.
Brian Dawson, DevOps evangelist at CloudBees, said the survey results show that despite the gains being made by some organizations, there is much work to done in terms of overcoming technical hurdles and cultural tribalism. For example, 35 percent of all respondents said their No. 1 challenge in application security is that they find out about the problem too late in the process. Nearly half (46 percent) of organizations without a DevOps practice do not have application-level credentials encrypted. Half of respondents (50 percent) with a cloud infrastructure said they rely on their cloud service provider to secure their IT environment, and 28 percent of respondents admitted they do not protect secrets such as passwords, API keys and certificates.
The core issue most organizations need to focus on first is to firmly establish a DevSecOps mindset, said Dawson. Much progress has been made in driving adoption of DevOps practices. Now those principles need to be extended to include automating the implementation of security controls as applications are being built and deployed. The survey noted that 63 percent of organizations that have elite DevOps practices are now informing developers of cybersecurity issues directly within the tools provided to the DevOps team. Armed with that information, it becomes much easier to incorporate security fixes within the overall application lifecycle management process.
It may be a while before best DevSecOps process are widely implemented. But as cybersecurity issues continue to be incorporated within the quality assurance process, it’s now only a matter of time before applications become much more secure.
For most teams, the adoption of DevOps and an agile mindset involves many twists and turns on the path to maturity. It’s not uncommon for teams to experience a “trough of disillusionment” period as:
Initial euphoria and “us-against-the-world” team mentality ushered in by adoption of new work patterns wanes with time.
Client demands and pressures to produce intensify.
Sustained leadership engagement and support fails to materialize.
Key evangelists move on to new roles.
Apologists for organizational status quo never tire.
And before you know it, what once felt like a cultural movement seems mired in the realities of metrics, tools and keeping the operational wheels on.
Pushing through these challenges and self-doubt is a critical step in DevOps maturity that can refine and strengthen a team’s character. Getting to the other side of “the trough” requires a team and its leaders to reconnect with why they embarked on the DevOps journey in the first place, and remind themselves that a lasting shift in behaviors is the ultimate goal. Where behaviors change, results follow.
What are these behaviors? With so much written by thought leaders in the DevOps community, teams have the luxury of expressing the behaviors they intend to model in a way that resonates with their situation. Ultimately four actions, practiced faithfully, will build the team muscle memory necessary to get to the other side.
Choose Wisely
Overcoming the urge to please others and having the courage to say “no” to work that is unlikely to add customer value is perhaps the most important behavior of all. Doing it in a way that preserves credibility and trust is a skill that can only be learned through deeply understanding who your customers are and what problems they are trying to solve. Choose wisely and prioritize ruthlessly. Flawless IT delivery of unnecessary features is a shared failure for both customer and DevOps team alike.
Commit Boldly
Go “all in” on the work you choose to take on. Ensure that the desired outcomes and acceptance criteria of your work are clear before committing, and rely on those closest to delivery to determine how to get it done. Once established, game on! Commit boldly to small deliveries that add customer value and invite accountability for what you take on—knowing that customer satisfaction is the ultimate measure of your success.
Adapt Quickly
Sometimes, we fail. In big and small ways teams employing DevOps don’t always meet their own expectations or those of their customers. Our failures don’t define us; how we respond to them does. Learn. Adapt. Move on. Welcome the changes to requirements you know will come and adapt quickly by leaning in to the power of small and frequent deliveries to place you back on track quickly.
Share Freely
Teams employing DevOps are part of a larger community of like-minded individuals committed to delivering customer value quickly and making their organizations better. Take your obligations to continuous improvement seriously, share best practices freely and solicit expertise to help improve the pace and quality of your delivery and those around you. Helping others succeed has the remarkable effect of making yourself better in the process!
DevOps’ aim is to create efficiencies in the software delivery process. But, it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by consultant Viktor Farcic about siloed departments.
Scenario:
A real obstacle is cultural and caused by having siloed departments. When we start looking at a system as a whole and detecting areas of improvements, the answer tends to be, “This is not our department; we don’t know them, they won’t speak with us.” Those siloes are the main argument in favor of DevOps; it tries to remove them. But, in reality, most companies fake it by creating yet another silo called the “DevOps Department.” DevOps in practice directly contradicts DevOps ideas.
More than a half century ago, while working on compilers, Melvin Conway nailed what is now the crux of the problem we are seeking to solve with modern software development and DevOps practices: “Organizations which design systems,” he stated, “are constrained to produce designs which are copies of the communication structures of these organizations.”
This concept, known as Conway’s Law, suggests how failure to break down silos and organize cross-functional, collaborative teams lingers today as one of the most overlooked steps to implementing DevOps and, despite all best intentions, results in dysfunctional software.
Most companies are organized by function—with business, project management, project owners, developers, quality assurance, app deployment, infrastructure and security groups all focusing on their own areas of expertise.
DevOps initiatives attempt to break down these silos. But it’s easier said than done. Most organizations embarking on DevOps journeys have to deal with an “us versus them” mentality.
Organizational issues create tension, resulting in difficult, friction-laden handoffs between teams. The problems often show up in the software delivery process and in the software itself in the form of quality gaps, inconsistent user experiences and other issues.
Picture this: Input to the application is coming from multiple nonaligned sources, each with its own particular variant on the user experience. If a business analyst and a developer interpret a customer requirement for the customer differently, the end product won’t meet expectations. If you have one team responsible for login and authentication and another responsible for the primary landing page, you can have a login identification that looks and feels different than the core landing page does.
Organizations need to acknowledge that culture will have a direct bearing on DevOps success and commit to taking on those challenges. It won’t be easy. It won’t happen right away. But, with some work, it can get done.
Here are three strategies for removing organizational blockers and avoiding this particular DevOoops.
Alignment – Think about how to best align real or virtual teams organized around products, product functionality or features instead of skills sets or functional groups. For example, construct a small agile team around middleware that includes representatives from the business, developers, quality and operations input.
Communication – Continually encourage communication, trust and transparency. This can be done by using real-time communication tools such as Slack and by centralizing and sharing data using tools such as Jira. Teams need to break down the lingering feelings of distrust. It’s OK to not be perfect. Organizations should set up reporting mechanisms that include successes and failures—all in the name of creating a better final product.
Inclusion – Involve key stakeholders early and continuously. Once you have built these teams, as you embark on your DevOps journey, make sure all stakeholders in the software have a voice. You need to create a common understanding of what their individual needs are and align those around team needs, which sets the stage for ongoing collaboration and communication.
Takeaways
Organize your teams around subjects such as product and functionality and include all the stakeholders in the software development process in these product or feature teams.
Culture in DevOps is just as important as process, practices, tools and technology, yet it’s one of the most difficult things to change.
Takeaways from a recent conference suggest more work is needed in DevOps adoption among organizations
This was a question explored by CIO Platform Nederland and the DevOps Agile Skills Association (DASA) in a recent event I attended, hosted by Transavia. Many organizations have already embarked or are about to embark upon Agile transformation initiatives, driven by growing business demands. This event focused on sharing experiences, learning from the practical examples (successes and lessons learned) from Transavia and ABN AMRO, and exploring the way forward to successfully create a high-performing IT culture that is “Fit for the Future.”
The event was well-attended—unsurprising, really, when you remember the wise words from the late Robert Stroud, former Analyst at Forrester: “To unlock the promise of DevOps, CIOs must lead and support a cultural change within their technology management organization, but as any leader knows, changing institutionalized behavior is the toughest of all management challenges. CIOs must recognize when a trend becomes an imperative for survival. DevOps has become this imperative.”
DevOps and Agile transformations being seen as imperative for business survival. But not just business survival, according to the “2017 State of DevOps” report, which declared, “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations’ digital leadership teams.”
Rik Farenhorst, CIO of Transavia and chairman of the DASA Leadership Forum, presented lessons learned from Transavia. He stressed the importance of changing mindsets and giving equal, if not more attention, to creating desired “behavior” change and not relying too heavily on structure, processes and automation and hoping that the right culture emerges. “People first,” “live your values” and “lead by example” were his key messages, stressing that values must be translated into corresponding desired behaviors as well as unlearning old ways of doing things. Another important discovery was accepting that not all teams are so far and not all teams can mature at the same speed.
A great example of “lead by example” was the leadership team’s daily standups and visualization of strategic priorities, gaining more situational awareness and shared understanding and commitment.
Matthijs Dee and Ivo Carton from ABN AMRO presented their journey from Agile to DevOps, also pointing out the fact that teams mature at different speeds and the need to share lessons learned between teams. ABN AMRO adopted a DevOps playbook that teams could use, containing a kind of toolkit for improving team capabilities. Equally, Dee and Carton noted ABN AMRO’s challenges in moving from “command and control” and traditional hierarchic management models to new leadership styles and behaviors. In ABN AMRO’s case, the organization needed to focus on BizDevOps—it’s not all about code, it’s about delivering business value.
Farenhorst mentioned “values” in his presentation, while Dee and Carton mentioned “principles.” Both of these concepts need translating into team behaviors and leadership behaviors must help empower teams to make this shift.
My presentation, “DevOps Will Fail … Unless,” I revealed the findings culled from more than 400 global teams having participated in the Phoenix Project DevOps simulation and gave a Top Three set of takeaways.
The first was the need to focus on BizDevOps—making the shift from “features” to “outcomes” and from a flow of “code-to-commit to a flow of “idea-to-value.” This was also one of the top challenges for organizations at the most recent DevOps Enterprise Summit.
My second tip mirrored advice from Transavia and ABN AMRO: The need to define, agree and commit to “desired behaviors” and what is effective collaboration (here is an example of what I mean).
My third tip was embedding continual learning and improvement as core behaviors. DevOps is new and all about experimenting—it is a long journey and needs to be done as iterative improvements. At the same time, the pace of change coming at us is increasing, as is the amount of change. Teams need to be able to respond to change. I also quoted Mike Orzen, author of “Lean IT”: “Improving your work is just as important as doing your work”; as such, impediments and improvements should also be made visible on the backlog of work for teams and leaders must ensure adequate time is given and reserved for work-in-progress, for teams to improve.
Following the presentations were three break-out sessions: “Value Creation and Management,” “Organizational Design – Agile/DevOps at Scale,” and “Sustainable Culture Change.”
These are some of the challenges raised in the Culture break-out session I attended as a follow-up to my presentation:
An interesting discussion was around “control” and how to empower the management layers and have them empower the teams. As a CIO, this layer between you and the work floor can diffuse your vision by the time it reaches the shop floor and at the same time leave you unaware of what is happening.
Management by walking around was suggested, as a CIO being visible on the shop floor and soliciting feedback. Listen to concerns and enter into a dialogue about vision, goals and what it means.
Teams often are faced with fear, uncertainty and doubt (FUD) about new ways of working. What will it mean to their job? To their responsibilities? And will they be able to do it? Leaders must remove the FUD and create a safe environment. Safe to take risk, to experiment, to fail, to give feedback. Managers must foster a “no blame” culture.
A study by Google (2012) called Project Aristotle, named in tribute to Aristotle’s quote, “The whole is greater than the sum of its parts,” looked into why some teams work and others don’t. One of the key conclusions was that “psychological safety” was the “it” factor: A sense of confidence that the team will not embarrass, reject or punish someone for speaking up.
Change team behaviors. We adopt these approaches and don’t see the behaviors we expect. Not everybody wants to change. Leaders need to be able to identify the silent killers, what is under the water, what are drivers for working against. At the same time, teams need to be willing and able to work effectively as a team and need team training (not just individual training). Team training also needs to address soft skills such as communication, feedback and collaboration.
Leaders also need new skills in organizational behavior management to be able to identify behavior drivers and how to create sustainable behavior change.
Lead by example was named, but what is that? What behaviors are expected from leaders? Sometimes managers are only one step ahead in this new working. DevOps is new, there is no fixed framework; it is about experimenting what is right for your organization. This is all new.
Ask yourself, “Why are we doing Agile, DevOps, Lean? Are we adopting these approaches as a goal in themselves? Or are they to solve a specific set of problems or deliver a specific set of results? And how does this relate to strategic goals?” This must then be translated into, “What is in it for ME?” for the various stakeholders (business, IT management, teams, customers). Success can only be achieved end to end, which means all end-to-end stakeholders must understand the “why” if they are to commit to the journey.
Build empathy and trust and asking questions to get open feedback—not telling (command and control).
Authenticity: To build trust, be empathetic to FUD; listen to concerns; demonstrate (words and deeds) how these will be taken seriously; and recognize achievements. Empathy must be authentic; if your behaviors are not seen as real, you will lose the trust you are trying to create.
Shift from a (real or perceived) “blame culture.” Failure is OK as long as we learn and improve as a result. If there is no trust, then people fear consequences of making mistakes and are less willing or likely to want to experiment or speak up.
Leaders should show vulnerability, too. Set an example of having failed and learned. Admit in these turbulent times when new ways of working really are new, managers don’t have all the answers—it is a journey of experimenting, learning and improving and giving feedback.
Allow teams time to learn and discover themselves. Recognize that not all teams mature at the same speed, appropriate leadership styles and coaching for teams. Use external (to the team) coaches.
Be prepared for the difficult discussions and decisions. Not everybody can and not everybody wants to change. If you are dependent upon heroes, get rid of them—literally or metaphorically. One example noted was a toxic working culture with teams held to ransom by the hero, while another example was having the hero share knowledge, coach and create more T-shaped capabilities.
Look for “change champions” in teams—peer level respect. Empower them with the right skills and recognize their contributions and the team successes. “Give a podium” to success from desired behaviors such as experimenting, learning and improving.
From the discussions it was clear that C-level managers need to learn new skills and demonstrate new behaviors, while management layers must learn more behavioral and organizational change skills. Teams must not only learn DevOps technical skills, but also team-building and soft skills.
CIO Platform Nederland and DASA will be organizing future events and sharing content and experiences from member organizations to address these needs as a follow up to this kick-off event.
A survey of 336 developers, web product managers and DevOps practitioners published by SolarWinds, a provider of IT management tools, finds they spend most of their time troubleshooting applications—so much so that there is little time for strategic projects such as transitioning to DevOps practices.
Joe Kim, executive vice president and global chief technology officer for SolarWinds, said that while senior IT leaders continue to preach about the critical role DevOps will play in changing their IT culture, IT staff still spends too much time dealing with tactical problems to implement any of the goals defined within a longer-term strategic plan. More than half of respondents (53 percent) said troubleshooting application issues is the top task completed on a given day.
Because most of their time is spent on troubleshooting applications and not on managing and/or deploying applications, Kim said, many employees suffer from low morale. Then, before many IT leaders realize it, their best people are being lured away by the promise of working for an organization that enables them to focus more on the parts of IT process they enjoy, he said.
In fact, Kim noted, in this situation IT leaders that embrace DevOps are in danger of losing credibility. If IT leaders tout the benefits of DevOps, and yet the internal IT staff is spending most of their time on application troubleshooting, chances are good many of the rank-and-file IT professionals will conclude their leaders are out of touch with their everyday reality. IT leaders would be well-advised to engage more with their IT staffs to make sure their strategies are more informed by that IT reality, he advised.
The survey also finds respondents are spending less than 25 percent of their time proactively optimizing performance of their IT environments. Being able to spend more time on this aspect of IT is critical, because if more time is spent on proactively discovering issues, less time must be spent on troubleshooting, said Kim.
IT teams should also be trying to push the boundaries of IT automation to reduce as much of the manual drudgery associated with IT as possible, added Kim.
To get a better appreciation for real state of the IT environment, Kim advised IT leaders gather and share metrics with their teams. Not only does that provide IT teams with some sense of accomplishment as specific goals are achieved, the metrics make it easier to get IT teams to rally behind transitioning to DevOps best practices, he said. That’s crucial because any transition to DevOps is likely to fail because of cultural rather than technical challenges.
It’s unlikely the need to troubleshoot applications ever will go away entirely. The developers that create those applications, after all, are only human. But it is possible to make working in IT a lot more fun and rewarding than what most IT professionals currently experience today.
Carmen DeArdo is well-known in the DevOps space mostly because of his work at Nationwide, where he was DevOps leader and technology director. In August of last year, Carmen moved to Tasktop, where he works with other DevOps luminaries Mik Kersten and Dominica DeGrandis, helping organizations along their transformational journeys.
In this DevOps Video Chat, we speak with Carmen about his move to Tasktop and the culture shifts that are almost universal in embracing DevOps.
The video is immediately below, followed by the transcript of our conversation.
Transcript
Alan Shimel: Hi, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and welcome to a DevOps Video Chat. Today’s guest on our DevOps Video Chat is well-known DevOps personality, Carmen DeArdo, as well as huge Pittsburgh Steelers fan. Carmen, welcome to DevOps Video Chat.
Carmen DeArdo: Thank you, Alan. Great to be here. Glad to be talking to you.
Shimel: Absolutely. So, Carmen, to paraphrase a religious question, this one’s a little different than all other chats we’ve done before, in that we’ve always spoken to Carmen from Nationwide Insurance DevOps team, where you really pioneered a lot of DevOps – not only at Nationwide but set the tone and the course for a lot of DevOps journeys throughout the industry. But now you come to us as Carmen DeArdo as part of Tasktop.
DeArdo: As you said, I always enjoyed, within the DevOps community, talking to other companies. You know, we talked a lot at conferences, where I was a reference customer for some other customer – companies, like IBM, and I was just really intrigued by Mik’s book, which I know we’ll be talking about, and some of the concepts that Tasktop was getting involved with because we were a customer. And I really liked their vision and now I’m here on the Tasktop roster, so to speak, working with customers to help them go through the kind of journey that we went through at Nationwide and also to bring in some of the concept of Mik’s book, which I think can take it beyond, to the next level of what we were doing at Nationwide.
Shimel: That’s fantastic, Carmen, and I – first of all, I wish you a lot of luck in the new position, new sort of career path, and I’m sure you’ll do great because you’re just a great person with this. Well, you mention Mik’s book and I happen to have a copy of it here. It’s called Project to Product. It’s by Mik Kersten. And, not to brag, but I happen to have a signed-by-the-author copy of it, you know?
And I will tell you, Carmen, I was fascinated with some of the concepts in the book and I’ve discussed some of these concepts with people, as part of my devopsy.kinsta.cloud communication, and it really is – this concept of moving from a project-centric IT environment, right, to a product-centric is really – I mean, it’s fundamental, in a lot of ways, right?
DeArdo: Yeah, it is. So I think, you know, we were at DOES London and that’s where they galley copies of the book came out, so the official copy, I think, the coming out party, officially, I think is gonna be in the Vegas conference, and then the book will be available shortly after that to everybody. But – through IT Revolution, but it is a fundamental change and it was something even that Mik and I had been talking about and we had been trying to work through at Nationwide because, before I worked at Nationwide, I worked at Bell Labs for quite a while. And one of the things that struck me when I came to Nationwide was we went – at Bell Labs, we already had this concept of kinda like products, right, because you had your switches, you had your network control points, you had – so work really came to the team. And so we were always working on the next set, for example, way back when I worked on Advanced 800 features, and so we had a team that stayed together and the work came to us and we were very focused on our product, that 800 service concept and then the systems that we build.
So I came to Nationwide, what struck me was almost all the work was done through projects and these projects seemed to have a life cycle, a lot of ’em, that got funded as part of the funding for that year. They started up. You brought all these people to the team. You went through the “norming and forming and storming” kind of parts of the project. And then, once they were done, the project dispersed and the people went. And what you would do – it was very difficult to do – was, if a project touched five or six systems, you would start to allocate – you know, we were using Clarity – two people, three people, a half a person, a quarter person, whatever you thought you needed from those systems.
And so it was a very fragmented kind of experience because the project was temporal; the systems people were all separated and spread out among these projects. Sometimes, a person might be working on a half-dozen projects at the same time. And, you know, you were constantly forming in these teams and so one of the ideas of the product focus is you have a set of people that are focused on that product. They’re gonna live, eat, drink around that product. And it may be more than one team – we’d like to say it’s a two-pizza team that can solve the world’s problems, but we know, sometimes, it’s gonna be more than one team – but still their focus is on the product and now the whole idea of bringing them a continuous flow of work, you’re bringing it to this established product teams.
And the other thing that came up with projects a lot was there was this tension between doing the feature work versus the defects and the risk items and some of the technical debt because project managers were very focused on “You have a budget. You have a set of scope. You’re supposed to deliver that.” So who wants to pay if there’s a vulnerability in a Struts library or making the next thing faster, right? Technical debt is always about – is really about investing to make the next thing go faster, so, if you don’t really have a long-lived relationship with what you’re working on, you tend, just because of the nature of business –
Shimel: You make short-term decisions.
DeArdo: Make short-term decisions. Right.
Shimel: Yeah. I mean, you know, Carmen, that was one of the things I realized in talking to people, about shifting from a project focus to a product focus, and, to me, it was almost – I learned a lesson. I helped take a company public back in the dot-com days – ’99, 2000 – and the difference in how we managed the company as a pre-public, as a private company, to a public company just blew me away. Right? All of a sudden, we were managing in three-month cycles, right? ‘Cause we were managing quarter to quarter ’cause we had to just, you know, go to the street and tell them our results. And you start making decisions that ultimately led to the failure of the company, frankly, because we were making very, very short-term decisions that were maybe good for the short term but not for the long term.
DeArdo: Right. And you know I’m a – you know, we talk about Deming a lot and Deming warned about things like that, right, about the short-term thinking and lack of systems thinking, but you’re right. It drives you that way because the market’s saying, “Well, what’s your next earnings per share? What’s your next report gonna be?” So it does take more – it’s a different way of thinking about things and, really, you’re more in it for the long haul. Now that’s not to say you can’t do things well in the short term because absolutely you can. You can apply all the DevOps practices we’ve talked about. You can run your experiments or ideas of what you think is gonna get value. You can get that feedback much more quickly.
So you can still apply all the things that we’ve talked about and everything, over the last five, six years, but now you’re doing it for the benefit of that product and you can start to tie the business in with how the value’s actually flowing, right? Because, a lot of times, you don’t – a business doesn’t wanna go up and show ’em an Agile team board, with a bunch of different stories, if they can’t relate it to “What’s the value of what I’m getting through this value stream? Right? Where is my value? What’s going on with what I’m trying to do?”
And, with that product focus, it kinda tears down that wall and opens up that black box of IT and says, “Okay, all the way from the idea through deployment, we’re gonna make this work visible and we can see it and we can make decisions and priority calls based on how we’re gonna deploy our resources. And it’s across the features, the risks, the defects, and the investment in debt items so that we can continue to make this product good, not just now but in the future.”
So it seems subtle, but, actually, it’s a very different way of thinking and it even gets back into how things are funded. So I think, in the past, with Agile and DevOps, we stayed away from that left-hand side of the value stream, closer to the customer – we’re kind of in our comfort zone – but now it forces you to engage the business and really have them come along to the journey with you. So that’s another change, I think, in how this applies, versus some of the things we’ve done up to this point.
Shimel: Good. Now – I mean, Carmen, I think you’re so right on with that, though, in that it sounds subtle, but yet it’s so fundamentally different. And I guess one of my biggest fears around it is people are so resistant to change.
DeArdo: Right.
Shimel: And, thought – you know, maybe that it sounds subtle will help sneak it in, right, as a Trojan horse or whatever, but to tell people that they’re gonna have to shift the way they think about this, about the way they go about doing IT in products, knee jerk is “No, we’re not.” Right? It’s –
DeArdo: Right.
Shimel: And that makes it hard.
DeArdo: It makes it hard. And that’s an excellent point, Alan, and I think we’ve been talking about this since I joined, which it’s almost two months now, about the fact that, just like – when I was at Nationwide and gave some of the talks with Cindy and Jim, we talked about the journey. We even had that mountain which showed you climbing to the summit. This is a journey, right? And, if you start out with what you just said and saying, “Okay, you can’t even get started until you do all these things and they involve your business,” you’re right. You’re gonna lose folks right there because “Wait. I gotta change my organization. I gotta change how we do funding. I gotta change this.”
So the way you have to present it and the way I think we’re looking at is we’re taking the customer on a journey. And that journey starts, really, with – and I used to talk about this with you – thinking horizontally. Right? So, when you think of a value stream, a value stream starts and ends with the customer. And if you start to get people to think horizontally and the concept across that value stream and the artifacts that flow from your PPM, your portfolio planning, through your build cycle, through your deploy, and the fact that, in order to get flow, your tools have to be connected, which is where Tasktop essentially was started, was around this concept of connecting your tools, making your tools talk, synchronizing artifacts from tools so, when a defect is created in quality center, it can flow into the backlog of your Agile team. When an instance’s created in ITSM, it can become a defect; it can flow.
So it’s really starting with this concept of “Let’s start with getting the tools talking. Let’s get your artifacts flowing.” And even – and this may sound heresy – even if those artifacts initially are really focused on the work of an initiative or a team or even a project, dare I say, you can still start to utilize the concepts in Mik’s book around the flow framework and see things like lead time, flow time, the flow efficiency, flow velocity. You can still see those things as part of this journey.
Now I think, ultimately, yes, you need to turn the boat more to get that product focus, to get some of the things – advantages we just talked about, but you can start this journey, I think, from wherever you are, as a enterprise, and just take those incremental steps in order to get to the point where then you’re ready to experiment with one product. Right? And then you just wanna start with one product. Pick one product, get one business owner that is interested in this.
And then, from that, you even call it maybe an “experiment,” get some results, and then that’s what starts to change the culture. Right? It’s not gonna be me talking or giving talks; it’s gonna be – you know, I used to joke about this at Nationwide, right? When I would go around to different areas and talk, that didn’t have near the impact of getting one team to actually do something and get results ’cause then that opened people’s eyes and said, “Okay, that is possible. It’s possible right here, right now, and I wanna create my own story.” So you’re absolutely right. It needs to be structured in a way that allows companies’ teams to go on that journey; otherwise, the impact of the change is gonna be too much and you’re not really gonna get any traction.
Shimel: I agree. You know, Carmen, that was one of the first lessons I learned in business a long, long time ago, which is nothing succeeds like success.
DeArdo: [Chuckles]
Shimel: No one wants to jump on a sinking ship; everyone wants to be with the winner. And if you could show that you’re winning somehow, that you’re succeeding, you have upside, everything kinda flows from there. Right? People wanna be associated with that and they’re willing to tolerate and even embrace change, in some –
DeArdo: Yeah. Absolutely. And, you know, and some of the – you know, you can’t just decide you’re gonna wake up today and change culture, right? I mean, you have to do things through what you’re talking about, right? Creating those experiences, those successes, those stories. Stories are very powerful. That’s what ultimately will change the culture, right? Sharing credit. Right?
If it looks like – you know, at Nationwide, if it looked like, “Well, this is Carmen’s show,” that’s not gonna get anywhere. It has to be everybody’s show. It has to be Jim’s show and Jane’s show and Donna’s show and everybody’s show, where they’re now showing how they can be successful. That then really has the impact of changing the culture and showing people that, yeah, this is possible because that team on the fifth floor’s doing this, right? It’s not in this book. It’s not in this conference. It’s that team. It’s that team on the fifth floor and why can’t we do it?
Shimel: Agreed. Agreed. So, Carmen, you mentioned you’ve been at Tasktop now two months. We spoke a bunch about the book, Project to Product, by Tasktop CEO, Mik, but the Tasktop people have really amassed a great Murderers’ Row lineup there. They’ve got you; they’ve got Mik. And I can’t –
DeArdo: Dominica.
Shimel: Dominica DeGrandis, of course. You know, that’s a lot of brainpower. Lot of brainpower over there. What else are you working on?
DeArdo: Well, I mean, it’s obviously a pleasure. There’s a lot of – it’s a great company, it’s a great culture, and Dominica beat me over here by a few months, but to be able to collaborate with folks like Dominica and Mik and then just the whole community, right? Because they were folks that were already in the great community that has been built, that Gene’s built, other folks like yourself have helped build this whole community. I mean, yeah, it’s a pleasure. It’s an honor to be able to talk about these ideas. I mean, Dominica’s concepts of flow and visibility naturally just relate to what we’re talking about.
I thought it was interesting, in the London talk, the London conference, I mean, you go to all these conferences and you can sorta see, over year to year, how things are going and what topics now are becoming more in the mainstream. And there’re a number of talks this year where flow was either in the title or key concept. Jonathan Smart talked about it; Fin had a talk about flow. So flow is kinda where it’s at, I think, now, as people are seeing, “If I’m gonna accelerate my delivery, become more responsive, I gotta understand flow and, as my ex-boss, Tom Payter, would say, where things aren’t flowing.”
And so, with Dominica’s great work that she’s done with making work visible and concepts of flow, her book, Mik’s book now, I mean, yeah, it’s a great opportunity to be involved with just a great company, great culture, and great folks, to see how we can help other companies succeed in the marketplace.
Shimel: I think it also speaks to the coming maturity of the DevOps marketplace, where we’re seeing sort of leaders kinda come together, community leaders come together and combining in ways that it’s sorta synthesis, right? _____ synthesis and I think it’ll lead to yet another wave of innovation, another wave of thinking about these things.
DeArdo: Yeah, I agree. I mean, I think people like Gene have done a great job of bringing people – creating forums for people to come together and the community, as you know and as you contribute to, is very open about sharing ideas. Right? So, for example, I would use – Topo Pal, we both know at Capital One – I would use his example all the time at Nationwide, when people would say, “Well, we can’t do this. We’re not a unicorn. We’re a regulated industry.” I’d go, “Well, let’s see what Capital One has to say about this,” right? They’re a bank. They had a great presentation with auditing, where I think Jennifer Brady was the auditor that came up on stage with Topo and gave a talk.
So it’s just such a great community where people are willing to share and – you’re right – that allows everybody to get better and the whole community is kinda propelling each other forward. So it’s really been a great blessing, honor to be involved, getting to know all these folks and getting to do this kind of work.
Shimel: Absolutely. Well, Carmen, we’re about out of time, but I wanna wish you luck at Tasktop. I think they’re really, really lucky to have you. I’m sure we’ll run into you somewhere on the DevOps conference circuit and we’ll catch up in person. But congratulations on the move to Tasktop. You’ll make us proud, Carmen. Keep innovating. Keep leading. And thanks for being our guest today.
DeArdo: Well, I appreciate the kind words, Alan. It’s always great talking to you and I look forward to keep the conversation going. Thank you very much.
Shimel: All right. Carmen DeArdo – you know, Carmen, I forgot to ask, what’s your title at Tasktop?
DeArdo: My title is senior value stream management strategist.
Shimel: Carmen DeArdo, senior value stream management strategist at Tasktop, our guest today on DevOps Video Chat. Thanks, everyone. Have a great day.
DevOps is arguably one of the most disruptive forces driving transformation of IT processes in recent memory. But a new survey finds that while IT organizations are making progress, adoption of DevOps processes remains uneven at best.
A survey of 2,419 IT professionals conducted by OverOps, a provider of code analysis tools, finds that 82 percent of organizations have partially adopted DevOps practices, while only 17 percent of respondents claim to have fully adopted DevOps. Obviously, not every workload requires an organization to embrace DevOps processes to manage it.
But OverOps CTO Tal Weiss said the survey shows that as organizations increase the rate at which application workloads are being deployed, a fair amount of chaos is starting to reign. According to the results, more than 90 percent of companies deploy code at least once a month, and more than 60 percent release code at least once every two weeks. Nearly 40 percent, however, indicate that moving too quickly is a primary reason errors make it into production.
Worse yet, more than 50 percent say they rely on customers to tell them about errors in production, even though 60 percent of respondents say they use automated tooling for error notification and to ensure application quality.
Not surprisingly, IT teams are still spending an inordinate amount of time troubleshooting their environments. More than a quarter of respondents say they still spend at least roughly one full workday per week troubleshooting errors. Another 42 percent of respondents spend 10 percent to 20 percent of their time troubleshooting.
The good news is that IT is increasingly being viewed as a team sport. A full 67 percent of respondents believe their entire team is to blame when an application breaks or has an error, and 73 percent said both developers and operations are equally accountable for ensuring the overall quality of an application.
Weiss said more organizations need to implement a more structured approach to DevOps based around the same continuous integration/continuous deployment (CI/CD) platform. Too many organizations today are adopting DevOps processes from the bottom up when it’s become clear that top-down initiatives for driving consistent adoption of best DevOps practices are now required. One of the biggest challenges facing IT leaders, however, is identifying and implementing the metrics that will be employed to identify those best practices, he said.
Change, of course, comes hard to any IT organization. It may take IT organizations years after acquiring new tools to define their processes. Inertia surrounding existing waterfall and ITIL-based processes can be incredibly high. But as the amount of software that needs to be deployed and updated continues to increase, risk levels associated with deploying applications also are starting to increase. Many organizations are now essentially experimenting on their end customers. How much tolerance customers have for trial-and-error approaches to applications development and deployment remains to be seen. In the meantime, IT organizations would be well-advised to resolve to do DevOps better in 2019.
