devopsy.kinsta.cloud

Tag: devops culture

Embracing a DevOps Culture, Part 2
In Part 1 of this series, we discussed the fact that more organizations are looking to embrace the “DevOps culture.” However, unlike ITIL, implementing DevOps into an organization does not happen overnight. DevOps is more a philosophy than a procedure, and choosing the right leadership team is as important as choosing the right tools.

In Part 2 we look at other important elements of organizations successfully embracing the DevOps culture.

Without Proper Collaboration, DevOps Initiatives Will Fail

A good test to determine if your organization has truly embraced the DevOps culture is how well your IT department responds when a major incident occurs in the product environment. For example, an e-commerce company’s dramatically slow website, a hospital’s unresponsive electronic medical record (EMR) system, and a malfunctioning airline reservation system. In these chaotic situations, every minute counts and all Dev and Ops hands need to be on deck to keep your organization afloat.

When incidents occur, it’s critical that your organization’s Dev and Ops teams work together to resolve the problem and restore failed service as quickly and efficiently as possible.

Incident Response: The Critical Incident Team

An international company with global operations may have network operations centers (NOCs) in the United States and the U.K. that support more than 50 mission-critical business applications, development centers in China and the United States and a service desk in the United States. When an incident occurs, the company’s service desk is flooded with alerts, ranging from various IT monitoring tools and customer phone calls to open tickets from online users and inquiries from stakeholders. When that happens, the only concern of the service desk is to fix the issue—not tracking down a department head on the other side of the world that may be responsible for the problem.

DevOps should enable an organization to assemble its critical incident team in only a few minutes. The Ops team should know exactly what team(s) to contact for a given incident, and who from those teams is assigned to application support. They should know who is on call based on the time zone and day of the week, and have the correct phone numbers easily accessible.

Next, the team should review the technical details of the issue provided by the incident manager. The universally used DevOps toolset enables everyone to speak the same language, quickly identify the root cause, put a remediation plan together to have approved by the emergency cab, deploy the fix, test it out and confirm service is back up and running.

IT Communication: The Glue That Holds Dev and Ops

To be successful, organizations need to be able to respond as quickly as possible during a major IT incident to limit the negative impact on the business. This is achieved when Dev and Ops works together to create a central communication and collaboration center to ensure proper communication with key stakeholders.

An organization’s ability to effectively mitigate the impact of an IT issue relies heavily on its ability to access and communicate critical information, and to ensure the right people can analyze it and initiate the appropriate actions to keep the business running smoothly. The communication hub equipped with an IT communication solution will facilitate:
- Reaching out to the right on-call people among all the different teams: infrastructure, server, system administration, middleware, network, DBA, QA, support team, service desk and the application developers.
- Because emails don’t wake up people, they will be able to leverage the IT communication solution to reach out to people via multiple channels until they respond (voice, SMS, email, push notification app, paging, etc.), or automatically escalate to the next resolver on the on-call list.
- Providing the right information so the IT resolvers start investigating the issue, identify the root cause and put a resolution plan together without wasting time.
- Get people to collaborate together using the same telecommunication and collaboration tools whatever the time zone and wherever they may be located.
- Contacting a third-party vendor in case the problem is not attributable to the company but caused by an external piece of software.
- Informing the other departments if the business impact grows so big that it affects the company profitability or reputation, such as a cyberattack leading to a data breach for instance. The CEO, legal and marketing may need to be informed to anticipate the consequences.
- Informing end users or customers to limit the number of incoming calls into the help desk.
The benefits of quick collaboration and not just handoff between the teams don’t even have to be demonstrated. Removing the broken pathways, obsolete delivery methods and redundant platforms will improve significantly the organization’s ability to communicate during critical moments. Becoming more efficient and removing the time wasted will automatically have a huge impact on the mean time to know (MTTK) and, in turn, on the mean to resolve (MTTR), hence minimizing the disastrous impacts on the business and keeping the company’s execs sleep at night.

DevOps Culture Can Happen

Embracing a DevOps culture is not a Big Bang change that happens overnight. This is a journey that will be more or less difficult to pursue based on the existing culture and the history and maturity of the IT organization. Even more importantly, it will depend on the level of commitment from upper management to introduce a new culture. Leaders need to embark on this journey keeping in mind that not only will it require new common tools and new common processes, but also they’ll need to convince people of the value of such transformation. To achieve this goal, management should consider IT communication as a key element, which will be the glue that binds Dev and Ops together. And because there always will be room for continuous improvement, the road to DevOps will have no end.

About the Author/Vincent Geffray

Vincent Geffray is Senior Director of Product Marketing at Everbridge with focus on IT Service Alerting & Communications Automation and IoT.

He has more than 14 years of experience in the information technology business, designing, promoting and selling Enterprise IT Operations Management solutions, including Critical Communications, Application Performance Management, IT Process and Workload Automation. He also has international experience as he started his career in Europe. Vincent holds a Master of Science (Mechanical Engineering and Computer Science) and executive certificates from the MIT – Sloan School of Management.

Vincent’s LinkedIn: www.linkedin.com/in/vgeffray

Vincent’s Twitter: @vgeffray
May 10, 2016
State of DevOps Survey – The Ultimate Feedback Loop

If there’s any consistent criticism of the larger DevOps movement, it’s that the talking heads and industry gadflies are driving the conversation, not the people in the trenches.

All the thought leadership panels and “how-to” content (including the written and AV materials on this very site) make for great watercooler discussion, but lack the input of those dev, ops and IT management pros who are actually living the DevOps experience, some critics have said.

Well, fair enough. Talking heads are paid to talk. Thought leaders are chosen to lead the debate at conferences and on webcasts. Bloggers blog. Reporters report. This is simply the way of the world. Certain voices are heard via the channels they frequent. It is, as so many people are fond of saying these days, what it is.

So what about everyone else? What of the people reading this stunning prose and listening to these talks while rolling their eyes based on the connectivity of it all to their real-world experience? How can their voices be heard?

Independent of jumping into the blogsphere or applying to speak at said industry events – which you should, btw – there’s one significant opportunity available to anyone who cares to take the time – participating in industry research.

Over the next few business days, citizens of the DevOps world will have their last chance to contribute to the industry’s most widely read and respected research project, the State of DevOps Report.

If you haven’t done so already, take the time to fill out this year’s survey, and make sure that your voice is heard when the subsequent report is published in June. Sure, it takes a little time, about 15-20 minutes; but that’s because it goes to the trouble of exploring the real-world issues that weigh most heavily on today’s practitioners.

Why is this particular project so important, or even credible? Consider that the minds behind the survey and analysis resulting in the 2016 State of DevOps Report are those authors and talking heads that actually get the most respect.

Backed by DevOps Research and Assessments (DORA), the independent think tank operated by a partnership of Jez Humble, Gene Kim, and Nicole Forsgren – perhaps the most respected voices on the landscape, along with Nigel Kersten, the project is categorically aimed at surfacing the truth about DevOps adoption today – not merely promoting the interests of vendor sponsors.

Like previous iterations of the report, the 2016 edition will focus on topics including IT performance, technical practices, culture, lean management, and organizational performance. This year’s survey also examines several new areas, including the ROI of IT performance, security and compliance, and containers.

To further entice participation, project leaders are also offering free tickets to the DevOps Enterprise Summit, along with forthcoming books from some of the industry’s leading advocates to a lucky few who take the time to get involved.

DevOps success is fundamentally based on the notion of increased collaboration and continuous feedback among the people scattered across the SDLC. Likewise, the legitimacy of this research project is utterly dependent on the input of as many real-world practitioners as possible.

If you’re one of those individuals, and you really care whether or not everyone else involved in the advancement of DevOps understands what this all means to you, do yourself a favor and participate.

Much like an application being built using DevOps practices, this quality of this project will be directly determined by the degree to which anyone with insight is willing to get involved and build toward optimal results.

Take the 2016 State of DevOps Survey… today!

April 15, 2016
DevOps’ Cultural Myths and Logical Fallacies

Remember that guy you knew in college who was working on his fifth or sixth degree with no plans to stop? At some point the idea of real work must have terrified him, and instead what appealed to him was a continual state of learning. If you have the funding to master five or six disciplines, and perhaps not the ambition to get going in any one of them, why not stay and explore the next one?

But if you work in IT, chances are you did not stay in college for more than a few areas of study, and since you entered the work force, your future options dwindled down to two: become a specialist (deep thinking in a singular discipline, but not too broad) or become a generalist (broad thinking across multiple disciplines, not too deep in any one of them).

Managers tend to start their careers as specialists who do so well they are promoted to manager, where their career shifts and they become generalists. From time to time they dabble in a particular area of study to speak the lingo, use the right buzz words and understand what is coming, but the role they play is inherently generalist, as they are forced to deal with a number of topics beyond the realm of any one particular area of study.

We tend to associate the title “architect” as being the specialist who is deep in an area we are forced to contend with. Examples might include, “We need to hire a network architect, or a database architect, or a software architect …” the list continues. When we do this, we have certain expectations. We are looking for the “guru” who can solve our network problems or database problems or security problems—wherever the skill set was that caused us to hire this specialist. If our new employee cannot solve our problems, we begin the search for another who can solve them.

But the candidate has expectations as well: The network guy may have a little knowledge about databases but does not expect to have to perform at the level of his peers in that discipline. If we hire a network guy and then ask him to do everything in every other discipline, we are the ones out of line and he will likely flee our employment in haste, calling us crazy on the way out.

The Cultural ‘Frankenstein’

To create a true master of the multiple disciplines that exist within IT would require creating a modern Frankenstein in networks, servers, operating systems, storage, database, middleware, software and mobility. And, of course, this list is not limited to one expression of each type, but indeed multiple expressions of each type. So for example, our database discipline likely includes deep expertise in Oracle, DB2, SQL Server, NoSQL, MySQL … you name it. If it is popular or relevant to our current stack, our Frankenstein must know that one.

As DevOps has gained in popularity, we have conflated certain aspects of it and driven ourselves off the highway of logic. Too often, proponents have taken the concept of co-location from the agile beginnings of DevOps and applied it to the operational endings of DevOps. Taking down silos is mistakenly applied to co-locating IT operational staff. This conflation occurs billed as the panacea it once was to co-locating business staff, with the developers producing products for them. In the strictly agile context, this works well. Applying it to IT personnel on the back half of delivery is the least effective means of creating Frankensteins. Co-location is a good way to share ideas, from user back and forth to innovator. In general, sharing ideas between a network guy (with limited interests outside his discipline) and middleware guy (who could care less how the network is configured in detail) has limited success.

These operational IT disciplines do interact, but only as they have need, which is significantly less than having to sit in the cube 4 feet away from a counterpart involved in a differing discipline. Co-location is not the cure for breaking down silos or increasing communication—common backbones are. Imagine the example of the medical profession: A hospital provides the common backbone for the neurosurgeon, the orthopedic surgeon and the heart surgeon. These disciplines are inherently separate, and if you have a particular illness, you are going to want the one who specializes in the thing you need done. While each doctor understands general health aspects, they specialize for a reason—a reason you want and need—when it is your time to seek help.

The hospital provides the common facility, the common support staff, the common electronic systems and tools we use. Our medical records are stored in one place, no matter what type of treatment we seek, so each specialist has to enter them there and can read them there. But distinctions by practice remain. There is a special surgical floor for heart surgeries, where they do not conduct orthopedic work or neurosurgical work. The tools may be slightly different; the medical journals are. The training is different. The experience is different. And none of these skills transfer by osmosis of co-location if you put all three doctors in one central cubical cluster. It takes years to get where they are. The same is true of our IT specialists.

If instead of co-location we looked more at common systems for moving, documenting and articulating change. If we gave limited access to our DevOps tooling to the specialists, confined to the niche they have expertise in, then instead of co-location we could have remote-work-concurrence. Experts from all over the country could work on the movement of change without ever physically seeing each other, but with full visibility into what is happening in real time through a common lens. Network operations center (NOC) personnel remain physically in the NOC, but can see the entire stack from that location and change only their portion as changes are required from there.

It is the common backbone of DevOps systems that breaks down the silos of change transparency and change participation. Prior to DevOps we were all using different tools to move change and track change. With DevOps we can begin to articulate change and document it, in the same tool or interface. Meaningful access is granted only to the portions of tools and only where it is needed. Database guys have full access to that platform and nobody else does (other than to see it). The same model works for every other specialty. And before you know it, you have a concert—or rather a symphony of moving parts—instead of disjointed sounds no one can comprehend.

Organizations that hold to the idea that staff will become Frankensteins, equipped with deep skills in all areas because they share cubicle clusters, would do better buying lotto tickets. Where the culture will require attention is in demonstrating to the impacted teams why participating in the common backbone approach has merit for each of them. For engineers who have never had to use “something different” before, this takes work, but is the real win.

When complete, what has truly happened is a conversion to a culture of unity. The work of the genius is still protected in this method; in fact, it is amplified as now it is visible to the entire organization. So even the prima donnas should be happy with the approach. And the ability to orchestrate change never will have been so high, with so much real-time interaction, as after the embrace of a common backbone of DevOps systems. This should offer executives new insights into product creation and evolution. In this sense, the cultural changes reach from top to bottom of the organization, and they are achievable.

To continue the conversation, feel free to contact me.

April 6, 2016
Tales Of DevOps Discovery: Q&A from the Field
DevOps is a buzzed-about industry term, yet many companies are still trying to figure out how to start the journey to a DevOps transformation. Each organization’s journey will be different, but planning and starting the journey always should include assessing the current state of a company’s software delivery process, quantifying the objectives and plotting the “route.” This begins with a discovery process.

This multipart series will review questions commonly asked during the DevOps discovery process, highlight common answers and provide examples of the resulting guidance. It will show that DevOps is really about optimizing software development and delivery on the planes of process, tools and technology, across multiple stakeholders. Based on experience with a broad cross-section of real-world organizations, the series draws on actual interactions that have led to successful DevOps transformations.

Starting the Discovery

Often, organizations embarking on a DevOps transformation want to “pass go” and get straight to, “Tell us how to do it!” Some initial statements frequently heard are:
- “We already do and understand [CI, CD, etc.], don’t waste time reviewing that.”
- “We are practicing [Scrum, ITIL, Six Sigma…]; we are just like everyone else, so there is no need for interview or discovery.”
- “We don’t need to discuss [process, culture, or technology], just tell us what tools we need.”
The eagerness to “just go” helps fuel excitement for the transformation; however, it is important to slow the rush to a solution and start with the basics. This includes:
- Setting a baseline definition of practices: Agile, continuous integration, continuous delivery and DevOps are highly talked-about industry practices that often have multiple definitions and are misunderstood, which fosters resistance or ends up being implemented poorly.
- Starting with foundational discovery: Ask the most fundamental questions about the current state of the organization’s delivery process and the objectives of the DevOps transformation. These questions must be asked of each stakeholder, including the Business, Development, Quality Assurance (QA), Change Management and Operations.
Taking these initial steps can be like pulling teeth, yet they spawn insightful discussion, surface key differences in perspective and begin the process of gaining alignment between stakeholders. With alignment among the stakeholders, shared priorities become clear and actionable.

Once stakeholders align on both the current state and objectives and identify shared priorities, only then is it possible to plot a “route” for the DevOps transformation. At this point a mission statement can be formed, which will act as the North Star or guiding light for the journey. For example, a resulting mission statement may be one of the following:
- We will increase the stability of our releases while increasing Developer, QA and Operations productivity and job satisfaction. This will improve our ability to retain employees, satisfy customers and be more competitive.
- We will do this for our existing and new mobile and web products through the use of iterative development, continuous integration, test automation and continuous delivering applications to a production-like environment.
- We, as Development, QA and Operations, will work as one toward this common objective.
- We will transform incrementally, empirically establishing best practices and expertise within teams and sharing them across the organization.
The mission statement may seem like a trivial part of the process, but in reality it is absolutely vital to achieving the cross-fictional buy-in necessary for DevOps. It works to not only outline the strategy, but also gives the teams something to refer back to during the transformation process.

The other key step to starting the transformation, defining DevOps and related terms, will be discussed in the next post.

About the Author/Brian Dawson

Brian is currently a DevOps evangelist and practitioner at CloudBees, where he helps the community and customers in implementation of Agile, CI, CD and DevOps practices. Prior to CloudBees Brian spent 22-plus years as a software professional in multiple domains including QA, Engineering and Management. Most recently he led an Agile Transformation Consulting practice helping organizations small and large implement CI, CD and DevOps.

Prior to CloudBees, Brian worked at CollabNet, VA Software, Sony Computer Entertainment, Sega, Namco and Apple. His roots are as a C/C++ developer, but his primary job has always been gathering and distributing knowledge and using shared solutions to solve unique problems.
April 1, 2016
Culture Change: Metrics, Tools Make a Difference
Change is in the wind.

This time of the year, as winter weather gives way to springtime renewal, this concept isn’t a cliché but rather a concrete, visceral experience. You go outside, smell the earth defrosting and know for certain that change is happening all around you. You can feel it.

Such is the case on the current DevOps landscape. Seemingly every conversation, with everyone from end users to industry analysts, reaffirms that people are making significant progress toward greater adoption. For an increasing number of organizations, DevOps is maturing from a set of budding goals into a tangible reality.

At the center of this massive shift within the software factory is the issue of cultural change; as we know in all things DevOps, culture trumps all else. So how are organizations affecting, empowering and measuring all this seismic transition, in particular the human elements?

During the recent DevOps Virtual Summit, leading practitioners representing industries ranging from banking to travel and technology shared their views on the progress of cultural reinvention.

From management tips, to tools and even metrics useful for driving transformation, here are some of their observations:
1. What’s the most critical cultural aspect of planning?
  Kazi Whitfield, director, Quality Engineering, MUFG Union Bank, N.A.: Scalability on this is the key; if you’re doing this at a new organization, you need to understand how to scale it, and starting at some smaller level helps you get there. It also helps each individual and those components that are changing to truly understand what’s changing, and why it’s changing; I think that’s a huge part of the conversation that has to happen.
1. What’s the biggest obstacle to cultural change?
  Eric Warheit, senior manager, Environment Management, Southwest Airlines: The challenges that we’re working through are organizational in nature. It goes back to the question of localization, and people maintaining a mindset that they’ve previously had, and what we do to create the conversation around what’s possible and the key impacts that we want to have. Ultimately, it’s about being productive with quality, and creating those common values.
1. How can tools enable cultural change?
  Eugene “Bear” Lehenbauer, group manager, Information Technology, Citrix: If I have to pick one that links them all, a ring to rule them all, that’s when you have an orchestration platform that can take what individual teams are already using and plug right in, or you can take what needs to be automated and automate it; that orchestration harness can now become that common thing that links them.
1. How can you ensure cross-culture collaboration?
  Warheit: Creating that concrete example, that proof case tying analytics to it, so that we have real measurable ways to make good business decisions, and being able to foster that perspective so that we can look across the organization, as opposed to in a localized way, is really critical to be able to foment that collaboration that’s needed.
1. How do you measure cultural change?
  Daniele Romano, product owner, Continuous Delivery Pipeline, ING Bank: The most important thing as an organization is to able to apply an analytical approach, meaning that for each team or department you should look at the current situation and have in mind the target situation, and then have different means to guide everyone toward that target.
1. What’s a key cultural change metric?
  Lehenbauer: We’ve a developed an agility rating scale that allows people to earn scores for something as basic as change management. Based on how those key metrics behave, and there are several of them, but they are fact-based and they’re user impact driven for the most part, they reward the right behaviors. Those types of things can draw out some of DevOps principles results, but also have hard data behind them.
1. How do you recognize advocates of cultural change?
  Whitfield: Being able to phase reward and recognition into the process is key. Organizations are doing a lot of great things and making millions of dollars, and as the individuals that are coding or testing are driving it, you have to understand how you get to those people’s hearts, to make them passionate about what they’re doing. That’s something that becomes key and it’s something that’s different per organization.
1. What’s the best way to gauge successful cultural transformation?
  Romano: There is a really only one critical human metric, which is happiness. So, if engineers are happy and they can deliver fast, at a high quality and with intense focus—that’s a very unique but important metric.
So there you have it, some of today’s most critical elements in driving the cultural transformation necessary to succeed in the DevOps world.

If you missed the DevOps Virtual Summit you can still view recordings of all the sessions including this customer panel, and stay tuned to this space for more excerpts and highlights.
March 30, 2016
The Forgotten Fourth Question
A very well-written paragraph about DevOps appears in Mike Loukides’ book, “What is DevOps”:

… Modern applications, running in the cloud, still need to be resilient and fault tolerant, still need monitoring, still need to adapt to huge swings in load, etc. But those features, formerly provided by the IT/operations infrastructures, now need to be part of the application, particularly in “platform as a service” environments. Operations doesn’t go away, a good part of it becomes part of the development. And rather than envision some sort of Uber developer, who understands big data, web performance optimization, application middleware, and fault tolerance in a massively distributed environment, we need operations specialists on the development teams. The infrastructure doesn’t go away – it moves into the code; and the people responsible for the infrastructure, the system administrators and corporate IT groups, evolve so that they can write the code that maintains the infrastructure. Rather than being isolated, they need to cooperate and collaborate with the developers who create the applications. This is the movement informally known as DevOps.

These days, recruiters to managers, engineers to analysts and, of course, CXOs all are interested in DevOps. They have questions such as, “What is DevOps?” “What are the DevOps requirements?” “What are DevOps implementations challenges?” But the key question is, “Why does DevOps remain difficult to implement?”

Adopting DevOps does not simply mean adopting new processes or toolsets. “Why DevOps” can be understood better by focusing on the benefits it will give you. So let’s dive deeper into one of the key best practices we discovered in our journey into DevOps.

Agile to DevOps

Business challenges, complexities and competition pushed companies to innovate and solve their existing business problems in a better way, adding more features and integrated solutions. Ways of implementing solutions vary from organization to organization, adoption of tools and technologies also differ by small and large organizations based on complexity of problems, but it’s clear that many have adopted agile practices in the past two decades. Agile was made clear by a few well-defined processes such as iteration planning, daily scrum meetings, scrum of scrums (in larger enterprises), iteration demos, retrospective meetings and small, empowered teams.

One of the key mindset changes most of the IT world went through while transforming themselves into agile execution was to have more verbal communications and daily scrum meetings. Daily scrum meetings required the scrum master to ask three questions:
- What was completed yesterday?
- What will be completed by end of today?
- Any blockers?
These three simple questions kept things simple and quick.

As agility is extended to Ops now, there is a need to revisit whether these three questions are enough. There is a need to revisit the attendees in scrum meetings too. There is a need to add some more responsibility to the role of the scrum masters’ communication radius.

Before we look at the fourth question, lets let look at the key reasons to adopt DevOps implementations in organizations:
- Achieve higher speed of time to market without compromising Ops stability
- Better resource utilization across Dev, Test and Ops
- Be more responsive to customers
Why would speed of delivery hamper stability in Ops? In our firsthand experience, the communication between Dev and Ops is not that rich in most of the organizations today. If one studies the root cause analysis of problems faced by Ops in trying to keep the environment stable and healthy, one will find some if not all the following issues:
- Developers are not deployment-aware
- Developers do not think of exposing or exporting the right controls for allowing Ops to configure app dynamically (Unintentionally. They simply don’t know.)
- Many configurations are embedded deep inside the app
- Adding specific third parties, changing their versions is sometimes missed in Ops communications
- Ops is unable to clearly articulate what they require from Dev, even when everyone is co-located
- Continuous firefighting in Ops results in fast fixes, better debugging, but not proactive processes
- Even if weekly meetings between Dev and Ops happen, they do not have a clear agenda or metrics
While there are many more symptoms and their causes, and while there are multiple solutions to fix processes and problems, we have found particularly one solution that seems effective.

We added a fourth question to daily scrum meeting. That fourth question is: “Are you blocking Ops?”

This question is a direct question to each developer in daily scrum meetings. This question essentially means to ask:
- Have you written in any code that requires a new configuration?
- Have you written any code that requires change in existing configuration?
- Have you written any code that is creating a new binary or file that needs to be installed in production?
- Have you recompiled existing code with new third party libraries because older ones are unsupported, even though there may be no other change in functionality?
- Have you hard coded any IP address or other configurations into the code which cannot be changed without rebuilding?
- And so on…
Ask the Fourth Question

The above list of questions is developed based on the root cause analysis of app stability in Ops. This list will be different for different developers/teams. The scrum master communicates the appropriate actions to developers based on the answers to these questions. The scrum master also alerts the Ops team and gives them a heads up on what’s coming their way, and provides timely feedback to the developers.

Is this question important enough to be the fourth question in scrum meetings?

Or is this question covered as part of the “Any blockers?” already being asked? One can debate either way. The blocker question in scrum meetings is thought of in a limited Dev/QA scope, and that scope has not included the Ops team until now. Given that Ops team faces customers, given that the DevOps movement is about improving synchronization between teams, this Ops question needs to be a first-class citizen in daily scrum meetings.

Lastly, it’s critical to to look at root cause analysis-related Ops problem as early as possible. The scrum master becomes more mature as he asks this question to his team while interacting with Ops on hearing them out. If teams are funded enough to have someone from Ops into the scrum meetings, then he/she can be the primary representative for Ops and can steer the delivery to be Ops-friendly.

Irrespective of how one achieves it, taking care of configuration issues proactively in daily meetings, and thinking about Ops on a daily basis in Dev meetings is key to successful DevOps adoption. Configuration issues are not the only focus, but they do form a large chunk of Ops issues.

From our metrics, customers have seen following benefits after adding the fourth question their scrum meetings:
- Shipping code 7x more frequently
- 45 percent fewer failures
- 12x faster ability to recover earlier in the tool chain
Please feel free to share your findings with us @ info@opexsoftware.com.

Happy DevOps!

About the Author/Sanju Burkule

Sanju Burkule, a pioneer, visionary and creator of Products in the DevOps Space @OpexSoftware. He writes about automation and strong, connected, integrated IT and tweets often about these topics.
March 25, 2016
How to Kick-Start Your DevOps Adoption

A recent survey by CA Technologies found Australian organizations are below par in terms of implementing measures to adopt and achieve successful DevOps strategies.

According to Ashok Vasan, vice president of DevOps Solutions Strategy at CA Technologies Asia Pacific and Japan, organizations are generally aware of the DevOps methodology and its value, but they either don’t know how to start or they’ve begun but then failed to move to rapid or broad adoption.

A lot of companies ask CA about the prerequisites for a successful DevOps adoption, he said. It needs to be considered as more than just an IT change, and success requires an understanding and acknowledgement that the business side must lead the transition.

Starting the DevOps Adoption Journey

The business perspective is about launching new products or services and gaining market share. To do that, companies will ask IT to not only deliver certain facilities, but deliver them quickly, so the business should see the adoption of DevOps as a first step in achieving those goals. If senior executives are looking at a particular strategic initiative, there is a greater possibility of getting the required resources across the organization, including IT.

Ideally, that first project will be relatively small, yet visible and significant to the rest of the organization.

It is important to establish a common set of goals and shared objectives, covering areas including development, testing, APIs, product management and product marketing. “Everybody needs to be measured in the same goal,” said Vasan.

Then it is time to look at the IT capabilities and activities needed to make it happen. Where are the bottlenecks? What can be automated? How should testing be performed?

The next step is to identify the weakest link in the existing product or service chain, and work to improve that. If the release process is the bottleneck, then automating that is a good early step to take. Vasan has seen speed improvements of 70 percent to 80 percent from the introduction of release automation.

One of the main reasons for the improvement is that release management can be a complex task, and automation reduces the incidence of errors, sometimes to the point of practically eliminating them. No errors means no time is wasted on remediation, so significant time savings are possible.

Reduced downtime is obviously a benefit to the business, but it can even pay dividends in terms of employees’ work/life balance. Making the release process quicker and smoother means there is no need to do it overnight, on weekends or during holidays.

Indeed, there is a very high return on automation efforts in general, he said. It is important to realize that it doesn’t matter how strong the rest of the links are: For example, even if the marketing people are doing everything expected of them, if a new app receives low user ratings, the organization has failed.

Successful DevOps Needs More than Tools

Back on the IT side, a technology platform should be designed and implemented so the people involved have the right tools, the right scalable technology and so on. A commitment to tooling is necessary to realize the benefits of DevOps, Vasan said. But it’s not just about tech, he warned, but rather a combination of the right processes, the right technology and the right people skills for successful DevOps adoption.

Where an organization is struggling to gain traction with DevOps, Vasan recommends making sure that the process really is business-led; checking that the right employees have been assigned and the right vendors and consultants engaged; ensuring good project management practices; using the lessons learned from initial projects to inform the rest of the organization (possibly by establishing a DevOps center of excellence to spread the word to other divisions and projects); and using an enterprise scorecard to help keep everyone on the same journey.

The rate of change demanded by today’s environment can be an issue for some organizations, said Vasan, but Australian banks have been very progressive and have done a lot to deliver world-class IT products.

Even outside of the banking industry, “there are pockets of [DevOps] excellence … that can serve as examples for others in Australia.” There are “exciting” things happening in the government sector, he added.

CA is working to pass on to a wider audience the experience and best practices developed by pioneers in the DevOps field. “We have seen how companies have started to work in all these [technical] areas, and we take these experiences to our customers.”

March 25, 2016
Experts Weigh In on the Evolution of DevOps

DevOps is an ethereal concept. It means different things to different organizations and individuals. It also is going through an evolution, which means that as quickly as we might think we have a grasp on what it is and how to effectively employ it, DevOps has moved on. There’s no way to accurately predict whether DevOps will even still be a thing in another five years, or if it will have faded into oblivion—replaced by the next hot tech or development model. We can, however, take a look at how DevOps has changed in the past year, and what the immediate future holds for it.

It isn’t an easy question to answer because first we have to agree on what DevOps is. Are we talking about DevOps as a culture or business model, or are we referring to the evolution of the tools and applications generally used in a DevOps environment? I reached out to the DevOps community to get some expert insight.

Current State of DevOps

“The technology question is a common one when it comes to DevOps. We are still faced with the problem of a clear definition of DevOps. So what makes this difficult is … I’m squarely in the camp that believes DevOps is more about culture than tools or tech,” explains Andrew Storms, vice president of Security Services at New Context. “That being said, a well-rounded and mature DevOps organization does require a shift in people, processes and technology. A lot of people will likely point to orchestration of infrastructure as the most important DevOps tool. However, I think the bigger driver of orchestration has been virtualization and cloud.”

One of the defining characteristics of DevOps from my perspective—aside from the IT culture aspect—is the fact that we continue to raise the bar for automation. Each new automation tool further streamlines the process and puts a spotlight on the next bottleneck. Eventually, a new solution is developed to automate that as well.

Chef CEO Barry Crist believes that one of the most important evolutions in the world of DevOps in the past year is continuous delivery (CD)—or the process of rapidly developing software and iterating through quick, small-batch releases. “Continuous delivery was previously exclusive to the web innovators like Facebook, Amazon and Google, but over the last year companies have realized it’s a primary key to unleash an organization’s innovation engine. When the whole stack—infrastructure, compliance, security, microservices and apps—are all expressed as code, IT is able to deliver value more quickly.”

DevOps also has introduced more common ground between developers and security/compliance officers. In high-velocity organizations that have embraced DevOps, IT can deal with security threats more quickly by assessing a vulnerability, creating a fix, testing it and deploying it rapidly before damage is done. DevOps is also starting to automate and define compliance. When compliance is defined as code, the software can be managed and tested like any other piece of code in the organization.

The Evolution of DevOps

Storms stresses that the immediate future of DevOps revolves around it becoming more mainstream—the new normal. “We are already seeing the business value of DevOps. Companies are getting code out the door faster and at a more efficient means. Other departments, like those of security, are starting to see the value of DevOps and looking at ways to incorporate key points into their departments,” he says.

Crist sees DevOps as evolving beyond IT itself. “In the next few years, the DevOps tent will expand beyond ‘devs’ and ‘ops’ to include executives and other business units,” he says. “DevOps is really about building high-velocity organizations. Mature DevOps organizations see both innovation and quality increases that drive more profit and shareholder value, as well as advances in security and safety.”

Only time will tell how DevOps will actually adapt and evolve. I tend to agree with Chef’s Crist that the natural evolution is to take the core principles and practices of DevOps and expand them beyond development and beyond IT to streamline all aspects of the organization. I guess the real question is whether or not we’ll still refer to it as “DevOps” at that point.

March 24, 2016
First We Fixed Dev, But Let’s Not Forget About Ops

Since its beginning in 2008, DevOps has gone from an underground rebellion by developers to a mainstream movement with representatives in every industry and at every scale. You can see the change just by looking at how people attending DevOps events dress: the black T-shirts are still there, but these days there are lots of suits and ties as well. Another hallmark of this success is companies trying to fit DevOps into their existing conceptual frameworks—hence, the plethora of “ITIL and DevOps” papers out there.

Despite all this success, though, the DevOps movement is still young. Of course, things move fast in IT, but we aren’t talking about a technology or a technique that can be adopted rapidly and spread very widely. Committing to DevOps is a major cultural shift for organizations, and that takes time. Some organizations have been doing DevOps for five years and have fully re-engineered their processes around this new approach, but most are still only at the beginning of their DevOps journey. As William Gibson said, “The future is here; it’s just not evenly distributed.”

First Things First: Speed Up Releases

The early phase of DevOps adoption focuses on activities around releasing code to production, because that was the bottleneck that the DevOps movement was originally created to break. While it still may be too soon to declare victory, it’s safe to say the improvement in the release process is enormous. Today, any new project that does not propose to implement DevOps principles would have to have a very good reason and explanation.

Notice that qualification about new projects. It’s easy to adopt DevOps methodologies and expectations in a greenfield situation; however, it is much harder in a brownfield project, where previous cultural norms have been expressed in tooling and processes that are hard to change in-flight. The good thing is that over time, the old way of doing things will die off naturally, as code bases get re-engineered, ported and replaced. Eventually, every project will be a DevOps project.

Are We Running So Fast That Others Can’t Keep Up?

DevOps, however, is not just about release and deployment. It also covers operation of what has been deployed. It’s right there in the name, after all! Coming from an operations background, though, there can be a feeling that DevOps is not for “people like me.” Some simplistic interpretations of DevOps focus on the sometimes adversarial relationship between Dev and Ops, forgetting that good Ops people are just as incentivized as Devs to get Devs more involved in Ops.

One of the difficulties that arise when engineers try to “do the DevOps” without involving Ops is that Ops are not ready to join in. When Dev are doing non-agile releases on an 18-month cycle, Ops have a certain set of assumptions about what a release represents for them. Doing these sorts of monolithic releases is a major pain for Ops for a while, but there will be plenty of time to get ready for it, and other required activities can be scheduled around that release.

One of those important dependencies is preparing the monitoring and management of the code that is being deployed. What needs to be instrumented, where do those feeds need to go, what needs to be in place to make sense of those feeds when they arrive, who needs to jump when one of the feeds goes red, and so on. There is a lot going on in the NOC that Devs are not always aware of.

If Devs suddenly show up with a more Agile cadence of a release every couple of weeks, Ops are simply not going to be able to adapt their existing tools and process on the fly to that new cadence. When you have months of lead time, it’s feasible to write and modify extensive sets of rules and filters to make sense of monitoring data, filter out the cruft, and route the remainder correctly. When your lead time on a release is measured in days or hours—not so much.

Devs should remember that “infrastructure as code” also includes debugging that code. Adopting release automation, containerization or whatever will be the next cool tool will not solve the problem on its own. Bridget Kromhout has a wonderful articulation of this mechanism:

Containers will not fix your broken culture. Microservices won’t prevent your two-pizza teams from needing to have conversations with one another over that pizza. No amount of industrial-strength job scheduling makes your organization immune to Conway’s Law, which states that “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

Leaving aside the communications barriers that all too often still exist between Dev and Ops, one of the constraints that Ops teams have to deal with is that Ops is segmented into many different teams, each of which deals only with one area or one technology. These days, though, problems tend to have symptoms that cross these domain boundaries, and so operations teams have to communicate with each other.

The historical ways of doing this are not very efficient, relying on ticket ping-pong, escalations, war-rooms and good old blame-storming. What do you think happens if you suddenly accelerate a system like that? The answer is that it just torques itself into scrap.

Completing the DevOps Culture Shift

The good news is that others have been where you are now, and there are both tools and techniques that can help you operate what you have developed. In common with the rest of the DevOps movement, it’s as much a cultural shift as a technological one. Ops teams need to start thinking of operations in a new way.

One aspect of that change is the ongoing shift to immutable architecture, also known as puppies vs. cattle. Another is to enable efficient communication across support domains. Today, like in the parable of the blind men and the elephant, support teams often get hold of different bits of the problem, misidentifying them because they only have partial information, and therefore cannot truly resolve the problem. Nobody really has a complete view of the whole elephant, except perhaps for some savant who can be reached only after many escalations—and really has better things he or she should be doing.

If Ops teams can work side by side with Dev, on the other hand, there is a virtuous cycle that becomes possible, with efficient operations removing even more friction from the development process. Communications are the key, whether that means machine-to-machine feeds, human-to-human conversation or machine-to-human interaction. Organizations that get this right reap enormous rewards in both Dev and Ops.

Dominic Wellington serves as Moogsoft’s Chief Evangelist

March 23, 2016
Continuous Improvement: The Key DevOps Distinguisher
DevOps has become a well-accepted and growing component for delivery success, shown in the State of DevOps Report among other respected publications. But even as it becomes more ubiquitous, it’s naive to think every organization will cease having delivery concerns. Implementations will differ widely, and adherence is essentially self-reported. So what is it, then, that separates those organizations that struggle with effective software delivery for years from those that consistently shoot further past their competitors?

The fuzzier nature of the culture and relationships aspects of DevOps obstruct the potential for an ISO- or CMMI-style certification—which is likely a good thing, though it also opens the door for DevOps to be simply a re-branding exercise without any teeth. As a guideline, the CAMS model is still an excellent overview of key DevOps principles. It lays the groundwork for measurement, feedback and sharing that set the stage for improvement, yet stops before making continuous improvement a top-level concern.

The rate of continuous improvement is the metaphorical equivalent of compounding interest. It takes just a simple illustration to show how starting further behind, but focusing on continuous improvement, can take you past those that have more initial resources.

Prioritizing DevOps for Continuous Improvement

DevOps with a focus on Kaizen or continuous improvement can catapult a company’s ability to effectively deliver software. Unfortunately, it falls squarely in the Important/Not Urgent quadrant in Stephen Covey’s model, where good ideas and good intentions are continuously frozen. An inability to keep momentum and traction in this state affects many companies.

To reach our goal of continuous improvement in the DevOps implementation in a company, first we must examine how to successfully drive internal initiatives in general. As a default behavior, companies commonly relegate themselves to just making it through quarter to quarter, delivering mostly incremental customer features. More ambitious initiatives draw lots of excited talk in executive suites, yet often stall and get delayed in actual implementation. Why is that an all-too-common occurrence?

There are three main hurdles:
- Inconsistency of shared goals and priorities across all levels of the organization: There’s often a gap somewhere around either middle management or line management where initiatives can tend to slip. It doesn’t hurt to over-communicate and ensure it goes across all levels, especially when important changes are occurring.
- Lack of individual accountability and empowerment: If communication is adequate, do individuals know not just the goals but also what they can and should be doing to reach them? This aspect is in principle tackled by official systems such as management by objectives (MBO) or objectives and key results (OKR). You can find success or failure in many systems, so rather than debating the merits of various people management systems, make sure to focus on the key goals and actions chosen.
- Do people actually have the budget of time and potentially funding to take action? Without time for action, there is only a nebulous intention—and perhaps a slight feeling of guilt at not having made progress.
Simply identifying the above obstacles and addressing their existence can be eye-opening and likely will cause common-sense measures to emerge to align the organization. After the initial quick wins, there certainly will be more difficult steps to follow that are specific to each organization. The end result is certainly worthwhile.

Unblocking the organizational obstacles to DevOps implementation, not as a one-time project but an evolving program of continuous improvement, blazes the path for long-term success.

As a further thought, imagine you also accelerate the rate of continuous improvement so it’s not a constant. An amazing example of the power of acceleration is the notion of a solar sail: Accelerating its velocity just 1 millimeter per second seems imperceptible, yet after one day, it’s moving 195 mph, and after 12 days, it’s up to 2,300 mph. “Astronomical” is an apt description of the delivery speed of some famous tech startups. That’s the power of continuous improvement!
March 22, 2016