Tag: DevOps Success

  • The Fallacy of Move Fast and Break Things

    The Fallacy of Move Fast and Break Things

    Ever since Mark Zuckerburg uttered the phrase “move fast and break things,” it has become the motto of many development teams. Companies wanting to be the next unicorn decided this must be the way to operate. The race was on to release more, ship faster, never stop. If moving slowly and methodically wasn’t working, doing more had to be the key to success.

    These companies would cherry-pick statistics from reports such as the Accelerate: State of DevOps Report. The 2019 report showed elite companies have 46 times more frequent code deployments than low-performing teams and a 2,555 times faster lead time to move from code commit to deploy to support their initiatives to move faster.

    The problem is moving fast doesn’t work across industries or for all teams. To effectively move fast, you need processes in place to support the velocity. The consequences of moving too fast and not being able to fix things when they break are high. 

    How We Got Here

    What has gotten us to the point where we are moving too fast? In short, we have. How do you feel when it takes months or a week for a PR to be resolved compared to hours or days?  As consumers and end-users of software, our expectations are continually rising. We have to ship quickly because that is what we as consumers expect. As customers, we are pushing companies, which in turn push their employees to meet the increasing expectations.

    What are the consequences when you try to move fast and not just break things, but fail? 

    • It takes longer to resolve incidents.
    • You lose customer confidence and sales.
    • Employees burnout and you have a high turnover rate.

    Let’s go back to the stats from the Accelerate: State of DevOps Report. Yes, elite teams ship faster. But, their changes are 1/7 as likely to fail and they recover from incidents 2,604 times faster than low-performing teams. It’s not just about moving fast and breaking things; it is about having the right systems and processes in place to support this way of working.

    Setting Yourself Up for Success

    You need two things to effectively move fast: a culture of psychological safety and smart investments in tooling. Employees need to feel empowered to speak up if things are moving too fast, if they are concerned about why a feature is being built and to identify gaps in the processes. They need to feel they won’t be blamed when something breaks. Building this requires empathy, open communication and teamwork.  This psychological safety is the foundation of being able to move quickly and quickly recover when things break. 

    Next up is selecting the right tooling and processes. Invest in tools that make things easier. Tools should be useful, usable and change the underlying problems, not create more. 

    Think about the tools in place to quickly resolve incidents when something fails:

    • Observability and monitoring tools to identify and notify when things go wrong.
    • Incident management tools to route, track and escalate issues.
    • Feature management tools to enable circuit breakers and load shedding to turn off features quickly. 

    The culture and tools are part of the equation; the final piece is having the right processes in place to effectively use the tools and support the people. 

    What are some processes you can implement to enable safety at speed?

    You should attempt to move fast, you should try to break things—but only when you have the right protections and processes in place. Using a combination of the right tools and processes, you can deliver more value faster without sacrificing quality or your employees’ health.

    — Dawn Parzych

  • Untapping the Potential of DevOps, Part 3: Keys to Success

    Untapping the Potential of DevOps, Part 3: Keys to Success

    Over the course of this series, we’ve talked about some of the common mistakes we have seen organizations make as they either launch or continue their journey into the world of DevOps. But just as it is important to understand the pitfalls that may happen during a DevOps transformation, it is perhaps even more important to know what the key indicators are that drive success. With that in mind, here are a few tried-and-true tactics for making the most out of your DevOps initiatives.

    Start With Automation

    One of the most powerful areas of impact for DevOps is in driving automation. It accomplishes several things: it increases the velocity of the development pipeline, it can reduce the number of errors or potential errors in code and it often frees up time for developers, allowing them to focus more on creating value and less on repeatable and mundane tasks. By ensuring that DevOps automation goals, including CI/CD, test automation, monitoring, Infrastructure as Code and GitOps are included at the outset of projects, you can better set up your organization for successful software development and delivery—both in the short term and for the long haul.

    Strong Communication

    When it’s all boiled down, DevOps is like any other advanced business tactic as it relies on strong communication to deliver the most success. As the name implies, DevOps engineers must bridge the worlds of development and operations, so to be truly effective they need to possess fluency in both worlds. Can you have success if you have engineers who are only strong on one side of the equation? Technically, yes. But you will not maximize the amount of value from your DevOps initiatives without stakeholders who can adeptly manage both sides. Additionally, communication with the business and other non-technical sides of the organization is critical. The entire organization should be on board with DevOps, with clear visibility into the transformation—your successes and challenges will come from frequent, effective communication.

    Adhere to Processes

    DevOps is process-driven, particularly as organizations scale. To work effectively, and to serve real enterprise environments, creating and adhering to processes is key. While automation is important for the rapid delivery of value, effective change management is important for sustainable development and system operations. Process and compliance are an important part of sustainability for large environments. It is also extremely important for organizations where compliance is vital—think industries such as finance, medical and government. By implementing and adhering to well-defined processes, you can avoid future headaches or rework that will slow down the pipeline.

    Leadership Buy-in

    The most impactful changes have one thing in common: strong leadership. Leadership dedicated to delivering customer value and understanding the software’s prominent role in delivering that value is key. Organizations that cling to outdated methods of development and delivery (i.e. waterfall) lead to slower delivery of value, which is the ticket to extinction. DevOps also benefits greatly from having leaders within the organization who serve as champions for change. By keeping a finger on the pulse of your organization from within, you can better practice change management, identify potential weaknesses earlier on and make your DevOps initiatives work for you.

    DevOps can be an extremely effective way to deliver the most customer value from your software pipeline. By keeping these four key indicators in mind, you can better position your team for success and enjoy the benefits of your DevOps journey.

    — Jonathan Fries

  • Untapping the Potential of DevOps, Part 2: Common Mistakes

    Untapping the Potential of DevOps, Part 2: Common Mistakes

    Many companies are turning to software to differentiate themselves and drive success. But with an increased focus on software comes ever-increasing customer expectations on performance and updates. One of the best structures that IT organizations turn to is DevOps—a strategy that is designed to deliver value quickly and strategically.

    Being successful in DevOps takes careful planning and execution. Is the organization structured to execute on DevOps principles? Is the company aligned in culture and goals? But just as there are commonalities for success, there also are common threads that can lead to failure. When it comes to DevOps, it indeed holds true that failure is an opportunity to learn.

    With that in mind, here are three of the most common mistakes organizations encounter when diving into DevOps.

    Misunderstanding Goals and Definitions

    Many organizations are still grasping at a firm definition of DevOps and what it means for them. Simply labeling existing IT as DevOps isn’t going to make a difference. Furthermore, having clearly defined goals is key for just about any meaningful business endeavor. How can you know that you have succeeded if you don’t have a clear picture of what success looks like?

    Before jumping onto the DevOps train, first understand what it means for your organization and decide on concrete, measurable goals to make sure the transformation is adding value. One of the key benefits of DevOps is it lends itself extremely well to data and measurement. After all, DevOps is about delivering value with velocity. How fast can you get your solutions into the hands of your customers? How close to perfect can you get your releases through automation? This measurability is what makes DevOps so satisfying—embrace it. It is a great feeling to watch the performance of a new streamlined development pipeline improve.

    Fear of Change

    Being fearful of rapid changes and new technologies is a killer of innovation and success. An overly cautious approach with too much manual intervention will stifle the progress of a DevOps transformation. Understand there will be change and the change may be significant for your organization, particularly depending on the current structure and processes. One of the biggest changes may be culturally—DevOps certainly requires different tools and structures, but also requires a change in understanding about how your organization fundamentally views and delivers software. Use DevOps as a catalyst for change—the speed may be daunting but can reap major long-term benefits.

    Overestimating DevOps

    DevOps alone can’t re-calibrate applications or change corporate culture—real leadership is required to push through organizational inertia and bring teams together. This goes back to the cultural change mentioned above. Simply putting DevOps into place and then sitting back and waiting for results will not work. Teams will need DevOps champions, not just at the beginning, but always. DevOps is a constantly growing and changing strategy, built to adapt to customer demands. It holds great power to help teams perform at new levels, but people are always the most important. Empower teams to grow with this new strategy and enjoy the benefits of a sound DevOps strategy.

    Adopting DevOps is necessary to stay competitive in any market. However, while uniting people, tools and processes sound simple enough it can be a much more challenging feat than meets the eye. A DevOps transformation does not happen overnight and bumps in the road are to be expected. However, keeping in mind some of these common challenges can help keep the journey on the right track.

    Bottom line: Take it one day at a time and make sure to optimize a transformation for organizational goals and long-term strategy.

    — Jonathan Fries

  • Five Tips for Solving the Technology Skills Gap

    Five Tips for Solving the Technology Skills Gap

    The insightful, and sometimes breathless, predictions regarding digital transformation are everywhere. We are right to be mindful of both the potential and the challenges posed by digital transformation. Digital transformation is increasing the rate of change in every business, in every line of business and even in every role.  

    Digital transformation requires new development practices, organizational culture changes, customer intelligence and focus, empathy and, above all, new ways of thinking. Only when everyone is committed to improving the way work gets done, can an organization thrive in a competitive environment. As organizations continue to grapple with how to prepare their workforce for an increasingly digital world, they have a technology skills development problem.

    Below are five considerations for bridging the technology skills gap brought on by digital transformation.

    Identify the Skill or Development Problem You Are Trying to Solve

    It’s essential that the technology skills development program be aligned in purpose and design with the intended business objective. That may sound obvious, but organizations have a way of accelerating beyond the original scope and intentions. 

    Ultimately, what you are trying to do will determine the combination of tools, content and even organizational commitment needed to be successful. For instance, are you trying to deepen technical knowledge for a set of employees? Or, are you tasked with improving specific performance objectives in data, DevOps and/or security?

    Before investing in training, it’s important to establish the targeted employees, the range of competencies to be covered, the degree of proficiency required and the time frame to complete the initiative. In all cases, build a program that can solve the problem in a time frame that meets the business needs.

    Establish the Performance Objective 

    For nearly all technology skill problems, the goal is performance: the target learners becoming sufficiently competent to perform at the required level. Understanding what the performance standard or measure of success is or will be helps focus all stakeholders on the same goal.    

    If specific performance objectives are required, monitoring a specific key performance indicator (KPI) can be the most valuable approach to determining training success or impact. If the KPIs are not available, using post-assessments can be a useful measure of knowledge gained and can help demonstrate the impact of the training.   

    Leverage Authoritative Content

    It’s important to understand the sources used to create the course material as well as who authored or collaborated on the material. This matters not just because the learning content should be accurate, but also because adult learners are sensitive to the credibility of the source material. Especially in technical areas, IT professionals expect to be provided with material from authoritative sources.

    IT professionals consider current practitioners, or well-known consultants the most authoritative voices on how to use technology. In some cases, industry associations or trade groups have established themselves as authoritative sources in their specific technical areas. Regardless, the source material should be highly reputable and authoritative. 

    Put User Experience First 

    Making learning as convenient for the target learner as possible is essential to a successful learning program.  

    Convenience has often meant anytime/anywhere learning—learners can take courses at home, while commuting, from their desk or wherever they work. This experience should be much like watching a movie on Netflix or Hulu: Learners should be able to start the course at one location, on one device and continue it at another time on a different device from just the right point in the course. 

    Training can also be seamlessly incorporated into the workflow of some applications. Think of it like a more sophisticated, less annoying version of “Clippy,” the Microsoft assistant that provided in-context help for Microsoft Office through the mid-2000s. For some types of applications and technical roles, having a context-sensitive learning engine that can identify and serve up nuggets of learning appropriate to the task can be very useful. 

    Offer the Right Mix of Push Learning and Pull Learning 

    Employers are beginning to realize the benefits of pull learning where IT professionals identify and select their own learning paths. This model allows motivated professionals to acquire skills, gain more responsibilities and, in some cases, promotions and new job opportunities. Employers have realized that when employees self-select a course, it is a strong indicator of self-motivation, and a strong predictor of professional dedication and success.  

    But pull learning isn’t appropriate for every problem to be solved. For example, maybe you need a group to learn the same thing about cloud security or compliance. In such situations, it’s not practical to wait until the target learners are motivated to take the specific courses. It makes more sense to solve these challenges by assigning specific courses to the right groups of target learners, assigning completion dates and monitoring progress.  

    Depending on what problem you are trying to solve, either push learning or pull learning may be more appropriate. 

    Setting Your Organization Up for Success

    Ultimately, successfully navigating digital transformation requires a team with well-aligned IT roles and skill sets. People are the make-or-break element of a high-performing IT organization. There is simply no replacement for people with the right skills, attitudes and traits. CIOs don’t have the luxury of hiring a totally new roster of people with “the right stuff,” so they will have to be creative in using a combination of hiring, mentoring and training, contractors, and partners to create the needed mix of capabilities. 

    Once you have hired and trained the talent in your organization, what’s your plan to help them continue their learning journey for the duration of their career?

    — Mike Hendrickson

  • DevOps Needs Collaboration and a Safe Place for Success

    DevOps Needs Collaboration and a Safe Place for Success

    When it comes to DevOps adoption, “[many organizations] tell a similar story,” said Scott Van Kalken, systems engineer at F5 Networks. It takes more time and effort to develop the right culture than you probably expect.

    Van Kalken has a long history in tech, including development and operations roles, and has spent several years practicing DevOps. He is part of the open source and meetup communities, and organizes some of the largest DevOps events in Australia.

    “DevOps typically starts in a corner of the organization, where people want to deliver high-quality software really quickly,” said Van Kalken.

    When that team gets good results, the organization decides to scale up its DevOps efforts, but tradition and silos get in the way.

    “At least two things are needed to overcome those obstacles and achieve DevOps success,” suggested Van Kalken.

    Collaboration and Safe Places

    The first thing you need to achieve DevOps success is a collaborative team environment. One of the central ideas of DevOps is you don’t throw software over the wall between functions, such as development and testing, so there needs to be an understanding and acceptance that everyone in the team is on the same side.

    Second is the provision of a safe place to fail and the realization that failure is just an iterative step on the path to implementing the best ideas.

    A financial services company that Van Kalken works with was adopting GitOps but needed to convince the service management team this was the right move.

    “Initially, it was quite challenging because service management thought it [GitOps] was uncontrolled chaos, even though that isn’t the case,” he said.

    Its support was gained after running a series of workshops that showed the underlying processes were actually the same, even though the tools used to implement them were changing.

    Service management is still the final gatekeeper, but it now approves releases in the repository rather than on paper.

    Maturity

    “There is a maturity in accepting change does involve risk,” said Van Kalken.

    Things such as GitOps provide a quick and easy way of reverting to a previous release if something goes wrong. So the “safe place to fail” idea can be implemented in a way that protects the organization in a practical sense as well as staff members in a psychological (and career protecting) sense.

    Another way DevOps can protect the organization is that frequent releases generally involve relatively minor changes, which in turn have a smaller blast radius in the event something goes wrong. Again, this is about having the maturity to recognize risk and dealing with it, rather than trying to avoid it in the first place.

    A sometimes overlooked aspect of increasing the release cadence is the effect on users, who are being asked to cope with frequent changes.

    Van Kalken pointed out that increasing the frequency does not necessarily mean multiple releases a day–it could be from two releases a year to three or four, if that’s what suits the organization.

    Not all changes have a direct impact on users. Some are under the hood, improving performance or addressing rarely-encountered bugs. But when users’ interaction with the system is changed, he suggested canary deployments as a way of checking the acceptability of the new approach among a larger pool of users than those brought into the development process, before it is released to the entire user population.

    “This approach also has a place in DevSecOps as another way of limiting the blast radius,” he said.

    Accepting Failure

    Perhaps the biggest challenge to an organization’s culture is the adoption of chaos engineering because if you’re going to kill a container or flood a network with data in order to check that the wider system can cope, you absolutely need to be in a safe place to fail.

    You also need to realize everyone involved–including developers, security people and those involved in the business side–wants to achieve high availability, and that means designing systems that can handle (partial) failures.

    It isn’t enough to do this in pre-deployment testing. Ongoing testing provides confidence that production systems will continue to cope with such failures.

    “It’s pretty cool when you get it right,” said Van Kalken. However, “your culture has to be OK with this iterative approach.”

    “DevOps is really just a methodology to achieve a better outcome,” stated Van Kalken. That involves collaboration, iteration and embedding security and other considerations up front. But the wider organization needs to understand and tolerate this, and the risk that goes with it.

    “Education, awareness and experience can all contribute to dispelling the traditional view that risk and failure are inherently bad,” he suggested.

    Are the Metrics Appropriate?

    A related point is that whether one outcome is better than another, depends on the metrics adopted. “[Metrics are] probably the biggest things organizations need to change,” said Van Kalken.

    For example, a customer who ordered a pizza doesn’t care whether a particular system involved in taking the order, cooking the pizza and then delivering it, achieved a certain uptime target. They just want the pizza to arrive promptly.

    When that doesn’t happen, he believes a collaborative approach is needed to determine why the delivery was late, exactly what caused the problem and how to quickly roll out a fix to stop it recurring.

    When it comes to practical advice, Van Kalken emphasized the need to get people together early in a project, otherwise there is a risk that sub-groups will demonize each other. In addition, you get more buy-in and everyone accepts they have to help with the pedaling, not just sit back and be passengers.

    “If people aren’t talking, you have a bad culture,” he said. But he acknowledged it can be difficult to get past existing adversarial relationships. Breaking the ice between teams that haven’t previously engaged with each other can also be tricky.

    Returning to an earlier theme, he stressed the importance of adopting an analytical, rather than punitive, approach when failure occurs. Everyone involved must know the organization accepts risk and understands the consequences.

    — Stephen Withers

  • Agile and DevOps Success Needs Continuous Testing

    Agile and DevOps Success Needs Continuous Testing

    One of the central ways to progress an agile transformation is to introduce continuous testing. With all such changes, it is cultural as much as technical. The expected outcomes are less broken builds and reduced time from break to fix, as well as a yardstick for measuring the team’s ambition for improvement.

    The Testing Bit and the Continuous Bit: Testing should be part of any agile developer’s skillset; the shift is to apply it to the whole build pipeline. Continuous implies automation, with the build featuring as a living thing whose state is always changing. The infrastructure options have expanded with the advent of cloud computing and virtual containers.

    Let’s say your teams’ developers work on components, independently buildable and testable chunks of the whole app or service. They receive stories and tasks to improve a customer journey and then add functionality to implement these. And when done, they check their work into your team’s code repository.

    Starting with One Developer, One Laptop and One Test

    The initial testing focus for a single developer (or pair) working on code will be the unit test and the coverage test. Whether writing the test first or code first, the unit test pokes the bounds of the new code and the coverage test checks that all code paths are exercised.

    As an example, maybe the developer has implemented the “customer wants to deposit money” story on an accounts component of a banking system. The tests should check what happens if a user deposits zero, negative or fractional money and the coverage confirms the tests have wandered down all possible the code paths of the new feature. In this way, coverage tests also prevent a developer from adding code without introducing the corresponding tests.

    At this point, it is worthwhile making sure tests are run as part of the build so every time the developer builds the code all the unit tests are run. This is a fairly standard thing for most build systems to do. The bottom line is the chance of checking-in breaking code is greatly reduced.

    Now the Bigger Picture

    The new code worked on your developer’s laptop, the tests passed and they checked-in the code. But in a busy enterprise team, other developers are also working on the same component, maybe even on the same file. This is no different from working on a book chapter with several authors. Your developer worked with a stable snapshot of the code. It is quite possible changes were made by others that conflict with the changes your developer just made. So how do we proceed?

    This is the purpose of the central build. The job of the central build is to check out all code for that component, build it and run all associated tests whenever new code is checked in. In today’s world, the likely path would be to create a container on a cloud server, where the component can be built from scratch, and then all the tests run against it. Traditionally, a tool such as Jenkins would watch for git check-ins and kick off a build. The result is the team always knows if the overall build is green (working) or red (broken). A developer must understand the central build is the true source of truth, not what is happening on their laptop. So, the latest check-in leads to a correctly built component. What next?

    There are three general types of testing needed to develop a software system. Tests that make sure new features work correctly and don’t break the component, tests that make sure the components still work with each other and those confirming the customer journeys your component participates in all work as expected. There would be no point in making a deposit function without a customer journey test that proves a customer can deposit cash with the system.

    The second type of testing, usually referred to as integration testing, needs an environment where all current successfully built components can communicate with each other. Today, that may mean some form of container orchestration with the system’s components placed and managed in their containers. Tests of a customer journey may involve emulating the customer’s manipulation of a website or mobile app, depending on how the application or service is accessed. These are the tests stakeholders are most interested in as they confirm what has been requested in the stories actually work. As with integration testing, these tests may be run less often.

    I Mock Thee

    Mock (or fake) services and data are a necessary part of the testing process, and usually, must be maintained and extended as part of the testing. A feature story may require a developer to spend more time developing or extending a mock service than writing the code it helps test. For example, a mock account with some transaction history is needed to test a new deposit function.

    Live environments can never be entirely simulated in test environments. Sometimes a live service uses a paid for third-party service, and it would be too expensive to use in testing. It is usually the case that the development staff are not permitted to work with live customer data.

    The Build Is Alive

    All the structures mentioned are just products of one simple attitude: “How do I know if this works?” For continuous testing to thrive, engineers should always question whether every part of the system is being tested and whether the tests are being evolved at the same pace as feature development. When tests or mocks are missing, an agile team refers to technical debt. This is as it implies, something that must be paid for later.

    When team members realize they cannot just check-in code and go to lunch without checking the central build, that is a good sign. Information radiators such as large screen displays of the build state should be used to get the extended team to understand the state of the build is everyone’s concern, throughout the lifecycle.

    — David Eastman

  • Measuring Success in DevOps: Top Four DevOps KPIs

    Measuring Success in DevOps: Top Four DevOps KPIs

    There is no news in DevOps concept. Many companies have integrated DevOps to improve and accelerate software development and help drive their digital transformation, and that number continues to grow exponentially. There are now entire tool ecosystems, methodologies and transformation models—as well as endless resources—available to guide companies along the DevOps journey.

    But DevOps success can be difficult to measure. DevOps isn’t a formal framework; it’s more of a culture and a set of practices, and there is limited guidance available to ensure you’re doing it properly or accurately measuring your successes and failures. DevOps also looks different in every organization—no two DevOps shops will be the same.

    However, despite the nebulous definition of what a DevOps organization looks like, there are some core key performance indicators (KPIs) that should be common to all DevOps environments: asset management, monitoring, continuous integration/continuous deployment and continuous security.

    Asset Management

    In this context, asset management means measuring what is automated and how effectively it is controlled. It doesn’t matter if you are using tools such as Puppet, Chef, Ansible or Terraform; what matters is how effectively you use them. These tools allow you to automatically provision bare-metal or cloud assets, as well as join a load balancer and receive traffic. However, you can achieve similar results from Kubernetes or chaining lambda functions. Automation is really about controlling assets to achieve your business goals.

    Whether you’re using on-premises virtualization or cloud workloads, taking the time to streamline your processing cycles down to the minimum hardware required can significantly reduce monthly costs. The higher the percent of your servers, containers, services, etc. are automated, the less time you’ll spend fixing and maintaining your infrastructure. Some of your applications may be uniquely configured to your particular organization, and therefore cannot be automated. However, if applications cannot be automated, it might be time to retire them and build next-generation applications.

    Monitoring

    Should you be monitoring all of your resources? Yes. Should you also filter out the noise? Absolutely. Monitoring agents should be deployed to all of your servers, and if your PaaS tools and other providers have integrations, you should be using them and parsing out unneeded data. The only thing worse than not monitoring anything is monitoring everything without proper filtering.

    Good monitoring should always be coupled with utilizing the information acquired. Capturing logs, parsing them and using tools such as ELK, Datadog or Sumo Logic, for example, give you a large kit that can not only capture and glean useful information from a tremendous amount of data, but can also make decisions and perform predictive analyses. You might learn your stack is operating inefficiently, or not scaling fast enough during peak utilization periods.

    Disparate systems should be aggregated into a common tool, lest you spend valuable time combining information across systems or pulling information from data warehouses to get your insight. Let the tools do that work for you. You should also be using tools such as Logstash to pull and understand application, database and other logs in your stack for meaningful data. If not, you’re limiting your insight into the service’s health.

    Continuous Integration/Continuous Deployment

    Continuous integration and continuous deployment (CI/CD) is the difference between releasing software once a quarter (or year) and performing dozens of releases to production within a day.

    But it’s one of the most difficult indicators to measure, since every environment is different. It is important to understand how many code commits don’t make it past your integration environment. These blockages could be the result of poor programming practices, poor code quality or poor testing. However, if features stay in dev or integration for too long, there is surely an underlying issue. In most cases, it centers around automated testing.

    So, how do you measure good testing? This is one of the most difficult questions enterprises face, especially when automated testing is added after monolithic applications are already written. Essentially, good testing is about creating good business logic and integrating testing that meets most use cases, while covering each function and class. Immediate feedback on failed or successful tests expedites the automation process. The less automated testing is built, the more QA has to test by hand. Failing to integrate good business and integration testing slows down the entire CD part of the pipeline significantly. Also remember shooting for 100% test coverage is unrealistic—over 80% should be sufficient.

    It’s also important to remember applications that are not in a pipeline and must be manually tested will never keep up with the pace of cloud-born innovation. If you are dealing with a giant, monolithic legacy app, it’s time to design its replacement (starting with testing). Pipelines should also have notifications built into them, with chaining logic to react to failures. Tools such as Jenkins, CircleCI, TravisCI and most code repository solutions, such as GitLab, offer rich feature sets to aid in pipeline creation and management.

    Continuous Security

    The easiest way to ensure security compliance is to shift left and resolve security issues in the development stage. Too often, security is applied at production, which means it’s not part of the environment’s end-to-end process. Securing applications and networks at the development level will give you more confidence that applications will interoperate properly at the production level.

    After shifting left, make sure you continuously deploy and monitor your clusters, nodes and pods in a secure manner. Ideally, you have a tool that provides a real-time summary of your cluster’s compliance and security status by looking at workload security and governance checks, cluster worker node CIS Kubernetes Benchmark checks, cluster Ingress controller security best practices and istio security checks.

    DevOps Success Never Ends

    Success in measuring your core DevOps KPIs is not about getting to full automation immediately—it’s about understanding your biggest issues, continuously securing them and improving them over time. DevOps has no end; success requires continuously measuring your progress and improving your environment with the most effective and up-to-date tools. Take control, implement thorough and meaningful monitoring, deliver with confidence and do it securely.

    — Gadi Naor

  • Survey Finds Wide Gap Between DevOps Adoption and Success

    Survey Finds Wide Gap Between DevOps Adoption and Success

    A recent HBR survey shows there is work to be done to advance DevOps in organizations

    A survey of 654 subscribers to the Harvard Business Review found that while 86 percent of respondents said it is important for their organizations to be able to build and deploy software quickly, only 10 percent said they are successful at it.

    Conducted by Harvard Business Review Analytic Services on behalf of Google, the survey found nearly half (48 percent) of respondents said their organization always relies on DevOps methodologies when deploying software, while another 21 percent rely on DevOps selectively. An additional 8 percent said their organization plans to adopt DevOps practices.

    Among those respondents employing DevOps practices, the top benefits cited were increased speed to market (70 percent), productivity (67 percent), customer relevance (67 percent), innovation (66 percent) and product/service quality (64 percent). Despite those benefits, however, most organizations still have a significant amount of work to do in terms of implementing best DevOps practices—the three biggest hurdles organizations face are organizational silos (50 percent), legacy technologies (49 percent) and resistance to change (46 percent).

    Melody Meckfessel, vice president of engineering for Google Cloud Platform (GCP), said these issues are precisely why Google is sharing how it operates in the form of site reliability engineering (SRE) principles, which are a prescriptive set of guidelines for implementing best DevOps practices. Not every organization needs to implement those principles to the same level, but just about every organization can benefit from adopting a set of well-defined engineering principles, said Meckfessel.

    The challenge many organizations face is the fact that most of the IT staff are not engineers by trade. Most IT staffs are made up of administrators who tend to have limited programming skills. But Meckfessel said even IT organizations that don’t necessarily have a lot of engineering expertise still can benefit from automating operations based on the SRE principles.

    Many of those organizations will need to slow down the rate at which they are building applications today to be able to speed up tomorrow. It takes time for organizations to master the cultural change required to implement DevOps practices. But once they begin to master DevOps, the benefits to the organization compound quickly. Meckfessel said two of the most important attributes of DevOps is adopting an immersive approach to training and establishing a no-blame culture within the IT organization when it comes time to conduct post-mortems. That latter goal is especially difficult to achieve sometimes, because most interactions between developers and IT operations teams historically have been less than positive.

    Meckfessel also noted it’s critical now for organizations to incorporate cybersecurity into its DevOps processes as well, which requires not only additional training for developers but also closer alignment with cybersecurity teams.

    It’s now only a matter of time before the complexity associated with emerging technologies such as microservices based on containers force many organizations to dive deeper in DevOps. The real challenge now is charting a path of least resistance that also generates the maximum level of benefit for all concerned.

    — Mike Vizard

  • The Top 5 Ways DevOps Fails – and How to Prevent Them

    The Top 5 Ways DevOps Fails – and How to Prevent Them

    The working hypothesis for a Gartner report on how to avoid failure with your DevOps effort is that, unless the trajectory changes, “Through 2023, 90 [percent] of DevOps initiatives will fail to fully meet expectations.” The reason? You could boil it down to management not fully understanding DevOps tenets and nuances—things such as targeting business value, measuring success, meeting cultural/organizational challenges, fostering collaboration, understanding the inherently iterative process and managing for realistic expectations.

    DevOps is about people from multiple disciplines coming together to solve problems streamline workflows, develop new processes, break down silos to develop business value. “A lot of people fixate on the tools,” said George Spafford, senior director analyst, Gartner and co-author of the report. “They identify best-of-breed, lay out their toolchain and say, ‘Ta-da!’”

    The tools are important, but they are not the difficult part of DevOps. They are not a key obstacle. To succeed with DevOps, you often have to do something much more difficult: Change mindsets. Gartner identified these five top reasons for the failure of DevOps initiatives:

    1. Doing DevOps for DevOps’ Sake

    It’s important for DevOps initiatives to be grounded in generating business value. If the main goal is to achieve DevOps, you won’t be able to demonstrate sufficient value to sustain support. Failure to identify a business goal and focus the effort on attaining that goal can shut down the DevOps initiative  in only a few months. Identify business value by working closely with business stakeholders on a project-by-project basis to identify opportunities or obstacles and define the business value that will be developed. Avoid the temptation to rush DevOps implementation by taking too large a leap. Trust the DevOps process.

    2. Don’t Ignore the Culture

    DevOps can be intimidating in the early stages. Done right, it usually involves organizational change. People need to understand from leadership that this is not optional. They also need to understand what DevOps is and ultimately how it will affect them. Otherwise, what they imagine will tend to be much worse. It’s also of chief importance to communicate the business goals, both for instituting DevOps and the business value that you hope to generate. Senior leaders must recognize that DevOps teams represent “a fundamental shift away from traditional command-and-control hierarchical management models,” according to the report. “To improve agility, decision-making must move to where the information is.” Relaying information up and down the chain of command to make a decision isn’t feasible with DevOps. The teams need to be empowered to make decisions themselves. For that to work, key DevOps team members need to be fully aware of senior management’s strategic priorities.

    DevOps thrives in an open, honest environment where a frank exchange of perspectives is discussed, analyzed constructively by the group leading to decision-making. Many DevOps initiatives fail because they give short shrift to selecting people with the right temperament to help manage this process. Gartner suggests choosing people who you would describe as being team players, trustworthy, motivated, accountable, smart, experienced and effective communicators. Other words that come to mind include responsible, high emotional IQ, fair-minded, open-minded, aboveboard, not prone to casting blame, respectful and collaborative. In the end, for DevOps to work, you need to foster the type of environment people from multiple disciplines, with multiple agendas, can come together in a small team and get things done.

    3. What We Have Here is a Failure to Collaborate

    DevOps should not be limited to just IT and Ops. For it to succeed in generating business value, DevOps teams must work with other groups and stakeholders. This work “requires a systemic perspective and involvement, not uncoordinated silos,” according to the report. That said, while there is no magic number of DevOps team members, some have recommended five to 10 people. When you get into the range of 20 people, you may find that you DevOps effort is less effective. (Aside: It is possible to have more than one DevOps team.)

    In considering collaboration, don’t forget senior management. Look to gain the support of an executive who can help champion the effort. Finally, you know that DevOps is working when DevOps team members are collaboratively compromising deferring to each other’s needs. The trick is to solve multiple problems with every decision.

    4. DevOps is Iterative

    Too many companies get caught up in the notion that they can launch DevOps in a single bound. According to the report, historically, traditional transformation approaches have a high failure rate. “DevOps involves too many variables for this type of approach to be successful in a large IT organization,” Gartner noted. “An incremental, iterative approach lets organizations focus on continual improvements and avoid the risks” of what is often termed the “big bang” approach. DevOps by its very nature is focused on continuous improvement with an iterative process, rolling out smaller improvements more quickly. Any thought of launching it all in one fell swoop misses the point and shows a distinct lack of understanding about the principles of DevOps.

    5. Set Realistic Expectations

    DevOps and DevSecOps are enjoying the huge level of interest and popularity among enterprises. When that happens, the hype tends to overinflate expectations about the outcomes. Gartner advises that you manage expectations by agreeing on objectives and success metrics. Establish your starting point with whatever metric you adopt, then pursue the goal iteratively. According to the report, “DevOps is not a one-time effort; rather, it’s about trying over and over again.”

    — Scot Finnie