Tag: DevOps KPIs

  • Transposit Survey Shows DevOps’ Impact on CIO, CTO Relationship

    Transposit Survey Shows DevOps’ Impact on CIO, CTO Relationship

    A survey of 415 CIOs and CTOs conducted by Transposit, a DevOps process orchestration platform provider, finds that as organizations continue to adopt DevOps processes more widely, friction between these types of senior IT leaders is rising within many organizations.

    As a C-level role, ‘CTO’ first gained traction in the wake of the dot com boom. In more recent years, many traditional enterprises added CTOs alongside CIOs as they realized how dependent the organization had become on software. In the wake of the COVID-19 pandemic, there is now more focus than ever on digital business transformation initiatives that are dependent on the timely delivery of software.

    Transposit CEO Divanny Lamas said that as more organizations’ structures revolve around product teams, there are a lot more CTOs. In some cases, CIOs are reporting to CTOs.

    The majority of organizations surveyed have both roles. A full 83% of CIOs said there is also a CTO at their organization, while 86% of CTOs said there is also a CIO. Just under a quarter of respondents (24%) said they perform both roles.

    At the same time, 98% of survey respondents reported increased focus on digital transformation within their organization in the past three years, with three-quarters (75%) noting that focus has increased since the start of the pandemic. A full 85% report their budget has increased over the last three years.

    Not surprisingly, 92% of respondents said that their responsibilities have changed in the last three years, with over half (56%) reporting their role changed significantly. Both are now graded according to key performance indicators (KPIs) related to managing revenue, with 66.3% of CIOs and 78% of CTOs saying their position is now either directly involved in or required to lead a business strategy.

    But only 34%, or just over a third, said that ownership of most aspects of digital transformation initiatives is a shared responsibility between the CIO and CTO. Specifically, just under a third (32%) noted ownership of software development in their organization is a shared responsibility. A full 88% of respondents apply DevOps practices, and 83.4% believe that DevOps is important to digital transformation at their organization. Almost all (98%) plan to expand their DevOps practice, and 83% already have a plan in place to do so.

    Lamas says software engineering-first principles are changing not just how applications are built but also how IT teams are structured in a way that advances innovation.

    More than a third (35%) of respondents said ownership of software operations developed in-house is a shared responsibility in their organization, while 28% said ownership of managing SaaS or operating commercial off-the-shelf (COTS) software in their organization is a shared responsibility. A third (33%) also noted ownership of the technology for the external customer experience in their organization is a shared responsibility.

    A full 95% of survey respondents also confirmed that the performance of their team is measured by key performance indicators (KPIs) related to revenue, specifically revenue growth (75%), rate of innovation (60%) and uptime (54%).

    It’s too early to say how the relationship between CIOs and CTOs will evolve, but the one thing is clear: The pressure for these roles to collaborate more effectively is on the rise.

  • Choosing the Right Metrics for DevOps Adoption

    Choosing the Right Metrics for DevOps Adoption

    DevOps adoption has skyrocketed in recent years. Its impact on traditional development platforms has long been well-documented, but in more recent years it has started to influence low-code platforms, too. In its recent survey on the state of Salesforce DevOps, Gearset found that 69% of teams are already using source control and another 20% plan to start this year. Additionally, 45% of companies have implemented CI/CD, and that number is set to increase by 33% this year. However, for high-performing developer teams to recognize the benefits of DevOps and strive for further DevOps adoption, they must use the right KPIs.

    While it can be tempting to take a ‘big bang’ approach to DevOps adoption, teams see more success when they take it one step at a time, with each stage building on the last.

    That said, to experience the full ROI of DevOps adoption, it’s still critical to see the journey through to the end. It’s not uncommon for teams who begin the process of DevOps adoption to stop halfway through or, worse still, lose motivation and halt the process entirely. DevOps adoption is a process full of technical and cultural change, and one of the most common causes of its failure is an emphasis on the technical at the expense of the cultural.

    The best way to tackle this is to encourage teams to take a product mindset and apply it to DevOps adoption. Simply put, rather than tasking your teams with “adopting CI” for instance, they should be asked to solve a specific business problem. For example, “When we finish work on a story, it takes weeks for it to get released!” It sounds obvious, but focusing teams around business impact rather than on individual tasks makes sure that there’s a direct correlation between the work that’s happening and the desired business outcome. This is the crux of adopting a DevOps mindset, rather than a purely technical and perfunctory approach to DevOps adoption.

    One of the key things a team can do to drive this type of change—and to avoid common causes of failure—is to use the right metrics at each stage of their DevOps journey.

    There are a number of well-recognized metrics for measuring DevOps success, including:

    • Deployment time
    • Change failure rate
    • Release cadence
    • Lead time
    • Recovery time

    These metrics serve two purposes. They’re well-documented and benchmarked indicators of performance that can help teams focus on their biggest problem areas. And they’re measurable outcomes that prove the business benefits of DevOps, such as agility, improved collaboration, higher quality systems and resilience, not to mention significant time and cost savings and efficiency.

    Choosing the Right Metrics

    Teams must choose the right metrics at the right time to identify their opportunities for improvement and to measure the impact of each stage of DevOps adoption. The right metric empowers a team to solve their most pressing challenges, and to demonstrate the value of their work to their leadership.

    Here’s how the benefits of each stage of DevOps maturity should be measured: 

    Stage of DevOps maturity

    Most relevant metric(s)

    Choosing your release tools

    Deployment time

    Adopting version control

    Change failure rate

    Recovery time

    Automating testing

    Change failure rate

    Automating deployments (CI/CD)

    Release cadence

    Lead time

    Integrating backup into DevOps tools and process

    Recovery time

    Let’s expand on each stage and the appropriate KPI.

    Choosing Your Release Tools

    This is more relevant to low-code platforms, but still broadly applies to the wider tech world. Even in the most mature of development processes, there will be manual steps. Getting changes into version control, reviewing changes, resolving conflicts, triggering releases—each of these is a manual step, and the effort involved in each of these manual steps is heavily dictated by the tools you use.

    Finding the right tool for each stage of your development and release process can therefore have a massive impact. The right deployment tool, for instance, can cut down deployment time by as much as 90%. This step alone increases efficiency and ultimately saves companies vast amounts of time and money. In extreme examples, teams that save so much time by picking up a couple of new tools to patch up their manual processes can be tempted to stop there. This is why teams need to look at other metrics as they proceed to adopt more DevOps processes.

    Adopting Version Control

    For most teams, a world without version control is unthinkable, having built their development and release processes around it a decade or more ago. For others, particularly in the low-code space where it’s yet to reach ubiquity, it can feel like an unnecessary burden. The introduction of version control is an extra step in the releases process and another level of complexity, with new tools and practices to learn. But version control is the foundation of an automated DevOps workflow. This step allows for more team collaboration and visibility into that collaboration, with code reviews and a detailed change history. Version control also allows teams to deal with merge conflicts early, rather than in the middle of a release. 

    After adopting version control, teams will see a reduction in their change failure rate due to code reviews and their recovery times will be shorter as version control offers rollback and a quasi-backup of the code and config captured by their version control system. But the real benefits of version control are in the DevOps processes it unlocks. 

    Automating Testing  

    It goes without saying that the purpose of testing is to improve the quality of releases. It does this in a few ways:

    • Unit testing – Exercising individual methods and functions in isolation generally results in more maintainable code, which has positive benefits for developer productivity and even things like handling merge conflicts.
    • Static code analysis – While this is not testing in the strictest sense, automatically verifying that code adheres to a set of best practices similarly improves code maintainability.
    • Integration testing – This type of testing ensures that combinations of classes interact and generate the expected results when combined together. This brings another level of validation that changes being made are likely correct.
    • End-to-end testing – UI testing is an example of this; exercising the system in the way a consumer would and verifying that the results are deterministic and correct. This also includes regression testing for key workflows and sometimes even acceptance testing.

    The more of these testing approaches a team uses, the more confidence they’ll have in their changes and the less human verification and intervention will be required. As teams’ DevOps processes mature, it’s often the human element that becomes the biggest bottleneck, so confidence in deploying changes is very valuable.

    When teams introduce more testing and test automation to their release pipeline, the right metric to watch is change failure rate—the proportion of releases that include a bug or error. The immediate benefit of higher-quality code translates to savings in the long term as less time is spent finding and fixing bugs or reworking code.

    Automating Deployments (CI/CD)

    If choosing the right deployment tool significantly reduces deployment times, automating deployments with a CI/CD workflow eliminates the time it would usually take to run those deployments manually. However, deployment times aren’t the key metric for measuring the success of CI/CD implementation. The most important metrics to pay attention to here are release cadence and lead times, which are dramatically improved thanks to painless, automated deployments. 

    With automated deployments, teams can increase their release cadence; rather than releasing huge batches of changes once a month, they might begin to release smaller packages every week. Ultimately, the goal is to have a steady stream of incremental improvements being released on-demand, and usually multiple times per day. Accelerated release cadences then result in reduced lead times—the period between work being finished and finally released to production. This is beneficial for developers and their end users as more frequent releases create more opportunities to act on user feedback earlier in the process. 

    Integrating Backup Into DevOps Tools and Process

    Recovery times are a key metric for DevOps success, as robust DevOps processes will improve the speed at which teams can restore after an incident. For even better resilience and faster recovery times, teams should integrate backup into their DevOps tools and processes.

    Restoring from a backup is comparable to a deployment, and this means that restore processes also can harness the power of DevOps so they feel like a familiar process—not something unusual that teams struggle to execute at the point when it matters most. It’s also important to make it as easy as possible to back up on demand just before a risky release.

    Downtime and data loss are massive costs to any business, so improving recovery times is a huge benefit of a fully mature DevOps process.

    See the Benefits of Maturing your DevOps Process

    When teams pay attention to the right metrics, they understand the benefit of each DevOps process individually and they are motivated to keep maturing their DevOps processes as a whole. It’s all well and good to take the pain out of deployments and releases, but high-performing DevOps teams effectively make deployments an asset for releasing more often and getting feedback earlier. As a result, they are better equipped to deliver more value to their companies—sooner.

  • Measuring Success in DevOps: Top Four DevOps KPIs

    Measuring Success in DevOps: Top Four DevOps KPIs

    There is no news in DevOps concept. Many companies have integrated DevOps to improve and accelerate software development and help drive their digital transformation, and that number continues to grow exponentially. There are now entire tool ecosystems, methodologies and transformation models—as well as endless resources—available to guide companies along the DevOps journey.

    But DevOps success can be difficult to measure. DevOps isn’t a formal framework; it’s more of a culture and a set of practices, and there is limited guidance available to ensure you’re doing it properly or accurately measuring your successes and failures. DevOps also looks different in every organization—no two DevOps shops will be the same.

    However, despite the nebulous definition of what a DevOps organization looks like, there are some core key performance indicators (KPIs) that should be common to all DevOps environments: asset management, monitoring, continuous integration/continuous deployment and continuous security.

    Asset Management

    In this context, asset management means measuring what is automated and how effectively it is controlled. It doesn’t matter if you are using tools such as Puppet, Chef, Ansible or Terraform; what matters is how effectively you use them. These tools allow you to automatically provision bare-metal or cloud assets, as well as join a load balancer and receive traffic. However, you can achieve similar results from Kubernetes or chaining lambda functions. Automation is really about controlling assets to achieve your business goals.

    Whether you’re using on-premises virtualization or cloud workloads, taking the time to streamline your processing cycles down to the minimum hardware required can significantly reduce monthly costs. The higher the percent of your servers, containers, services, etc. are automated, the less time you’ll spend fixing and maintaining your infrastructure. Some of your applications may be uniquely configured to your particular organization, and therefore cannot be automated. However, if applications cannot be automated, it might be time to retire them and build next-generation applications.

    Monitoring

    Should you be monitoring all of your resources? Yes. Should you also filter out the noise? Absolutely. Monitoring agents should be deployed to all of your servers, and if your PaaS tools and other providers have integrations, you should be using them and parsing out unneeded data. The only thing worse than not monitoring anything is monitoring everything without proper filtering.

    Good monitoring should always be coupled with utilizing the information acquired. Capturing logs, parsing them and using tools such as ELK, Datadog or Sumo Logic, for example, give you a large kit that can not only capture and glean useful information from a tremendous amount of data, but can also make decisions and perform predictive analyses. You might learn your stack is operating inefficiently, or not scaling fast enough during peak utilization periods.

    Disparate systems should be aggregated into a common tool, lest you spend valuable time combining information across systems or pulling information from data warehouses to get your insight. Let the tools do that work for you. You should also be using tools such as Logstash to pull and understand application, database and other logs in your stack for meaningful data. If not, you’re limiting your insight into the service’s health.

    Continuous Integration/Continuous Deployment

    Continuous integration and continuous deployment (CI/CD) is the difference between releasing software once a quarter (or year) and performing dozens of releases to production within a day.

    But it’s one of the most difficult indicators to measure, since every environment is different. It is important to understand how many code commits don’t make it past your integration environment. These blockages could be the result of poor programming practices, poor code quality or poor testing. However, if features stay in dev or integration for too long, there is surely an underlying issue. In most cases, it centers around automated testing.

    So, how do you measure good testing? This is one of the most difficult questions enterprises face, especially when automated testing is added after monolithic applications are already written. Essentially, good testing is about creating good business logic and integrating testing that meets most use cases, while covering each function and class. Immediate feedback on failed or successful tests expedites the automation process. The less automated testing is built, the more QA has to test by hand. Failing to integrate good business and integration testing slows down the entire CD part of the pipeline significantly. Also remember shooting for 100% test coverage is unrealistic—over 80% should be sufficient.

    It’s also important to remember applications that are not in a pipeline and must be manually tested will never keep up with the pace of cloud-born innovation. If you are dealing with a giant, monolithic legacy app, it’s time to design its replacement (starting with testing). Pipelines should also have notifications built into them, with chaining logic to react to failures. Tools such as Jenkins, CircleCI, TravisCI and most code repository solutions, such as GitLab, offer rich feature sets to aid in pipeline creation and management.

    Continuous Security

    The easiest way to ensure security compliance is to shift left and resolve security issues in the development stage. Too often, security is applied at production, which means it’s not part of the environment’s end-to-end process. Securing applications and networks at the development level will give you more confidence that applications will interoperate properly at the production level.

    After shifting left, make sure you continuously deploy and monitor your clusters, nodes and pods in a secure manner. Ideally, you have a tool that provides a real-time summary of your cluster’s compliance and security status by looking at workload security and governance checks, cluster worker node CIS Kubernetes Benchmark checks, cluster Ingress controller security best practices and istio security checks.

    DevOps Success Never Ends

    Success in measuring your core DevOps KPIs is not about getting to full automation immediately—it’s about understanding your biggest issues, continuously securing them and improving them over time. DevOps has no end; success requires continuously measuring your progress and improving your environment with the most effective and up-to-date tools. Take control, implement thorough and meaningful monitoring, deliver with confidence and do it securely.

    — Gadi Naor