devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

How Automated Testing Closes the Development and Delivery Gap

For the past two years, companies have leaned into digital transformation projects to keep their digital offerings at pace with their in-person operations. Rather than become a stopgap, this digital approach became the new standard for the modern employee and customer experience. It also creates friction between a company’s development team and its user base. How long should customers wait for new or updated applications in our increasingly fast-paced business environment? Companies need time for development and testing to deliver quality products, but will the demands of their users force them to cut corners?

Consider each stage of a customer’s journey. Visiting a website or app and putting something into the shopping cart is only the beginning. Behind the scenes, systems that monitor availability, sourcing, loyalty and promotions come into play. Following checkout and payment, there are systems for picking the product, packing it and delivering it with timely notifications along the way.

Businesses find themselves in a bind when they have to decide which of those areas to cut back on. There’s really no good answer—enter the continuous delivery mindset. We like to say there is no continuous delivery without continuous testing, but those two areas are often at odds with each other.

The good news is that there is a way to close the gap between customer demand and necessary development time: Test automation. Here are three ways that test automation can help companies satisfy both the needs of their business and their customers.

Relieving the Testing Bottleneck

Moving fast is becoming a competitive advantage for many companies. That speed-to-market opportunity has been heightened as cloud-based, low-code/no-code apps like Salesforce, MS Dynamics, Workday and Oracle have become an industry standard. Today, while plenty of development activities may be handled by low-code/no-code platforms, the test automation of these apps remains a high-code, high-maintenance activity.

To keep from becoming a bottleneck, teams should plan automation and testing strategies early in the process. This is where continuous development includes continuous testing. Suppose companies can make test automation accessible to more people by removing the need for specialized knowledge and daunting technical constraints. In that case, they will be able to streamline ongoing development cycles.

Ensuring Higher-Quality Output; Aligning Development and Testing

One of the keys to putting out a higher-quality product is by adopting in-sprint testing, which aligns development and testing efforts. Each feature is tested and automated as soon as it is implemented so that teams can more easily catch errors early rather than at the end of a development phase.

The automation also elevates the area of regression testing, as teams make sure new changes haven’t broken existing functionality. As automation moves into the mainstream, a wider range of collaborators become more involved in testing each additional feature.

Creating a Better User Experience

Historically, testing has been extremely focused on function and is very technical while user experience efforts remain focused on the stylistic and usability end of the spectrum. But creating satisfying end-to-end user experiences is a critical part of any application adoption, so the two sides should work toward a common goal.

That common link lies in a company’s business processes. Each process is the starting point for building and designing a test scenario. From start to finish, everyone understands the user’s journey and works together toward making that path as effortless as possible.

Conclusion

Today’s modern customers don’t expect a measurable difference in experience between online and in-person. Expectations are at an all-time high, leading companies to push updates to their websites and apps more quickly and more frequently. As companies increase their dependence on software, the way to test that software is evolving to keep pace with development speeds.

Automated testing can simplify and speed up the testing phase, catching many of the basic errors and leaving testers to focus on more complex functionality.

July 13, 2022
CodeLogic Toolkit Increases Visibility Into App Dependencies

CodeLogic launched today a toolkit that enables developers to scan binaries, runtime application behavior and database connections and then leverage graph technology to identify connections and dependencies in real-time.

Brian Pierce, CodeLogic CEO, said the goal is to make it simpler for developers to identify the relationship between application elements to increase overall productivity and simplify the addition of new members to an application development project.

In general, most developers add code to applications without really having much visibility into how that code might impact other components. The toolkit created by CodeLogic is an extension of the company’s existing continuous software intelligence (CSI) platform that employs server software to store and process data collected via its agent software. The goal is to make it easier for developers to directly consume those insights via a toolkit they can access directly from within, for example, an integrated development environment (IDE), said Pierce.

As applications become more complex in the age of microservices, it becomes more challenging to understand all the dependencies between application components. Developers require greater visibility into applications to better manage the break/fix process that inevitably occurs when new code is added to any environment. The CodeLogic CSI platform provides an automated impact score for methods, classes and the overall health of an application as code is developed to provide that visibility.

That information also plays a critical role in enabling IT teams to understand the implications any newly discovered vulnerability might have on the application environment, added Pierce.

Overall, the level of technical debt that organizations accumulate over time saps developer productivity, he noted. The more difficult it becomes to add code to an application environment the less likely it is that developers will enthusiastically make the effort, said Pierce. Before too long, the number of updates made to an application environment starts to decline, he noted.

Developer productivity has become a major issue simply because the number of applications that organizations now want to build and deploy is exceeding the financial resources they have available to hire or contract developers. The more time developers spend studying how an application has been constructed the less time there is to write code and drive innovation. In fact, the more frustrating it becomes to add code the more likely it becomes developers will move on to a company with much less friction involved in building and deploying applications, said Pierce.

There are many senior IT leaders that also need visibility into application environments before approving projects. Many IT teams discover that there are multiple projects or other teams writing duplicate code or replicating existing functions. Regardless of the reason, more developers are voting with their feet when they feel their time and effort is not fully appreciated.

July 12, 2022
Rezilion Launches Vulnerability Prioritization Platform

Rezilion today announced general availability of a platform that enables DevOps teams to better prioritize remediation efforts by identifying which vulnerabilities both run in memory and actually impact a class or function that can be executed.

Liran Tancman, Rezilion CEO, said the biggest DevSecOps challenge organizations face today is that the bulk of vulnerabilities that developers are tasked with remediating don’t constitute a significant cybersecurity threat.

Rezilion has created a searchable proprietary next-generation vulnerability database (NGVDB) that identifies which issues impact specific classes and functions. That capability enables DevOps teams to de-prioritize close to 95% of software vulnerabilities, said Tancman.

In addition, the Rezilion platform will surface remediation suggestions that lower the amount of disruption and/or cost, he said. A fully-automated remediation capability, for example, will automatically upgrade and test vulnerable software components and packages within a continuous integration (CI) pipeline that were found to be exploitable. In the longer term, Rezilion is also working toward surfacing remediation suggestions that might be addressed using an alternative to patching, said Tancman.

The Rezlion platform is also integrated with IT service management platforms such as ServiceNow and project management tools such as Jira to make it easier to address vulnerabilities as part of existing workflows, he added.

Most developers today are simply overwhelmed by the number of vulnerabilities that cybersecurity teams routinely ask them to patch, noted Tancman. The goal should be to enable developers to focus their limited time and effort on remediating the vulnerabilities that truly matter to ensure application security, he said.

In addition, developers need to have more confidence in the fact that the patch they are being asked to make won’t break their existing applications, added Tancman.

Vulnerability prioritization is, of course, at the heart of the long-standing divide that exists between DevOps and cybersecurity professionals. Spreadsheets filled with long lists of vulnerabilities are regularly created, but there is usually no context provided in terms of how severe a threat any given vulnerability poses. As a result, developers would allocate a small portion of their time to applying patches that are prioritized based on intuition and level of complexity rather than severity.

Naturally, developers are going to allocate most of their time to creating new applications and features. For any DevSecOps initiative to succeed, it’s critical to ensure optimal use of the time allocated to remediation. Cybersecurity teams will continue to compile lists of vulnerabilities, but that list can now be triaged in a way that causes the least amount of friction.

In the wake of a series of high-profile security breaches, there is now more focus than ever on securing software supply chains. The challenge is to find a way to better secure applications both before and after they are deployed in a production environment. There will always be vulnerabilities that are discovered after an application has been deployed. However, just because a vulnerability exists, that doesn’t mean it’s exploitable. The trouble with application security today is that, all too often, remediation efforts turn out to be a waste of time.

July 12, 2022
5 Best Practices for Software Development in the Hybrid Workplace

Over the years, software teams have gradually become more dispersed as technology advancements facilitated hybrid work and allowed teams to collaborate, create and innovate remotely. However, this trend accelerated and became more of a mainstream norm over the last two years during the COVID-19 pandemic, requiring newly dispersed teams to develop a more streamlined hybrid approach to the development process.

When seeking to implement a hybrid environment, it’s important for development leaders to consider how best to coordinate a distributed workforce across time zones, how to track projects effectively and ensure accountability. At Lenovo, these are considerations the Cloud & Software team has had to navigate since its inception due to the global nature of our business. To be effective in a hybrid environment, we adopted certain processes and tools that facilitated open communication and cross-collaboration no matter the location. After evaluating various hybrid methods, here are five best practices that stand out.

1. Adapt the development process to suit a hybrid setting

In a hybrid work environment, it can be challenging to consistently bring the entire team together at a specific point in time to estimate work, refine issues or plan sprints. Coordinating between busy calendars and multiple time zones is an inefficient use of time, especially when many software projects are deadline-driven. Moreover, many developers are highly creative and want fewer coordinated meetings and more time to contribute when they want, at their own pace.

To navigate these challenges, program managers can adapt the development process to be less regimented and more flexible. Facilitating such events as estimation, sprint planning and backlog refinement in an asynchronous manner within a virtual team workspace can allow all team members, no matter where they are located, to contribute, feel heard and seen and receive feedback from their teammates. Such programs can also enable an automatic way to log decisions and house required software documentation.

2. Establish a single source (or tool) of truth

Many organizations relate to the challenge of having project data spread across multiple tools. While the development team tracks day-to-day work in one tool, other stakeholders are tracking goals and team KPIs in an entirely different set of tools which can lead to misalignment, something that can be exacerbated in a hybrid environment.

To overcome this challenge, development teams can establish one central location using a work management tool where past, current and future projects live and that all team members can access, reference and share. By leveraging this tool as a single source of truth, team managers can bring visibility and transparency to the development and program management processes while creating stronger alignment.

3. Enable full transparency in decision making

Our team at Lenovo follows a strict process of documenting the decisions taken throughout the development process and we make that documentation available for all team members to access within the team workspace.

This can be a beneficial practice for project managers that want to strengthen accountability. Encouraging such transparency enables team members and stakeholders to reference the reasoning behind a specific decision, especially if concerns arise later in the project life cycle process. The result is better alignment across the organization.

4. Establish a facilitator for each ritual

To maintain a healthy hybrid environment, assigning an owner to each Agile event can make all the difference. By serving as a facilitator, that owner would then be responsible for aligning everyone to a pre-meeting agenda and goal, allowing teams to prepare and review issues and tickets ahead of time. This added organizational step can result in a more streamlined meeting cadence and more productive discussions.

5. Automate and leverage efficient tools

Finding the right combination of tools to support a team’s specific collaboration needs is imperative to a successful hybrid development environment. Seamless and automated collaboration is the key, but there is no one-size-fits-all solution mix. Development leaders should consider tailoring their portfolio of tools around specific needs and look for programs that can help keep a dispersed team on track and synchronized (via direct messaging, ongoing team conversation chats or face-to-face virtual meetings, for example), as well as those that can ensure visibility and improve efficiencies. Whatever external programs are selected, however, should be hosted internally behind the company firewall and VPN for added security.

The modern workplace has changed dramatically in the last two years—in many ways permanently—with flexible, remote and hybrid work driving the greatest transformation of collaboration for software development teams across all sectors. By adapting to new hybrid development strategies, enabling transparency in decision-making and leveraging automation for greater efficiency, the development teams of today and tomorrow will be well equipped to more easily adapt to future disruptions in the workplace.

July 12, 2022
Connections Matter: Gaining Visibility Into APIs

Digital transformation is at the center of modern enterprise. Yet, as cloud initiatives take shape and the complexity of connecting systems and applications multiplies, the need to tie everything together and automate processes grows. APIs, otherwise known as application programming interfaces, fuel this automation. These small pieces of code—which connect and interconnect clouds, containers, microservices and applications—help organizations compress what used to be years of project development into months or weeks.

As businesses connect functionality—often across companies and service providers—APIs lead to an enhanced user experience. Simply put, they make it possible for much of the magic of the internet to take place, including seamless authentication, data sharing, payments and collaboration.

Today, it’s next to impossible to find an enterprise application that doesn’t rely on APIs. Yet, despite all the gains, APIs can cause plenty of pain. The slow performance or complete failure of an API can lead to an array of problems. These include customers who can’t complete transactions, employees who can’t collaborate through critical apps, and incomplete or incorrect supply chain data.

Today, a typical organization relies on more than 1,200 cloud-based applications and hundreds or even thousands of APIs. For many businesses, 50% to 90% of revenues are linked to APIs. If a system underperforms or breaks, there’s often no way to identify the root cause and fix it promptly. Reputational damage and lost sales often ensue.

When Good APIs Go Bad

It’s important to recognize that the monolithic applications of the past are a vanishing breed. Today’s highly-distributed software and services environment taps a universe of more than 24,000 APIs. It isn’t unusual for a transaction to pass through several cloud containers and across numerous companies en route to the desired outcome. Along the way, these APIs encounter different hardware, code and conditions.

When APIs work well, a process or transaction typically finalizes within milliseconds—and the digital experience usually excels. However, if a page loads slowly, a promotional code doesn’t work or payment doesn’t go through, the odds are high that somewhere along the line something went astray with an API.

For many organizations, there’s no simple and fast way to identify the source of the problem, understand the current impact, and take steps to fix the failure post haste. In fact, it isn’t unusual for several thousand or more failed transactions to pile up before the issue hits IT’s radar and anyone begins to notice a problem. By then, shoppers have likely abandoned shopping carts and sales have evaporated.

Adding to the challenge is the fact that APIs can fail in several ways. Not surprisingly, the number one problem is availability. If the API doesn’t function, the interaction or transaction cannot take place. In addition, an API might perform slowly or erratically, thus delivering highly inconsistent and frustrating experiences for users along with a problem that is difficult to diagnose and fix. Yet another problem is misconfigured or poorly functioning APIs that return incomplete data or the wrong data.

The real-world fallout can be significant. In practical terms, a customer shopping at a clothing retailer’s website may find it impossible to add socks to a shoe order or use one or more payment services. It may be impossible to cash in points from a loyalty program or view delivery information. Broken APIs can disrupt or break shopping carts, customer accounts, supply chain data, collaboration links, security tools, and much more.

Of course, when things go haywire, finger-pointing and conflicts with vendors, business partners and customers often ensue.

Deep Visibility

In today’s API-centric world, detecting problems early and often is critical. What makes the task so difficult is that API transactions do not unfold in a linear way. They bounce across servers and companies—often a couple dozen of them—scattered all over the world. Further complicating things: This routing can change at any given instant.

Identifying specific issues that trigger a problem—traffic bottlenecks, bandwidth issues, code problems and more—requires the right tools. It’s vital to understand how an API-based transaction behaves across a dozen or even two dozen hops and touchpoints that extend through clouds, containers and microservices.

Passive visibility tools, while useful for getting closer to the end-user, do not offer a comprehensive and granular view of events and processes. Today, businesses require broader and deeper observability capabilities that extend across an ecosystem of APIs and into the entire IT landscape—from network to device. It’s vital to understand how customers use websites, apps and other services.

Making Integrations Matter

As organizations delve deeper into clouds and digital interconnectivity, a more nuanced approach that incorporates both passive and active observability is in order. This approach makes it possible to emulate a user journey and quickly identify problem points. A business can detect problems as they pop up—after only a handful of failed transactions—rather than thousands or even tens of thousands of transactions later.

With a best-practice observability and remediation framework in place, an enterprise can establish baselines and triggers for alerts and notifications. It can assemble a complete picture within a single pane of glass and troubleshoot across the delivery chain. Whether the API involves a collaborative task within the organization or a financial transaction that stretches across several companies, it’s possible to view the entire transaction—and experience.

With both passive and active monitoring in place, a business can evolve beyond constant fire-fighting mode. Armed with the information and insight required to make vendors more accountable, businesses ensure they are getting the level of service (SLA) they are paying for. Make no mistake, as digital transformation initiatives accelerate and cloud complexity grows, it’s critical to ensure that APIs, microservices and other digital components are up to speed.

July 11, 2022
The Quest To Unify With GraphQL, The Modern Data API

There is an ongoing trend in enterprise software: Using GraphQL as an aggregation layer to unify access to underlying data sources and REST APIs. Some call this the supergraph, and others refer to it as a meta layer. The idea is that by providing a more consumable layer to a wide range of internal sources, frontend developers are better equipped to access whatever data they need when constructing applications. But the quest to unify all data sources into a single schema doesn’t come without its share of trials.

GraphQL adoption is steadily increasing, with significant large enterprises making up some of the early adopters. For example, Netflix engineers recently shared how one approach gaining traction is “GraphQL microservices (GQLMS) as a backend platform facilitating rapid application development.” As the use of GraphQL increases across a large enterprise, though, scalability and security hurdles present themselves. These include retaining proper authorization throughout multiple layers and mitigating the risks of an inherently open schema.

I recently met with Hasura CEO Tanmai Gopal to discuss why GraphQL should be used as a middleware element to access and expose underlying datasets and APIs. According to Gopal, GraphQL is becoming the data API for modern applications—this could be a massive positive shift. However, the potential downside is around the time involved in integration. Thus, the key will be automating the integration of data sources so that developers and DevOps teams can focus on the apps and services they’re building instead of maintaining the data model itself.

The Underlying Issue: Data Access

“Data access is a bottleneck,” said Gopal. “It needs to be self-service.”

On one side, there are data producers, such as databases, internal services and external SaaS services. On the other side, there are the clients that consume this data, such as mobile apps, web apps, microservices and APIs externalized for third-party developers.

The problem, according to Gopal, is that the link between both sides is increasingly challenging to implement. It takes time to integrate with HTTP-based APIs and parse JSON payloads. The exchange protocols are dissimilar and you must solve for performance issues, caching and concurrency in data access. What’s more, the authorization context must be maintained for all data sources. These issues are exacerbated since not all developers are accustomed to the nuances of maintaining integrations.

Frontend developers want to communicate with a centralized data store, but also need to mix and match various sources. At this point, Gopal explained, a data access layer is necessary to handle the communication between both sides. Such a layer requires some integration automation to truly get APIs to developers at the edge. And it also requires a simple, unified schema—this is where GraphQL comes in.

Solution: A GraphQL-Based Data API

“GraphQL is an absolutely delightful API to consume,” said Rajoshi Ghosh, co-founder of Hasura. “Using GraphQL as a data API can remove a lot of requirements of legacy knowledge of integration systems and how to securely fetch data.”

The concept involves using GraphQL as an intermediary data API—this would consume all sources, whether they are RESTful APIs, GraphQL services, an Amazon Aurora database or a PostgresSQL-style database—and unify them in a more accessible, pleasing construct. To create a unified API over these disparate sources, a middleware solution must map API resources to logical models and methods and add authorization rules to logical models, explained Gopal.

Although GraphQL enables developer agility, it’s the part in the middle that’s difficult to assemble. Building these connectors from scratch requires a lot of relearning and tedium to maintain, Ghosh said. And to truly cater to all data sources, organizations will likely need to craft custom plugins for legacy sources, he added.

Solving Unified Graph Problems

Data-driven initiatives are a common technology goal, but the breadth of data sources is rising. GraphQL can play an important role in uniting these sources and providing flexibility for frontend developers to call precisely what they need. A unified graph is thus a boon for developer experience but will be challenging to set up from scratch.

Arguably, authorization is the biggest challenge when approaching GraphQL as an aggregation layer for multiple APIs. Any system that unifies multiple backend APIs must ensure that only the person with the correct permissions can access underlying data. A unified GraphQL layer must retain granular permissions that are endpoint-centric and unique to the specific caller.

“Most enterprises don’t have a good authorization story,” Gopal said. Implementing authorization for data APIs is hard, Gopal said, as it’s unlike REST where the user ID can literally be specified within the URL request. These kinds of fine-grained attribute level entitlements must be recreated as a policy engine and applied to the graph API as well. For example, Hasura implements authorization by using a token that scopes the entire experience to the particular user.

The Future of Unified API Strategies

Looking to the future of this area, Ghosh noted that frontend stacks are becoming more robust and code in the backend is essentially becoming business logic deployed as serverless functions. And in this new reality of increasing abstractions, the role of modeling experts will likely become more important.

Furthermore, GraphQL must go beyond read-only capabilities to embrace another trend—real-time. Although most GraphQL APIs retrieve persistent data, GraphQL-based data brokers must also evolve to implement streaming, said Gopal, as real-time data and event-based subscriptions are becoming more common.

Today, you probably don’t need to unify all your sources into a unified data model all at once. This activity will likely be relegated to particular business domains or products to help them accelerate and demonstrate a proof-of-concept. But in the coming years, there will be mounting pressure to adopt a unified approach, Gopal said. “In a few years, if you don’t have a unified API strategy, you’ll have a massive problem.”

July 11, 2022
BMC Adds Support for DORA Metrics to Mainframe Tools Portfolio

BMC this week announced it has added support for DevOps Research and Assessment (DORA) metrics within its portfolio of DevOps tools for mainframe environments.

John McKenny, senior vice president and general manager for intelligent Z optimization and transformation at BMC, said the latest edition of BMC Compuware zAdviser now includes a DORA KPI Dashboard that aligns with the widely-adopted framework for measuring DevOps efficiency defined by an arm of Google.

The DORA metrics track deployment frequency, failure rate, lead time for changes and mean-time-to-recovery (MTTR). Tracking those metrics is crucial for IT teams that have mainframes because they need to make sure applications continue to be deployed on the platform versus being usurped by another platform that is perceived to provide a faster alternative for builds and deployment, noted McKenny.

BMC also added support for automated webhook notifications to its BMC Compuware Abend-AID offering. That includes diagnostic and root cause analysis to resolve defects in test and production faster. In addition, BMC provides a connector to the machine-identity management platform created by Venafi. That integration promises to eliminate certificate-related outages that can derail an application development initiative.

In general, DORA metrics are being used to identify areas where DevOps teams can improve. It is up to each organization to determine to what degree those metrics will be used to coach teams versus enforce productivity mandates. However, given the overall shortage of available DevOps talent, few organizations are employing metrics to force DevOps teams to improve. In fact, there is no universal set of DevOps goals that every organization is trying to achieve. Rather, most organizations are trying to determine what set of DevOps best practices makes the most sense to implement given their own goals and internal culture. In the case of the mainframe, most organizations start with a few projects before they begin to employ DevOps more widely, noted McKenny.

The metrics defined by DORA framework are, of course, only a subset of the total metrics that DevOps teams should track. BMC, for example, makes it possible to track tools usage to provide DevOps teams with insights into what tools may no longer be relevant or required, noted McKenny.

Mainframe environments also have a long history of using waterfall-based approaches to building and deploying applications that many of them will continue to use alongside DevOps practices, depending on the type of application being built and deployed. In other instances, distributed applications built using DevOps practices are being integrated with mainframe applications that now need to be updated using the same workflow. The mainframe is no longer a platform that is managed in complete isolation from distributed computing environments.

The major challenge is that many organizations using mainframes often overestimate how difficult it is to adopt DevOps best practices, McKenny said. Regardless of the platform, there is no doubt the overall rate at which all applications are being built and deployed will continue to increase. The mainframe is proving to be no exception to that rule.

July 8, 2022
Planview Outlines VSM Strategy After Acquiring Tasktop

Planview this week announced it completed its acquisition of Tasktop, a provider of a value stream management (VSM) platform that enables DevOps teams to align their efforts more closely with strategic business goals.

Louise Allen, chief product officer for Planview, said the goal is to tighten integration between the Tasktop VSM platform and the project management applications that Planview already provides to software development teams and business executives.

It’s still early days as far as adoption of VSM within DevOps workflows is concerned, but as more organizations realize how dependent they are on software, they need to better align application development projects with their strategic business goals, said Allen. VSM platforms make it possible to analyze data collected from various DevOps tools and enable business and IT leaders to, for example, visualize how a delay to one project might impact revenue goals for an upcoming quarter.

Given the current weakness in the overall economy, organizations are looking to more precisely understand the impact software development decisions will have on the business, she added. As part of an effort to address that requirement, Planview will also make it easier for its applications to consume data that was originally created in Tasktop, she explained.

VSM enables more organizations to function like a software company—aligning application updates with revenue goals. As the number of digital business transformation initiatives being launched increases, organizations are finding they need to operate more like a software company that either manufactures something or provides a service.

The concept of value stream management traces its lineage back to lean manufacturing methods, which called for each step of a manufacturing process to be continuously measured. As software development has evolved from being a craft to an automated process using DevOps best practices, there is growing appreciation for the value of monitoring things such as the impact of missed software development deadlines on the business.

At its core, a VSM platform will automatically convert metrics collected from a DevOps workflow to track a set of key performance indicators (KPIs) defined by the business. Once those metrics are gathered and KPIs established, organizations are then able to apply advanced analytics to align the resources needed to achieve a strategic goal by, for example, devoting more developers to a specific project. The data collected via a VSM platform can also then be shared with business intelligence (BI) applications that business leaders typically use to manage the business.

It’s not clear just how hard business leaders are looking for deeper insights into how software development initiatives are progressing, but at the very least, DevOps leaders need a simpler way to understand all the dependencies that exist between initiatives. Otherwise, they risk missing deadlines simply because no one realized a delay that occurred in one project had a cascading impact across multiple other initiatives. Regardless of who consumes the data generated by a VSM platform, the need to increase visibility across the software development life cycle had never been greater.

July 8, 2022
Dev Job Phisher Steals $540M | Patch OpenSSL NOW | Systemd Dev Joins Microsoft

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Spear-phishing causes $540 million loss, a high severity bug in OpenSSL might be “worse than Heartbleed,” and Lennart Poettering is now working for Microsoft. (more…)

July 7, 2022
5 Mean-Time Reliability Metrics To Follow

Most folks working in DevOps or SRE roles are familiar with metrics like mean-time-to-recovery (MTTR). Keeping track of the average time a team takes to respond to incidents is crucial to identifying bottlenecks in the support process. It’s also something executives like to show higher-ups when sharing a snapshot of overall platform performance. However, focusing on one single metric might be missing the greater picture.

For example, how long did it take to discover the incident? How long did it take from discovery until action was taken? What was the timeframe between filing a ticket and updating all clients that had a bug? When can you say you’ve completely resolved an issue? As you can see, there are many potential metrics that could inform software reliability and the platform engineering process. “One number is never going to tell you a complete story,” said Emily Arnott, community relations manager, Blameless.

I recently met with the Blameless team for a closer look into mean-time reliability metrics. Below, we’ll explore the nuances behind five different types of MTTX metrics and consider the business value of keeping tabs on each type.

1. Mean-Time-To-Detect

Mean-time-to-detect is a measurement of how long it takes, on average, to detect that an incident is present. This often happens automatically within a monitoring system. Perhaps a tool sends out an alert when latency meets a certain threshold, for example. But, detection could also come from other sources, such as a customer complaint.

For example, consider Log4j. Perhaps a runtime vulnerability scanning tool noticed one of your components is impacted by a novel CVE and automatically sent a notification to the appropriate team channel.

2. Mean-Time-To-Acknowledge

Now, just because an incident is detected doesn’t mean it’s immediately acknowledged. Mean-time-to-acknowledge, then, is a measurement of how long it takes a human being to realize the incident and begin to act on it.

In our CVE example, this would be the time between vulnerability detection and initial response. For example, the on-call incident manager received a vulnerability notification, read the exposure details and then filed a JIRA ticket and contacted the relevant team members.

Many factors might extend mean-time-to-acknowledge. Perhaps someone isn’t logged into Slack, or a hardware issue stalls the notification. Alert fatigue may also get in the way, or there may be a reluctance to file a ticket and introduce more toil. Depending on the severity of the incident, someone simply might not believe the alert is worth responding to.

3. Mean-Time-To-Recover

Now, the following metrics are a little more open to interpretation, but we’ll do our best to define each. Mean-time-to-recover is the average time it takes to introduce a temporary fix after discovering an incident.

For example, if a particular region is experiencing outages, engineers might temporarily divert traffic to a more stable server. This is not a permanent fix, but the system recovers and operations are generally unaffected. This helps maintain the status quo while a more permanent solution is ideated, tested and applied.

4. Mean-Time-To-Repair

Mean-time-to-repair (or restore) is the mean time it takes to issue a permanent repair to a system after discovering an incident. In order for a system to be considered fully repaired, it has to not just be working, but working robustly.

For example, let’s say the incident in question involves performance issues. Perhaps a patch is issued to the core branch to remove bulky code that’s causing slow load times for clients. Mean-time-to-repair introduces a more permanent solution but still might not be the complete resolution to the problem.

5. Mean-Time-To-Resolve

Mean-time-to-resolve can be thought of as the average time between when an incident occurs to when the issue is completely resolved. Not only is the core codebase patched, but all clients reliant upon the software have been updated, as well. Lessons are learned and mitigation plans are set to respond to similar incidents or vulnerabilities in the future.

Mean-time-to-resolve is about resolving the incident entirely, said Jake Englund, senior site reliability engineer, Blameless. This includes addressing the underlying fundamental contributors, completing all logs that remain on the back burner and following up with a retrospective.

Using Mean-Time Metrics

Mean-time metrics can provide a quantitative picture of incident response performance, which can be valuable for overall business operations. Mean-time-to-acknowledge, for example, can expose gaps in the remediation process, such as cognitive strife in reporting incidents, said Matt Davis, intuition engineer, Blameless. Understanding these technical and human factors are the first step to making the resolution process swifter.

Of course, the above metrics rely on incident data, which may not always be a top priority. According to Davis, encouraging a culture that declares incidents—even minor ones such as a configuration change—can improve knowledge sharing within a team. “If you declare an incident, you could enact more systemic change,” added Arnott.

Limitations of Mean-Time Metrics

These mean-time metrics do have some limitations. “A number is only one part of the story,” said Davis. As a result, teams might struggle with deciding precisely what to detect. MTTR can be a helpful metric, but it’s the context that matters, he explained. Therefore, tracking multiple metrics can help provide a more sophisticated, nuanced picture. This involves looking beyond averages to consider outlier events, added Englund.

There are also semantic nuances between the MTTX metrics defined above. “There’s a lot of ambiguity around these words,” said Davis. As a result, organizations might compute each figure differently. Some of these figures use time-markers which may not be technically possible to track consistently, especially since each incident is unique. Demarcating the precise moment an incident began might require guestimation. You might know when a service is fully restored, but the lasting customer perception is harder to gauge.

Also, another potential downside is that mean-time metrics could easily be manipulated or misinterpreted, whether deliberately or inadvertently. Operations leads might selectively recall specific windows when showing MTTX metrics to higher-ups, leaving out other statistics that paint a different picture.

Treat MTTX As A Guidepost

Improving incident response is becoming mission-critical to maintain fully-functional systems as things like outages, downtime, slow speeds and zero-day vulnerabilities can negatively impact user experience. Sometimes, these issues must be addressed immediately to maintain SLAs.

But tracking reliability averages isn’t all that simple, and they will likely mean something different to each organization. “It’s about embracing complexity and asking the right questions,” said Davis.

In summary, mean-time reliability metrics provide helpful insight into the ongoing state of incident response. Yet, such metrics shouldn’t be imposed as a strict target — instead, they should be viewed as an informative guidepost. “Metrics can help you find what is discussion-worthy, but it’s not a discussion itself,” said Arnott.

July 7, 2022