devopsy.kinsta.cloud

Tag: observability

  • Digital Experience and the Future of Observability

    Digital Experience and the Future of Observability

    Over the past two years, there’s been significant hype around observability in the IT operations and DevOps world. As shorthand for providing better insights, observability has seen a rise in lockstep with distributed architectures, cloud-native applications and the increasingly complex IT environments associated with them.

    As we round out 2022 and move into the new year, there’s evidence observability is gaining mainstream adoption by IT operations and has begun to take off, with the market reaching a better understanding of what observability is and why it matters. This broader adoption offers a moment to reflect on the impact of observability so far and how it will continue to evolve in the year ahead.

    What is Observability?

    Though there’s ongoing debate in the industry about what observability actually means and what an observability platform should include, most people in the industry will argue observability is the evolution of monitoring and promises to offer more complete answers than standalone monitoring. Gartner, for example, first asserted in 2020 observability is “the evolution of monitoring.” But the two aren’t mutually exclusive, and observability and monitoring will continue to coexist and fill distinct needs for different teams based on what information they need about their environments and how they need to act on it.

    So far, observability has had the most considerable impact on application performance and digital experience monitoring.

    The days of monolithic applications are over. Instead, modern applications are highly distributed, are built natively in the cloud using container technology, are organized around microservices and leverage databases in multiple cloud environments. These increasingly distributed software stacks have introduced enormous flexibility for developers but huge complexity for operations teams, making it harder to deliver an optimal experience to the end user. The DevOps teams tasked with building and managing custom web applications are under pressure, and the result is, as one recent article so concisely put it, “Devs don’t want to do ops.”

    Observability has provided a welcome respite for DevOps teams who need intelligent visibility into their environments to determine how applications are performing. This has made application performance a key focus for many observability providers.

    An Intolerance of Bad Digital Experience

    Ensuring external customer-facing applications are performing optimally is the key driver behind this. Digital businesses with external applications core to the company’s success need to ensure those applications perform optimally—customers are fickle. They’re intolerant of bad digital experiences, and alternatives are plentiful. Any e-commerce vendor knows no website equals no sales.

    Traditional application performance monitoring (APM) tools look at telemetry to flag application issues or outages needing to be addressed. But understanding the health of your applications from this narrow standpoint isn’t the same as understanding how the applications perform from a user perspective. In other words, if you aren’t considering how applications are performing (or not!) for customers, employees and other users, you’re not seeing the forest for the trees. Observability helps solve this by elevating APM into a more holistic digital experience approach by adding visibility into how the end user engages with your application.

    By providing a view into how an application is performing with the context of what’s happening throughout the entire stack—including networks, databases, systems, and more—and how issues are impacting the user experience, DevOps teams can shed light on previous blind spots.

    Elevating the importance of digital experience through observability enables teams to understand and respond to issues more efficiently and provides critical data to help DevOps teams further improve their apps beyond just debugging and issue remediation. By understanding how users engage with and experience the application, you can gather critical data to improve or simplify the digital user experience. In our e-commerce example above, this could mean fewer clicks or an easier checkout experience (better digital experience = more sales).

    Cloud migration efforts, coupled with ongoing hybrid and remote work and understaffed tech teams, make prioritizing what matters most even more important. Focusing on the experience of the end user—but with end-to-end visibility across the environment—allows stretched DevOps teams “do more with less.”

    2023 Will be the Year of Digital Experience Observability

    An observability approach prioritizing digital experience is the first step toward building a complete customer-centric environment. This eliminates the potential risks of missing issues directly impacting application performance and creates an enhanced, complete experience for every user. This will make 2023 the year of digital experience observability.

  • Best of 2022: Day in the Life of a Site Reliability Engineer (SRE)

    Best of 2022: Day in the Life of a Site Reliability Engineer (SRE)

    As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    By now, most of us are familiar with the concept of site reliability engineering (SRE). The term was originally coined by Google and SRE has been gaining traction in recent years as a role dedicated to increasing the resilience of digital ecosystems. To accomplish this, a big part of the SRE doctrine is “automating your job away.”

    Observability and monitoring tools are extremely important to DevOps and for SRE. But what is the day-to-day experience actually like for your average SRE? What activities does an SRE do? And what percentage of their time does an SRE spend on particular tasks?

    I recently met with James Curtis, lead site reliability engineer at a large multinational company. According to Curtis, the SRE approach takes a certain type of person—someone with Ops in their blood and a passion for eradicating repetitive tasks. Below, we’ll use his input and the input from other sources to better understand what it’s like being in the shoes of an SRE.

    Understanding the SRE Role

    Organizations have interpreted the role in different ways. But in a general sense, SREs attempt to maintain high reliability and availability for software applications and respond to incidents as they occur. To aid their efforts, an SRE tries to streamline and automate as many operations as possible to remove opportunities for human error.

    A hallmark SRE goal is to reduce “toil.” Curtis defines toil as “tedious actions that really have no enduring value.” For example, say an admin must manually restart a service in Microsoft Exchange every time a triggering event interrupts the service—this action could certainly be automated away. SREs spend much of their time eliminating toil by coding automation and configuring internal tools to better interact with software infrastructure.

    SREs usually are also in charge of logs and setting benchmarks using a tool like Splunk or Datadog to observe and ingest data. Curtis himself uses Cribl, which offers an observability pipeline to parse and route log data. Since SREs oversee internal service-level indicators, they are typically in charge of normalizing behavior and setting SLOs and SLAs.

    Common SRE Activities

    An SRE juggles a lot of tasks. For proof, read the hour-by-hour day in the life by Yonatan Schultz, SRE at New Relic. Schultz’s average day is spent configuring infrastructure, jumping from project to project, and, of course, hopping into many meetings. Here are some other tasks an SRE might perform on a daily basis:

    Monitoring service-level indicators (SLIs). An SLI could be the number of successful requests out of total requests. Having a high SLI, in this case, would be a target. SREs track other metrics such as availability, uptime performance, latency, error count and throughput. Regularly monitoring systems is essential to ensure proper resource utilization of containers and to avoid out-of-memory (OOM) errors.

    Setting SLOs and SLAs and determining error budgets. Once you have determined baseline system performance, you can set service-level objectives (SLOs). These are typically internal targets like 99.99% availability. While SREs typically oversee functional metrics, some teams set goals for non-functional metrics, as well. SREs help determine service-level agreements (SLAs), which are more legally binding and typically partner-facing.

    Responding to incidents. On-call SREs will be tasked with finding the root cause of issues as they arise. When triaging an incident, it’s helpful to have all the necessary logs and tools immediately at hand. This is one area where automation can assist by pulling relevant details to instantly build a case, said Curtis.

    Writing postmortems. After an incident has been dealt with, it’s important to learn from it. Postmortems are common in cybersecurity practice and often fall under the responsibility of an SRE. These reviews seek to answer set criteria to get to the heart of an incident and identify the root cause(s) of an issue to prevent it from happening again.

    Automating other system tasks. SREs will spend significant time coding and building tools for engineers to interact with infrastructure. For instance, an SRE might generate reliability reports that consider performance over long time periods.

    Cross-department collaboration. SREs don’t tend to own application code. Instead, they support multiple software divisions. This means checking in with other developers, disseminating best practices and reviewing new architectures to represent the reliability side of the equation.

    As you can see above, the SRE role might blend many different activities, and keeping track of them all may be part of the job itself. Anika Mukherji, an SRE at Pinterest, shared that, at Pinterest, there is a weekly meeting where SREs share what they spent time on. For another helpful “day in the life” story, take this account of an average OpenShift SRE’s day. Nikita spends her day responding to open JIRA cards, handling incidents, pushing code to GitHub and syncing with SREs in other regions when shifts change.

    Time Well Spent

    So, how does an SRE allocate their time? As Curtis explained, the ideal goal is a 50/50 split between time spent in work mode and time trying to automate that work away. Of course, this is more like a sliding scale, he admits. When things are broken, attention naturally shifts toward more manual work, says Curtis. While there may be less automation upfront, the scale balances out the more you build.

    This is similar at other institutions too. For example, in an interview with DZone, Paul Greig explained that half of his time is spent on service reliability upkeep and half on toil reduction. John Turner, SRE at Squarespace, said some 70% of his time is spent writing code—much of it automation code.

    Who’s Fit to Be an SRE?

    As mentioned above, the SRE job requires a specific attitude toward solving operational problems. Curtis described this person as “someone who hates the monotony of doing something over and over.” This person should have a drive to continuously solve new problems because as soon as you’ve automated one thing, you move on to the next, he said.

    Don’t assume a role with high automation equals an easy paycheck. We’ve all heard the story of the system admin who secretly automated their job away and never told a soul. While this may fly in other roles, the job of an SRE is never finished. “You’re never going to run out of stuff to optimize,” Curtis said.

    Again, the SRE role is a blend of many different activities. You will likely interface with different developer teams, too. Since the role involves this type of interaction, communications skills and understanding are a must.

    Final Thoughts

    The SRE practice is wide and varied—each company may have its unique flavor. Looking to the future, there are plenty of use cases for machine learning to further empower SRE practices, said Curtis. This is especially relevant in security automation, where algorithms could be trained against real attacks to flag suspicious behavior. Or, with the right amount of data, predictive analytics could be applied to anticipate high CPU peaks, informing server utilizations.

    In both scenarios, Curtis stressed the importance of observability. “It really gives you the ability to look at data and to ask a question later you didn’t realize you needed to ask,” he said. This power hinges on easy data transformation to make things speak the same language. That’s why his team opted for Cribl, that allowed them to normalize that data on the fly and replay it later. “The ability to change data and morph it—that’s the power observability gives you.”

  • Opvizor Extends Monitoring Reach to the Cloud

    Opvizor Extends Monitoring Reach to the Cloud

    Opvizor, a unit of Codenotary, has extended its monitoring capabilities to cloud computing environments starting with support for Amazon Web Services (AWS) and Microsoft Azure.

    Codenotary CTO Dennis Zimmer said Opvizor aggregates data to provide a more efficient approach to applying analytics to surface performance and risk issues.

    Previously, Opvizor was optimized for instances of VMware vSphere running in an on-premises IT environment. Now the unit of Codenotary, a provider of a service for generating software bills of materials (SBOMs), is extending those monitoring capabilities to the cloud.

    That transition comes as many organizations are looking to strike a balance between monitoring pre-defined metrics and observability platforms that make it possible to surface unknown potential issues that could adversely affect IT environments.
    It’s still early days as far as observability is concerned, so Opvizor is focused on meeting most IT teams where they are today as they further embrace cloud computing, said Zimmer.

    Observability, in general, promises to query not just logs and metrics but also distributed traces that are applied to specific processes. In that way, observability allows organizations to detect patterns that have not been predefined. In effect, an observability platform makes it simpler to debug application environments where the number of dependencies that exist between services are becoming too complex to track manually, much less decipher. Without the aid of an observability platform it can take weeks to discover the exact cause of an application’s performance degradation.

    Of course, those applications will need to be instrumented using agent software capable of generating the data observability platforms require. Fortunately, open source projects such as OpenTelemetry are helping to drive down the total cost of instrumentation.

    However, in the absence of observability, most IT teams will rely on existing monitoring tools. Opvizor is making a case for centralizing monitoring across multiple platforms to reduce costs, noted Zimmer.

    Heading into 2023, each IT team will need to decide how and when to rely on traditional monitoring tools versus observability platforms. Many observability platform providers are justifying the acquisition of a new platform based on cost and the ability to eliminate the need for multiple monitoring tools. However, it remains to be seen whether IT organizations will prefer to employ both monitoring tools to track predefined metrics alongside observability platforms.

    Regardless of approach, the reality is that IT environments are becoming more complex to manage. In addition to running workloads in multiple cloud platforms and on-premises IT environments, organizations are deploying emerging cloud-native applications based on containers alongside existing monolithic applications deployed on legacy virtual machines. As workloads are deployed on multiple platforms, the total cost of IT starts to rise. One way to contain those costs is to standardize on a set of tools that can be applied to multiple platforms.

    The pressure to rationalize toolsets only increases during any period of economic uncertainty. The challenge, of course, is convincing one team to give up their preferred tool in favor of another to reduce costs.

  • New Relic Extends Observability Reach to More DevOps Platforms

    New Relic Extends Observability Reach to More DevOps Platforms

    New Relic today expanded its ecosystem by adding integrations between its observability platform and a wide variety of DevOps and cybersecurity tools. The New Relic observability platform now integrates with offerings from Atlassian, Amazon Web Services (AWS), CircleCI, Confluent, GitHub, JFrog, Lacework and Snyk and the open source Jenkins continuous integration/continuous delivery (CI/CD) platform.

    Peter Pezaris, senior vice president for strategy and experience at New Relic, said the goal is to make it simple to share analytics based on telemetry data collected from IT operations environments with the tools organizations rely on to build and deploy applications.

    In total, there are now more than 500 integrations with the New Relic observability platform, he noted.

    Only 27% of respondents to a recent New Relic survey reported that they have achieved full-stack observability and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints from end users. More than half (52%) of respondents said they experienced high-business-impact outages once per week or more and 29% said they take more than an hour to resolve those outages.

    On the plus side, nearly three-quarters (72%) of respondents expected to maintain or increase their observability budgets next year, with more than half (52%) expecting observability budgets to increase over the next year.

    Observability, in one form or another, has always been a tenet of any DevOps practice. For the most part, the best that could be achieved until recently was continuous monitoring of a set of pre-defined metrics. With the arrival of observability platforms, it’s becoming much simpler to not only monitor metrics but also launch queries that surface issues long before an IT service can be disrupted.

    An observability platform achieves that goal by making it possible to query logs, metrics and distributed traces in a way that surfaces patterns that have not been pre-defined. In effect, an observability platform makes it simpler to debug application environments where the number of dependencies that exist between services are becoming too complex to track or decipher manually.

    Of course, those applications will need to be instrumented using agent software capable of generating the data observability platforms require. Fortunately, open source projects such as OpenTelemetry are helping to drive down the total cost of instrumentation by making agent software more accessible to a wider range of application developers.

    It’s not clear how quickly IT teams will be embracing observability platforms, but there is now a clear opportunity to narrow the divide between IT operations teams and application developers, noted Pezaris. As each DevOps platform makes it easier to collect data via application programming interfaces (APIs), richer telemetry data can be collected, he added.

    In the meantime, as application environments continue to become more complex in the cloud-native era, most DevOps teams will need additional tools. Existing legacy monitoring tools don’t make it easy to discover, for example, which microservice might be adversely impacting the overall performance of an application because calls are being rerouted. The frustrating thing, as always, is that it can take days or weeks to pinpoint an issue that might take only a few minutes to fix.

  • A History of Distributed Tracing

    A History of Distributed Tracing

    Organizations are increasingly using distributed tracing to monitor their complex, microservice-based architectures. Distributed tracing has become essential in microservice applications, cloud-native and distributed systems. Microservices and serverless applications can grow exponentially, which makes observing them at scale very challenging. The traditional logging method becomes expensive, and if there’s an issue, time-series data can reveal symptoms but not the root cause(s). With modern usage patterns of Kubernetes service mesh like Istio and Envoy, for example, there can be billions of microservices calls daily.

    What is Distributed Tracing?

    Traces describe the progress of a transaction or workflow through a system. Distributed tracing provides visibility into service dependencies and their interdependencies by providing end-to-end visibility. Visualizing transactions in their entirety allows you to compare non-normal traces with normal ones to determine differences in behavior, structure and timing. Through distributed tracing, you can identify bottlenecks in your system. As a traced request flows through a network, it can be tracked. 

    As requests flow from frontend to backend and to databases, traces provide diagnostic techniques that reveal how a set of services coordinate to handle an application request. Using distributed tracing, developers can troubleshoot requests with errors and high latency. A single trace typically shows the activity of a request or transaction within an application. A trace shows what has changed at each step. As you aggregate the traces, you can see which backend service or database has the most significant impact on performance. 

    The History of Distributed Tracing

    Dapper, a large-scale distributed systems tracing infrastructure, was introduced by Google in 2010. Two years after Dapper was made public, Twitter open sourced Zipkin, which was designed for application performance tuning. Zipkin was the first open source distributed tracing project. Zipkin trace data can be collected and visualized using a UI. In 2015, Uber launched Jaeger, which was inspired by Dapper and named after the German word for hunter. Later in 2017, Uber published a blog post, called Evolving Distributed Tracing at Uber Engineering, explaining the reasons for the architectural choices in Jaeger. In addition to creating Jaeger, its author, Yuri Shkuro, wrote a book about distributed tracing called Mastering Distributed Tracing. 

    In 2016 Ben Sigelman, founder of Lightstep, wrote a blog post called Toward Turnkey Distributed Tracing, describing OpenTracing as a single standard. Some people refer to this as the OpenTracing Manifesto. OpenTracing allows developers of application code, open source packages and open source services to instrument their code without binding themselves to any particular tracing solution. The goal of OpenTracing was to solve a standardization problem. Trace context must pass through all the components, including application code, dependent libraries, standalone open source services (Nginx, MySQL) and other vendor-specific libraries and services, to collect a complete distributed trace. Collecting a full path without a standard API to define the collection and passing of trace context is difficult. OpenTracing aims to solve this problem by defining a standard API that can be implemented by components from different tracing solutions, allowing the collection of end-to-end tracing data. The Cloud Native Computing Foundation (CNCF) accepted OpenTracing as its third project in October 2016. Two months later, OpenTracing 1.0 was released. After OpenTracing, Jaeger joined the CNCF. 

    W3C tracing context specification was proposed in November 2019, bringing distributed tracing closer to standardization. Over ten years, distributed tracing evolved from one paper to an active community. With standardization on all the layers, it is moving from just tracing to overall observability, ranging from latency optimization to root cause analysis and application performance management. It is moving from a single backend system to an end-to-end solution that spans multiple systems. 

    OpenTracing and OpenTelemetry merged in 2019. Using OpenTelemetry, distributed tracing can be implemented end-to-end. It released version 1.0 in 2021. CNCF’s OpenTelemetry project is one of the fast-growing projects, and OpenTelemetry has become the de facto standard for tracing, metrics and logging.

    How it Works

    Traces in distributed tracing consist of tagged time intervals called spans. It is possible to think of a span as a unit of work. Traces are directed acyclic graphs (DAGs) composed of segments known as spans. According to OpenTelemetry, There are four major components of a trace:

    • traceID
        • A unique 16-byte array to identify the trace that a span is associated with
    • spanID
        • Hex-encoded 8-byte array to identify the current span
    • Trace Flags
        • Provides more details about the trace, such as if it is sampled
    • Trace State
        • Provides more vendor-specific information for tracing across multiple distributed systems. Please refer to W3C Trace Context for further explanation.

    The spans in a trace consist of contiguous segments of work that are named and timed. There are start and end times for spans, as well as metadata such as tags or logs that help classify them. Parent-child relationships may be present between spans to show how particular transactions traverse the application’s numerous services and components. 

     

     

    A trace represents an end-to-end request; it can contain one or more spans. Spans indicate work performed by a single service with associated time intervals and metadata. The purpose of traces is to provide a request-centric view of spans through tags. Despite microservices enabling teams to work independently, distributed tracing offers a central resource that allows all teams to understand issues from a user’s perspective.

    Conclusion

    In distributed systems, response latency can have a significant commercial impact. Identifying bottlenecks and understanding how a request moves through a complex system is complicated. Other techniques, like logging and monitoring metrics, don’t provide insight into distributed applications such as those created with microservices architecture. Several open standards and tools are emerging within the distributed tracing space, like OpenTracing and commercial tools that could compete with existing APM solutions. Implementing distributed tracing for modern cloud-native services poses several challenges. OpenTelemetry tracing is necessary for processing high volumes of trace data while generating meaningful insight.

  • Logz.io Unveils Managed Open 360 Observability Platform

    Logz.io Unveils Managed Open 360 Observability Platform

    At the AWS re:Invent 2022 conference, Logz.io launched an Open 360 platform that combines multiple open source technologies to provide observability across both existing monolithic and emerging cloud-native application environments.

    Existing Logz.io offerings included in the Open 360 platform are Logz.io Kubernetes 360, a managed observability platform based on open source tools such as OpenSearch analytics software, Prometheus monitoring tools and Jaeger distributed tracing agents.

    In addition, Open 360 includes Logz.io Telemetry Collector agent software, Logz.io Data Optimization Hub dashboard, Logz.io LogMetrics Index for converting log data into metrics and the Logz.io Trace Sampling Wizard tool for configuring open source OpenTelemetry Collector agent software.

    Logz.io CEO Tomer Levy said rather than requiring organizations to combine a set of disjointed open software tools to create an observability platform, instances of open source tools have been curated by Logz.io. The goal is to streamline the number of tools and platforms that DevOps teams will need to observe multiple classes of applications, he noted.

    It’s still early days as far as the adoption of observability platforms is concerned, but it’s apparent that as application environments become more complex the monitoring tools DevOps teams rely on today will need to evolve. Observability platforms, in general, promise to unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue rather than simply tracking a set of pre-defined metrics.

    The rate at which DevOps teams will embrace observability will naturally vary, but the choice many DevOps teams are trying to navigate is how much to rely on proprietary observability platforms rather than using open source software to construct their own. Logz.io is making a case for finding a middle ground between those two extremes by relying on a managed service based on open source tools.

    Observability has always been a core DevOps tenet, but achieving and maintaining it is challenging. Most DevOps teams today aspire to maintain some level of continuous monitoring. However, as it becomes easier and less costly to instrument applications, interest is rising in observability platforms that simplify the investigation of anomalies indicative of an issue that could disrupt a distributed application environment.

    More challenging still, DevOps teams are also now managing a much wider range of application types as cloud-native applications based on microservices deployed on Kubernetes clusters are deployed alongside legacy monolithic applications running in the cloud and on-premises IT environments. The Logz.io approach enables DevOps teams to manage those applications at a lower total cost, noted Levy.

    Regardless of the approach to observability, the increasing complexity of application environments has become a major challenge for organizations that are increasingly sensitive to costs. Many of them can’t afford to hire additional DevOps engineers. Instead, the focus is on increasing the productivity of existing DevOps teams using a new generation of observability platforms that should cost less to acquire and deploy than to expand existing teams of DevOps engineers. That may not necessarily eliminate the need to hire additional DevOps engineers, but it should ensure organizations are getting the most out of the DevOps teams they already have.

  • Nobl9 Simplifies SLO Identification

    Nobl9 Simplifies SLO Identification

    Nobl9 today added a Replay capability to its service level objectives (SLO) management platform that makes it simpler to collect the historical data a DevOps team needs to get started.

    Nobl9 COO Kit Merker said Nobl9 Replay makes service level indicator (SLI) data instantly available in minutes using 30 days of historical data collected from observability and application performance monitoring (APM) platforms. Previously, DevOps teams would have had to wait as long as a month before the Nobl9 platform could collect enough spreadsheet data to create a graph that identified the relationship between various services.

    Nobl9 now enables historical data to be merged with data collected in real-time to allow DevOps teams to view SLO charts and generate reports shortly after creating an SLO within a Git-based workflow.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing them that has garnered support from Dynatrace, Sumo Logic and others. SLOs, of course, are not a new idea; they have been used as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications.

    The Nobl9 platform is designed to enable DevOps teams to employ an SLO-as-code approach that makes it simpler to gather the metrics that confirm whether SLOs are being achieved.

    It’s not clear whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. A recent survey conducted by Dimensional Research on behalf of Nobl9, however, found a full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The issue is that most organizations have limited visibility into their IT environments. The survey found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult, even though 45% said they already employed 11 or more observability and monitoring tools. Trying to track SLOs using tools such as Grafana dashboards that DevOps teams need to set up and maintain is simply too difficult, noted Merker.

    As applications are increasingly constructed using services that drive digital business transformation initiatives, it’s now only a matter of time before SLOs are more widely tracked. That will enable DevOps teams to not only ascertain their current level of performance but also more easily work toward achieving their actual performance goals, he added.

    The challenge is that defining an SLO in a highly dynamic application environment where services are added and replaced with much greater frequency is often a lot easier than maintaining it.

  • Of Max and Min: When Performance Engineering Plans Go Awry

    Of Max and Min: When Performance Engineering Plans Go Awry

    Modern software developers have access to powerful tools and services which allow them to quickly develop, demo and deploy fully functional applications. But what happens when the single-user prototype satisfies all required functionality but the user says the system seems slow? Or if the initial implementation turns out fine for one user, but multiple users start to experience delays? Or if users are happy, but the cost of auto-scaling grows prohibitive?

    The Problem with Performance Engineering

    Concern over non-functional aspects of computing systems—response times, resource utilization and costs—falls under the domain of computer systems performance engineering (CSPE). Unfortunately, the practice of CSPE has evolved into something more akin to art than engineering—with few standardized principles and practices. Because software is constantly evolving its languages and abstractions, CSPE practitioners appear obligated to evolve and change their abstractions and languages, too.1 As a result, part of our jobs as performance engineers is to understand what the heck other performance engineers really intend or mean.

    There are many reasons for this state of affairs, but ultimately this lack of a common language and engineering standards for CSPE is a deficiency in the education and training of software engineers. There is very little in the typical university software engineering curriculum that prepares software engineers for doing CSPE. This sometimes leads to practices that are not backed by scientific or engineering principles. As a result, practitioners are forced to “wing it”—creating their own art or hitching their wagons to someone else’s star and learning (or not) through trial and error.

    The PMWG – A POSIX for Performance?

    The Performance Management Working Group (PMWG) was started in the late 1980s during the “UNIX wars,” when computer vendors were feuding over claims to the UNIX leadership mantle and had formed two factions: UNIX International and the Open Software Foundation. The group was unique in that its members included computer systems vendors on both sides of the “war.” Its members put aside those differences to try to come up with performance management standards and practices that would put UNIX on par (at least) with the performance management tools found on mainframes.

    In retrospect, the PMWG was probably the best chance UNIX had for establishing, promoting and enshrining some basic performance engineering principles—a kind of POSIX for performance. Unfortunately, the group failed—in large part, in my opinion, because it put too much priority on the latter stages of the performance management data pipeline and did not spend enough effort toward establishing key primitives. Today, in the absence of established rules and practices for performance engineering, history is repeating itself. Current efforts in observability continue to focus mainly on data presentation and data transport and storage, without much focus on the right metrics and data quality based on basic performance engineering principles. Presentation, transport and storage are important, but the efforts around metrics, logging and tracing coverage and data quality are sporadic at best.

    Inertia, Expediency and Superstition

    Why are more formal, engineering-based practices around CSPE needed? Consider the use of CPU utilization as a key performance indicator (KPI) of system performance. I’m a member of Bloomberg’s Trading Solutions SRE team, which manages hundreds of machines on which the company’s Trading Solutions software runs. We often see or hear statements like “This host’s CPU is too busy,” or “That host is out of CPU resources.” I’ve found that these observations are not actually very helpful and sometimes divert our attention from the real problem(s). When misused as a system KPI, focusing solely on CPU utilization can lead us down the wrong path and draw incorrect conclusions. To understand why, let’s first look at how we assess the performance of real-world, non-computing systems.

    Take fast-food restaurants. One of the most important attributes of fast-food restaurants (besides their health inspection rating) is that they are fast. But we’ve all been to fast food restaurants that are anything but fast. Without actually standing in line and waiting, we can often tell that we will have a slow experience at a fast food joint if there is a long line. The line length gives us a lot of information. Our assessment of the situation is different whether there are 100 customers versus 10 customers versus a single customer waiting in line.

    In the real world, this line length (or queue length) assessment is almost universal. Supermarket checkouts, ATMs, airport security screening, toll booths, elevators and COVID-19 testing are all areas where we apply this semi-conscious assessment. In contrast, we almost never use “cashier utilization” or “self-serve soda fountain utilization” as a way of assessing how slow a fast-food restaurant is. We always say “the lines are long” or, if we actually choose to wait in line, we’ll make the more direct statement that “service is slow.”

    Which brings us back to our computing systems. Why do we reflexively look to CPU utilization when computing systems are slow while, in the physical world, we rely on queue lengths? The question is even more relevant when we consider that the concept of CPU is less well-defined today. Are we measuring the CPU, its cores or hardware threads?2 I believe the contributing factors to this misuse include inertia, expediency and superstition.

    In the early days of computing, CPUs were perhaps the most expensive component of the large monolithic computer systems of the time. Measuring how much these expensive components were being used was an important part of maximizing the financial investment in these large machines. Fortunately, measures of CPU utilization were relatively easy to estimate. Ease of implementation meant that CPU utilization measures were almost universally available.

    The universal availability of this simple metric, combined with some correlation (sometimes weak) between high CPU utilization and system slowness made it a default indicator of systems performance. When a belief becomes ingrained to the point of superstition, even weak correlations are enough to validate and perpetuate it. Better metrics—for example, run queue lengths at the CPU(s)3—can help us shed our superstitions to better understand our systems and lead us in the right direction in identifying and correcting problems.

    Dispelling Superstition and Other Irrational Practices

    Which brings us to the title and purpose of this series. As a title, “Of Max and Min” is meant to evoke John Steinbeck’s classic novella “Of Mice and Men,” which in turn got its title from the famous line in Robert Burns’ poem “To a Mouse”: The best-laid plans of mice and men oft go awry. The two main characters in “Of Mice and Men” fail in their efforts to better their lives during the Great Depression because they are tragically ill-equipped to do so. Similarly, we should not be surprised if plans go awry when software engineers are tasked with doing performance engineering without the proper training in performance engineering principles.

    “Max and Min” also highlights the importance of data analysis and statistics in understanding systems behavior. Effective CSPE requires us to be familiar with concepts in statistics, numerical analysis, and even operations research—in addition to the more standard computer science areas of algorithmic complexity and computer architecture.

    With “Of Max and Min,” in addition to calling out superstitions, I will be pointing out some common mistakes we make and blind spots that we have.4 Through case studies, I’ll illustrate how our blind spots can lead to mistakes that can persist for years. I will also try to cover some basic CSPE principles that I believe are skipped in most software engineering training. Like everyone else, I’ve been winging it for 40+ years. And I’m still learning. If you disagree with any of my points or have further insights into the topics that I am presenting, please let me know.

    1But, as in the application of the principles behind algorithmic complexity, there are some performance engineering principles that can and should be adhered to regardless of the system and language du jour.

    2There are in fact some bizarre implementations of “CPU utilization”. Here’s an example of how AIX “broke” the meaning of CPU utilization on their multiprocessor systems running in hyper-threading mode.

    3UNIX and Linux do in fact have metrics that could be used as CPU queue length indicators.  In a future installment, I will go into the quirks of some of the CPU queue length implementations and further argue for more consistent mathematical bases for queue length metric implementations.

    4For example, have you ever wondered why, after so many outages caused by logging, malloc, DNS and other low-level services, we still don’t have good, out-of-the-box visibility into these and other low-level components? We are accepting key blind spots as a fact of life when we really shouldn’t.

    About Me

    Like many software engineers, I started off studying a different field (Physics). Unlike most software engineers, I’ve always wanted to be a performance engineer. When I started taking computer classes as an undergraduate in Columbia College’s Physics program, I did not wish for a career writing programs in Fortran (yes, it was that long ago). But I did envision myself making systems run better and faster.

    To that end, when I added a Computer Science major to my undergraduate studies, I also added a minor in Industrial Engineering and Operations Research. For those unfamiliar with IEOR, it is a multidisciplinary engineering field that leverages mathematical and analytical methods to address complex system problems like resource allocation, supply chains, wait times, fairness, etc. Tools and methodologies from IEOR are invaluable to CSPE.

    At the start of my career, I joined the Bell Labs Computer Center as a computer systems performance engineer, where I made the somewhat contrarian decision (for 1980) to work in the group supporting their in-house, low-key UNIX systems (as opposed to their commercially popular MVS mainframes). I didn’t realize it at the time, but having access to the source code of the system you support/study can be a boon to understanding. At Bell Labs, I had the opportunity to contribute to the UNIX System V kernel, where I co-developed the first general-purpose kernel and user-land tracing system for UNIX and made significant performance enhancements to the virtual memory subsystem, and participated in the PMWG.

    Since then, I’ve spent the bulk of my professional career working on financial software systems—ranging from market data to messaging to transaction management. I’ve worked on monitoring systems on an enterprise level. I’ve made and witnessed many CSPE mistakes. In my current role as an SRE on the Trading Solutions team at Bloomberg, I’m seeing and helping remedy lots of operating system issues that stem from our runtime scale.

    Thanks

    Thanks to everyone who reviewed this document and provided constructive feedback. Special thanks to Nate McNamara and Peter Wainwright for suggesting substantive organizational and structural improvements. Peter also reined in my default stream of consciousness, run-on style. None of them had any hand in the writing of this paragraph.

     

     

  • Gigamon Delivers Pivotal Advancements Across Deep Observability Pipeline with GigaVUE 6.0 Release

    IT organizations applaud next-generation container network visibility, application metadata integration, and cloud-scale performance for centrally processing network telemetry data

    SANTA CLARA, Calif.- October 11, 2022 — Gigamon, the leading deep observability company, today announced an expansive set of new capabilities to the Gigamon Hawk Deep Observability Pipeline. GigaVUE 6.0, the software powering Gigamon Hawk, democratizes security delivery across networking, security, and cloud operations teams while dramatically reducing cross-platform cost and complexity. Gigamon Hawk helps IT teams scale up and scale out with confidence, eliminate security and performance blind spots across cloud traffic and unmanaged hosts, and gain application transparency to secure against threats-in-motion across distributed hybrid and multi-cloud infrastructure. Gigamon Hawk amplifies the power of observability by accessing network traffic at the source, whether virtual, container, or physical, and efficiently aggregates, optimizes, and enriches it while serving actionable, network-level intelligence to any tooling, anywhere.

    Changing Requirements in a Cloud-First World

    According to Flexera, 92 percent of large organizations have embraced a multi-cloud infrastructure strategy. In addition, according to market research firm EMA, 96 percent of large organizations expect to integrate network packet and observability intelligence, with 77 percent planning to use network-level intelligence to fortify hybrid cloud security. Consequently, 81 percent of organizations have experienced a cloud-related security incident in the past year.

    A New Frontier: Deep Observability

    Today’s release introduces a series of major advancements including:

    ●  Next-Gen Container Network Visibility – GigaVUE now enables the acquisition of container traffic, delivering the industry’s first universal container tap (UCT) solution deployable with any container network interface (CNI) and any container orchestration, powered by the enhanced Berkeley Packet Filter (eBPF) for network-level observability with minimal overhead. Deep observability for security and performance can now follow the workload no matter the ephemerality or scale of container-based micro-services.

    ●  Application Metadata Integration Framework – Gigamon Hawk application metadata engine integrates directly with leading observability platforms, including Dynatrace, New Relic, and Sumo Logic, using JavaScript object notation (JSON) and Kafka. This integration enables existing observability tools to perform new security functions, such as identifying rogue services, activities, and illegal crypto mining.

    ●  Cloud-Scale Network Telemetry Processing – The new GigaVUE-HC1 Plus visibility appliance, an underlying component of Gigamon Hawk, delivers twice the pipeline processing performance in half the physical footprint and power requirements, compared to the previous generation. This purpose-built appliance can efficiently and economically process network telemetry by aggregating telemetry from physical, virtual, and/or container workloads, including telemetry from public cloud workloads.

    “We’re excited about the new universal container tap (UCT) solution in the latest release of GigaVUE Cloud Suite as it shows how Gigamon is continuing to innovate to stay current with modern software architectures, including a wide range of Kubernetes-based container deployments,” said Steve Watt, principal, president at Layer8. “Gigamon is heading in the right direction.” Layer8 is a leading value-added Gigamon channel partner that provides comprehensive strategy, design, implementation, staffing, and support services for IT applications and infrastructure.

    “As we have moved workloads to the cloud over the last few years, one of the challenges has been maintaining visibility into key applications no matter where they are running,” said Jerry Hulbert, senior vice president of high impact technology solutions at Sev1Tech. “The Gigamon Hawk Deep Observability Pipeline brings actionable network-level intelligence to all our SIEM, observability, and performance tools, speeding up our troubleshooting and lowering operational costs to secure complex hybrid and multi-cloud IT infrastructure.”

    “IT organizations are increasingly relying on multiple sources of telemetry, including metrics, events, logs and traces to analyze, troubleshoot, and optimize hybrid and multi-cloud IT infrastructure,” said Akhil Kapoor, vice president of cloud and product partnerships at New Relic. “Our partnership with Gigamon enriches our telemetry data platform with application metadata intelligence that spans more than 5,000 attributes to provide cloud and security operations teams with the deep observability they need to identify and eliminate security blind spots and performance bottlenecks between virtual machines, containers, and within and between on-premises, public, and private clouds.”

    “Today’s GigaVUE 6.0 software release provides IT organizations with next-generation, actionable network-level intelligence that delivers granularity up to the application layer for all workloads including private and public cloud and all container environments,” said Michael Dickman, chief product officer at Gigamon. “Our deep observability pipeline is the realization of our vision to bring rich, ‘ground truth’ network intelligence to our customers, allowing them to solve their most pressing security and operational challenges to manage distributed hybrid and multi-cloud infrastructure efficiently and effectively.”

    Join the immersive, virtual deep dive technical sessions at Gigamon Visualyze Bootcamp to learn more about the latest innovations and how they bring deep observability to hybrid and multi-cloud environments.

    About Gigamon

    Gigamon® offers a deep observability pipeline that harnesses actionable network-level intelligence to amplify the power of observability tools. This powerful combination helps IT organizations to assure security and compliance governance, speed root-cause analysis of performance bottlenecks, and lower operational overhead associated with managing hybrid and multi-cloud IT infrastructures. The result: modern enterprises realize the full transformational promise of the cloud. Gigamon serves more than 4,000 customers worldwide, including over 80 percent of Fortune 100 enterprises, 9 of the 10 largest mobile network providers, and hundreds of governments and educational organizations worldwide. To learn more, please visit gigamon.com.

    © 2022 Gigamon. All rights reserved. Gigamon and the Gigamon logo are trademarks of Gigamon in the United States and/or other countries. Gigamon trademarks can be found at www.gigamon.com/legal-trademarks. All other trademarks are the property of their respective owners.

  • ServiceNow Acquires Era Software to Unify Observability

    ServiceNow Acquires Era Software to Unify Observability

    ServiceNow today announced it has agreed to acquire Era Software as part of an effort to add support for log data to the Lightstep observability platform the company acquired last year.

    Ben Sigelman, general manager for the Lightstep business unit at ServiceNow, said rather than reinventing the same log management capability that Era Software already has, it makes more sense to integrate that capability with the Lightstep platform that collects metrics and traces.

    The integration of those two platforms will then make it possible to unify observability rather than requiring DevOps teams to navigate two separate silos, he added.

    As observability continues to evolve and mature, the most immediate goal is to make it simpler to navigate the alert storm that inevitably ensues any time there is a disruption, said Sigelman. The alert storms generally occur after there has been a change made, but unraveling how exactly that change impacted an IT environment is challenging, he noted.

    An observability platform reduces the time and effort necessary to achieve that goal by surfacing a set of statistics that guide DevOps teams through that process, added Sigelman. The challenge, of course, has always been finding an efficient means of storing all the telemetry data that’s generated. In fact, as organizations start to deploy microservices-based cloud-native applications, many of them are starting to see the amount of telemetry data increase exponentially.

    Era Software uses an object storage system to index and analyze log data collected from cloud applications, Kubernetes and IT infrastructure platforms. The company has also made available in beta an Era Streams tool that reduces the volume of observability data collected without sacrificing fidelity by dynamically reconfiguring data pipelines in real-time.

    Once the acquisition is completed, log data analytics will then be shared with the IT service management (ITSM) tools that ServiceNow provides alongside the Lightstep observability platform. The goal is to make it easier to correlate those IT events with their potential impact on the business, said Sigelman.

    Era Software CEO Todd Persen said the long-term 10-year goal is to get to the point where issues are both automatically detected and remediated. In the meantime, the amount of context switching that IT teams encounter can be sharply reduced via a unified observability platform.

    There is, of course, no shortage of unified observability platforms these days. DevOps teams have also been relying on legacy monitoring tools that track specific metrics for years. The issue is that those legacy monitoring tools only track a narrow range of pre-determined metrics. As IT environments become more complex, the need to launch more open-ended queries to determine the root cause of IT disruption has become more pronounced.

    It may be some time before observability platforms are widely adopted, but the need for them is generally apparent to anyone managing a modern IT environment. Less clear is the degree to which observability platforms will remove the need for legacy monitoring tools; however, right now the focus should be on gaining as much visibility and transparency as possible into what are still opaque IT environments.