devopsy.kinsta.cloud

Category: Application Performance Management/Monitoring

  • Building an Open Source Observability Platform

    Building an Open Source Observability Platform

    Enterprises are under constant pressure to ensure their IT infrastructure and applications are available throughout the year. The complexity of modern architectures (containers, hybrid cloud, SOA, microservices, etc.) is constantly growing, generating vast volumes of unmanageable logs. We need intelligent application performance management (APM) and observability tools to achieve production excellence and meet the availability and uptime goals. These include analyzing application health, performance and user experience. Adopting machine learning techniques to identify anomalies and behavior patterns will help detect the root cause early and meet customer service level agreements (SLA).

    The APM and observability tools market is undoubtedly hot. These tools ingest multiple telemetry data feeds and are powerful analytics platforms providing critical insights into application and infrastructure health, including system performance. The software development teams that adopt observability are much better equipped to release their application code iteratively. As per “MarketsandMarkets” research, the market size of observability tools and platforms is expected to grow from $2.4 billion in 2023 to $4+ billion by 2028 at a compound annual growth rate (CAGR) of 11.7%.

    What is Observability?

    Observability is the ability to collect data about the distributed applications, infrastructure and communication among its internal and external components and services, allowing teams to debug their systems diligently. It enables the site reliability engineering (SRE), software engineering and operations teams to analyze the customer impact and triage a service outage. Observability and monitoring are sometimes used interchangeably. Observability (proactive) makes the data accessible and allows you to ask any system question to know how the code behaves more deeply. Monitoring (reactive) is the task of collecting and displaying that data and the ability to determine the system’s overall state.
    Observability can be further broken down into three key pillars: Logs, traces and metrics, which are essential for SRE observability.

    • Logs help us diagnose the issues and tell us why they happened.
    • Traces help us isolate issues and tell us where they happened.
    • Metrics help us detect the issues and tell us what happened.

    Market Tools, Capabilities and Challenges

    Gartner’s magic quadrant for APM and observability has identified 20+ vendor products offering APM and observability capabilities, including self-hosted, vendor-managed or SaaS deployments. These products provide multiple features, including application performance metrics, monitoring and alerting on events, traceability, anomaly detection and vulnerabilities, etc.

    An enterprise business application comprises homegrown applications (such as .NET, Java, Python, SQL, NoSQL DB, etc.), third-party/off-the-shelf products (such as Salesforce, HubSpot, etc.), and integrations (such as Stripe, PayPal, etc.). Homegrown applications are hosted in an on-premises data center or by cloud vendors like AWS, GCP, or Azure. The off-the-shelf products are SaaS-based or integrated via APIs. There are highly distributed applications spanning tens and hundreds of nodes, services and instances.

    • Too Many Tools: Enterprise applications use various tools for monitoring application health and performance (such as New Relic, Data Dog, etc.), Error logging (such as Splunk), and cloud vendor-provided tools (such as CloudWatch). These products overlap in functionality and maintaining and managing these tools (procurement, learning curve, etc.) can be cumbersome.

    • Unpredictable Data Volume: Imagine the volume of observability data (logs, traces, metrics) collected based on the application traffic, usage, dependency on external products, etc. The amount of data storage required to consolidate these data feeds can quickly grow out of control.

    • Pricing is Complex: These vendor products also offer different pricing models such as charge per host (such as Splunk, Data Dog, Dynatrace), per-user (such as New Relic), per-ingestion (such as SumoLogic, AppDynamics). The complexity of pricing models makes it challenging to compare total cost of ownership (TCO) among vendors and determine the right tool that fits your requirements and budget.

    Why Choose Open Source Observability Platform?

    Open source-based observability tools aim to provide a standard, vendor-agnostic approach for ingesting, transforming and sending data to an observability backend. The open source observability tools can be an alternative to save on licensing costs and consolidate multiple APM tools with the tool that fits your requirements and budget.

    However, maintaining open-source systems may require efforts to set up and maintain and will add to your initial operational cost. But in the long term, you will save on licensing fees and avoid vendor lock-in and contractual agreements.

    Gartner predicts that, by 2025, 70% of new cloud-native application monitoring will use open-source instrumentation rather than vendor-specific agents for improved interoperability, and 70% of new cloud-native applications will adopt OpenTelemetry for observability rather than vendor-specific agents and software development kits (SDKs).

    Scale Observability Using the Open Source Ecosystem

    The open source landscape for observability is quite dynamic. There are multiple Cloud Native Computing Foundation (CNCF) open source tools for observability and monitoring. This post will primarily focus on the OpenTelemetry framework and LGTM technology stack.

    OpenTelemetry:
    The “too many tools” challenge described above brings up a new struggle in telemetry data collection. Each tool vendor has its own APIs, SDKs, agents and collectors for logs, metrics and traces. We need a unified telemetry collection using the OpenTelemetry framework to create and manage telemetry data such as logs, traces and metrics.

    The OTEL project under the auspices of the CNCF offers a unified set of vendor-agnostic APIs, SDKs, and tools for generating and collecting telemetry data and exporting it to various analysis tools. You get one API and SDK per programming language to extract your application’s observability data, a standard collector, a transmission protocol (OTLP) and more.

    LGTM:
    The most popular open source-based observability and monitoring is implemented using the LGTM technology stack.

    In the LGTM stack, we leverage:
    • Loki for log aggregation
    • Grafana dashboards for telemetry visualizations
    • Tempo (or Jaeger) for trace aggregation
    • Managed Prometheus for metrics aggregation

    Conclusion

    Observability is about complete visibility across your systems and tying business metrics with technical data. Monitoring is about understanding if things are working correctly, and AIOps is about getting meaning from that visibility. Observability and monitoring are critical to ensure the smooth functioning of applications and meet customer SLAs. In conclusion, by investing in open source OTel frameworks and LGTM tools, SRE teams can effectively monitor their applications and gain insights into system behavior and potential issues. These tools offer cost-effectiveness and customization to suit specific requirements. It promotes vendor neutrality, which can be critical for avoiding vendor lock-in.

  • Survey Surfaces Lack of Significant Observability Progress

    Survey Surfaces Lack of Significant Observability Progress

    A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to truly observe application environments, especially as they become more complex with each passing day.

    The survey, conducted by Logz.io, a provider of an observability platform, found only one in 10 respondents reported they have full observability into their application environments.

    Logz.io CTO Asaf Yigal said while more DevOps teams than ever are collecting logs, metrics and traces, most of them have yet to determine how to operationalize all the data being collected. As data is ingested, it doesn’t just need to be stored; it also needs to be correlated to the various services that make up an application, he noted.

    The more organizations deploy cloud-native applications in production environments the more pressing this issue becomes. All the microservices that make up those applications are now generating a massive amount of telemetry data that results in more alerts than ever being generated.

    Not surprisingly, the biggest challenges organizations encounter when managing Kubernetes clusters in production environments are monitoring/troubleshooting (40%), followed closely by security (37%) and networking (33%).

    Many organizations simply don’t have the skills required to manage cloud-native applications. Nearly half of respondents (48%) specifically cited a lack of knowledge as the biggest challenge they encountered when trying to observe these types of applications. From an IT management perspective, most microservices all use the same basic template, so it’s difficult for DevOps teams to identify which microservices are likely to have the biggest impact on service level objectives (SLOs) and service level agreements (SLAs) in the event of a disruption, noted Yigal.

    In the absence of the ability to determine the actual root cause of an issue, alert storms conspire to increase fatigue—and that eventually results in higher levels of burnout across the DevOps team, noted Yigal. In fact, 82% of respondents said their mean-time-to-resolution (MTTR) during production incidents was more than an hour.

    The survey also noted that more than half of respondents (52%) worked for organizations that are simultaneously trying to rein in monitoring costs. More than three-quarters (76%) of respondents also noted that open source OpenTelemetry (OTEL) or OTEL-centric tooling was at least somewhat important to their overall observability strategy.

    In addition, 87% of respondents said their organization is already using some form of platform engineering to manage DevOps workflows at scale.

    It’s not clear how quickly organizations are adopting observability tools and platforms, but too many DevOps teams lack the visibility needed to identify the root cause of an issue. As a result, variations of the same problems often keep manifesting themselves because previous remediation efforts simply didn’t go deep enough to resolve the core issue.

    Of course, there may come a day when machine learning algorithms, along with other forms of artificial intelligence (AI), will make it simpler to surface these issues. The challenge, in the meantime, is laying down the observability foundation today to provide access to the data that will be required to train those AI models.

  • Survey Sees Cloud Developer Environment Adoption Gaining Momentum

    Survey Sees Cloud Developer Environment Adoption Gaining Momentum

    A global survey found that while 95% of developers and business leaders are familiar with cloud development environments (CDEs), the reasons for adopting one vary widely. The survey polled 223 developers and business leaders working for organizations with more than 2,000 workers and 250 developers.

    Conducted by Coder, a provider of a CDE, the survey, for example, found that while 15% cited optimizing the developer environment setup process as a reason for adopting a CDE, another 14% cited making management tasks such as onboarding or improving remote access easier.

    Coder CEO Rob Whiteley said while CDEs clearly improve the productivity of application development teams and improve security by reducing the amount of source code running on notebook and desktop workstations, organizations looking to adopt CDEs need a significant amount of cultural resistance. Application developers have historically set up their own development environments on their own local machines. That approach, however, means that in addition to making it more challenging to collaborate also requires organizations to spend significant time and effort locking down each endpoint being employed by their developers, noted Whiteley.

    In fact, those issues are why many platform engineering teams formed to centralize the management of DevOps workflows are implementing CDEs, he added. The ultimate goal is to establish a set of guardrails while still enabling developers to self-service their tooling requirements, noted Whiteley.

    In fact, many DevOps teams have already built a homegrown version of a CDE that they currently maintain. Coder and other providers of CDEs are making a case for using a platform supported by a vendor rather than devoting resources to supporting a custom CDE, said Whiteley.

    As DevOps workflows continue to evolve in the age of artificial intelligence (AI), it’s only a matter of time before more organizations embrace CDEs, he added. The sheer volume of data that developers will be working with will make it impractical to continue to use local machines to build applications, said Whiteley.

    Regardless of the motivation for adopting CDEs, significant cultural changes are coming to DevOps workflows that need to evolve as application development continues to become more distributed. In the post-COVID-19 era, it’s now not uncommon for application development teams to consist of IT professionals working in different time zones. That creates a pressing need to leverage cloud services to provide the level of collaboration those teams need to achieve.

    It’s not clear how many CDEs have been deployed, but as the pace at which applications are being built and deployed continues to accelerate, the days when organizations could allow each developer to maintain their own development environment are coming to an end. Organizations need to determine as quickly as possible whether the code being created will actually run in a production environment. The more time spent trying to reconcile differences between development environments and the production environment where the application runs, the longer it will take to complete any given project.

    Unfortunately, time is the one thing that most DevOps teams no longer have the luxury of spending on application development initiatives that, in addition to multiplying, are also becoming much more complex to manage with each passing day.

  • OpenTelemetry Project Maintainers Add Code Profiling Capabilities

    OpenTelemetry Project Maintainers Add Code Profiling Capabilities

    This week at the KubeCon + CloudNativeCon Europe conference, the maintainers of the open source OpenTelemetry observability agent software project being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) revealed they have added profiling capabilities to enable DevOps teams to identify the root cause of issues down to a specific line of code.

    Austin Parker, director of open source for Honeycomb and a maintainer of OpenTelemetry, said while several providers of IT observability platforms have in the past year added code profiling capabilities, it makes more sense to provide this as a capability surfaced by OpenTelemetry in a way that any IT platform can easily consume.

    The overall goal is to reduce the amount of time it takes to troubleshoot IT issues by identifying not just the code at issue but also the development team responsible for it, noted Parker.

    OpenTelemetry defines a format that makes it simpler to centrally manage the collection metrics, logs and traces using open source software. In general, OpenTelemetry continues to gain traction as DevOps teams look to rationalize the amount of commercial agent software they need to deploy, update and manage.

    In effect, OpenTelemetry is aggregating agent software that, in terms of observability, provides no differentiated value, said Parker. That’s critical because as application environments become more complex, it’s all but impossible to manage them without being able to broadly apply the agent software required to instrument them.

    In the future, OpenTelemetry is expected to add additional capabilities such as data pre-processing, filtering or routing to redact sensitive data in a way that still fosters observability.

    Ultimately, as OpenTelemetry becomes more widely employed, it makes it significantly easier for developers to understand how their application is performing, noted Parker. Over time, armed with those insights, developers will build and deliver higher-quality applications, he added.

    It’s still early days as far as adoption of OpenTelemetry is concerned, but it’s clearly well on its way to becoming a de facto standard for enabling observability. The number of developers who have access to not just metrics and logs but also traces should steadily increase as the cost of instrumenting code declines.

    Of course, finding a way to collect that data inexpensively will also be crucial for applying artificial intelligence (AI) to DevOps workflows. After all, before an AI model can be trained, it needs access to verified data.

    In the meantime, DevOps teams might want to consider at what rate they can replace existing commercial agents deployed across the enterprise. Much of that agent software is more mature than OpenTelemetry, but over time, the economic arguments for using open source agent software become compelling. Many of the providers of observability platforms have already made support for OpenTelemerty available by default.

    Regardless of approach, the need for observability has always been apparent. The issue is that there is a world of difference between monitoring a pre-defined set of metrics and being able to launch queries to identify the root cause of an issue that metrics alone are not going to be able to surface.

  • JFrog Survey Surfaces Raft of DevSecOps Challenges

    JFrog Survey Surfaces Raft of DevSecOps Challenges

    A global survey of 1,224 security, development, and IT operations professionals published today by JFrog finds that 60% typically spend four days or more remediating application vulnerabilities in a given month.

    JFrog security researchers, however, noted it’s probable most of those vulnerabilities are not as severe as they have been rated. After analyzing 212 vulnerabilities, the JFrog Security Research team downgraded the severity of 85% of the vulnerabilities rated as critical and 73% of one rated as high.

    JFrog researchers also found that 74% of the reported common vulnerabilities and exposure (CVEs) with high and critical CVSS scores assigned to the top 100 Docker Hub community images weren’t actually exploitable.

    Only 17% of the vulnerabilities analyzed enabled remote code execution, compared to 44% that could enable a denial of service (DoS) attack, the researchers noted.

    Stephen Chin, senior director for developer relations for JFrog, said that analysis suggests organizations are arguably spending too much time on application security at the expense of productivity. Tools that generate alerts based on severity rather than also rating exploitability are creating too much noise simply because they don’t provide enough contextual analysis, he added.

    Conversely, in the absence of that level of detailed analysis, it’s possible a low-level vulnerability that might be especially severe in a specific IT environment, also known as a false negative, might not be further investigated, noted Chin.

    Security concerns, however, also limit innovation, with 40% of the survey respondents noting that because of security reviews, it typically takes a week or longer to get approval to use a new package/library.

    Nearly half of IT professionals (47%) say they use between four and nine application security tools, with 90% reporting they use some type of tools that rely on machine learning algorithms or other forms of artificial intelligence (AI) to scan, identify and remediate vulnerabilities.

    Less than a third (32%), however, work for organizations that use AI to write code. That relatively low level suggests many organizations are still not comfortable with the quality of the code generated, noted Chin. Of course, individual developers may still be using those tools without approval.

    More than half (53%) say their organization uses between four to nine programming languages, with nearly a third (31%) reporting they use 10 or more.

    The most frequently used tools are static application security testing (61%), dynamic application security testing (58%), software composition analysis testing (56%), and application programming interface (API) security (56%).

    A full 89% said their organization has adopted a security framework such as OpenSSF or Supply-chain Levels for Software Artifacts (SLSA).

    Finally, the survey finds little consensus concerning when to run security scans, with 42% saying it’s best to perform security scans as code is being written versus 41% that prefer to perform scans on new software packages before installing them. A total of 41% said runtime is the least desirable place to run scans. More than half (56%) said their organization applies security scans at both the code and binary scanning levels.

    The degree to which DevSecOps best practices are being adopted will naturally vary from one organization to the next. The challenge is to prioritize remediation efforts in a way that enables developers to spend more time writing, for example, business logic versus patching vulnerabilities are not actually a material threat. In that context, the quality of the vulnerability analysis generated by scanning tools matters because of the unnecessary toil that might be eliminated, noted Chin.

    It may be a while before most organizations are able to strike the right balance when it comes to DevSecOps, but as a greater appreciation for the nuances of application security continues to take hold, the less stress there should hopefully one day be for all concerned.

  • Leveraging eBPF for DevSecOps

    Leveraging eBPF for DevSecOps

    eBPF is a revolutionary technology originating from the Linux kernel. It is used to safely and efficiently extend the kernel’s capabilities without requiring changing the kernel source code or loading kernel modules/extensions.

    Today, eBPF is used extensively to:

    • Provide high-performance networking and security in modern data centers and cloud-native environments
    • Extract fine-grained security observability data at low overhead
    • Provide insights for performance troubleshooting
    • Provide preventive application and container runtime security enforcement

    eBPF-Enriched Context

    eBPF is the base data layer that is needed in runtime cloud security. eBPF gives you information at a high level on Layers 3, 4 and 7, allowing you to see all network communication and correlate it with processes and specific applications that are communicating on those ports.

    To fully leverage eBPF, you need to pair this raw data with additional cloud infrastructure context to gain a full understanding of which resources and services are in use.

    For example, by leveraging eBPF, you can determine if you have 10 different applications all communicating on port 443, whereas a traditional network monitoring tool would tell you that you have a certain amount of traffic on this port but wouldn’t be able to segment this traffic and show how much of that traffic is coming from each individual application.

    Knowing which communication is coming from an individual application increases visibility. It allows you to see what your applications are doing in real-time and identify signs of compromise or decreased performance.

    Increasing Observability With eBPF

    eBPF provides users with intrinsic, comprehensive insight into communication flows across the entire system. The applications for enhancing security observability with eBPF are vast, ranging from enabling highly detailed process visibility to facilitating end-to-end observation of processes and flows.

    When dealing with Layer 4 to Layer 7 traffic involving multiple protocols, eBPF emerges as an optimal solution. Through eBPF, TLS traffic traversing pods, nodes and cloud platforms can now be uniformly instrumented across layers.

    In other words, with eBPF, you can observe your application and your node and gain a vast amount of raw data, which, with the right tools, you can correlate to understand your entire network topology.

    Beyond eBPF Raw Data to Context and Correlation: Layer 7 and Beyond

    Specifically for observability, you can use eBPF to apply filters directly in the kernel, which reduces the overhead and allows you to ingest significantly more raw data. eBPF’s architecture allows you to do all of this, observe network traffic and Layers 3, 4 and 7 without compromising your security or performance.

    With eBPF, you are able to understand the execution events, system call events and all of the network ingress and egress. But it’s not always enough.

    You will need to correlate it with Cloud APIs, Kubernetes identities, pods and namespaces to understand the actual identities and relationships between the individual workloads, all the way down to the application layer. By doing so, you are able to understand application-level identity and use this context to understand your application behavior as well as your most critical risks and threats.

    Leveraging eBPF for Threat Detection and Response

    Security and networking observability are inherently interconnected. eBPF helps you to gain insights such as abrupt shifts in behaviors, spikes in activity within specific processes indicating potential exploitation or attacks, service-to-service communication trends and more. These projects enable users to examine network transactions, identifying the pods, processes and system calls involved.

    With eBPF, you can see OS-level runtime threats. eBPF gives you the ability to listen to system calls but also to go one step further and block or terminate any process, PID (process identifier), system call or network packet. This allows for a more granular approach to automated threat detection and response without damaging your system or hurting your infrastructure.

    You can also use eBPF to implement prevention rules, meaning you can instruct eBPF to automatically block any malicious processes or pre-designated system calls or network packets, going beyond traditional uses of eBPF for monitoring and security, to become a more proactive, preventative tool for securing your infrastructure and applications.

    Real-World Benefits of eBPF

    eBPF presents a number of real-world benefits, going beyond traditional monitoring, observability and security to help with overall alert noise reduction and improved forensic analysis.

    How eBPF Contributes to Noise Reduction

    By itself, eBPF does not reduce noise. In fact, it increases because of the influx of raw data from every system call. However, with the right tools, you can leverage this raw data and correlate it with additional context to understand your infrastructure and application behavior. This effectively reduces the alert noise that you would otherwise receive because you can view more detailed insights into resource behavior.

    By filtering and aggregating this data, you receive more insights about applications, overall giving you less noise because you’re focusing on what actually matters.

    By having this eBPF raw data at runtime, you can apply intelligent filters to filter through the most important data and use it accordingly, resulting in better prioritization, more focused efforts and increased operational efficiency.

    Forensic Analysis

    Another important real-world use case for eBPF is conducting forensic analysis. By leveraging eBPF’s raw runtime data and applying intelligent filters, you can quickly pinpoint problem origins and conduct forensic analysis at the application level.

    By leveraging eBPF’s raw data from Layer 7, you gain a granular understanding of applications and the process-level communication within an application, allowing you to foresee the potential blast radius if an application were compromised in terms of who it could interact with and what damage could occur.

    Conclusion

    In summary, eBPF is a powerful technology that can be effectively leveraged for monitoring, observability and security purposes. However, when eBPF’s raw data is combined with context from cloud APIs, it can be leveraged even further with intelligent filters to provide unparalleled cloud infrastructure and application context, allowing you to understand application-layer identity, provide real-time security, pinpoint problem origins and stop threats with surgical, active response.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation, Techstrong Group and the entire cloud-native community in Paris, France at KubeCon+CloudNativeCon EU 2024 – March 19-22, 2024.

  • Sumo Logic Eliminates Ingest Fees for Observability Log Data

    Sumo Logic Eliminates Ingest Fees for Observability Log Data

    Sumo Logic this week announced it will no longer charge a fee for ingesting log data into its observability platform to encourage DevOps teams to apply analytics more deeply.

    Michael Cucchi, vice president of product marketing for Sumo Logic, said the Sumo Logic Flex Licensing plan eliminates what amounts to a tax for using an observability platform. Flex Licensing is being made available immediately to new customers and will be offered to existing Sumo Logic customers later this year. There are no hidden monthly charges, feature restrictions, performance tradeoffs or user limitations being applied under the plan.

    The licensing plan only applies to log data, but DevOps teams will be able to ingest all structured, semi-structured and unstructured data for a lower cost that Sumo Logic is working to store more efficiently, said Cucchi. The overall goal is to make available a disruptive approach to licensing that lowers the total cost of observability, he added.

    Storing log data is going to be increasingly critical as organizations begin to apply multiple types of artificial intelligence (AI) models to further automate IT operations, he added. Those AI models promise to make observability platforms more accessible because algorithms will be able to surface issues that need to be addressed. Historically, the value of an observability platform has been tied to how proficient a DevOps team became when using its query language.

    IT teams that have adopted observability platforms have been, to varying degrees, limiting the amount of data they collect and retain to keep storage costs from exploding. The challenge is that as DevOps teams deploy more microservices-based applications, the amount of log data generated has exploded. Determining the root cause of any issue involving those applications can be problematic if log data isn’t readily available. Log data is, after all, the atomic unit for observability, noted Cucchi.

    In general, IT environments are becoming too complex for humans to manage without the aid of observability platforms augmented by AI models, noted Cucchi. Each application environment is fairly unique, so it is pivotal for those AI models to be exposed to as much data as possible to ensure optimal results.

    It’s not clear whether observability platforms might one day replace the need for many of the monitoring tools that IT teams rely on today to track a set of pre-defined metrics. However, one thing that is certain is that observability platforms, at the very least, present an opportunity to rationalize some of those tools. The biggest challenge is finding ways to fund the acquisition of an observability platform in the first place, which then creates an opportunity to consolidate those tools later.

    Given the complexity of highly distributed computing platforms, it’s only a matter of time before most IT teams have some type of observability capability in place. Of course, observability has always been a core DevOps tenet. The trouble is that the tools being used to achieve it are legacy monitoring platforms that don’t provide the level of depth required to successfully manage modern application environments.

    Image source: Joshua Sortino via Unsplash.