devopsy.kinsta.cloud

Tag: application performance monitoring

  • Survey Surfaces Lack of Significant Observability Progress

    Survey Surfaces Lack of Significant Observability Progress

    A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to truly observe application environments, especially as they become more complex with each passing day.

    The survey, conducted by Logz.io, a provider of an observability platform, found only one in 10 respondents reported they have full observability into their application environments.

    Logz.io CTO Asaf Yigal said while more DevOps teams than ever are collecting logs, metrics and traces, most of them have yet to determine how to operationalize all the data being collected. As data is ingested, it doesn’t just need to be stored; it also needs to be correlated to the various services that make up an application, he noted.

    The more organizations deploy cloud-native applications in production environments the more pressing this issue becomes. All the microservices that make up those applications are now generating a massive amount of telemetry data that results in more alerts than ever being generated.

    Not surprisingly, the biggest challenges organizations encounter when managing Kubernetes clusters in production environments are monitoring/troubleshooting (40%), followed closely by security (37%) and networking (33%).

    Many organizations simply don’t have the skills required to manage cloud-native applications. Nearly half of respondents (48%) specifically cited a lack of knowledge as the biggest challenge they encountered when trying to observe these types of applications. From an IT management perspective, most microservices all use the same basic template, so it’s difficult for DevOps teams to identify which microservices are likely to have the biggest impact on service level objectives (SLOs) and service level agreements (SLAs) in the event of a disruption, noted Yigal.

    In the absence of the ability to determine the actual root cause of an issue, alert storms conspire to increase fatigue—and that eventually results in higher levels of burnout across the DevOps team, noted Yigal. In fact, 82% of respondents said their mean-time-to-resolution (MTTR) during production incidents was more than an hour.

    The survey also noted that more than half of respondents (52%) worked for organizations that are simultaneously trying to rein in monitoring costs. More than three-quarters (76%) of respondents also noted that open source OpenTelemetry (OTEL) or OTEL-centric tooling was at least somewhat important to their overall observability strategy.

    In addition, 87% of respondents said their organization is already using some form of platform engineering to manage DevOps workflows at scale.

    It’s not clear how quickly organizations are adopting observability tools and platforms, but too many DevOps teams lack the visibility needed to identify the root cause of an issue. As a result, variations of the same problems often keep manifesting themselves because previous remediation efforts simply didn’t go deep enough to resolve the core issue.

    Of course, there may come a day when machine learning algorithms, along with other forms of artificial intelligence (AI), will make it simpler to surface these issues. The challenge, in the meantime, is laying down the observability foundation today to provide access to the data that will be required to train those AI models.

  • Datadog Updates Aim to Improve Logging, Testing Efficiency

    Datadog Updates Aim to Improve Logging, Testing Efficiency

    At its recent DASH 2023 conference, Datadog revealed Flex Logs and Intelligent Test Runner, two major new capabilities aimed at improving logging and testing efficiency for engineering teams.

    Omri Sass, director of product management at Datadog, said Flex Logs introduces a new middle tier of log storage between real-time querying and cold archives. As Sass explained, this provides a cheaper option for retaining logs beyond the typical two weeks for occasional analytics and investigations. By decoupling storage from compute for reading logs, teams can independently scale each based on their needs. This enables retaining rich log data for months or years to uncover long-term trends, he said.

    In addition, Sass noted that Flex Logs helps address common log analysis limitations around storage costs and rehydrating archives. The feature better aligns log pricing with specific use cases beyond just troubleshooting. According to Sass, this evolution in log storage tiers will likely shift more logs from legacy on-premises systems to its cloud platform.

    The second capability, Intelligent Test Runner, leverages observability signals to optimize CI test suites. As Sass explained, comprehensive test automation in CI pipelines often increases deployment times and flakes. Intelligent Test Runner inspects code changes to determine associated tests, ignoring irrelevant ones, which helps avoid long wait times for minor changes just to exercise unaffected areas.

    By tapping into existing telemetry around code commits, test health and runtimes, the feature adapts testing to each code modification. Sass explained that observability data has matured enough to enable real-time optimization like this. The end result is faster CI validation, so engineering teams can ship updates more rapidly.

    Sass highlighted how Flex Logs and Intelligent Test Runner are Datadog’s attempts to address real pain points developers face around managing log data life cycles and CI speed bumps. Sass said he believes this combination of depth in core disciplines like logs and application performance monitoring (APM), along with expanding into adjacent developer workflows, is key to solving these issues through innovation.

    Looking to the future, the observability space continues to evolve rapidly as emerging technologies and practices expand monitoring beyond traditional IT infrastructure. Companies handling modern workloads expect their observability platform to provide comprehensive visibility as development toolchains and architectures grow more complex, with the next generation of observability likely incorporating advanced analytics, automation and AI to help engineers tame systems at ever-increasing scale and speed. 

  • Is Your Monitoring Strategy Scalable?

    Is Your Monitoring Strategy Scalable?

    Every company is trying to get better insights into its operational effectiveness, and they are running into the same problem: Scale. So what does a scalable monitoring strategy look like and how can you safeguard against the most significant issue in observability?

    What is a Scalable Monitoring Strategy?

    We’ll begin by identifying the two things most impacted by scale: Cost and performance. Cost can be broken down into storage and computation. It is obvious that to hold more data, more storage is needed, but what about compute power? This is important because queries need to search through more data—and thus, use more compute power—to return a result.

    This creates a trade-off between performance and cost-effectiveness. It becomes more and more challenging to have queries that resolve instantly as the dataset increases, so engineers will often simply tolerate a performance decline. This further impacts usability and the general usefulness of the system. If it takes 10 seconds to resolve every query, you’ve just cut down your data discovery efficiency by a factor of 10.

    Scaling Means Performance and Cost-Effectiveness

    To scale a monitoring strategy, a wise architect needs to break this standoff between performance and cost and approach the problem like a data scientist. A scalable monitoring strategy starts with a simple question: What is the use case for this data?

    Some data is only ever ingested because it might be needed. It spends its entire lifetime undisturbed and is eventually compressed or deleted. On the other hand, some data is queried every minute of every day and is integral. Knowing how data is used means an engineer can decide the value of that data.

    In the battle to create a scalable monitoring strategy, there’s a three-step approach.

    Step One: Track Data Usage

    Which data is ingested and never queried? Which data never spends any time at rest? Build a map of how often data is consumed so that any strategic choices are informed by the use case.

    It is most common to group data into three different use cases:

    Frequent Access – Data that is constantly queried and needs to be available at a moment’s notice.

    Monitoring – Data that drives dashboards or trains machine learning models but which is largely useless after it has been processed.

    Compliance – Data that is held in case it is needed, but it is only queried sometimes. For example, audit logs.

    Step Two: Optimize Storage and Consumption

    Once the use case of data—whether it’s logs, metrics or traces—is understood, the next stage is to optimize. This means creating different storage and query solutions for the different usages we’ve seen above.

    Frequent Access – Rapid queries that can be optimized and tuned. OpenSearch is a good option, although management overhead can be painful, especially at scale.

    Monitoring – This is largely about transforming data. For example, ingesting logs, converting them into metrics and deleting the original log. This is very powerful because metrics take up far less space than logs and can be stored much more cost-effectively.

    Compliance – Low-cost storage like Amazon S3 is a good option, but this data must still be accessible, even if simply by reindexing.

    How Easy is it to Build These Capabilities?

    It is trivial to create an OpenSearch cluster, but it’s challenging to manage it at scale. Likewise, it’s simple to convert logs to metrics, but doing so in a performant way at scale is complex. Holding onto compliance logs as they scale near infinitely can be difficult, and reindexing this data is a non-trivial operation when dealing with large data volumes.

    However, the key takeaway from this analysis should be that these are capabilities that any organization should have if they intend to scale their monitoring solutions. If they can be attained through a SaaS vendor, then this is a serious option to consider because it enables companies to immediately take advantage of this capability without the upfront, unpredictable and often ongoing cost of in-house engineering.

  • Observability Costs are too Damn High

    Observability Costs are too Damn High

    Today, any business that deploys software faces an obscene amount of expenses. It has to pay for cloud hosting, assuming of course that it’s among the 92% of companies that use the cloud. It needs trustworthy networks to connect its apps to its employees and customers. It probably pays for a suite of cybersecurity software in a bid to stay ahead of the ever-growing list of cybersecurity threats. All of these costs, which are unavoidable for any company that wants to host modern software, add up to higher overall IT bills.

    But there’s a source of excess spending that has grown tremendously in recent years: Observability. If a company deploys software, they also need to monitor and observe that software to ensure that it meets performance and availability requirements. In recent years, the cost of observability has skyrocketed.

    In fact, according to Charity Majors of Honeycomb, the total cost of observability at an organization today is equivalent to up to 30% of total infrastructure spending. That’s a truly astounding figure, if you think about it. For every dollar spent on the infrastructure that powers applications, 30 cents ends up paying just for the tools and services needed to make sure the apps running on that infrastructure are doing what they’re supposed to.

    On top of this, spending on observability tools has become very unpredictable, meaning companies have no idea what they are going to be paying next week, next month or next year to get the insights they need to manage their applications and infrastructure. In this sense, observability is worse than taxes. With taxes, at least you know ahead of time what you’re going to have to pay in most cases.

    Getting Observability Spending Under Control

    Observability costs won’t go down on their own,  but there are practical steps that organizations can take to get observability spending under control–and they can do it without sacrificing the critical visibility they need to manage complex systems.

    To prove the point, we’ll discuss the reasons why observability has become so expensive. It’s partly about technology, but it also has to do with who gets to make decisions about adopting observability tools and how cultural inertia inside organizations impedes the ability of practitioners to migrate to more cost-efficient observability solutions.

    Then, we’ll talk about actionable strategies for getting observability costs under control. By taking advantage of some new tools and strategies, observability spending can be reined in while also–and this is the best part–actually increasing the organization’s ability to observe and manage complex systems.

    Why Observability Costs so Much

    There’s no one simple reason why observability costs for the typical organization today have increased. Instead, several factors have conspired to bloat observability bills.

    Drowning in Data

    One issue is the sheer volume of observability data that organizations have to collect.

    That’s partly because businesses deploy so many individual applications–207 on average, according to Okta–each of which generates observability data, to say nothing of the infrastructure on which the apps are hosted. But it’s even more so because of the fact that cloud-native architectures have vastly increased the number of application components and services that we need to observe.

    Instead of having one monolithic app and one host server to monitor, today, companies might have two dozen microservices, a Kubernetes orchestration plane, a bunch of nodes and perhaps an underlying IaaS platform, all of which they have to observe just to keep a single app running healthily. That adds up to tremendously more observability data than businesses had to manage in the past.

    Viewed from one perspective, having more observability data is a good thing. The more data there is, and the more granularity that can be associated with particular parts of the stack, the more insights DevOps teams can glean, at least in theory. But the problem is that, in many cases, only a small fraction–perhaps as little as 1%–of the data that is collected and ingested into observability tools is actually ever used to detect or troubleshoot performance issues.

    Granted, part of the reason we collect so much data but use so little is that it’s impossible to know ahead of time which observability data we’ll actually need. But that’s not an excuse for paying for data that never serves a useful purpose. If we’re going to spend so much on observability data, we might as well use more of it while also finding ways to pay less for it.

    Inefficient Data Collection

    The problem of ever-growing data volumes is compounded by the challenge of inefficient data collectors, which translate to higher infrastructure costs.

    Here’s why: The traditional way to deploy observability tools is to rely on agent-based data collectors that run essentially as standalone applications alongside the applications they are monitoring. As a result, the observability agents suck up a non-insignificant amount of CPU and memory, which require more infrastructure resources to support them and, thus, higher costs.

    Hence, one of the grand ironies of our times: To figure out whether their applications are consuming resources efficiently, businesses deploy observability software that actually decreases resource efficiency, while also bloating costs.

    Complex Pricing Models

    The pricing models of the typical observability tool are, in a word, complex. Most vendors charge in part based on how much observability data is ingested into their tools (this is usually the biggest contributor to overall cost), but they might also charge based on how many agents are deployed, how many users the tool has within the organization, and which features are wanted to access. The costs might also vary depending on where the data is chosen to be stored, how long of a retention period, how often it needs to be accessed  and so on.

    Complex pricing makes it hard in many cases for organizations to cost-optimize their observability spending. Determining exactly which features and usage tiers are needed can be a tough job, and companies might easily end up overspending as a result (after all, who wants to risk underspending and ending up with observability gaps that lead to critical failures?). Plus, unlike, public cloud services, there are no cost-optimization tools designed to help businesses “rightsize” their observability tool deployments or architectures.

    Organizational Inertia

    Historically, most observability and APM solutions were designed first and foremost for developers. To use them, developers had to instrument observability into applications. Then, it fell on the operations or DevOps team to connect the apps to APM tools and figure out what to do with the observability data they generated.

    There’s nothing wrong with involving developers in the observability process. However, the pitfall of the developer-centric approach to APM and observability that organizations have tended to follow is that developers typically don’t actually know very much about what the operations team needs to observe. After all, developers build apps; they don’t monitor or troubleshoot them. The result of this is that developers design apps to generate all sorts of observability data that may or may not be useful (which is why, again, organizations end up paying for a whole bunch of data they don’t actually use).

    This might not be much of an issue if it were easy for the operations team to come in and say, “Hey, we’re collecting all this data and it’s dumb. Let’s be more strategic.” But sadly, it’s not. Organizations being organizations, effecting change is not easy, especially if the change means getting people to adopt new tools or practices. So, businesses end up stuck with observability solutions that may have worked well in the past, but that are not efficient by modern standards, due simply to organizational inertia.

    Cost Unpredictability

    As mentioned, the problem with observability costs today is not only that they’re too damn high. It’s also that they’re too damn unpredictable.

    Observability vendors don’t hide their costs, so those are easy enough to predict. But those costs, again, are based largely on how much data is ingested into the observability tool. And how many engineers do you know who can predict with any kind of accuracy how many logs and metrics their applications will generate from one day to the next?

    Most can’t because data volume is tied to application demand, and that’s inherently unpredictable. After all, if we knew exactly how many requests our applications were going to receive from one moment to the next, we probably wouldn’t really need observability at all. But we don’t, and part of the core purpose of observability tools is to ensure that we can detect the issues that arise due to unexpected software usage patterns.

    I am not exactly criticizing observability vendors here, but rather pointing out that their pricing models capitalize on a fundamental limitation of modern applications: The amount of observability data they’ll produce is unpredictable in most cases, which can lead to wild fluctuations in observability spending.

    Keeping Observability Costs in Check

    Just as there’s no one single cause of high or unpredictable observability costs, there’s no one trick that can rein the spending in. But there are several strategies that, used in unison, will bring observability costs under control:

    Analyze Data at the Source

    The more data is moved in order to observe it, the more it will end up costing. By the same token, the more data analyzed at its source, the lower the cost will be.

    This is why, for instance, Kubernetes observability will cost much less if the data is collected and analyzed right inside nodes, instead of shipping it out to an observability platform first. Not only will the results be faster (because there is no need to wait for the data to move) but data that isn’t critical will be able to be ignored , which reduces how much data is ingested into the observability tools and cuts down on overall costs.

    Keep Data at the Source

    In a similar vein, not all data needs to leave its source at all. There might be datathat needs to be kept handy in case it is needed later, but  can not be analyzed right now. Instead of moving it into external storage, which will cost more, it should be kept right where it originated – inside the nodes. It’s there if needed, but there is no need to  payi extra for it.

    Add Efficiency With eBPF

    I mentioned before the grand irony in which organizations end up wasting infrastructure resources to deploy observability tools: There’s a pretty simple solution for that. It’s called eBPF, and it’s an ultra-efficient way to collect observability data. Unlike conventional observability and APM software, eBPF data collectors run directly inside the operating system kernel instead of running as user space applications. That leads to much lower levels of CPU and memory consumption. With eBPF, there is  all the observability needed, without sacrificing resources in the process.

    Affordable Observability

    The cost of deploying software these days is going up overall, but observability costs can actually go in the opposite direction. Thanks to new observability tools and techniques, it’s possible to collect all of the data neeeded, without allocating a tremendous portion of the overall IT budget to it.

    So, say goodbye to bloated budgets, and say hello to affordable observability.

  • 5 Technologies Powering Cloud Optimization

    5 Technologies Powering Cloud Optimization

    Cloud optimization is the process of improving the performance, cost-effectiveness, and reliability of cloud computing resources and services. It involves identifying and addressing inefficiencies in cloud architecture, application design and resource utilization, as well as implementing best practices for security, scalability and disaster recovery. 

    The goal of cloud optimization is to maximize the benefits of cloud computing while minimizing the risks and costs associated with it. It is a multi-disciplinary process that may involve several key elements, such as:

    • Capacity planning: Evaluating the current and future resource requirements of your cloud environment and ensuring that the right amount of resources are available to meet your needs.
    • Resource utilization: Monitoring and managing the utilization of cloud resources to ensure that they are used efficiently and effectively.
    • Cost optimization: Minimizing the cost of cloud resources and services by identifying and addressing areas of inefficiency and waste, such as over-provisioning or underutilized resources.
    • Performance optimization: Improving the performance of cloud applications and services by optimizing network bandwidth, storage performance and database performance.
    • Security optimization: Enhancing the security of cloud resources and data by implementing best practices for access control, encryption and threat mitigation.
    • Scalability optimization: Making sure that your cloud environment can scale up or down as needed to meet changing resource demands.
    • Disaster recovery: Establishing a disaster recovery plan to ensure that your cloud environment can quickly recover from failures or outages.

    Why is Cloud Optimization Important?

    Optimization is important for several reasons, including:

    • Cost savings: By optimizing the use of cloud resources, organizations can reduce their cloud computing costs and ensure that they are only paying for what they actually need.
    • Improved performance: By optimizing cloud applications and services, organizations can improve the performance and responsiveness of their systems, leading to a better user experience and increased productivity.
    • Increased reliability: Optimization helps organizations to improve the reliability and availability of their cloud environment, reducing the risk of downtime and ensuring that their systems are always available when they are needed.
    • Better security: By implementing best practices for security, organizations can better protect their cloud resources and data, reducing the risk of data breaches and other security incidents.
    • Scalability: Cloud optimization enables organizations to scale their environment up or down as needed, ensuring that they always have the resources they need to support their growing business.
    • Improved agility: Organizations can be more agile and responsive to changing business needs, enabling them to quickly and effectively respond to new opportunities and challenges.

    5 Technologies Powering Cloud Optimization

    Cloud Cost Management

    Cloud cost management is a critical component of optimization that helps organizations to monitor and manage their cloud spend. The goal is to ensure that organizations are only paying for the cloud resources they actually need and that they are using those resources efficiently.

    Cloud cost management solutions (such as Azure Cost Management or AWS Trusted Advisor) provide detailed cost reports, usage analytics and cost optimization recommendations to help organizations understand their cloud spend and identify areas of inefficiency. For example, these solutions can:

    • Monitor and report on cloud resource usage: By monitoring the usage of resources, cloud cost management solutions can provide detailed cost reports that show how much an organization is spending on each resource, such as compute instances, storage and network traffic.
    • Automate cost optimization: Many cloud cost management solutions can automate cost optimization tasks, such as turning off unused resources or right-sizing instances to reduce waste.
    • Provide cost analysis and forecasting: These solutions can provide detailed cost analysis and forecasting to help organizations understand the impact of changes to their cloud environment and plan for future spending.

    Autoscaling

    Autoscaling is a technology that enables organizations to automatically scale their cloud resources up or down as needed to meet changing demands. The goal of autoscaling is to ensure that organizations always have the right amount of resources to support their workloads while minimizing costs and ensuring that their systems are always available when they are needed.

    Autoscaling works by monitoring the performance and usage of cloud resources, such as compute instances, storage and network traffic, and automatically adjusting the size of those resources to meet changing demand. For example, autoscaling can:

    • Add or remove instances as needed: If demand for a resource increases, autoscaling can automatically add more instances to meet that demand. If demand decreases, autoscaling can automatically remove instances to reduce waste.
    • Optimize resource utilization: Autoscaling can optimize the utilization of cloud resources by right-sizing instances to meet demand, ensuring that resources are not over-provisioned or underutilized.
    • Improve system availability: By automatically adding or removing resources as needed, autoscaling can ensure that systems are always available and responsive, reducing the risk of downtime and improving the user experience.

    API Gateway

    An API gateway is a server that acts as an intermediary between an application and one or more microservices. The API gateway is responsible for request routing, composition and protocol translation, which enables microservices to communicate with each other securely and efficiently.

    An API gateway can help with cloud optimization in several ways:

    • Load balancing: The API gateway can distribute incoming API requests to multiple microservices, improving the overall performance and reliability of the system. It can also provide features such as automatic failover and load balancing, which helps to ensure that the system remains available and responsive even in the face of high traffic volumes or other performance challenges.
    • Resource optimization: By acting as a single entry point for API requests, the API gateway can help to reduce the overall resource usage of the system. For example, it can cache frequently used data and serve it directly to clients without having to go back to the microservices, reducing the load on the underlying system.
    • Monitoring and management: API gateways provide a centralized management interface for the microservices, making it easier to monitor and manage the system as a whole. This includes features such as logging, monitoring and alerting, which can help to identify and resolve performance issues quickly and effectively.

    GitOps Platforms

    GitOps is a paradigm that helps automate application deployment and infrastructure provisioning. It typically involves using Git, an open source version control system, as a single source of truth for declarative infrastructure and applications. 

    GitOps aims to improve the developer experience by letting teams use their familiar software development tools to manage application and infrastructure changes.

    With GitOps, cloud optimization can be achieved by:

    • Automating the deployment process: GitOps automates the deployment process, reducing the risk of human error and speeding up the deployment process. This helps to ensure that resources are deployed consistently and correctly, improving the reliability and performance of the environment.
    • Managing cloud infrastructure as code: With GitOps, cloud infrastructure can be defined and managed as code, making version control and collaborating on configurations easier. This helps to optimize the use of resources and improve the overall efficiency of the cloud environment.
    • Improving collaboration and communication: GitOps enables teams to collaborate and communicate more effectively, reducing the risk of conflicts and delays. With Git, changes can be easily tracked, discussed and approved, improving the speed and quality of deployments.
    • Enforcing governance and compliance: GitOps provides a centralized and auditable record of all changes to the environment, making it easier to enforce governance and compliance requirements. This helps to improve the security and reliability of the cloud environment, reducing the risk of breaches and other security incidents.

    APM

    Cloud application performance management (cloud APM) is the practice of monitoring, analyzing and optimizing the performance of cloud-based applications. The goal is to ensure that these applications are available, responsive and performing optimally to meet the needs of users.

    These solutions provide a range of tools and capabilities for monitoring and optimizing cloud-based applications, including:

    • Performance monitoring: Cloud APM solutions monitor application performance in real-time, providing visibility into performance metrics such as response time, resource utilization and error rates.
    • Root cause analysis: These solutions can help identify the root cause of performance issues by providing detailed performance data and tracing information.
    • Capacity planning: Cloud APM solutions can provide detailed performance data and analytics to help organizations plan and optimize their application deployments.
    • Performance optimization: These solutions can provide recommendations for performance optimization, such as right-sizing cloud resources, reducing resource utilization and improving application design.
    • Continuous performance monitoring: These solutions can continuously monitor application performance, providing real-time visibility and alerting to help organizations quickly identify and resolve performance issues.

    Conclusion

    In conclusion, cloud optimization is critical for organizations looking to maximize the value and minimize the costs of their cloud deployments. By using technologies such as cloud cost management, autoscaling, API gateways, GitOps platforms and APM, organizations can improve availability, performance and efficiency while reducing costs and ensuring that they are always prepared to meet changing business needs. 

    These technologies provide the tools and capabilities necessary to monitor, analyze and optimize cloud-based applications, helping organizations to ensure that their deployments are always delivering the best possible value. By embracing optimization, organizations can stay ahead of the curve and ensure that their cloud deployments are always aligned with their business goals and objectives.

  • Dynatrace Embeds DX Monitoring in Observability Platform

    Dynatrace Embeds DX Monitoring in Observability Platform

    Dynatrace has enhanced its analytics capabilities to enable digital experience monitoring, including session replays, based on the logs, metrics and traces it collects via its observability platform.

    Steve Tack, senior vice president of product management at Dynatrace, said this update makes it simpler for DevOps teams to track user journeys across the multiple components and microservices that make up a modern application environment. Rather than trying to, for example, determine the impact of an issue by parsing through log data without any context, Tack said Dynatrace’s analytics capabilities will automatically fire up a session replay to enable DevOps teams to view the exact nature of the application issue being experienced.

    Previously, DevOps teams would have needed to deploy the Dynatrace session replay offering separately. The overall goal is to remove much of the friction DevOps teams encounter when trying to fix an application issue by providing context in a way that is more accessible, he added.

    In general, as observability continues to advance, it’s now become much less about the amount and types of data that can be collected, said Tack. The data is table stakes for achieving observability, he added. The focus now needs to be on the richness of the actionable analytics being surfaced once that data is collected, he said.

    In fact, as more organizations start to build and deploy applications based on highly interdependent microservices, the richness of the analytics is now more critical than ever, noted Tack. While microservices-based applications are more resilient than monolithic applications, troubleshooting them is more challenging whenever application performance degrades, he added.

    It’s not clear at what rate DevOps teams are embracing observability. The concept has always been a core DevOps principle, but, for the most part, more mature DevOps teams have only been able to achieve what amounts to continuous monitoring of pre-defined metrics. Observability platforms promise to surface anomalies indicative of IT issues before they escalate; DevOps teams can then launch queries to better determine their root cause and potential severity.

    Dynatrace is making a case for an observability platform, accessed as a cloud service, that employs the company’s Davis AI engine and machine learning algorithms to surface the root cause of application issues.

    As observability platforms continue to evolve, the days when IT teams determined the root cause of an IT issue by process of elimination are finally coming to an end. DevOps teams will be gaining more visibility into their application environments in ways that surface more meaningful actionable intelligence. That intelligence is especially critical for externally-facing applications driving digital business transformation initiatives.

    There is, of course, no shortage of options when it comes to observability platforms. In some cases, IT organizations are opting to rely on cloud services while others are extending DevOps platforms they built themselves and continue to maintain. Regardless of the approach, the issue will no longer be how to collect data as much as what to do with it once it’s collected.

  • Deciphering the Observability Market

    Deciphering the Observability Market

    Observability has become one of the most overused buzzwords in IT and cybersecurity. Today, the term is used by vendors to refer to everything from application performance to network monitoring, cybersecurity and data and analytics.

    While the term’s ubiquity has created confusion for everyone from end users to journalists, startups in the space have also attracted over $2 billion dollars in venture capital investment over the past two years. This momentum prompted TechCrunch to ask if this area, specifically data observability, was effectively recession-proof.

    This observation triggers two questions. First, what is observability? And second, what are the different kinds or variants?

    What is Observability?

    Observability, as a broad practice or capability, originated in the 1960s as part of industrial control theory. The idea is that by watching the output of a system, you can figure out what’s happening inside the black box. This sounds a lot like monitoring, a popular practice for IT and security teams. However, observability takes a different approach than monitoring’s alert-based methods; it allows you to ask questions about a system that dig deeper than the pre-defined thresholds from your monitoring system. 

    Observability requires collecting massive amounts of data from systems, networks and applications to feed its discovery process. As systems and applications become more complex, figuring out what went wrong and why becomes much more challenging. At a high level, these are the kinds of challenges many startups in this area are addressing. Each takes a different approach and solves a different part of the observability problem.

    Types of Observability

    The most frequently mentioned type is data observability, which is concerned with the health and quality of data passing through data pipelines for analytical use cases like feeding a data warehouse. Vendors like Acceldata, Bigeye or Monte Carlo Data typically target their products to data engineers tasked with building and operating analytical data pipelines. 

    Another group of companies addresses applications. These firms, like Honeycomb and Observe.ai, collect data from applications to help site reliability engineers (SREs) understand performance issues and aid them in debugging and troubleshooting. 

    There are also companies specializing in machine learning observability, and these are concerned with the performance and drift of models in production. These companies, like Arize and WhyLabs, target data scientists. 

    Finally, there are companies in the observability data space which is comprised of logs, events, metrics and traces essential for all other forms of observability and monitoring to work. Companies in this space, like Cribl, use specially-designed pipelines to connect the sources and destinations of observability and security data. These pipelines allow companies to route data to multiple destinations, enrich data in flight and reduce data volumes before ingestion.

    As you can see, there are dozens of companies in the space. Each category of vendors addresses a unique challenge in today’s sprawling IT and security landscape.

    Beware Observability-Washing

    As the hype around this topic grows, companies hoping to differentiate themselves from their competitors may suddenly rebrand as observability companies. This is already happening. Legacy monitoring and alerting companies are calling themselves observability companies, hoping to attract the same kind of attention as others in the space that actually do have differentiated products and positioning.