devopsy.kinsta.cloud

Tag: observability

  • Why Your Data Lake Strategy is Broken for Observability

    Why Your Data Lake Strategy is Broken for Observability

    Someone once said, “Life is better at the lake.” That may be true in the physical world, but when it comes to using data lakes for observability, that’s not typically the case.

    Just about every organization in every industry has a digital presence, whether it’s a website, an app or another digital service. This makes observability a universal need; it helps teams monitor software performance, optimize performance, detect potential security issues and quickly monitor for bugs if and when they occur. These responsibilities typically fall on engineering teams and require as close to real-time response as possible.

    Some organizations also have business intelligence (BI) needs, aimed at generating insights from multiple data sources for business decision-making. These responsibilities typically fall on business decision-makers; they are not real-time decisions, but rather made by looking back over long time periods.

    All-In on a Data Lake Strategy

    Companies that go all-in on a data lake strategy may ignore the needs of the engineering teams working on the real-time mission-critical aspects of the business and force all of the company’s data, queries, dashboards, etc. into a data lake solution. That doesn’t actually meet the needs of the engineering teams. The result? Engineering teams use unsanctioned solutions, fracturing the company’s data strategy from the ground up.

    By using a standards-based observability solution built on open source InfluxDB 3.0, companies can give their engineers the observability tools they need while preserving their data lake strategy.

    Here are three main challenges that teams come across when using data lakes for time-series, and how to solve the problem with InfluxDB 3.0.

    Problem One: Inefficient and Slow Ingest
    Data sources generate time-series data at an incredibly high rate, easily producing hundreds or thousands of data points every second; some sources even do so as quickly as every nanosecond. Data lakes can store massive volumes of data, but they aren’t optimized to quickly ingest and make it readable on a millisecond time scale.

    Data lake ingest jobs can be measured in minutes or even hours, which is far too late for any meaningful interventions that can prevent catastrophes such as downtime. Customers will notice the outage before the data is even ingested into the data lake, much less acted upon.

    Problem Two: Time-Series Life Cycle
    Full-fidelity data is necessary for real-time analytics and observability, but the value of the full-fidelity data fades over time. Storing the full-fidelity data in a data lake may be convenient, but it is costly and not necessary for typical BI purposes.

    InfluxDB is designed for the time-series data life cycle. This life cycle entails:

    1. The leading edge of data. That is, data written in the last two hours or so. This is the data that is most likely to be queried, so the full fidelity data is kept in memory ready to query instantly.
    2. Compacted data. That is, data older than the leading edge that is still likely to be queried and requires full fidelity is compacted off to disk in object store, which still permits fast querying but does not incur the costs of storing data in memory.
    3. Retention policies. That is, data that has lost its value in its full fidelity form. For example, for observability data, data older than a few weeks can be aggregated into a much smaller data footprint. Then, this aggregated data is ingested into the company’s data lake while the full fidelity data is automatically dropped from InfluxDB 3.0.

    Without such a system, it is necessary to store the full fidelity data in the data lake and incur those massive costs without a reasonable way to manage the life cycle of the data. With such a system, the correct amount of observability data can be ingested into your data lake to support your BI needs.

    Problem Three: Slower Query Performance
    Engineers can experience a high degree of frustration when they are troubleshooting an issue in real-time, but their dashboards won’t load or their queries won’t return in a reasonable amount of time. Data lakes simply are not designed to produce fast results for observability, and so query performance is a primary complaint of engineers trying to use a data lake for observability. Waiting minutes for a query to return when you are running a BI report may be annoying, but it could spell extended downtime for an engineering team.

    Fortunately, InfluxDB is built on the Apache Arrow standards, which means that it can be directly queried by your engineering team. Via protocols such as Delta Sharing and Iceberg, anyone can query into InfluxDB 3.0 from their data lake and join the real-time observability data stored in InfluxDB 3.0 with the data lake data. This is akin to having your cake and eating it too.

    The Bottom Line

    If you are trying to use a data lake strategy to fulfill your engineering team’s observability needs, you are going to frustrate those teams because the data lake is not designed for data ingestion, the data life cycle, or the query needs of those teams. This will cause those teams to create rogue solutions that are inconsistent with your company’s data lake strategy.

    By providing those teams with open source InfluxDB 3.0, engineering teams will have a first-class observability platform that is designed to interoperate sensibly with your company’s data lake. This enables the critical mission of your observability platform while maintaining your overall data lake strategy for your BI needs.

  • How AI Addresses DevOps Monitoring and Observability Challenges

    How AI Addresses DevOps Monitoring and Observability Challenges

    In my recent article Revolutionizing the Nine Pillars of DevOps with AI-Engineered Tools, I explained that the continuous monitoring pillar involves monitoring observable systems and application performance in real-time so that teams can detect and react to problems quickly, often before they impact users. It’s crucial for maintaining application health and informing future improvement efforts.

    In this article, I explain how AI can analyze logs and metrics to predict potential system failures or performance degradation, allowing for proactive maintenance and issue resolution.

    Continuous Monitoring and Observability Use Cases

    CI Anomaly Detection: AI can analyze historical data to detect anomalies during the continuous integration phase. Any unusual change can be flagged for review before it proceeds to the next phase. Tools like IBM Watson Anomaly Detection can help identify these anomalies by using AI to detect patterns and irregularities.

    Code Quality Assurance: AI can be used to analyze the code in the development stage to ensure its quality, which can help reduce bugs and vulnerabilities. Tools like DeepCode and Codota use AI to identify potential problems and suggest improvements based on a learned database of code and solutions.

    Test Case Optimization: AI can help optimize the selection of test cases in continuous integration (CI). Using historical test data, AI can identify which test cases are most likely to find new defects. Tools like Testim.io can help with this by using AI to prioritize testing based on risk and change impact.

    Predictive Analytics in CD: AI can analyze historical deployment data and predict potential issues during continuous delivery (CD). This can help preemptively solve problems, reducing downtime. Tools like Splunk use AI and machine learning to provide predictive analytics for operations data.

    Automated Rollbacks: In the case of continuous deployment, AI can be used to automatically roll back deployments that are causing issues. Tools like Harness use machine learning to understand typical application behavior, automatically reverting to the last stable state if anomalies are detected.

    Infrastructure Optimization: AI can help optimize the usage of resources in cloud environments. Tools like CAST.AI and Turbonomic use AI to continuously optimize your infrastructure, ensuring better performance while reducing cost.

    Incident Management: AI can help automate the incident management process, from detection to resolution. Tools like BigPanda and Moogsoft AIOps use AI to aggregate, correlate and analyze alerts from various sources, reducing noise and speeding up incident resolution.

    Log Analysis: AI can analyze logs and identify patterns that would be difficult for humans to spot. Tools like Logz.io use AI to provide cognitive insights into log data, providing a deeper understanding of the data.

    Security Threat Detection: AI can analyze patterns and detect security threats more effectively. Tools like Darktrace use machine learning to detect unusual behavior in real-time, thereby detecting potential threats before they cause damage.

    Network Monitoring: AI can predict network outages by analyzing traffic patterns. Tools like Kentik use AI to proactively identify potential network issues before they affect users.

    Challenges When Transforming Continuous Monitoring to use AI

    Here are some challenges organizations might face when transitioning existing CI/CD pipelines to incorporate AI into continuous monitoring and observability, along with possible solutions:

    Data Quality and Availability: The effectiveness of AI-engineered tools largely depends on the quality and quantity of the data they’re given. Insufficient or poor-quality data can result in inaccurate insights or predictions. Implement effective data governance and management practices to ensure data quality and accessibility. Data should be thoroughly cleaned and properly labeled to facilitate the training of AI models.

    Skills Gap: Adopting AI-engineered tools requires new skills that the existing IT team may not possess. There might be a lack of understanding of how to use these tools effectively. Provide comprehensive training to your DevOps team to bridge the skills gap. Also, consider hiring AI specialists or working with an experienced vendor to help implement and manage the AI tools.

    Resistance to Change: As with any significant transformation, resistance to change can be a substantial barrier. Employees may be concerned about job security or the perceived difficulty of adapting to new tools. Communicate clearly and transparently about the benefits of the AI transition, both at an organizational and individual level. Ensure staff that AI is there to assist them, not replace them. Organize workshops and training sessions to ease the transition.

    Integration with Existing Systems: AI tools need to integrate seamlessly with existing DevOps tools and workflows to ensure that they add value without disrupting operations. Choose AI tools that are compatible with your existing infrastructure or consider implementing integration middleware. Conduct a proof-of-concept (PoC) to ensure the new AI tools integrate smoothly.

    Cost of Implementation: Deploying AI tools can require a significant upfront investment, both for the tools themselves and for necessary infrastructure upgrades. Conduct a thorough cost-benefit analysis to understand the return on investment (ROI) that the AI tools can deliver. Consider starting with lower-cost or open source tools or use cloud-based AI services to reduce initial investment in infrastructure.

    Summary

    As the world of DevOps evolves, the integration of AI in monitoring and observability becomes increasingly important. Whether it’s during the phases of continuous integration, continuous delivery or continuous deployment, or for applications, infrastructure and pipelines, AI can provide exceptional benefits. From anomaly detection in CI, code quality assurance and test case optimization to predictive analytics in CD, AI can transform your operations, delivering rapid and more reliable results.

    However, transitioning to an AI-optimized DevOps environment is not without its challenges. Issues such as data quality, the skills gap, resistance to change, system integration and cost implications must be considered and addressed. But fear not, solutions are at hand. With effective data governance, comprehensive training, transparent communication, smart tool selection and thorough cost-benefit analysis, you can navigate these challenges and reap the rewards of AI integration in your DevOps journey. So, are you ready to embrace the future of AI-driven DevOps? There’s no time like the present to start exploring the possibilities.

  • Leaders in Resilience See ROI From Observability

    Leaders in Resilience See ROI From Observability

    Observability is maturing throughout the software industry. Observability is often defined as merging metrics, logs and traces to get a fuller picture of an application’s state. This can help pinpoint disruptions or security incidents and offer a clearer path to remediation. Observability is touted to advance the application monitoring practices of yesteryear, but are investments paying off?

    According to recent findings, many organizations are using observability tools and seeing a positive outcome. A full 73% of organizations reported using observability tools for over a year. And impressively, 86% of observability leaders said the ROI on their observability tools has exceeded their expectations. The practice appears to be paying dividends for organizations that have gone “all-in” with mature observability processes.

    The State of Observability 2023 report released by Splunk demonstrated how observability leaders are reaping a positive return from observability tools. According to the report, observability is helping to meet performance expectations, avoid downtime and enhance resilience in general. However, not all organizations are on the observability maturity bandwagon. Below, we’ll examine some of the key trends from the report and consider the benefits of refining your observability efforts.

    Now that observability is more commonplace, dedicated roles are emerging to support these initiatives. In fact, 87% of respondents now employ specialists who work exclusively on observability projects. Site reliability engineers likely make up a portion of these experts, as observability data can be used to advance efforts around meeting service-level objectives (SLOs).

    But although observability might be more common than a few years ago, not every observability practice is mature or standardized. Just 40% have a formal approach to resilience that has been institutionalized organization-wide. Because of this, only 43% are confident in their team’s ability to meet its application reliability and performance objectives. It should be noted that this number increases significantly for those with more mature observability adoption, which we’ll expand on below.

    Furthermore, not all applications are cloud-native. Organizations reported operating and maintaining, on average, 165 internally developed business applications, half of which are hosted in the public cloud and half of which are on-premises. Due to the persistence of legacy modalities, observability tools must continue to support a hybrid architecture if they are to be comprehensive and useful.

    And although we’ve previously covered how many organizations are drowning in a sea of observability data, observability leaders now say their ROI on observability tools far exceeded their expectations. Perhaps this indicates that, with more open standards, analysis centralization and fewer false positives, teams are finding ways to avoid observabiliy tooling sprawl.

    Lastly, the report unveiled desires to loop AI and ML into the observability pipeline. In fact, 66% of observability teams already use AI/ML, while 26% are in the process of deploying the technology. And 91% said AIOps is an important enabler of their observability goals. Certainly, AI is becoming a pervasive instrument to accelerate many areas, including observability.

    Benefits of Observability

    An impressive 64% reported that their investment into observability has exceeded their expectations. So, what are some of the top benefits of observability? Well, the report found the topmost benefits to be areas like decreasing problem detection and resolution times, identifying “true positive” incidents, and enhancing application security.

    Observability is being deployed as a tool to evolve traditional security efforts, as it uncovers a deeper understanding of application behaviors. This can help security teams correlate errors to specific diagnoses and spot potential threats in a quick manner. As such, 59% said observability helps them uncover security issues thanks to the intelligence capabilities of observability solutions. And 55% say they can uncover more security vulnerabilities thanks to the visibility afforded by observability.

    Observability can also be used to improve visibility into hybrid architectures. A full 86% said it’s important to have flexible observability solutions that cover hybrid architectures, though half identify this as an area for improvement. Although we like to think that cloud-native development is the status quo, many enterprises still oversee a mixture of on-premises, private cloud and multi-cloud environments. A unified, agnostic observability layer is thus essential to peek into all these disparate environments to grapple with disparate data outputs.

    The Leaders of the Observability Pack Win

    Service interruptions can have negative consequences for organizations, including lower customer satisfaction, loss of revenue and loss of customers. In fact, 76% of respondents reported downtime can cost up to $500,000 per hour.

    Thankfully, observability has emerged to help improve overall resilience and avoid these negative repercussions. But, to truly reap the benefits, you must have a mature practice in place. As defined by the report, “observability leaders” have a more formal approach to resilience and have standardized this across their organization. And 89% of those leaders are completely confident in their ability to meet their application availability and performance requirements. The catch-22 with observability is that you must invest more energy into formalization and knowing when the effort is worthwhile is a moving target.

    A solid observability practice is supported by standardized metrics, logs and traces across the organization. Furthermore, to make observability behave correctly, you will need flexible tooling that can leverage data from multiple sources, such as the network, infrastructure, servers and databases.

    The State of Observability 2023 report surveyed 1,750 observability practitioners from around the globe, making it the most extensive study on observability to date. For more insights, you can pick up a copy of the report behind an email gate here.

  • The Tragedy of the Data Commons: Balancing Innovation With Risk

    The Tragedy of the Data Commons: Balancing Innovation With Risk

    Clean drinking water and fisheries are classic examples of the “tragedy of the commons,” the problem in economics and ecology where an individual is incentivized to consume as much of a shared resource as possible at the expense of the larger society. With fisheries, for example, a logical fisherman would like to make as many catches as possible. However, this can ruin the environment for everyone, as overfishing can endanger an entire species and even threatens whole ecosystems. The modern IT environment has given us a new case to consider related to the tragedy of the commons: Data.

    Data, much like drinking water, is a critical resource. Developers need to use this data and, logically, they want to use it as much as possible to build innovative applications and products. However, too much data flowing around a modern distributed enterprise environment carries risks to the larger company.

    The data developers use often includes customers’ personal data like names and birth dates. The risks of this information being damaged or stolen can be catastrophic. This is where database administrators (DBAs), who are tasked with managing and securing this data, come in. There is an inherent tension between developers, who want as much access as possible, and DBAs, who want to limit that access for security and privacy reasons. How data is used within an enterprise is critical, because it must be used to create new solutions and be kept secure. An overly open data policy could lead to breaches and leaks, while an overly restrictive policy limits innovation.

    Thankfully, there is a solution that is helping ease this tension. Observability can automatically analyze massive amounts of information across a business’s entire IT environment. Users can use it to pinpoint the causes of outages or performance issues and receive actionable insights to resolve these problems quickly. Modern observability solutions even use AI to predict and proactively prevent problems before they occur. DBAs and developers can use observability to strike a balance while boosting innovation and improving productivity. Here are the three biggest ways that observability can solve the tragedy of the data commons and ease the tension between DBAs and developers.

    Ensuring Data is Secure

    One of the biggest concerns that DBAs have regarding data access is security. Sensitive production data are no longer stored exclusively on a company’s on-premise servers, where DBAs have full visibility into how it is being used and who has access. Instead, it is now stored across various private and public clouds as well as on-premises, which has reduced visibility for DBAs.

    This sensitive data is the same information that developers often need to build and launch applications in a continuous delivery model, where developers build software in short cycles so it can be released at any time. This involves working across different clouds, including AWS and Azure, while leveraging complicated cloud-enabled technologies like microservices and Docker containers. When data is deployed across multiple clouds, the attack surface increases while visibility decreases, leading to new security concerns.

    Observability provides DBAs with a better understanding of how data is being used by developer teams so they can more easily ensure that the data remains secure and protected. Additionally, many modern IT environments deploy threat intelligence solutions that observe actions and detect threat signals to help predict security incidents and respond before they happen. Some threat intelligence applications are powered by AI, which can automatically gather and organize complex log data from the entire network. This data is then compared with an out-of-the-box threat database, helping to detect suspicious activity early. Observability and threat intelligence working in tandem provide a comprehensive overview to help relieve some of the security concerns for DBAs and create a more secure environment for data to be shared between teams.

    Remaining Data Compliant

    Ensuring that data is compliant is top of mind for every organization. Compliance regulations such as GDPR, CCPA and HIPAA carry severe penalties for data leaks, meaning that companies must maintain the integrity and availability of sensitive and regulated data.

    Ensuring compliance while managing requests for information from developers working across geographies and clouds can be a daunting task for DBAs. Not only must they remain compliant, but many businesses must demonstrate continuous compliance, which is only possible using complex, dynamic and automated systems.

    Observability allows teams to clearly see and follow the complex flow of data as it moves between DBAs, developers and other teams. Additionally, observability can be deployed to help ensure that any sensitive data that is required to be anonymized is done so before it is shared. DBAs can further use observability to track the lineage of data as it moves between teams to monitor that it is not changed or transformed in any way that risks a compliance violation. Making compliance a regular part of the IT environment should not be viewed as a burden. In fact, it provides many benefits, including improving data quality, which can be used to further innovate and build better products.

    Making Teams More Productive

    Observability has shown it can create harmony between DBAs and developers by helping mitigate security and compliance concerns. Not only is it addressing these issues, but it is being used to make each team more productive on their own.

    For DBAs, this means gaining additional insights that help ensure applications and services are running properly. Observability collects information from across hybrid and multi-cloud environments and presents it in a single view to allow DBAs to respond quickly and resolve issues. When DBAs gain the insights that observability provides, they can better manage critical data and anticipate future problems before they arise.

    Observability is key to helping developers build applications as productively as possible. The ongoing shift left has necessitated the use of observability as a way to gain single-pane-of-glass visibility into the build process. Developers are also provided insights into the performance of web applications and can move quickly to address any problems. Finally, observability is improving the speed at which teams can innovate, as it provides live code profiling that can be used to predict and prevent future problems.

    Observability provides a real solution to ensure that data is not a resource that falls victim to the tragedy of commons, where its use can harm an entire organization. Instead, it is powering teams to use data securely to work more productively and build innovative applications.

  • Chronosphere Adds Professional Services to Jumpstart Observability

    Chronosphere Adds Professional Services to Jumpstart Observability

    Chronosphere has added a professional services capability through which it will provide the resources and expertise needed to deploy and manage its observability platform.

    Ian Smith, field CTO for Chronosphere, said the Chronosphere Quick Start program will make it easier for organizations to transition away from legacy monitoring tools that only track a set of pre-defined metrics and embrace an observability platform that enables DevOps teams to surface issues before an IT environment is disrupted.

    The scope of the services provided spans everything from data ingestion to providing training so that management of the Chronosphere observability platform can eventually be handed over to an internal DevOps team.

    The professional services team will also guide DevOps teams through setting up data control mechanisms and retention polices in addition to crafting dashboards to monitor specific processes.

    Finally, there’s also a self-paced online learning platform, dubbed Chronosphere University, that provides a comprehensive curriculum to help DevOps teams navigate fundamentals, advanced features and best practices.

    It’s still early days as far as the adoption of observability platforms is concerned. But it’s apparent that as application environments become more complex, the plethora of monitoring tools that DevOps teams rely on will need to be streamlined. Observability platforms promise to unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue.

    The rate at which DevOps teams will embrace observability will naturally vary, but the biggest technical obstacles are deploying and configuring the platform, loading data and managing the cost of storing all the data that is collected.

    In addition, DevOps teams also need to understand what queries to launch to investigate specific issues. Many DevOps teams lack the expertise required to frame those queries today, but it’s expected that machine learning algorithms and other forms of artificial intelligence (AI) will automatically identify issues that might lead to disruption.

    In the meantime, DevOps teams should be evaluating where it makes sense to continue to rely on monitoring tools. In many instances, an observability platform might obviate the need for that tool, but there may also be plenty of instances where a simple monitoring tool is sufficient for the task at hand.

    One way or another, however, more advanced observability capabilities will be needed as application environments continue to become more complex in the cloud-native era. Application environments now consist of a mix of monolithic and microservices-based applications that gain more dependencies with each passing day. An application may not be as prone to crash as it once was, but determining the root cause of a performance issue can still take weeks given all the dependencies that exist between various services.

    Regardless of approach, observability has always been a core tenet of DevOps. As organizations’ DevOps maturity increases, the greater the appreciation for it becomes. The challenge has been that observability is one of many DevOps things that are easier said than done.

  • Is Your Monitoring Strategy Scalable?

    Is Your Monitoring Strategy Scalable?

    Every company is trying to get better insights into its operational effectiveness, and they are running into the same problem: Scale. So what does a scalable monitoring strategy look like and how can you safeguard against the most significant issue in observability?

    What is a Scalable Monitoring Strategy?

    We’ll begin by identifying the two things most impacted by scale: Cost and performance. Cost can be broken down into storage and computation. It is obvious that to hold more data, more storage is needed, but what about compute power? This is important because queries need to search through more data—and thus, use more compute power—to return a result.

    This creates a trade-off between performance and cost-effectiveness. It becomes more and more challenging to have queries that resolve instantly as the dataset increases, so engineers will often simply tolerate a performance decline. This further impacts usability and the general usefulness of the system. If it takes 10 seconds to resolve every query, you’ve just cut down your data discovery efficiency by a factor of 10.

    Scaling Means Performance and Cost-Effectiveness

    To scale a monitoring strategy, a wise architect needs to break this standoff between performance and cost and approach the problem like a data scientist. A scalable monitoring strategy starts with a simple question: What is the use case for this data?

    Some data is only ever ingested because it might be needed. It spends its entire lifetime undisturbed and is eventually compressed or deleted. On the other hand, some data is queried every minute of every day and is integral. Knowing how data is used means an engineer can decide the value of that data.

    In the battle to create a scalable monitoring strategy, there’s a three-step approach.

    Step One: Track Data Usage

    Which data is ingested and never queried? Which data never spends any time at rest? Build a map of how often data is consumed so that any strategic choices are informed by the use case.

    It is most common to group data into three different use cases:

    Frequent Access – Data that is constantly queried and needs to be available at a moment’s notice.

    Monitoring – Data that drives dashboards or trains machine learning models but which is largely useless after it has been processed.

    Compliance – Data that is held in case it is needed, but it is only queried sometimes. For example, audit logs.

    Step Two: Optimize Storage and Consumption

    Once the use case of data—whether it’s logs, metrics or traces—is understood, the next stage is to optimize. This means creating different storage and query solutions for the different usages we’ve seen above.

    Frequent Access – Rapid queries that can be optimized and tuned. OpenSearch is a good option, although management overhead can be painful, especially at scale.

    Monitoring – This is largely about transforming data. For example, ingesting logs, converting them into metrics and deleting the original log. This is very powerful because metrics take up far less space than logs and can be stored much more cost-effectively.

    Compliance – Low-cost storage like Amazon S3 is a good option, but this data must still be accessible, even if simply by reindexing.

    How Easy is it to Build These Capabilities?

    It is trivial to create an OpenSearch cluster, but it’s challenging to manage it at scale. Likewise, it’s simple to convert logs to metrics, but doing so in a performant way at scale is complex. Holding onto compliance logs as they scale near infinitely can be difficult, and reindexing this data is a non-trivial operation when dealing with large data volumes.

    However, the key takeaway from this analysis should be that these are capabilities that any organization should have if they intend to scale their monitoring solutions. If they can be attained through a SaaS vendor, then this is a serious option to consider because it enables companies to immediately take advantage of this capability without the upfront, unpredictable and often ongoing cost of in-house engineering.

  • How to Choose an Observability Data Pipeline

    How to Choose an Observability Data Pipeline

    A complete full-stack observability platform combines data from multiple sources to gain holistic insights into system behavior. In fact, organizations without a full observability solution could find themselves facing departmental silos since teams with different data sources cannot share insights.

    While a key step in observability is the ability to combine logging data with metrics, companies that have developers viewing logging data and their metrics elsewhere are not collaborating on their data in the way observability requires. The solution is an observability data pipeline that best fits your business needs.

    What is an Observability Data Pipeline?

    An observability data pipeline does two things—centralizes data from a number of different sources and provides teams with the right tools to turn data into business insights. Observability data pipelines are typically user-friendly so that you can easily set up and manage your data.

    Observability data pipelines integrate data from a range of sources, including AWS, FluentD and Kubernetes. They have capabilities for searching and viewing log data and notifying engineers through email and even Slack.

    An observability pipeline ingests logs, so they can be viewed in a log viewer. Pipelines can also analyze data in real-time, allowing engineers to quickly spot critical security issues before they affect critical systems.

    There are a number of full-stack observability platforms, including Coralogix, Splunk, Elastic and even options like Azure Log Analytics and Google Cloud Logging. Coralogix, for example, is a popular observability data pipeline that can analyze data in real-time using Streama and use Kafka Connect to integrate data from a variety of sources and extract trends.

    Business Benefits of Observability Data Pipelines

    Using observability data pipelines has the potential to transform the way a business does DevOps. Let’s go through a few key benefits of this technology:

    ● Reducing Cost of Observability Data
    Many organizations use systems like Cloudwatch to analyze logging data. However, these often charge per GB of logging data analyzed, raising the question of whether Cloudwatch is really cost-efficient. And it’s not just Cloudwatch. For many systems, logging costs can be increased due to slow query times, noisy logs and the wrong type of storage. Observability data pipelines help organizations manage high volumes of logging data and move away from expensive legacy platforms. This enables enterprises to increase their observability for a modest price.
    ● Reduced Onboarding, Increased Problem-Solving
    Pipelines are straightforward to learn. Unlike options such as ELK, which can be quite specialized and difficult to maintain, observability data pipelines are easy to use and easy to set up. This reduces the time needed to onboard new engineers. Less time onboarding means more time solving problems and serving clients, which increases business revenue.
    ● Better Data Shaping and Merging
    Data from different sources is often stored in multiple formats and structures. Because observability depends so much on well-integrated data, it’s vital to transform this data into standardized formats. Observability data pipelines use log parsing to store key features of logging data for search and analysis.
    ● Reducing MTTD/MTTR
    Having an optimized, user-friendly observability data pipeline reduces downtime by helping engineers solve problems faster.
    ● Enabling DevSecOps With Observability
    We’ve written about the rising prominence of DevSecOps, a paradigm that involves shifting security left in the development life cycle. Key to DevSecOps is the fruitful collaboration between DevOps and Security. This can only be achieved by high levels of observability. Observability data pipelines facilitate this with their ability to integrate data from multiple different sources and present it intuitively. This means security teams can apply their expertise to log analytics that would otherwise have remained siloed.

    Wrapping Up

    Observability, the ability to use logs, traces and metrics to “picture” the internal state of a system, is vital to any DevOps team. Observability data pipelines can help by integrating and presenting data from different sources. This mitigates the effects of organizational silos and allows DevOps teams to solve problems faster.

  • ServiceNow Adds Observability Platform to SaaS Portfolio

    ServiceNow Adds Observability Platform to SaaS Portfolio

    At its Knowledge 2023 conference, ServiceNow launched an observability service that runs on its software-as-a-service (SaaS) platform.

    Based on the observability platform the company gained with its acquisition of Lightstep, the ServiceNow Cloud Observability platform makes it possible to collect and analyze data collected from software running on infrastructure and also from workflows created using ServiceNow applications.

    Pablo Stern, senior vice president and general manager for technology workflows products at ServiceNow, said the goal is to make it simpler to investigate IT issues involving applications and workflows that have multiple dependencies. That makes it possible to quickly diagnose an issue, determine the root cause and then dispatch the right site reliability engineering (SRE) team to troubleshoot it, he added.

    ServiceNow Cloud Observability uses both open source OpenTelemety agent software to collect logs, metrics and traces in addition to a log management tool that ServiceNow gained last year with its acquisition of Era Software.

    In addition, ServiceNow is employing a Service Graph Connector that makes it simpler to pull data from, for example, OpenTelemetry and Kubernetes clusters into the ServiceNow IT Operations Management (ITOM) without requiring any additional tooling. ServiceNow is also making available its Unified Query Language to make it possible for IT teams to query the data residing in ServiceNow Cloud Observability.

    As application environments become more complex, the need for entire IT teams to have access to an observability platform is growing. While observability has always been a core tenet of DevOps best practices, Stern said it’s clear that these capabilities also need to be made available to IT administrators. In fact, as observability becomes more democratized, it will drive the rise of services operations that will enable IT teams to be better aligned around specific processes, he added.

    Most IT organizations today employ a wide range of tools that enable them to monitor a set of pre-defined metrics. Observability platforms, in contrast, enable IT teams to collect data from systems in a way that can be investigated using a query tool.

    It’s still early days as far as the adoption of observability platforms is concerned, but as the collection of logs, metrics and traces becomes more unified, an opportunity to better converge IT practices based on DevOps principles and the ITIL framework that many IT administrators rely on to manage IT will emerge.

    The challenge, regardless of what methodology is used, is that IT teams need to be able to frame the queries they use to discover the root cause of an IT issue before any potential disruption occurs. In the long term, however, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify those issues.

    Regardless of how an IT issue arises, the metric that matters most is how quickly it can be resolved—no matter who in the IT team is tasked with resolving it. The issue is finding a way to provide IT teams with better tools to achieve that goal.

  • Four Trends Defining the Observability Market in 2023

    Four Trends Defining the Observability Market in 2023

    Observability is a market that has grown exponentially over the past few years. It started as a buzzword in late 2015 and early 2016, and since then we’ve seen dozens of insurgent companies rush in, eager to stake their claim in a new (albeit loosely defined) market. Incumbent companies also rebranded their wares, hoping to capture some of the momentum that comes from the new term.

    Venture capital saw the market opportunity, too. By my conservative estimate, observability companies raised nearly $2 billion over roughly the last two years alone (I’m including some cybersecurity companies, which I’ll explain later).

    In the seven or so years since its inception, the observability market has fragmented and specialized into categories such as application, data, network and edge observability. On the artificial intelligence and machine learning side, there is model observability and AIOps. Connecting it all are observability pipelines.

    While more businesses than ever know about observability and have built it into their tech strategy, category awareness still has a long way to go. And though the dollars were flying fast and furious a couple of years ago, the economic cycle has turned against risk, and growth funding rounds are harder to come by. It’s time to step back and examine the key trends defining today’s observability market.

    Staffing Shortages: A Blessing and Curse to Observability

    Many companies in the observability space, regardless of their subcategory, represent net-new work for IT and security teams. A new application performance monitoring (APM) platform may offer a marginal improvement around APM’s golden signals, but that marginal improvement is a substantial lift for the end users tasked with implementing and deploying the new platform. While software may get sold, overworked ITOps and SecOps teams may decide the effort isn’t worth the perceived ROI, leaving the software on the shelf.

    On the other hand, observability products that reduce toil for operators will continue to outpace market growth rates. IT departments face acute shortages in skilled staff, leaving companies with little choice but to seek out software that automates high-effort, low-value tasks or otherwise offers a force multiplier to existing teams. Subcategories like AIOps and observability pipelines can address those needs.

    Consolidation is Needed to Simplify the Market

    Another looming hurdle for the nascent observability market is feature similarity between companies in a subcategory. Two subcategories where this is apparent are application and data observability. In the application observability space, several companies were founded in the wake of the availability of extended Berkeley Packet Filter (eBPF) technology. eBPF allows you to extend the capabilities of the Linux kernel at runtime, and a common use case is to trace applications running in user space. While this is a great technology with several uses, there are simply too many companies with the same idea, the same go-to-market strategy and same limited feature set.

    Data observability is another subcategory riddled with clones. Data observability started as a reaction to operational brittleness in the modern data stack, and there are only a few companies with marked differentiation here. I agree with Matt Turck’s assessment that data observability products will drift into the well-defined (but arguably less sexy) data quality category.

    CFOs are Looking to Reduce Tool Sprawl and Cut Costs

    The challenges with raising growth rounds have been well documented. VC funding represents the supply side of the finance puzzle and IT budgets are the demand side. The demand side is getting squeezed as CFOs clamp down on budget growth. Deals are taking longer to close and often involve more parties eager to reject a budget request from another team because it means more money for their team.

    A common refrain from CFOs today is that any new spending must be more than offset by corresponding savings somewhere else. This puts the products representing net-new work with little marginal utility at risk. Observability vendors today need a strong consolidation (e.g. can reduce tool sprawl, serve multiple teams, etc.) and cost savings story to survive in a difficult budget climate.

    Cybersecurity remains a budget bright spot. According to a recent survey from IDC, 27% of respondents stated cybersecurity budgets were immune from budget cuts. With changing SEC rules highlighting the need for board-level cybersecurity expertise and reporting of breaches, coupled with the new National Cybersecurity Strategy, the priority around preserving these budgets is clear.

    Cybersecurity is Pure Greenfield for Observability

    A few short years ago, SecOps teams, threat hunters and incident response teams didn’t have enough data to create an accurate picture of their environment. Endpoint data was sparse and incomplete, cloud-based application deployments were still uncommon, the network remained between the office and data center, and so on.

    Today’s SecOps teams, threat hunters and incident response teams have too much data to understand their environment. There are too many alerts and too little context, making finding the signal through the noise a difficult, if not impossible, proposition. From a compliance perspective, much of this data must be retained for longer periods, driving up storage costs.

    This is where the principles of observability benefit cybersecurity teams. Much like site reliability engineers need to trace an error well after it occurred, cybersecurity professionals need to trace a breach or find threats over huge volumes of data. They need robust exploration tools, rational data storage and routing options, and, most of all, the ability to do this while maintaining budgets. This is where observability and cybersecurity converge.

    Seven years young, observability remains a nascent market despite its rapid growth. Though the market will face growing pains, the opportunities and upside are bright.

  • New Relic Taps Generative AI to Simplify Observability

    New Relic Taps Generative AI to Simplify Observability

    New Relic has added a tool that takes advantage of generative artificial intelligence (AI) to make it simpler to query the telemetry data collected by its observability platform via a natural language interface (NLI).

    Jemiah Sius, director of developer relations for New Relic, said New Relic Grok will make the New Relic One platform more accessible to a wider range of IT professionals, especially those that haven’t mastered New Relic’s proprietary programming language that enables DevOps teams to identify the root cause of an issue.

    New Relic Grok automatically pinpoints code-level errors in integrated development environments (IDEs) in addition to analyzing code, stack traces and production telemetry to suggest fixes. New Relic is providing this capability via integration with the large language models (LLMs) created by OpenAI, which is providing the foundation for a broad range of generative AI capabilities through Microsoft. New Relic has a long-standing alliance with Microsoft, added Sius.

    Using plain language, anyone can generate a system or app health report complete with anomalies, issues and recent deployments in a way anyone can understand without having to create and filter dashboards, he noted.

    New Relic Grok also makes it simpler to identify instrumentation gaps that can then be addressed using the instructions it provides. It can also set up missing alerts and automate alerts using the open source Terraform infrastructure-as-code (IaC) tool.

    Finally, IT teams can also use New Relic Grok to manage accounts, users and user access, data retention rules, usage, billing and other administrative tasks.

    New Relic previously invested in machine learning algorithms to automate observability tasks. These AI extensions to the New Relic platform are part of a larger effort to extend the reach of the New Relic One observability platform further left toward developers and IT administrators rather than just site reliability engineers (SREs), said Sius.

    However, New Relic doesn’t anticipate that generative AI will eliminate the need for SREs to use its programming tool to solve more complex problems, he added.

    The overall goal is to provide development teams with frictionless access to observability data at every stage of the software development life cycle to reduce mean-time-to-detection (MTTD) and mean-time-to-resolution (MTTR) of issues.

    In general, most organizations are still in the early stages of achieving full-stack observability. A recent New Relic survey found only 27% of respondents have achieved full-stack observability and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints, the survey found. On the plus side, the survey also found nearly three-quarters of respondents said C-suite executives in their organization are advocates of observability, and more than three-quarters of respondents (78%) saw observability as a key enabler for achieving core business goals. However, more than half (52%) of respondents said they experienced high-business-impact outages once per week or more and 29% said they take more than an hour to resolve those outages.

    AI, of course, promises to improve reliability. At the very least, it could make it easier to identify issues long before a major disruption occurs.