devopsy.kinsta.cloud

Tag: LogDNA

  • Mezmo Adds Observability Pipeline to Analyze DevOps Data

    Mezmo Adds Observability Pipeline to Analyze DevOps Data

    At the KubeCon + CloudNativeCon North America conference this week, Mezmo launched an Observability Pipeline platform that promises to make it simpler to manage, enrich and correlate machine data.

    Previously known as LogDNA, Mezmo is expanding the scope of its reach to add the ability to augment and analyze data on top of its core log management platform. Over time, that capability will be extended to add support for metrics and traces using the open source OpenTelemetry agent software.

    Mezmo CEO Tucker Callaway said the goal is to provide a framework for centralizing the flow of observability data that is now being generated by a wide range of sources across a DevOps workflow. It’s clear that DevOps teams require a means to enrich and correlate high volumes of data in motion that goes beyond simply pre-processing so it can be stored more cost-effectively, he added.

    The Mezmo Observability Pipeline achieves that goal by ensuring the right data is flowing into the right systems in the right format for analysis, noted Callaway.

    As the volume of data being collected by DevOps teams continues to dramatically increase the need to embrace a set of DataOps best practices is becoming more apparent. DataOps has emerged recently to describe a set of data engineering practices that applies many DevOps principles to the way data should be managed. Mezmo’s Observability Pipeline aims to make it simpler to apply DataOps best practices specifically to the data collected by DevOps teams.

    Of course, each DevOps team will need to decide for themselves how much data they want or need to store long term. However, increasingly DevOps teams are holding onto data much longer as IT environments become more complex, noted Callaway. That creates a need to more efficiently analyze as much data as possible and store it more cost-effectively, he said. Those twin goals, however, often create a physics problem in the sense that they can be diametrically opposed.

    In the longer term, it’s not quite clear how much DevOps teams are embracing observability. Most DevOps teams make use of monitoring tools that enable them to track a set of pre-defined metrics. Observability platforms promise to make it easier for DevOps teams to query a wide range of data types to discover potential issues before they disrupt an IT environment. The challenge is that many DevOps teams may not know what queries to form to discover those issues. It’s likely that, over time, machine learning algorithms will be increasingly applied to observability data to make it simpler to surface potential issues.

    In the meantime, however, the need for DevOps teams to find ways to manage all the data being generated by multiple types of applications is becoming a higher priority. In fact, in the absence of any meaningful way to manage all that data, the benefits of an investment in an observability platform is likely to be limited, at best.

  • LogDNA Survey Surfaces Observability ROI Concerns

    LogDNA Survey Surfaces Observability ROI Concerns

    A Harris Poll of 200 senior engineering professionals suggests it may be a while before most DevOps teams are able to achieve true observability in their IT environments.

    The survey was conducted on behalf of LogDNA, a provider of an observability platform, and found nearly three-quarters of respondents (74%) are struggling to achieve their observability goals despite investing hundreds of thousands of dollars. In fact, well over a third (38%) said they are already investing $300,000 annually on observability tools.

    Despite those investments, less than half of respondents are very satisfied with their ability to use log data. Key areas of frustration include the challenges of collaboration with colleagues spanning multiple teams (67%), the fact that the tools are not easy to use (66%) and the routing of security events (58%).

    Nevertheless, the bulk of survey respondents said they are optimistic about observability. A full 85% of participants said they believe true observability is possible.

    LogDNA CEO Tucker Callaway said the core issue that many organizations fail to recognize is that observability is really a data literacy and management challenge. Organizations are collecting logs, metrics and traces from a wide range of tools and platforms without thinking through optimal data pipeline construction. The result is an aggregation of data that has not been normalized to simplify surfacing actionable intelligence, he said. In fact, many organizations quickly discover that all they’ve really accomplished is increase their data storage costs, said Callaway.

    In the absence of that data normalization, Callaway noted it becomes exceedingly difficult for cross-functional teams to query all the data being collected within a repository.

    At its core, observability promised to make it easier to identify IT issues before they cause a disruption. Observability in one form or another has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Observability platforms make it possible to correlate events in a way that makes it easier for analytics tools to identify anomalous behavior that might be indicative of the root cause of an IT issue.

    In contrast, traditional IT monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability platforms combine metrics, logs and traces—a specialized form of logging—to make it easier for IT professionals to interrogate data being generated by a wide range of DevOps tools and platforms.

    Of course, the assumption is that DevOps teams actually know what queries to launch to determine what the root cause of an IT issue might be. In the longer term, machine learning algorithms should make it easier to identify anomalies within IT environments. In the meantime, however, achieving observability still requires a fair amount of DevOps expertise that is not always readily available.

    Eventually, observability will become much more automated than it generally is today. As that transition occurs, IT teams should find the ROI of observability is steadily improving.

  • LogDNA Previews Streaming Module for Observability Platform

    LogDNA Previews Streaming Module for Observability Platform

    LogDNA is making available on a limited basis the ability to process streaming data to its observability platform as part of an effort to give DevOps teams more control over log data.

    Tucker Callaway, LogDNA CEO, said the data ingestion pipeline that LogDNA already provides can ingest, parse and normalize massive amounts of structured and unstructured log data. LogDNA Streaming adds the ability to automatically send log data to any application or analysis tool.

    That capability also helps reduce storage costs by limiting the number of copies of log data that might otherwise be created, he added.

    LogDNA also expects that DevOps teams will also be able to analyze data in flight as part of an effort to make it simpler to surface issues even before log data is stored, noted Callaway.

    While there are multiple streaming data platforms available, Callaway said LogDNA opted to build one on its own to meet the requirements of an observability platform. Ultimately, the goal is to shift observability left at the data level to provide more granular insights in real-time, he said.

    That capability will also lay the foundation for a wide range of autonomous processes that would be driven by the observability data collected via LogDNA, added Callaway.

    LogDNA

    Observability has always been a core tenet of any DevOps best practice. Most DevOps teams typically focus on some form of continuous monitoring to more proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Monitoring tools are designed to consume predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization.

    Observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without having to rely solely on a limited set of metrics that have been pre-defined to monitor a specific process or function. DevOps teams can more easily employ queries to interrogate data in a way that makes it easier to discover the root cause of an issue. An observability platform correlates events in a way that makes it easier for analytics tools to identify anomalous behavior indicative of an IT issue.

    It’s now only a matter of time before providers of the platform widely employ machine learning algorithms to analyze machine data faster. There may even come a day when so-called “war room” meetings—that today need to be convened to identify the cause of an IT issue via a painstaking process of elimination—are no longer required.

    In the meantime, the percentage of applications that are instrumented in an IT environment needs to increase to achieve that goal. Fortunately, thanks to the rise of open source agent software, the cost of instrumenting applications is starting to decline. At the same time, DevOps teams should expect to see a marked increase in the volumes of incoming data, and they will need to find somewhere to store that data as well as determine the length of time it should be retained.

  • LogDNA Adds Usage Quota Tools to Help Control Costs

    LogDNA Adds Usage Quota Tools to Help Control Costs

    LogDNA announced today it will enable IT teams to apply usage quotas to help rein in log management costs.

    Tucker Callaway, CEO, LogDNA, said as applications become more complex – thanks, in part, to the rise of microservices – the amount of log data that a DevOps team can collect has significantly increased. IT teams can now assign DevOps teams both hard and soft quotas to limit the amount of log data collected during a specific time period, Callaway said.

    IT teams can set a daily or monthly hard limit on the volume of logs stored or, alternatively, they can also set soft daily or monthly quotas that can apply throttling logic to ensure mission-critical log data will continue flowing through LogDNA as usage volume approaches the limit threshold. It will then be up to each IT team to determine, based on the tasks at hand, whether to enable a DevOps team to exceed those thresholds, Callaway said.

    IT teams can also make use of soft limits to reallocate storage resources from one DevOps teams to another based on how much log data might be generated in a specific time period, Callaway noted.

    LogDNA will notify team members any time a quota is hit, or when an exclusion rule is triggered because a quota has been reached. Those alerts can also be integrated with email systems or the Slack messaging platform. Logs that are not retained are not counted toward the monthly bill, but are displayed and will trigger other relevant alerts.

    In the wake of the economic downturn brought on by the COVID-19 pandemic, IT organizations are, generally, more sensitive to costs. At the same time, many of them are encouraging developers to assume more responsibility for application life cycle management as they embrace DevOps best practices. The usage quotas are intended to enable IT teams to strike a better balance between what can be two competing agendas, Callaway said. IT leaders are not necessarily trying to discourage DevOps teams from collecting log data as much as they are trying to make sure costs are predictable, Callaway added.

    Log data, of course, is one of the core elements of any observability strategy. While observability has always been a core DevOps principle, the number of organizations that have been able to meaningfully achieve it has been limited. However, as organizations start to deploy microservices with multiple dependencies, the need to provide context when troubleshooting those applications has become more acute.

    It’s not clear to what degree more granular control over log data might sway an IT organization toward one platform versus another. However, when it comes time to justify a choice to senior IT and business leaders, it’s always a good idea to address, at some point, how one decision versus another might have a more positive impact on the bottom line.

  • Instana Announces APM Integration with LogDNA Log Analysis Solution

    One-Click Navigation from Automated APM Solution Into LogDNA Analysis Tool

    Chicago, IL – March 17, 2020 – Instana, the leading provider of real-time application performance management (APM) solutions for cloud-native microservice applications, today announced an extension of the company’s one-click integration with Log Analysis tools to now support LogDNA.

    The Instana / LogDNA integration provides a seamless experience for Dev and Ops teams when troubleshooting microservices and cloud native applications. When application problems occur, Instana automatically isolates the service in trouble, as well as the triggering event and problematic application code, microservice or infrastructure platform causing the problem.

    “While applications continue to grow in complexity, individual infrastructure platforms and microservices are also their own complex ecosystems,” said Chris Farrell, Technical Director and APM
    Strategist at Instana. “The integrated solution of Instana and LogDNA provides the full range of monitoring, tracing and log analysis that enables any application stakeholder to identify, isolate and fix application problems before service levels are impacted.”

    Instana’s integration with LogDNA is the latest one-click navigation from the automated APM solution into a log analysis tool, handled via buttons in the Instana GUI on key component dashboards (such as Servers, Containers and Kubernetes). When the button is clicked, Instana starts up an instance of the log analysis tool, along with a pre-executed analysis filter into the exact logs of the application components and associated infrastructure needing examination. Timing is synchronized, meaning users won’t have to search for the specific log or timeframe needed to solve problems.

    Unlike traditional APM tools, Instana’s automated Application Performance Monitoring (APM) solution discovers all application service components and application infrastructure, including Cloud infrastructure such as Azure, orchestration infrastructure like Kubernetes and Docker, application services and DevOps processes. Instana automatically deploys monitoring sensors for each part of the application technology stack and traces all application requests – without requiring any human configuration or even application restarts. The solution detects application and infrastructure changes in real-time, adjusting its own models and visualizing the changes and impacts to performance in seconds.

    The integration with LogDNA and other log analysis tools is available today in all Instana APM solutions. Additional information can be found at https://www.instana.com.

  • LogDNA Announces New Log Management Capabilities on IBM Cloud

    LogDNA enables developers to quickly locate faults and debug code within terabytes of data on the IBM Cloud to reduce downtime

    Mountain View, CA: LogDNA today announced a new offering that enables developers to troubleshoot coding bugs and quickly fix development errors in applications, potentially saving businesses millions of dollars in development hours, downtime and associated revenue loss. IBM Log Analysis with LogDNA integrates LogDNA services to allow developers to aggregate all their log data across applications in the IBM Cloud. LogDNA features such as Live Tail, multi-channel alerting, and natural language queries enable developers to more easily query, filter and gain insight from their log data, helping them quickly identify and resolve issues to reduce downtime and quickly fix performance issues.

    “As businesses’ application and cloud environments grow, they shouldn’t have to sacrifice service availability while the IT team processes vast quantities of log data,” said LogDNA co-founder and CTO Lee Liu. “IBM Cloud customers will now be able to find, debug and fix production issues in their applications faster than ever before so there’s no lag in service for the end users.”

    IBM Cloud developers will be able set up and configure their log sources and retention plans directly within their IBM Cloud console. LogDNA’s fast, responsive, and intuitive interface will provide users with the ability to view and search through their logs to effectively find and isolate issues within the applications they build and run on the IBM Cloud.

    “The availability of LogDNA on IBM Cloud offers leading cloud-native log analysis capabilities,” said Jason McGee, Vice President and CTO, IBM Cloud Platform. “Our clients can now more quickly gain insights from their increasingly complex cloud applications.”

    For more on the LogDNA and IBM Cloud integration, read the IBM blog post: https://ibm.co/2ybcxZw.

    About LogDNA:
    LogDNA provides detailed insights into the production environment. With LogDNA, development teams are free to focus on in-the-cloud or hybrid solutions. LogDNA enables engineering and DevOps professionals to easily and quickly gather all system and application logs into one intuitive platform and get back to focusing on what is most important: building great products. For more information visit LogDNA.com

    LogDNA is an IBM Business Partner in PartnerWorld, IBM’s premier partner program that empowers ecosystem members with the tools and resources that help transform clients into industry leaders.