devopsy.kinsta.cloud

Tag: monitoring

  • Building an Open Source Observability Platform

    Building an Open Source Observability Platform

    Enterprises are under constant pressure to ensure their IT infrastructure and applications are available throughout the year. The complexity of modern architectures (containers, hybrid cloud, SOA, microservices, etc.) is constantly growing, generating vast volumes of unmanageable logs. We need intelligent application performance management (APM) and observability tools to achieve production excellence and meet the availability and uptime goals. These include analyzing application health, performance and user experience. Adopting machine learning techniques to identify anomalies and behavior patterns will help detect the root cause early and meet customer service level agreements (SLA).

    The APM and observability tools market is undoubtedly hot. These tools ingest multiple telemetry data feeds and are powerful analytics platforms providing critical insights into application and infrastructure health, including system performance. The software development teams that adopt observability are much better equipped to release their application code iteratively. As per “MarketsandMarkets” research, the market size of observability tools and platforms is expected to grow from $2.4 billion in 2023 to $4+ billion by 2028 at a compound annual growth rate (CAGR) of 11.7%.

    What is Observability?

    Observability is the ability to collect data about the distributed applications, infrastructure and communication among its internal and external components and services, allowing teams to debug their systems diligently. It enables the site reliability engineering (SRE), software engineering and operations teams to analyze the customer impact and triage a service outage. Observability and monitoring are sometimes used interchangeably. Observability (proactive) makes the data accessible and allows you to ask any system question to know how the code behaves more deeply. Monitoring (reactive) is the task of collecting and displaying that data and the ability to determine the system’s overall state.
    Observability can be further broken down into three key pillars: Logs, traces and metrics, which are essential for SRE observability.

    • Logs help us diagnose the issues and tell us why they happened.
    • Traces help us isolate issues and tell us where they happened.
    • Metrics help us detect the issues and tell us what happened.

    Market Tools, Capabilities and Challenges

    Gartner’s magic quadrant for APM and observability has identified 20+ vendor products offering APM and observability capabilities, including self-hosted, vendor-managed or SaaS deployments. These products provide multiple features, including application performance metrics, monitoring and alerting on events, traceability, anomaly detection and vulnerabilities, etc.

    An enterprise business application comprises homegrown applications (such as .NET, Java, Python, SQL, NoSQL DB, etc.), third-party/off-the-shelf products (such as Salesforce, HubSpot, etc.), and integrations (such as Stripe, PayPal, etc.). Homegrown applications are hosted in an on-premises data center or by cloud vendors like AWS, GCP, or Azure. The off-the-shelf products are SaaS-based or integrated via APIs. There are highly distributed applications spanning tens and hundreds of nodes, services and instances.

    • Too Many Tools: Enterprise applications use various tools for monitoring application health and performance (such as New Relic, Data Dog, etc.), Error logging (such as Splunk), and cloud vendor-provided tools (such as CloudWatch). These products overlap in functionality and maintaining and managing these tools (procurement, learning curve, etc.) can be cumbersome.

    • Unpredictable Data Volume: Imagine the volume of observability data (logs, traces, metrics) collected based on the application traffic, usage, dependency on external products, etc. The amount of data storage required to consolidate these data feeds can quickly grow out of control.

    • Pricing is Complex: These vendor products also offer different pricing models such as charge per host (such as Splunk, Data Dog, Dynatrace), per-user (such as New Relic), per-ingestion (such as SumoLogic, AppDynamics). The complexity of pricing models makes it challenging to compare total cost of ownership (TCO) among vendors and determine the right tool that fits your requirements and budget.

    Why Choose Open Source Observability Platform?

    Open source-based observability tools aim to provide a standard, vendor-agnostic approach for ingesting, transforming and sending data to an observability backend. The open source observability tools can be an alternative to save on licensing costs and consolidate multiple APM tools with the tool that fits your requirements and budget.

    However, maintaining open-source systems may require efforts to set up and maintain and will add to your initial operational cost. But in the long term, you will save on licensing fees and avoid vendor lock-in and contractual agreements.

    Gartner predicts that, by 2025, 70% of new cloud-native application monitoring will use open-source instrumentation rather than vendor-specific agents for improved interoperability, and 70% of new cloud-native applications will adopt OpenTelemetry for observability rather than vendor-specific agents and software development kits (SDKs).

    Scale Observability Using the Open Source Ecosystem

    The open source landscape for observability is quite dynamic. There are multiple Cloud Native Computing Foundation (CNCF) open source tools for observability and monitoring. This post will primarily focus on the OpenTelemetry framework and LGTM technology stack.

    OpenTelemetry:
    The “too many tools” challenge described above brings up a new struggle in telemetry data collection. Each tool vendor has its own APIs, SDKs, agents and collectors for logs, metrics and traces. We need a unified telemetry collection using the OpenTelemetry framework to create and manage telemetry data such as logs, traces and metrics.

    The OTEL project under the auspices of the CNCF offers a unified set of vendor-agnostic APIs, SDKs, and tools for generating and collecting telemetry data and exporting it to various analysis tools. You get one API and SDK per programming language to extract your application’s observability data, a standard collector, a transmission protocol (OTLP) and more.

    LGTM:
    The most popular open source-based observability and monitoring is implemented using the LGTM technology stack.

    In the LGTM stack, we leverage:
    • Loki for log aggregation
    • Grafana dashboards for telemetry visualizations
    • Tempo (or Jaeger) for trace aggregation
    • Managed Prometheus for metrics aggregation

    Conclusion

    Observability is about complete visibility across your systems and tying business metrics with technical data. Monitoring is about understanding if things are working correctly, and AIOps is about getting meaning from that visibility. Observability and monitoring are critical to ensure the smooth functioning of applications and meet customer SLAs. In conclusion, by investing in open source OTel frameworks and LGTM tools, SRE teams can effectively monitor their applications and gain insights into system behavior and potential issues. These tools offer cost-effectiveness and customization to suit specific requirements. It promotes vendor neutrality, which can be critical for avoiding vendor lock-in.

  • Survey Surfaces Lack of Significant Observability Progress

    Survey Surfaces Lack of Significant Observability Progress

    A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to truly observe application environments, especially as they become more complex with each passing day.

    The survey, conducted by Logz.io, a provider of an observability platform, found only one in 10 respondents reported they have full observability into their application environments.

    Logz.io CTO Asaf Yigal said while more DevOps teams than ever are collecting logs, metrics and traces, most of them have yet to determine how to operationalize all the data being collected. As data is ingested, it doesn’t just need to be stored; it also needs to be correlated to the various services that make up an application, he noted.

    The more organizations deploy cloud-native applications in production environments the more pressing this issue becomes. All the microservices that make up those applications are now generating a massive amount of telemetry data that results in more alerts than ever being generated.

    Not surprisingly, the biggest challenges organizations encounter when managing Kubernetes clusters in production environments are monitoring/troubleshooting (40%), followed closely by security (37%) and networking (33%).

    Many organizations simply don’t have the skills required to manage cloud-native applications. Nearly half of respondents (48%) specifically cited a lack of knowledge as the biggest challenge they encountered when trying to observe these types of applications. From an IT management perspective, most microservices all use the same basic template, so it’s difficult for DevOps teams to identify which microservices are likely to have the biggest impact on service level objectives (SLOs) and service level agreements (SLAs) in the event of a disruption, noted Yigal.

    In the absence of the ability to determine the actual root cause of an issue, alert storms conspire to increase fatigue—and that eventually results in higher levels of burnout across the DevOps team, noted Yigal. In fact, 82% of respondents said their mean-time-to-resolution (MTTR) during production incidents was more than an hour.

    The survey also noted that more than half of respondents (52%) worked for organizations that are simultaneously trying to rein in monitoring costs. More than three-quarters (76%) of respondents also noted that open source OpenTelemetry (OTEL) or OTEL-centric tooling was at least somewhat important to their overall observability strategy.

    In addition, 87% of respondents said their organization is already using some form of platform engineering to manage DevOps workflows at scale.

    It’s not clear how quickly organizations are adopting observability tools and platforms, but too many DevOps teams lack the visibility needed to identify the root cause of an issue. As a result, variations of the same problems often keep manifesting themselves because previous remediation efforts simply didn’t go deep enough to resolve the core issue.

    Of course, there may come a day when machine learning algorithms, along with other forms of artificial intelligence (AI), will make it simpler to surface these issues. The challenge, in the meantime, is laying down the observability foundation today to provide access to the data that will be required to train those AI models.

  • New Relic Streamlines Observability Data Management

    New Relic Streamlines Observability Data Management

    New Relic this week made available a live archives capability that provides instant access to historical logs for up to seven years and eliminates the need to rehydrate, reload, re-index or move data.

    At the same time, the company also announced its observability platform is now integrated with infrastructure and observability tools provided by Sentry Software to help IT teams reduce costs and collect additional sustainability metrics alongside logs, metrics and traces.

    Jemiah Sius, senior director of developer relations for New Relic, said overall, New Relic is moving to make it simple to analyze and store telemetry data regardless of when it was collected. For example, rather than requiring historical data to be accessed via a cold storage service in the cloud, the New Relic platform now makes that data available directly within the database that is at the core of its observability platform.

    That approach also provides the added benefit of not incurring ingress and egress fees for moving data in and out of cloud storage services, noted Sius.

    IT teams can simply create a New Relic Query Language (NRQL) to identify which logs to route to a live archive in as little as 30 seconds without having to add an additional log collector.

    The overall goal is to make it simpler to investigate incidents faster, given the urgency that is created whenever there is an outage or audit request that needs to be immediately addressed, noted Sius. That’s critical because DevOps teams are always for time, he added.

    In general, New Relic is reducing the overall toil associated that many DevOps teams encounter today when managing telemetry data. Rather than being confronted with a complex set of data engineering tasks, telemetry data from multiple applications and infrastructure platforms can automatically be collected, analyzed and stored.

    It’s still early days as far as the adoption of observability platforms is concerned, but there is already no shortage of options. The challenge is that in addition to logs, metrics and traces, IT teams are now starting to also track cost and energy consumption metrics. Observability platforms provide the means to centralize telemetry data that is only going to increase in volume as more applications are deployed. The challenge has been framing the queries needed to surface meaningful insights. Fortunately, with the rise of generative artificial intelligence (AI) tools that make it simpler to create queries, the overall level of skill required to invoke an observability platform is dropping. In effect, generative AI is making it possible to democratize DevOps best practices.

    Ultimately, existing DevOps teams should be able to manage highly distributed application environments without having to hire and retain a small army of additional software engineers. The issue is that, in many cases, the size of the overall code base is increasing rapidly as developers take advantage of various generative AI tools to help them write code faster. Hopefully, advances in AI will arrive soon enough to enable DevOps teams to cope with all that code before DevOps teams are inevitably overwhelmed.

  • Chronosphere Acquires Calyptia to Process Log Data More Efficiently

    Chronosphere Acquires Calyptia to Process Log Data More Efficiently

    Chronosphere today announced it has acquired Calyptia to add additional pipeline management capabilities to its observability platform.

    Calyptia created Fluent Bit and Fluentd, a set of open source log processors that are now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

    Chronosphere CEO Martin Mao said these log processors will be embedded into the company’s namesake observability platform to enable the routing, transformation and optimization of log data at scale. That issue has become problematic because cloud-native applications based on microservices create exponentially more telemetry data than legacy monolithic applications. If it becomes too costly to analyze that data, then DevOps teams will begin to limit the number of applications they will observe, noted Mao.

    The Calyptia platform achieves that goal using filters at the data source to reduce the volume of data that might otherwise be shared by 30% or more.

    The shared data can then be analyzed in real time rather than waiting for it to be stored and indexed. In addition, DevOps teams can enrich that data in flight to include additional context.

    Calyptia cofounder Anurag Gupta added that the primary mission for the company has always been to make log data more accessible to developers as they build and deploy applications.

    The acquisition of Calyptia comes on the heels of a log management alliance between Chronosphere and CrowdStrike through which a log storage and virtualization capability, dubbed Logs powered by CrowdStrike, has been added to the observability platform. The overall goal is to simplify the management and analysis of logs via a single click of a graphical tool or within the context of a larger DevOps workflow based on a command line interface (CLI) without requiring data engineering expertise, said Mao.

    It’s not clear how fully DevOps teams have embraced observability, but it’s apparent the legacy monitoring tools that collect a set of predefined metrics are not able to keep pace with the complexity of cloud-native application environments. An observability platform makes it simpler to query log data and discover the root cause of an issue—hopefully before a catastrophic event occurs. The challenge, of course, is providing that capability without incurring massive increased storage costs for log data.

    Despite those concerns, the need for observability is becoming a more pressing issue with each passing day. Organizations have never been more dependent on applications to drive revenue. A degradation in performance is usually an early indicator of a disruption that often has a direct impact on revenue and profitability.

    There is, of course, already no shortage of observability platforms. The issue most DevOps teams encounter is finding the funding needed to employ one at a time when organizations have become much more sensitive to IT costs. There may be an opportunity to rationalize some monitoring tools to help pay for the cost of a more modern observability platform, but regardless of approach, an ounce of prevention is always going to be worth a pound of proverbial cure.

  • Unlocking Accountability: How Real-Time App Monitoring Empowers Engineering Teams

    Unlocking Accountability: How Real-Time App Monitoring Empowers Engineering Teams

    In today’s fast-paced software development environment, traditional infrastructure monitoring is no longer sufficient. As engineering teams take on more responsibility for code ownership and software quality, the need for comprehensive real-time application monitoring becomes crucial. This isn’t just an addition to standard infrastructure checks; it’s a fundamental shift toward a more accountable and empowering engineering culture.

    A Case for “Real-Time Application Monitoring?”

    Infrastructure monitoring is ubiquitous, and we’ve gotten really good at monitoring the health of our servers, our containers and our data stores. APM tools have come a long way and give us insights into how pieces of our code perform from a speed and error context.

    At the same time, the business side of the house is monitoring things like traffic, leads, conversions and an important performance metric—sales. However, for most businesses, this monitoring is rarely in real-time, and visibility gaps are created in the spaces between where problems are introduced, discovered and resolved.

    Let’s look at a concrete example using an auction marketplace. Core to this business is the ability for users to place, accept and pay for accepted bids. The business probably looks at these metrics on a daily basis, at best. What if a delivered change to their application broke and removed the ability for users to conduct one of these actions?

    Placing a bid probably has a “normal” volume pattern, and a change could impact this in a positive or negative way. If the volume of bids drops after a deployment or even spikes, what happens? How quickly can that change be reported, seen and acted upon, if necessary? How quickly can they get back to “normal” before a customer reports it or before the rest of the business notices?

    If this business is limited to once-daily monitoring, how quickly can they realistically expect their engineering team to be able to identify and remediate such an issue?

    How valuable would it be to the business if their engineering team could instrument these metrics with their own real-time tooling?

    Key Metrics for Trust and Satisfaction

    Customer trust and satisfaction are critical for any software product. The most obvious indicator of customer satisfaction is churn–the retention rate of your customers. This metric, while influenced by various factors, remains a vital sign of how well we are doing. At Stoplight (acquired by SmartBear in August 2023), we focused on reducing customer bug reports. A spike in these reports prompts an immediate investigation into the root cause. It’s about understanding what changed and how to address it.

    Moreover, new customer acquisition isn’t just about retention; it’s also about word-of-mouth referrals. Satisfied customers often become your best advocates. Thus, keeping an eye on churn rates and incoming bug reports and setting realistic targets for these metrics is crucial. It’s about balancing resource allocation without striving for impractical perfection.

    Net Promoter Score (NPS) scores, although often dreaded by customers, are a critical metric. As engineers, we might dislike them, but they offer a holistic view of user satisfaction. If you’re deploying daily and consistently breaking things, expect your NPS scores to reflect that.

    We can also demonstrate real-time ROI of real-time application monitoring in a security context using login successes and failures. Tracking the volume of these activities in our engineering tools allows us to quickly identify and assess the threat of anomalous behavior. Did the rate of successful logins drop? Did we break something with a deploy (I’ve broken logins before). What if either successes or failures spike? Are we under attack? How can we mitigate it to protect our users?

    Employee Retention and Developer Satisfaction: Essential in an Era of Real-Time Monitoring

    In the realm of software development—particularly with the advent of real-time application monitoring—employee retention, especially of developers, is paramount. Their deep understanding of the nuances of our applications and their ability to respond swiftly to the insights provided by real-time monitoring are invaluable.

    Maintaining a team of satisfied, engaged developers is crucial in this context. It’s not just about reducing turnover; it’s about fostering a culture where the engineers feel invested in the continuous improvement and success of our products. When developers are genuinely satisfied with their work and their environment, it reflects in the quality of their output. They become proactive in identifying and addressing issues, often before they escalate, thanks to the real-time data at their fingertips.

    The shift toward more dynamic monitoring practices has underscored the need for a supportive, collaborative environment. A culture where developers are encouraged to share insights and take initiative leads to a more responsive and adaptable team. This environment not only supports the technical aspects of our work but also enhances the overall morale and commitment of our developers.

    Engineers will seek out organizations that adhere to better practices. Interestingly, when good practices are in place, developers are more open to being on call. They accept responsibility for their code and their peers’ code as a trade-off for the autonomy and value they can deliver through frequent, quality deployments.

    Starting Small: Embracing the Journey of Incremental Improvements

    The journey to adopting best practices in engineering, such as real-time application monitoring, often begins with small yet significant steps. Reflecting on my past experiences, particularly at a shipping marketplace, the path to continuous deployment was transformative. Initially, we had no automated tests, and the challenges were substantial. However, the commitment to deploy continuously led to an evolutionary process in our practices.

    This approach isn’t just about technology; it’s about mindset. In my earlier days, inspired by the continuous deployment practices I observed in financial institutions, I realized the power of incremental changes. We began with what we had, deploying continuously, and with each step, we uncovered new areas for improvement. It was a process of building our capabilities piece by piece–from integrating automated tests to refining our deployment strategies.

    The essence of this journey lies in its continuous nature. Much like the principles of agile and DevOps, it’s a perpetual process of evolution. You don’t reach a finish line and stop; it’s about relentlessly pushing for betterment, identifying gaps, and adapting to new challenges. It’s about creating a culture where continuous improvement is the norm, where each small step contributes to a larger goal.

    Building a Culture of Continuous Improvement and Excellence

    Adopting real-time application monitoring is more than implementing a new set of tools; it’s about fostering a culture of continuous improvement, accountability, and excellence. In my experience, this cultural shift has a profound impact on every aspect of our work. It’s about ingraining a mindset that constantly seeks to exceed our standards with each release.

    Creating this culture involves a holistic approach to our engineering practices. It’s not just about the technical side of things; it’s about enhancing the collaborative and innovative spirit of our teams. It’s important for engineering leaders to do their part to foster an environment where every team member feels empowered to contribute ideas and take initiative. The ROI of this approach can lead to significant improvements in your team’s processes, as well as a deeper sense of ownership and pride in their work.

    Moreover, this culture of continuous improvement often extends well beyond the engineering department. It can influence the entire organization, encouraging departments like sales, marketing, and customer support to adopt similar principles. By doing so, we create a unified approach to product development, customer service, and business growth.

    Conclusion

    Embracing real-time application monitoring is not just about adding another layer to our development practices; it’s about fundamentally shifting our mindset toward a culture of accountability and continuous improvement. By focusing on key metrics, understanding the psychology of developers, and starting small, we can drive significant improvements in both product quality and employee satisfaction. As engineers, we are in a unique position to influence this change, and it is imperative that we lead the charge in adopting these essential practices.