devopsy.kinsta.cloud

Category: Observability

  • Chronosphere Acquires Calyptia to Process Log Data More Efficiently

    Chronosphere Acquires Calyptia to Process Log Data More Efficiently

    Chronosphere today announced it has acquired Calyptia to add additional pipeline management capabilities to its observability platform.

    Calyptia created Fluent Bit and Fluentd, a set of open source log processors that are now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

    Chronosphere CEO Martin Mao said these log processors will be embedded into the company’s namesake observability platform to enable the routing, transformation and optimization of log data at scale. That issue has become problematic because cloud-native applications based on microservices create exponentially more telemetry data than legacy monolithic applications. If it becomes too costly to analyze that data, then DevOps teams will begin to limit the number of applications they will observe, noted Mao.

    The Calyptia platform achieves that goal using filters at the data source to reduce the volume of data that might otherwise be shared by 30% or more.

    The shared data can then be analyzed in real time rather than waiting for it to be stored and indexed. In addition, DevOps teams can enrich that data in flight to include additional context.

    Calyptia cofounder Anurag Gupta added that the primary mission for the company has always been to make log data more accessible to developers as they build and deploy applications.

    The acquisition of Calyptia comes on the heels of a log management alliance between Chronosphere and CrowdStrike through which a log storage and virtualization capability, dubbed Logs powered by CrowdStrike, has been added to the observability platform. The overall goal is to simplify the management and analysis of logs via a single click of a graphical tool or within the context of a larger DevOps workflow based on a command line interface (CLI) without requiring data engineering expertise, said Mao.

    It’s not clear how fully DevOps teams have embraced observability, but it’s apparent the legacy monitoring tools that collect a set of predefined metrics are not able to keep pace with the complexity of cloud-native application environments. An observability platform makes it simpler to query log data and discover the root cause of an issue—hopefully before a catastrophic event occurs. The challenge, of course, is providing that capability without incurring massive increased storage costs for log data.

    Despite those concerns, the need for observability is becoming a more pressing issue with each passing day. Organizations have never been more dependent on applications to drive revenue. A degradation in performance is usually an early indicator of a disruption that often has a direct impact on revenue and profitability.

    There is, of course, already no shortage of observability platforms. The issue most DevOps teams encounter is finding the funding needed to employ one at a time when organizations have become much more sensitive to IT costs. There may be an opportunity to rationalize some monitoring tools to help pay for the cost of a more modern observability platform, but regardless of approach, an ounce of prevention is always going to be worth a pound of proverbial cure.

  • Observability: The Central Watchtower Your All-Seeing IT Needs

    Observability: The Central Watchtower Your All-Seeing IT Needs

    If there’s something you need to keep secure, it’s natural to want to keep it sealed away under lock and key. However, even keeping something in a vault, far from prying eyes and ill intent, may not be enough to offer peace of mind. After all, how can you have the peace of mind of knowing it’s safe if you don’t regularly check to ensure it hasn’t bolted, been burgled or broken down?

    Now, multiply the volume and value of things being monitored tenfold — and you get a sense of the daily inefficiencies and anxieties IT professionals battle.

    Instead of jewels, these teams safeguard, manage and maintain something far more valuable: The data that underpins everything from business transactions to system performance to threat detection to service delivery and beyond. To make matters more complicated, today’s digital ecosystems are more complicated, distributed and disparate than ever before.

    Databases are critical to managing a successful IT department — and it’s time to discard the widespread industry’s perception of them as an unknowable black box. To be sure, databases represent the most difficult component of the IT ecosystems to observe, tune, manage and scale. But the database isn’t a Magic 8 ball in terms of its internal processes; We no longer have to shake the mystery toy and accept one of the six answers it’s designed to spit out.

    The fact of the matter is that database specialists and IT teams need a clear view of database performance telemetry if they want any hope of maintaining the health, stability and scalability of their services.

    Observability Tools

    This is where observability tools can play a transformative role. Taking its cues from the all-seeing panopticon watchtower, observability eliminates the dark corners of the database dungeon by providing a comprehensive view across the full cloud-native, on-premises and hybrid technology stacks.

    The word panopticon derives from the Greek word panoptes, or “all-seeing.” The panopticon model features a central, lighted guard tower at the center of a circular room — allowing 360-degree lines of sight that allow the guards to observe every surrounding cell from a single, centralized vantage point.

    English philosopher Jeremy Bentham had no way of foreseeing the challenges of today’s database professionals when he first conceived of the panopticon in the 1700s. The concept would lay the blueprint for a prison system designed around one core concept: Enabling the minimum number of guards to effectively monitor the maximum number of inmates. Bentham’s system also works as a strategy for ensuring the health and performance of your organization’s critical databases.

    The panopticon was designed so that every cell could be monitored from one central point — ensuring not only better and more straightforward security but preserving resources in saved time, manpower and effort expended. Think of your ITOps, database, DataOps and DevOps teams as the wardens of your IT performance. Traditional monitoring can be likened to patrolling a traditional prison: Rows of cells facing out into a hallway that is patrolled on a schedule or used to spot and address misbehavior as it’s noticed.

    How can they be sure they’re focusing their efforts in the right place? Or trust that Cell Block A won’t stage a breakout while they’re breaking up an argument in Cell Block B?

    In short: They can’t. Odds are, the team is either running from outage to outage in an everlasting firefighting mode or relying on random spot checks for something as vital as the health of our IT systems.

    Observability, as opposed to monitoring, puts teams into that central panopticon watchtower, allowing them to not just see everything, but to see it within the context of the bigger picture. This comprehensive visibility and always-on observation enable IT teams to identify critical issues as they occur, even those caused by complex dependencies between the database, operating system, storage subsystem and network.

    Database Performance Issues

    Furthermore, database performance issues are a starting gun for costly bottlenecks or serious outages that can handcuff your company’s ability to compete or grow. Without complete and precise database monitoring and observability, IT operations struggle to accurately determine an application’s root cause of performance issues. This increases the risk of downtime, data loss and poor customer experiences.

    Discovering the root cause of performance issues is quite literally a system-critical undertaking, whether you’re simply running your enterprise, scaling your operations or deploying new code. Observability also offers teams the rare opportunity to get ahead of performance issues since you can see performance anomalies in advance of a service disruption rather than relying on real-time remediation.

    With traditional methods, DevOps and IT teams must manually analyze the data presented to them, correlate it to the problem, and locate the error before they can finally begin addressing it. In contrast, observability collects data to provide information on what’s not performing as expected and why. This allows teams to take a proactive approach to solving downtime and bottlenecks and proactively preventing them.

    In the software world, that translates directly to more time spent where it matters: Developing new business value within your applications and infrastructures, fueling innovation and exceeding customer expectations.

    Given how complicated the modern database has — and continues to — become, arming your enterprise with the right observability solutions can provide a boost to efficiency and performance and pardon your teams from a sentence to time-consuming, mundane and error-prone work.

  • How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

    How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

    In a digital world, data is often the differentiator between success and failure. Whether it’s defending against cybersecurity threats, improving application performance, or resolving full-blown outages, data is critical to the modern DevOps team.

    In fact, there are plenty of situations where DevOps engineers, SRE teams and other observability-focused departments need to explore reams of data rapidly and flexibly. An SRE trying to figure out the root cause of latency may need to dissect the movement of data across endpoints to find the malfunctioning one. Another engineer may need to compare historical and current performance to find anomalous behavior. Lastly, a DevOps engineer may have to dissect user performance metrics to understand the scope of a slowdown—whether it is global or regional.

    Every use case requires dashboards that can provide detailed data through a wide variety of visualizations, including choropleth maps, stack areas, pie charts and bar graphs. Ideally, a dashboard will enable teams to isolate dimensions, apply filters, and dive into data for deeper insights.

    Unfortunately, not all dashboards can support this type of fast, flexible analysis. Many still utilize outdated technologies, which lack the flexibility, agility and scalability required to seamlessly explore data. Many were also designed without the urgency that today’s data requires—after all, in previous years, most uses for dashboards (such as internal reporting) were not time-sensitive.

    Other dashboards are constrained by templates, which offer a finite array of widgets, tools and drill-down capabilities. These dashboards may feel unwieldy and sluggish, unsuited for a rapidly evolving situation like an application outage.

    Requirements

    To truly fulfill the promise of self-service, ad-hoc analytics across huge datasets, teams need to work in real-time—and so they need a database capable of timely responses.

    When an application goes down, an SRE may not know what to look for and, hence, needs to quickly comb through lots of data. While some dashboards rely on workarounds for faster queries, such as pre-aggregations, precomputing, or rollups, this is not possible in this instance because the SRE simply won’t know what to look for. After all, they can’t necessarily predict what will go wrong, and even if they work on assumptions from previous failures, this current issue could be much different.

    Therefore, dashboards must provide plenty of functionalities and visualizations. Users should be able to filter data by time, isolate variables like location, and zoom into specific time intervals with a few clicks. Dashboards should also accommodate diverse data types, including intricate parent-child relationships and nested columns.

    Further, dashboards must offer a depth of insight. An online streaming media platform may need to assess user metrics (such as latency or load times) across different devices, operating systems, and regions to fine-tune performance. A cloud provider has to monitor their physical hardware for high temperatures, slowed network switches or devices, and other anomalies.

    Because data is now so important to success, many more people across a company, including data scientists, product managers and external users, themselves require data-driven insights. In these situations, dashboards must handle the increased user traffic and query activity, maintaining fast responses even under load. This is especially important considering that a single-user operation (such as a zoom) will require multiple queries on the backend to execute.

    A database also must scale seamlessly. If an organization’s environment generates millions of events an hour, that equates to billions of events per day or week—challenging for any database to ingest, store, analyze and query. In fact, many databases cannot successfully provide fast response times while managing large datasets and high query volume. As an example, transactional databases (OLTP) can often query rapidly but cannot execute analytics at scale, while analytical databases (OLAP) can analyze massive volumes of data but not at speed.

    Apache Druid for Independent, Ad-hoc Data Exploration

    This is where open source Apache Druid comes in. Combining the scale and advanced analytics of an OLAP database with the speed of an OLTP database, Druid offers swift, real-time data exploration.

    Upon ingesting data, Druid makes it immediately available for querying and analysis, removing the need to first batch or aggregate the data in some way. In addition, Druid natively integrates with streaming technologies like Apache Kafka and Amazon Kinesis, removing the need for workarounds or connectors.

    Druid powers interactive visualizations, providing millisecond response times, enabling more versatile exploration, expanding the range of available dimensions and filters, and even maintaining subsecond speeds in the face of surging user and query volumes.

    Druid’s unique architecture also enables easy scaling. By separating key duties among separate node types—data nodes for storage, master nodes for data ingestion and availability, and query nodes for executing queries and returning results via the scatter/gather method—Druid ensures that nodes can be independently scaled based on need. Afterward, Druid also automatically rebalances traffic to ensure consistent performance.

    Salesforce: A Druid Success Story

    Salesforce pioneered the customer relationship management (CRM) space, serves 150,000 customers worldwide and earns billions in annual revenue. 

    The Edge Intelligence Team is the division of Salesforce that tackles the massive task of ingesting, processing, filtering, aggregating, and querying the entirety of their log data — anywhere from billions to trillions of lines daily. Each minute, Salesforce ingests 200 million metrics, while each day, Salesforce processes five billion daily events globally. In total, Salesforce accumulates dozens of petabytes of data in their transactional store, five petabytes of logs in their data centers, and almost 200 petabytes in their Hadoop storage.

    Salesforce teams use Druid to unlock real-time insights into product performance and user experiences, diving into large datasets instantly. Anyone from engineers to account executives can query a wide variety of dimensions, filters and aggregations to better understand trends, troubleshoot any issues that arise and set strategies for the future.

    By using Druid’s compaction abilities, Salesforce also decreased the number of Druid rows by 82%, leading to an overall reduction of their storage footprint by 47% and accelerating their query response times by 30%.

  • Observability Leaders: The Mighty Minority Holding Their Own Against Outages

    Observability Leaders: The Mighty Minority Holding Their Own Against Outages

    The job of IT teams has become much harder thanks to the new realities of hybrid IT, with workflows occurring across on-premises and cloud-based networks and applications. Yet so much rests on your ability to respond to these challenges, all while managing the expectations of your customers and the health of your bottom line. With the growing complexity of ecosystems comes the need for better visibility, both into specific systems and across your entire stack.

    Observability solutions address this by providing visibility across your network, infrastructure, systems, application, database, digital experience and log monitoring.

    In other words, observability enhances your ability to observe your digital ecosystem with more transparency, more effective detection and more intelligent insights. Take, for example, the very timely and pressing issue of outages. On average, the typical business experiences nine brownouts or outages every month, lasting roughly twelve hours each and averaging $13.7M annually, according to new research we conducted at SolarWinds.

    Over the course of the year, that can add up to nearly 1,300 hours, with outages chipping away at your organization’s productivity, reputation and competitive edge.

    It’s not enough to monitor for these outages and address them once they take down your systems. Repeated or prolonged outages signify a gap in the enterprise’s ability to comprehensively monitor and understand its network’s internal state and performance. A robust observability strategy can preemptively detect anomalies and potential issues—before they become full-blown outages.

    According to new data from the 2023 IT Trends Survey released this month by SolarWinds, companies who’ve embraced observability tend to not only lead in detecting, remediating and preventing outages—they’re experiencing better business and IT outcomes, too.

    The technology professionals surveyed shared how observability is helping them achieve better performance, compliance and resilience in digital environments—and how you can follow their lead.

    Observability Drives Better Business Outcomes

    The observability leaders surveyed were 300% more likely to say their organization is doing extremely well with growing revenue and 250% more likely to say the same about their organization’s speed of innovation. Interestingly, they also reported a better employee experience: They were 388% more likely to rate their organization as doing “extremely well” with employee experience and also reported fewer and smaller skill gaps on their teams.

    So, how does embracing observability actually translate into better business outcomes? According to the data, observability leaders are…

    • Investing in top priorities: Data shows organizations using observability solutions to support the priorities most critical to their growth and success: Improve their customer experience (96%), enable faster innovation (71%), reduce time spent solving (71%) and detecting (60%) issues and increase operational efficiency (55%).
    • More automated and integrated: Observability leaders embracing automation and investing in tools that provide enhanced efficiency are 214% more likely to say they are doing extremely well with operational efficiency, 750% more likely to say they are doing extremely well with auto-remediation of complex alerts and 300% better at automatically collecting background diagnostic data for IT support staff.
    • Ahead on IT: The data found that those ahead of the curve on observability are also leading by huge margins when it comes to monitoring, detecting and resolving issues that could otherwise bring the business to a screeching halt. When it comes to IT, they are 233% better at auto-escalation of tickets, 213% better at auto-remediation of simple alerts and 36% better at settling alert levels based on historical behavior.

    How to Reap Observability Benefits in Your Organization

    All told, observability adoption is still in its early days. Only 1% of enterprises are already up and running using observability tools on a regular basis, while another third are undergoing the implementation process.

    That’s good news for other businesses who are ready to reap the rewards—it’s not too late to be an observability pioneer.

    If you’re interested in bringing observability to your own organization, start by addressing a few important factors that can help you sidestep challenges—such as the accelerating pace of change in technology (72%) or insufficient budget (58%)—that might otherwise obfuscate your observability opportunities:

    • Make sure your observability solutions are appropriate for multi-cloud environments: Multi-cloud environments have become standard for many companies because of the many upsides, but they can be challenging for IT teams to manage. Strategically accelerate your organization’s cloud migration with an observability solution designed for complex hybrid and multi-cloud environments​ to save a lot of headaches down the road.
    • Give your teams autonomy—and build toward autonomous operations: Observability brings your enterprise a step closer to autonomous operations. Autonomous operations, where IT systems can govern themselves, providing a proactive approach to addressing issues, rather than reactive, won’t replace your human teams but rather free up their time and brainpower to support the business and drive innovation.
    • Don’t ignore the database: Databases represent the most difficult ecosystems to observe, tune, manage and scale. They require significant computing power, tend to be memory-intensive and likely make up the bulk of your organization’s cloud spending. Tasks crucial to the health and functionality of your business—including troubleshooting, root cause analysis and remediation—simply cannot happen without full, transparent observability across the numerous resources a database needs to engage to do its job.

    Observability helps balance competing priorities and demystify application requirements while increasing your overall DevOps agility, efficiency and speed to market. Observability not only fosters transparency and collaboration but also encourages a system of ongoing review, which benefits the organization, IT teams and—crucially—your customers.

  • How Observability and Explainability Benefit the SDLC

    How Observability and Explainability Benefit the SDLC

    Software systems require continual modification to meet changing requirements, address errors, improve performance and enhance reliability. Yet, maintaining and modifying complex and distributed software systems as they change and grow can present challenges. When systems are designed with observability and explainability in mind, adapting, expanding and troubleshooting software becomes easier and more effective. Observability refers to the ability to discern and monitor software systems’ real-time behavior and performance. This allows developers to identify and fix issues before they become critical. Explainability articulates how and why the software system operates.

    Both observability and explainability can contribute to the health of the software development life cycle by improving visibility and transparency, helping developers better understand the system’s overall behavior, and allowing the easy identification of errors, all of which contribute to the software system’s overall stability.

    Overview and Benefits of Observability

    Observability combines data from various sources, including application performance management (APM) monitoring, infrastructure monitoring, distributed tracing, third-party integrations and machine-language data. The data is then structured and presented in an accessible way. The first step to building an observable workflow is to identify which metrics best measure the behavior and performance of a software system. This will vary based on the specifics of the system. As a software system operates, logs and traces keep track of activities, requests and responses. They can be used to find the causes of errors and performance bottlenecks. Application performance monitoring (APM), infrastructure logs, machine logs and application traces are just a few examples of logs and traces. These records provide the necessary data to create dashboards and visualizations, which can then be presented to organizational stakeholders.

    While observability is often compared to software monitoring, which generates data focusing on various metrics related to application performance, it differs from monitoring in several respects. Monitoring tracks raw data related to a few performance metrics. It provides information that is best used by the operations team. Observability uses a large amount of structured data to provide a more robust, accessible picture of the whole system, from front end to back end, that can be understood by employees throughout the organization—even those who do not work in development or lack a technical background entirely.

    Advantages of Explainability

    Explainability refers to transforming machine data into a clear articulation of how a software system works such that others outside the organization can understand it. This capability empowers developers and users alike by enabling them to understand what a software system does and how and why it does it, providing a deeper understanding of why errors occur and how to fix them. Presenting data in a visual format is often a vital part of explainability.

    Explainability is especially important in artificial intelligence (AI) and machine learning (ML) systems, where it constitutes a cornerstone of ethical AI development. Model tracing, which refers to monitoring an ML model as it executes to understand its decision-making process better, is a key component of explainability in ML.

    Best Practices and Common Issues With Observability and Explainability

    Before adding observability or explainability to the software development life cycle, it is important to revisit software life cycle best practices and examine what kind of data is being generated. Not all data is meaningful or useful in observability and explainability. Structuring and ordering data from logs and traces is time-consuming but vital to correctly analyze and draw conclusions from the data. For optimal results, it is important for traces to be appropriately tagged and named consistently according to conventions.

    Additionally, the tools used to implement observability and explainability can be complex. For organizations to achieve maximum benefits from these strategies, it’s important for employees to familiarize themselves with the tools and develop specialized knowledge and skills, which can take time. Finally, there are costs associated with implementing observability and explainability. On one hand, commercial software tools come with licensing costs, which can pose a barrier for small businesses. On the other hand, open source alternatives often do not perform well because they lack the funding and resources required to continuously update and enhance the software.

    Harnessing the Power of Observability and Explainability

    Observability and explainability have numerous benefits for organizations that use flexible, ever-changing and complex software systems. Any company interested in building smoother-functioning software applications stands to benefit. Though both observability and explainability require upfront investments of time and money, these investments pay off in the form of quicker and easier error correction, with resolution occurring in minutes rather than hours or days, and greater application stability. In customer-facing applications, this results in fewer crashes and a better experience for the user. Increased stability also makes it easier to scale applications without losing performance. As these benefits become more apparent, a significant number of companies will likely develop software solutions that are reliable, cost-effective and accessible. This will increase the adoption of observability and explainability, which will continue to influence the evolution of software systems.

  • Using Observability to Create a Smart Data-Driven Culture

    Using Observability to Create a Smart Data-Driven Culture

    The first words that come to mind when talking about smart technologies are often artificial intelligence, machine learning and automation. While these smart technologies play a crucial role in smart-driven cultures, they are not the sole factor in making workplaces “smart.” Organizations that have a smart-driven culture, or data-driven culture, also promote innovation. An innovative organization actively encourages creativity, exploration and the pursuit of new ideas, continuously developing and implementing groundbreaking solutions, products and processes. Innovative companies with a culture of learning and growth are often seen as desirable employers and have high employee retention rates, attracting top talent.

    Let’s go over the benefits of a smart-driven culture and how an observability platform can help your company become data-driven in five steps.

    Why is Smart-Driven Culture Important

    You can increase efficiency and productivity among your employees by fostering a smart-driven culture. Smart technologies are able to automate repetitive tasks and streamline processes, which helps free up workers’ time and focus on other issues instead.

    A data-driven culture can also drastically improve decision-making on a company level. Your organization will begin to strategize and act on objective insights rather than guesswork or assumptions. Through collecting, analyzing and interpreting data, you will not only be able to make informed decisions but also know how to optimize processes and mitigate risks.

    Use data to better understand your customers’ preferences and improve your products to meet their demands more effectively. For example, leverage smart technologies and create high-tech touchpoints to tailor each of your customer’s experiences. A culture that encourages continuous learning, experimentation and adaptation is ready to navigate market shifts, changing customer needs and emerging technologies.

    How to Create a Data-Driven Culture in Five Steps

    According to the 2023 Data Analytics Survey conducted by New Vantage Partners, 79.8% of data leaders identified people, business processes and organizational alignment as the principal challenge to becoming data-driven. These folks cited cultural issues, including organizational receptivity to change, communication and organizational alignment, as the greatest obstacles to realizing business value.

    One way to overcome these challenges is by harnessing the power of observability, building data literacy and organizational transformation. Observability refers to the practice of gaining real-time insights into systems, applications, and processes through the collection and analysis of relevant data. Real-time insights will enable your organization to make data-driven decisions, understand your customers better and quickly identify areas of optimization to improve your products.

    These characteristics are fundamental to smart-driven companies, enabling them to stay competitive and drive innovation in the market. Take the following five steps to ensure you’re maximizing the potential of observability and paving the way toward becoming a smart-driven organization:

    1. Define Clear Goals

    Start by defining what a smart-driven culture means for your company. Write down clear outcomes that you want to achieve with a new culture, and identify the specific areas where observability can make a significant impact on your team.

    Focus on areas where real-time insights derived from observability can generate substantial business value. Set SMART (specific, measurable, achievable, relevant and time-bound) goals that can be tracked over time. For example, you could aim to reduce website loading time by 20% in the next two months in order to increase customer conversion.

    2. Implement Robust Observability Tools

    Evaluate and choose observability tools that align with your organization’s specific needs and goals. Look for tools that offer features such as real-time monitoring, log management, distributed tracing, advanced analytics and visualization capabilities.

    Consider factors like scalability, ease of use, integration with existing systems, and the ability to handle diverse data sources. Remember that the availability of accurate and timely insights promotes a culture that values data and encourages performance for achieving business objectives.

    Ask your IT department to help set up your observability tools to gather the relevant data. For our previous example goal of improving website loading times, this will mean gathering real-time data on website performance, such as server response times, page load speeds, and user interactions.

    3. Promote Data Literacy

    Your observability tools are of little help if no one understands how to use and derive helpful insights from the data you gather. Educate all of your employees on good data practices and the role of analytics in decision-making.

    Provide training and support on the data tools you have chosen and encourage your employees to understand, interpret and utilize data effectively in their day-to-day work. For instance, in meetings, you can ask for data points to back up business decisions.

    4. Encourage Collaboration and Knowledge Sharing

    Creating cross-functional teams and encouraging collaboration among different departments, such as IT, operations and analytics, is crucial for the successful implementation of observability practices.

    Taking an interdisciplinary approach allows for a better understanding of the systems and data within the organization. This can lead to the identification of optimization opportunities, help break data silos and practice collective problem-solving as a result.

    5. Iterate and Improve

    Embrace a culture of experimentation and continuous improvement by using observability data to identify areas for optimization and innovation. Encourage teams to test hypotheses, implement changes and measure the impact, fostering a cycle of learning and refinement. Track progress against the goals you set at the beginning and adjust your trajectory if needed.

    Coming back to our goal to improve website loading time. Now that your interdisciplinary team implemented some changes and reduced the loading times, you can assess the impact on user experience and business metrics, such as bounce rates, conversion rates and customer satisfaction.

    Becoming a smart-driven organization will grant you a great competitive advantage. You will make better and faster decisions, employ the best talent, maximize their productivity, and provide an enhanced experience for your customers.

  • Why Your Data Lake Strategy is Broken for Observability

    Why Your Data Lake Strategy is Broken for Observability

    Someone once said, “Life is better at the lake.” That may be true in the physical world, but when it comes to using data lakes for observability, that’s not typically the case.

    Just about every organization in every industry has a digital presence, whether it’s a website, an app or another digital service. This makes observability a universal need; it helps teams monitor software performance, optimize performance, detect potential security issues and quickly monitor for bugs if and when they occur. These responsibilities typically fall on engineering teams and require as close to real-time response as possible.

    Some organizations also have business intelligence (BI) needs, aimed at generating insights from multiple data sources for business decision-making. These responsibilities typically fall on business decision-makers; they are not real-time decisions, but rather made by looking back over long time periods.

    All-In on a Data Lake Strategy

    Companies that go all-in on a data lake strategy may ignore the needs of the engineering teams working on the real-time mission-critical aspects of the business and force all of the company’s data, queries, dashboards, etc. into a data lake solution. That doesn’t actually meet the needs of the engineering teams. The result? Engineering teams use unsanctioned solutions, fracturing the company’s data strategy from the ground up.

    By using a standards-based observability solution built on open source InfluxDB 3.0, companies can give their engineers the observability tools they need while preserving their data lake strategy.

    Here are three main challenges that teams come across when using data lakes for time-series, and how to solve the problem with InfluxDB 3.0.

    Problem One: Inefficient and Slow Ingest
    Data sources generate time-series data at an incredibly high rate, easily producing hundreds or thousands of data points every second; some sources even do so as quickly as every nanosecond. Data lakes can store massive volumes of data, but they aren’t optimized to quickly ingest and make it readable on a millisecond time scale.

    Data lake ingest jobs can be measured in minutes or even hours, which is far too late for any meaningful interventions that can prevent catastrophes such as downtime. Customers will notice the outage before the data is even ingested into the data lake, much less acted upon.

    Problem Two: Time-Series Life Cycle
    Full-fidelity data is necessary for real-time analytics and observability, but the value of the full-fidelity data fades over time. Storing the full-fidelity data in a data lake may be convenient, but it is costly and not necessary for typical BI purposes.

    InfluxDB is designed for the time-series data life cycle. This life cycle entails:

    1. The leading edge of data. That is, data written in the last two hours or so. This is the data that is most likely to be queried, so the full fidelity data is kept in memory ready to query instantly.
    2. Compacted data. That is, data older than the leading edge that is still likely to be queried and requires full fidelity is compacted off to disk in object store, which still permits fast querying but does not incur the costs of storing data in memory.
    3. Retention policies. That is, data that has lost its value in its full fidelity form. For example, for observability data, data older than a few weeks can be aggregated into a much smaller data footprint. Then, this aggregated data is ingested into the company’s data lake while the full fidelity data is automatically dropped from InfluxDB 3.0.

    Without such a system, it is necessary to store the full fidelity data in the data lake and incur those massive costs without a reasonable way to manage the life cycle of the data. With such a system, the correct amount of observability data can be ingested into your data lake to support your BI needs.

    Problem Three: Slower Query Performance
    Engineers can experience a high degree of frustration when they are troubleshooting an issue in real-time, but their dashboards won’t load or their queries won’t return in a reasonable amount of time. Data lakes simply are not designed to produce fast results for observability, and so query performance is a primary complaint of engineers trying to use a data lake for observability. Waiting minutes for a query to return when you are running a BI report may be annoying, but it could spell extended downtime for an engineering team.

    Fortunately, InfluxDB is built on the Apache Arrow standards, which means that it can be directly queried by your engineering team. Via protocols such as Delta Sharing and Iceberg, anyone can query into InfluxDB 3.0 from their data lake and join the real-time observability data stored in InfluxDB 3.0 with the data lake data. This is akin to having your cake and eating it too.

    The Bottom Line

    If you are trying to use a data lake strategy to fulfill your engineering team’s observability needs, you are going to frustrate those teams because the data lake is not designed for data ingestion, the data life cycle, or the query needs of those teams. This will cause those teams to create rogue solutions that are inconsistent with your company’s data lake strategy.

    By providing those teams with open source InfluxDB 3.0, engineering teams will have a first-class observability platform that is designed to interoperate sensibly with your company’s data lake. This enables the critical mission of your observability platform while maintaining your overall data lake strategy for your BI needs.

  • The Best (and Worst) Reasons to Adopt OpenTelemetry

    The Best (and Worst) Reasons to Adopt OpenTelemetry

    It was a rainy day in Seattle at KubeCon + CloudNativeCon North America in December 2018 when I first encountered the term ‘OpenTelemetry.’ At that time, I was an active member of a working group focused on developing W3C Trace Context, a standard now extensively employed for context propagation in distributed systems. It is likely no coincidence that some key individuals in this working group also played a pivotal role in orchestrating the merger between two prominent open source telemetry products: OpenCensus and OpenTracing. I never would’ve guessed that, in 2023, OpenTelemetry would become the second-largest project in the CNCF, sitting just behind Kubernetes.

    As is the case with any popular new technology like OpenTelemetry, the hype can make it difficult to discern signal from noise. During my years working in observability, I’ve had the chance to chat with many organizations that were thinking about using OpenTelemetry or already actively use it. Through those conversations, I’ve heard many reasons for adopting OpenTelemetry, some good and some bad. In this post, I will share examples of the best and worst reasons to adopt OpenTelemetry to help you cut through the noise.

    Poor Reason One: Following the Crowd

    It’s never a good idea to adopt something solely because others are doing it. Just because you saw a conference talk praising a new technology as the greatest innovation since sliced bread doesn’t necessarily mean that technology is suitable for your specific use case.

    Speaking of which, have you clearly defined your use case? To start, ask yourself the following three questions:

    1. Are you able to determine at any given moment whether your systems are performing as expected, such as meeting their service level objectives (SLOs)?
    2. When something goes wrong, can you easily pinpoint the root cause?
    3. Are you able to drill into previously unidentified problems, also known as “unknown unknowns”?
    If you answered these questions with “no,” you certainly have observability gaps worth tackling, and OpenTelemetry may help close some of them.

    Poor Reason Two: Building Your Own Observability Product

    Gathering telemetry data can be a challenge, and with OpenTelemetry now handling essential signals like metrics, traces and logs, you might feel the urge to save your company some cash by building your own system. As a developer myself, I totally get that feeling, but I also know how easy it is to underestimate the effort involved by just focusing on the fun parts when kicking off the project. No joke, I’ve actually seen organizations assign teams of 50 engineers to work on their observability stack, even though the company’s core business is something else entirely.

    Keep in mind that data collection is just a small part of what observability tools do these days. The real challenge lies in data ingestion, retention, storage and, ultimately, delivering valuable insights from your data at scale. Chances are, a homegrown solution won’t give you a return on investment that your company would appreciate.

    However, whether you build your own system or use an existing product, OpenTelemetry may be a good fit for your observability stack.

    Poor Reason Three: ‘Vendor Neutrality’

    People I chat with often cite vendor neutrality as a major reason for their interest in OpenTelemetry. While there’s some truth to this, which I’ll touch on shortly, it’s important to take this argument with a grain of salt to avoid over-promising your management.

    If you’re a developer, you’ve likely worked with database abstraction libraries and have probably mentioned to someone that they allow you to swap out the database at any time–using MySQL today and Oracle tomorrow, for instance. While that’s the promise, I’ve never seen this happen in my career. There’s more to consider other than the SQL dialect when switching a database backend.

    In the same vein, the idea of vendor neutrality suggests that you can easily switch your observability backend. While this might be true in theory, in practice, vendor lock-in often occurs on the backend, where people and processes become closely tied to the tool currently in use.

    So, if you believe you’ve achieved vendor neutrality by going all-in with OpenTelemetry, I’d encourage you to think twice and consider building a stronger case. Vendor neutrality should not be your only reason.

    With all that said, it’s time to focus on the good reasons to use OpenTelemetry today. Trust me, there are plenty, and they’re definitely worth considering.

    Good Reason One: Bridging Gaps With Manual Instrumentation

    Nearly every observability tool out there today relies on some degree of auto-instrumentation. This means once you’ve added their respective language SDK to your code, the library will utilize a technique called “monkey patching” to insert observability code that runs automatically.

    At the very least, auto-instrumentation will search for recognized libraries and APIs and then add some code to indicate the start and end of well-known function calls. Additionally, auto-instrumentation takes care of capturing the current context from incoming requests and forwarding it to downstream requests.

    However, auto-instrumentation falls short when it comes to pure userland code, particularly for dynamic languages like JavaScript. It won’t catch your blocking Fibonacci function that’s slowing down all your requests by 50%. In some cases, you might also have to add your own context-propagation logic to RPC calls not captured by the auto-instrumentation.

    That’s where manual instrumentation comes in. In the past, vendors provided an API for their SDK, allowing developers to manually add instrumentation code to segments that weren’t auto-instrumented. This often resulted in code being littered with vendor-specific instructions. This brings us back to the topic of vendor neutrality. While adding a dependency to your project is usually no big deal, it becomes cumbersome when 20% of your codebase is made up of vendor-specific observability instrumentation.

    OpenTelemetry addresses this issue by offering both auto-instrumentation and a vendor-neutral API for manual instrumentation. This allows observability vendors to access the data while maintaining vendor neutrality within your codebase.

    Good Reason Two: Sending Data to Multiple Vendors

    I’ve yet to come across a large organization that relies on just one observability tool. Different parts of the organization have unique needs and perspectives when it comes to data, and it’s rare that a tool designed for SREs satisfies all use cases developers have and vice versa.

    At the same time, it’s not practical to instrument code specifically for each vendor. Doing so would lead to performance issues and other problems, such as double-tracing. For example, if multiple vendors modify an outbound HTTP request to add context, which one takes precedence?

    OpenTelemetry addresses this by offering a unified way to instrument code and multiple integration points where vendors can capture the collected data and send it to their respective backends. As a result, a single application can send data to multiple observability backends with a minimal added performance impact.

    Good Reason Three: Gaining Insight Into Third-Party Libraries and Services

    In an era where many applications depend on third-party libraries and services, gaining insight into all your application dependencies has become increasingly difficult. It’s also challenging for observability vendors to keep up with every new library and service that’s released.

    Traditionally, vendors would try to reverse-engineer these services and apply the aforementioned monkey patching to add in their code. However, since every vendor must do this for each service, it ends up being a lot of redundant work. A key goal of the OpenTelemetry project is to make it easy for infrastructure and library vendors to add instrumentation hooks to their code. This way, their offerings can be monitored without the need for reverse engineering. The observability community is gradually making progress on this goal. One successful example is Istio, which emits OpenTelemetry data that can be easily consumed by observability backends to understand what’s happening inside the service mesh.

    Unsurprisingly, cloud vendors like AWS, Microsoft and Google are among the founding members of the OpenTelemetry project. They share a strong interest in defining a format that allows observability vendors to seamlessly consume data from cloud services. OpenTelemetry Protocol, or OTLP, is set to become the standard for emitting observability data from these services that would otherwise be black boxes.

    Wrapping it Up

    “If all you have is a hammer, every problem looks like a nail.” This statement holds especially true when selecting tools in the tech industry. While there are many good reasons to adopt OpenTelemetry, it’s crucial to approach technology choices pragmatically.

    Avoid jumping on the bandwagon for its own sake; instead, carefully evaluate your use case and assess the available tools. OpenTelemetry is undoubtedly here to stay, and if you’re new to the topic, taking a closer look at the project and adding it to your toolbelt makes perfect sense.

    And don’t forget: OpenTelemetry is open source. Your contribution is always appreciated.