devopsy.kinsta.cloud

Tag: observability

Chronosphere Acquires Calyptia to Process Log Data More Efficiently

Chronosphere today announced it has acquired Calyptia to add additional pipeline management capabilities to its observability platform.

Calyptia created Fluent Bit and Fluentd, a set of open source log processors that are now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

Chronosphere CEO Martin Mao said these log processors will be embedded into the company’s namesake observability platform to enable the routing, transformation and optimization of log data at scale. That issue has become problematic because cloud-native applications based on microservices create exponentially more telemetry data than legacy monolithic applications. If it becomes too costly to analyze that data, then DevOps teams will begin to limit the number of applications they will observe, noted Mao.

The Calyptia platform achieves that goal using filters at the data source to reduce the volume of data that might otherwise be shared by 30% or more.

The shared data can then be analyzed in real time rather than waiting for it to be stored and indexed. In addition, DevOps teams can enrich that data in flight to include additional context.

Calyptia cofounder Anurag Gupta added that the primary mission for the company has always been to make log data more accessible to developers as they build and deploy applications.

The acquisition of Calyptia comes on the heels of a log management alliance between Chronosphere and CrowdStrike through which a log storage and virtualization capability, dubbed Logs powered by CrowdStrike, has been added to the observability platform. The overall goal is to simplify the management and analysis of logs via a single click of a graphical tool or within the context of a larger DevOps workflow based on a command line interface (CLI) without requiring data engineering expertise, said Mao.

It’s not clear how fully DevOps teams have embraced observability, but it’s apparent the legacy monitoring tools that collect a set of predefined metrics are not able to keep pace with the complexity of cloud-native application environments. An observability platform makes it simpler to query log data and discover the root cause of an issue—hopefully before a catastrophic event occurs. The challenge, of course, is providing that capability without incurring massive increased storage costs for log data.

Despite those concerns, the need for observability is becoming a more pressing issue with each passing day. Organizations have never been more dependent on applications to drive revenue. A degradation in performance is usually an early indicator of a disruption that often has a direct impact on revenue and profitability.

There is, of course, already no shortage of observability platforms. The issue most DevOps teams encounter is finding the funding needed to employ one at a time when organizations have become much more sensitive to IT costs. There may be an opportunity to rationalize some monitoring tools to help pay for the cost of a more modern observability platform, but regardless of approach, an ounce of prevention is always going to be worth a pound of proverbial cure.

January 22, 2024
A New Year’s Resolution for Observability in 2024

As we usher in the dawn of 2024, it’s only fitting that we set ambitious resolutions for the tech landscape. One area that has long been at the forefront of innovation and evolution is observability. As organizations continue to embrace digital transformation, the need for robust, scalable and cost-effective observability solutions becomes paramount. In 2024, the new year’s resolution needed for the observability field is crystal clear: It is the year for frictionless integration and scalability, reduced costs and predictability in pricing, granular visibility and insights and compatibility with open source standards.

Frictionless Integration and Scalability

In 2024, observability should seamlessly integrate into existing infrastructures, eliminating the complexities that often accompany implementation. The ideal observability solution should be like a chameleon, effortlessly adapting to diverse tech stacks, cloud environments and hybrid architectures. Scalability, a perennial challenge, should no longer be a stumbling block. When an organization undergoes significant expansion, observability tools must scale proportionally without compromising performance.

Predictable Pricing and Reduced Costs

Gone should be the days of $65 million observability bills, obscure pricing models and hidden fees. In 2024, organizations deserve clarity and predictability in observability costs. A transparent pricing structure ensures businesses can plan and allocate resources more efficiently. Predictable pricing also fosters a trustful relationship between observability service providers and their clients, allowing organizations to focus on innovation without the constant worry of unforeseen financial burdens.

Cost efficiency remains a perennial concern for organizations. In 2024, observability tools should not only provide enhanced functionalities but also reduce overall costs. Research has shown that observability has become an incredible burden for companies monitoring application performance: A full 9% of IT budgets are being spent on observability, an average cost of $1.72 million. The price is so high that over 70% of teams do not even opt to have an APM tier in place. But when companies do not have an observability tool, the price is high as well. On average, companies suffer two high-impact outages per month with an average MTTR of 30 minutes or more. These outages result in more than $7 million in losses per year due to hidden costs such as troubleshooting resources, customers dropping out, damaged reputation and more. In 2024, observability will become an asset rather than a financial burden.

100% Visibility and Granular Insights

True observability means having a comprehensive view of the entire system. In 2024, I anticipate observability tools offering 100% visibility into applications, infrastructure and user experiences. This level of insight empowers organizations to identify and resolve issues swiftly, minimizing downtime and enhancing overall system reliability. Granular insights will not only facilitate root cause analysis but also enable proactive measures to prevent potential problems before they escalate.

Compatibility With Open Source Standards

Observability data locked in at some third party will not be tolerated in 2024. Every organization will be packing some open source components for metrics, logs or other parts of its observability stack. Vendors will have to operate alongside these solutions, supporting open standards like OpenTelemetry and making sure customers own their own data.

As we embark on the journey through 2024, let us commit to realizing this resolution. The future demands a monitoring infrastructure that is not just powerful but also user-friendly, cost-effective and future-proof. By achieving these goals, organizations can fortify their digital ecosystems, ensuring resilience, innovation and sustained success in an ever-changing technological landscape. Cheers to a year of observability excellence.

January 12, 2024
How Supergraphs Help Consolidate Data Access
GraphQL, a query language for APIs, has become more of a common tool for new web-based development. GraphQL can help streamline data access by providing all the resources a developer requires in a single request. Compared to the traditional REST approach to web-based data access, GraphQL has a more standardized and traversable schema and arguably better developer experience.

But GraphQL is not just great for improving developer usability for a single data model. If used to aggregate various data sources and APIs into a unified graph, GraphQL could become ‘one schema to rule them all‘ for enterprises. Some refer to this architecture as a supergraph.

I recently caught up with Tanmai Gopal, CEO of Hasura, to explore the concept of a supergraph and see how it can help make data more accessible for enterprises. Below, we’ll consider the expansion of the supergraph concept and what the benefits are of deploying such a strategy.

Understanding the Need for Supergraphs

In large software architectures, product teams need access to various data sources from different domains. “There is a need for integrative experiences, which are becoming more and more important for the end user,” explained Gopal. For instance, an e-commerce application might be composed of product catalog data, login functionality, payment data, shipping information and other necessary information.

However, there are hurdles in the way of pulling in data from these multiple domains. Different domains are often built as microservices, supported by a dedicated team and using a bespoke developer portal and API schema to externalize the services, said Gopal. The friction involved in API integration and the lack of a common registry of services could slow down development.

Not only do microservices that expose discrete endpoints have issues, but the lines between internal and external are blurring, necessitating a zero-trust approach. And if we begin to consider data as a product, explained Gopal, we need to access multiple domains in a secure, consistent and scalable way.

Benefits of Adopting GraphQL and Supergraph

A supergraph can solve some of these difficulties, explained Gopal. By exposing graphs from different domain services, you can automatically unify data into a supergraph, which could be accessed from various teams. And the great thing about using GraphQL is that the act of documenting is the act of building the API, meaning that the description and schema are more standardized.

By composing disparate services into a centralized registry, you could work across domains in a systematic but federated way, said Gopal. Implementing a layer here could also be a good area to enforce role-based identity or authentication.

Andrew Carlson, principal field architect at Apollo GraphQL, similarly proposed using GraphQL to centralize data access control. “When used as a layer to aggregate and orchestrate existing APIs,” Carlson said. “It’s an ideal location in our architecture to centralize access control and authorization down to the field level, providing field-level observability into which clients request what data.”

Gopal shared a handful of other specific advantages for using GraphQL in this context:
- Helps you write less code: Since GraphQL enables you to fetch whatever field you want, aligning on GraphQL as a common interface could reduce the number of requests required to integrate data. Also, certain developer experience perks, such as auto-complete, could enhance schema discoverability and have exponential benefits when sources are intertwined in a combined graph.
- Increases agility for new development: The need to quickly understand the overall schema and what it looks like is becoming especially relevant, especially for AI applications, said Gopal. There’s a massive agility boost when using a supergraph schema, he added, such as enabling software teams to fetch the context they require to quickly put things into motion.
- Improves data and service discoverability: Software teams are facing tooling sprawl, and the number of APIs in use within an organization keeps increasing. A supergraph can enable a traversable data mesh, greatly aiding discoverability and, in turn, the ability to innovate. “The benefit from the ability to understand and use data from different domains is massive,” said Gopal.
- Unifies teams from various domains: Another potential benefit of the supergraph concept is improved inter-departmental collaboration. Helping teams work together could enhance reporting and reduce risk. This new shared operating model helps to clean things up when updating code and the schema, said Gopal.
Next Step: Protocol Agnosticism

The idea of wrapping disparate data sources and formats in a standard, unified layer is an alluring concept. “The agility impact is massive,” said Gopal. Especially, he adds, if you have the ability to automate the creation of a supergraph around various styles of legacy databases.

However, investment into a supergraph can be time-intensive. The benefits are unlocked down the line, but you will need an internal champion to initially advocate for it and bring it to fruition, he cautions. Furthermore, enterprises may face hesitancy in going all-in on GraphQL due to past investments in other technology types.

While GraphQL is becoming more popular, various API styles are still commonly applied and will continue to be supported for some time. For instance, a 2023 study by API management and testing company Postman found that GraphQL use had eclipsed the mainstay API format of SOAP. Yet, REST is still the dominant API style, used by 86% of respondents, followed by Webhooks (36%), GraphQL (29%) and SOAP (26%).

As such, the next step is for the supergraph to be agnostic to GraphQL, said Gopal. Instead, it should be able to work with various protocols since most enterprises are utilizing multiple styles.
January 11, 2024
Broadcom Survey Surfaces Raft of IT Automation Challenges

A survey of 501 IT leaders published today by Broadcom suggested that islands of automation that operate independently of each other are major contributing factors to organizations failing to meet service level agreements (SLAs).

The survey found that 98% of respondents encountered at least one automation issue, with 61% noting that SLA issues are a monthly occurrence. More than a third (35%) said these issues occur weekly. A full 88% indicated SLA breaches directly impacted customers in a significant way.

The survey also found nearly three-quarters (74%) lacked visibility across their data center and cloud environments, with an equal number lacking end-to-end visibility of their automated business processes. Only 28% possessed tools that could predict automation issues, and more than two-thirds (68%) were unable to model or predict whether a new or updated automation job would actually meet SLAs. A full 69% said they are unable to properly size automation workload resources, with 61% having no way to simulate automation workloads before implementation, the survey found.

Only slightly more than half (55%) of respondents, however, could provide SLAs and key metrics in a format that is appropriate for business stakeholders.

Aline Gerew, head of automation for the Agile Operations Division at Broadcom, said that while it’s routine for IT teams to leverage a range of automation tools and platforms to manage IT, it’s clear they are challenged to manage those platforms in a comprehensive fashion. As a result, whenever there is an issue with one tool or platform, it creates a bottleneck that impacts customer experience in a way that ultimately adversely impacts revenue, she noted.

In fact, many IT teams are now starting to identify automation architects within their IT teams to enable DevOps teams in a more holistic manner, added Gerew.

Overall, the survey found that 80% of respondents are using three or more automation platforms, with just over a quarter (27%) using six or more.

Nearly three quarters (74%) are also relying on three or more observability solutions, with just under a quarter (24%) using six or more. Just under three-quarters of respondents noted they are consequently subjected to alert storms that ultimately serve to make it more challenging to resolve issues. More than half (56%) of respondents are blind to the downstream effects of an initial alert and, as a result, do not inform response teams of which systems, jobs, applications and business processes are affected.

In some ways, IT automation has become too much of a good thing. Every provider of an IT platform makes automation tools available, but as they are invoked, each one creates yet another island that IT teams are expected to manage in isolation. However, all it takes for an SLA not to be met is for an issue to arise that IT teams can’t easily determine the root cause of because one or more automation platforms obscure the underlying issue in the absence of any standardization, noted Gerew.

Given the complexity of modern IT environments, automation is now a requirement. The issue, as always, is finding a way to apply it all in the most transparent way possible.

January 9, 2024
Nerd/Noir Allies With DX to Improve Developer Productivity

Nerd/Noir today announced it has partnered with DX, a provider of a platform for measuring developer experience, to identify areas where organizations can improve developer experiences in less than six weeks.

The consulting firm plans to leverage the DevEx 360 platform created by DX to survey developers before recommending a plan.

Nerd/Noir CEO David Laribee said that approach will make it simpler for DevOps teams to pinpoint where changes to workflows will have the most impact on 25 different key performance indicators (KPIs).

In addition, DevOps teams will be able to compare their results against more than 120,000 anonymized data samples collected across multiple industries and sectors.

There’s a lot of focus on developer productivity these days, largely because more organizations realize the extent to which they have become dependent on software to drive revenue and improve profitability. The challenge is organizations can’t afford to hire an unlimited number of developers, so many are now embracing platform engineering as a methodology for managing DevOps workflows at scale.

Most DevOps teams track metrics, but they are generally focused on software engineering issues rather than actual developer productivity. Measuring developer productivity tends to be a less exact science. Once upon a time, an organization might have tracked the lines of code being written, but the total number of lines of code written may not be especially good. In more recent times, metrics have been developed that, among other things, keep track of how much time developers spend actually coding versus maintaining their software development environment.

The issue that organizations are contending with is that more responsibility for managing everything from development environments to provisioning and securing cloud infrastructure has shifted, and the cognitive load that each developer is required to carry has increased. Every minute developers spend on these tasks, however, is, in theory, one less minute developers are potentially spending writing the business logic that drives an application.

Of course, there’s no way developers are going to write business logic for eight hours straight every day, so there are maintenance windows. However, application development is as much art as it is science. No one knows for sure when inspiration might strike, but if developers are busy maintaining code that they created to provision infrastructure, the less probable it becomes for some flash of brilliant insight into how an application might be made better is likely to occur.

DevOps teams need to strike a delicate balance. Developers like to be able to experiment with new tools and platforms, so any approach to platform engineering that smacks of totalitarianism will be resisted, and IT leaders need to proceed with caution. If talented developers start heading for the exits, IT leaders will find it difficult to replace them because great developers, even in an economic downturn, are still difficult to hire and retain. The challenge and the opportunity are to convince developers to embrace changes to workflows based on their input versus imposing one that benefits the DevOps teams more than the developers those teams are supposed to support.

January 9, 2024
Senser Extends AIOps Reach to Manage SLOs and SLAs

Senser is extending the reach of its artificial intelligence for IT operations (AIOps) platform to now include an ability to define and maintain service level agreements (SLAs) and service level objectives (SLOs).

SLOs are a set of internal performance goals that require access to telemetry data from service level indicators (SLIs), while an SLA is a formal commitment to maintaining specific levels of service.

Senser CEO Amir Krayden said the company’s AIOps platform collects data from SLIs and then applies predictive AI models to enable IT teams to achieve SLOs and SLAs. The Senser AIOps platform leverages extended Berkeley Packet Filter (eBPF) and graph technology to gain visibility into the entire IT environment versus requiring IT teams to deploy agent software. Machine learning algorithms are then used to aggregate and analyze that data to define thresholds for predicting performance in addition to recommending benchmarks for tracking SLOs and SLAs.

That approach provides a single source of truth for identifying the actual level of service being provided based on a topology of the infrastructure, network, applications and application programming interfaces (APIs) that makes it possible to identify the root cause of issues and the potential impact of an outage for degradation of performance.

IT teams have been attempting to achieve and maintain SLAs and SLOs for decades, but given all the dependencies that exist in a distributed computing environment, it’s difficult to achieve that goal. Senser is making a case for applying AI within the context of a platform for automating the management of IT to define and maintain SLOs and SLAs to make it possible to consistently manage SLAs and SLOs in a way that reduces the level of cognitive load that would otherwise be required. Senser is also working toward adding generative AI capabilities to provide summaries that explain what IT events have occurred.

Collectively, the goal is to provide IT teams with a more efficient holistic approach to monitoring and observability that legacy platforms are not going to be able to achieve and maintain, said Krayden.

At the core of that capability is eBPF, a technology that allows software to run within a sandbox in the Linux microkernel. That capability enables networking, storage and observability software to scale at much higher levels of throughput because they are no longer running in user space. That’s especially critical for any application that needs to dynamically process massive amounts of data in near-real-time.

As the number of organizations running the latest versions of Linux continues to increase, more hands-on experience with eBPF will be gained. IT teams may not need to concern themselves with what is occurring in the microkernel of the operating systems, but they do need to understand how eBPF ultimately reduces the total cost of running IT at scale.

Ultimately, the goal is to reduce the current level of complexity that today makes effectively managing highly distributed computing environments all but impossible for IT teams to manually maintain in an era where the pace at which applications are being built and deployed only continues to accelerate.

December 21, 2023
KubeCon 2023: Elevating Observability

Speaker 1: This is Techstrong TV.

Alan Shimel: Hey everyone. We’re live back here at KubeCon. I hope you’ve been enjoying our coverage today, Mitchell Ashley, our CTO and a GM of Techstrong Research. And I have been kind of splitting the duties. Just we’re that booked with people. I don’t know, there’s over 10,000 people here. I’m going to guess there’s 12 to 15,000 people.

Alex Burkhardt: Yeah, there’s a lot of people.

Alan Shimel: … At the event. Maybe more I’ll know more. I’ll find out tonight. But anyway, let me introduce you to our next guest. His name is Alex Burkart. Alex is with Observe Inc. Alex, welcome to Techstrong TV. Thanks for being here with us at KubeCon.

Alex Burkhardt: Yeah, thanks Alan. I appreciate you guys taking some time to talk to me today.

Alan Shimel: Absolutely. So let’s start with this. Let’s start with Alex.

Alex Burkhardt: Yeah.

Alan Shimel: What do you do at Observe?

Alex Burkhardt: Yeah, so at Observe I run our technical customer facing team. We call it field engineering. And we work with customers from the first interaction when they’re in the sales cycle to figure out if observes a great solution for them, all the way to success and renewal and helping them realize value from their investment in our partnership.

Alan Shimel: Okay. I mean, so you’re at that role, but let’s tell a little bit about more your personal journey.

Alex Burkhardt: Oh yeah. So I started as a computer scientist actually in the Navy.

Alan Shimel: Really?

Alex Burkhardt: Yeah. I went to the Naval Academy, got a computer science degree, and they said, that’s great, go drive some ships.

Alan Shimel: Such is life in the Navy, what do you expect?

Alex Burkhardt: And then my journey brought me to Silicon Valley after I did some cybersecurity work for the Navy. And what I love more importantly than anything else is helping people find a way to use technology to solve a problem that makes their life better.

Alan Shimel: Excellent. Now, when did you come to Observe?

Alex Burkhardt: I’ve been at Observe for more than three years now. Yeah, I was employee 28. I think if my numbers are right, pre customers, in fact.

Alan Shimel: Really early.

Alex Burkhardt: I think when I started, we signed our first customer.

Alan Shimel: Good for, good for you. Good for them. So we don’t have to assume, I know for a fact there are people out here who don’t know, Observe from, have never observed Observe. Tell them. How would you describe what Observe is and does to our audience?

Alex Burkhardt: That’s a great question, Alan. So at Observe, what we realized is observability is a data problem. So we started by building a data company. What we wanted to figure out is can we bring a solution to market that allows customers to send any data they’d like to correlate it and interact with it in a way that allows them to get more value out of it more quickly. And what we’ve done is we’ve brought a solution to market that unifies the observability tooling into one platform that allows you to send logs, metrics, traces, and any other type of data structure or unstructured that you’d like to keep in your observable universe into one tool, one data store.

Alan Shimel: Love it. And before we jump into news and stuff, for people who want to just say, hey, I’d like to just see what their website is, what’s the URL?

Alex Burkhardt: Yeah, our website is www.observeinc.com.

Alan Shimel: So it’s observeinc.com.

Alex Burkhardt: Observeinc.com. We are Observe Inc technically. So observeinc.com.

Alan Shimel: Makes sense to me. All right, we’ve got all that out of the way, Alex, let’s jump in. Now, you guys recently announced some new product feature releases. Why don’t we jump in right there and see what’s that about?

Alex Burkhardt: Our last release was called Hubble, and Hubble is significant for a couple of reasons. One, naming after the Telescope, one of the things that we want to do is help people elevate their observability. And that came in two specific forms.

Alan Shimel: I get it. I just got to ask, is the next release called Web?

Alex Burkhardt: We have considered other satellites, and in fact, we had Erica on our launch. She was actually a NASA scientist.

Alan Shimel: Very cool. All right.

Alex Burkhardt: One of the things that we know is very important to our customers is if we bring a solution to market and it’s very, very powerful, is it easy to use and are we taking advantage of the latest and greatest technology that’s emerging in the market? On the ease of use side, one of the things that this Hubble launch and we did, we call it a launch included, was the recognition that when we introduced our logs in our metrics and our tracing explorer, we’ve seen an enormous increase in users engaging in the platform. And we supercharge that by adding one additional layer, which is how we leverage GPT or AI in our platform. And that happens in four specific locations. We have a chat bot that allows you to get guidance on how to use the tool, will actually allow the AI to write any of our query language for you if you’d like to do that. We call that copilot.

We also have two cool functionalities in terms of making your data more actionable. You can click on a field and observe and ask Ollie or Ollie GPT specifically to explain what an error means and it will actually return an answer that helps you stop having to copy and paste into Google and do outside investigations. The last thing that’s most interesting is we leverage GPT to allow users to separate out their data and do those hard parsing and Reg X challenges that no one likes writing Reg Xs and anyone that has written them knows that regx101.com is their friend, and we wanted to kind of bring that into the product and make that easy for customers to use.

Alan Shimel: Very cool. Very cool. Now this was released not here at the show, it was released earlier in the month.

Alex Burkhardt: October 4th.

Alan Shimel: What’s been the feedback?

Alex Burkhardt: Well, we’ve more than doubled our monthly active users since we’ve released that.

Alan Shimel: Really?

Alex Burkhardt: Yeah.

Alan Shimel: Since October 4th?

Alex Burkhardt: Technically since late September. But yes.

Alan Shimel: Wow.

Alex Burkhardt: It’s really simple. We listen to our customers and we really care about what they say and how they experience ourselves on our platform. And when we say we’re focusing on ease of use, we mean it.

Alan Shimel: So the entrepreneur in me just has to ask you.

Alex Burkhardt: Okay.

Alan Shimel: What were you waiting for?

Alex Burkhardt: It’s very easy to make very quick, bad decisions. It’s a lot harder to take a data-driven approach to how to build a product better. And when we approach any problem at Observe, and it’s built into our culture as we first of all figure out this is an important problem, how can I measure the impact I can make on this problem? And that’s how we measured usability. Now for AI, we kind of knew that it was going to be something special and it could help. We’re still figuring out exactly how to measure the impact there, but every decision we make at Observe is grounded in what’s best for our customers and how do we help make the market better.

Alan Shimel: So how much of this phenomenal growth in such a short period of time do you think is just attributed to riding the AI wave, if you will?

Alex Burkhardt: That increase in active user growth actually wasn’t tied to the AI release. It was more to the ease of use functionality that we brought. And I think that’s important for any product, but more specifically, one as powerful as ours. When you’re solving big data problems, people get exposed to big data. And one of the things we thought very deeply about is how do we take the experience that a user has to have to get to the outcome they want and narrow that down and give them an obvious path to go through so that they don’t have to deal with all of the problems if they just want to come in and check their logs and see what’s not working. Sometimes it’s the simplest question of, hey, is it easy to do this in the tool? And when we decided ourselves that it wasn’t easy enough, then we started making some really serious changes and that was backed by every single one of our customers agreeing with us back in the active user growth.

Alan Shimel: Sure. That’s fine. That’s phenomenal, man. Alex, I can’t look for those out here. This is impressive. Let’s talk a little bit about KubeCon here now. I mean, it’s only day one of the expo floor open and everything. What are you hearing from the crowd, from the attendees?

Alex Burkhardt: Yeah, that’s a great question. This is my second KubeCon. I was new to it last year when we were in Detroit, and I really love this conference, Alan.

Alan Shimel: Yeah, it’s great. It’s a great community.

Alex Burkhardt: It’s a great community. It’s a great conference. And frankly, when people come up to our booth and want to talk to us, they want to talk about what we do. There’s not a lot of tire kickers. Everyone that’s here seems to be operators, people that are experienced and steepen the problem. And I love whenever someone comes up and talks to us. I think the other thing I’ve seen in this particular venue specifically, the room is packed. It’s crowded, so crowded that I think that we’re going to have to have a bigger footprint next year.

Alan Shimel: Observers or the whole?

Alex Burkhardt: Both.

Alan Shimel: Yeah. Well, the problem is there’s a pretty big convention center. I mean, short of breaking into sort of a two different exhibit halls, which RSA used to do that. Now it’s one big continuous one. It’s hard. The other thing I will tell you, so we do the North American and the European. So in Amsterdam in the spring, they actually said, look, I think it was 12,000 was the number. At 12,000 they stopped selling tickets. They did a waiting list and a lot of people didn’t get to the show. They were on the waiting list. As anti-business as that sounds, I wouldn’t be surprised if they did something similar.

Alex Burkhardt: Yeah, that makes sense. I mean, the quality of conversations and the quality of attendee keep the quality super high. And you asked about some of the conversations we’re having when people come by. It’s a variety of things and a couple of themes that I see. I’m curious if other people that have come on the show have said the same, but people continue to talk to us about saving money. They want to do more with less.

Alan Shimel: That’s the big thing today.

Alex Burkhardt: [inaudible 00:09:23].

And I think they also want to figure out, do we need all of the tools? Is there a way we can both get some hard dollar savings and some soft dollar savings. By compressing some tooling and aligning teams we’re also seeing a lot of push and the DevOps teams are getting more involved in security conversations as well. Which I guess-

Alan Shimel: No, no, that there’s no doubt.

Alex Burkhardt: As a former security practitioner, if you had asked me five years ago if I thought these two markets would converge, I would not have said yes. Now I think it’s inevitable.

Alan Shimel: I think it’s already happened, right? So there’s a lot of convergence going on. There’s security with DevOps into what we call DevSecOps. But there’s also, there was a time, and I’ve been going to these conferences now before Covid, so I don’t know, four or five years. There was a time where cloud native was very developer-centric, not as much centric, but it was very distinct from DevOps. There was DevOps and there was cloud native and then there was security. So you look at a company like ours, right? We have devopsy.kinsta.cloud, cloud native now, security boulevard.

And I always knew there was an inherent overlap between them, but we’ve seen much more than an overlap now. We’ve seen a conversion around this with security being a primary concern of people. And you don’t do cloud native today without doing DevOps.

Alex Burkhardt: Agreed.

Alan Shimel: And everything getting pushed out today, especially Greenfield, not Brownfield stuff. It’s predominantly probably 85% or better cloud native.

Alex Burkhardt: Yeah, I think that’s another thing I’ve seen too as well. Many, many customers are adopting the cloud native terminology, setting goals in those languages and words.

Alan Shimel: Now, multi-threaded, no doubt. Container based, the whole cloud native thing. And now what I think we’re starting to see and we see, look, you’re naming your company’s Observe. It’s not just the architecture and development being cloud native, but the cloud native aspect of the operations of ops. Forget dev, forget dev ops, ops.

Alex Burkhardt: It even touches revenue. So one of the things that we see very often is do the nature of how our platform can process data. We take technical telemetry from a customer’s application. We also take business data. Sometimes customers ask us, how can you Observe, help us give our customers a better experience? Well, one of the ways is we can tell you when a particular customer is hurting, when a customer has a challenge or help you reframe all these very deep technical jargon and things that are happening into the context of how does this impact the customer’s experience? How am I able to get proactive? Is what I’m doing in my infrastructure plan having a net positive impact on my customers, on my users?

And those are questions that previously companies I don’t think they thought they could answer. And when they start working with us, they start seeing a path to that. And I love that. We have some customers even completely away from the DevOps side, that send us shipping data just because they want to know, can I make a support dashboard for my low level support employees so that they can be more efficient with their time and our data? It’s such a cool convergence. I agree with you. I love that word convergence.

Alan Shimel: It really is. Let’s talk a little bit about observability though too. So over the last two years, so there was a KubeCon in LA during around Covid time that was a little dicey. Then there was the one in Europe, then got canceled. Then there was Detroit, then Amsterdam here. We’ve really seen the observability market just explode, right?

Alex Burkhardt: Yeah.

Alan Shimel: Prometheus. What’s the other, I’m drawing a blank. The other big observability open source. OpenTelemetry.

Alex Burkhardt: OpenTelemetry. Yes. Everyone-

Alan Shimel: I think of the second and third biggest projects of the gazillion projects in CNCF now. Right?

Alex Burkhardt: That’s such a good point. I remember when OpenTelemetry was OpenTracing and it was still kind of a fractured project. If you asked us three years ago if we thought that open telemetry would be the way, we weren’t super sure a year into that. So two years ago we were like, yeah, this is the direction. It’s solid.

Alan Shimel: No doubt about it. You look and you can’t see this at home guys because all you’re seeing is here, but there’s a-

Alex Burkhardt: All our beautiful faces.

Alan Shimel: … Big floor out there, but so much of it is based now on this. Really good stuff, really good stuff.

Alex Burkhardt: I think the other thing, observability as a vernacular is kind of emerging even at the higher level in the business. By definition observability is, can I tell something about a system by looking at its outputs? And one of the things as we’re moving closer to cloud native is more and more systems have outputs and that kind of builds on that data problem I mentioned and why we were founded. So I look forward to seeing how this problem space changes in the next year.

Alan Shimel: Oh, I think we’re just, look, when you start, you guys are already doing it with gen AI and stuff. You start adding gen AI to this stuff, we start putting more focus on it. We’re in for an interesting times. What’s the Irish proverb? May you live in interesting times.

Alex Burkhardt: Yes.

Alan Shimel: Definitely interesting times. Anyway, Alex, thanks for joining us here live at KubeCon. This is Observe Inc. So let me again observeinc.com. Check it out. Hey, they’re doing amazing things. This new release, something that gets twice as many new visitors right? In a couple of weeks, a month. It’s got to doing something right.

Alex Burkhardt: Yeah.

Alan Shimel: All right, we’re going to take a break. We’re live at KubeCon in Chicago. I think we have one or two more videos before we wrap for today. We’ll be right back.

December 18, 2023
KubeCon 2023: Telemetry and Data Management

Speaker 1: This is Techstrong TV.

Alan Shimel: Hey everyone. We’re back here at KubeCon and it’s bustling, man. It’s a crazy big floor with a lot of people walking around, so we hope you like this. This is a little different than we usually do. Usually people can’t see the background because the cameras face this way, but we usually have sort of the typical background that you see at some of these other booths. But we wanted to give people kind of a flavor that we’re alive here.

Tucker Callaway: We’re in the middle of it.

Alan Shimel: We’re in the middle of it. So kind of like if you ever watch Sunday night football, they always have that. So that’s what we were shooting for. But anyway, let me introduce you to Tucker Callaway. Tucker is the CEO of a company called Mezmo. Now, some of you may have heard of, I think it was Log DNA.

Tucker Callaway: Before. Yep.

Alan Shimel: And they pivoted. Was it about four years ago?

Tucker Callaway: About two years ago now.

Alan Shimel: Two years ago.

Tucker Callaway: I’ve been here for four, but big change was two years ago, renamed.

Alan Shimel: Renamed to be Mezmo and a slightly different mission, let’s call it. And then I had a chance to catch up with Tucker at RSA last, I guess it was last April.

Tucker Callaway: April. Yeah.

Alan Shimel: This year it’s May. So I think it was around then last year. And for those of you who want to catch that interview, it’s actually available on Techstrong TV. If you look up RSA 2023, it’s there. But so Tucker, before we get into that, let’s back up. Mezmo. Not everyone out here. Look, not everyone out here knew Log DNA either, so don’t worry about it.

Tucker Callaway: Fair enough. Yeah.

Alan Shimel: Let’s start there. Let’s talk about the mission.

Tucker Callaway: Yeah, so the mission of Mezmo is essentially to help people with their telemetry data. So we’ve found that there’s really kind of three main problems in the telemetry data space. There’s too much of telemetry data. It’s not in the right format. It’s not in the right location. So that was kind of one of our founding kind of principles of solving.

Alan Shimel: Yeah, I think that.

Tucker Callaway: Solving the problem.

Alan Shimel: Those were three rock solid.

Tucker Callaway: Hard to debate.

Alan Shimel: Hard to debate. That’s the good way of putting that. And also, quite frankly, all three problems are like they compound each other.

Tucker Callaway: They do.

Alan Shimel: So it’s like you have too much data. That’s a big problem, but we have too much data that we don’t know how to categorize, so that exponentially makes that problem worse. So these are-

Tucker Callaway: Compounding problems. No question about it.

Alan Shimel: No doubt about it. So when we were at RSA, you felt like you were kind of on the cusp of a realization, right? You were going to help customers. Why don’t you? Well, you tell the story better than me.

Tucker Callaway: So as we were talking about earlier, we had just launched this new product capability, the telemetry pipeline at RSA. We’ve got a lot of great customers and great experience since then. When we think about the core value of what it provides, it helps people with cost, it helps people get better insights and helps people enforce compliance. What we were surprised by though in the last couple months, is how little people actually understand of their data.

In fact, we’ve put a lot of time and effort as an industry into getting control of our applications and getting control of our infrastructure, but we haven’t really gotten control of our telemetry data yet, and that’s like the big opportunity that we see. But with that, in order to get that value I described, people didn’t need to understand their data better. And so that was a big revelation for us.

And we just launched our data profiling capability that helps people categorize and understand baseline, to do the schema management for their data, which is a really strong foundation for data ops principles applied to telemetry data. Then the next thing we found after we solved that problem was now that people understood their data and they were able to optimize it because they understood it. They also realized that the context and telemetry data changes rapidly. So for example, you don’t want to store all this data because you don’t need it until you need it, and then you really need it.

Alan Shimel: I’ll take a little, they do want to store the data because people are data hoarders.

Tucker Callaway: The data hoarder. Yeah.

Alan Shimel: It’s when they get hit with the bill for storing all that data, they say, “Wait a second, I don’t need all this data. Not at that price.”

Tucker Callaway: Something has to give somewhere.

Alan Shimel: Exactly.

Tucker Callaway: And so what we realized though was if you take the bookend of this value, we think of as understand, optimize and respond, we talked about understand and optimize the response stage was a really important one for us. So when new context comes in, the pipeline can change its behavior and change its optimizations to account for the current flow of data and the current needs. So if you’re sampling down to 20% and you see an event, you see performance, you can rehydrate, reprocess, dynamically change the responsiveness of the pipeline and get all the data to the right location for the right people, which helps address the cost problem, but also gives you the full visibility that you need.

Alan Shimel: If one was going to do a business school case study on this, someone who’s not from this industry would say, “Hey, just the fact that you discovered that most of these organizations don’t even know how much data or what data they have would be enough to build a company around.”

Tucker Callaway: Yeah, I’d like to think so.

Alan Shimel: No, it’s logical. It makes sense, but it’s not, it’s the threshold. Because I think you’d hit the nail on the head here, Tucker, which is once they recognize that, now there’s a whole new vista of realization for them of, “Holy mackerel, I got all this data, now I want to start understanding it. Wow. I’m seeing stuff I never thought I’d see. Maybe I intuitively thought I had, but I got the goods now.”

Tucker Callaway: We like to think of that as being mesmerizing, right?

Alan Shimel: Okay, I love it. Mesmerized by Mezmo.

Tucker Callaway: I couldn’t not take the shot, but that was just-

Alan Shimel: Yeah, that was your shot.

Tucker Callaway: That’s the derivative of the name.

Alan Shimel: I feel used. I feel used.

Tucker Callaway: I got back into it a little bit.

Alan Shimel: So now you’re mesmerized and now like, “Oh my God, what can I do with this?” And that really becomes the business. And I guess that’s now where you are.

Tucker Callaway: The next step for us is clearly that. Yeah, and I think there’s two plays on that, too. There’s a, “What do I do with it? Help me, tell me what to do. Give me the best practices that from all of your customers, what should I do?” And so we’ll be coming out probably later in the spring with something that actually allows you to simulate how that data might look when optimization’s applied to it so you can start to make those decisions more effectively.

But then the second phase of that, too, is that also won’t be sufficient for certain organizations that want to look at their data as a strategic advantage. And they’re going to be looking for ways to take actions and drive more insights out of that data. And so the way we see that, we see that through recipes. We’ll give you the standard offerings, but then you have the ability to go in and edit as code and go do the things you need to do to go drive more value out of that data. If you’re ready for advanced yoga moves in the data space.

Alan Shimel: Got it, got it, got it. My background is security.

Tucker Callaway: Yep.

Alan Shimel: Started a security company in 2001, and it was right when we were moving from IDS to IPS. So intrusion detection to intrusion prevention. Same thing with vulnerability management. We were moving from just finding vulnerabilities to patching, remediating, not always patching, remediating. Seemed like who wouldn’t want that? No brainer, right? Just don’t tell me I’m getting attacked. Block the thing. By the time you tell me, the attack went through. Don’t just tell me you found a vulnerability, make sure I’m not getting exploited. But a funny thing that I learned in that was a lot of people say, “Go slow here. I can’t afford, because sometimes the vulnerability or the attack or the data that you want me to act on can affect something that I consider more valuable.”

Tucker Callaway: Yeah, fair enough.

Alan Shimel: And I like to go slow when it comes to letting a program or an application or a product, just do things. Just give me my menu and I’ll decide what I want to do. Now, of course, that was before the world of AI and ML and a lot of the automations. And quite frankly, the speed of business was probably a little slower then. Do you see a way station where people are going to want that? Just give me best practices and I’ll decide when, how, and where I want to implement it. Or do we move right to, “Hey man, make it happen for me.”

Tucker Callaway: I think there’s a way station, when we think about the data management, telemetry, data management in general, the word that’s the most important to us is trust, right? So it is one thing to optimize the data, but you have to trust that data is constantly being optimized and handled in the right way for all those varying needs you described.

So I do see that the first step is a suggestion with a human in the loop. You want to hit that button still to say “go”, and then as you build trust in that data and you know that the algorithms are working, then you’ll let that go. Probably with some backstops on the side, I’m going to let it go, but I’m [inaudible 00:09:52] just in case. And then you’ll start to optimize it more and more. And I think naturally, almost like a realtime data platform, operating system will start to evolve.

Alan Shimel: So really the key there is trust.

Tucker Callaway: Key is trust.

Alan Shimel: To build trust in the next. So as the CEO, how do you build that trust?

Tucker Callaway: Well, trust is always a tough.

Alan Shimel: I’m interested, you tell me.

Tucker Callaway: Yeah. I think like anything else in life, you deliver trust through consistent delivery. And so that’s where I think that human in the loop step is important. It has to be sustaining. It can’t be a moment in time, it can’t be the very first implementation. It’s something that you earn over time where people can count on you or could count on the data or count on the ways that you treat data to get you to the outcome that you want to go through. So it’s really only through the delivery of outcomes to the customer that they will trust the systems and all those things.

Alan Shimel: Makes sense?

Tucker Callaway: Yeah.

Alan Shimel: All right. Got to do a little housekeeping.

Tucker Callaway: Okay, let’s do it.

Alan Shimel: People who want to know more about Mezmo.

Tucker Callaway: Come to the website. We’ve got an offer out there to profile anyone’s data for free right now.

Alan Shimel: Do you really?

Tucker Callaway: Yeah. Come to mezmo.com or stop by our booth.

Alan Shimel: M-E-Z-M-O.com.

Tucker Callaway: M-E-Z-M-O.com

Alan Shimel: Out there in TV land or if you hear a coupon.

Tucker Callaway: Come by the booth. I forget the number, but it’s over there.

Alan Shimel: You know what, if you look on the, I don’t have it. If you look on the back of your ID thing, you could actually look stuff up like that.

Tucker Callaway: Yeah, it’s back there. If you connected to the wifi here at KubeCon, it’s Mezmo Data.

Alan Shimel: I’ve noticed that.

Tucker Callaway: Is the wifi.

Alan Shimel: I was like, “Wow, what a cool thing.” Yeah.

Tucker Callaway: You have see us. Come see us. Profile your data will give you insight and understanding into that and talk to you about the next steps.

Alan Shimel: All right, check it out at Mezmo. Tucker, thanks.

Tucker Callaway: Thanks for having me.

Alan Shimel: Actually, we’ll see you. Are you going to be at AWS?

Tucker Callaway: Yeah, we’ll be there. Yeah, I’ll be there. Yeah.

Alan Shimel: We’ll be doing videos, not on the show floor. It’ll be a little quieter in a private suite. And then of course you’ll be at RSA.

Tucker Callaway: We’ll be at RSA.

Alan Shimel: We’ll be there as well.

Tucker Callaway: We might not be on the show floor at AWS just because of-

Alan Shimel: It’s a little crazy there isn’t it? A little pricey.

Tucker Callaway: Yeah.

Alan Shimel: Yeah, I hear you. That’s why we’ll be at the wind in the studio.

Tucker Callaway: I’ll see you there.

Alan Shimel: Come on up. I’ll have coffee.

Tucker Callaway: Sounds good.

Alan Shimel: But RSA will be at broadcast alley and we’ll be doing our DevSecOps thing on Monday. And the usual kind of RSA craziness.

Tucker Callaway: Looking forward to it.

Alan Shimel: Absolutely.

Tucker Callaway: Okay.

Alan Shimel: Check out Mezmo.com.

Tucker Callaway: Thanks everybody.

Alan Shimel: Tucker Callaway here on Tech Drunk TV. We’re going to take a break. I guess it’s almost lunchtime here, but it’s still busy.

Tucker Callaway: Near enough.

Alan Shimel: Stay tuned. We’re here all day. We’ll be back. Bye-Bye.

December 18, 2023
Data Observability and its Importance: Everything you Need to Know

In today’s tech-driven world, companies rely on almost incomprehensible amounts of data to dictate their business operations and decisions. Enterprises are focused on building multiple data repositories and data pipelines to process, store, manage and utilize data from various sources. Given the ever-growing size and complexity of the enterprise data environment, providing the accuracy and completeness of data is becoming increasingly challenging. So how can you understand how the data is performing across the entire IT infrastructure, all at once? The answer is data observability.

What is Data Observability?

Data observability is a process that aims to alert you to your data’s reliability and health while also delivering the insights and analysis required to identify and resolve issues before they affect your entire organization.

Monitoring operations in the database serves as a proactive defense mechanism against potential security threats.
This ensures accurate, complete, secure and valuable data and eliminates data downtime. Let’s dive deeper into the observability infrastructure and explore some of the best data observability platforms that can help you optimize and secure your data operations.

Data Observability Framework

We know that observability has three pillars: Logs, traces and metrics.

However, data observability has five pillars, which, together, provide key insights into the reliability and quality of your data.

Incorporating each pillar can help you craft an effective observability strategy.

1. Recency: Also referred to as “freshness,” recency involves confirming if the data is up to date. It also analyzes if there are unusual temporal gaps in the data tables. This helps prevent timeliness issues in data pipelines.

2. Volume: Volume involves checking if the amount of data intake in the database meets the expected thresholds. This ensures that data sets are complete.

3. Distribution: This measures the quality of data at the field level and confirms if the data values are within expected ranges. Unexpected fluctuations in the distribution patterns are indicative of a data issue.

4. Schema: This involves monitoring and auditing changes to data tables and data organization to look for signs of broken data. Schemas are extremely important as changes in the source data’s structure are often the cause of data downtime.

5. Lineage: Lineage collects metadata and provides a complete picture of your organization’s data landscape, including upstream sources, downstream, and which teams can access the data at which stages. This process helps data teams troubleshoot where breaks occur in the data.

Data Observability Benefits

Data observability provides better visibility into your IT system’s internal state, including its behavior, performance and interactions with other systems. This can be beneficial in the following ways:
● Facilitates Root Cause Analysis: With end-to-end data visibility and monitoring across multi-layered IT infrastructure, data observability enables data teams to quickly spot issues in data sets with less effort. This also increases the chances of identifying new issues regardless of where they originate.

● Faster Mean Time to Detection and Resolution: As data observability monitors a wide range of output, data teams can actively triage and debug their systems. By providing valuable insights into how the data interacts and moves within the IT architecture, data observability helps teams spot problems they didn’t know about, resulting in quicker mean time to detection (MTTD) and mean time to resolution (MTTR).

● Automates Security Management: Data observability not only provides real-time visibility into the security posture but also makes it easy to automate parts of the triage process. This helps detect data health issues or data downtime instantly.

Data Observability Challenges

Depending on their existing IT architecture, organizations may face the following challenges when it comes to data observability:
● Data Silos: Due to the presence of several agents, silos monitoring tools and disparate data sources, it becomes difficult to comprehend interdependencies among applications, digital channels and various clouds.

● Integration with Entire Data Ecosystem: For data observability to work, the tool requires insights into the complete data pipeline and the servers, databases, software and applications involved. However, some organizations may find it challenging to connect all of the systems to a data observability platform.

● Manual Instrumentation and Configuration: Data observability tools aim to standardize telemetry data and logging guidelines to efficiently correlate information. However, as large organizations maintain several data sources (hundreds or even thousands), data from these sources might have varied standards. This requires manual effort to standardize data.

Data Observability vs. Data Governance

In today’s digital-first landscape, where data security is a global concern, there’s a growing focus on data governance. Data governance assists in the formation of policies and procedures required to keep a check on how an organization collects, analyzes, stores, shares and uses its data.

A strong data governance program ensures data availability, integrity, usability and security. It eliminates data integration issues, data silos and poor data quality, thereby resolving the challenges of data observability.
With data sets now scaling with more tables, more data sources and more complexity, data engineers and developers are under pressure to keep up with the data consistency, availability and security requirements. Any amount of downtime can lead to wasted resources and time while deteriorating confidence in decision-making.

Data observability, along with data governance, helps organizations manage data security and quality problems in a streamlined manner.

Data Observability vs. Data Quality

Data quality measures the completeness and accuracy of data sets to determine if they can be used in analytics and operational applications. Data observability, on the other hand, allows organizations to detect and fix issues in the data pipeline efficiently and quickly.

For effective data management, an enterprise must take both into account. Earlier manual checking and cleansing of data sets helped determine data quality. But today, many of these tasks have been automated, thanks to modern data stacks.

As a result, there’s been a shift in focus from data quality to data observability. With data observability, businesses can effectively monitor and troubleshoot their data pipeline. Without it, they are at the risk of relying on incomplete or inaccurate data to make decisions. This can lead to expensive mistakes.

Finding the Right Data Observability Tools

Undoubtedly, data observability is a critical feature of any enterprise that uses data. However, not all data observability tools are equally beneficial to your business. Look out for the following traits when choosing a data observability platform:
● Compatible: The tool must be compatible with your data lakes, databases and cloud storage solutions.

● Autonomous: Autonomous technology responds to stimuli without any human intervention. This is essential in a data observability tool as it helps in the early detection of anomalies and an instant response to alerts.

● Timely: A data observability tool must help you identify errors as early as possible. The best platform constantly monitors data health from the moment the data is added to the ecosystem to the end of its lifecycle. Early detection of errors stops issues before they become critical.

● Sophisticated: The best platforms leverage machine learning (ML) and artificial intelligence (AI) to identify hard-to-find issues.

● Other features: The tool must be able to collect, sample, review and process telemetry data across several data sources. It should serve as a centralized data repository, offer comprehensive data monitoring services and provide data visualization.

Eventually, the right data observability platform depends on your organization’s observability engineering needs and unique IT architecture.

Top Data Observability Tools

Data observability platforms are still a budding product category. The good news is that over a half-dozen data observability companies now offer commercial tools with excellent features. These include the following:

Observability Vendors, Overview, Pros and Cons

Monte Carlo Data is headquartered in San Francisco. Monte Carlo is the creator of the industry’s first end-to-end data observability tool.

● Provides comprehensive data observability capabilities.
● Offers a high level of features, including automated alerting, data catalogs, etc.
● Supports fully automated setup.

● High data volumes can lead to user interface issues.
● Huge amounts of variables restricted by different constraints can lead to computational inefficiency.

Bigeye is an industry-leading data observability platform that allows teams to improve, measure, and communicate quality data clearly and quickly at any scale.

● An easy-to-use interface that facilitates data configuration while ensuring consistency and accuracy.
● Provides powerful API integration capabilities.
● Features a versatile dashboard with real-time tracking and monitoring of data quality metrics by multiple people.

● It can be expensive for smaller organizations.
● The tool might crash at times, requiring performance improvements.

Acceldata offers tools for Hadoop, cloud services, and companies which include end-to-end data reliability, data pipeline monitoring, and multi-layer data observability.

● Provides fully automated data reliability checks.
● Offers a drag-and-drop interface to analyze data pipelines across several layers and platforms.

● Initial setup can be complex and difficult.
● Node addition and node removal require human intervention.

Databand is an IBM company that offers proactive capabilities to identify and resolve data issues in the early stages of the development cycle.

● Provides cross-stack visibility. This means you can get an overview of all the data tasks from start to end.
● Offers standardized DataOps and end-to-end data lineage, ensuring data reliability and accuracy.

● The program requires a significant amount of space, which makes it difficult to install it on the user’s system.
● It requires constant software updates making it much heavier.

Datafold has a unique ability to proactively detect, investigate, and prioritize data quality errors before they affect production.

● Allows you to turn SQL queries into smart alerts, keeping you updated on any issues that may arise.
● Automates regression testing by integrating the CI process through GitLab and GitHub.

● Limited integration options.
● Does not provide any support for data analysis and data science.

Closing Thoughts

Data observability is the backbone of data engineers’ ability to be agile with their products. If you want to modernize your data management practices and improve your data quality, data observability is the way forward. Without it, your data team cannot rely on its tools and infrastructure, as issues can’t be detected efficiently and quickly.

December 12, 2023
5 Reasons to Move Beyond SRE to Observability

Recently, I was asked on a podcast to explain the relationship between site reliability engineering (SRE) and observability and why I believe organizations need to move their focus from the former to the latter. Those are great questions, but to answer them accurately, I first need to point out that many organizations claim to be doing SRE or claim to be doing observability but aren’t.

One big thing that SRE and observability have in common is that they are both often erroneously equated with monitoring.

SRE ≠ Monitoring

Google is credited with pioneering the SRE movement with the publication of the “Site Reliability Engineering” handbook in 2016. As the SRE movement gained momentum, organizations of all sizes pored over the handbook and began applying its principles. However, very few organizations are of the size and sophistication of Google regarding the scale of their IT operations and engineering resources.

So, as these “normal” organizations realized how difficult it was to follow the Google SRE approach in its entirety, they often opted instead to simply apply what they could. For many, the chapter on monitoring became the focus, so much so that monitoring has become synonymous with SRE in far too many organizations today.

Let me be clear: The role of a site reliability engineer is not to monitor alerts. The role of an SRE is to define how the engineering team should take ownership of their service. SREs are responsible for establishing a culture and creating engrained processes that are focused on the quality and reliability of infrastructure.

SREs guide the engineering team on how an application or service should be observed and monitored, including what parameters should be looked at, what alerts they should have, and what routing path the alerts should go through. An SRE instructs on best practices, explores the tools that are being used and determines how to gather reliable data with consistency throughout the organization.

Many organizations we work with say they want to do SRE this way, but they’re not there yet. They are still stuck on monitoring every single metric they can find.

Observability ≠ Monitoring

Similarly, a lot of companies I meet say that they do observability, but what they are doing is just monitoring. They monitor every single CPU of every node of every pod of every machine that is running. They have alerts for some of these, and they may even have a playbook for some of them. This is not how SRE is supposed to work, and it’s certainly not what observability is all about.

More importantly, it’s not scalable as an organization grows to hundreds or thousands of developers and different teams that all share the same IT environment.

Two Viewpoints: Horizontal and Vertical

Now that we’ve established that SRE and observability are alike in that neither should be equated with monitoring, let’s contrast the two. SRE and observability are different in that they have different perspectives on the IT domain.

SREs see the world horizontally. They conceive of their IT domain on a horizontal plane where they can observe the capacity of resources being used. They see the Kubernetes cluster, and they see the cloud utilization. And they care about ensuring that what is deployed into the production environment is secure.

Although they see the pieces they manage on the horizontal infrastructure plane, they don’t see the ultimate impact on the application or customer on the planes “above.”

Observability is looking at the world vertically—that is, seeing how an application is or isn’t meeting customer needs. The whole goal of observability is to know if my application is serving my customers at the service level that I agreed to. With a vertical view from application up to customer needs and business impact, I don’t care how many pods are being run, how many nodes are being deployed or what comprises the cloud infrastructure underneath my application, as long as it works.

I do care about security insofar as I want to avoid introducing security vulnerabilities into the organization. But beyond that, all I need to know is that the application can meet the customer’s expectations and the service level objectives (SLOs) that we have established are being met, as well.

Therein lies the heart of the matter: For a company to make the transition to observability, it needs to build the observability system around establishing and meeting SLOs.

The Challenges of Transitioning to a Vertical Perspective

Companies must move beyond an SRE and DevOps organization to being an observability and SLO organization. Only by doing this will organizations be capable of understanding the trade-offs within their IT environments between cost, security and availability and the ultimate impact those trade-offs will have on business objectives.

Today, making this transition — moving beyond SRE to observability — is difficult for a wide variety of reasons, not the least of which are:

Alert Fatigue. Companies today are monitoring too many things they shouldn’t care about, and as a result, engineers are suffering from alert fatigue. With this, you face the danger of a higher probability of missing the alerts that you do care about and making sure that you address them promptly.

Ideally, observability solutions eliminate the irrelevant and unimportant alerts, add context to alerts that might need investigating, and correlate response data and docs to accelerate and coordinate the appropriate response of on-call responders.

Tool sprawl. Many of the companies we work with use a wide variety of separate tools for observability. Observability requires access to the data all of these tools produce, and problems can persist if the tools operate in data silos.

However, the problem can be solved if there is unified data collection across the toolset, ideally using OpenTelemetry. Cross-tool and cross-stack insights are particularly important for security purposes. Another issue with tool sprawl is that true observability requires that these tools, which each typically address just one aspect of observability, somehow all work together.

Unfortunately, current systems and processes lack the “connective tissue” to offer a consolidated observability solution.

Complexity. As we’ve seen in the results of our annual DevOps Pulse survey, the mean-time-to-resolution (MTTR) is going up instead of down, despite the number of tools that are available and the advancements made by the industry in recent years.

Complexity is a large part of the problem, specifically concerning Kubernetes. In our 2023 DevOps Pulse Report, we found that almost 50% of respondents cited Kubernetes as their main challenge to gaining full observability into their environment.

The adoption of Kubernetes is happening at lightning speed—even faster and higher than the adoption of cloud technology. But it’s not just the adoption of one Kubernetes—there are so many different flavors of Kubernetes, whether it’s Kubernetes running on managed services like AWS EKS or AKS or Kubernetes running on serverless with AWS Fargate. And that’s just to name a few of many, many types of Kubernetes out there.

All of these options create flexibility, but wherever you create flexibility, you also create complexity. Complexity in the infrastructure increases the time it takes to troubleshoot and resolve issues.

Cost. Most of us have figured out that products that include every feature under the sun are a waste of money. We don’t need expensive and extraneous clutter. Likewise, gathering and processing too much irrelevant monitoring data adds significantly to costs. Investing in observability platforms that are overloaded with bells and whistles you simply don’t need is similarly costly.

The ideal observability platform taps AI and automation to help you easily see what’s most important to the business and respond quickly to core issues with precision and advanced insights.

Culture change. Going from monitoring to observability is akin to making a cultural quantum leap. Observability requires a different organizational structure and assigns different responsibilities to the teams. The transition requires change management skills and patient, thoughtful execution.

Though these challenges are significant, the observability industry is making rapid progress and organizations can expect to see solutions and technology providers that overcome these challenges shortly. So, be on the lookout for great opportunities coming down the pike.

In the meantime, I challenge you to change your perspective from horizontal to vertical. Set your organization on the path to move beyond DevOps and SRE to true observability, where your teams will not just be alerting and chasing issues for troubleshooting but working to establish real SLOs and driving toward those to enable the strategic goals of the business.

December 12, 2023