devopsy.kinsta.cloud

Author: Shahar Azulay

  • Scrimmage for Control: On-Premises Vs. SaaS in the Observability Super Bowl of 2024

    Scrimmage for Control: On-Premises Vs. SaaS in the Observability Super Bowl of 2024

    Observability means data. It encompasses a set of practices and tools designed to track the performance and health of software applications, providing insights into the “why” behind system behaviors, not just the “what.” At its core are data pillars we’ve learned how to work with and hopefully get the most out of—metrics (quantitative data on system performance), logs (recorded events within the system) and traces (detailed views of request paths through the system) that offer a comprehensive view of the system’s operation in real-time. SaaS is where the industry has been turning towards for the past decade. It’s easy to operate and upgrade and takes away maintenance and manpower requirements.

    But, using a SaaS solution where data is stored on the vendor’s infrastructure can be compared to playing a football game at your opponent’s home field. Your team does not own the field and must play along with conditions that are outside their comfort zone. While your team can still strategize, execute plays and score points, the control over the playing environment, including the surface, lighting and even some rules, lies with the owner of the stadium (in this case, the SaaS vendor).

    You don’t get the home-field advantage and the comfort of your familiar surroundings, and you also can’t control the audience’s shouts and cheers in the stands across the arena.

    Just as a visiting team must trust the host’s management of the stadium and adapt to its conditions, organizations using SaaS solutions entrust their operational data to third-party vendors. They must trust its data protection policies and privacy agreements to ensure that the organization’s data is handled securely and in compliance with relevant regulations.

    And even more importantly, they have to adhere to its hosting costs.

    Let’s Leverage the Home-Field Advantage

    The alternative is clear: While playing on your own field, you, the coach—hell, the entire team—know this field by heart, and you know best how to keep and enforce your own rules.

    Just as a football team controls the ball and decides on the play, having on-premises data means a company or organization keeps its data within its infrastructure, offering direct control over its storage, management and security. The organization decides how to deploy, use and protect this data and even how to manage the tradeoff of data costs.

    The downsides? Playing on your home field at all times means you’re running slower than the world out there. You’re not quick to update your software to the newest common practices and you’re tasked with maintaining another stack—your observability stack—on top of building your product, which serves your users.

    There’s Another Way. Play the Data on the Scrimmage Line

    In football, the scrimmage line is the imaginary line across the field where the ball is placed at the start of each play. This line serves as the point of engagement between the two opposing teams, with specific rules governing how players can position themselves and how the play begins.

    What if we can keep the data on-premise, in the home field, a little like having the ball on our side of the scrimmage line, within our team’s possession and control? We have the ball, and we decide and strategize how to advance the ball or score from the position behind the scrimmage line, but we can still interact with the outside world (or the opposing team) under a set of rules to which we both adhere.

    This is exactly where observability solutions are heading: A combination of the advantages of SaaS—easy to update and maintain value, with an easy way to distribute the solution inside the organization, alongside the advantages of on-premise—data privacy, data control and full transparent and controllable data costs.

    I call this approach inCloud. It is a modern on-premises approach—data in your cloud environment, protected and controlled by the user, but still governed, managed and served with a guaranteed SLA just like the best SaaS offerings out there.

    Just like the scrimmage line in football that provides a clear boundary and set of rules for how the game is played, inCloud acts in a similar way. Data is kept on-premises, allowing the organization full control over retention, privacy and data integrations. And on the other hand, inCloud defines a set of strict rules under which a control plane is allowed to operate and pass the scrimmage line, or past the organization’s cloud borderline, to manage and access the data like a SaaS solution does best.

    So when we sit down to watch the Super Bowl this Sunday—let’s use that halftime show to think about where we’ll be taking our organization’s observability capabilities in 2024. Will the data be in our hands, crossing that invisible scrimmage line? Or will we lose to our opponent’s home-field advantage?

    May the best team win.

  • Observability Costs are too Damn High

    Observability Costs are too Damn High

    Today, any business that deploys software faces an obscene amount of expenses. It has to pay for cloud hosting, assuming of course that it’s among the 92% of companies that use the cloud. It needs trustworthy networks to connect its apps to its employees and customers. It probably pays for a suite of cybersecurity software in a bid to stay ahead of the ever-growing list of cybersecurity threats. All of these costs, which are unavoidable for any company that wants to host modern software, add up to higher overall IT bills.

    But there’s a source of excess spending that has grown tremendously in recent years: Observability. If a company deploys software, they also need to monitor and observe that software to ensure that it meets performance and availability requirements. In recent years, the cost of observability has skyrocketed.

    In fact, according to Charity Majors of Honeycomb, the total cost of observability at an organization today is equivalent to up to 30% of total infrastructure spending. That’s a truly astounding figure, if you think about it. For every dollar spent on the infrastructure that powers applications, 30 cents ends up paying just for the tools and services needed to make sure the apps running on that infrastructure are doing what they’re supposed to.

    On top of this, spending on observability tools has become very unpredictable, meaning companies have no idea what they are going to be paying next week, next month or next year to get the insights they need to manage their applications and infrastructure. In this sense, observability is worse than taxes. With taxes, at least you know ahead of time what you’re going to have to pay in most cases.

    Getting Observability Spending Under Control

    Observability costs won’t go down on their own,  but there are practical steps that organizations can take to get observability spending under control–and they can do it without sacrificing the critical visibility they need to manage complex systems.

    To prove the point, we’ll discuss the reasons why observability has become so expensive. It’s partly about technology, but it also has to do with who gets to make decisions about adopting observability tools and how cultural inertia inside organizations impedes the ability of practitioners to migrate to more cost-efficient observability solutions.

    Then, we’ll talk about actionable strategies for getting observability costs under control. By taking advantage of some new tools and strategies, observability spending can be reined in while also–and this is the best part–actually increasing the organization’s ability to observe and manage complex systems.

    Why Observability Costs so Much

    There’s no one simple reason why observability costs for the typical organization today have increased. Instead, several factors have conspired to bloat observability bills.

    Drowning in Data

    One issue is the sheer volume of observability data that organizations have to collect.

    That’s partly because businesses deploy so many individual applications–207 on average, according to Okta–each of which generates observability data, to say nothing of the infrastructure on which the apps are hosted. But it’s even more so because of the fact that cloud-native architectures have vastly increased the number of application components and services that we need to observe.

    Instead of having one monolithic app and one host server to monitor, today, companies might have two dozen microservices, a Kubernetes orchestration plane, a bunch of nodes and perhaps an underlying IaaS platform, all of which they have to observe just to keep a single app running healthily. That adds up to tremendously more observability data than businesses had to manage in the past.

    Viewed from one perspective, having more observability data is a good thing. The more data there is, and the more granularity that can be associated with particular parts of the stack, the more insights DevOps teams can glean, at least in theory. But the problem is that, in many cases, only a small fraction–perhaps as little as 1%–of the data that is collected and ingested into observability tools is actually ever used to detect or troubleshoot performance issues.

    Granted, part of the reason we collect so much data but use so little is that it’s impossible to know ahead of time which observability data we’ll actually need. But that’s not an excuse for paying for data that never serves a useful purpose. If we’re going to spend so much on observability data, we might as well use more of it while also finding ways to pay less for it.

    Inefficient Data Collection

    The problem of ever-growing data volumes is compounded by the challenge of inefficient data collectors, which translate to higher infrastructure costs.

    Here’s why: The traditional way to deploy observability tools is to rely on agent-based data collectors that run essentially as standalone applications alongside the applications they are monitoring. As a result, the observability agents suck up a non-insignificant amount of CPU and memory, which require more infrastructure resources to support them and, thus, higher costs.

    Hence, one of the grand ironies of our times: To figure out whether their applications are consuming resources efficiently, businesses deploy observability software that actually decreases resource efficiency, while also bloating costs.

    Complex Pricing Models

    The pricing models of the typical observability tool are, in a word, complex. Most vendors charge in part based on how much observability data is ingested into their tools (this is usually the biggest contributor to overall cost), but they might also charge based on how many agents are deployed, how many users the tool has within the organization, and which features are wanted to access. The costs might also vary depending on where the data is chosen to be stored, how long of a retention period, how often it needs to be accessed  and so on.

    Complex pricing makes it hard in many cases for organizations to cost-optimize their observability spending. Determining exactly which features and usage tiers are needed can be a tough job, and companies might easily end up overspending as a result (after all, who wants to risk underspending and ending up with observability gaps that lead to critical failures?). Plus, unlike, public cloud services, there are no cost-optimization tools designed to help businesses “rightsize” their observability tool deployments or architectures.

    Organizational Inertia

    Historically, most observability and APM solutions were designed first and foremost for developers. To use them, developers had to instrument observability into applications. Then, it fell on the operations or DevOps team to connect the apps to APM tools and figure out what to do with the observability data they generated.

    There’s nothing wrong with involving developers in the observability process. However, the pitfall of the developer-centric approach to APM and observability that organizations have tended to follow is that developers typically don’t actually know very much about what the operations team needs to observe. After all, developers build apps; they don’t monitor or troubleshoot them. The result of this is that developers design apps to generate all sorts of observability data that may or may not be useful (which is why, again, organizations end up paying for a whole bunch of data they don’t actually use).

    This might not be much of an issue if it were easy for the operations team to come in and say, “Hey, we’re collecting all this data and it’s dumb. Let’s be more strategic.” But sadly, it’s not. Organizations being organizations, effecting change is not easy, especially if the change means getting people to adopt new tools or practices. So, businesses end up stuck with observability solutions that may have worked well in the past, but that are not efficient by modern standards, due simply to organizational inertia.

    Cost Unpredictability

    As mentioned, the problem with observability costs today is not only that they’re too damn high. It’s also that they’re too damn unpredictable.

    Observability vendors don’t hide their costs, so those are easy enough to predict. But those costs, again, are based largely on how much data is ingested into the observability tool. And how many engineers do you know who can predict with any kind of accuracy how many logs and metrics their applications will generate from one day to the next?

    Most can’t because data volume is tied to application demand, and that’s inherently unpredictable. After all, if we knew exactly how many requests our applications were going to receive from one moment to the next, we probably wouldn’t really need observability at all. But we don’t, and part of the core purpose of observability tools is to ensure that we can detect the issues that arise due to unexpected software usage patterns.

    I am not exactly criticizing observability vendors here, but rather pointing out that their pricing models capitalize on a fundamental limitation of modern applications: The amount of observability data they’ll produce is unpredictable in most cases, which can lead to wild fluctuations in observability spending.

    Keeping Observability Costs in Check

    Just as there’s no one single cause of high or unpredictable observability costs, there’s no one trick that can rein the spending in. But there are several strategies that, used in unison, will bring observability costs under control:

    Analyze Data at the Source

    The more data is moved in order to observe it, the more it will end up costing. By the same token, the more data analyzed at its source, the lower the cost will be.

    This is why, for instance, Kubernetes observability will cost much less if the data is collected and analyzed right inside nodes, instead of shipping it out to an observability platform first. Not only will the results be faster (because there is no need to wait for the data to move) but data that isn’t critical will be able to be ignored , which reduces how much data is ingested into the observability tools and cuts down on overall costs.

    Keep Data at the Source

    In a similar vein, not all data needs to leave its source at all. There might be datathat needs to be kept handy in case it is needed later, but  can not be analyzed right now. Instead of moving it into external storage, which will cost more, it should be kept right where it originated – inside the nodes. It’s there if needed, but there is no need to  payi extra for it.

    Add Efficiency With eBPF

    I mentioned before the grand irony in which organizations end up wasting infrastructure resources to deploy observability tools: There’s a pretty simple solution for that. It’s called eBPF, and it’s an ultra-efficient way to collect observability data. Unlike conventional observability and APM software, eBPF data collectors run directly inside the operating system kernel instead of running as user space applications. That leads to much lower levels of CPU and memory consumption. With eBPF, there is  all the observability needed, without sacrificing resources in the process.

    Affordable Observability

    The cost of deploying software these days is going up overall, but observability costs can actually go in the opposite direction. Thanks to new observability tools and techniques, it’s possible to collect all of the data neeeded, without allocating a tremendous portion of the overall IT budget to it.

    So, say goodbye to bloated budgets, and say hello to affordable observability.