devopsy.kinsta.cloud

Tag: Complexity

  • 5 Reasons to Move Beyond SRE to Observability

    5 Reasons to Move Beyond SRE to Observability

    Recently, I was asked on a podcast to explain the relationship between site reliability engineering (SRE) and observability and why I believe organizations need to move their focus from the former to the latter. Those are great questions, but to answer them accurately, I first need to point out that many organizations claim to be doing SRE or claim to be doing observability but aren’t.

    One big thing that SRE and observability have in common is that they are both often erroneously equated with monitoring.

    SRE ≠ Monitoring

    Google is credited with pioneering the SRE movement with the publication of the “Site Reliability Engineering” handbook in 2016. As the SRE movement gained momentum, organizations of all sizes pored over the handbook and began applying its principles. However, very few organizations are of the size and sophistication of Google regarding the scale of their IT operations and engineering resources.

    So, as these “normal” organizations realized how difficult it was to follow the Google SRE approach in its entirety, they often opted instead to simply apply what they could. For many, the chapter on monitoring became the focus, so much so that monitoring has become synonymous with SRE in far too many organizations today.

    Let me be clear: The role of a site reliability engineer is not to monitor alerts. The role of an SRE is to define how the engineering team should take ownership of their service. SREs are responsible for establishing a culture and creating engrained processes that are focused on the quality and reliability of infrastructure.

    SREs guide the engineering team on how an application or service should be observed and monitored, including what parameters should be looked at, what alerts they should have, and what routing path the alerts should go through. An SRE instructs on best practices, explores the tools that are being used and determines how to gather reliable data with consistency throughout the organization.

    Many organizations we work with say they want to do SRE this way, but they’re not there yet. They are still stuck on monitoring every single metric they can find.

    Observability ≠ Monitoring

    Similarly, a lot of companies I meet say that they do observability, but what they are doing is just monitoring. They monitor every single CPU of every node of every pod of every machine that is running. They have alerts for some of these, and they may even have a playbook for some of them. This is not how SRE is supposed to work, and it’s certainly not what observability is all about.

    More importantly, it’s not scalable as an organization grows to hundreds or thousands of developers and different teams that all share the same IT environment.

    Two Viewpoints: Horizontal and Vertical

    Now that we’ve established that SRE and observability are alike in that neither should be equated with monitoring, let’s contrast the two. SRE and observability are different in that they have different perspectives on the IT domain.

    SREs see the world horizontally. They conceive of their IT domain on a horizontal plane where they can observe the capacity of resources being used. They see the Kubernetes cluster, and they see the cloud utilization. And they care about ensuring that what is deployed into the production environment is secure.

    Although they see the pieces they manage on the horizontal infrastructure plane, they don’t see the ultimate impact on the application or customer on the planes “above.”

    Observability is looking at the world vertically—that is, seeing how an application is or isn’t meeting customer needs. The whole goal of observability is to know if my application is serving my customers at the service level that I agreed to. With a vertical view from application up to customer needs and business impact, I don’t care how many pods are being run, how many nodes are being deployed or what comprises the cloud infrastructure underneath my application, as long as it works.

    I do care about security insofar as I want to avoid introducing security vulnerabilities into the organization. But beyond that, all I need to know is that the application can meet the customer’s expectations and the service level objectives (SLOs) that we have established are being met, as well.

    Therein lies the heart of the matter: For a company to make the transition to observability, it needs to build the observability system around establishing and meeting SLOs.

    The Challenges of Transitioning to a Vertical Perspective

    Companies must move beyond an SRE and DevOps organization to being an observability and SLO organization. Only by doing this will organizations be capable of understanding the trade-offs within their IT environments between cost, security and availability and the ultimate impact those trade-offs will have on business objectives.

    Today, making this transition — moving beyond SRE to observability — is difficult for a wide variety of reasons, not the least of which are:

    Alert Fatigue. Companies today are monitoring too many things they shouldn’t care about, and as a result, engineers are suffering from alert fatigue. With this, you face the danger of a higher probability of missing the alerts that you do care about and making sure that you address them promptly.

    Ideally, observability solutions eliminate the irrelevant and unimportant alerts, add context to alerts that might need investigating, and correlate response data and docs to accelerate and coordinate the appropriate response of on-call responders.

    Tool sprawl. Many of the companies we work with use a wide variety of separate tools for observability. Observability requires access to the data all of these tools produce, and problems can persist if the tools operate in data silos.

    However, the problem can be solved if there is unified data collection across the toolset, ideally using OpenTelemetry. Cross-tool and cross-stack insights are particularly important for security purposes. Another issue with tool sprawl is that true observability requires that these tools, which each typically address just one aspect of observability, somehow all work together.

    Unfortunately, current systems and processes lack the “connective tissue” to offer a consolidated observability solution.

    Complexity. As we’ve seen in the results of our annual DevOps Pulse survey, the mean-time-to-resolution (MTTR) is going up instead of down, despite the number of tools that are available and the advancements made by the industry in recent years.

    Complexity is a large part of the problem, specifically concerning Kubernetes. In our 2023 DevOps Pulse Report, we found that almost 50% of respondents cited Kubernetes as their main challenge to gaining full observability into their environment.

    The adoption of Kubernetes is happening at lightning speed—even faster and higher than the adoption of cloud technology. But it’s not just the adoption of one Kubernetes—there are so many different flavors of Kubernetes, whether it’s Kubernetes running on managed services like AWS EKS or AKS or Kubernetes running on serverless with AWS Fargate. And that’s just to name a few of many, many types of Kubernetes out there.

    All of these options create flexibility, but wherever you create flexibility, you also create complexity. Complexity in the infrastructure increases the time it takes to troubleshoot and resolve issues.

    Cost. Most of us have figured out that products that include every feature under the sun are a waste of money. We don’t need expensive and extraneous clutter. Likewise, gathering and processing too much irrelevant monitoring data adds significantly to costs. Investing in observability platforms that are overloaded with bells and whistles you simply don’t need is similarly costly.

    The ideal observability platform taps AI and automation to help you easily see what’s most important to the business and respond quickly to core issues with precision and advanced insights.

    Culture change. Going from monitoring to observability is akin to making a cultural quantum leap. Observability requires a different organizational structure and assigns different responsibilities to the teams. The transition requires change management skills and patient, thoughtful execution.

    Though these challenges are significant, the observability industry is making rapid progress and organizations can expect to see solutions and technology providers that overcome these challenges shortly. So, be on the lookout for great opportunities coming down the pike.

    In the meantime, I challenge you to change your perspective from horizontal to vertical. Set your organization on the path to move beyond DevOps and SRE to true observability, where your teams will not just be alerting and chasing issues for troubleshooting but working to establish real SLOs and driving toward those to enable the strategic goals of the business.

  • Managing Risk

    Managing Risk

    We have built some beautiful toolchains that crank out a finished product on the fly without needing anything close to the level of intervention that was historically required. The most advanced organizations on an automation journey could change a line of code and then wait for the new version to hit production without doing a thing. I say “could” because I don’t know of a single org that does this; there are, of course, checks in place and it’s not single-line changes but projects that release. But the infrastructure is there that would allow them to do this. It’s a high probability that one day we all will.

    The problem is that most organizations only look at risk when building these toolchains and don’t bother with contingency plans. But over the long life of any given toolchain—toolchains that are made up of many moving parts—there will inevitably be the need for contingency plans.

    And it’s certainly not only toolchains—remember when popular libraries were purposely corrupted by their author to make a point? And unless you’ve done your best to ignore it, you’re also aware of faked versions of popular libraries that insert code before continuing on to the original…

    Most organizations do not consider a scenario in which critical libraries are suddenly unavailable or a scenario where a tool is crippled by movement in the market—a vendor making harsh licensing changes, the underlying tech making changes that impact the tool, etc. While it is comforting to think that open source protects us from many changes (and it forking does), that is not complete coverage that entirely protects our systems and toolchain. We need to know what is absolutely critical and how the organization will keep delivering in case that absolutely critical element goes away.

    When I have done disaster planning—of which this is an adjunct—I have done so with a vague generality in mind. We have plenty of work to do, so all I ever wanted was a list of critical parts and an idea of how we would adapt without them. I worked at a place that kept a backup data center in case of total regional disaster—the backup data center was intentionally located on the other side of the country—but most of us don’t need that kind of DR. Certainly not if we can go, “This OSS library is used in 60% of our apps; if we have to, we could adapt this other one,” instead of building infrastructure.

    For OSS dependencies, we have two great options that work together. Generate SBoMs, then correlate usage from those across the entire app portfolio. Then set up and maintain a local repository with point-in-time copies of those libraries that are deemed critical.

    A similar, “Our CI tool is critical, but we could easily port processes and YAML to this other one,” is enough, too—it is absolutely not going to be easy if you ever have to do it, but “We looked at this six months ago and we know what to do,” is far better than “Oh crap, now what?”

    You built it, you operate it, you are rocking it. Just take some time to protect it and you are covered.

  • DevOps: The Things We Must Stop

    DevOps: The Things We Must Stop

    Anytime there is a popular movement in technology (and I would argue any other field), people get carried away. We’ve seen it over and over, and DevOps attracts this kind of attention. In fact, DevOps caters to those who get carried away by mainstream folks screaming, “THIS CHANGES EVERYTHING!” That type of hyperbole pushes those who get carried away even further. And marketing always takes things to whatever place they have to in order to, well, sell their ideas to the market.

    Rack ’em Stack ’em Robots

    My all-time favorite purposeful over-the-top is the tendency to pretend that prior to the cloud, every project was started with “rack ’em, stack ’em.” This is demonstrably false, as by the time the cloud was taking off, VMware’s market penetration was significant. And yet, the number of blogs/amount of marketing material that starts out with the premise that every application required waiting for physical server delivery and configuration is insanely high.

    No. Enterprises, in particular, had to wait for VM allocation and configuration.

    Another that isn’t really mainstream—and I’ve mentioned in other locations—is the AWS TCO calculator. The simple calculator is not a bad tool, and I’ve used it numerous times in the past, but the TCO calculator makes some assumptions about deployment costs that are a little wild, such as that TCO includes the cost of VMware and its supporting hardware for an individual app. Of course, that’s not how enterprises do it. There is a VMware infrastructure and it is shared across apps. While it would increase the complexity a tiny bit, a simple checkbox that asks, “Do you already have a VMware cluster?” would go a long way—or better, one that asks how many apps you are running on VMware and divides the overhead by that number before including it in TCO. But of course, if you are getting your TCO comparisons from a competitor, expect the TCO will be stilted, no matter who is generating it.

    Oversimplification

    Whether it is infrastructure orchestration or app deployment, there is a lot of content out there that talks about how much easier infrastructure as code will be, and how much less documentation we will need. This, sadly, is a bogus premise. Whether you’re using the 50,000 service offerings of AWS or the power of HashiCorp’s Terraform—or any of half a dozen other competitors—it is complex. We’re not doing anyone any favors by glossing over the complexity that inevitably occurs in enterprise-class deployments, and we’re doing a massive disservice every time someone claims that documentation needs will be less because it’s “self-documenting code.” Self-documenting code sucks without a ton of comments in App Dev, and it’s worse in deployment. Employees new to the process need, and should be provided, solid documentation on how the system works and where different pieces fit.

    We’re deploying complex apps on complex infrastructures with evermore complex compliance and security requirements. It is not simple.

    All or Nothing

    There is a belief among too many aficionados that their way is the “One True Way.” The fact of the matter is that all enterprise application portfolios are complex and there is a metric ton of extant code that must be managed. A startup with one web-based app (no matter how large it may be) or a games developer focused on Android as its only platform might well have agile and DevOps as their only development/deployment methodology. No one else does. The enterprise running a hundred or a thousand different apps on several platforms is not going to flick a switch and say, “We’re all X now,” no matter what X is. Anyone who has tried to standardize the database in an enterprise knows that just doesn’t happen (I did, as an architect in a financial services org. It was ugly).

    Some projects are simply great fits for agile and DevOps—those with a high rate of change, where a little bit of instability in return for rapid feature rollout—while others simply are not—ones that maintain data that must be rock-solid, ones with a low level of change, etc. Any given enterprise is a mix of these things. Back in the days of the language wars, I used to say, “Right tool for the job,” and today, in the days of the architecture/automation wars, I say the same thing. If the project is rapidly evolving and you want lots of small changes in a constant stream, agile and DevOps are the right tools. If the project moves slowly and has an emphasis on stability or data quality, it’s not a great candidate for these tools at this time.

    I’ve talked about some of the bad vendors in this post, so I’ll call out IBM for being a good vendor on this point. Looking at the company’s model for moving the mainframe is a good example of how you could look at all of your IT, because IBM doesn’t say, “This changes everything!” It says, “Here’s how to tell what offers the most bang for the buck in moving to agile and DevOps.”

    Just Keep Rocking It

    Working in IT? You’ve been keeping your org running. Don’t start second-guessing your gut now just because the new shiny is being waved in front of you. Learn, change, do. But don’t rush into something with misinformation. Obey your gut and keep rocking your org’s world.

    — Don Macvittie