devopsy.kinsta.cloud

Tag: SRE

  • Our Infrastructure is Still Expanding

    Our Infrastructure is Still Expanding

    You would think that sooner or later, marketers and early adopters would take a rest, but it seems that if one group slows down for a bit, another picks up. Our infrastructure is expanding in almost every possible way, and this creates more of a burden on every aspect of DevOps, specifically in IT.

    We have more security needs because we have a larger footprint, for example. We have long had APIs, but now we have massive API groups that are used and re-used, and while standard code scanning works with the source, the companies doing “API security” start with security of the interface, not the code. That is slowly changing, but API management is also impacted, and organizations that determined they didn’t need API management because adjacent markets were serving enough of their needs are slowly discovering that with proliferation comes the need to look at API management again.

    I stay up to date on most new technology because, as an industry analyst, what is coming can inform my discussions of what a given technology space looks like today. But even for me—someone who is not spending a huge chunk of my day coding or working on the toolchain or perfecting the new deployment platform—it can be hard to stay abreast of everything. While one of the foundational tenets of DevOps was no specialization and a shared workload, I think everyone now agrees with what a few of us were saying from day one: Some specialized skills are still needed, except in corner cases. But we certainly are sharing workloads more broadly and far more responsive, so DevOps is absolutely a massive success. Site reliability engineering (SRE) extends that success but focuses more on Ops. I’m really not a huge fan. I like that DevOps wanted to merge the processes and make the management of applications more holistic; I don’t think re-dividing it is the best idea out there. Like every title, “SRE” is interpreted pretty broadly. Some roles really involve DevOps (but SRE sounds cooler), and other organizations are just replacing ‘developer’ with DevOps and ‘operations’ with SRE. However, most are using SRE to manage the entire environment in which the app runs.

    Where are we going? Constant change. That’s the answer. I’ve waited for the breather for a couple of decades, but since VMs were popularized, it has been non-stop and will continue to be.

    This means a lot to you, to your viability as an employee, and to hiring. With the glut of people on the market, we’re seeing a resurgence of “must know every tool in our toolchain” type position postings. Of course, you must not, in a world where the combination of languages, frameworks, CI/CD tools, alerting, and monitoring tools would make a great college-level combinatorics equation. You add value by being adaptable. Both to your current employer and to future ones. The key thing you should seek to demonstrate is your ability to pick up and master new technologies when required. Do it on your current job to show worth, and keep a record of it for your next job. I’ve been a hiring manager off and on forever, and when the market is flooded with good candidates, that is the type of differentiator that gets the team’s attention. Do it; make it something worth doing and document that you did it. You won’t regret it.

    A group of us were helping a recent grad figure out what to do with their career a couple weeks ago, and the consensus was “avoid AI until it shakes out.” This is not the first time I’ve heard that advice—it’s not even the first group I was a part of giving that advice. And I agree in the general sense, but certainly in places where it is being implemented by all vendors in the space and it adds value, you do need to be aware of how it works and what you can do with it, and a project is one way to get that knowledge.

    It is still a rolling opportunity, and you are still keeping the lights on. I’ve seen people who worry they’re not doing enough. Any employer worth working for wants you to keep the systems running first, and everything else after—so if you’re doing that, you’re doing enough. Now go beyond. And keep rocking it. This new technology hasn’t implemented itself, and it isn’t self-maintaining (yet) … That’s all you.

  • Unlocking Accountability: How Real-Time App Monitoring Empowers Engineering Teams

    Unlocking Accountability: How Real-Time App Monitoring Empowers Engineering Teams

    In today’s fast-paced software development environment, traditional infrastructure monitoring is no longer sufficient. As engineering teams take on more responsibility for code ownership and software quality, the need for comprehensive real-time application monitoring becomes crucial. This isn’t just an addition to standard infrastructure checks; it’s a fundamental shift toward a more accountable and empowering engineering culture.

    A Case for “Real-Time Application Monitoring?”

    Infrastructure monitoring is ubiquitous, and we’ve gotten really good at monitoring the health of our servers, our containers and our data stores. APM tools have come a long way and give us insights into how pieces of our code perform from a speed and error context.

    At the same time, the business side of the house is monitoring things like traffic, leads, conversions and an important performance metric—sales. However, for most businesses, this monitoring is rarely in real-time, and visibility gaps are created in the spaces between where problems are introduced, discovered and resolved.

    Let’s look at a concrete example using an auction marketplace. Core to this business is the ability for users to place, accept and pay for accepted bids. The business probably looks at these metrics on a daily basis, at best. What if a delivered change to their application broke and removed the ability for users to conduct one of these actions?

    Placing a bid probably has a “normal” volume pattern, and a change could impact this in a positive or negative way. If the volume of bids drops after a deployment or even spikes, what happens? How quickly can that change be reported, seen and acted upon, if necessary? How quickly can they get back to “normal” before a customer reports it or before the rest of the business notices?

    If this business is limited to once-daily monitoring, how quickly can they realistically expect their engineering team to be able to identify and remediate such an issue?

    How valuable would it be to the business if their engineering team could instrument these metrics with their own real-time tooling?

    Key Metrics for Trust and Satisfaction

    Customer trust and satisfaction are critical for any software product. The most obvious indicator of customer satisfaction is churn–the retention rate of your customers. This metric, while influenced by various factors, remains a vital sign of how well we are doing. At Stoplight (acquired by SmartBear in August 2023), we focused on reducing customer bug reports. A spike in these reports prompts an immediate investigation into the root cause. It’s about understanding what changed and how to address it.

    Moreover, new customer acquisition isn’t just about retention; it’s also about word-of-mouth referrals. Satisfied customers often become your best advocates. Thus, keeping an eye on churn rates and incoming bug reports and setting realistic targets for these metrics is crucial. It’s about balancing resource allocation without striving for impractical perfection.

    Net Promoter Score (NPS) scores, although often dreaded by customers, are a critical metric. As engineers, we might dislike them, but they offer a holistic view of user satisfaction. If you’re deploying daily and consistently breaking things, expect your NPS scores to reflect that.

    We can also demonstrate real-time ROI of real-time application monitoring in a security context using login successes and failures. Tracking the volume of these activities in our engineering tools allows us to quickly identify and assess the threat of anomalous behavior. Did the rate of successful logins drop? Did we break something with a deploy (I’ve broken logins before). What if either successes or failures spike? Are we under attack? How can we mitigate it to protect our users?

    Employee Retention and Developer Satisfaction: Essential in an Era of Real-Time Monitoring

    In the realm of software development—particularly with the advent of real-time application monitoring—employee retention, especially of developers, is paramount. Their deep understanding of the nuances of our applications and their ability to respond swiftly to the insights provided by real-time monitoring are invaluable.

    Maintaining a team of satisfied, engaged developers is crucial in this context. It’s not just about reducing turnover; it’s about fostering a culture where the engineers feel invested in the continuous improvement and success of our products. When developers are genuinely satisfied with their work and their environment, it reflects in the quality of their output. They become proactive in identifying and addressing issues, often before they escalate, thanks to the real-time data at their fingertips.

    The shift toward more dynamic monitoring practices has underscored the need for a supportive, collaborative environment. A culture where developers are encouraged to share insights and take initiative leads to a more responsive and adaptable team. This environment not only supports the technical aspects of our work but also enhances the overall morale and commitment of our developers.

    Engineers will seek out organizations that adhere to better practices. Interestingly, when good practices are in place, developers are more open to being on call. They accept responsibility for their code and their peers’ code as a trade-off for the autonomy and value they can deliver through frequent, quality deployments.

    Starting Small: Embracing the Journey of Incremental Improvements

    The journey to adopting best practices in engineering, such as real-time application monitoring, often begins with small yet significant steps. Reflecting on my past experiences, particularly at a shipping marketplace, the path to continuous deployment was transformative. Initially, we had no automated tests, and the challenges were substantial. However, the commitment to deploy continuously led to an evolutionary process in our practices.

    This approach isn’t just about technology; it’s about mindset. In my earlier days, inspired by the continuous deployment practices I observed in financial institutions, I realized the power of incremental changes. We began with what we had, deploying continuously, and with each step, we uncovered new areas for improvement. It was a process of building our capabilities piece by piece–from integrating automated tests to refining our deployment strategies.

    The essence of this journey lies in its continuous nature. Much like the principles of agile and DevOps, it’s a perpetual process of evolution. You don’t reach a finish line and stop; it’s about relentlessly pushing for betterment, identifying gaps, and adapting to new challenges. It’s about creating a culture where continuous improvement is the norm, where each small step contributes to a larger goal.

    Building a Culture of Continuous Improvement and Excellence

    Adopting real-time application monitoring is more than implementing a new set of tools; it’s about fostering a culture of continuous improvement, accountability, and excellence. In my experience, this cultural shift has a profound impact on every aspect of our work. It’s about ingraining a mindset that constantly seeks to exceed our standards with each release.

    Creating this culture involves a holistic approach to our engineering practices. It’s not just about the technical side of things; it’s about enhancing the collaborative and innovative spirit of our teams. It’s important for engineering leaders to do their part to foster an environment where every team member feels empowered to contribute ideas and take initiative. The ROI of this approach can lead to significant improvements in your team’s processes, as well as a deeper sense of ownership and pride in their work.

    Moreover, this culture of continuous improvement often extends well beyond the engineering department. It can influence the entire organization, encouraging departments like sales, marketing, and customer support to adopt similar principles. By doing so, we create a unified approach to product development, customer service, and business growth.

    Conclusion

    Embracing real-time application monitoring is not just about adding another layer to our development practices; it’s about fundamentally shifting our mindset toward a culture of accountability and continuous improvement. By focusing on key metrics, understanding the psychology of developers, and starting small, we can drive significant improvements in both product quality and employee satisfaction. As engineers, we are in a unique position to influence this change, and it is imperative that we lead the charge in adopting these essential practices.

  • Broadcom Survey Surfaces Raft of IT Automation Challenges

    Broadcom Survey Surfaces Raft of IT Automation Challenges

    A survey of 501 IT leaders published today by Broadcom suggested that islands of automation that operate independently of each other are major contributing factors to organizations failing to meet service level agreements (SLAs).

    The survey found that 98% of respondents encountered at least one automation issue, with 61% noting that SLA issues are a monthly occurrence. More than a third (35%) said these issues occur weekly. A full 88% indicated SLA breaches directly impacted customers in a significant way.

    The survey also found nearly three-quarters (74%) lacked visibility across their data center and cloud environments, with an equal number lacking end-to-end visibility of their automated business processes. Only 28% possessed tools that could predict automation issues, and more than two-thirds (68%) were unable to model or predict whether a new or updated automation job would actually meet SLAs. A full 69% said they are unable to properly size automation workload resources, with 61% having no way to simulate automation workloads before implementation, the survey found.

    Only slightly more than half (55%) of respondents, however, could provide SLAs and key metrics in a format that is appropriate for business stakeholders.

    Aline Gerew, head of automation for the Agile Operations Division at Broadcom, said that while it’s routine for IT teams to leverage a range of automation tools and platforms to manage IT, it’s clear they are challenged to manage those platforms in a comprehensive fashion. As a result, whenever there is an issue with one tool or platform, it creates a bottleneck that impacts customer experience in a way that ultimately adversely impacts revenue, she noted.

    In fact, many IT teams are now starting to identify automation architects within their IT teams to enable DevOps teams in a more holistic manner, added Gerew.

    Overall, the survey found that 80% of respondents are using three or more automation platforms, with just over a quarter (27%) using six or more.

    Nearly three quarters (74%) are also relying on three or more observability solutions, with just under a quarter (24%) using six or more. Just under three-quarters of respondents noted they are consequently subjected to alert storms that ultimately serve to make it more challenging to resolve issues. More than half (56%) of respondents are blind to the downstream effects of an initial alert and, as a result, do not inform response teams of which systems, jobs, applications and business processes are affected.

    In some ways, IT automation has become too much of a good thing. Every provider of an IT platform makes automation tools available, but as they are invoked, each one creates yet another island that IT teams are expected to manage in isolation. However, all it takes for an SLA not to be met is for an issue to arise that IT teams can’t easily determine the root cause of because one or more automation platforms obscure the underlying issue in the absence of any standardization, noted Gerew.

    Given the complexity of modern IT environments, automation is now a requirement. The issue, as always, is finding a way to apply it all in the most transparent way possible.

  • 5 Reasons to Move Beyond SRE to Observability

    5 Reasons to Move Beyond SRE to Observability

    Recently, I was asked on a podcast to explain the relationship between site reliability engineering (SRE) and observability and why I believe organizations need to move their focus from the former to the latter. Those are great questions, but to answer them accurately, I first need to point out that many organizations claim to be doing SRE or claim to be doing observability but aren’t.

    One big thing that SRE and observability have in common is that they are both often erroneously equated with monitoring.

    SRE ≠ Monitoring

    Google is credited with pioneering the SRE movement with the publication of the “Site Reliability Engineering” handbook in 2016. As the SRE movement gained momentum, organizations of all sizes pored over the handbook and began applying its principles. However, very few organizations are of the size and sophistication of Google regarding the scale of their IT operations and engineering resources.

    So, as these “normal” organizations realized how difficult it was to follow the Google SRE approach in its entirety, they often opted instead to simply apply what they could. For many, the chapter on monitoring became the focus, so much so that monitoring has become synonymous with SRE in far too many organizations today.

    Let me be clear: The role of a site reliability engineer is not to monitor alerts. The role of an SRE is to define how the engineering team should take ownership of their service. SREs are responsible for establishing a culture and creating engrained processes that are focused on the quality and reliability of infrastructure.

    SREs guide the engineering team on how an application or service should be observed and monitored, including what parameters should be looked at, what alerts they should have, and what routing path the alerts should go through. An SRE instructs on best practices, explores the tools that are being used and determines how to gather reliable data with consistency throughout the organization.

    Many organizations we work with say they want to do SRE this way, but they’re not there yet. They are still stuck on monitoring every single metric they can find.

    Observability ≠ Monitoring

    Similarly, a lot of companies I meet say that they do observability, but what they are doing is just monitoring. They monitor every single CPU of every node of every pod of every machine that is running. They have alerts for some of these, and they may even have a playbook for some of them. This is not how SRE is supposed to work, and it’s certainly not what observability is all about.

    More importantly, it’s not scalable as an organization grows to hundreds or thousands of developers and different teams that all share the same IT environment.

    Two Viewpoints: Horizontal and Vertical

    Now that we’ve established that SRE and observability are alike in that neither should be equated with monitoring, let’s contrast the two. SRE and observability are different in that they have different perspectives on the IT domain.

    SREs see the world horizontally. They conceive of their IT domain on a horizontal plane where they can observe the capacity of resources being used. They see the Kubernetes cluster, and they see the cloud utilization. And they care about ensuring that what is deployed into the production environment is secure.

    Although they see the pieces they manage on the horizontal infrastructure plane, they don’t see the ultimate impact on the application or customer on the planes “above.”

    Observability is looking at the world vertically—that is, seeing how an application is or isn’t meeting customer needs. The whole goal of observability is to know if my application is serving my customers at the service level that I agreed to. With a vertical view from application up to customer needs and business impact, I don’t care how many pods are being run, how many nodes are being deployed or what comprises the cloud infrastructure underneath my application, as long as it works.

    I do care about security insofar as I want to avoid introducing security vulnerabilities into the organization. But beyond that, all I need to know is that the application can meet the customer’s expectations and the service level objectives (SLOs) that we have established are being met, as well.

    Therein lies the heart of the matter: For a company to make the transition to observability, it needs to build the observability system around establishing and meeting SLOs.

    The Challenges of Transitioning to a Vertical Perspective

    Companies must move beyond an SRE and DevOps organization to being an observability and SLO organization. Only by doing this will organizations be capable of understanding the trade-offs within their IT environments between cost, security and availability and the ultimate impact those trade-offs will have on business objectives.

    Today, making this transition — moving beyond SRE to observability — is difficult for a wide variety of reasons, not the least of which are:

    Alert Fatigue. Companies today are monitoring too many things they shouldn’t care about, and as a result, engineers are suffering from alert fatigue. With this, you face the danger of a higher probability of missing the alerts that you do care about and making sure that you address them promptly.

    Ideally, observability solutions eliminate the irrelevant and unimportant alerts, add context to alerts that might need investigating, and correlate response data and docs to accelerate and coordinate the appropriate response of on-call responders.

    Tool sprawl. Many of the companies we work with use a wide variety of separate tools for observability. Observability requires access to the data all of these tools produce, and problems can persist if the tools operate in data silos.

    However, the problem can be solved if there is unified data collection across the toolset, ideally using OpenTelemetry. Cross-tool and cross-stack insights are particularly important for security purposes. Another issue with tool sprawl is that true observability requires that these tools, which each typically address just one aspect of observability, somehow all work together.

    Unfortunately, current systems and processes lack the “connective tissue” to offer a consolidated observability solution.

    Complexity. As we’ve seen in the results of our annual DevOps Pulse survey, the mean-time-to-resolution (MTTR) is going up instead of down, despite the number of tools that are available and the advancements made by the industry in recent years.

    Complexity is a large part of the problem, specifically concerning Kubernetes. In our 2023 DevOps Pulse Report, we found that almost 50% of respondents cited Kubernetes as their main challenge to gaining full observability into their environment.

    The adoption of Kubernetes is happening at lightning speed—even faster and higher than the adoption of cloud technology. But it’s not just the adoption of one Kubernetes—there are so many different flavors of Kubernetes, whether it’s Kubernetes running on managed services like AWS EKS or AKS or Kubernetes running on serverless with AWS Fargate. And that’s just to name a few of many, many types of Kubernetes out there.

    All of these options create flexibility, but wherever you create flexibility, you also create complexity. Complexity in the infrastructure increases the time it takes to troubleshoot and resolve issues.

    Cost. Most of us have figured out that products that include every feature under the sun are a waste of money. We don’t need expensive and extraneous clutter. Likewise, gathering and processing too much irrelevant monitoring data adds significantly to costs. Investing in observability platforms that are overloaded with bells and whistles you simply don’t need is similarly costly.

    The ideal observability platform taps AI and automation to help you easily see what’s most important to the business and respond quickly to core issues with precision and advanced insights.

    Culture change. Going from monitoring to observability is akin to making a cultural quantum leap. Observability requires a different organizational structure and assigns different responsibilities to the teams. The transition requires change management skills and patient, thoughtful execution.

    Though these challenges are significant, the observability industry is making rapid progress and organizations can expect to see solutions and technology providers that overcome these challenges shortly. So, be on the lookout for great opportunities coming down the pike.

    In the meantime, I challenge you to change your perspective from horizontal to vertical. Set your organization on the path to move beyond DevOps and SRE to true observability, where your teams will not just be alerting and chasing issues for troubleshooting but working to establish real SLOs and driving toward those to enable the strategic goals of the business.

  • Google Taps Nobl9 to Improve Services Reliability

    Google Taps Nobl9 to Improve Services Reliability

    Google announced this week it has added a capability for tracking reliability based on how service level objectives (SLOs) are being met that is based on a platform developed by Nobl9.

    Nobl9 launched a Nobl9 Reliability Center earlier this year that provides a single source of truth about the state of reliability in an application environment. Included in that platform is SLOgpt.ai, a generative artificial intelligence (AI) tool for programmatically creating SLOs and launching natural language queries to determine how well they are being adhered to across a distributed computing environment. The generative AI tool developed by Nobl9 is based on Vertex, a foundational large language model (LLM) provided by Google.

    Kit Merker, chief growth officer for Nobl9, said the generative AI tool also makes managing SLOs more accessible to a wider range of stakeholders.

    The expansion of the alliance with Google is the latest in a series that will make it simpler to manage SLOs across multiple ecosystems, including, most recently, partnerships that have been established with Microsoft and Cisco, noted Merker.

    The Nobl9 Reliability Center is already integrated with more than 50 DevOps, observability and incident management tools. It collects metrics, events, logs, traces and alerts to track incidents, releases, rollbacks, runbooks and other documentation. Those integrations will make it simpler for IT teams to create SLOs using historical data that would otherwise be challenging to define from scratch.

    In time, there is no doubt tools for managing SLOs will become more widely available, but given the dependencies that exist between applications and systems, DevOps teams will need an approach that can be employed across hybrid IT environments, said Merker.

    In general, as more microservices-based distributed applications are built and deployed, it’s becoming more challenging to maintain SLOs across them since they have many more dependencies than legacy monolithic applications. Each feature and application programming interface (API) added over time can adversely impact performance and reliability. The challenge is that it’s not so much that any given service will outright fail as much as it is determining where degradations are occurring that result in SLOs not being met.

    Each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services by making it simpler to track whether SLOs are achieved and maintained. Not every service needs to necessarily meet a stringent requirement, but IT teams do need to have confidence in the fact that, on average, certain levels of service are being consistently maintained.

    When it comes to SLOs, the challenge is first determining which dependencies might impact the level of reliability that can be maintained in the event of an outage and then, second, making sure whenever necessary workarounds—such as rerouting application programming interface (API) calls to other services—can be easily invoked.

    SLOs, of course, as a concept, have been around for decades, but with the rise of digital services that span multiple organizations, the level of reliability DevOps teams are expected to provide is now being written into business contracts. As a result, the overall level of stress related to managing SLOs is only going to increase in an era where businesses have never been more dependent on software.

  • Nobl9 Allies With Microsoft to Simplify SLO Creation

    Nobl9 Allies With Microsoft to Simplify SLO Creation

    Nobl9 this week revealed it has integrated its reliability center platform for managing service level objectives (SLOs) as code with the Microsoft Azure Monitor service to make it simpler to both create and track service levels.

    The Nobl9 Reliability Center provides access to dashboards for a single source of truth about the state of reliability in an application environment based on the metrics collected from the SLOs that have been defined.

    Nobl9 Chief Growth Officer Kit Merker said the integration with Microsoft Azure Monitor is the first in a series that will ultimately make it easier to define SLOs using historical data collected by third-party tools.

    The Nobl9 Reliability Center is already integrated with more than 50 DevOps, observability and incident management tools through which it collects metrics, events, logs, traces, alerts to track incidents, releases, rollbacks, runbooks and other documentation.

    Those integrations will ultimately make it simpler for IT teams to create SLOs using historical data that today are still challenging to define from scratch, noted Merker. In fact, the data tracked by Nobl9 can be used to automate remediations via those integrations, he noted.

    As more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Each feature and application programming interface (API) added over time conspires to adversely impact performance. It‘s not so much that any given service will outright fail as much as it is determining where degradations are occurring that result in SLOs not being met.

    Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services by making it simpler to track whether SLOs are achieved and maintained. Not every service needs to necessarily meet a stringent requirement. Instead, IT teams need to have confidence in the fact that, on average, certain levels of performance are being consistently maintained.

    That’s crucial because as more organizations depend on software to deliver digital services, any disruption is likely to have a direct impact on profit and revenue. As such, the reliability of application environments has become a major concern in an era where IT teams are now trying to manage distinct services rather than individual applications.

    The challenge and the opportunity now is to find a way to proactively reduce that stress by programmatically managing SLOs as code rather than having to overprovision infrastructure resources to ensure maximum availability at the highest cost possible. At a time when more organizations are encountering economic headwinds, that’s no longer a luxury they can afford.

    Each organization will need to determine for themselves how aggressively they need to manage SLOs. Of course, SLOs as a concept have been around for decades, but with the rise of digital services that span multiple organizations, the level of reliability DevOps teams are expected to provide is now being written into business contracts. Not surprisingly, the overall level of associated stress DevOps teams are experiencing has correspondingly increased.