devopsy.kinsta.cloud

Author: Andreas Grabner

Empowering BizDevOps Teams With SLOs

When BizDevOps teams are drowning in logs and false alerts and are spending more time on manual processes than on building and deploying new applications and features, the work suffers, morale suffers and the organization as a whole suffers. Releasing better and more secure software, faster and with real business impacts starts with a robust BizDevOps practice. A crucial step IT leaders can take to align the developer, operations and business teams is to create a shared language of service-level objectives (SLOs).

SLOs are metrics for validating that software is meeting its technical and business objectives. Defining and implementing SLOs help align developer, operations and business teams, not only ensuring everyone is moving toward the same goals, but also providing a way to reliably measure how BizDevOps teams are progressing toward these goals.

Here are three of the most impactful ways SLOs can empower your BizDevOps teams.

1. Establish a Common Language Across the Delivery Cycle

The best SLO gives you precise and actionable insight into your system. Take performance and resiliency engineering, for example. By defining SLOs for performance and error rates, teams can use common measurements to track progress toward meeting their quality criteria to ensure they are staying within their error budget.

In this way, teams can use SLOs as a common language across the delivery cycle – something everyone from developers to operations to business teams can rally around, ensuring everyone is working toward the same outcomes. This aligns teams into a coordinated BizDevOps practice, where all sides have better insight into how each piece of the business feeds into the others. If an incident occurs, operations can use an AI-based observability platform to observe its trends and impacts across different product or service releases and automatically feed that information back to the developers, who can modify their builds accordingly. When everyone is using the same SLOs to speak the same language, BizDevOps teams can coordinate and communicate more easily and effectively.

2. Creating a Culture of DevOps Resiliency by Shifting SLOs Left

BizDevOps isn’t just a technical practice defined by tools and platforms. It’s also a culture change and a shift in mindset. SLOs can play a major part in this change. Shifting SLOs left into the delivery pipeline means giving developers feedback about the outcome of their work and real-time visibility into their business KPIs using consistent SLO measurements – ideally, on every code commit. This consistency gives them answers at their fingertips to make data-backed decisions that consistently deliver better business outcomes. Using SLOs for developer feedback is a fundamental building block of DevOps, but it’s also key to creating a culture and mindset of resiliency within BizDevOps.

A resilient BizDevOps culture is more than just making changes to your technology stack; it’s about changing the mindsets of individual engineers and developers. Resiliency means strength and reliability; the trust that when something goes down, it’ll go right back up. It’s about building the processes and systems that guarantee flexibility for anything from managing crisis events to delivering service continuity. Shifting SLOs left helps facilitate this culture change by giving developers greater exposure to the impact their work has when it’s in production and when it’s deployed. It makes everyone feel they’re in the same boat, and recognizes developers, operations and the business teams as one cohesive unit.

3. Integrating SLOs Into Remediation

BizDevOps teams can use SLOs to enhance remediation and incident response efforts. With everyone working with the same metrics, BizDevOps knows with certainty when something is broken, can identify root causes more quickly and can develop agreed-upon incident responses to accelerate time to resolution. BizDevOps teams can then feed this incident response knowledge back into quality gates earlier in the development cycle to avoid future problems. This develops SLOs that have been “battle tested” and can be easily automated to make quality gate testing better on the development side, and incident response better and faster for the operations side.

SLOs bridge the gaps between BizDevOps

SLOs already serve a useful role in production, but they also can influence the ability to create a robust, resilient BizDevOps practice. Shifting SLOs closer to developers and using them as a shared language across the delivery cycle provides space for greater experimentation, more innovation and a culture of resiliency. SLOs can also be defined along business-relevant metrics, such as conversion rates, user experience or feature adoption. As a result, SLOs empower BizDevOps teams to respond more quickly to issues and deliver better digital experiences and business outcomes.

SLOs aren’t just performance metrics – they’re a powerful tool for bridging the divides between the developer, operations, and business teams. SLOs can help cultivate a BizDevOps culture that delivers innovative, secure software faster and with tangible business value.

May 13, 2021
Has the Pandemic Made DevOps Tougher or Easier?

How has COVID-19 impacted the way DevOps professionals approach their work?

The current, unpredictable economic climate as a result of the global COVID-19 pandemic has exposed a glaring need for organizations to be able to do more with less. They have fewer resources and even less margin of error now to maintain customer relationships, keep up the pace of innovation and stay ahead of competitors. This global crisis has shifted more end users toward digital services, increasing the pressure further on organizations to deliver robust digital services to both their customers and employees. Meeting that need in a short amount of time is, suffice it to say, extremely challenging.

All of this has shaken up norms for DevOps professionals and created an even greater need to accelerate continuous automation, break down silos between teams and cultivate greater cross-team collaboration, to help their organizations continue business as usual—in other words, a greater need for DevOps, period.

Let’s take a look at how today’s new normal has in some ways made DevOps transformations easier and more widely accepted, and in other ways made DevOps more complicated—and what leadership can do about it.

Improved Trust and Collaboration Among IT Teams

Trust and collaboration form the base of true DevOps cultures. Anything an organization can do to make collaborating easier and eliminate unfair advantages between teams makes DevOps easier to facilitate. If half your team is working in the office and half is working remotely, that is not an even playing field; there are advantages to sharing an office and being able to communicate in-person that those who work remotely don’t have.

It’s also a matter of trust. Trust is not always assumed for employees working remotely. People in the office might feel skeptical that those at home just aren’t working as hard as they are. The advantage of everyone having to work remotely for the last several months is that we’re now all on a level playing field; we’ve all been in the same work-from-home boat. That has eliminated unfair advantages and helped boost trust among team members.

The more you trust each other, the easier it is to collaborate and delegate work, which removes some of the typical hurdles for DevOps projects.

Increased Use of Automation

Alongside trust and collaboration, automation is another key pillar of DevOps. In the past, teams may have been skeptical about automating their work, figuring it was just easier to do it themselves.

But, the massive spike in demand for IT services this year put a lot of pressure on traditional IT teams to get services up and running, provision machines and troubleshoot access problems, among other things. The only way to scale up to meet these demands without being overworked was to adopt continuous automation.

Now automation skeptics have seen for themselves the value of automating tasks to make work more manageable during unexpected digital activity peaks.

Growing DevOps Skills Gaps Around Automation, Kubernetes

The good news is, DevOps projects have been excelling in the new normal thanks to these accelerations in collaboration, trust and automation. But the flip side of more automation is a greater need to pick up new skills for making automation work.

Those working remotely without kids to look after might find more free time to spend on leveling up their automation skills, such as learning Python and other new script languages. But those who have to pull double-duty between their own job and other household tasks demanding immediate attention—such as homeschooling and looking after young children—might be less likely to have that opportunity. This could put many people at a long-term disadvantage as skills gaps continue to grow around automation. We may be looking at the possibility of certain demographics of employees (i.e. those who aren’t currently raising families or school-aged children) having more access to skills and educational opportunities than those who do.

This is also an issue with Kubernetes. Enterprise adoption of Kubernetes is growing rapidly as the container platform of choice for managing dynamic multi-cloud environments. But adoption is moving so quickly that even the teams who work with Kubernetes can find themselves behind the curve before they know it. This Kubernetes education gap is getting bigger every day, and the people who may have time now to keep themselves informed about Kubernetes will be at a greater advantage than those who just don’t have the time for after-work training.

The Role for DevOps Leaders in This Moment

For all the instability going on today, DevOps projects seem to generally be in a good spot, thanks to growing trust and collaboration between teams and a greater acceptance of the need for continuous automation. But as we start returning to normalcy and heading back to the office, leaders must not allow their teams to lapse back into old habits.

Leadership will need to come up with new rules of engagement for working in the office to eliminate unfair advantages that used to exist. For example, at my company, we’ve established that once offices are fully reopened, meetings will only be held in-person if everyone invited is present. If one person invited to the meeting is remote, then everyone will join remotely. Organizations should look to adopt similar rules, as the transition from working virtually to in-office is going to be a long and challenging one, with not everyone returning all at once.

In addition, organizations and leadership teams need to dedicate more official work time to training and educating employees on new tech such as automation and Kubernetes to even the playing field, develop more highly skilled teams and close these skills gaps. Team leaders must ensure that everyone is on the same page and that it’s not just those who had the most free time while quarantining who are coming out ahead.

The behaviors we’ve picked up during social distancing—new opportunities for continuous learning, wider adoption of automation, elimination of unfair advantages for remote workers—are making DevOps better. But these are practices we have to continue carrying forward in the new normal and beyond.

July 14, 2020
Digital Transformation Is More About a BizDevOps Culture than Tech

Digital transformation is top of mind for every organization now. The problem is, many of these organizations take the phrase too literally and think it’s all about adopting new technical capabilities and tools. Tech is certainly a part of it, but real digital transformation is also about changing company cultures and ways of working. Tech and culture go hand in hand for digital transformation; you can’t change one without the other.

Successful digital transformations mean redefining the way IT works, breaking down the silos between the development, production, UX and business teams in a way that cultivates greater collaboration, unlocks new business value and simply makes work better for everyone. All of that comes down to pivoting to a BizDevOps culture that creates more visibility, and positive reinforcement, for engineers and developers.

Building a Cultural Transformation Around BizDevOps

The goal for BizDevOps is to move development and production teams away from nursing applications and tediously staring at dashboards, and align them closely with the business side, so that they’re putting their time and energy toward delivering more business-relevant services. Shifting gears to a BizDevOps structure is its own culture transformation. Eliminating the silos between teams encourages better cross-collaboration and opens up new visibility for developers to see how their work is being received by users and impacting business value for the organization as a whole.

But, breaking down these silos to get engineers more engaged in production introduces another challenge: Those developers are already busy building products and features that they’re responsible for. Why would they want to get involved in production on top of all their other work? Overcoming this hurdle means making concerted efforts to give developers more visibility into production environments. More than that, it means exposing developers to more positive feedback.

Encouraging Engineers Through Constant Feedback

What makes engineers feel proudest about their work? When they see how it’s being used and enjoyed by their target audience. The problem is engineers rarely get that kind of feedback. The feedback that is typically sent back to the engineers are critical comments and complaints; they rarely see or hear the good things, just the problems they have to fix for the next iteration.

When you consider that, who could blame engineers for feeling discouraged or unwilling to want to take on more work? From their perspective, it’s not just about adding more to their plate, it’s also taking on more negativity and more criticism.

But it doesn’t have to be that way. What if engineers were provided with immediate feedback on all the new applications, features and services they were creating? Both the good and the bad, in real time? That’s part of the key to facilitating a BizDevOps culture: When engineers are seeing how the products they put out into the world are being enjoyed, as well as the positive impact they have on business value, they’ll naturally begin to feel more appreciated and also more reaffirmed that they are doing good, meaningful work and that they’re having an appreciable, positive impact on users and the whole business. This doesn’t mean putting on blinders to negative feedback, but it’s easier to act on negative feedback when it’s balanced out by positive feedback, too. Exposing engineers to more feedback about the impact of their work might also help inspire them to have new ideas and drive new innovations. The more information you’re armed with, the better position you’re in to innovate.

Organizations need to prioritize building easy ways for developers to have this visibility in their environments where they’re getting immediate feedback from end users, both the good and the bad. The more feedback they receive, the more encouraged by, and invested in, their work they become and, consequently, the more responsibility they feel for their contributions to the business. With that responsibility comes the culture change you need to digitally transform.

Top-Down Support

This culture change can’t happen without executive buy-in. The C-level needs to provide the necessary tooling and clarity on new organizational structures to break down silos between the development, operations, business and support teams, and then shepherd those teams into a more collaborative and visible environment. That support creates the foundation for a wider culture change, one that encourages a transition from operations to development and enables a bigger shift to BizDevOps.

Imagine the silos between teams as vertical walls. Now, shift those walls 90 degrees. That changes the structure so that your platform and autonomous cloud teams are sitting on the same level together, providing services across the organization, delivering software into production at faster speeds while simultaneously achieving visibility across the whole environment.

The democratization of data enabled by this level of visibility is helpful in another way. When all teams have access to the same data, and can see what each other is working on and how all that work moves toward a shared goal, it drives everyone to find new ways to make each team’s workload lighter–such as with automation. When teams are not just automating their own work, but automating each other’s because they see how lifting one group lifts the whole organization–and has a concrete impact on business value–that’s what BizDevOps is all about.

A Transformed Culture

In the end, every engineering team should think, feel and act like their own startup, responsible for everything from end to end. That spans from what they’re building, to how they implement it, to the impact it has on the end user and the business. Every engineering team needs to be working like this, keeping top of mind considerations like, how can a new application or feature or service be optimized so that it not only runs well, but also runs efficiently? Not just efficient in terms of performance, but in scalability as well.

Think about how cloud providers have potentially endless resources to deploy code. Just because something might run well doesn’t mean it’s running efficiently, and just throwing more code at a situation will inevitably raise costs and hurt overall business value. That’s something that engineers and developers should be keeping in mind as part of their work–not just how to deliver value in terms of new features or services, but how to bring down the total cost of ownership as well.

That’s what digital transformation looks like: A culture shift where new BizDevOps organizational structures, visibility across production environments and immediate access to feedback, both positive and critical, enables all teams to work together in a cohesive way, driven by what creates value for both the business and the end users.

April 29, 2020
Threading the Needle on Kubernetes Complexity with AI-Powered Observability

If you run down the laundry list of major industries–transportation, retail, healthcare, automotive, financial services–you’ll see that the major players in each have essentially become software companies. What they do and how they serve their customers rests entirely on their software working perfectly. Underpinning this is a massive digital transformation that has shifted organizations away from the old, static ways of enterprise IT monoliths. Instead, today’s businesses are working in more dynamic IT environments–agile, multi-cloud and, especially for new digital services, based on containers and microservices. All of which means embracing more IT complexity than ever before.

Kubernetes Brings More Data to the Table, but that Impacts Observability

A key part of this more complex environment is the soaring adoption rate of Kubernetes over the last few years. In a survey of 5,000 enterprises, 40% said they were currently running Kubernetes in production and over 80% reported that they’d adopted Kubernetes over any other container management system. Another recent survey revealed that 68% of CIOs are already using containers and a total 86% expect to deploy them over the next year. Enterprise IT is increasingly gravitating to containers and especially to Kubernetes for orchestrating their containerized applications.

What drives organizations toward containers and Kubernetes is the flexibility given to development teams to release services, features or apps faster. But faster doesn’t necessarily mean better or higher quality, unless you also apply cloud-native architectural patterns, include quality gates into your delivery pipeline, embrace new deployment models such as canary or feature toggling and provide full visibility and short feedback loops from production back to engineering.

Kubernetes itself provides great visibility into node or pod health. Kubernetes even ensures the health of your pods through its built-in health check mechanisms. Thanks to components such as service meshes, you also get live information about dependencies between services running in your pods. This data can, in theory, provide organizations with the necessary insights into their environments to understand whether their organization is releasing better value faster to their end users and unlocking new business value.

The reason I said these are theoretical benefits is because, in practice, all that extra data doesn’t necessarily mean more visibility. Often it can mean less visibility–and with that, less insight into how the different elements of your environment may be impacting speed of production deployments, overall system performance and the end-user experiences. I’ve seen Kubernetes environments that are packed with literally billions of interdependencies between applications, containers, microservices and clouds. Asking human beings to keep track of that level of data manually is an impossible task now. And relying on traditional, manual approaches to manage and monitor IT environments shortchanges the value that organizations should be getting out of Kubernetes.

Kubernetes Environments Need Full Stack Observability—that Means Needing Deterministic AI

Getting the full potential out of Kubernetes means having observability across the full technology stack and those billions of interdependencies. But as more organizations are discovering, that can’t happen without AI. Specifically, a deterministic AI engine. In fact, in a 2019 survey, 88% of CIOs said they were looking to AI as a critical solution for overcoming IT complexity. As long as Kubernetes is fueling complexity, overcoming that complexity means making deterministic AI an integral part of any Kubernetes environment.

A deterministic AI applies specific algorithms to detect system anomalies and apply its collected dependency information toward detecting the root causes behind those anomalies. When that deterministic AI engine ingests critical Kubernetes events, such as state changes and workload changes, it provides visibility into the Kubernetes stack’s interdependencies. That AI-powered, full stack observability matters because it gives IT new capabilities for their environments, including instrumenting and mapping containers, managing and allocating container resources and drilling down into container runtimes at work. All of that is possible due to the level of observability that you can only get with deterministic AI; traditional metrics and dashboards just can’t compete.

By providing full stack visibility and interdependency mapping across a Kubernetes environment, deterministic AI is empowering IT teams to identify, in real time, the root cause of degradations in business service availability–from reliability and resource usage to system performance and user experience–faster than they ever could manually. That level of automated speed and visibility also makes it possible to immediately resolve those issues at the source before they can impact the end user. The automation provided by deterministic AI also works its way into DevOps’ production pipelines, accelerating the release and feedback cycles of new deployments and allowing teams to more quickly incorporate user feedback into the next iteration for even better experiences.

In short, a deterministic AI engine makes life better for both IT and the customers they’re aiming to serve. Deterministic AI takes work that used to be time-consuming, tedious and counterintuitive when done manually, and performs it fast and automatically. That then frees up the teams that used to do that to focus on work that is more satisfying and drives new business value.

AI-Powered Observability Relies on Quality Data and Quality Monitoring

All that said, a deterministic AI’s ability to provide reliable, precise full stack observability in a Kubernetes environment relies on it being given consistently quality data it can process. The algorithms are dependent on the data that’s ingested by the AI engine. If there’s no standard monitoring capability being used across the board, then you invite a scenario where different teams bring their own preferred monitoring tools into the fold. The bigger a Kubernetes cluster gets, the more diverse the set of monitoring tools can be. And the greater the diversity of monitoring tools, the more inconsistent the data being fed into the deterministic AI engine.

The way out of this predicament is to leverage a single monitoring tool as a self-service capability into Kubernetes clusters, one that can support the diversity of technologies that are introduced into the stack and provide a baseline standard of system and event data that developers, IT administrators and the deterministic AI all lean on. In other words, when you feed consistent, quality data into the AI, you make it smarter. And a smarter deterministic AI engine provides the full stack Kubernetes observability that organizations need.

Building a Positive Feedback Loop Between Kubernetes, Deterministic AI and Observability

Kubernetes and deterministic AI are a natural fit. Kubernetes provides a platform for managing and instrumenting containerized applications. The deterministic AI provides a tool for mapping out the Kubernetes stack, offering insights into interdependencies and degradations where they arise and resolving them before they can affect end users. The data from Kubernetes makes AI better, and a smarter deterministic AI engine means more precise observability across your technology stack.

It’s a self-reinforcing feedback loop that eliminates the usual pitfalls of container-induced IT complexity. Instead, it creates new standards of visible, consistent and quality data that organizations can leverage for faster and better deployments–and in turn, better user experiences and greater business value.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

— Andreas Grabner

March 23, 2020