devopsy.kinsta.cloud

Category: Blogs

  • Evolution of Toil Identification and Mitigation – A Classic SRE Challenge

    Evolution of Toil Identification and Mitigation – A Classic SRE Challenge

    Ingo Averdunk, leading architecture and solutions for cloud service management and site reliability engineering for IBM Cloud, contributed to this article.

    Christine White, architect at IBM Consulting, also contributed to this article.

    When many people think of operations, they often associate it with manual, labor-intensive work. This is because operations traditionally encompass a wide range of activities that involve managing and executing tasks related to producing, distributing and maintaining goods and services. Information technology (IT) is not different: Manual work has traditionally been a significant component of various tasks and processes. Examples are server maintenance, software installation, troubleshooting and network configuration, which historically required manual intervention and human expertise. Much of the work is inherently repetitive by nature, time-consuming and prone to errors. We call this work “toil”, the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value and that scales linearly as a service grows.

    Toil

    This toil has always been there. And toil leads to burnout, increased errors and reduced scalability. Only how we have addressed it has changed. While toil remains, toil mitigation has evolved.

    In the beginning, toil was addressed through labor arbitrage. Between 2000 and 2010, we leveraged labor arbitrage to reduce the cost of manual work by outsourcing and out-tasking activities to near-shore and later off-shore countries. When there is a problem, the person who can solve it steps up. Then, to manage it better, we started to do workload or work item management. You have systems, like prioritized backlogs, and we started putting a structure on how we solve these problems. But still, it was all resource-based; everything was resource-intensive. And that is why the engineers became more of the subject matter experts (SME). Being a subject matter expert took time, and being critical to operations became exhausting. And the repetitive manual activities required took time and effort they didn’t have.

    As issues became repetitive, we started thinking about scripting. We evolved from a complete labor arbitrage mentality to writing spot scripts. Engineers began to use scripts to solve a small part of the most oppressive problem.

    Tools started to be developed to enable and automate some of the work. Organizations purchased tools, and education of the tools commenced. It became important to be an SME in your area and an SME of the tools in your area. However, the tools were specific to certain functions and features that they provided, and the software development life cycle (SDLC) had a lot of nuances that many of the tools did not capture. So even when the tools helped, it created this mass of white spaces that could not be integrated easily. Tools were very format-specific and didn’t have the flexibility end to end integration. Newer problems emerged as formats were different, speeds were different, and other differences manifested. Even as this initial approach was an improvement over the labor arbitrage mentality and reduced the amount of manual effort required, it often resulted in complex and brittle systems that were difficult to maintain and debug.

    Automation

    From 2015 to 2020, automation became more prevalent. On one side, the cost levers of labor arbitrage were reduced, and due to low skill, quality suffered. Thanks to technological advancements, automation in various flavors became a viable alternative: Runbook automation, robotic process automation, etc.

    Aligning tool isolation with SDLC nuances required an end-to-end automation philosophy. We need to do automation across the board. Toil elimination was a goal of end-to-end automation, and that obsession led to SRE thinking and the principle of capping operational load. An SRE spends only 50% of his time on operational activities, and the other 50% on “engineering” (improvements): Identification and elimination of problem areas, improving and integrating tooling, etc. And thus, the genesis of the next wave of reliability engineering principles began.

    Parallel to this, adopting DevOps and Agile practices as well as cloud computing enabled IT teams to define and manage infrastructure and application configurations through code, allowing for automated provisioning and scaling. Applications are designed for reliability and instrumented for observability, providing insights into the production use. Traditional organizational barriers are broken down and teams cooperate across the entire SDLC with a shared goal that balances velocity and quality.

    The new focus on leveraging cloud-native technologies and practices to build scalable, flexible and resilient systems allowed SRE teams to adopt cloud-native technologies such as Kubernetes, containers and serverless computing to build systems that are designed to be highly available, fault-tolerant and easy to maintain.

    Toil mitigation and the SDLC lifecycle have evolved from reactive handling of problems to a proactive approach to operations, where alerting is maturing, and other sophisticated instrumentation is implemented. A proactive approach underlined the importance of observability and monitoring. SRE teams are using observability tools such as Prometheus, Grafana and New Relic to monitor their systems in real-time. The ability to identify potential issues before they become incidents emerges.

    AI Infusion

    The next step is being more prescriptive and predictive. That calls for a good bit of AI infusion. We have historical knowledge management systems that will tell us what went wrong, how it went wrong, how it got resolved and how long it took. We can augment that knowledge base with publicly available and curated knowledge.

    Starting in 2020, AI and machine learning are being used in IT for tasks such as anomaly detection, predictive maintenance, risk assessment and automated incident response, reducing the manual effort required for monitoring and maintaining systems.

    AI systems can learn from all this and not only predict issues, but when issues happen, they also auto-resolve. An extension of AI, Generative AI, will disrupt this event more. Prediction of a problem will also lead to the generation of executable runbooks and playbooks, which are agnostic to specific types of automation or automation tools. New challenges such as data quality, model training and explainability have arisen simultaneously and human intervention is still required until we can reach a certain comfort level. There will need to be a manual effort to look at Generative AI proposed solutions, approve them and prove the solutions in lower-level environments, before applying them into production environments.

    In the next 18 – 36 months, we will see that morph into risk-taking, where Generative AI will become digital operations. If we look at an Ops team or a SRE team, there are different personas, and we firmly believe that those personas will be assisted by an AI bot. These bots will participate in daily standups, and they will learn. They will participate in issue-resolution processes, and they will learn. They will start participating in retrospectives, and they will learn. This “AI persona” will become a digital worker in a highly integrated digital team. The manual tasks that once were solved through AI are now becoming complex decision-making processes. We are entering the 4th wave of automation improvements using Generative AI.

    Be Prepared

    Just as the invention of fire, tires, the printing press, the telephone and the lightbulb, what we do with this advancement will greatly shape our future. We must be prepared. And the best way to be prepared is to upskill our practitioners with newer technology skills like AI. The upskilling needs to include all the core and peripheral benefits and concerns to help them manage and improve upon the current day. We need to clarify roles amongst practitioners so there is not an “I’m being replaced by a bot” mentality, and most importantly being open-minded about the possibilities, building on those possibilities and embracing risk rather than fighting it!

  • Container Image Tags

    In a Kubernetes environment, you encounter a situation where a service’s configuration is not consistent across multiple clusters. What common configuration element could be the source of this issue?

    1. Pod resource requests and limits
    2. Container image tags
    3. Ingress controller settings
    4. Service account permissions
    5. Node affinity rules

    Correct Answer: 2) Container image tags

    154 people answered this question with 20% getting it right.

    In a Kubernetes environment, maintaining consistency across multiple clusters is a vital component of reliable and efficient operations. When you encounter a situation where a service’s configuration is not uniform across these clusters, identifying the root cause is key. In this blog post, we’ll delve into the common configuration element that could be the source of this issue and explain how to address it.

    The Role of Configuration Consistency

    Consistency in configuration ensures that your applications behave predictably and reliably in any Kubernetes cluster, regardless of its location or characteristics. It simplifies management, monitoring, and scaling while reducing the likelihood of unexpected issues. When configuration discrepancies arise, they can result in erratic behavior and operational headaches.

    Container Image Tags: The Likely Culprit

    Container image tags (Answer 2) are a common source of configuration discrepancies. Image tags specify the version or specific build of a container image to use. They serve as a fundamental element in the deployment of applications in Kubernetes. When different clusters reference different image tags, it can lead to several issues:

    1. Version Mismatch: Each image tag represents a specific version of the application. If clusters use different tags, they might run different versions of the application, leading to inconsistent behavior.
    2. Security Concerns: Different image tags may have varying security vulnerabilities or updates, exposing your clusters to potential risks.
    3. Operational Complexity: Managing different image tags across clusters increases operational complexity, making it challenging to roll out updates and troubleshoot problems.

    Addressing the Issue: Container Image Tags

    Let’s explore how you can ensure consistency by addressing the container image tag issue. Below are some code samples and steps for clarification:

    Step 1: Define a Consistent Image Tag

     To achieve consistency, define a consistent image tag for your application across all clusters. This tag should represent the desired version of your application. For example, if you’re using a Docker image, here’s how you specify the image tag in your Kubernetes Deployment:

    Unset
    apiVersion: apps/v1 kind: Deployment
    metadata:
    name: my-app spec:
    template:
    spec:
    containers:
    - name: my-app-container
    image: my-app:v1.0.0 # Use a specific version tag

    Step 2: Automate Image Tag Updates

    To avoid manual updates and ensure that your clusters are always using the correct image tag, consider automation. You can utilize Continuous Integration/Continuous Deployment (CI/CD) pipelines to automate the image build and deployment process. In your CI/CD script, you can set the image tag dynamically based on your application version:

    Unset
    # Example CI/CD bash script APP_VERSION="1.0.0" # Your application version IMAGE_TAG="my-app:$APP_VERSION"
    docker build -t $IMAGE_TAG . docker push $IMAGE_TAG

    Step 3: Implement a Release Strategy

     Use a consistent release strategy that defines when and how new versions of your application are deployed to different clusters. For example, you can use rolling updates to minimize disruption during deployments. Here is an example of a yaml:

    Unset
    apiVersion: apps/v1 kind: Deployment
    metadata:
    name: my-app spec:
    strategy:
    type: RollingUpdate rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25% template:
    spec:
    containers:
    - name: my-app-container
    image: my-app:v1.0.0 # Consistent image tag

    By adhering to these best practices, you can ensure that your Kubernetes clusters remain consistent in terms of container image tags, preventing configuration discrepancies and promoting stability.

    Conclusion

    In the intricate world of Kubernetes, maintaining consistency is paramount. When a service’s configuration varies across multiple clusters, it’s often the container image tags that are the culprits. By establishing a consistent image tagging strategy, automating updates, and implementing a release strategy, you can ensure that your clusters operate smoothly and predictably, regardless of their location or size. This consistency ultimately paves the way for efficient management and scaling, reducing the likelihood of unexpected issues and contributing to the reliability of your Kubernetes environment.

    But there are always exceptions. And when there is a difference in config across clusters use Webb.ai to identify the difference(s). Make Webb.ai part of your runbooks.

  • How to Navigate App Modernization Strategies for the Cloud

    How to Navigate App Modernization Strategies for the Cloud

    In today’s fast-paced digital landscape, organizations continuously seek ways to innovate and enhance customer experiences while optimizing efficiency. One of the most transformative strategies is modernizing applications for the cloud. App modernization isn’t just a technology upgrade. It’s a holistic business transformation. The cloud’s elasticity and scalability present opportunities to create deeper customer engagement, foster seamless partner integration and unlock valuable data insights. However, this requires a thoughtful approach that aligns with business goals and technological aspirations.

    Let’s look at some best practices for modernizing apps for the cloud and navigating potential challenges.

    The App Modernization Process

    It’s important to determine whether the cost of migration will lead to better efficiency and customer engagement that will yield an increase in return on investment (ROI). This approach allows businesses to address current issues and future-proof their data stacks.

    After assessing the cost and ROI considerations, organizations need to follow a systematic process to successfully modernize an app for the cloud. The first step is to evaluate the existing app’s strengths, weaknesses and areas for improvement. It’s crucial to assess an app’s compatibility with cloud platforms and set clear modernization goals.

    Following the assessment, design a cloud-native architecture that aligns with the cloud’s scalability, elasticity and distributed nature. This may involve rearchitecting the app to include such cloud-native components as microservices, containers, or serverless computing. Once the architecture is in place, migrate the app’s data from on-premises or legacy databases to cloud-based storage solutions to ensure data availability, scalability and ease of management in the cloud environment.

    The next step involves rewriting or modifying the app’s codebase to take advantage of cloud services and APIs. Specific modules or functionalities may need to be re-engineered to enhance performance, scalability and integration capabilities. Perform comprehensive automated load and security testing to verify reliability and performance in the cloud.

    Once the modernized app has been tested, deploy it on the selected cloud platform. Implement monitoring and logging mechanisms to gain insights into the app’s behavior, performance and user experience so it operates effectively in the cloud environment.

    Navigating App Modernization Challenges

    While the benefits of app modernization are significant, challenges can arise during the process.

    One of these hurdles involves dealing with legacy constraints. Legacy applications often have complex architectures that complicate modernization. Careful planning is required to update outdated technology stacks and move from a tightly coupled system to a loosely coupled one.

    Another obstacle pertains to data migration complexities. Transferring large amounts of data from on-premises systems to the cloud demands meticulous planning and execution to maintain data integrity, security and minimal downtime.

    Integration presents another potential difficulty. Modernized apps often need to integrate with other systems, both on-premises and in the cloud. Many cloud vendors offer tools and programs to aid in migrating data. These, coupled with credits to be used as apps are migrated, can significantly reduce costs.

    Security and compliance are critical considerations when moving to the cloud. Cloud environments require robust security measures to protect sensitive data, and organizations must comply with relevant regulations and industry standards.

    Effective change management strategies are another aspect to consider to ensure a smooth transition for users and stakeholders. App modernization is a business process modernization representing a significant change in business operations.

    In-House Vs. Outsourced App Modernization

    Choosing between in-house and outsourced app modernization depends on several factors.

    First, if an organization has skilled developers with cloud expertise, handling app modernization in-house is a viable option. On the other hand, if a team lacks the necessary skills, it may be more practical to outsource the task to a cloud service provider or a specialized development partner. It should be noted that if the development work is outsourced, it’s important to keep the app business owners involved to ensure that the modernized solutions yield the expected benefits.

    Another consideration is the availability of resources, such as time, budget and staffing. App modernization projects require dedicated efforts, and outsourcing becomes appealing if internal resources are limited. By entrusting the project to external professionals, organizations can ensure it receives the attention and resources needed without burdening the internal team.

    Evaluating whether app modernization aligns with an organization’s core competencies is also essential. If the application is not directly related to the core business, outsourcing can free up internal resources and redirect them toward more strategic initiatives. This allows an organization to concentrate on areas in which it has a competitive advantage and expertise while still benefiting from the app modernization expertise of external partners.

    Pioneering the Future

    Modernizing an app for the cloud is a transformative journey that holds the potential to enhance customer experiences, optimize processes and unlock valuable insights. Navigating challenges through meticulous planning, robust testing and adherence to standards are crucial, whether opting for in-house expertise or external collaboration. By embracing cloud-native principles and using advanced technologies, organizations can position themselves at the forefront of innovation, ensuring lasting success in the dynamic landscape of the digital era.

  • Improvement Cycles and AI

    Improvement Cycles and AI

    It’s that time of year when pundits start to crank out their predictions for the next twelve months! I’ve been asked to contribute to a private group (and occasionally, Techstrong Group asks me to contribute to their annual list), so I won’t thrill you with my predictions just yet; but I will discuss the cycles and trends that previous predictions have shown.

    Back when I was testing physical servers, we thought of throughput bottlenecks in terms of where the problem was. The circle would go disk->memory-> CPU, and in the case of things like NAS, insert network in there somewhere, too. When one bottleneck was resolved, another would crop up. And users will always seem to be pushed to the next bottleneck.

    We see similar cycles in DevOps. First, Agile made development faster, and that caused us issues throughout the rest of the process. Then, DevOps helped with the build and a little with the testing and deployment steps, but in the early days it was more standardization than speeding up. And since then, we have been in that cycle. We’re seeing GitOps speed up and automate deployment and upgrades—based upon the automation that came before—we’re seeing test and security tools feeding directly into the IDE to speed resolution, we’re seeing container automation cause “cloud-first development” (I hate that name; Kubernetes has cloud-like interfaces, but is not precisely cloud) to be fully automated, and thus faster.

    And we’re looking at that cycle continuing. No doubt you will hear a lot of predictions that start with “AI …” For the most part, AI is a tool and/or technique. Listen for what it is improving because it is doing very little that’s actually new. (Delete list of things that sounded alarmingly like those predictions I said I wouldn’t make.)

    All in all, we are still living in a very cool technological time. We can have a tool generate code snippets of the dull work for us so we can focus on the important bits. Some shy away from this use of “AI,” but honestly, we already use libraries to do the grunt work—I haven’t written to a screen buffer in forever—because we don’t have to. Same story—use the tools that make you productive. Some of the more obscure bits of Linux command lines are far better dealt with by asking an AI to give me examples than searching through tons of docs across several platforms. Never was much of a fan of man output for complex commands, this makes it easier. The nice bit is that bash and UNIX utilities don’t change much over time, so answers can be a bit dated and perfectly accurate. That’s not true for much of IT.

    The environment has consistently gotten more complex, while the building blocks have gotten easier to use. This allows things to keep moving and changing while it offers us better ways to serve the organization’s needs. This does change at some point—infinite change is indeed a thing, but any technology does have its limits—but I honestly can’t see the point at which we are content with our IT tools and stop looking for more. Oh, most organizations go through those cycles—organizationally, once an infrastructure and process works correctly, it is likely to hang around until better is found—but the industry will keep changing.

    And you’ll keep learning and changing. That’s one prediction I will make for next year. Teams and IT staff will grow and learn new things, because at its heart, that is what we do. We learn new things and keep the lights on. In good years, we offer sweet new tools and functionality to the company or our customers. But either way, we keep things online. Year after year. You all rock, so that’s an easy prediction.

  • EP 39: DevOps Building Blocks Part 2: Power Up Your Automated Testing

    EP 39: DevOps Building Blocks Part 2: Power Up Your Automated Testing

    Automation is critical to DevOps success. Test automation accelerates and simplifies the software development process, empowering DevOps teams to deliver high-quality, secure and reliable products to market faster without sacrificing crucial debugging and troubleshooting processes. Automated testing minimizes the chance of human error, removes redundancy and increases collaboration between DevOps and QA teams. On the other hand, over-automating can also be counterproductive. So how much test automation is too much? How do you find a balance? And what tools should you use? Join host Mitch Ashley and a panel of distinguished DevOps and QA leaders Tracy Ragan (DeployHub), Martin Klaus (Tricentis) and Raju Chavan (T-mobile) to discuss:
    – How automated testing empowers DevOps teams
    – How to incorporate automated and continuous testing into your DevOps workflow
    – How to scale automated testing
    – How to choose the right tools to make test automation smooth and easy