Category: Doin’ DevOps

  • BMC Mainframe Survey Results Show a Platform on the Rise

    BMC Mainframe Survey Results Show a Platform on the Rise

    What if I told you that the oldest active computing platform is also one of the most innovative, seen as a platform for growth and new workloads, not just by seasoned professionals, but by a new generation of stewards, too? The 2023 BMC Mainframe Survey shows just that. In addition to an overwhelmingly positive perception and growth mindset regarding the mainframe, the survey reveals that the majority of those who are most enthusiastic about its future are relative newcomers to the platform.

    Now in its 18th edition, the annual BMC Mainframe Survey gauges the opinions of more than 800 executive and technical professionals on a range of topics, from overall perception of the platform to top priorities, common practices and greatest challenges.

    This year’s results show that 94% of respondents have a positive perception of the mainframe, with 62% citing it as a platform to grow and attract new workloads. This tracks the 62% who report adopting DevOps practices on the platform. It’s no surprise, then, that two of the top five priorities cited by respondents are efforts to enhance automation and modernize applications. Organizations around the world recognize that the mainframe can be a leader in innovation, powering their digital transformations.

    Modern Management Approaches

    This innovation goes beyond the software and services developed for the mainframe to include those used to manage the platform. It’s reported that 35% of respondents listed the implementation of cloud technologies as one of their top four priorities for the coming year, with 41% saying they are considering cloud-based storage and backup. With the growing volume of mainframe data and the emergence of new regulations and standards governing data security and recovery, organizations are turning to cloud-based data management to reduce costs and improve recovery options.

    Year over year, the number of survey respondents reporting the use of AIOps hasn’t changed considerably, but those reporting its cross-platform use on both mainframe and non-mainframe systems has jumped 7% from 2021.

    Organizations are recognizing the need to take advantage of the greatest business values of AIOps — cited in the survey as improved uptime and the ability to detect problems quickly, understand trends and respond proactively — as part of a holistic approach to enterprise IT operations management.

    With increased cross-platform use, significant benefits and a decrease in the number of respondents reporting implementation challenges like cost, complexity and the need for specialized skillsets, BMC sees AIOps as an area ripe for significant growth in the coming year.

    Investment Pays Dividends

    Improving the availability, quality and efficiency of the mainframe, of course, requires investment. In our report on the results of the 2023 BMC Mainframe Survey, we compare the responses of maximizers (those planning to increase investment in the platform) with those of minimizers (those planning to decrease investment levels). The results are eye-opening.

    Maximizers have invested in future-proofing their workforce in a variety of ways, and the data bears that out; 81% have ten or fewer years of experience on the platform. The new generation of stewards sees the mainframe not as a legacy platform, but one of continued growth and innovation, worthy of the investment required not just to keep it modern, but to place it at the forefront of digital transformation.

    Another key difference between maximizers and minimizers relates to security, which was cited as the top overall priority of survey respondents for the fourth consecutive year. Maximizers are more likely than minimizers to have a dedicated resource to investigate security events and to have dedicated ransomware recovery capabilities in place. It should come as no surprise, then, that maximizers were 20% more likely than minimizers to report that their mainframe security controls for ransomware are extremely effective.

    These results show that the differences between maximizers and minimizers go beyond outlook, producing tangible effects on outcomes. The maximizers increasing their investment in the mainframe report fewer challenges with staffing and skills shortages and a stronger stance against increasingly sophisticated cyberthreats. The minimizers decreasing their investment in the platform not only have greater difficulty in maintaining a skilled workforce, which increases the risk of falling behind competitively, but it also puts their organizations at serious risk of potentially catastrophic security breaches.

    A Bright Future

    In short, the results of the 2023 BMC Mainframe Survey reflect a positive outlook for the platform’s future. By investing in their workforces and the tools required to advance and protect the platform while adopting modern techniques and practices, organizations can reduce risk and promote innovation that delights their customers.

  • Cutting Costs, Boosting Efficiency: The Economic Benefits of Framework-Defined Infrastructure

    Cutting Costs, Boosting Efficiency: The Economic Benefits of Framework-Defined Infrastructure

    In a world that’s constantly modernizing, developers are at the forefront of progress, embracing cutting-edge tools that match the pace. Industry-standard practices like infrastructure-as-code (IaC) are getting an update: Framework-defined infrastructure (FdI) is an approach to deploying applications that is quickly gaining popularity among developers, where the deployment environment automatically provisions infrastructure derived from the framework and the applications written in it. For developers, this serverless architecture means more reliability, increased productivity, top-of-the-line security and reduced costs.

    To modernize development practices, understanding the role of framework-defined infrastructure in defining and automating infrastructure—and its impact on the development experience in open source frameworks—is crucial.

    The Evolution From IaC to FdI

    Infrastructure-as-code (IaC) is the industry-standard practice for provisioning infrastructure in a repeatable and reliable way. Framework-defined infrastructure (FdI) is an evolution of IaC, where the deployment environment automatically provisions infrastructure derived from the framework and the applications written in it.

    In the early days of servers and the internet, we SSH-ed (or worse) into individual machines, running install scripts or even configuring production software using graphical user interfaces to provision infrastructure. As the web scaled, it became clear that this approach wouldn’t scale with it, and together with the DevOps movement, IaC became the best practice for provisioning infrastructure. The code in IaC represents a version-controlled description of the desired state of the infrastructure—which, when executed, will create the described system state. However, as the web continues to scale, developers must apply new processes and architectures to increase the efficiency of their workflows.

    Enter framework-defined infrastructure.

    Framework-defined infrastructure abstracts over cloud primitives such as servers, message queues and serverless functions, making them mere implementation details of the frameworks’ concepts. Frameworks use well-established patterns to provide structure and abstraction to applications, making them easier to write and understand by managing the high-level application flow while the developer writes code within the hooks provided by it.

    Framework-defined infrastructure takes advantage of both this inversion of control and the predictable structure of framework-based applications like Next.js to automatically map framework concepts onto the appropriate infrastructure without the need for explicit declaration or configuration of the infrastructure.

    Optimization for Serverless Environments

    Since framework-defined infrastructure lets the framework itself define the production behavior, local development can just use the framework’s own tooling, while production infrastructure is automatically configured to be a scaled and optimized representation of the same underlying behavior.

    IaC uses machine-readable files, which developers can version much like software, to provision infrastructure such as physical servers, virtual machines and higher-level services like databases or applications. Serverless architecture—which, despite its name, still uses servers—eliminates the need for developers to manage specific physical or virtual servers.

    Serverless products still require developers to define and deploy their respective serverless primitives, much like more traditional infrastructure. For example, in the archetype of serverless products, AWS Lambda, developers need to create a lambda function and deploy code to it.

    While serverless architecture greatly simplifies managing production infrastructure, the systems that run serverless code are often complex and proprietary, which can cause problems with local development. In a worst-case scenario, local development isn’t possible, and developers must test local changes by making comparatively slow deploys to the production infrastructure. In a best-case scenario, developers must create a local simulation of the complex serverless stack, which must stay up-to-date with the production environment. Framework-defined infrastructure completely eliminates this problem, as it is capable of automating the process for users.

    Framework-Defined Infrastructure and Immutable Deployments

    These days, application code is developed almost exclusively within version control systems such as Git. IaC maintains infrastructure definition in either the same or a sibling version control system. However, production infrastructure itself can only represent limited points in time of the infrastructure’s version history. This is fundamentally incompatible with framework-defined infrastructure, which expects that production infrastructure directly maps to the framework code at the same commit hash.

    Immutable deployments solve this impedance mismatch between an application’s and the infrastructure’s version history. Instead of deploying onto a limited number of infrastructures, immutable deploys create a completely new infrastructure for every single deployment that is ever made. For developers, this means lower IT complexity and failures, improved security, and easier troubleshooting, given there are fewer chances for unpredictable behaviors or unintended consequences of code changes.

    Naturally, immutable deployments are impossible in a world where infrastructure is mapped to finite physical resources. However, in a serverless world, where unused infrastructure can scale to zero, immutable deployments become possible to perform, as unused deployments do not take up physical compute resources beyond the baseline storage needed for their contents.

    Conclusion

    It goes without saying that framework-defined infrastructures rely on IaC as the ultimate infrastructure interface and, hence, could not work without the amazing work of its community. In order to modernize, it’s imperative for developers to leave behind infrastructure constraints and embrace a world where scaling applications becomes effortless, enabling them to optimize efficiency without sacrificing performance. For developers growing frustrated with scaling issues or the burdensome process of their development process, widespread adoption of framework-defined infrastructure is key to improving the developer experience.

  • Salesforce Winter Release: What you Need to Know

    Salesforce Winter Release: What you Need to Know

    With the Salesforce Winter release dates behind us, it’s time to evaluate all of the new innovations added to your Salesforce ecosystem. Let’s just say that many users will be excited about the features added to this release cycle; plus, there were many anticipated updates that have finally come to fruition.

    Let’s dive right in and get to the meat of the release that will appease the administrator community, automate internal processes and close many operation gaps, and promote more team collaborations for you and your clients.

    Search Capabilities and Admin Processes

    These updates make an admin’s day-to-day work more streamlined, automated and convenient when it comes to getting things done.

    ● Configure Searchable Objects: This update allows admins to determine searchable objects available by Profile. This will allow users to broaden their search capabilities of records and more quickly and easily find what they need. For example, you can search for a picklist value and see all the records with that chosen value. Users, especially those in larger organizations, have been wanting this as it will streamline their process for customers. *This is in beta.
    ● Refine Your Search Results With Sort: Another win in the Search department – now you can sort your searches by different attributes and column headers. A nice “sigh of relief” feature for users, allowing them to find what they need faster. *Created because of users’ input on IdeaExchange.
    ● Dynamic Forms on Standard Objects: Administrators will very much appreciate this update as it streamlines the admin process with enhanced page performance and optional visibility rules. This will be rolled out everywhere except in three areas: Campaigns, products and tasks (though we really look forward to when this will be available here.) *Created because of users’ input on IdeaExchange.

    AI

    Since Einstein was introduced to the Salesforce community, it has helped users work smarter and faster, which is why seeing some of these key updates come to light, as noted below, has been a game-changer for many.

    ● Ability to Report on Emails Captured by Einstein Activity Capture (EAC): When EAC was first introduced, it was a huge weight off of admins because there was no more logging and it helped synchronize all conversations in one place. Over time, you couldn’t access or report on the data because it lived outside of SF, so you couldn’t compare it to other data in Salesforce, i.e., how many conversations does it take to close a deal? However, with this new update, you can! This is an enormous feature gap that has been filled, and will ultimately save people time and money as many would outsource this work to third-party vendors. *Created because of users’ input on IdeaExchange.● Call Summaries Powered by Einstein: This is fully available, and just a generally cool feature where Einstein will record voice and video calls and transform them into summaries that are editable and include next steps and customer feedback.
    ● Conversation Sentiment Analysis: This feature goes hand in hand with the call summaries because now, customer success agents can listen and view sentiments of a conversation between agent and customer, distinguishing if they are happy, calm, disgruntled, etc. There’s a sentiment score and visuals provided for easy reporting as well. This is a great feature for agent performance grading and identifying coaching opportunities.

     

    Flows

    Flows continues to be a priority and focus for Salesforce, and they certainly delivered on giving users what they want with these improvements.

    ● Screen Flows With Reactive Components: Users and developers will very much appreciate this update as it reduces the number of screens they have to click through. With reactive components, the user interface can react to the user and what it puts on the screen. Previously, in order to use the choice a user selected, you’d have to click “next” for it to recognize and load the next screen. This could turn into three to four “next” clicks to get where you need. Now, the choice can be calculated on the fly and the dependent choice will appear below, without the need for extra click-throughs!
    ● Custom Error Messages in Record-Trigger Flows: What was once an automated feature, now puts the user in the driver seat. With a custom error message, you can now create targeted – and user-friendly – error messages for your customers and explain what went wrong and/or how to correct it. This is a big win in our opinion, which will not only save the end user headaches and blockers but will also save your admin or consultant time researching where and what happened. This is now generally available – during the Summer release, this was still in beta. *Created because of users’ input on IdeaExchange.
    ● Autolaunched Flows Based on Data Changes in Data Cloud: Based on a data model object or calculated insight object, you can start a data cloud-triggered flow. This allows you to apply flows to your data cloud, storing all of your business data, and of course Salesforce data too.

    Callouts

    It wouldn’t be fair to close out the review of this release update before highlighting a few noteworthy callouts that will surely resonate with certain users, which include:

    ● Automated Site Check-Ins and Check-Outs: For mobile workers, this new feature protects their privacy while also tracking the times they arrive and leave job sites. It provides automated timestamps for location-based actions and gives you a report based on those insights. This is being piloted so not fully available yet.
    ● Controlled Tabs and Sub-Tabs: This change is currently in beta, but will resonate with the day-to-day user as it makes it immensely easier to manage these tabs with the methods they provide.

    There’s a lot to experience and discover within the Winter ‘24 release, from increased workflow to automated operations; any user will be happy. But Salesforce also delivered for its users. Many of the ideas posted on IdeaExchange are now reflected in these new features, which just goes to show that Salesforce is listening. Let’s continue using these innovations and work with Salesforce to continue bettering the overall experience for us and our customers. We can only imagine what Salesforce will conquer with the Spring ‘24 release!

  • Transitioning Legacy Systems to DevOps: The XDR Support Paradigm

    Transitioning Legacy Systems to DevOps: The XDR Support Paradigm

    In today’s rapid and evolving technological landscape, DevOps emerges as a frontrunner in development methodologies, promising agility, efficiency and continuous delivery. While companies are keen to leverage its benefits, the transition from legacy systems can be fraught with challenges. Integrating extended detection and response (XDR) while making this transition can serve as a powerful tool to manage these challenges. In this article, we will delve deep into the potential pitfalls of this transition and explore the remedial measures with the aid of XDR solutions.

    The Legacy System Quagmire

    Legacy systems are often monolithic, siloed and heavily dependent on older technologies. According to a report by Forrester, a staggering 72% of corporate architectures still run on legacy systems. These systems, while operational, often lack the flexibility and scalability required for a DevOps-driven environment. They can become roadblocks, slowing down processes and becoming vulnerable points of failure.

    The Inherent Challenges

    Scalability Concerns: DevOps emphasizes scalability. However, legacy systems weren’t built with this level of dynamism in mind. Most legacy systems face scalability issues when introduced to DevOps practices.

    Security Vulnerabilities: The older tech stacks are more vulnerable to modern-day threats. Without continuous updates, these systems become an attacker’s paradise. This is where XDR solutions come into play, offering real-time threat detection and response across the infrastructure.

    Integration Hurdles: Legacy systems often struggle to integrate with modern tools used in DevOps, from container orchestration tools like Kubernetes to configuration management tools like Ansible.

    XDR to the Rescue

    Holistic Threat Detection: XDR’s primary strength lies in its comprehensive threat detection across networks, clouds, endpoints and applications. This compensates for the potential vulnerabilities of legacy systems, providing a safety net.

    Real-time Responses: XDR doesn’t just detect threats; it actively responds to neutralize them. As per a study by IDC, integrating XDR can reduce incident response times by up to 50%, ensuring that any security breach on legacy components is swiftly managed.

    Forensic Capabilities: Understanding the root cause of issues is paramount. XDR solutions come with forensic capabilities, allowing teams to understand the “how” and “why” behind incidents, an essential feature when dealing with older systems that may lack documentation.

    Migrating With Minimal Friction

    Training and Skill Development: One of the most underrated challenges is the lack of skills in managing and migrating legacy systems. Investing in training and reskilling can bridge this gap. A recent survey by DevOps Institute revealed that 72% of organizations that prioritized skill development found transitions, including to DevOps, smoother and more efficient.

    Pilot Programs: Before a full-blown transition, it’s wise to run pilot programs. This involves migrating a smaller, non-critical segment of the legacy system and integrating XDR support. The lessons learned here can guide the broader migration.

    Continuous Monitoring: Post-migration is not the time to be complacent. Using XDR’s monitoring capabilities, teams should continuously oversee the newly migrated segments for any anomalies or performance issues.

    Infrastructure-as-Code (IaC) and Legacy Systems

    Bridging the Gap with Automation: DevOps heavily leans on the principle of automation, and infrastructure-as-code (IaC) plays a pivotal role in achieving this. However, legacy systems often don’t align seamlessly with IaC principles. By using IaC platforms like Terraform or CloudFormation, organizations can create templates for legacy systems, making them more malleable for DevOps operations.

    Ensuring Compliance: One challenge with legacy systems is maintaining compliance, especially when regulatory standards evolve. IaC, when coupled with XDR’s continuous monitoring, can ensure that these older systems remain compliant. According to a 2021 report by Puppet, companies that adopted IaC methodologies witnessed a 60% faster compliance update implementation time on their legacy systems.

    Cultural Shifts and Organizational Dynamics

    The Human Element: While the technical challenges of migration are tangible, the human element is often overlooked. Employees used to legacy systems might resist transitioning to DevOps practices, fearing obsolescence or due to comfort with older methods.

    Change Management: Addressing this requires an organized change management strategy. Workshops, seminars and constant communication can assuage fears and foster a culture of continuous learning. A study by the Harvard Business Review highlighted that organizations that integrated change management strategies had a higher success rate in technical transitions, including shifts to DevOps models.

    Collaborative Approach: It’s essential to foster a collaborative environment where both legacy system experts and DevOps enthusiasts work together. XDR’s collaborative tools can be instrumental here, providing a platform where teams can discuss XDR solutions, share insights and collaboratively respond to threats.

    Conclusion

    Transitioning from legacy systems to a DevOps model is not a mere technical challenge; it’s a transformative journey for the entire organization. The challenges, while significant, are not insurmountable, especially with tools like XDR providing support. By understanding the pitfalls and leveraging advanced solutions, businesses can ensure a smooth transition and harness the power of DevOps to its fullest.

  • Policy-Based Governance: Modernizing Authorization for the Enterprise

    Policy-Based Governance: Modernizing Authorization for the Enterprise

    As we’ve moved away from on-prem environments to cloud-based ones and policy managed via code, we’ve inadvertently changed the way the infrastructure is managed. The infrastructure that used to be managed by IT/security teams has now shifted to being managed by the engineering operations teams. With that shift, engineering teams have found themselves now in control of new environments, including the access management part. This new change of control and management of access in the privileged environment requires a new approach to the solutions we use for privileged access governance.

    Shift to Cloud-Based Environments

    Cloud adoption has grown rapidly in the past few years, and while many organizations were already moving to the cloud from on-prem, the COVID-19 pandemic accelerated the transition, causing remote working to become the norm and changing the way responsibilities are managed.

    These organizations found they needed to adopt new ways of working to be able to support and provide critical services to off-site workers. As a result, more than 90% of organizations have adopted some form of cloud-based infrastructure, changing the way they work.

    The cloud is prevailing for tech stacks that need to elevate access governance over the cloud infrastructure. In fact, a whopping 94% of enterprises report they are using cloud services today, and 75% say security is a top concern.

    Cloud Shift, Worldwide, 2019 – 2025

    “The shift to the cloud has only accelerated over the past two years due to COVID-19, as organizations responded to a new business and social dynamic,” said Michael Warrilow, research vice president at Gartner.

    Shift to a Fast-Paced, Customer-Value Mindset

    Since switching to a new agile working environment, customers have gotten used to expecting very fast service. Instead of having a change management process that deploys to production once every few weeks, it’s now common to deploy hundreds of times to production every day.

    Access: required more often, more dynamically.

    Dynamic Changes: With the shift to the agile pace in which the new company works, access has been complicated a hundredfold. (Being able to support customers, fix bugs and being able to move very quickly. Needs to support these hundreds of changes to the customer’s product in their hand.

    Scale of Access has Increased: With the shift to cloud-based infrastructure, it’s now possible to create 5000 databases, for example, with a click of a button. All of them require access management because each one requires an understanding of who needs access and in what way.

    New Solutions are Needed

    The cloud providers, AWS, Azure and GCP, are now the way that people work. And most resources, whether databases or the machines that are running are doing so in the cloud environment. However, companies that shifted to the cloud are dissatisfied with what their legacy PAM solutions can provide them for the cloud resources. These solutions are ill-suited for their new needs. Instead, they need a solution that is agile and scalable to match their new working environment.

    With a perceived lack of options, companies are choosing to have the engineering team put in time to create new solutions. These solutions are meant to provide the following benefits: A frictionless way to gain permissions, an easy way to integrate with the tools their team uses today, and instead of manual provisioning and manual access requests, these solutions should automate those tasks.

    They are creating these in a variety of ways, including the following examples:

    1. Dev Portals: Dev portals are internal developer portals that act as a storehouse of information for the developers in a company. They work by centralizing the data in an internal portal, which offers developers and managers the ability to construct a system that facilitates monitoring of the software ecosystem, receiving alerts about alterations, and visually representing interconnected details. This portal additionally simplifies the task of overseeing diverse facets of the software development lifecycle, encompassing aspects like microservices interconnections, ownership statuses, dependencies, resource allocation, and beyond. By utilizing these, companies are delivering and gaining permissions to cloud environments.

    2. Slack Bots: Many times, a developer will need elevated access to resources or databases, so they need an easy way to request and be granted the permissions they need when they need them.

    3. Terraform Automation: Terraform is an infrastructure-as-code tool that lets you build, change, and version cloud and on-prem resources safely and efficiently. Terraform policies and infrastructure code/policies could also be leveraged to define access control policies. The way it works is that engineers or support teams are required to request a change in the Terraform code in order to gain the access they need. If there is a Terraform code in place describing the policy of groups to role access to the relevant infrastructure they would like to gain access to, they can request that access by requesting a Terraform code change. They would need to understand which parts of the code to change and submit a “pull request” to make the desired change which will resolve in the permissions change. Once a Terraform code change is approved, it changes the policy of access to that infrastructure which will change the permissions that the group has.

    (a) Static: The Terraform state is static, meaning that permissions granted in this way are essentially meant to be long-lasting, standing permissions. In addition, these permissions are usually granted to a role and are very coarse-grained(not granularly defined, as they need to suit the whole role’s needs). This creates an environment of over-privileges. That is why, in many cases where the permissions should be granted to a specific task or person or for a limited time, teams choose to use one-off permission granting to manually grant the permissions. However, these one-off permissions create drifts in the environment essentially if they are not revoked. And since there is no process in place for these one-off just-in-time (JIT) permissions, the revocation needs to also be manual and needs to be remembered to be done.

    (b) “Drifts”: (A drift in Terraform is when the environment isn’t accurately reflected in the Terraform policy). Therefore, by changing the access outside of Terraform, you’re creating a drift because it doesn’t match up correctly, making it a poor answer to one-off permissions. For anything needing high privileges, you want it to be dynamic instead of remaining as part of Terraform.

    4. Jira Service Management With Your Cloud Provider: Integrating Jira with GCP (as a cloud provider example), allows the IT security team the ability to automate servicing access tickets by creating forms where teams can request a certain role or be granted access to a certain project.

    5. Manual JIT Cloud Role: This is a method that allows self-serve permissions to engineers to get the permissions they need on demand. The role needs to be predefined, and essentially, as long as the engineer is part of the group, he or she will be able to gain those privileges on-demand. This reduces the standing privileges in the environment at any given point; however, roles granted are usually for a generic cloud admin rather than being granular, as it’s hard to determine ahead of time exactly what permissions will be needed.

    Problems With DIY Solutions

    Because of the lack of the right tool, companies are spending expensive resources creating their own solutions for access control. Team members know they need something that leverages cloud capabilities rather than their PAM for on-premises. These solutions are time-consuming; companies need to build then maintain the infrastructure. They are also very limited in what they can do since they are not the company’s main concern.

    Policy-Based Governance

    With the growing adoption of cloud-based tools, it’s more important than ever that companies learn to adapt. Engineers have now found themselves in control of permissions in new environments they were not looking to manage. It’s clear that a new approach to access governance is necessary.

    The quickest and most sure-fire way to do this is by utilizing tools deliberately made for the new cloud-based era that address the challenge in a way that is suited to the new, fast-paced, dynamic cloud environment. These new solutions are policy-based and leverage the fact that with the shift to new ways of managing infrastructure, it’s necessary to give the IT and Operations teams more control, granularity and automation.

    Platforms such as Apono for cloud-based permissions management allow organizations to benefit from centralized permissions management, enforce dynamic role-based access controls, monitor and audit privileged user activities and mitigate security risks effectively.

    Final Thoughts

    In addition to the shift we have witnessed in the control of the environment and access governance, organizations are also shifting to delegating more ownership to the engineers themselves. This means providing engineering teams more control of their code — all the way to production. However, with that comes the need to give them only granular control over the environment and provide access management to their specific scope without creating a burden on the teams in managing/granting the access. This leads the operations teams to search for solutions to address the need for access management delegation in the environment. New solutions and platforms addressing the new environment’s privileged permissions management challenge need to take that into consideration as well.

    As a final note, I believe the shift in mindset occurred a few years ago already. However, we are only now realizing some of its effects. One of them is the shift in privileged access challenges and all that comes with that. Seeing the many internal solutions that small-to-large companies have created for themselves gives us an understanding of how this evolved market isn’t being addressed with the same platforms that have existed for well over a decade.

  • Evolution of Toil Identification and Mitigation – A Classic SRE Challenge

    Evolution of Toil Identification and Mitigation – A Classic SRE Challenge

    Ingo Averdunk, leading architecture and solutions for cloud service management and site reliability engineering for IBM Cloud, contributed to this article.

    Christine White, architect at IBM Consulting, also contributed to this article.


    When many people think of operations, they often associate it with manual, labor-intensive work. This is because operations traditionally encompass a wide range of activities that involve managing and executing tasks related to producing, distributing and maintaining goods and services. Information technology (IT) is not different: Manual work has traditionally been a significant component of various tasks and processes. Examples are server maintenance, software installation, troubleshooting and network configuration, which historically required manual intervention and human expertise. Much of the work is inherently repetitive by nature, time-consuming and prone to errors. We call this work “toil”, the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value and that scales linearly as a service grows.

    Toil

    This toil has always been there. And toil leads to burnout, increased errors and reduced scalability. Only how we have addressed it has changed. While toil remains, toil mitigation has evolved.

    In the beginning, toil was addressed through labor arbitrage. Between 2000 and 2010, we leveraged labor arbitrage to reduce the cost of manual work by outsourcing and out-tasking activities to near-shore and later off-shore countries. When there is a problem, the person who can solve it steps up. Then, to manage it better, we started to do workload or work item management. You have systems, like prioritized backlogs, and we started putting a structure on how we solve these problems. But still, it was all resource-based; everything was resource-intensive. And that is why the engineers became more of the subject matter experts (SME). Being a subject matter expert took time, and being critical to operations became exhausting. And the repetitive manual activities required took time and effort they didn’t have.

    As issues became repetitive, we started thinking about scripting. We evolved from a complete labor arbitrage mentality to writing spot scripts. Engineers began to use scripts to solve a small part of the most oppressive problem.

    Tools started to be developed to enable and automate some of the work. Organizations purchased tools, and education of the tools commenced. It became important to be an SME in your area and an SME of the tools in your area. However, the tools were specific to certain functions and features that they provided, and the software development life cycle (SDLC) had a lot of nuances that many of the tools did not capture. So even when the tools helped, it created this mass of white spaces that could not be integrated easily. Tools were very format-specific and didn’t have the flexibility end to end integration. Newer problems emerged as formats were different, speeds were different, and other differences manifested. Even as this initial approach was an improvement over the labor arbitrage mentality and reduced the amount of manual effort required, it often resulted in complex and brittle systems that were difficult to maintain and debug.

    Automation

    From 2015 to 2020, automation became more prevalent. On one side, the cost levers of labor arbitrage were reduced, and due to low skill, quality suffered. Thanks to technological advancements, automation in various flavors became a viable alternative: Runbook automation, robotic process automation, etc.

    Aligning tool isolation with SDLC nuances required an end-to-end automation philosophy. We need to do automation across the board. Toil elimination was a goal of end-to-end automation, and that obsession led to SRE thinking and the principle of capping operational load. An SRE spends only 50% of his time on operational activities, and the other 50% on “engineering” (improvements): Identification and elimination of problem areas, improving and integrating tooling, etc. And thus, the genesis of the next wave of reliability engineering principles began.

    Parallel to this, adopting DevOps and Agile practices as well as cloud computing enabled IT teams to define and manage infrastructure and application configurations through code, allowing for automated provisioning and scaling. Applications are designed for reliability and instrumented for observability, providing insights into the production use. Traditional organizational barriers are broken down and teams cooperate across the entire SDLC with a shared goal that balances velocity and quality.

    The new focus on leveraging cloud-native technologies and practices to build scalable, flexible and resilient systems allowed SRE teams to adopt cloud-native technologies such as Kubernetes, containers and serverless computing to build systems that are designed to be highly available, fault-tolerant and easy to maintain.

    Toil mitigation and the SDLC lifecycle have evolved from reactive handling of problems to a proactive approach to operations, where alerting is maturing, and other sophisticated instrumentation is implemented. A proactive approach underlined the importance of observability and monitoring. SRE teams are using observability tools such as Prometheus, Grafana and New Relic to monitor their systems in real-time. The ability to identify potential issues before they become incidents emerges.

    AI Infusion

    The next step is being more prescriptive and predictive. That calls for a good bit of AI infusion. We have historical knowledge management systems that will tell us what went wrong, how it went wrong, how it got resolved and how long it took. We can augment that knowledge base with publicly available and curated knowledge.

    Starting in 2020, AI and machine learning are being used in IT for tasks such as anomaly detection, predictive maintenance, risk assessment and automated incident response, reducing the manual effort required for monitoring and maintaining systems.

    AI systems can learn from all this and not only predict issues, but when issues happen, they also auto-resolve. An extension of AI, Generative AI, will disrupt this event more. Prediction of a problem will also lead to the generation of executable runbooks and playbooks, which are agnostic to specific types of automation or automation tools. New challenges such as data quality, model training and explainability have arisen simultaneously and human intervention is still required until we can reach a certain comfort level. There will need to be a manual effort to look at Generative AI proposed solutions, approve them and prove the solutions in lower-level environments, before applying them into production environments.

    In the next 18 – 36 months, we will see that morph into risk-taking, where Generative AI will become digital operations. If we look at an Ops team or a SRE team, there are different personas, and we firmly believe that those personas will be assisted by an AI bot. These bots will participate in daily standups, and they will learn. They will participate in issue-resolution processes, and they will learn. They will start participating in retrospectives, and they will learn. This “AI persona” will become a digital worker in a highly integrated digital team. The manual tasks that once were solved through AI are now becoming complex decision-making processes. We are entering the 4th wave of automation improvements using Generative AI.

    Be Prepared

    Just as the invention of fire, tires, the printing press, the telephone and the lightbulb, what we do with this advancement will greatly shape our future. We must be prepared. And the best way to be prepared is to upskill our practitioners with newer technology skills like AI. The upskilling needs to include all the core and peripheral benefits and concerns to help them manage and improve upon the current day. We need to clarify roles amongst practitioners so there is not an “I’m being replaced by a bot” mentality, and most importantly being open-minded about the possibilities, building on those possibilities and embracing risk rather than fighting it!