devopsy.kinsta.cloud

Tag: it ops

  • IT Ops Trends: Cloud, Containers and COVID

    IT Ops Trends: Cloud, Containers and COVID

    Almost all companies are modernizing their IT infrastructure to remain competitive, with the focus firmly on migration to the cloud and the use of dynamic infrastructures. In a recent Trends in IT Ops 2021 survey, more than 160 IT professionals were asked how the implementation of IT strategies affected their day-to-day work. The results provided an impressive insight into the current work priorities of system administrators and IT Ops in general.

    According to the survey, most administrators believe that monitoring cloud (79%) and container (73%) environments will take on a more important, or much more important role for them in the future. Additionally, 59% of those surveyed agree that their job role has evolved over the years due to cloud computing and containerization.

    Not all IT Teams are Equal

    However, the data also shows that the changes are not affecting every IT team equally. For example, 37% of all respondents say that cloud solutions are not relevant—or less relevant to them on a day-to-day basis. This response shows that on the one hand, system administrators see cloud computing as an essential topic. However, on the other, the impact in practice is not felt to the same extent by every administrator. These responses show that not every administrator has to deal with cloud platforms on a day-to-day basis, with some being ‘siloed’ into dedicated teams that may not interact with the broader corporate cloud strategy. In addition, not every organization has fully committed to the cloud, with some finding their migration hindered by applications that are not suited to cloud migration.

    In the survey, 45% of respondents also said they believe the importance of monitoring physical infrastructure will not change in the next few years. However, 43% believe that monitoring physical systems will become more important or much more important. Conversely, a mere 1% replied that they thought physical infrastructure monitoring was less critical. 

    Taken together, these responses demonstrate that, despite the excitement around the cloud, many organizations take a hybrid approach to infrastructure. IT Ops and administrators recognize the importance of monitoring cloud systems and dynamic assets while, at the same time, the monitoring of local systems remains relevant or is even expected to gain in relevance.

    Network Management and Automation

    Cloud and containers enable faster growth of IT environments and promote digitalization. However, they also lead to a sharp increase in data traffic and more complex networking of systems. As a result of these changes, the demands on local networks are growing. This increasing demand demonstrates the importance of network management tools. 81% of respondents said that solutions in this area are relevant, or very relevant to them in their day-to-day work.

    This year, the move to a home office joined the trend towards migration to the cloud, containers, and network monitoring. In the wake of the COVID-19 pandemic, many IT departments had to adapt their infrastructure to allow employees to work from home offices. However, most administrators were relatively stress-free, with 54% saying they had no trouble adapting infrastructure to meet remote work requirements.

    Automation is also a key issue. For example, 81% of all respondents say projects that deliver deployment and configuration automation have been relevant in their organizations in the past 12 months. In addition, 92% say that automation tools are relevant, or very relevant to them on a day-to-day basis. 

    Cloud, Containers and COVID

    The Trends in IT Ops 2021 survey shows that system administrators are concerned with cloud computing and containers and several other topics. In addition, most administrators recognize the increasing importance of monitoring cloud environments and containers. That said, administrators continue to see monitoring networks and other physical infrastructure assets as critical, and some system administrators see increasing importance in monitoring this vital area. 

    In addition to the topics outlined here, respondents also provided feedback on topics such as virtualization, application performance monitoring and network management.

  • 3 Pillars of Autonomous IT Ops

    3 Pillars of Autonomous IT Ops

    Imagine a future where human IT operators and intelligent systems work together in a virtual operations room; the systems do the heavy lifting while the humans supervise. This may seem like science fiction, but the truth is we are already on the way to that future with AIOps. While actual AI-augmented teams are still quite a way off, AI/ML tools are slowly but surely becoming more ubiquitous within IT Ops, NOC, DevOps and SRE teams and leading us into this not-too-distant future.

    In the last five years, the number of internet users grew by 50%. In the last two years alone, more data was generated than in the entirety of human history up to that point. Today, 5 quintillion (that’s 18 zeros) bytes of data are produced by smart devices daily. These massive volumes of data are having a huge impact on digital operations.

    Looking at the enterprise, we see similar trends: More tools generating more data and with it, greater complexity. This is due to many factors, including the transition to the cloud, M&A activity that introduces new tools and data into an existing organization and the adoption of new technologies and methodologies such as CI/CD, microservices and more. At some point, there is so much data and everything becomes so complex that IT Ops teams reach a breaking point where they can no longer manually manage the data being generated.

    And when that point is reached, outages and service disruption become an everyday reality, as IT teams are pulled away from their strategic, business-critical projects into what is essentially firefighting. 

    Implementing more observability and monitoring tools simply exacerbates the problem, and scaling teams isn’t cost-effective in the long term. The more forward-thinking solution is automation, and even more so, autonomy. 

    The Road to IT Ops Autonomy 

    Autonomy is the ability of IT Ops to operate with minimal to no human intervention. It relies on the automation of the incident management life cycle so that humans are needed only for supervision, in the same way that a single operator can supervise a modern and highly complex robotic car assembly line.

    How can we hope to reach IT Ops autonomy?  The following three proposed pillars can act as a guide: 

    Pillar One: Automate

    Pillar Two: Predict and prevent 

    Pillar Three: Democratize

     Pillar One: Automate

    We must automate as much as possible across every step in every aspect of incident management. It won’t happen overnight, but with time and with the help of AI/ML it is doable.

    Consider the autonomous driving analogy.

    Just a few years ago, cars were manual, with little to no automation. Then we moved through partial autonomy to where we are today–conditional autonomy–where the car can drive itself in ideal conditions. This is a level of automation that we are beginning to accept; some of us cautiously, others with more enthusiasm! Meanwhile, progress continues toward higher levels of autonomy, in which vehicles will be able to drive themselves in most conditions. One day, maybe, we’ll get to the holy grail of full autonomy, where the car will drive on its own–perhaps we won’t even need a steering wheel!

     

    Similarly, in IT Ops, automation is a process and it will take time to build not only the capabilities required but also the necessary trust in AI/ML technology. Such a journey typically begins with human-driven automation, as “if-then-else” types of automation are added manually. For example, if a certain alert appears, a ticket will automatically be assigned to a specific person. As our confidence grows, the next stage could be AI-driven suggestions, whereby AI applies what it has learned from previous human actions to suggest what to automate, but the human expert decides whether to accept the suggestion or not. 

    At some point, we will progress to allowing AI to partially automate some tasks—perhaps those that are less risky, such as collecting diagnostic information and adding it to the alert, so that the incident responder automatically receives both together. We may even automate some parts of our system; perhaps within a staging environment that isn’t production-critical and, as such, is a good place to validate that the automation is working as expected. Over time, as our AI and ML capabilities develop and our trust in them grows, we may eventually reach full AI-driven automation, with no human involvement at all—the AI will simply detect the fault and resolve it. 

    Pillar Two: Predict and Prevent 

    Typically, when assigned an incident, a responder will take note of the incident’s priority. But they will also rely on their common sense and past experience in deciding how to respond. So, for example, if a P3 incident had recently escalated to a P1 in 30 minutes and a similar P3 incident now appears again, our responder will probably prioritize it, even over other incidents, to avoid a similar escalation.

    While this is one of the advantages of human logic, the flip side is that the IT environments in which we work are vast and complex—no human can possibly remember every potentially relevant detail. What’s more, if a different responder dealt with the earlier incident in our example, the current responder would not have the same insight. What is needed is an element of enrichment that can input leading indicators into the platform for the next operator to see and act upon to prevent an incident.

    This is where AI can help. 

    By identifying patterns in the data haystack—linking different incidents to the same root cause, for example—AI not only generates insights that help resolve these incidents faster but can also predict a certain outcome. This is done by anticipating a future chain of events so that appropriate preventative action can be taken before the event occurs. For now, this preventative action may be manually taken by the operator, but at some point in the future automation may enable the problem to be identified and fixed by the AI with no human intervention before it results in an outage.

    Pillar Three: Democratize

    The last pillar that will help us achieve full autonomy in IT Ops is democratization of AI. AIOps should not just be accessible to advanced, mature organizations with an army of data scientists working behind the scenes. Just as anyone can accomplish almost anything online these days with just a few clicks, any organization should be able to adopt AIOps and enjoy its benefits! 

    Democratization is about making relevant information accessible to anyone who needs it, when they need it, and in a manner that is easily applicable and actionable. AI should be accessible through simple user interfaces, with no need for coding; the benefits of machine learning should be available to people that don’t (need to) understand machine learning models. 

    This democratization can be achieved by making AIOps platforms simple and accessible to all—from the admin to the user. For example, an AIOps platform might automatically suggest how admins should prioritize incidents for operators, to make triage easier and provide instructions that even a non-expert can follow. The more such features are built into AIOps platforms, the more democratized AIOps will become.

    Data volumes will continue to grow exponentially in the coming years, and the only way to eventually conquer the tyranny of overwhelming data will be to reach autonomy in IT Ops. 

    If you’re an AIOps vendor, keep the three pillars in mind so your customers can continue to progress towards autonomy.  If you’re an IT Ops leader or tooling architect, and IT Ops autonomy is part of your vision, make sure that your AIOps vendors’ platforms and tools have been architected with these pillars and their principles in mind. 

    How far are we from achieving autonomy? Only time will tell. But one thing is for sure—we will get there, and if the COVID-19 pandemic has taught us anything it’s that we may be seeing machine-augmented teams sooner rather than later. 

  • IT Ops Takes Center Stage

    IT Ops Takes Center Stage

    When people talk about the last 18 months, most will remember it as a disruptive time when nothing felt certain. Yet, for many IT Ops and network operation center (NOC) leaders, it was the year they stepped out of the darkness and into the spotlight. Suddenly, it mattered deeply who was maintaining those services we relied on to shop, work, eat, live – and it became clear that IT Ops teams – who were previously thought of only as the number you called for a password reset – had been keeping our businesses running all along.

    While there may have been some recognition before the pandemic, that awareness certainly gained momentum during 2020 and, now, shows no signs of stopping.

    As Ben Narramore, senior manager of operations with Sony Playstation Network said at a recent customer conference, “Today, I am able to talk to leaders across the company that I would never have talked to a year ago. They didn’t care about operations, they didn’t care about the NOC, didn’t care about incident management, and now they’re like, ‘Wow, this is great stuff,’” he said. “I’m excited right now. We’re at a place in operations where my team has evolved, doing more than just problem-solving.”

    I am excited to hear sentiments like that, because for the last 10 years or so, application development has been the golden child of IT. The faster you could crank out code, the faster the business could move. For years, every dollar and every resource went to building out amazing applications and services, leaving the IT Ops people to keep it all available and performing. Sure, some companies like Hulu and Expedia had the dynamics worked out and built a modern, optimized organization that could run as fast as it built, but most companies paid a high price in time, focus and resources to keep their services up and running.

    For most companies, it felt like a war of speed versus availability. Engineering wins when they can deliver what the business needs, quickly. They do not need to look beyond their group for much and – if it works as intended – they are good to go. IT Ops, on the other hand, needs a broad view of the entire technology landscape. They need to think about the availability of all services, which requires them to have visibility across the entire enterprise stack, often dealing with the downstream implications of outages and incidents that cross networks, hardware, software and clouds.

    Now, it’s 2021, and the war is over. And it turns out, the rise of operations wasn’t about a revolution. It was about recognition. IT Ops teams needed a seat at the table. They finally got the chance to explain that they want to move fast, too – to streamline operations, consolidate awareness, automate wherever possible and help support and enable the velocity the business demands, side by side with their partners in engineering.

    Jeremy Bragg is an infrastructure automation architect at Abbott Labs. I thought his role description was really compelling; here’s how he explained it: “Probably the most interesting aspect about being in automation is that we break down silos. We cannot just be a server team or a network team or a mobility team. We work across the board with all the teams to try to further their causes. We ask ourselves: where are we doing manual work today that we could automate?”

    Over the next five years, I believe more teams will start to take advantage of AI and machine learning to set the pace for the modern IT organization; more of the mundane work will be automated so they are free to lead innovation, not just react to it.

    Ben Narramore also has a lot of passion for that very topic. “If I look at my teams in the last ten years, we were firefighters; we’re good at it out of necessity. Now, we’re at a point with cloud computing [and] with AI that gives us a chance to reassess our team. I’m excited about how our team has evolved from day-to-day problem solving to looking at our business with the future in mind. Being able to automate manual operations and workflows gave us the ability to do new, more exciting work, as well as helping out with incident management and problem management.”

    My own take on IT is that the shift is happening, and I am excited to have a front-row seat as IT Ops teams step out of the shadows and into the light, taking their well-deserved place as strategic partners at the table, and helping drive the business forward.

  • How IT Ops Can Exceed Service Level Objectives in Digital Transformations

    How IT Ops Can Exceed Service Level Objectives in Digital Transformations

    The pace of change can be managed successfully by defining service level objectives and more in dev environments

    Mobile applications, data lakes, microservices, data visualizations, SaaS integrations, automations, IoT data streams, machine learning models—in proof of concepts, pilots and scaling production environments, for customer-facing capabilities and employee workflows—all of these technical capabilities are developed, deployed and enhanced faster today more than ever before.

    I spoke to Jason Walker, field CTO at AIOps platform BigPanda, about how the speed of deployment, the breadth of technology services transforming businesses are developing, the greater security threats, and the increase in reliability and performance requirements impact IT Ops.

    Walker believes that of all the things we’re trying to do in IT—more, faster, smarter, safer, innovative, secure, reliable—it’s the speed that’s the driving force. “The most significant impact is velocity; the dev-test-deploy cycle time is drastically reduced,” he said. “Without the right guardrails, that breeds unnecessary complexity and a gradual loss of operational awareness.”

    He explained that much of the barriers that once slowed down development teams are addressable today when developing service-based architectures on the cloud. “Developers realize that the traditional constraints, either technical dependencies or organizational capability, are much reduced,” Walker noted. “Developing in the cloud for a cloud-based service, leveraging an ecosystem of microservices for inputs, an agile team can move very quickly.”

    Why IT Ops Can’t Slow Down Transformations

    IT Ops can’t easily say “no” or “slow down” to business stakeholders investing in digital transformation to improve customer experiences and gain competitive advantages with data, analytics and machine learning. Some IT leaders attempted that command-and-control approach during the early days of public clouds, but today, DevOps practices, SRE responsibilities and AIOps capabilities are integral to mainstream IT Ops teams in keeping up with transformational velocities.

    So instead of saying “no,” progressive IT Ops teams say “yes, but” by defining service level objectives, (SLOs), capturing service level indicators (SLIs) and managing to error budgets.

    Walker agreed. “SLIs, SLOs and error budgets are a very useful way to manage the critical inputs and outputs at the interfaces between microservices and at a high level across the business service, allowing developers to keep changing the ‘interior’ pieces.“

    These tools change the operating model and mindset across the entire IT organization by exposing trade-offs to business stakeholders. For example, if a web application has a 99.9% SLO, the whole IT team has a 0.1% error budget. If the SLO is missed, a service level policy identifies areas of investment to improve performance, reliability, security and automation or to address technical debt.

    What IT Teams Should Do to Implement Service Level Objectives

    Defining service level objectives helps bring business stakeholders, development teams, SREs and IT Ops together and align on reliability objectives and trade-offs. It’s an important step, but not sufficient for teams that want to exceed service levels during digital transformation.

    Walker offered several technical recommendations for IT Ops groups transitioning to service level objectives:

    1. Where there are dependencies between microservices and interfaces, frequent check-ins between adjacent teams are required.
    2. APIs, inputs and outputs, need to remain consistent over time and changes communicated and receipt confirmed.
    3. Knowledge management processes delivering accurate, up-to-date documentation have to be baked into the SDLC to prevent the generally small, modular teams from sprawling away from each other and developing incompatibilities.
    4. Consolidated change awareness for operations is also a must-have. Whether changes are human or automated, they have to be tracked and relatable to service events and alerts.
    5. The later phases of the SDLC, when cloud-native, containerized microservices are running and supporting customers, have to be monitored. A monitoring strategy is necessary to ensure effectiveness and minimize the work and noise involved.
    6. Synthetics and client telemetry can be very useful macro-indicators of overall service performance. As with all monitoring efforts, actionability is key. Signal-to-noise in monitoring has to be measured.

    These are balanced recommendations, with the first two focused on how development teams engineer microservices and the last two on how IT Ops teams use monitoring and AIOps to implement actionable SLIs. The middle two recommendations on knowledge and change management processes help the entire organization stay in sync through a fast-paced operating environment.

    “The velocity, flexibility and variability of this type of development means leaders at all levels need to understand and align the strategic and tactical goals, and to prevent their teams from drifting away from business goals,” Walker said.

    Why AIOps is a Force Multiplier for IT Transformation

    So, the business leaders align on service level objectives, development teams engineer observable containerized microservices, IT Ops executes a monitoring strategy and the CIO ensures communication, collaboration and knowledge sharing. Is that all that’s required?

    The issue is that most IT Ops teams are understaffed and get overwhelmed supporting the new cloud-native microservices, legacy systems and everything in between. To address the gap, IT leaders are investing in AIOps, and even hundred-year-old enterprises are successful at adopting machine learning and automation to accelerate IT Ops.

    Automation and machine learning event correlation applied to monitors, alerts and observable artifacts are the force multipliers. Open-box machine learning enables IT Ops to triage incidents and improve their mean time to resolution, while automation reduces manual efforts and keeps teams in sync. Organizations that are modernizing applications and supporting hybrid clouds require these capabilities to manage the complexities, run at business speed and manage databases, microservices and applications to higher service level objectives.

    As Walker noted, increasing velocity is important for responding to customer opportunities and changing conditions. Driving faster digital transformations is more achievable today when IT Ops leverages automation and AIOps to stabilize speed with reliability and performance.

  • Incident Resolution for Remote Teams

    Incident Resolution for Remote Teams

    People working in IT support and incident management right now are faced with unusual difficulties supporting large remote workforces and managing unpredictable workloads. On Reddit, system admins and other IT pros are bemoaning the hiccups and hassles of working in isolation while trying to resolve issues and maintain high SLAs. You can’t go grab your indispensable SME for troubleshooting, because that person is also home and inundated with messages and alerts from many different tools.

    Networks, VPNs and servers are being hammered extra hard right now from higher usage of SaaS apps and collaboration tools. User expectations are even more grandiose than normal: The home-based IT environment is a lifeline to whatever remains of human interaction. IT organizations are expected to make applications and services work flawlessly–overnight. Undoubtedly, IT pros are working with assiduity to keep the wheels of work moving for their employers during a time of crisis. Yet, working longer hours isn’t the answer. We need new ways of working—ones which may prove better than the current status quo for the long term.

    In many organizations, the process of incident management and resolution begins with people looking at screens full of metrics. You see a spike in something, such as network I/O, and immediately begin to investigate it, assuming the worst. A more adverse scenario is when users call out problems on Twitter, Reddit or other popular tech communities, creating a noxious PR situation which IT and the business must swiftly respond to and manage. This ad hoc, reactive fire drill takes time, and resolving that particular issue may not be the most important item of the day. In software development, it’s akin to the bad practice of designing the architecture of a system first and then the user experience.

    Given everything that’s at stake in the foreseeable future of stay-at-home mandates, SREs and IT support technicians need to flip the incident management process. Rather than beginning with the metrics and working backwards to uncover the problem, if there even is one, start by looking for the most important problems to solve. Then, discover the resources and components which support that end-user process and finally, analyze the metrics which are most relevant. For example:

    1. A B-to-C provider of a streaming video service is now seeing a 200% increase in daytime traffic. This is resulting in sluggish load times for users and glitches. 
    2. IT has tagged the components beneath that service, so IT operators know exactly which VMs, databases and load balancers come into play. 
    3. Next, the operations team can identify the most relevant metrics, for instance CPU idle time, to measure and take steps to correct. 

    The Problem-First Approach

    It sounds so simple: flip the process. Yet, this is not the case. IT operators and SREs are not accustomed to thinking in terms of which problems to solve or prevent. Being proactive or predictive hasn’t been the common workflow.

    Here is how IT operations and DevOps teams can take a proactive approach to managing the availability and health of enterprise services.

    Coordinate Across Departments

    Identifying the most critical problems to solve right now is the hardest step and the one which will take the longest. Common problems these days are related to cost and scale: How do I right-size my environment to reduce costs or handle more workloads? Broaden your field of inputs to narrow down to the specific issue, such as slow page loads on the financial reporting website. Product managers, account managers and business unit leads who are the closest to the customer experience can deliver feedback on the top issues affecting customer/user satisfaction. Teams should also review recent support tickets to identify common themes of pain.

    Fine-Tune Your Approach to Metrics

    The Utilization Saturation and Errors (USE) Method is one way to approach the problem-first incident management process. As detailed by Brendan Gregg, a senior performance architect at Netflix, this methodology begins by posing questions, seeking answers and working backwards to the metrics. For each resource that you want to measure, identify three metrics: one for utilization, one for saturation and one for errors.

    “The USE Method has made you aware of what you didn’t check: What were once unknown-unknowns are now known-unknowns,” Gregg explained.

    Standardize Workflows

    Create a common process for incident management across all your teams. Without the advantage of having almost everyone in the same room to huddle together in an ad hoc fashion when big issues crop up, it’s imperative to institute clear steps and roles. Doing so will prevent the frustrations, confusion and oversights that needlessly delay resolution. Since most incidents are composed of multiple contributing factors, teams need to adopt a small number of user-friendly tools to document and organize the information. At our company, we’re now using tools such as Miro, an online whiteboard application, to replace our physical whiteboarding sessions. Of course, there’s also Zoom, Slack, Jira and a host of other cloud-based tools already in place at many organizations. Mandate which tools everyone should use, with some guidelines on how to use them.

    Increase Automation

    In some organizations, scaling requirements in response to demand have increased tenfold. Automation is playing a critical role now; moving away from a web GUI, for example, is more scalable and aligns with modern tools such as Chef and Puppet. User tickets can be auto generated, for instance, from emails and linked to code management systems like GitHub. Modern development and operations teams are also expanding automation in unit testing and provisioning. 

    Watch for Burnout

    Whether there’s more work and/or a need to fill the hours during long quarantine days, many software engineers, testers and architects are working longer days right now. Yet, exhaustion and burnout can lead to errors and oversights along with low morale. It’s up to managers to make sure that employees are taking breaks, working reasonable days and having the time and energy to attend to personal needs.

  • Unlocking the Power of AIOps

    Unlocking the Power of AIOps

    AIOps is becoming quite the hot topic lately. It seems as though everyone is realizing that Ops is difficult. As we have to deal with more big data at greater velocity with more automation and complexity, there is no way humans alone can tame the beast. AIOps seems to be the answer.

    I recently sat down with Dave Link, CEO and founder of ScienceLogic to discuss how to unlock the power of AIOps to empower IT ops. We video taped the conversation, along with some accompanying slides that help explain what the challenges are that ops faces and what are some of the ways to tackle this using AIOps.

    I should mention that Dave and I are old friends, having worked together 20-plus years ago at a dotcom era company called Interliant. Since 2003 or so I have watched Dave build ScienceLogic into the powerhouse it is today. I couldn’t be more happy and proud of him.

    ScienceLogic has been named by EMA as a top vendor in the AI/ML for IT Operations. Dave gives us a good background on ScienceLogic as well as a good background on the use cases and challenges that IT Ops faces that AIOps seems to be the answer for.

    Dave also talks about the differences between AI and machine learning or ML. This seems to be an area of confusion today as well.

    The video is about 30 minutes and excuse Dave and I reminiscing, but there is some really great information here.

    Video on AIOps

     

    — Alan Shimel

  • Operations Management Puts the ‘Ops’ in DevOps

    Operations Management Puts the ‘Ops’ in DevOps

    Awareness and anticipation of operational management, best practices can improve your app at every stage of its life cycle

    How important or interesting you find the “operations management” aspects of DevOps methodologies likely depends on your background, experience and how many operational horror stories you have lived through. If you have been on the hot seat when services are impacted, woken up in the middle of the night when major outages occur or dragged into blame-battle war room calls while you and your team scramble to restore service, then the importance you place on operations management may be very high.

    However, if you have avoided such experiences, it would surprise no one if the operations management aspects of DevOps are not top of mind. When everything is smooth sailing, then it is understandable that you might be reluctant to make any significant changes to your daily routine. However, in agile, fast-moving environments, things can go wrong very fast, even if your organization separates duties intentionally to create an “operational firewall” to keep ill-advised code or configuration changes to a minimum. Using effective ways to lend your app expertise to assist during impactful situations can make your life much easier and limit disruption to conducting your other duties.

    You may work in an organization which is new to owning and managing operations management tools. It is becoming more common for organizations outside of a central IT operations team to do so as scopes of responsibility expand. Several analyst firms have published recent studies that nearly half of purchase and usage of operations tools is already done outside of IT operations teams. This trend is predicted to grow to the majority in a few years.

    Configuration and automation tools are also becoming more commonly owned outside the data center. Such shifts in responsibility make efforts of operational skill-building, incident readiness and resolution more challenging. As the number of different groups deploying these tools expands, the likelihood of overload of notifications, alerts, events and general operational noise starts to rise dramatically. Note: I will be presenting a webinar on how new operations management tools can help you carry out the best practices, join me in this conversation Thursday, Oct 26.

    Organizations deploying new apps and management tools in the cloud naturally look to manage those workloads from the cloud as well. Other organizations with legacy applications often are managing digital transformation leading to transformative projects for operations management. Whether your organization is already leveraging cloud-based tools or you are part of an organization undergoing transformation, you may have the challenge of a hybrid set of workload deployments, both cloud-based and on-premises. For your organization, this could mean finding a pragmatic complementary approach to deploy and manage the new, cloud-based applications and their associated management tools. Such an approach may not need to be isolated from your current tools and methods. Sufficient tool interoperability could support a federated, phased path.

    In this blog series, we will explore the common operational challenges many DevOps teams are facing today, how traditional IT operations best practices could be leveraged for use in a DevOps methodology and how new operations management tools can help you carry out those best practices to meet your goals on an on-going basis.

    Here’s a preview of the forthcoming blog topics:

    Let the Noise Wail Without Going Deaf

    Unless your applications and services exist in a vacuum sealed chamber of silence, agile development, continuous rollouts and updates and infrastructure system changes create a plethora of events that range in severity of impact, validity and meaningfulness. Those events tend to come from multiple monitoring and configuration tools or reflect many different parts of your applications and underlying services. This mixture of events and alerts leads to what is commonly known as operational “noise.”

    Smart Ops Action: Taking the Right Action for Ops Traction

    Empowering anyone in your organization who contributes to restoring service when incidents occur can ensure their time is well spent, with the intended results. All too often, operations teams effectively identify and triage incidents, yet fall short of empowering first responders to actually resolve problems. Bolstering your team’s efforts to take the actions necessary to resolve the situation at hand and restore service promptly is achievable—if done right.

    Boring is Best! How Can You Thrill Your DevOps Stakeholders with Boredom?

    Strive for stability and structure in production without stifling development agility. Don’t try to thwart the innovation and chaos that results from agile development and constant rollouts—get structure and stability where you have control so you can focus your talented team members to be supportive, valuable contributors to innovation and agility.

    Fuel innovation and make your stakeholders smile with boredom by effectively turning chaos into stable operations.

    If you’re a veteran IT operations practitioner with years of experience helping your company’s applications, services and underlying infrastructure run efficiently, these tips may be obvious. Hopefully you’ve been successful advocating these best practices to your development and line of business colleagues.

    If you are new to operations—and especially if you are amongst the growing population of DevOps professionals who are understandably reluctant to conduct triage, investigation, diagnosis or resolution of operational incidents—rest assured there is relief. Tried-and-true best practices carried out with new, easy-to-use tools can empower your team to conduct these unwanted tasks like a seasoned veteran, regardless of operational experience. Learn more.

    About the Author / James Moore

    James Moore is Principal Offering Manager, IBM, responsible for offering management and strategy for IBM HybridCloud Operations Insight solutions including IBM Cloud Event Management, Runbook Automation and Alert Notification. James joined IBM from the Candle acquisition in 2004, where he was product manager for Candle’s Application Response Time product lines. James has over 15 years’ experience in event management, application performance management, and business service management. Connect with him on LinkedIn and Twitter.

  • AIOps: Managing the Second Law of IT Ops

    AIOps: Managing the Second Law of IT Ops

    Or, why IT systems keep breaking

    The tension between the two halves of DevOps is long-standing, going back long before the term was even invented. What underlies the split between Development and Operations is a fundamental difference, like an actual physical law. Let’s explore that analogy together.

    IT Ops is fundamentally a constant fight against entropy, meaning disorder. Sysadmins expend significant energy trying to keep everything up and running, in a process which can sometimes feel like trying to shore up a sandcastle while the tide is coming in. Systems are constantly breaking down; users need changes made that introduce risk; and, as if that wasn’t enough, the whole shebang is constantly under attack from malicious outsiders. It’s no wonder that sysadmins have a reputation for being a bit grumpy!

    Homer: In this house, we obey the laws of thermodynamics!
    It’s not just a good idea, it’s the law.

    In contrast, Dev actually increases the level of entropy. Every new release, be it as small as a one-line fix or finding that errant semicolon, is a change that has to be made, reducing the order that Ops fights so hard to maintain. Dev is entropogenic, embracing change as a positive force.

    The interaction between Dev and Ops is analogous to the second law of thermodynamics: Entropy can only increase over time, unless energy is introduced to the system from outside. In other words, unmanaged change will eventually cause something to break, unless people put in continuous effort to prevent those breaks.

    Thermodynamics: Lots of Heat, not Much Light

    These conflicting goals, taken to extremes, can get unhealthy, with Ops actively fighting change or Dev going around Ops to increase their velocity of change. Left unchecked, this tension can turn into a vicious cycle, as frustration with the “Department of No” drives developers into unmanaged “shadow IT.”

    The good news is that the analogy with hard laws of physics is not perfect. The process of increasing entropy is not irreversible, and does not require constant inputs of energy to prevent. New agile approaches to IT Ops are emerging that are light and flexible, taking their structure from that tension. In the physical world, this is called “tensegrity,” and was originally formulated by Buckminster Fuller, one of those annoying polymaths who seem to make breakthroughs in every field they touch.

    Tensegrity structure
    Tensegrity structure

    IT Ops needs its own equivalent to tensegrity, a flexible, holistic structure that is able to embrace change while maintaining its shape. Structures that try to resist change through rigidity are brittle and ultimately fragile.

    Today, this flexible strength is enabled by AIOps, a term coined by Gartner to describe a new way of operating IT, based on augmenting human skills with automated analysis enabled by modern algorithmic and machine-learning approaches.

    Complexity Breeds Complexity

    Modern IT is too complex for any single person or even team to keep track of, instead requiring multiple specializations. In turn, each of these functions requires its own specific support tools that enable that particular role. Beyond a certain point, it becomes very difficult to combine these partial views to gain a holistic understanding of what is actually happening in the environment.

    Because of the scale of modern IT environments and the sheer rate of change that businesses require, old-style rules-based “Managers of Managers” can no longer keep up, requiring more effort to build and maintain their worth. Especially as infrastructure becomes more and more self-modifying in response to changing conditions and demand, IT Operations needs to accelerate just to keep up.

    AIOps brings developing issues to the attention of sysadmins before they become visible to customers and end users, and gives them the tools to collaborate effectively between different teams and specialist roles to avoid outages or restore service quickly and efficiently. In this world, automation is used to augment the capabilities of the human operators to make them more effective, letting people concentrate on adding value, not just fighting fires.

    This is how Ops can get out of the business of fighting Dev, and instead become a supportive partner. AIOps can catch issues that are affecting services and flag them to the right people early on, including the DevOps engineers that are responsible for production support.

    For developers, one of the fears around DevOps is that in “picking up the pager,” they are opening themselves up to a constant stream of irrelevant alerts. Old Ops hands nod wryly at this, or maybe try to share their personal approach to filtering the noise. Instead, AIOps ensures that only valid, actionable alerts get as far as being shown to a human being and taking up their valuable time.

    Eyes on the Prize

    It’s not all roses. As the AIOps market begins to mature, there is a certain amount of “algorithm-washing” going on, with more and more vendors trying to claim that either their legacy tool was AIOps all along, or that their one-trick pony is actually a whole three-ring circus. What gets lost in the excitement is what all of those algorithms are actually supposed to achieve.

    Me too!
    Me too!

    In considering a shift toward AIOps, the goal should not be simply to build a better mouse trap, meaning a smarter filter for monitoring events. The success of the project can be measured not so much in the mathematical results of the algorithms, but in the smoother exchange between the human users. This easier collaboration is especially important across team boundaries, whether these be different specialist units within the Ops function or developers and software engineers who need to work with Ops to understand the performance of their code in production.

    The benefit of transforming operations this way—moving from sequential, waterfall Ops to abile, collaborative Ops—is to turn DevOps-enabled teams into a sort of perpetual-motion machine, able to continue operating without requiring constant massive inputs of energy from outside the system. Modern applications and the complex infrastructures that support them cannot be managed with the sorts of rigid, cobbled-together, industrial approaches that were appropriate in the mainframe days. AIOps is the agile, flexible approach that can not only accomodate that complexity, but thrive on it.

    — Dominic Wellington

  • Are We Leaving Developers Out of DevOps Spinoffs?

    Are We Leaving Developers Out of DevOps Spinoffs?

    SecOps. DataOps. NetOps. Reading these terms, you get the sense that the key to IT efficiency is to make IT Ops work with everyone else. But that’s a mistake, because it leaves developers out of the picture.

    It’s no secret that the DevOps movement has generated a number of other *Ops initiatives, including those listed above. The driving idea behind all of them is that IT Ops teams should work more closely with security teams (in the case of SecOps), data analytics teams (in the case of DataOps) or networking teams (in the case of NetOps). These are only a sampling of *Ops movements; the list could go on.

    To be sure, there are advantages to gain by having IT Ops collaborate more closely with other parts of the organization. But there are an equal number of advantages in doing the same thing with developers.

    To optimize security operations, you need developers, IT Ops and the security team to work together. If Dev is left out of the picture, it’s much harder for the other two parts of this equation to make sure that the code the organization deploys and manages is secure.

    For similar reasons, DataOps, NetOps and almost all the other *Ops you can come up with work well only when Dev also participates. The people who actually design and write the applications are important.

    Rethinking Ops

    Why does this matter? My point is not merely that terms like SecOps and DataOps are misleading because they don’t mention developers.

    On the contrary, giving equal weight to developers is important because if you don’t, you risk isolating developers. Some are already complaining that DevOps is “killing” them. The trend of defining new *Ops practices that don’t mention developers is not helping.

    So, maybe we should start mentioning and thinking about developers more prominently when we are talking about ways to extend the DevOps philosophy to other parts of the IT organization. This is already being done somewhat in the case of SecOps, which you sometimes hear called SecDevOps. But Google lists nearly 13.5 times as many results for “SecOps” as for “SecDevOps.” The numbers are similar if you search for “NetOps” as compared to “NetDevOps.” And I’ve never heard anyone say “DataDevOps.”

    The takeaway is this: Developers are as important as IT Ops to DevOps. Don’t leave them out.

    — Chris Tozzi