devopsy.kinsta.cloud

Tag: incident management

  • 7 Important Truths About Chaos Engineering

    7 Important Truths About Chaos Engineering

    As a relatively new practice, chaos engineering has plenty of myths surrounding it, from randomly shutting down production systems to requiring huge investments of time and money. There’s a lot of confusion over the purpose, the value and the practice of chaos engineering. This presents a problem for DevOps teams, especially since more than half of the teams surveyed in the 2019 Gartner DevOps Survey listed improving system reliability and release quality as one of their top five DevOps objectives.

    In this article, we’ll clear up some of this confusion by presenting seven important truths to help you make an informed decision about chaos engineering and how it can help your engineering organization.

    Truth 1: Chaos Engineering is not Chaotic

    What comes to mind when you think of chaos engineering as a practice? If the answer is causing random production outages, you’re not alone. Tools such as Chaos Monkey popularized this idea, but for most teams practicing chaos engineering, it is a well-planned, controlled process that aims to mitigate chaos rather than cause it.

    The goal of chaos engineering isn’t to add chaos, but to mitigate chaos.

    It’s true that it involves creating potentially harmful conditions on otherwise healthy systems. For example, we might test our application’s ability to handle load by increasing CPU usage on our servers. With enterprise solutions we have full control over which systems are affected by this test (known as the blast radius), how much CPU we consume (known as the magnitude) and how long it runs for. We can also immediately stop the test and roll back its impact in case of unexpected consequences.

    We also know exactly what conditions we’re introducing into our systems and can revert the changes at any time. Yes, we’re still causing harm, but we’re doing so in a way that helps us learn about our systems and reduces the risk of failures, both real-world and induced.

     Truth 2: Developers Care About Reliability

    Developers aren’t only interested in building new features. While development teams often prioritize feature development, this is likely due to business-driven initiatives resulting in rapid release schedules. Developers—especially those who have responded to incidents or worked through bug reports—understand the value of resilient software. They just don’t have time to build it.

    The problem is that this creates a reliability gap in our applications. Rapidly building features without adequately testing their resilience creates failure modes, leading to problems that developers need to go back and fix at the expense of new projects. The longer we go before finding these failure modes, the more likely we are to experience unexpected behaviors and outages in our applications, and the more expensive a solution will be.

    Increasing development velocity without testing for reliability increases the likelihood of failures, causing our expected availability to drop.

    To prevent this gap from forming, we need to prioritize and provide the appropriate tooling so developers can test their code and improve the resilience of our applications. This leads to our next truth.

    Truth 3: You Have Enough Time for Chaos Engineering

    Chaos engineering saves you the time you would spend responding to, troubleshooting and fixing production incidents. As you build resilience through chaos engineering, you’ll reduce the engineering time, effort and costs spent on incidents and can focus more on your core competencies.

    That said, adopting chaos engineering does require some time investment. Engineers will need to learn the principles and practices, adopt new tools, run experiments and implement fixes. However, this is relatively minor compared to the cost of production outages. Without chaos engineering, your engineers will need to pause their regular work and rush to restore service. This is time they could be spending on feature development and other value-creating tasks.

    A few minutes of proactive testing can uncover defects that would’ve taken hours to fix in production.

    Truth 4: Your Systems Support Chaos Engineering

    You don’t need to use specific hardware, software, or cloud platforms to benefit from chaos engineering. While some tools have specific requirements, most only require access to the host operating system. Some can even target specific resources, such as containers.

    Truth 5: You Don’t Need Comprehensive Observability (But It Helps)

    You don’t need a comprehensive monitoring or observability practice in place to benefit from chaos engineering. Basic metrics provided for free by your cloud provider can still provide important insights into how your systems are responding to experiments, and some chaos engineering tools provide basic monitoring for you.

    That said, chaos engineering provides significantly more value when used with monitoring and observability. We want to know how our systems are responding to our experiments, and the way we do that is by collecting metrics and traces. Even if you’re currently implementing a monitoring solution, chaos engineering can help streamline this process by helping you determine which metrics to track, fine-tune your alerts and avoid alarm fatigue.

    Truth 6: Chaos Engineering Pays for Itself

    Like any new tool or practice, chaos engineering requires an upfront investment of time and money. Solutions need to be evaluated and purchased, and engineers need time to warm up to the practice. However, the value it provides far exceeds these costs. Using chaos engineering can help you:

    • Avoid losses in revenue and productivity due to incidents and engineering time spent resolving incidents.
    • Improve your ability to generate sales by helping you build reliable systems and prepare for periods of increased demand.
    • Save costs by helping you optimize resource allocation and test for non-essential infrastructure.

    Truth 7: You Can Get Started Today

    Getting started with chaos engineering can be as easy as deploying an agent to your systems and choosing a resource to target for experimentation. You don’t need to have a stable, mature SRE practice or enterprise-grade infrastructure. In fact, you can use chaos engineering to help establish your SRE practice or accelerate migration to a new platform.

    The most important truth is that no system is perfect, and there are failure modes hidden all throughout our applications and infrastructure. Chaos engineering lets us uncover these failure modes and transform reliability from myth to reality.

  • Mattermost Offers DevOps Command Center in the Cloud

    Mattermost Offers DevOps Command Center in the Cloud

    Mattermost today launched Mattermost Cloud, a software-as-a-service (SaaS) platform for managing DevOps workflows using pre-built playbooks that comes integrated with a wide variety of third-party developer tools.

    Chandar Venkataraman, executive vice president and chief product officer for Mattermost, said Mattermost Cloud enables IT teams to take advantage of a single-tenant DevOps platform that they don’t have to maintain in a way that forces developers to give up their preferred tools.

    In contrast, other DevOps platforms that are managed as a service require developers to make use of a narrow range of tools provided by the platform vendor, he said.

    Built on top of an instance of Kubernetes, the single-tenant capability also ensures that any data or artifacts stored on the platform remain private. The Mattermost Cloud Professional edition, available today, provides a full set of collaboration and DevOps features, while the Mattermost Cloud Enterprise edition, currently in beta, adds dedicated single-tenant cloud infrastructure, enterprise-grade secure network topology and strict data residency options.

    By building the platform on Kubernetes, he noted, it becomes much easier to scale the Mattermost Cloud platform up and down as required.

    Mattermost Cloud is essentially a DevOps command center that provides single pane-of-glass pre-built workflows that address most common DevOps processes. IT teams can add their own custom workflows as their DevOps processes mature, Venkataraman said, noting plugins for Jira, Opsgenie, PagerDuty, GitLab, GitHub and Jenkins are already available.

    The cloud platform also includes a pre-built best-practices incident management playbook to enable DevOps teams to address issues as they arise rather than trying to resolve them using a legacy ticket-based approach to IT service management (ITSM), he added.

    Venkataraman noted Mattermost doesn’t expect DevOps processes to supplant existing ITSM platforms based on ITIL-frameworks. However, many DevOps teams are looking for ways to resolve incidents within their existing workflows using, for example, a messaging platform versus a workflow designed for IT support staff.

    In general, Venkataraman said, many organizations that are relatively new to DevOps don’t want to manage the underlying platform. At the same time, he noted there are also many IT organizations that want to allocate more resources to writing code versus managing a DevOps platform.

    Each IT organization will need to decide when it makes sense to stand up their own platform for managing DevOps workflows and playbooks instead of building and maintaining their own control plane. Many of those IT teams are already making use of artifact repositories and continuous integration/continuous delivery (CI/CD) platforms that are accessed via the cloud, so a DevOps command center is only one more logical extension.

    Just as importantly, isolating the DevOps command center from the rest of the underlying DevOps platforms ultimately makes it easier for IT teams to switch out tools and platforms over time as they best see fit. The challenge now may be in finding a way to decouple workflows that are deeply intertwined with a specific platform.

  • The CMDB Is Dead – Long Live the CMDB

    The CMDB Is Dead – Long Live the CMDB

     Many enterprise IT organizations have a configuration management database, or CMDB, in place. These systems are important for overall configuration management, governance and DevOps coordination. Yet, these legacy data repository systems can fall short in our dynamic IT environment. They require manual updating and can’t always support real-time decision-making. Furthermore, we have new requirements since the onset of widespread WFH workplaces. In this article, we talk about the past and current role of the CMDB, how it is evolving and how integrating it with other IT operations systems can modernize infrastructure monitoring and configuration management.

    History Lesson

    Remember the enormous fuss over Y2K? IT execs were concerned that the new century turnover would break their systems, which led to a secondary concern that they weren’t sure what was even on the network. The 2001 release of ITIL v2 best practice led to the notion of a CMDB, which would enable the tracking of every technology asset and its configurations and show the relationship between those assets. But getting all that data in one place proved to be a problem—and one that has persisted to this day.

    Is the CMDB Still Relevant?

    CMDBs have been around for a long time, and they still play a valuable role in the enterprise IT organization. In fact, a CMDB has never been more essential than now. The year 2020 has been chaotic for most IT organizations as employees quickly transitioned to working from home and are using a wide array of personal devices and networks, all of which need to be monitored, maintained and secured. As shadow IT persists and organizations move increasingly toward rapid deployment and release models that incur frequent, ongoing change, governance and control become critical.

    Contrary to what some may think, you can’t safely go faster without a level of control. This is a balance that DevOps practitioners must strike, and it entails a deep understanding of all elements within the IT environment and how they relate to one another. If the enterprise doesn’t have a CMDB (or something akin to it) to provide visibility into infrastructure components, configurations and relationships, the IT operations team is working at a disadvantage. Such lack of knowledge slows down developers—and nobody wants that. Even worse, without always updated, reliable infrastructure data, DevOps teams risk a catastrophic result with each software release.

    Modern CMDB Requirements

    Today’s CMDB is critical for incident management and must be supported by a mature, well-understood change enablement practice in multi-cloud and hybrid environments, yet legacy platforms aren’t designed to monitor systems residing outside of the corporate firewall. More importantly, modern applications are built on containers and ephemeral microservices that appear and disappear frequently. It is nearly impossible for IT staff to manually update the CMDB under such constraints, and the truth is, people weren’t typically doing this regular updating years ago. Today, the only viable way to maintain state and configuration data to support accurate, real-time decision-making is through automation.

    Most CMDBs on the market today lack automation; they still look and act much like a data repository. The appetite for heavy manual work is razor-thin, and IT organizations simply don’t have the people power to do it regardless. Simultaneously, the role of the IT operations professional has evolved dramatically—it is no longer as focused on installing, patching and rebooting systems. Rather, these individuals are now expected to manage applications and platforms in an end-to-end fashion, serving the business with their unique requirements spanning distinct business units and use cases. Therefore, the more we can use technology to reduce the load of traditional incident management tasks and enable operations to be a true partner with development, the better we can serve the business and end customers.

    So, What Now?

    Integrating modern and legacy tools is one practical solution for resolving these issues. Newer IT operations management (ITOM) technologies that incorporate artificial intelligence and machine learning (AIOps) to automate the discovery of assets and topology mapping can bring the CMDB into the modern age. An open ITOM framework for integrations can keep the CMDB always up-to-date, without manual intervention. This enables IT operations staff to focus on optimizing and improving infrastructure for reliability and performance, versus digging into one dashboard after another to make sense of the status quo.

    Understand the requirements and scope of your CMDB. Tracking the configurations of components that are not business-critical only expands on the effort and resources needed to successfully maintain the CMDB, creating waste in the process. Keep it simple and iteratively improve the CMDB over time. Most importantly, get started now. Even seemingly small, simple improvements can have a positive cascading effect down the road.

    IT environments will only get more complex—even though the end goal is simplicity, cost savings and efficiency. To keep pace, enterprise IT organizations will need to rethink the traditional CMDB. By integrating the CMDB with modern IT operations monitoring, management and automation systems, IT operations teams can finally have a reliable repository of data on all of the company’s on-premises assets and multi-cloud resources, while also gaining the ability to automate manual tasks that impair productivity and eventually remediate where appropriate with the right approach to AI and machine learning.

     

    This article was co-authored by Curt Thorin, a solutions consultant at OpsRamp. He has 20+ years’ experience in enterprise IT software and services. Curt can be reached online at LinkedIn.

  • What Is Resilience Engineering?

    What Is Resilience Engineering?

    Admitting things will go wrong isn’t easy for anyone or any team. But modern engineering practices have moved beyond this fear, giving birth to a new practice in DevOps and site reliability engineering (SRE) known as resilience engineering.

    Resilience engineering today isn’t thought of as a function. However, just as DevOps was a description of culture before it was a role and site reliability was an extension of operations before it was a focus, I wouldn’t be surprised if resilience engineering became a function in the new future. The first question most will ask however is, “Isn’t this just SRE?” The purpose of the term is to change the focus from simply reacting to incidents to developing long-term response strategies for them.

    Because the expectation in these environments is that things will break, resilience is the responsibility of existing DevOps and cloud operations teams. When applications and services do break, a “fly by the seat of your pants” response strategy will not work.

    A Focus on Frameworks

    Resilience engineering, while rooted in engineering practices, is largely focused on building strategies and a framework for their execution. This leaves the process of building resilience into a largely unestablished system in part because each system is unique. And, how you respond to issues in that system will likely be unique, even if the management plane that reports issues is not.

    The job of resilience engineering is to:

    Establish procedures, habits and decision trees

    When things break, fight is the only option. Operators and on-call engineers need to address issues in a systematic and repeatable way and do their best to remove emotion and fear from the equation. This not only helps triage and resolve issues, but it also makes sure the activity associated with the issue leads to meaningful insights in post mortems and future collaboration. Part of that is establishing habits and decision-making processes for those who are on-call. The processes help prioritize what to focus on, help catch details because details are critical.

    Be data-driven

    Resilience engineering must rely on data. This is another place where traditional SRE practices grow with a focus on resilience. In typical SRE environments, the focus is on the now, using real-time dashboards of the current state. When you think about resilience you are thinking about the past’s impact on the future, not the now. The only way to do that is to make sure the data supports it; thus, part of resilience engineering is making sure the data is there. Instead of data silos that most organizations have across their delivery chain. Resilience engineering should ensure that telemetry across the entire delivery chain is captured, correlated and shared. The reason this is critical is that what happens early in the delivery chain directly impacts incidents. That activity can be the source of answers, it can be the triggers for rollback, or it can be the clarity needed to prevent similar issues in the future. Without continuity between each stage of the delivery chain, it’s easy to miss correlated events that can lead to more systemic problems.

    Engineering out of reproducible incidents

    For most, the best part of resilience engineering is taking what is learned from previous incidents and finding ways to automate future resolution. Learning from data and having consistency in habit leads to the ability to create runbooks and automate remediation for known issues. Often incident response audit trails can read like playbooks for addressing issues of a particular type. When the resolution is not directly related to code and the potential of issues to surface again in the future is inevitable, being able to build intelligence to address it saves waking someone up at midnight and much shorter impact on customers.

    The Resilience Stack

    Organizations looking to embrace resilience engineering need to have a toolkit built for it. It’s pretty straightforward but worth highlighting. The resilience stack will include:

    1. Observability and/or monitoring tool.
    2. Incident response tool.
    3. On-call strategy documentation.
    4. Post mortem process and documentation.
    5. Documented processes including recommend response steps.
    6. Path to automation.

    For those with a relatively mature and automated environment, the next step is chaos engineering—embracing chaos as a way to get ahead of incidents before they happen in the wild.

    Get your markdown skills to maximum level. There is a lot of documenting that needs to happen with comprehensive resilience engineering. But these documents should not be shelf-ware—they should be living and ultimately lead to the implementation of automation or feedback to development.

    As for monitoring, observability and incident response tools, it’s not enough to simply implement them. Knowing how data will be collected, consumed and actualized is also necessary. So is establishing an on-call strategy with purpose, not just because having everyone on-call is the “cool thing to do.”

    It may seem obvious to say that organizations want system resiliency. What is not obvious is how to execute it. When organizations see the gaps (and are often embarrassed by them), they understand that resilience is a focus for either current functions or new ones in the future. That is why it’s worthwhile to talk about resilience engineering and what makes it effective.

  • PagerDuty Looks to Streamline IT Incident Management

    PagerDuty Looks to Streamline IT Incident Management

    PagerDuty has added more automation and analytics capabilities into its portfolio of incident management tools, at a time when organizations are looking to significantly reduce the total cost of IT.

    Jonathan Rende, senior vice president of product at PagerDuty, said the general release of Intelligent Triage advances the level of AIOps that can be applied to incident management by eliminating a raft of manual tasks.

    Intelligent Triage aggregates signals from more than 375 monitoring, observability and ticketing tools, which are then grouped based on relationship and elevated depending on the impact on the business. That level of contextual data makes it possible for IT teams to increase productivity while dramatically reducing alert fatigue, said Rende.

    At the same time, PagerDuty is adding support for integration with Microsoft Teams alongside existing support for Slack. IT teams are increasingly relying on these collaboration tools to process requests for help from end users, he noted.

    In some cases, IT teams are communicating with one another via these media or employing built-in tools embedded in the PagerDuty incident management platform. In either case, workflows across IT teams are now occurring in near real-time, said Rende.

    Finally, PagerDuty is adding a Dynamic Service Directory to give DevOps teams a central knowledge base that identifies owners and experts for each service and their associated run books, in addition to making available a mobile status dashboard to allow remote teams to collaborate more effectively when incidents arise. That capability is especially critical at a time when many IT staffs continue to work from home as part of the effort to combat the COVID-19 pandemic, said Rende.

    In general, IT organizations are looking to automate incident response whenever possible to both minimize costs and potential disruptions. Businesses are more dependent on IT services than ever. However, as more applications are deployed the nosier the IT environment becomes. Discerning which alerts are stemming from the same root cause issue becomes more relevant as organizations either limit headcount or even look to reduce it during a time of widespread economic uncertainty.

    Of course, regardless of the state of the economy, most IT professionals would rather not have to waste time on manual tasks that don’t add a lot of value to the business. Most IT professionals need to free up more time to achieve more pressing goals. Every minute spent on chasing down an alert that winds up being yet another wild goose chase just piles on more stress.

    Advances in automation and AIOps have been coming at a steady pace for the last several years. The adoption of these new capabilities to manage incidents has been uneven at best. In the wake of the economic downturn brought on by the pandemic, it’s now more a question of how soon IT organizations will be tapping into these capabilities to streamline the management of IT. After all, many IT managers are going to conclude there’s no sense in letting a good crisis go to waste.

  • Incident Resolution for Remote Teams

    Incident Resolution for Remote Teams

    People working in IT support and incident management right now are faced with unusual difficulties supporting large remote workforces and managing unpredictable workloads. On Reddit, system admins and other IT pros are bemoaning the hiccups and hassles of working in isolation while trying to resolve issues and maintain high SLAs. You can’t go grab your indispensable SME for troubleshooting, because that person is also home and inundated with messages and alerts from many different tools.

    Networks, VPNs and servers are being hammered extra hard right now from higher usage of SaaS apps and collaboration tools. User expectations are even more grandiose than normal: The home-based IT environment is a lifeline to whatever remains of human interaction. IT organizations are expected to make applications and services work flawlessly–overnight. Undoubtedly, IT pros are working with assiduity to keep the wheels of work moving for their employers during a time of crisis. Yet, working longer hours isn’t the answer. We need new ways of working—ones which may prove better than the current status quo for the long term.

    In many organizations, the process of incident management and resolution begins with people looking at screens full of metrics. You see a spike in something, such as network I/O, and immediately begin to investigate it, assuming the worst. A more adverse scenario is when users call out problems on Twitter, Reddit or other popular tech communities, creating a noxious PR situation which IT and the business must swiftly respond to and manage. This ad hoc, reactive fire drill takes time, and resolving that particular issue may not be the most important item of the day. In software development, it’s akin to the bad practice of designing the architecture of a system first and then the user experience.

    Given everything that’s at stake in the foreseeable future of stay-at-home mandates, SREs and IT support technicians need to flip the incident management process. Rather than beginning with the metrics and working backwards to uncover the problem, if there even is one, start by looking for the most important problems to solve. Then, discover the resources and components which support that end-user process and finally, analyze the metrics which are most relevant. For example:

    1. A B-to-C provider of a streaming video service is now seeing a 200% increase in daytime traffic. This is resulting in sluggish load times for users and glitches. 
    2. IT has tagged the components beneath that service, so IT operators know exactly which VMs, databases and load balancers come into play. 
    3. Next, the operations team can identify the most relevant metrics, for instance CPU idle time, to measure and take steps to correct. 

    The Problem-First Approach

    It sounds so simple: flip the process. Yet, this is not the case. IT operators and SREs are not accustomed to thinking in terms of which problems to solve or prevent. Being proactive or predictive hasn’t been the common workflow.

    Here is how IT operations and DevOps teams can take a proactive approach to managing the availability and health of enterprise services.

    Coordinate Across Departments

    Identifying the most critical problems to solve right now is the hardest step and the one which will take the longest. Common problems these days are related to cost and scale: How do I right-size my environment to reduce costs or handle more workloads? Broaden your field of inputs to narrow down to the specific issue, such as slow page loads on the financial reporting website. Product managers, account managers and business unit leads who are the closest to the customer experience can deliver feedback on the top issues affecting customer/user satisfaction. Teams should also review recent support tickets to identify common themes of pain.

    Fine-Tune Your Approach to Metrics

    The Utilization Saturation and Errors (USE) Method is one way to approach the problem-first incident management process. As detailed by Brendan Gregg, a senior performance architect at Netflix, this methodology begins by posing questions, seeking answers and working backwards to the metrics. For each resource that you want to measure, identify three metrics: one for utilization, one for saturation and one for errors.

    “The USE Method has made you aware of what you didn’t check: What were once unknown-unknowns are now known-unknowns,” Gregg explained.

    Standardize Workflows

    Create a common process for incident management across all your teams. Without the advantage of having almost everyone in the same room to huddle together in an ad hoc fashion when big issues crop up, it’s imperative to institute clear steps and roles. Doing so will prevent the frustrations, confusion and oversights that needlessly delay resolution. Since most incidents are composed of multiple contributing factors, teams need to adopt a small number of user-friendly tools to document and organize the information. At our company, we’re now using tools such as Miro, an online whiteboard application, to replace our physical whiteboarding sessions. Of course, there’s also Zoom, Slack, Jira and a host of other cloud-based tools already in place at many organizations. Mandate which tools everyone should use, with some guidelines on how to use them.

    Increase Automation

    In some organizations, scaling requirements in response to demand have increased tenfold. Automation is playing a critical role now; moving away from a web GUI, for example, is more scalable and aligns with modern tools such as Chef and Puppet. User tickets can be auto generated, for instance, from emails and linked to code management systems like GitHub. Modern development and operations teams are also expanding automation in unit testing and provisioning. 

    Watch for Burnout

    Whether there’s more work and/or a need to fill the hours during long quarantine days, many software engineers, testers and architects are working longer days right now. Yet, exhaustion and burnout can lead to errors and oversights along with low morale. It’s up to managers to make sure that employees are taking breaks, working reasonable days and having the time and energy to attend to personal needs.

  • PagerDuty Focuses Incident Response Efforts on the Cloud

    PagerDuty Focuses Incident Response Efforts on the Cloud

    After helping to pioneer the adoption of incident management software in on-premises IT environments, PagerDuty is now focusing its attention on the cloud with the launch of PagerDuty for Cloud Operations.

    Jukka Alanen, senior vice president of business development and corporate strategy for PagerDuty, said that as cloud computing evolves to include containers and serverless computing frameworks, it’s becoming apparent that managing IT incidents on public clouds is fundamentally different than on-premises IT environments. As such, demand for incident management platforms optimized for specific cloud platforms is increasing, he said.

    PagerDuty initially will focus its efforts on the Amazon Web Services (AWS) platform because that’s where most of the initial demand is coming from, Alanen noted. As part of that effort, an instance of the PagerDuty digital operations management platform running on AWS has been integrated with Amazon CloudWatch, AWS Personal Health Dashboard, Amazon EventBridge, AWS Security Hub, Amazon GuardDuty and AWS CloudTrail cloud services.

    Rather than replace all the capabilities already being provided by cloud service providers, Alanen said PagerDuty is committed to making them more accessible within the context of an incident management platform. With the rise of DevOps, the types of IT professionals who need to be able to access an incident management platform in the cloud era is also expanding. To resolve issues as quickly as possible, Alanen said it’s more critical than ever to make sure the right problems are being presented to the right teams within an IT organization.

    In terms of making that transition, however, IT organizations are all over the map. Many are trying to extend existing IT operations processes developed for on-premises IT environments. To the extent that organizations are essentially lifting and shifting monolithic applications running on-premises into a public cloud, that approach is feasible. However, once organizations also begin deploying cloud-native applications based on microservices and serverless computing frameworks, most will need to rely more on management and monitoring tools provided by cloud service providers.

    The next challenge then becomes defining workflows that automate responses to potential service disruptions or a security breach. At the same time, many organizations are adopting best DevOps practices that shift responsibility of many aspects of incident response on to the shoulders of developers who now need access to an incident management platform. The existing reactive approach to managing IT needs to be replaced with a proactive approach that automates responses to incidents whenever possible, Alanen said.

    To help organizations accelerate their move down that path, PagerDuty has crafted a Digital Operations Maturity Model, which provides benchmarking data and best practices for organizations looking to modernize their IT operations as part of a larger transition to increased reliance on automated workflow processes.

    Of course, once IT organizations define these workflows it won’t take long before IT organizations find themselves needing to extend more proactive approaches to incident management across multiple cloud computing environments. The challenge then becomes finding a way to weave all the application programming interfaces (APIs) and services exposed by multiple cloud service providers into what one day will become a truly hybrid cloud computing environment.

    — Mike Vizard

  • Four Steps to Implement an Observability Strategy for Microservices

    Four Steps to Implement an Observability Strategy for Microservices

    You’ve finally decided. You’re making the move to microservices and re-architecting your entire infrastructure. Some services will still be on-premises (those databases and their legacy apps are never moving), but a cloud-based microservices architecture will allow your teams to build and bring new products to market faster.

    But how do you handle the complexity that comes with cloud and microservices? You know you need a strategy for monitoring and managing the complex dependencies and ephemeral nature of your new architecture, but you don’t know where to start. What’s more, you don’t want to spend time and money now only to realize a few months down the line that you’re on the wrong road. Your investment needs to be future-proof.

    You need to implement a strategy for observability. Observability goes beyond monitoring to give you greater operational insight, leading to shorter incidents, fewer quality problems, a better product and happier customers. Your company has unique observability needs, which means how you approach your observability strategy will also be unique. As is usually the case, it’s important to start with a roadmap and goals. Based on my experience, the following four things are key to making a considered, comprehensive observability strategy that’s built for the long haul.

    Choose an Observability Platform that Fits Your Future

    Perhaps counter-intuitively, the right platform isn’t necessarily the one that’s the right fit for you today. If you’re investing in new technologies such as microservices, containers and Kubernetes, you’re going to see a significant increase in the scale and ephemerality of your environment. A monitoring platform that may suit your current infrastructure likely won’t be able to keep up and meet your future needs. Instead, consider how your infrastructure will look three, four or even five years down the road, and then select the platform that best fits that long-term strategy. It doesn’t matter if you choose a commercial tool or an open source one: What matters is that you give considered thought to your future infrastructure and choose a platform that can address the future needs of your environment—and protects your investment at the same time.

    Monitor the Metrics that Really Matter

    If you’re still in the early days of your shift from monitoring to observability, you’re likely taking a service-based approach, monitoring specific pieces of your ecosystem with a focus on individual incidents. The real value of observability is the ability to provide visibility into big-picture issues, such as measuring user experience, performance against SLOs and other key metrics important to the business. In modern environments, small upticks in a microservice can cascade into increased latency for an entire product, a specific feature, or even a specific customer. Having an observability platform that measures and collects data across multiple dimensions–e.g. customer/business-specific, application/service specific and infra/host/container specific–will allow you to understand the telemetry in many different ways. The platform should also aggregate the interactions between your services and unlock a wealth of data for engineers to investigate when issues do arise, and provides visibility into the metrics the business cares about most

    Embed Observability into Your Incident Management Process

    Incident management is a hot topic right now. I speak with CTOs across organizations and industries who have fantastic tools that provide detailed telemetry, but everything breaks down when an incident occurs. They need to better embed observability as a part of their incident management process. For example, ask your incident management team how observability helped in a given incident and record those results. This keeps track of the ongoing benefit. Next, ask how observability could be improved to get the team closer to the crucial clues they need during an incident. And finally, make sure to fold these items into your remediation plans so the team has a clear playbook when incidents do occur.

    Establish a Culture of Observability Across Your Organization

    Simply deploying a new technology of any type is not enough. You need to ensure you’ve built the appropriate processes and norms around the new platform to ensure its success—this is especially true with observability. Modern observability platforms can be incredibly powerful in providing insight into the performance of your ecosystem, but without an investment in your organization’s ability to use the platform, you’ll barely scratch the surface of its true value.

    Observability requires ongoing investment and improvement. Start by building a core group of champions, then fold training into onboarding, continuing education or brown-bag sessions. Using subject matter experts to create the content is a good way to get everyone off on the right foot. Use the output from observability tools in company meetings, team syncs or support settings to keep everyone on the same page. This sort of standardization raises the collective skill of the entire organization and demonstrates the value of observability to those who are not already on board.

    Following these four steps will set you off in the right direction on your observability journey. If your organization is like most, you keep an eye on what you think could go wrong. While that’s unquestionably important, a mature observability strategy will give you an insight into previous unknowns and help you more quickly understand why incidents occur. And as you continue on your observability journey and your understanding of what and why things break improves, you’ll be able to implement increasingly automated and effective performance improvements that impact your company’s bottom line.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Arijit Mukherji

  • Atlassian Integrates DevOps Portfolio

    Atlassian Integrates DevOps Portfolio

    At its Atlassian Summit 2019 conference, Atlassian announced updates to all the DevOps tools that make up Atlassian Data Center as well as upgrades to its Opsgenie incident management software to provide deeper integration with the rest of the company’s portfolio.

    At the same time, Atlassian revealed it has rebranded the AgileCraft business planning application it acquired earlier this year to Jira Agile.

    New capabilities added to Atlassian Data Center include the ability to set up smart mirror farms for Bitbucket Data Center; content delivery network (CDN) support for Jira Software, Confluence, Bitbucket and Jira Service Desk Data Center; rate limiting for Bitbucket, Jira Software and Jira Service Desk Data Center; issue archiving for Jira Software and Jira Service Desk Data Center; templates for deploying on Microsoft Azure and Amazon Web Services public cloud; and a mobile application for Jira Server and Data Center.

    Atlassian Data Center now incorporates Jira Software 8.0, Confluence 7.0, Bitbucket 6.0, Jira Service Desk 4.0 and Portfolio 3.0. The latest version of Jira Software 8.0 now executes advanced searches using Jira Query Language (JQL) 31% faster on average, while agile boards load two times faster and re-indexing times are now 71% faster. The latest version of Confluence adds support for multimedia files and additional analytics tools to the content management platform.

    The Atlassian Cloud service has also been expanded to increase the user limit to 5,000, with an early access program signaling an intention to bring that up to 10,000 users soon.

    Meanwhile, specific capabilities being added to OpsGenie include support for incident timelines to track response activities and key events; postmortems to help discover root causes and track remediation activities, a user interface that aligns with the rest of the Atlassian portfolio that fosters bi-directional workflow, the ability to automatically trigger events, integration with Slack collaboration software, the ability to encrypt data and Statuspage services integration to make it possible to share alerts with third-party services. Atlassian has also rebranded its Automation Actions to Opsgenie Actions; the company acquired Opsgenie last fall.

    Berkay Mollamustafaoglu, co-founder and head of product for Opsgenie, said now that Opsgenie is part of the larger Atlassian product family there’s clearly more focus on helping organizations integrate best DevOps practices with more structured processes based on the ITIL format. The latest version of the ITIL format provides makes those processes more flexible, but there’s still plenty of room for Atlassian to improve the tenor of the conversation across what has become a bi-modal IT operations environment. As the level of synchronization across those processes increases, the amount of fatigue IT operations teams experience lessens as the number of alerts that need to be correlated across multiple platforms decreases, Mollamustafaoglu noted. In fact, rather than convening in a physical “war room” to resolve a issue, most incidents will increasingly be resolved via online chats, he said.

    It may take a while before a period of détente emerges between advocates of DevOps and ITIL processes inevitably emerges. In the meantime, the more IT services management (ITSM) platforms become integrated with one another, the easier it becomes to achieve that goal. In fact, the major obstacle that needs to be overcome are much more likely to cultural rather than the level of integration that can be achieved using any specific ITSM platform.

    — Mike Vizard

  • xMatters Looks to Meld DevOps and ITIL

    xMatters Looks to Meld DevOps and ITIL

    With the launch today of a flow design tool from xMatters, the divide between DevOps and ITIL-based approaches to managing IT is narrowing.

    xMatters Flow Designer is a drag-and-drop graphical tool that either developers or IT operation teams can employ to create incident management workflows on top of xMatters incident management software. Doug Peete, chief product officer of xMatters, said those workflows feed data into a variety of third-party IT management tools to enable IT operations and DevOps teams to consume the same data in different tools—for example, in an instance of ServiceNow for IT operations and as a message in Jira or Slack for DevOps teams.

    Peete said the goal is to enable organizations to develop a more consistent approach to incident management. A recent survey of more than 1,000 IT professionals conducted by xMatters and Atlassian found that while 81 percent of developers and operation teams share information and tools, only 17 percent provide open access to information without time restrictions. The survey also found that 50 percent of respondents said the tools, processes and steps they employ vary from incident to incident.

    Most of those organizations have very little in the way tools that enable them to effectively conduct a post-mortem analysis of those incidents, added Peete.

    The arrival of the xMatters Flow Designer tool comes at a time when organizations are trying to find ways to meld the highly structured ITIL processes that most IT organizations rely on today and the inherently more agile approaches based on best DevOps practices. The most recent iteration of the ITIL framework incorporates some DevOps principles, but it’s apparent the two approaches to managing IT will be employed side by side in many organizations for many years to come. Developers these days tend to prefer DevOps methodologies, while IT administrators tend to appreciate the consistency of an ITIL framework that was designed to address IT governance concerns. By providing a set of workflow tools that span both those approaches, Peete said IT organizations can more easily meld those two approaches on their own terms.

    At the same time, an extensible approach to incident management makes it possible for IT organizations to incorporate new tools into those workflows as their IT environment evolves. That capability also should make it easier to integrate incident management workflows across multiple organizations as well as incorporate a new IT environment as a result of acquiring another company.

    Regardless of the path chosen, it’s clear that as organizations increasingly realize how dependent their business processes are on software in the age of digital business transformation, the more likely it is they will embrace DevOps practices. However, that doesn’t mean organizations need to abandon ITIL; there are plenty of IT processes that benefit from a highly structured approach. In fact, arguably the most important thing is to not let a philosophical debate over the merits of DevOps versus ITIL wind up becoming a major distraction that creates two isolated cultures who refuse to collaborate with one another.

    — Mike Vizard