devopsy.kinsta.cloud

Tag: application performance management

  • Predictions 2020: The Future of APM Strategies and Tools

    Predictions 2020: The Future of APM Strategies and Tools

    As 2019 comes to an end, it’s safe to say that multi-cloud strategies are here to stay. We can predict that 2020 will bring an ever-increasing demand for application performance management (APM) tools and services.

    So what does the future of APM look like? Let’s dive in.

    Multi-Cloud Strategies Increase the Demand for APM Tool Adoption

    Companies are increasingly adopting more than one platform—either for financial leverage or to create a time-to-market or feature race between the platforms. To remain competitive, public cloud providers must offer unique features or capabilities differentiating them from competitors. This has created an upsurge in new and more complex technologies, increasing the need for APM tool adoption.

    Logs Will Be Recognized as Key Aspects of APM

    APM is a puzzle made up of user experience, metrics and traces, and logs—and each piece must be incorporated to achieve full-stack visibility and ultimately seamless app performance. However, a healthy app is not characterized solely by top performance, but also availability—making a log management solution that monitors availability, enables proactivity and reacts quickly to remediate problems key.

    In 2020, implementing solutions that achieve integrated application performance and availability management for any type of organization, including those operating on-premises and/or cloud-based applications across all mixes of cloud environments, will become increasingly important and prioritized.

    APM Skills in a Cloud-First World

    The nouns have changed, but the verbs remain the same: APM skills are among the top skills tech pros want to develop in the coming year, according to the recent SolarWinds IT Pro Day survey, Building Confidence for Tech Pros of Tomorrow. However, in 2020 tech pros will begin to recognize that many of the skills they already possess are sufficient to manage the new realm of integrated solutions in the cloud. Tech pros can leverage some of the same skills they could before as the nouns have changed, but the verbs remain the same. Where we used to have server or virtual machine, now we have containers. But the verbs? They’ve always been focused on optimizing performance, ensuring availability and planning capacity.

    In 2020, the extent to which tech pros will discover they don’t need to fear new nouns (more than they did when they moved from client virtualization) will increase.

    Universal Adoption of APM Tools

    As environments become more dynamic and complex, the opportunity for applications to break increases—creating the need for robust monitoring tools. In 2020, as buyers see the benefit of leveraging APM tools on a regular basis rather than only in critical moments, the market will adjust accordingly. We can expect easier-to-implement solutions that are more highly valuable and affordable. The paradigm that APM tools have to be expensive will be broken once and for all.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — David Wagner

  • Dynatrace Extends APM Reach Into Business Analytics

    Dynatrace Extends APM Reach Into Business Analytics

    A growing number of providers of tools widely employed by DevOps teams that are extending their reach into business analytics has been joined by Dynatrace. The provider of a namesake application performance management (APM) platform unveiled a set of business analytics applications aimed at application owners who need to understand how changes to the  application experience their organization is providing ultimately impacts the business.

    Ajay Gandhi, vice president of product marketing for Dynatrace, said Digital Business Analytics applications from Dynatrace extends the single-agent architecture Dynatrace created for its Software Intelligence Platform to make it easy to employ an APM service delivered via the cloud into the realm of business analytics.

    The difference between the applications being made available by Dynatrace and traditional business intelligence (BI) applications is the agent software Dynatrace developed enables application owners to track all events in real-time, said Gandhi.

    As organizations launch digital transformation initiatives, Gandhi said many discover they lack visibility into the actual end user experience. The suite of Digital Business Analytics applications provides access to a set of applications that enables application owners to track transactions as well as metrics such as conversions and revenue performance by product, customer segment and geography.

    Gandhi noted organizations don’t necessarily need to have embraced best DevOps practices to derive value from the company’s analytics applications. However, those that have embraced DevOps are likely to appreciate the value of these applications because they view themselves as a software company that happens to build a specific product or digital service.

    It’s not clear to what degree application owners may be willing to give up relying on spreadsheets and legacy BI applications they have relied on for decades to manage the business. However, a case can be made for aligning the business to IT more closely by tracking the events in real-time as they occur across application environments and the IT infrastructure on which they depend. In fact, the one thing that has always differentiated Dynatrace is its ability to leverage an artificial intelligence (AI) engine, dubbed Davis, to surface events and deliver actionable intelligence by identifying the root cause of an issue, said Gandhi.

    Other DevOps vendors that are venturing into the realm of business analytics include CloudBees and GitLab. As these vendors increasingly realize that business processes now frequently revolve around DevOps processes, they all see an opportunity to extend their franchises by providing analytics applications that are aimed well beyond their existing base of IT users. Less clear right now is which set of tools within a DevOps toolchain will serve as the primary source of data for those analytics applications. However, Gandhi did note that unlike continuous integration/continuous deployment (CI/CD) platforms, APM services can provide real-time visibility into the actual digital experiences that change as various events impact production applications.

    Regardless of the ultimate source of data, however, the one thing that is certain is the level of insight business users can have into application development and deployment processes trends is, for better or worse, about to increase substantially.

    — Mike Vizard

  • Understanding and Implementing the APM Puzzle – Performance and Availability

    Understanding and Implementing the APM Puzzle – Performance and Availability

    While every organization is unique, there’s one universal goal everyone can get behind—creating and delivering an exceptional end-user experience. For both classic IT organizations and companies built in the age of cloud, delivering continuous availability and appropriate application performance is key.

    Although digital transformation has become a buzzword some would say is meaningless, the idea of digitizing a traditional company and building new businesses whose foundations are established upon the latest web technologies is critical in today’s IT and non-IT organizations. Think about the prevalence of cloud migration efforts during the past several years. Traditional organizations typically prioritize on-premises applications with increasing quantities of hybrid applications. But, because they aren’t yet fully transitioned to cloud—whether IaaS, PaaS or SaaS—their IT organizations need to live in both universes.

    Monitoring these environments at both the infrastructure and application tiers and across each stage in this transition requires support of a full stack of tools to cover web uptime, user experience, transaction monitoring, application tracing, log analysis and appropriate infrastructure monitoring. That said, modern organizations that have fully embraced cloud and leverage modern microservice architectures have similar challenges, but the exact same level of monitoring visibility requirement.

    Whether public, private, hybrid or multi-cloud, nearly every organization has moved to the cloud in some capacity. Regardless of what the company produces or the services IT provides, it must maintain uptime, availability and seamless performance to create and continuously deliver a user experience rivaling competitors and exceeding user expectations.

    Because let’s face it, even though every business is different and virtually every market is saturated, consumers and B2B companies have more options than ever. Without delivering a solid end-user experience, there’s an ever-increasing plethora of alternative vendors and providers from which customers can choose.

    Public Cloud Platforms for Classic IT Organizations and the Cloud Generation

    Let’s explore a few differences between how traditional IT organizations and cloud-era companies interact with the cloud. At a high level, Azure is a natural choice for IT companies with significant investments in apps traditionally acquired (not built) by the company and managed by their IT team, that are now tasked with digitizing their business by leveraging public cloud platforms. These types of companies often have Microsoft Office, Microsoft Exchange and other Microsoft products already implemented on-premises, and Infrastructure-as-a-Service (IaaS) or Platform-as-a-Service (PaaS) solutions as well. If they’re now adding a public cloud platform to their mix, staff expertise and solution familiarity make adopting Microsoft Azure a natural next step. Microsoft has long recognized the need to make these types of transitions and mixtures easier, demonstrated by its Office365 suite (as an example).

    On the flip side, companies that were born fully in the cloud, (i.e., relatively newer organizations without a legacy anything on-premises, with respect to IT infrastructure) will likely have a higher propensity to choose a provider such as AWS or Google Cloud even though Microsoft, Oracle and others are certainly competing there as well. So, it’s not a matter of one being intrinsically better or more fully-featured, but about which is the best fit for the organization and its priorities.

    At the end of the day, however—regardless of organization type or public cloud platform implemented—every company needs monitoring tools and solutions that keep applications up, running and performing seamlessly. With an integrated set of simple, easy-to-use, quick to value and powerful solutions, organizations can achieve truly proactive lifecycle assurance of application availability and performance.

    Uptime and Seamless Performance: Putting Together the APM Puzzle Using All the Pieces

    Each and every piece of the APM puzzle must be incorporated to achieve full-stack visibility and ultimately top performance: user experience, metrics, traces and logs. Having a log management solution monitoring availability, enables proactivity and enables IT to react quickly to help identify and remediate problems is paramount—it’s table stakes that everything be available. After all, if a website or page isn’t accessible, business transactions cannot be processed and that business is effectively down.

    Creating and maintaining a healthy application involves not only ensuring it’s functionally available to users; it must also be performing appropriately—continuously delivering both a reasonably responsive user experience and supporting the required volume of transactional work. This means a complete view into all aspects of performance—from the website and webpages all the way into and across all relevant application components and their associated infrastructure—be proactively visible and monitored.

    There is thus a natural and increasingly urgent workflow affinity between logs and APM—across the development lifecycle. When errors occur, it’s critical to understand, precisely and immediately, where are the impacts on service performance, or if there even are any impacts. Conversely, when application performance issues arise, it’s critical to immediately know exactly which errors—across the complex and constantly changing infrastructure and application stack—are directly related to the problem at hand.

    Without this bi-directional workflow integration, not only can application performance and availability not be managed proactively, but any issues with application service availability and performance will take far longer to diagnose and repair—seriously impacting the business bottom line.

    Just because everything is available doesn’t necessarily mean it’s appropriately performing for the end user trying to use the application.

    With all this in mind, it’s paramount to implement powerful, easy-to-use solutions that’ll achieve integrated application performance and availability management for any type of organization operating on-premises and/or cloud-based applications across all cloud environments.

    Conclusion

    While traditional IT organizations and cloud-only companies are different in many ways, providing exceptional end-user experience and achieving seamless availability and performance (no matter what) is a common and increasingly critical business goal. To achieve business success, organizations should choose a complete suite of simple, powerful and integrated solutions to optimize and manage their cloud-hosted applications and platforms. Anything less leaves the APM puzzle unsolved.

    — David Wagner

  • Splunk to Acquire SignalFx to Gain APM Platform

    Splunk to Acquire SignalFx to Gain APM Platform

    Splunk today announced it will acquire SignalFx as part of an effort to extend its reach beyond IT operational analytics and into the realm of application performance management (APM).

    Valued at approximately $1.05 billion, the deal to acquire an APM platform delivered as a software-as-a-service (SaaS) offering will expand and accelerate Splunk’s overall transition to a recurring revenue model based on cloud services, said Splunk CEO Doug Merritt during a call with industry analysts.

    Splunk is currently on pace to generate over $300 million per year from the cloud version of its IT operational analytics platform. Splunk revenues for the second quarter were $517 million, up 33% year over year, with software generating $350 million of that total. However, as Splunk transitions to more of a subscription-based pricing model, the company has some financial management challenges. GAAP operating loss for the company’s second quarter was $87 million. However, non-GAAP operating income was $47 million, even though operating cash flow was negative $129 million. That cash-flow issue is a direct result of the transition away from a perpetual software license model based on the volume of data consumed by the Splunk platform. Overall, Merritt said Splunk is firing on all cylinders and is still on track to have 20,000 customers by the end of its fiscal year.

    Merritt said once the deal is approved, sometime in the second half of fiscal 2020, SignalFx will provide a complementary set of services for monitoring legacy monolithic applications as well as cloud-native applications based on containers, Kubernetes and serverless computing frameworks. There are very few customers at this point who have not adopted cloud-native technologies, but for now they remain a small percentage of the overall number of application workloads being managed, he added.

    The goal is to provide a suite of services that will enable IT teams to investigate, analyze, monitor and act on data in real-time, said Merritt.

    Splunk is already widely employed by internal IT operations teams, many of whom already rely on a complementary APM platform. Splunk clearly expects organizations will prefer to acquire an IT operational analytics platform and an APM service from the same vendor. However, competition between APM platform providers is already fierce. That said, Splunk appears to be counting on increased demand for real-time application monitoring to differentiate SignalFx from rivals.

    It’s not clear to what degree Splunk might converge the various services now being provided separately by both companies. It’s also not clear how or whether this deal will accelerate acquisitions of the remaining independent APM platform providers.

    In the meantime, Merritt said SignalFx and Splunk’s earlier acquisition of VictorOps, provider of incident management platform based on best DevOps practices, will expand the amount of visibility Splunk has with application development teams.

    Ultimately, acquisitions of smaller, DevOps-oriented companies by vendors that today are still more closely identified with legacy ITIL-based IT processes, such as Splunk, are only like to increase. What will be interesting to watch is whether the tools and services being acquired by those companies wind up being the DevOps tail that wags the proverbial ITIL dog.

    — Mike Vizard

  • Datadog Fetches Bevy of Additional DevOps Services

    Datadog Fetches Bevy of Additional DevOps Services

    At its Dash 2019 conference today, Datadog launched a series of DevOps initiatives that significantly expand the use cases for the agent software that drives the company’s monitoring services.

    Additions to the Datadog portfolio include Network Performance Monitoring and Real User Monitoring (RUM) services as well as general availability of a previously announced service for monitoring serverless computing frameworks.

    In addition, Datadog announced it is adding Tracing without Limits to its application performance monitoring (APM) and distributed tracing service that allows DevOps teams to search all traces without live sampling and store the traces matching bad user experiences for the long-term.

    Finally, Datadog is adding support for log rehydration and the ability to generate custom metrics from ingested log data residing within its log management service.

    Ilan Rabinovitch, vice president of product and community for Datadog, said the company continues to invest in expanding the core use cases for which its agent software can be applied. Over time, that approach will enable DevOps teams to rationalize the types of agent software they need to deploy to monitor applications, cybersecurity, infrastructure, networking and user experience, he said.

    Rabinovitch noted that interest in extending the reach and scope of Datadog services into the realm of networking is especially high because DevOps teams today have little visibility into how the various software-defined network (SDN) services provided by cloud service providers are impacting their applications.

    DevOps teams are also now being asked to present IT metrics in a way that reflects actual end user experiences. The RUM service being provided by Datadog achieves that goal by making it easier to track front-end latency metrics such as time to interactive and time to first byte, in addition to providing a geographic location, the operating system and browser types being employed by the end user.

    Rabinovitch noted that as organizations become more dependent on digital business processes, they are moving to instrument every aspect of delivering IT services. For example, APM in the past might have been limited to a handful of mission-critical applications running on-premises. Now, as more digital business applications become distributed across both cloud and on-premises IT environments, the need for visibility across interdependent applications and microservices has become more acute, he said, adding most organizations would prefer to consume monitoring as a service versus having to deploy and maintain that software themselves.

    It’s unclear how being able to apply a common base of agent software might accelerate the convergence of job functions within an IT organization. But it does make it easier for cross-functional IT teams to collaborate and troubleshoot IT environments so all members of a team can use a common management platform to identify issues. In many cases, the amount of time saved by eliminating the need to convene a “war room” meeting whenever a new issue is discovered might in and of itself be priceless.

    — Mike Vizard

  • Application Performance Management for DataOps

    Application Performance Management for DataOps

    Over the years, application performance management (APM) tools have been utilized typically to measure and monitor the performance of software such as web applications. They do so by tracking such things as request response times and error messages, thus providing relevant data to be analyzed by the DevOps team, which then can make the appropriate changes to optimize their software. But what about the effectiveness of the data workflows that contribute to the functionality in many of the applications out there today? How can performance monitoring toolsets be extended to provide these workflows with opportunities for continuous improvement?

    Data Pipelines and Performance Challenges

    Below, I will discuss the use of APM tools in the context of DataOps. I will delve into the metrics and insights an APM tool must provide to engineers so they can support data pipelines efficiently, showing how the effective use of data-centric APM tools can provide extensive value to a DataOps team and their organization.

    Data Pipelines and DataOps Engineers

    A successful DataOps implementation requires creating automated data workflows that analyze large amounts of data quickly, while providing high-quality analysis as an end product. It’s easy to see how the job of a DataOps engineer can be a complicated one. It is their job to build and manage these pipelines to help the business achieve certain goals that rely upon the resulting data analysis. In other words, DataOps engineers are responsible for making sure high-quality data is made available as quickly as possible.

    Consider a common scenario with which we are all familiar: online shopping. With each purchase you make from a particular e-commerce website, the system typically provides recommended products for you to buy. This is often the result of a data workflow that produces data analysis for a recommendation system, which is then leveraged by the application to provide the end user with these recommendations. This pipeline is likely to be made up of several processes.

    These processes include those for loading large sets of raw data into a distributed file system for storing data across a cluster. This would be followed by several processes that need to run on this cluster to format and analyze this data to produce the quality data for the recommendation engine being leveraged by the web application. A lot is being done with large sets of data. And, as is usually the case with so much happening, there is also much that can go wrong.

    The Challenges in Managing Data Pipelines

    There are several major performance challenges that come with the management of the types of pipelines mentioned above. Below are a few of the common threats to data pipeline performance:

    • Scalability: One of the more obvious challenges when dealing with the construction of a data pipeline is scalability—in other words, the effectiveness and efficiency of the pipeline when it needs to deal with a much larger dataset. It’s likely when the jobs were written to deal with such data transformation and analysis that they were tested against smaller sets of data in development environments. What happens when this is no longer the case? Maybe now instead of processing a relatively small amount of data in a test environment, it needs to process much more as it runs in production. Will the processes still perform adequately to deliver quality analysis in a time-efficient manner?
    • Application Failures: Data pipelines rely upon parallel processing. But what happens when you experience the failure of one or more of these parallel-running processes that are executing to clean and analyze the data? This can throw a wrench in your ability to relay quality, complete analysis to the business unit. Being able to resolve failures quickly and get the pipeline back up and running plays a huge role in the success of a particular data workflow.
    • Fragmentation: Most data pipelines consist of a variety of tools that handle different tasks (such as ingestion, storage, processing, analysis, etc.). With so many different tools in the mix, pipelines can become fragmented. Tool incompatibility issues can arise and data formats can vary. It’s important to be able to unify data within the pipeline, even as it moves between disparate tools.

    APM Tools to the Rescue

    Manual vs. Automated

    Enabling DataOps within your organization requires continuous feedback and continuous improvement to your data workflows. A good place to implement these strategies is with the use of APM tools to help manage the performance of these pipelines.

    Consider the processes for improvement that occur without the use of APM tools. It would be on the IT and DataOps folks to locate performance-related problems manually within these pipelines and begin digging through log files to find the cause. A significant amount of time likely would be added to resolving these failures due to the lack of a centralized location to view this information, the lack of automated alerts to notify the team to these performance snafus and the lack of assistance in deriving insights from collected performance data. APM tools can provide an intuitive UI for debugging such issues and can provide the type of insights needed to detect and resolve these problems quickly.

    That being said, monitoring for big data processes is quite different from other types of application monitoring. Certain APM software will be better-equipped for dealing with data pipeline-related challenges than others. Much of this depends on the ability of the APM tool to integrate with the platforms and frameworks used within the data workflow, as well as the metrics the performance management software collects and utilizes to help the DataOps engineers refine their processes.

    What Types of Metrics Can Help Determine the Causes of Issues Within a Data Pipeline?

    Earlier, I discussed data pipeline performance challenges stemming from scalability issues and process failures. Let’s discuss what types of metrics should be collected to resolve issues like these in a data workflow.

    Before delving into the metrics, it’s important to note that having full-stack data is essential. By full-stack data, I mean not just infrastructure metrics (which are the most obvious type of metric folks tend to think about when managing data pipelines), but also other data points such as user data, application data and resource data. All this data needs to be unified and correlated across all systems in the pipeline to manage it effectively.

    Consider scalability, for instance. The APM solution you are using must be able to track several key metrics to help identify scaling-related bottlenecks within the data pipeline. These include:

    • The Amount of Data Being Processed: How can we identify a scalability issue without knowing how much data is being processed by a particular run? The answer: We can’t. Being able to quantify the amount of data being handled by each iteration of the data pipeline is the first step in detecting if the workload is the problem.
    • The Length of Each Run for Each Individual Process: The APM software being used by an organization must be able to integrate with the structure of the data pipeline to report the length of each run for each individual job running within the workflow. If a particular job was completed in one minute when processing X amount of data but took 20 minutes when processing 10X, then we have diagnosed a potential scalability issue and narrowed it down to the root cause in a time-efficient manner. Maybe the fix is as simple as tuning some SQL, and the scalability issue can be rectified in relatively little time.
    • Resource Utilization: Certain processes may require more resources than others to complete in a timely fashion. The ability of the APM tool to track the amount of resources being utilized by each process is key in determining if simply reallocating certain resources can fix problems with performance.

    Example issues:

     

     

    Unravel is an example of an APM tool that can track all the above while organizing the data within a user interface that provides DataOps engineers with just about all they need to diagnose data workflow performance issues from one screen. In addition, Unravel employs AI solutions to help take the metrics they collect and provide the engineers with specific insights as to how they can optimize their pipeline. This helps decrease the amount of time DataOps personnel spend on managing existing processes, allowing them to spend more time creating new processes to improve the quality of analysis being provided to the business.

    Conclusion

    APM tools are no longer solely for ensuring application availability. APM tools for big data, such as that from Unravel, have allowed for performance monitoring and analysis within data workflows, helping to ensure the availability of crucial, high-quality data analysis. Any organization looking to improve the quality of their data analysis processes should be looking to extend their APM toolset to include those for big data.

    This sponsored article was written on behalf of Unravel.

    — Scott Fitzpatrick

  • Survey Finds Reliance on APM Mainly Reactive

    Survey Finds Reliance on APM Mainly Reactive

    A survey of 358 strategic IT decision-makers in enterprise organizations across North America and Europe conducted by the market research firm GigaOm on behalf of SolarWinds, a provider of IT management tools, finds that roughly half of the respondents consider themselves proactive in their use of application performance management (APM) tools.

    That definition is based on those respondents either strongly agreeing or agreeing that their organization was employing APM tools within the context of all the following criteria:

    1. Engaged in a digital business transformation initiative.
    2. Makes use of a hybrid or multicloud computing strategy.
    3. Has a public-cloud first IT strategy.
    4. Makes use of best practices for IT operations.
    5. Line of business teams are driving technology change.

    Denny LeCompte, general manager for application management at SolarWinds, said that while significant progress is clearly being made it terms of APM adoption, it’s clear that half or more of the potential market is still wrestling with a variety of challenges that often stem from the cost and complexity associated with acquiring and employing an APM platform at scale.

    LeCompte said most of the use of APM is still reactive in that most organizations don’t actively engage an APM tool until there is an actual issue. The survey finds the primary driver for adopting APM is to fix problems (61%), followed by keeping on top of complex application environments (57%), delivering the best possible customer experience (52%) and managing costs and delivering efficient operations (34%). However, it’s worth noting that among the half of the respondents claiming to be proactive about employing APM, customer experience was the primary driver for adopting APM (61%).

    Even so, LeCompte noted reliance on APM platforms is often uneven at best. When it comes to adoption, he noted, most organizations are on a journey. Initially, the focus is on IT troubleshooting. Over time, however, customer experience with digital business processes becomes a much bigger factor. Different teams with the same organization are at different stages of that journey, he said.

    SolarWinds views the rising tide of APM awareness as the justification for jumping into the category, LeCompte said, claiming the company is committed to providing APM capabilities an IT organization is likely to require, at a fraction of the cost of rival platforms. That’s critical because many organizations have limited the number of applications they track using APM tools because of cost, he said.

    As IT environments become more complex in the age of microservices built using highly ephemeral containers, LeCompte said it’s only a matter of time before more IT organizations will need to embrace APM platforms proactively to optimize overall application performance. Individual IT administrators won’t be able to keep track of all the dependencies that might be impacting a “cloud native” application without relying on an APM platform, he noted.

    The issue now is finding a way to provide access to an APM platform that everyone on the IT team can regularly employ without breaking the IT budget.

    — Mike Vizard

  • The Changing Role of APM in a Microservices World

    The Changing Role of APM in a Microservices World

    Microservices usage has skyrocketed. Fully 91 percent of organizations are using or have plans to use microservices and 92 percent expect to grow their use of microservices in the coming year. It is no longer if but when: 86 percent of organizations expect microservices to be the default within five years.

    With the growing use of microservices and containers, applications have become orders of magnitude more dynamic. It’s not unusual for containers or building blocks of the application to have a shelf-life of less than a week. However, what is great for application development can create challenges when it comes to monitoring ­and diagnosing performance issues in ever-changing applications. How does application performance monitoring (APM)  change to support microservices and dynamic cloud-native apps?

    More Moving Parts

    Small, specialized microservices may facilitate shorter development cycles, continuous releases and greater agility, but when deployed across dynamic cloud infrastructure, the sheer number of moving parts also makes it much more difficult to troubleshoot application performance. Highly distributed and transient containerized components need to be tracked and measured continuously, demanding a big data approach for APM tools. To be effective, monitoring containers and microservices requires high-definition visibility across all transactions, machine learning to pinpoint problems and a shift in focus to users from servers.

    High-Definition Data

    Because the dynamic, containerized environment is constantly changing, measurement data must be collected at high frequencies—think seconds rather than minutes. Collecting infrastructure performance metrics at one-minute intervals, or longer, can distort the real situation and sometimes miss entire events altogether. For example, at low definition, a pattern of recurring but brief spikes in CPU usage will either be averaged out or not register at all—but if detected would be a tell-tale sign of a systemic configuration issue. High-definition infrastructure metrics, collected at one-second intervals, are crucial for a comprehensive view of resource utilization and shared resource dependency issues.

    However, infrastructure metrics alone are not enough. Building a complete view of the operating environment calls for data from multiple sources, including application traces, system metrics, application logs and network behavior. The resulting data set is very large, necessitating big data collection methods and processes. In-line data compression, NoSQL back-ends, non-intrusive client agents and cloud scaling are foundational technologies for modern APM tools.

    Correlating Data End to End

    Big data analytics and visualizations are necessary to process APM data into human-scale patterns and events. Identifying root cause of a performance problem often involves following a transaction from end to end, through multiple containers, shared resources and downstream infrastructure dependencies. Next-generation APM tools dynamically map the flow of every single transaction, through all intermediate relationships. Machine learning and artificial intelligence, while still in their early stages, are proving essential to finding patterns and anomalies in the data, suggesting probable causes and flagging them for human investigation. Combining the best of machine and human skills results in faster problem identification, diagnosis and resolution. IT operations teams benefit with improved application performance, reduced repair times and higher uptime even in dynamic application environments.

    Putting the Focus on the End User

    Legacy APM tools tend to focus on infrastructure components and resources, monitoring the health and utilization of servers, storage and networks. Organizations today, driven by digital competition and customer satisfaction, need to concentrate on the user’s perspective. This means monitoring multiple types of user devices for health indicators and mapping each transaction to the backend system. Automatic instrumentation delivers real-time end user metrics that trigger alerts much earlier than server-side monitoring, and enables more effective prioritization. End user insights also help DevOps and product owners track feature and application adoption rates, compare performance before and after upgrades and directly relate technical issues and investments to business priorities.

    Monitoring Apps for Today and Tomorrow

    Clouds, containers and microservices require a different approach to performance monitoring that takes into account the speed, size and life cycle of application modules. Effectively watching these distributed and transitory processes demands a big data approach that can handle the scale of end-to-end transaction data in modern enterprise environments.

    Legacy APM tools that only take snapshots result in an incomplete view of application reality and lengthen problem resolution times. Instead, modern APM systems collect a complete data set, enabling DevOps to quickly resolve intermittent and complex issues that impact business productivity and user satisfaction. They augment their skills with machine learning to quickly identify anomalies and guide them to probable causes. The results are better application performance, enhanced competitiveness and perhaps most important, increased user satisfaction.

    — Gayle Levin

  • AppDynamics Applies AIOps to Automate IT

    AppDynamics Applies AIOps to Automate IT

    AppDynamics, a unit of Cisco, today announced it is melding the worlds of AIOps and application performance management (APM) with the launch of Cognition Engine.

    At the same time, AppDynamics is making good on a promise to integrate metrics drawn from Cisco Application Centric Infrastructure (ACI), the software-defined networking (SDN) that Cisco is embedding in all its networking equipment.

    Finally, AppDynamics also announced Serverless Agent for AWS Lambda, which marks the first time the company’s APM platform has been extended to include the ability to capture metrics from every single invocation of a serverless computing framework.

    Matt Chotin, senior director of technology strategy at AppDynamics, said Cognition Engine represents an ambitious effort based on machine learning algorithms and predictive analytics to create a central nervous system for IT spanning applications, infrastructure and the network. The days when applications and infrastructure were managed in isolation are now coming to an end, he said.

    Available at no extra charge to AppDynamics customers, Cognition Engine infuses the Business Transaction data model underpinning the AppDynamics APM services with machine learning algorithms to run diagnostics and automate root-cause analysis in a way that reduces resolution times from minutes to seconds. The machine learning algorithms evolve constantly as data is gathered continuously using streaming analytics to ingest, processes and analyze millions of records per second. The Cognition Engine automatically understands how metrics correlate and instantly knows when there’s been a break in correlation among datasets.

    Cognition Engine also automatically isolates metrics that deviate from normal behavior to identify the top suspects of root cause for any application issue. As an open platform, that information then can be shared with orchestration and automation systems embedded within third-party incident response, intelligent workload placement and cloud cost optimization tools and platforms.

    These types of capabilities are now becoming an absolute requirement because IT in general is becoming too large and dynamic to manage relying on manual processes, said Chotin, adding what’s needed is a shift toward an AIOps mindset that automates the management of routine IT tasks at scale.

    AIOps is, of course, not so much a thing that can be purchased as much as it is a collection of technologies spanning machine learning algorithms, predictive analytics and other technologies that are employed in unison to automate IT operations. It usually takes a while for algorithms to learn a specific IT environment. But once they do, they bring with them the benefits of a more sophisticated approach to managing IT at scale (and they never take a sick day or decide to take another job). Of course, AIOps is not likely to replace the need for IT staff entirely. But roles and functions within IT organizations most certainly will evolve in the months and years ahead.

    In the meantime, organizations will increasingly vet AI models before deploying them in production environments. While AIOps will advance the state of IT as it is known today, the most worrisome thing about any form of current AI technology is being catastrophically wrong at a level of scale that is nearly impossible to recover from.

    — Mike Vizard

  • xMatters Incorporates Stackdriver APM Metrics from Google

    xMatters Incorporates Stackdriver APM Metrics from Google

    As a set of application monitoring services on the Google Cloud Platform (GCP) for driving DevOps processes, Stackdriver has been steadily gaining adherents, especially since Google extended the reach of that service into the realm of on-premises systems. Today xMatters, a provider of incident management tools, announced it is integrating alerts and data generated by Stackdriver into its workflow processes.

    Company CTO Abbas Haider Ali said the xMatters incident management software will now automatically relay Stackdriver data to the correct people and systems to help coordinate and resolve incidents faster. Stackdriver insights will be embedded into notifications, collaboration invitations and service management tickets, said Haider Ali.

    The goal, he said, is to provide DevOps teams with more context concerning specific issues directly within the user interface employed to access the incident management platform.

    Google, via Stackdriver, not only provides access to a tool for application performance management (APM), it also effectively embeds best practices for site reliability engineers (SREs) inside the service. While the best practices defined by Google are highly opinionated, they are starting to gain traction as a specific methodology from implementing a DevOps process, said Haider Ali.

    Support for Stackdriver follows the recent release of an update to xMatters that added an Event Flood Control capability to suppress similar requests in close succession as part of an effort to reduce the level of noise generated by a single incident. The company also added an Unlimited Event Visibility feature allows organizations to unlock insights from past incidents to identify a potential root cause of a problem faster.

    Of course, while Google may have invented the SRE concept, it is clear other cloud service providers are coming up with their own flavors and many internal IT operations teams will likely create their own equivalents. But as the SRE concept continues to gain traction, the rate at which DevOps processes are adopted enterprisewide should increase. The challenge most enterprise IT organizations face today is their internal teams are largely made up of administrators rather than engineer that know how to programmatically manage infrastructure as code. Many of those organizations are either hiring former developers to become SREs, or investing in training to up level the skills of their existing IT administrators.

    But as more organizations embrace SRE concepts many of them will be looking to create closed-loop approaches to DevOps that enables them to significantly reduce the amount of time between when an issue or incident is discovered, and it gets assigned within a continuous integration/continuous deployment (CI/CD) environment to be programmatically addressed.

    In the meantime, any gaps that may have existed between DevOps processes and incident management systems are continuing to narrow. Not every organization necessarily has to hire SREs to achieve that goal. But those organizations that can afford to hire and retain SREs should expect they will insist on incorporating incident management with the context of a larger set of sophisticated DevOps processes.

    — Mike Vizard