devopsy.kinsta.cloud

Tag: apm

  • The Battle for the Frontend: Rethinking APM

    The Battle for the Frontend: Rethinking APM

    With the battle for the backend between data centers and cloud largely over, application development is the next frontier ripe for innovation. Studies project mobile app revenue at $935.2 billion by 2023 and Internet of Things (IoT) installations at $75.44 billion by 2025. It’s no surprise that software innovation is ramping up around application development in a big way. At the same time, the desktop is still a primary interface in the workplace, which has given rise to a wave of browser-based applications for enterprise collaboration and efficiency.

    Apps—both on devices and in-browser—are effectively what businesses run on. New JavaScript-based Single-Page Applications (SPAs) are replacing legacy cloud stacks, where the software runs primarily in a customer’s browser. However, this approach creates an observability gap for teams under pressure to deliver better software more quickly. Because no application is siloed, the distributed nature of modern applications adds further complexity—and increases the risk for defects—for developers, who must code faster, smarter and better to keep their businesses competitive in an application-centric world.

    The Battle for the Frontend

    IDC predicts that by 2022, more than 60% of global GDP will be digitized, with growth in every industry driven by digitally enhanced offerings, operations and relationships—and applications will be the primary method of delivery. Companies in all industries are battling for users, revenue and reputation only, but only those companies with the best software will remain contenders. To innovate continuously, ensure optimal user experiences, and ultimately, retain their competitive advantage, enterprises must ship code quickly, every day and with fewer errors.

    Application errors and crashes will dramatically spike churn rates, kill conversions and impact brand reputation and customer retention. As tolerance for faulty performance is rapidly diminishing, user experience becomes a business-critical concern that has a substantial impact on a company’s bottom line. While companies with faulty code are dealing with the fallout, those that rely on modern application performance monitoring (APM) are shipping better code more quickly and with less risk.

    These new business drivers, along with an explosion of devices—from smartphones to IoT devices—have led developers to move their code closer to customers to increase code speed and improve scale. But this shift means software is run on customer devices over which developers have no control, a reality that inevitably creates its own set of new problems to monitor. To win this battle for the frontend, development teams that create games, mobile applications, IoT and other high-performance applications need rich insights into application health in order to quickly and continuously fix software errors, all with minimal impact.

    Rethinking APM

    Unfortunately, many legacy application monitoring tools do not solve for the frontend. They are fragmented, or specialized, and missing support for JavaScript, mobile, desktop apps or embedded IoT devices—despite these platforms being a significant part of the application ecosystem. Legacy tools focus more on system behavior, offering performance metrics on availability, throughput and latency, but miss the mark when it comes to actionable insight that helps frontend developers make sense of complexity and quickly get to the root cause of errors.

    Modern applications require a holistic approach to monitoring that provides deep context and understanding about how to address issues when they arise, instead of simply displaying the symptoms of a problem. This approach should involve collecting and analyzing transactional traces, scaling search and storage to handle full traces, and building intelligent algorithms to detect and analyze issues. By moving monitoring from production environments to test environments, development teams can accelerate the software development lifecycle and reduce the impact on customers. APM must integrate with continuous delivery tools to monitor continuous health mechanisms.

    — David Cramer

  • Instana Releases APM Integration with Log Analysis Solutions Humio and Splunk with One-Click Navigation

    Instana is Only APM Solution Able to Jump Directly Into Log Analysis Tool with Contextual Filtering, Creating the Quickest Path to Examining the Exact Logs Needed to Solve Problems

    November 4, 2019, Orlando, FL at Microsoft Ignite conference – Instana, the leading provider of real-time application performance management (APM) solutions for cloud-native microservice applications, today announced new capabilities that make it the only application management solution that can jump right into a log analysis tool with all context of the application and component under investigation.

    “IT teams responsible for deploying and maintaining today’s mission critical applications must react quickly, especially when attacking service issues,” said Pete Abrams, COO and Instana co-founder. “it’s important to have tools that work together to optimize troubleshooting. Instana’s ability to kick off a log analysis solution, with a visualization of the exact logs needed for examination significantly reduces the time it takes to actually solve problems.”

    The first log analysis tools for which Instana is rolling out its one-click integration are Splunk and Humio. The interaction is handled via built-in buttons in the Instana GUI on key component dashboards (such as Servers, Containers and Kubernetes). When the button is clicked, Instana starts up an instance of the log analysis tool, along with a pre-executed analysis filter into the exact logs of the application components and associated infrastructure needing examination. TIming is also synchronized, meaning users won’t have to search for the specific log or timeframe needed to solve problems.

    “Log analysis is an important part of the troubleshooting process, especially when integrated with application performance management solutions like Instana. Humio’s unlimited plan enables users to log everything, giving them autonomy to correlate Instana metrics with relevant logs from their environment,” said Cinthia Portugal, vice president of marketing at Humio. “Humio’s one-click integration and simple query language makes it easy to get started. Our modern architecture runs real-time searches on millions of streaming events per second, and because Instana pulls the exact infrastructure logs associated with problem components, getting the root-cause analysis is fast and easy.”

    Unlike traditional APM tools, Instana’s automated Application Performance Monitoring (APM) solution discovers all application service components and application infrastructure, including Cloud infrastructure such as Azure, orchestration infrastructure like Kubernetes and Docker, application services and DevOps processes. Instana automatically deploys monitoring sensors for each part of the application technology stack and traces all application requests – without requiring any human configuration or even application restarts. The solution detects application and infrastructure changes in real-time, adjusting its own models and visualizing the changes and impacts to performance in seconds.

    The new log analysis integration is being highlighted, along with the recently announced support for .Net Core monitoring, in software demos at Instana’s Microsoft Ignite booth (number 2404). Additional information can be found at https://instana.com/log-analysis

  • Achieving Application Health Through Integrated APM

    Achieving Application Health Through Integrated APM

    Performance and availability mean everything to application users at both the enterprise and consumer level. Think about how much we rely on technology for our everyday communication—when Slack suffered from an outage this summer, it affected thousands of business professionals across the globe who use Slack to communicate, organize tasks and share information. In a sense, we are only as efficient and effective as the technology we use.

    With this in mind, it’s clear achieving and maintaining application health is paramount to overall business success. In part one of this series, we explored how creating and maintaining a healthy app involves not only ensuring it’s performing well via lifecycle APM, but also making sure it’s available when needed—making a log management solution that monitors availability, enables proactivity and reacts quickly to overcome problems is crucial.

    In order to help ensure a healthy application, we must first consider a few defining characteristics.

    Understanding a Healthy Application

    Even for humans, health is more than just being alive—if we’re not performing optimally, we’re not having the best experience possible. Optimum health for an app or website doesn’t mean it’s just up, but that it’s also performing in the way the market demands. For this, organizations need to implement an integrated APM solution to ensure top performance and continuous availability.

    However, achieving application health is easier said than done. Issues that arise for teams working in silos can lead to digital war rooms and blame storming. Organizational structure can get in the way of disparate teams working together to achieve performance and availability. How can organizations overcome this issue?

    Optimizing Performance and Availability

    As we know, each piece of the APM puzzle must be incorporated to achieve full-stack visibility and ultimately top performance: user experience, metrics, traces and logs. Following a few key best practices can help achieve overall application health.

    • Early Adoption: Implement application performance and logging tooling as early in the application development lifecycle as possible. Without integrating these tools from the beginning, organizations run the risk of having errors in availability and performance when they deploy the app. This is because one of the key benefits of a full-stack APM solution is the ability to analyze the performance characteristics of the application code and logic itself. If an application isn’t properly written, it can be so inefficient that no amount of resource in production can assure appropriate service delivery.
      Another critical benefit of early lifecycle APM adoption is the knowledge it delivers about app performance can be reviewed with business stakeholders to ensure complete alignment against shared goals—before going live in full production.
    • Integrated Solution: As mentioned above, the biggest challenge to achieving availability and performance is typically organizational structure. For most organizations of a reasonable size, separate teams monitor performance and availability; in some cases there may even be multiple teams monitoring performance and availability, specifically aligned with the individual technologies (e.g. compute, storage, network, web, cloud, etc.) underpinning each application. To get around the obstacles this creates, it’s key to have an integrated solution where every team is looking at the same set of information—the so called single pane of glass. The ability to share information proactively with management across teams is paramount to overall success and avoiding digital war rooms, or at least minimizing time spent in them.
    • Implement High-Value and Appropriately Priced Solutions: Powerful, easy-to-use solutions that will achieve integrated application performance and availability management for any type of organization operating either on-premises and/or cloud-based applications are key. Traditional APM solutions were either so complex to deploy or so expensive that ubiquitous adoption was not operationally or fiscally possible—depriving much of the organization and its applications the benefits of APM by restricting its application to only the most important subset. As a result, quick-to-value and affordable solutions are needed to enable tech pros who may be working in the organizational performance silo to get running quickly and show success to the organization.

    Conclusion

    Although apps serve their own unique purpose—whether that’s for consumer use enabling us to watch TV and movies on-the-go or at the business level to streamline work and communication—they’re all connected by a universal need: performance and availability. Companies must therefore prioritize application health, and in turn creating and delivering an exceptional user experience.

    — David Wagner

  • Dynatrace Extends APM Reach Into Business Analytics

    Dynatrace Extends APM Reach Into Business Analytics

    A growing number of providers of tools widely employed by DevOps teams that are extending their reach into business analytics has been joined by Dynatrace. The provider of a namesake application performance management (APM) platform unveiled a set of business analytics applications aimed at application owners who need to understand how changes to the  application experience their organization is providing ultimately impacts the business.

    Ajay Gandhi, vice president of product marketing for Dynatrace, said Digital Business Analytics applications from Dynatrace extends the single-agent architecture Dynatrace created for its Software Intelligence Platform to make it easy to employ an APM service delivered via the cloud into the realm of business analytics.

    The difference between the applications being made available by Dynatrace and traditional business intelligence (BI) applications is the agent software Dynatrace developed enables application owners to track all events in real-time, said Gandhi.

    As organizations launch digital transformation initiatives, Gandhi said many discover they lack visibility into the actual end user experience. The suite of Digital Business Analytics applications provides access to a set of applications that enables application owners to track transactions as well as metrics such as conversions and revenue performance by product, customer segment and geography.

    Gandhi noted organizations don’t necessarily need to have embraced best DevOps practices to derive value from the company’s analytics applications. However, those that have embraced DevOps are likely to appreciate the value of these applications because they view themselves as a software company that happens to build a specific product or digital service.

    It’s not clear to what degree application owners may be willing to give up relying on spreadsheets and legacy BI applications they have relied on for decades to manage the business. However, a case can be made for aligning the business to IT more closely by tracking the events in real-time as they occur across application environments and the IT infrastructure on which they depend. In fact, the one thing that has always differentiated Dynatrace is its ability to leverage an artificial intelligence (AI) engine, dubbed Davis, to surface events and deliver actionable intelligence by identifying the root cause of an issue, said Gandhi.

    Other DevOps vendors that are venturing into the realm of business analytics include CloudBees and GitLab. As these vendors increasingly realize that business processes now frequently revolve around DevOps processes, they all see an opportunity to extend their franchises by providing analytics applications that are aimed well beyond their existing base of IT users. Less clear right now is which set of tools within a DevOps toolchain will serve as the primary source of data for those analytics applications. However, Gandhi did note that unlike continuous integration/continuous deployment (CI/CD) platforms, APM services can provide real-time visibility into the actual digital experiences that change as various events impact production applications.

    Regardless of the ultimate source of data, however, the one thing that is certain is the level of insight business users can have into application development and deployment processes trends is, for better or worse, about to increase substantially.

    — Mike Vizard

  • Understanding and Implementing the APM Puzzle – Performance and Availability

    Understanding and Implementing the APM Puzzle – Performance and Availability

    While every organization is unique, there’s one universal goal everyone can get behind—creating and delivering an exceptional end-user experience. For both classic IT organizations and companies built in the age of cloud, delivering continuous availability and appropriate application performance is key.

    Although digital transformation has become a buzzword some would say is meaningless, the idea of digitizing a traditional company and building new businesses whose foundations are established upon the latest web technologies is critical in today’s IT and non-IT organizations. Think about the prevalence of cloud migration efforts during the past several years. Traditional organizations typically prioritize on-premises applications with increasing quantities of hybrid applications. But, because they aren’t yet fully transitioned to cloud—whether IaaS, PaaS or SaaS—their IT organizations need to live in both universes.

    Monitoring these environments at both the infrastructure and application tiers and across each stage in this transition requires support of a full stack of tools to cover web uptime, user experience, transaction monitoring, application tracing, log analysis and appropriate infrastructure monitoring. That said, modern organizations that have fully embraced cloud and leverage modern microservice architectures have similar challenges, but the exact same level of monitoring visibility requirement.

    Whether public, private, hybrid or multi-cloud, nearly every organization has moved to the cloud in some capacity. Regardless of what the company produces or the services IT provides, it must maintain uptime, availability and seamless performance to create and continuously deliver a user experience rivaling competitors and exceeding user expectations.

    Because let’s face it, even though every business is different and virtually every market is saturated, consumers and B2B companies have more options than ever. Without delivering a solid end-user experience, there’s an ever-increasing plethora of alternative vendors and providers from which customers can choose.

    Public Cloud Platforms for Classic IT Organizations and the Cloud Generation

    Let’s explore a few differences between how traditional IT organizations and cloud-era companies interact with the cloud. At a high level, Azure is a natural choice for IT companies with significant investments in apps traditionally acquired (not built) by the company and managed by their IT team, that are now tasked with digitizing their business by leveraging public cloud platforms. These types of companies often have Microsoft Office, Microsoft Exchange and other Microsoft products already implemented on-premises, and Infrastructure-as-a-Service (IaaS) or Platform-as-a-Service (PaaS) solutions as well. If they’re now adding a public cloud platform to their mix, staff expertise and solution familiarity make adopting Microsoft Azure a natural next step. Microsoft has long recognized the need to make these types of transitions and mixtures easier, demonstrated by its Office365 suite (as an example).

    On the flip side, companies that were born fully in the cloud, (i.e., relatively newer organizations without a legacy anything on-premises, with respect to IT infrastructure) will likely have a higher propensity to choose a provider such as AWS or Google Cloud even though Microsoft, Oracle and others are certainly competing there as well. So, it’s not a matter of one being intrinsically better or more fully-featured, but about which is the best fit for the organization and its priorities.

    At the end of the day, however—regardless of organization type or public cloud platform implemented—every company needs monitoring tools and solutions that keep applications up, running and performing seamlessly. With an integrated set of simple, easy-to-use, quick to value and powerful solutions, organizations can achieve truly proactive lifecycle assurance of application availability and performance.

    Uptime and Seamless Performance: Putting Together the APM Puzzle Using All the Pieces

    Each and every piece of the APM puzzle must be incorporated to achieve full-stack visibility and ultimately top performance: user experience, metrics, traces and logs. Having a log management solution monitoring availability, enables proactivity and enables IT to react quickly to help identify and remediate problems is paramount—it’s table stakes that everything be available. After all, if a website or page isn’t accessible, business transactions cannot be processed and that business is effectively down.

    Creating and maintaining a healthy application involves not only ensuring it’s functionally available to users; it must also be performing appropriately—continuously delivering both a reasonably responsive user experience and supporting the required volume of transactional work. This means a complete view into all aspects of performance—from the website and webpages all the way into and across all relevant application components and their associated infrastructure—be proactively visible and monitored.

    There is thus a natural and increasingly urgent workflow affinity between logs and APM—across the development lifecycle. When errors occur, it’s critical to understand, precisely and immediately, where are the impacts on service performance, or if there even are any impacts. Conversely, when application performance issues arise, it’s critical to immediately know exactly which errors—across the complex and constantly changing infrastructure and application stack—are directly related to the problem at hand.

    Without this bi-directional workflow integration, not only can application performance and availability not be managed proactively, but any issues with application service availability and performance will take far longer to diagnose and repair—seriously impacting the business bottom line.

    Just because everything is available doesn’t necessarily mean it’s appropriately performing for the end user trying to use the application.

    With all this in mind, it’s paramount to implement powerful, easy-to-use solutions that’ll achieve integrated application performance and availability management for any type of organization operating on-premises and/or cloud-based applications across all cloud environments.

    Conclusion

    While traditional IT organizations and cloud-only companies are different in many ways, providing exceptional end-user experience and achieving seamless availability and performance (no matter what) is a common and increasingly critical business goal. To achieve business success, organizations should choose a complete suite of simple, powerful and integrated solutions to optimize and manage their cloud-hosted applications and platforms. Anything less leaves the APM puzzle unsolved.

    — David Wagner

  • Splunk Adds Omnition to DevOps Portfolio

    Splunk Adds Omnition to DevOps Portfolio

    Splunk followed up on its pending acquisition of SignalFx, a provider of an application performance management (APM), by also acquiring Omnition, a startup building a distributed tracing tool accessed as a software-as-a-service (SaaS) application.

    Rick Fitz, senior vice president and general manager for the IT Markets Group at Splunk, said Omnition consists of a small team of developers who have been especially active in maintaining OpenCensus, an open source set of libraries for various languages that make it easier to collect application metrics and distributed traces and then transfer the data to a backend of choice in real-time; and OpenTelemetry, a project led by the Cloud Native Computing Foundation (CNCF) that merges OpenCensus and OpenTracing to standardize the instrumentation and telemetry data collection.

    Distributed tracing is emerging as a requirement for APM platforms because the IT environment is becoming both more complex and ephemeral. Serverless computing frameworks are especially challenging because the only way to know an event occurred involving, for example, cloud bursting is to employ distributed tracing. The challenge is, distributed tracing requires organizations to capture, store and analyze massive amounts of data to pinpoint the root cause of any application performance issue, noted Fitz. As such, distributed tracing tends to be employed only by the most elite DevOps practitioners, as implementing distributed tracing is nontrivial, he added. On the upside, distributed tracing reduces the need to deploy a large amount of agent software across the enterprise.

    Fitz said Omnition will provide a set of capabilities that over time will be added to the SignalFx platform. Long-term, Splunk hopes to reduce the costs associated with distributed tracing to the point where that capability becomes democratized, he added, noting that as organizations embrace microservices to build cloud-native applications that will run in hybrid computing environments, increased reliance on distributed tracing is now all but inevitable.

    In general, Fitz said Splunk is working toward combining the best elements of DevOps with traditional IT service management (ITSM). Most organizations will be employing both approaches for years to come simply because some applications don’t need to be updated regularly, so it doesn’t make any sense to apply DevOps processes to those applications, he noted. At the same time, as DevOps processes are embraced in enterprise IT environments, appreciation is growing for IT operational challenges that stem from multiple applications running at scale which need to share a limited amount of IT infrastructure resources.

    With the acquisitions of SignalFx and now Omnition, Splunk is betting that it will be able to combine IT operational analytics based on machine data generated by logs with an APM platform to correlate application performance against infrastructure events. It’s not clear how long it will be before Splunk can fully realize that vision. In the meantime, the rate at which application and infrastructure management platforms will converge will only accelerate as developers are held more accountable for applications long after they are deployed in a production environment.

    — Mike Vizard

  • Splunk to Acquire SignalFx to Gain APM Platform

    Splunk to Acquire SignalFx to Gain APM Platform

    Splunk today announced it will acquire SignalFx as part of an effort to extend its reach beyond IT operational analytics and into the realm of application performance management (APM).

    Valued at approximately $1.05 billion, the deal to acquire an APM platform delivered as a software-as-a-service (SaaS) offering will expand and accelerate Splunk’s overall transition to a recurring revenue model based on cloud services, said Splunk CEO Doug Merritt during a call with industry analysts.

    Splunk is currently on pace to generate over $300 million per year from the cloud version of its IT operational analytics platform. Splunk revenues for the second quarter were $517 million, up 33% year over year, with software generating $350 million of that total. However, as Splunk transitions to more of a subscription-based pricing model, the company has some financial management challenges. GAAP operating loss for the company’s second quarter was $87 million. However, non-GAAP operating income was $47 million, even though operating cash flow was negative $129 million. That cash-flow issue is a direct result of the transition away from a perpetual software license model based on the volume of data consumed by the Splunk platform. Overall, Merritt said Splunk is firing on all cylinders and is still on track to have 20,000 customers by the end of its fiscal year.

    Merritt said once the deal is approved, sometime in the second half of fiscal 2020, SignalFx will provide a complementary set of services for monitoring legacy monolithic applications as well as cloud-native applications based on containers, Kubernetes and serverless computing frameworks. There are very few customers at this point who have not adopted cloud-native technologies, but for now they remain a small percentage of the overall number of application workloads being managed, he added.

    The goal is to provide a suite of services that will enable IT teams to investigate, analyze, monitor and act on data in real-time, said Merritt.

    Splunk is already widely employed by internal IT operations teams, many of whom already rely on a complementary APM platform. Splunk clearly expects organizations will prefer to acquire an IT operational analytics platform and an APM service from the same vendor. However, competition between APM platform providers is already fierce. That said, Splunk appears to be counting on increased demand for real-time application monitoring to differentiate SignalFx from rivals.

    It’s not clear to what degree Splunk might converge the various services now being provided separately by both companies. It’s also not clear how or whether this deal will accelerate acquisitions of the remaining independent APM platform providers.

    In the meantime, Merritt said SignalFx and Splunk’s earlier acquisition of VictorOps, provider of incident management platform based on best DevOps practices, will expand the amount of visibility Splunk has with application development teams.

    Ultimately, acquisitions of smaller, DevOps-oriented companies by vendors that today are still more closely identified with legacy ITIL-based IT processes, such as Splunk, are only like to increase. What will be interesting to watch is whether the tools and services being acquired by those companies wind up being the DevOps tail that wags the proverbial ITIL dog.

    — Mike Vizard

  • Instana Pipeline Feedback™ Integrates Automated APM with CI/CD Application Releases

    DevOps and Developers Get Immediate Feedback on New Release Performance

    Solingen, Germany and San Mateo, CA – August 15, 2019 – Instana, the leading provider of automated Application Performance Management (APM) solutions for dynamic microservice applications, announced application release tracking and feedback capabilities that make it the first APM solution to include automatic release tracking.

    The new capabilities are part of Instana Pipeline Feedback™, an application delivery pipeline performance tracking and analytics solution, directly integrated with Instana’s automated APM solution. In addition to the new Pipeline Feedback system, today’s announcement covers specific new functionality:

    • Automatic tracking and performance feedback of Application Releases
    • Integration with Jenkins for automatic identification of new releases

    “As application teams strive for higher velocity in their deployment processes, they require immediate feedback whenever changes occur,” said Pete Abrams, Instana co-founder and COO. “Instana Pipeline Feedback tracks and isolates application release performance, notifying developers and DevOps of issues within seconds.”

    Pipeline Feedback is included with Instana’s automated APM solution that maps, visualizes and manages microservice applications, delivering the actionable data that application teams need to provide high quality applications. The new CI/CD process management capabilities are included at no additional cost.

    Instana’s automated Application Performance Monitoring (APM) solution discovers application service components and application infrastructure, including Jenkins, Kubernetes, and Docker. The tool automatically deploys monitoring sensors for each part of the application technology stack and traces all application requests – without requiring any human configuration or even application restarts. The solution detects changes in the application environment in real-time, adjusting its own models and visualizing the changes and impacts to performance in seconds.

    For deployment teams, the full system is a powerful tool to understand the quality of new software deployments. Whenever an official release is deployed, Jenkins notifies Instana Pipeline Feedback, which will isolate each new service or piece of code, provide Instana performance reports, and deliver a comparison of service and application performance and health, compared to the previous release.

    Like other Instana capabilities, Pipeline Feedback is automatic and provides actionable data in seconds – providing DevOps and Developers the immediate feedback they need to know that new releases are performing as expected. The new functionality is being showcased at the JenkinsWorld / DevOps World Conference. Learn more at instana.com.

    ABOUT INSTANA

    As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit https://instana.com to learn more.

     

  • Application Performance Management for DataOps

    Application Performance Management for DataOps

    Over the years, application performance management (APM) tools have been utilized typically to measure and monitor the performance of software such as web applications. They do so by tracking such things as request response times and error messages, thus providing relevant data to be analyzed by the DevOps team, which then can make the appropriate changes to optimize their software. But what about the effectiveness of the data workflows that contribute to the functionality in many of the applications out there today? How can performance monitoring toolsets be extended to provide these workflows with opportunities for continuous improvement?

    Data Pipelines and Performance Challenges

    Below, I will discuss the use of APM tools in the context of DataOps. I will delve into the metrics and insights an APM tool must provide to engineers so they can support data pipelines efficiently, showing how the effective use of data-centric APM tools can provide extensive value to a DataOps team and their organization.

    Data Pipelines and DataOps Engineers

    A successful DataOps implementation requires creating automated data workflows that analyze large amounts of data quickly, while providing high-quality analysis as an end product. It’s easy to see how the job of a DataOps engineer can be a complicated one. It is their job to build and manage these pipelines to help the business achieve certain goals that rely upon the resulting data analysis. In other words, DataOps engineers are responsible for making sure high-quality data is made available as quickly as possible.

    Consider a common scenario with which we are all familiar: online shopping. With each purchase you make from a particular e-commerce website, the system typically provides recommended products for you to buy. This is often the result of a data workflow that produces data analysis for a recommendation system, which is then leveraged by the application to provide the end user with these recommendations. This pipeline is likely to be made up of several processes.

    These processes include those for loading large sets of raw data into a distributed file system for storing data across a cluster. This would be followed by several processes that need to run on this cluster to format and analyze this data to produce the quality data for the recommendation engine being leveraged by the web application. A lot is being done with large sets of data. And, as is usually the case with so much happening, there is also much that can go wrong.

    The Challenges in Managing Data Pipelines

    There are several major performance challenges that come with the management of the types of pipelines mentioned above. Below are a few of the common threats to data pipeline performance:

    • Scalability: One of the more obvious challenges when dealing with the construction of a data pipeline is scalability—in other words, the effectiveness and efficiency of the pipeline when it needs to deal with a much larger dataset. It’s likely when the jobs were written to deal with such data transformation and analysis that they were tested against smaller sets of data in development environments. What happens when this is no longer the case? Maybe now instead of processing a relatively small amount of data in a test environment, it needs to process much more as it runs in production. Will the processes still perform adequately to deliver quality analysis in a time-efficient manner?
    • Application Failures: Data pipelines rely upon parallel processing. But what happens when you experience the failure of one or more of these parallel-running processes that are executing to clean and analyze the data? This can throw a wrench in your ability to relay quality, complete analysis to the business unit. Being able to resolve failures quickly and get the pipeline back up and running plays a huge role in the success of a particular data workflow.
    • Fragmentation: Most data pipelines consist of a variety of tools that handle different tasks (such as ingestion, storage, processing, analysis, etc.). With so many different tools in the mix, pipelines can become fragmented. Tool incompatibility issues can arise and data formats can vary. It’s important to be able to unify data within the pipeline, even as it moves between disparate tools.

    APM Tools to the Rescue

    Manual vs. Automated

    Enabling DataOps within your organization requires continuous feedback and continuous improvement to your data workflows. A good place to implement these strategies is with the use of APM tools to help manage the performance of these pipelines.

    Consider the processes for improvement that occur without the use of APM tools. It would be on the IT and DataOps folks to locate performance-related problems manually within these pipelines and begin digging through log files to find the cause. A significant amount of time likely would be added to resolving these failures due to the lack of a centralized location to view this information, the lack of automated alerts to notify the team to these performance snafus and the lack of assistance in deriving insights from collected performance data. APM tools can provide an intuitive UI for debugging such issues and can provide the type of insights needed to detect and resolve these problems quickly.

    That being said, monitoring for big data processes is quite different from other types of application monitoring. Certain APM software will be better-equipped for dealing with data pipeline-related challenges than others. Much of this depends on the ability of the APM tool to integrate with the platforms and frameworks used within the data workflow, as well as the metrics the performance management software collects and utilizes to help the DataOps engineers refine their processes.

    What Types of Metrics Can Help Determine the Causes of Issues Within a Data Pipeline?

    Earlier, I discussed data pipeline performance challenges stemming from scalability issues and process failures. Let’s discuss what types of metrics should be collected to resolve issues like these in a data workflow.

    Before delving into the metrics, it’s important to note that having full-stack data is essential. By full-stack data, I mean not just infrastructure metrics (which are the most obvious type of metric folks tend to think about when managing data pipelines), but also other data points such as user data, application data and resource data. All this data needs to be unified and correlated across all systems in the pipeline to manage it effectively.

    Consider scalability, for instance. The APM solution you are using must be able to track several key metrics to help identify scaling-related bottlenecks within the data pipeline. These include:

    • The Amount of Data Being Processed: How can we identify a scalability issue without knowing how much data is being processed by a particular run? The answer: We can’t. Being able to quantify the amount of data being handled by each iteration of the data pipeline is the first step in detecting if the workload is the problem.
    • The Length of Each Run for Each Individual Process: The APM software being used by an organization must be able to integrate with the structure of the data pipeline to report the length of each run for each individual job running within the workflow. If a particular job was completed in one minute when processing X amount of data but took 20 minutes when processing 10X, then we have diagnosed a potential scalability issue and narrowed it down to the root cause in a time-efficient manner. Maybe the fix is as simple as tuning some SQL, and the scalability issue can be rectified in relatively little time.
    • Resource Utilization: Certain processes may require more resources than others to complete in a timely fashion. The ability of the APM tool to track the amount of resources being utilized by each process is key in determining if simply reallocating certain resources can fix problems with performance.

    Example issues:

     

     

    Unravel is an example of an APM tool that can track all the above while organizing the data within a user interface that provides DataOps engineers with just about all they need to diagnose data workflow performance issues from one screen. In addition, Unravel employs AI solutions to help take the metrics they collect and provide the engineers with specific insights as to how they can optimize their pipeline. This helps decrease the amount of time DataOps personnel spend on managing existing processes, allowing them to spend more time creating new processes to improve the quality of analysis being provided to the business.

    Conclusion

    APM tools are no longer solely for ensuring application availability. APM tools for big data, such as that from Unravel, have allowed for performance monitoring and analysis within data workflows, helping to ensure the availability of crucial, high-quality data analysis. Any organization looking to improve the quality of their data analysis processes should be looking to extend their APM toolset to include those for big data.

    This sponsored article was written on behalf of Unravel.

    — Scott Fitzpatrick

  • Survey Finds Reliance on APM Mainly Reactive

    Survey Finds Reliance on APM Mainly Reactive

    A survey of 358 strategic IT decision-makers in enterprise organizations across North America and Europe conducted by the market research firm GigaOm on behalf of SolarWinds, a provider of IT management tools, finds that roughly half of the respondents consider themselves proactive in their use of application performance management (APM) tools.

    That definition is based on those respondents either strongly agreeing or agreeing that their organization was employing APM tools within the context of all the following criteria:

    1. Engaged in a digital business transformation initiative.
    2. Makes use of a hybrid or multicloud computing strategy.
    3. Has a public-cloud first IT strategy.
    4. Makes use of best practices for IT operations.
    5. Line of business teams are driving technology change.

    Denny LeCompte, general manager for application management at SolarWinds, said that while significant progress is clearly being made it terms of APM adoption, it’s clear that half or more of the potential market is still wrestling with a variety of challenges that often stem from the cost and complexity associated with acquiring and employing an APM platform at scale.

    LeCompte said most of the use of APM is still reactive in that most organizations don’t actively engage an APM tool until there is an actual issue. The survey finds the primary driver for adopting APM is to fix problems (61%), followed by keeping on top of complex application environments (57%), delivering the best possible customer experience (52%) and managing costs and delivering efficient operations (34%). However, it’s worth noting that among the half of the respondents claiming to be proactive about employing APM, customer experience was the primary driver for adopting APM (61%).

    Even so, LeCompte noted reliance on APM platforms is often uneven at best. When it comes to adoption, he noted, most organizations are on a journey. Initially, the focus is on IT troubleshooting. Over time, however, customer experience with digital business processes becomes a much bigger factor. Different teams with the same organization are at different stages of that journey, he said.

    SolarWinds views the rising tide of APM awareness as the justification for jumping into the category, LeCompte said, claiming the company is committed to providing APM capabilities an IT organization is likely to require, at a fraction of the cost of rival platforms. That’s critical because many organizations have limited the number of applications they track using APM tools because of cost, he said.

    As IT environments become more complex in the age of microservices built using highly ephemeral containers, LeCompte said it’s only a matter of time before more IT organizations will need to embrace APM platforms proactively to optimize overall application performance. Individual IT administrators won’t be able to keep track of all the dependencies that might be impacting a “cloud native” application without relying on an APM platform, he noted.

    The issue now is finding a way to provide access to an APM platform that everyone on the IT team can regularly employ without breaking the IT budget.

    — Mike Vizard