Author: Jonah Kowall

  • Gartner Evaluates APM Tools for DevOps

    Gartner Evaluates APM Tools for DevOps

    In 2009, Gartner analyst Mark Nicolett created the Critical Capabilities report. Over time, this document became paired to individual Magic Quadrant reports, and in 2014, Gartner mandated that Critical Capabilities be paired with each Magic Quadrant. Instead of graphically plotting vendors on a x- and y-axis, this research takes a deeper look at a product and its fit for specific use cases.

    So how does this pertain to DevOps?

    In its latest “Critical Capabilities of Application Performance Monitoring” research, Gartner notes, “Because of its wide lifecycle purview, the DevOps release team often leverages many of the capabilities found in APM suites.” Since the foundation of DevOps is to foster collaboration among the two, typically disparate teams, modern application performance monitoring (APM) solutions are geared perfectly to help encourage this feedback loop. “When examining the capabilities valued by this buyer, APM tooling is critical when attempting to quickly assess the quality (and effectiveness) of a recent release. Detecting and diagnosing issues must be done quickly, which often requires in-depth data sourced from multiple, integrated tools,” according to Gartner.

    “Much of the activity of APM technology is focused on detecting and providing insight into root causes. Identification of the impact on system and application resources is based on existing (or proposed) transaction-level trends,” the report continues. “This capability provides IT operations and DevOps release teams with the ability to tell the (line of business) and application owners the potential impact for any planned changes or continuation of resource consumption trends. Although capacity-planning tools have often filled this role, APM suites increasingly offer predictive analytics capabilities for shorter-term horizons”

    Of course, we can’t ignore the key to DevOps success: culture. However, no tool—no matter how good—can foster the right type of culture. John Rakowski, my current colleague and ex-Forrester analyst, wrote a compelling ebook on the intangibles that successful DevOps teams need.

    The Critical Capabilities research report provides a comprehensive and technical product analysis ranking APM suites from 15 vendors on critical APM capabilities across five different usage scenarios, including DevOps. The capabilities assessed for each use case included business analysis, service monitoring, anomaly detection, distributed profiling, application debugging and workload planning.

    The vendors evaluated were AppDynamics, AppNeta, BMC Software, CA Technologies, Dell, Dynatrace, Hewlett Packard Enterprise, IBM, JenniferSoft, ManageEngine, Microsoft, Nastel Technologies, New Relic, Riverbed Technology and Tinygun.

    The report is free of charge and can be downloaded here.

  • The Role of APM in Continuous Integration and Continuous Release

    Today’s software-defined and -driven business requires fast changes in business models—a phenomenon permeating successful businesses. What this means is almost every company is learning how to make small, rapid changes and adjustments within their business, especially within software systems. As a result, IT is feeling immense pressure to evolve, and the net result is a major uptick in private cloud environments (especially Apprenda, Pivotal Cloud Foundry and Red Hat OpenShift) and public clouds (especially Amazon and Microsoft) within our install base of large enterprises. These new platforms enable faster development, testing and releasing of software.

    Enterprise customers are getting much smarter by using automation to drive the software life cycle, including continuous integration, and even experimenting with and using continuous release processes. This is no longer a startup scenario; it’s a strategy for our large, enterprise customers. These companies are trying to build agile development and operations teams by changing people, process and technology to enable a DevOps feedback loop:
    image02

    Wikipedia describes the continuous delivery loop as follows:

    DeliveryLoop

    Within this loop, application performance management (APM) can be part of build, unit, regression and load tests to drive the automated acceptance testing before new code is pushed to production. In fact, at the recent AppSphere 2015 user conference we had several great talks on this subject.

    The first talk was presented by our customer The Container Store‘s  August Azzarello, who discussed, “How The Container Store uses AppDynamics in the Development Lifecycle.” The company began with APM in production, in an effort to improve software quality before production. It began the expansion by installing APM within test, integrating it into functional and performance test suites. It also enabled alerting from dev/test so that developers and testers both understood when there were performance deviations and issues before production.

    Key features the company leveraged in pre-production included dashboards such as this one:image03

    Comparative analysis views such as this one:

    image04

    And being able to understand between releases if they were improving or degrading:

    image00

    The Container Store is using open source tools including Selenium for functional testing and Locust.io for performance testing.

    During his discussion, August also explained some of the major benefits the company saw from implementing APM in pre-production. Here are some best practices outlined by August in his discussion:

    • Monitor everything
    • Test continuously
    • Performance test early in development life cycle
    • Empower development & QA team members

    Some benefits The Container Store has seen, according to August, are:

    • Set performance expectations before production deployments – about 40 percent improvement since it started using AppDynamics in test
    • Fine-tune alert and business transaction policies prior to production deployments
    • Identify testing requirements and testing gaps
    • Decrease performance test result reporting to 20 minutes from five hours

    One of my favorite talks (which is why I selected it for the deep dive track at AppSphere) was the talk given by one of our senior sales engineers Steve Sturtevant, who came to AppDynamics from PayPal two years ago. In his talk, titled, “DevOps and Agile: AppDynamics in Continuous Integration Environments,” he explained the integration work he did. He also delved into the questions, Can you scale and auto implement monitoring in dynamic environments? Do you quantify the impact of change? Steve also offered a demo with integration between Puppet, Jenkins, ServiceNow and AppDynamics in his examples, with automation throughout the life cycle.

    One of our partners Electric Cloud specializes in continuous delivery and embraces an open ecosystem of technology providers and open source software. It has put together some great resources for people looking to explore the rest of the feedback loop:

  • Performance Monitoring, another view

    Performance Monitoring, another view

    This is my first post to devopsy.kinsta.cloud, for those who haven’t read my other writings, I currently work as the VP of Market Development and Insights at AppDynamics. I joined in February 2015 after being at Gartner for 4 years as a Research VP covering all things monitoring. During my time as a Gartner analyst I would help buyers make decisions about purchasing monitoring software. When I read articles on the internet which blatantly disregard best practices I either smell that something is fishy (sponsorship) or they just didn’t follow a process. Conversely some people do follow a proper process. I specifically wrote this up due to reading a really fishy article.

    Step 1 – Admit it

    The first step to determining you have a problem is admitting it. Most monitoring tool buyers realize they have too many tools, and none of them help isolate root cause. This is a result of most buying happening at the infrastructure layer, when the application layer is either lightly exercised synthetically or disregarded. People soon realize you must find an application layer monitoring tool, or write your own instrumentation. The vendors who sell logging software would love for you to generate a lot of rich logs, so they can help you solve problems, but realistically this doesn’t work in most enterprises as they are using a combination of custom and packaged software, and forcing developers to write consistent logging is a frustrating and futile task.

    Step 2 – Document it

    The next phase is to document what you are using today, most organizations have over 10 monitoring tools, and they typically want to buy another one. As they consider APM tools they should document what they have today, and determine a plan to consolidate and eliminate redundant tools. Having tools within silos will happen until you can fix the organizational structure.

    Step 3 – Requirements

    I can’t tell you how many times vendors set requirements for buyers. You can tell when vendors do this versus actually understanding the requirements the user has, and if the solution will scale and be operable by the team. Purchasing a tool which requires full time consultants on staff to keep it running is no longer feasible today, sorry to the old big ITOM vendors (formerly the big-4) but people don’t want that anymore.

    Step 4 – Test it!

    If you are evaluating software you MUST test it, and make sure it works in your environment. Better yet load test the software to determine if the vendors claims of overhead and visibility are in fact true. I see far too many people buy software they do not test.

    Getting back to the article which prompted this post:

    https://devopsy.kinsta.cloud/features/devops-style-performance-monitoring-for-net/

    The author is located in Austria and I believe there is a connection to Dynatrace. Dynatraces’ R&D and technical leadership are in Austria, but this is just conjecture. The article is written with a bias, which is obvious to any reader who has used these tools. The evaluator did not test these products, or if he did he didn’t have a testing methodology.

    Finally the author’s employer is new and all employees have come from Automic Software in the past year. I don’t need to explain to those who work with Automic the state of things there. http://www.glassdoor.com/Reviews/Automic-Software-Reviews-E269431.htm#

    I think I’ve spent enough time on the background, here are my specific issues with the content.

    The screenshots are taken from various websites, and are not current or accurate. If someone writes an evaluation of technology, at least test it, and use your screenshots from the testing!

    This post was specifically to discuss DevOps, but yet the author doesn’t discuss the need to monitor microservices or asynchronous application patterns, which are clearly the current architecture of choice. This is especially the case in DevOps or continuous release environments. In fact Microsoft themselves announced new products last week at Ignite including nano server, and Azure Fabric Service for building these patterns. The difficulty with microservices is that each request externally spawns a large number of internal requests, these often start and stop out of order creating issues when monitoring these. This causes major issues for APM products, and very few handle this effectively today.

    The article calls out user interface and deployment models inconsistently, and fails to mention Dynatrace is an on-premises software product. Dynatrace is also a thick Java client, built on Eclipse. This is the definition of a heavy client, which the author calls out for other products in this article.

    Continuing the incorrect facts in this article the author calls out Dell (formerly Quest) Foglight. This is not Dell’s current product for modern architectures. Dell has moved to a SaaS product called Foglight APM SaaS, which is not what was evaluated. The evaluator should look at the current product (noticing a trend here).

    Finally the AppDynamics review is inaccurate:

    Editors note: the following is this authors opinions, not the opinions of devopsy.kinsta.cloud.  As the author states, he is employed by AppDynamics.

    “code level visibility may be limited here to reduce overhead.”

    The default in order to scale AppDynamics to monitor thousands of systems with one single on premises controller (something no other APM vendor delivers) is done based on advanced smart instrumentation, but every transaction is monitored and measured. All products do  limit the amount of data they request in some way. AppDynamics has several modes including one which captures full call traces for every execution.

    “ lack of IIS request visibility”

    AppDynamics provides excellent visibility to IIS requests, I’m not sure what the author means here. AppDynamics also supports implementation on Azure using nuget.

    “features walk clock time with no CPU breakdown”

    Once again, I’m not sure what the author is referring to here. The product provides several ways of viewing CPU usage.

    “Its flash-based web UI and rather cumbersome agent configuration”

    The agent installation is no different than other products, showing the author has not installed the product. Finally the product is HTML5, and has been for a while, there are still some flash views in configuration screens, but those are being removed with each release. I’d much rather have a web based UI with a little remaining flash than a fat client UI requiring a large download.

    AppDynamics has a single UI which goes far deeper into database performance than the other products in this review. AppDynamics also provides far more capabilities than what was highlighted in this article including synthetic. AppDynamics provides deployment of the same software via SaaS or on premises installation.

    Finally this review did not look at analytics, which is clearly an area of increasing demand within APM. That being said this review is far from factual or useful.

    Hopefully this sets some of this straight, please leave comments here, or @jkowall on twitter!

    Thanks.