Tag: monitoring

  • Culture Change: Metrics, Tools Make a Difference

    Culture Change: Metrics, Tools Make a Difference

    Change is in the wind.

    This time of the year, as winter weather gives way to springtime renewal, this concept isn’t a cliché but rather a concrete, visceral experience. You go outside, smell the earth defrosting and know for certain that change is happening all around you. You can feel it.

    Such is the case on the current DevOps landscape. Seemingly every conversation, with everyone from end users to industry analysts, reaffirms that people are making significant progress toward greater adoption. For an increasing number of organizations, DevOps is maturing from a set of budding goals into a tangible reality.

    At the center of this massive shift within the software factory is the issue of cultural change; as we know in all things DevOps, culture trumps all else. So how are organizations affecting, empowering and measuring all this seismic transition, in particular the human elements?

    During the recent DevOps Virtual Summit, leading practitioners representing industries ranging from banking to travel and technology shared their views on the progress of cultural reinvention.

    From management tips, to tools and even metrics useful for driving transformation, here are some of their observations:

    1. What’s the most critical cultural aspect of planning?
      Kazi Whitfield, director, Quality Engineering, MUFG Union Bank, N.A.: Scalability on this is the key; if you’re doing this at a new organization, you need to understand how to scale it, and starting at some smaller level helps you get there. It also helps each individual and those components that are changing to truly understand what’s changing, and why it’s changing; I think that’s a huge part of the conversation that has to happen.
    1. What’s the biggest obstacle to cultural change?
      Eric Warheit, senior manager, Environment Management, Southwest Airlines: The challenges that we’re working through are organizational in nature. It goes back to the question of localization, and people maintaining a mindset that they’ve previously had, and what we do to create the conversation around what’s possible and the key impacts that we want to have. Ultimately, it’s about being productive with quality, and creating those common values.
    1. How can tools enable cultural change?
      Eugene “Bear” Lehenbauer, group manager, Information Technology, Citrix: If I have to pick one that links them all, a ring to rule them all, that’s when you have an orchestration platform that can take what individual teams are already using and plug right in, or you can take what needs to be automated and automate it; that orchestration harness can now become that common thing that links them.
    1. How can you ensure cross-culture collaboration?
      Warheit: Creating that concrete example, that proof case tying analytics to it, so that we have real measurable ways to make good business decisions, and being able to foster that perspective so that we can look across the organization, as opposed to in a localized way, is really critical to be able to foment that collaboration that’s needed.
    1. How do you measure cultural change?
      Daniele Romano, product owner, Continuous Delivery Pipeline, ING Bank: The most important thing as an organization is to able to apply an analytical approach, meaning that for each team or department you should look at the current situation and have in mind the target situation, and then have different means to guide everyone toward that target.
    1. What’s a key cultural change metric?
      Lehenbauer: We’ve a developed an agility rating scale that allows people to earn scores for something as basic as change management. Based on how those key metrics behave, and there are several of them, but they are fact-based and they’re user impact driven for the most part, they reward the right behaviors. Those types of things can draw out some of DevOps principles results, but also have hard data behind them.
    1. How do you recognize advocates of cultural change?
      Whitfield: Being able to phase reward and recognition into the process is key. Organizations are doing a lot of great things and making millions of dollars, and as the individuals that are coding or testing are driving it, you have to understand how you get to those people’s hearts, to make them passionate about what they’re doing. That’s something that becomes key and it’s something that’s different per organization.
    1. What’s the best way to gauge successful cultural transformation?
      Romano: There is a really only one critical human metric, which is happiness. So, if engineers are happy and they can deliver fast, at a high quality and with intense focus—that’s a very unique but important metric.

    So there you have it, some of today’s most critical elements in driving the cultural transformation necessary to succeed in the DevOps world.

    If you missed the DevOps Virtual Summit you can still view recordings of all the sessions including this customer panel, and stay tuned to this space for more excerpts and highlights.

  • Flash Mob Inflection: Rugged DevOps Revolution

    Flash Mob Inflection: Rugged DevOps Revolution

    Truthfully, I was never a huge fan of the HBO series “The Sopranos.” It’s not that it wasn’t entertaining; I just didn’t agree with the “best ever” label that so many espoused during the show’s halcyon days.

    This had something to do with living in Hoboken, N.J., at the time and feeling the show was a heavily glamorized caricature of the real-life Mafioso I still perceived to be in my midst. More specifically, it’s hard for anything mob-related to stand up next to “Goodfellas” and “The Godfather”—among my favorite films, ever.

    Either way, I currently find myself identifying with the series’ defining sentiment (copped directly from The Godfather, but hammered home as The Sopranos main theme): “Just when I thought I was out … they pull me back in.”

    It wasn’t that I assumed that I’d never delve back into the domain of IT security after transitioning into the DevOps market last year; leveraging DevOps methodologies to improve security, policy compliance and related workflows stands as a core element of the overall movement.

    But, in recent months and weeks it has become increasingly obvious that the security facet of the larger DevOps revolution appears poised for explosion as the buzz around emerging best practices, business models and solutions providers grows from a low hum into a steady roar.

    After spending a few days talking DevOps security with Derek Weeks of Sonatype at CA World ’15 (we were booth neighbors and dinner mates) and giving more consideration to his company’s angle on securing the systems development life cycle (SDLC) supply chain, the internal gears really got spinning.

    Regardless of whether you buy into Sonatype’s security-heavy approach, the concept of aggressively and proactively addressing security across the DevOps spectrum makes infinite sense. The ultimate downfall of applications security has always been the challenge of securing all the underlying layers of code (and today, microservices), especially as applications are assembled using heavier doses of shared, open source componentry.

    As further evidence, nearly every major solutions vendor focused on the DevOps space—whether oriented toward the vibrant application performance management (APM), configuration management and continuous testing segments, among many others—seems to have launched some initiative aimed at integrating security into their products.

    The rise of security capabilities adjacent to—and supporting inclusion into—the DevOps world, such as container giant Docker’s Content Trust initiative, along with the advancement of related startups including Twistlock, Scalock and StackRoxs, offers more proof of this growing momentum. And that’s only the container security piece of the puzzle.

    Of course, there are also a number of emerging startups aimed at addressing the DevOps security opportunity, backed by well-known venture capitalists. A shortlist might include providers such as Immunio, Prevoty and UpGuard (formerly known as ScriptRock, which recently rebranded itself under the guise of a “DevOps security” provider), and we’re sure to see many more. And don’t forget the growing emphasis on DevOps security among established security industry stalwarts such as CyberArk, Snort and Tripwire, among others.

    For a comprehensive list of the various types of DevOps security best practices and tools that we should expect to see gain wider adoption, this October 2015 blog by noted security analyst Adrian Lane of Securosis serves as a basic framework. Each of the areas of focus that he cites clearly offers significant opportunity for both new and existing methodologies.

    Finally, one need look no further than this year’s RSA Security Conference, to be held in San Francisco at this end of this month, as tacit proof of the continued growth of what people now varyingly refer to as DevSecOps or Rugged DevOps (I prefer the latter as it just sounds cooler).

    In addition to multiple conference sessions, notably a “DevOps Throwdown” between security industry veterans Caleb Sima, Chris Wysopal and Gary McGraw, there’s also the 2nd Annual DevOps Connect: Rugged DevOps Edition. If last year’s event is any indication (standing room only, venue literally at max capacity), this daylong DevOps security track at RSA will be an even more high-profile element of the entire week.

    Further, if DevOps security—or, at the very least, the growing focus on adjacent segments such as container security—isn’t the unofficial theme of this year’s entire RSA show, I’d be extremely surprised.

    Meanwhile (self-serving), my employer CA Technologies absolutely has a lot going on that complements and contributes to this growing DevOps Security market momentum.

    For all of those reasons, here I find myself again, drawn closer back into the security space that I’ve called home for roughly the last decade. I guess that’s just how it goes, as DevOps and security are inarguably two of the most significant trends in IT these days and inextricably linked at their respective cores.

    There are definitely worse fates out there. But, just when you think you’re out …

  • OpsClarity Introduces Monitoring for Data-First Applications

    SUNNYVALE, Calif.–(Business Wire)–OpsClarity, the intelligent web-scale applications monitoring company, today announced that its Intelligent Monitoring solution now provides monitoring for the growing and popular suite of open source data processing frameworks. OpsClarity understands the extremely complex and distributed runtime characteristics of modern data processing frameworks like Apache Kafka, Apache Storm, Apache Spark as well as datastores such as Elasticsearch, Cassandra, MongoDB and others that act as sinks to these data processing frameworks. The solution enables DevOps teams to gain visibility into how these technologies are dependent on each other and troubleshoot performance issues.

    “Open source data processing frameworks have rapidly matured and gained enterprise adoption to provide immediate business value, whether it be to identify customer preferences on the fly, detect online fraud or IOT-enable the next electronic device in our homes,” said Amit Sasturkar, Co-Founder and CTO of OpsClarity. “OpsClarity has deep domain understanding of these distributed and complex data processing frameworks and how they work together and has built an intelligent assistant that visualizes the entire environment, detects and correlates failures, and provides guided troubleshooting.”

    Enterprises use big-data frameworks to process and understand large-scale data. Technologies like Apache Kafka, Apache Spark and Apache Storm are constantly expanding the scope of what is possible. However, most of these data processing frameworks are themselves a complex collection of several distributed and dynamic components such as producers/consumers and masters/slaves. Monitoring and managing these data processing frameworks and how they are dependent on each other is a non-trivial undertaking and usually requires an extremely experienced operations expert to manually identify the individual metrics, chart and plot them, and then correlate events across them.

    “Unresponsive applications, system failures and operational issues adversely impact customer satisfaction, revenue and brand loyalty for virtually any enterprise today,” said Holger Mueller, VP & Principal Analyst at Constellation Research. “The distributed and complex characteristics of modern data-first applications can add to these issues and make it harder than ever to troubleshoot problems. It is good to see vendors addressing this critical area with approaches that include analytics, data science, and proactive automation of key processes to keep up with the changes being driven by DevOps and web-scale architectures.”

    OpsClarity leverages an advanced data-science and real-time streaming analytics-based approach to ingest huge amount of metrics and events data from a disparate set of the open source frameworks and intelligently correlate metrics and events across them. OpsClarity synthesizes the various metrics, alerts and signals into an intuitive visual service topology along with overlaid health status. This radically simplifies the effort required by DevOps to set up and troubleshoot these modern data frameworks.

    The OpsClarity Intelligent Monitoring solution provides the following for data processing frameworks:

    • Auto-Discover: Automatically discover all the components of various data processing frameworks and automatically configure a deep and specific collection of metrics, events, alerts, process and network data. For example, Kafka brokers, Spark masters/slaves, Storm supervisors/workers are auto-discovered and auto-configured.
    • Visual Topology: Automatically discover the service connections and dependencies to generate a logical visual topology for these data processing frameworks.
    • Health Analysis: Enables immediate understanding of data processing framework component health, prioritized anomalies, and service-level metrics – all within the context of the topology.
    • Troubleshooting: a highly-specific and actionable anomaly detection and event correlation capability that allows for rapid root cause analysis.

    To learn more about OpsClarity and its Intelligent Monitoring solution, please click here.

    About OpsClarity

    OpsClarity is transforming modern software operations with data-science-driven monitoring solutions that help today’s DevOps teams continuously visualize, understand and troubleshoot performance of their rapidly changing web-scale application environments. Built by alumni from Yahoo, eBay, Facebook, Twitter and Google, the OpsClarity Intelligent Operations Platform is the only solution that can dynamically adapt to and learn every component of a company’s applications and infrastructure and extract key insights needed to maximize operational speed and agility. OpsClarity is privately held and headquartered in Sunnyvale, Calif. For more information about OpsClarity, visithttp://www.opsclarity.com.

    OpsClarity
    Sachin Agarwal, 408-462-9538
    Press@OpsClarity.com

  • Wix requires visibility, and insights are the key

    Wix requires visibility, and insights are the key

    One of the key tenants of the DevOps principle is that your environment has to be prepared for anything. If your shiny modern delivery chain does not support the adoption of new DevOps tools, or does not help you learn, and iterate, then it will ultimately fall into the same category of solutions as Waterfall, and will be on a clear path to Legacy.

    (more…)

  • Carving through the hype – take the DevOps maturity assessment

    Carving through the hype – take the DevOps maturity assessment

    As anyone who spends time with their family over the holidays can attest, the notion of maturity is a highly subjective concept.

    Whether it’s sizing up the next generation’s advancement or defaulting to time-honored inter-sibling behaviors, annual get-togethers offer tangible proof of what’s new and those things that likely won’t ever change.

    In the domain of DevOps – an umbrella concept that encompasses a massive array of processes and supporting capabilities – gauging the maturity of your organization’s standing is also a highly interpretive task.

    Given that “DevOps” means something different to nearly every organization, based on its unique makeup, methodologies, tools, and most importantly, business goals, attempting to measure progress against any outside model may seem an effort of limited value.

    That said, as an increasing number of organizations seek to embrace DevOps, and growing recognition that well-worn stories of leading-edge “unicorns” don’t relate closely to most people’s real-world expectations, many teams are left wondering where their own efforts stand, especially in comparison to their peers.

    To help acquit this pursuit, and allow organizations to scope their progress, CA recently launched the second iteration of its online DevOps Assessment.

    Split into three short surveys covering core limbs of the DevOps tree – “Develop and Test”, “Release and Deploy”, and “Manage and Monitor” – this free tool offers detailed insight into how your organization’s maturity stacks up to that of other practitioners.

    For instance, in the domain of development, how well do your teams fare in defining and managing requirements, and then presenting them to others in a structured manner? Related to release cycles, are your teams truly meeting business-driven expectations for more frequent application deployments, shorter lead times, and higher success rates? Or, when it comes to applications monitoring, are you able to distinguish the most critical and recurrent problems and how they truly impact business services?

    These are perhaps some of the most fundamental questions that one could apply to measurement of DevOps adoption in general, but further investigation attempts to take a closer look at more finite, underlying issues.

    Are your development teams accurately replicating realistic conditions from production systems to create virtual services and testing scenarios? Are your application release deployments fully automated across development, test and production environments? Can your teams monitor complete business services that link user experience, transactions and applications to infrastructure to pinpoint any root cause issues?

    The best thing about the survey is that it immediately provides visibility into how your answers compare to those of previous respondents. There’s no fuzzy math or editorial interpretation (though we may highlight trends in a report at some point), merely instant validation, and the option to create a customized report card charting where your efforts stand in relation to those of your peers.

    If you’re wondering precisely just how to achieve your long-term DevOps goals comparative to where you stand today, it might even be worth leveraging some of the involved questions for the purpose of evolving process and methodology.

    The only remaining questions are: Do you think that you’re ahead of the curve? Are you afraid to find out just how far you’re behind the crowd? And most importantly, what do you have to lose?

    Take the survey today and get a real-world fix on your overall level of DevOps maturity. Guaranteed it’s a far more informative activity than debating family and friends over whose turn it is to carve the turkey, or wash the dishes.

  • APM on-premises or SaaS: What’s the best fit?

    APM on-premises or SaaS: What’s the best fit?

    Today’s DevOps teams need in-depth monitoring that offers insights into every aspect of the application-delivery chain. When you decide for an application performance monitoring (APM) solution, you can choose between an on-premises approach or SaaS-based solutions (“Monitoring as a Service”). Every deployment option brings its pros and cons or has advantages for special use cases. An installation on-premises for example works best if you want to monitor an intranet that you want to keep isolated.

    Which APM deployment option makes most sense for your organization? There‘s no simple answer to this question. A lot depends on your environment and requirements. Let’s explore the points to consider in deciding between SaaS and on-premises monitoring:

    Server location

    With in-house, on-premises systems, all servers and data are physically under your control. When you use SaaS-based monitoring, performance metrics, reliability data, and detected issues are stored in the cloud. If your industry has strong regulatory and policy requirements, SaaS may not be an option for your organization—you may need to keep all such monitoring data on-premises.

    Implementation

    To deploy on-premises solutions you need to provide your own hardware and storage in a private datacenter or cloud. You install and configure the monitoring software, databases, and other components of the APM solution. In general, SaaS offerings can be installed and set up within minutes using a self-service system. Some SaaS APM tools even auto-detect your entire server landscape within minutes following installation.

    Software updates

    SaaS solutions are typically updated automatically on a frequent basis by the vendors who provide them to deliver support for new technologies and feature enhancements. On-premises installations require that you manage updates yourself and stay current with the release of new product updates. Updates for on-premises solutions typically come out less frequently, which means you sometimes have to wait longer for support of new technologies and bug fixes.

    Server and database backups

    Maintaining your own servers for an on-premises APM solution requires that you manage the servers, data storage, and data backups yourself. SaaS solutions are maintained and backed up by their vendors.

    Costs

    On-premises software is usually licensed for longer periods of time. This means that the initial investment is often higher, but costs are recouped over time. SaaS-based APM often features “pay as you go” pricing, so you only pay for what you use. This model can be cheaper, but it depends heavily on your usage and your tool vendor’s pricing model. If you run extra servers during periods of peak demand, it makes a big difference if your billing is based on the maximum number of servers you use rather than your actual usage. Because they’re built on cloud infrastructure and often don’t require long-term contracts, SaaS make it much easier for you to opt out of the system and cancel any time. As you aren’t required to purchase and maintain your own hardware, they’re also typically more flexible and easier to scale.

    Support

    When your monitoring solution runs on-premises it’s often harder to get vendor support because the system may not be reachable from outside your organization. With SaaS-based solutions you can grant access to your tool vendor’s support staff and work with them on resolving issues. This makes the SaaS approach faster and easier if you run into special cases that require vendor support.

    Mobility

    With on-premises solutions you must connect to your internal network via VPN to view monitoring data on mobile devices (unless your organization exposes this service to the Internet). Conversely, you can access monitoring data supplied from a SaaS-based solution anywhere you have an Internet connection.

    Customization

    With an on-premises APM solution, you can expect in-depth customizations that suit the exact requirements of your organization. Customization of SaaS solutions on the other hand is limited, and dictated by rules defined by your tool vendor.

    Distributed applications

    If you need to monitor applications that run in a public or hybrid cloud scenario, it’s likely that you’ll also want to run your monitoring in the cloud. SaaS-based solutions enable you to seamlessly monitor cloud-based environments because this is what they’re designed to do. You only need to install agent software on the involved servers.

    (No) single point of failure

    On-premises solutions are vulnerable to critical problems like power outages and broken network connections, potentially leaving you with no way of receiving performance and availability problem alerts. On the other hand, SaaS monitoring in the cloud runs in a different geographical location and so remains available to send you alerts whenever problems are detected.

    Reliability and redundancy

    When your APM monitoring solution runs in your own data center, your IT department can determine appropriate service availability levels. SaaS monitoring providers often use software architectures that can be updated without downtime and reach service availability levels of up to 100%. Well-designed SaaS solutions use redundancy to further improve availability. It’s recommended that you take a close look at any SaaS-based solution vendor’s claim of providing “always on” availability.

    Scalability

    SaaS tool vendors ensure that needed resources like disk space and CPU will scale up with your demand. This means you won’t need to budget for additional hardware purchases and typically leads to lower total cost of ownership (TCO) with cloud-based SaaS solutions.

    Conclusion

    Deploying monitoring software in house (on-premises) gives you full control over your data. “Monitoring as a Service” in the cloud brings new possibilities for many monitoring use cases. Depending on your requirements, it could even make sense to use both. Some tool vendors are now providing monitoring solutions that combine the best of both worlds.

    With the points covered in this article, you should now know everything you need to consider when deciding which monitoring approach makes most sense for your organization.

  • Ancestry.com Brings BlazeMeter into the Fold

    Ancestry.com Brings BlazeMeter into the Fold

    In order to gather and organize your lineage and the branches in your family tree, Ancestry has spent more than a generation (the company was born in 1983) wading through the 16B+ images and records buried in historical society data, government archives, and information caches of like organizations, groups, and communities the world over.

    Data in hand, Ancestry transcribed it and converted it into a searchable format for your investigation so you can connect the dots from great, great aunt Tildy to great, great, great grandpa Jebediah. “These historical records date back to the 11th century and form more than 30K collections that include birth, marriage, and death records as well as military service, immigration, and even incarceration records,” says Norman Warren of the Ancestry Performance Team, Ancestry.

    Ancestry’s platforms enable it to unite descendants and the curious with true tales from 70M family trees that subscribers build up by adding their own branches. Interestingly, Ancestry data now relies on a million plus DNA samples and counting.

    “Our servers respond to more than 75M searches per day,” says Warren, who coaches Ancestry’s teams in refining load testing practices as they release new testing scripts before and after sprints and based on their individual development timing.

    Load Testing Before & After

    When the Ancestry team went looking for deeper, broader load testing capabilities, they were hoping to add routine testing across the site 365-days a year. The folks at Ancestry wanted a malleable solution that would invite them to apply JMeter to vertical load testing and permit them to cut costs for load testing by leveraging the advantages of open source testing tools.

    Ancestry had been using a load testing approach that did not allow Warren’s teams to use home-brewed scripts. But in order to benefit from a testing solution that did embrace such scripts, Ancestry had to ensure a flow of updated quality scripts from every applicable internal group. Because Ancestry ultimately opted for JMeter and BlazeMeter, that presented challenges such as JMeter and BlazeMeter learning curves, the need for users to make JMeter requests in parallel, and Ancestry apps that are harder to script.

    Growing pains accompanied BlazeMeter adoption including the pains of stretching to embrace different devops teams in order to retrieve testing scripts. By pointing out the advantages of the new testing paradigm, making the argument for time saved by site wide load testing, and providing training in JMeter, Ancestry made the leap to the BlazeMeter supported approach. “BlazeMeter provided excellent JMeter training while we evangelized for teams to own their scripts,” says Warren.

    Ancestry solved its technical and learning curve challenges with steps including installing the JMeter Webdriver Set plugin and learning to debug and test scripts before they would run optimally in the BlazeMeter cloud service.

    “We had to work to get adoption, but it was well worth the effort to build accountability over different sections of the site. We now ask teams to take ownership of their own scripts to review, modify, or emulate user traffic flows,” says Warren.

    Results

    “BlazeMeter enabled more frequent load testing at a competitive cost and good site-wide load testing with the ability to add loads to stacks or to stop loads on one of the stacks during the test,” says Warren.

    BlazeMeter helped the Ancestry teams to foster a strong confidence that they could support heavy traffic loads at their most hectic times. “The BlazeMeter team helped us to translate the metrics we use for measuring traffic into a number of concurrent users for loading the site. In this way, BlazeMeter helped us to measure load test results against a baseline, such as traffic on our most recently busiest day,” says Warren.

    Ancestry leveraged BlazeMeter’s swift feedback about site throughput and the response times of each page to speed site responsiveness for each user. Ancestry was able to predict and automate the flows that are most commonly used by customers in order to streamline processes and speed the user experience even more.

    “We created a new baseline from new site wide load tests of additional traffic flows by loading flows incrementally using a special BlazeMeter feature that enables us to dynamically add more concurrent users in the midst of a load test,” explains Warren.

  • Digital Disrupters – Building a DevOps Leadership Profile

    Digital Disrupters – Building a DevOps Leadership Profile

    While the advancement of DevOps methodology remains fairly nascent for many organizations, an ever-expanding class of leading practitioners are evolving a wide range of related best practices.

    Beyond the well-worn concept of “unicorns” – the scant few organizations that stand as mythical examples of DevOps nirvana – this broader set of early adopters have begun to pull back the curtain on their efforts, lending further detail to the strategies that have bred this current model for success.

    Last week, industry analysts Freeform Dynamics published valuable research that highlights the ongoing work of such DevOps innovators, dubbed as “Digital Disrupters” for their accomplishments in pursuing larger digital transformation. The report “Exploiting the Software Advantage” – sponsored by CA Technologies, outlines a critical set of initiatives crucial to navigating the DevOps landscape.

    What are the specific hallmarks of those organizations leading the charge? Where are others struggling to catch-up? Many of the Freeform Dynamics survey results, based on 1442 respondents, reaffirm widespread DevOps observations, while other findings serve to shed new light, including:

    • Everyone builds software: Perception is, in fact, reality and to deeply engrain digital transformation, Digital Disruptors are increasingly identifying with this mantra. While 95% of leaders have committed to becoming more app-centric, only 51% of mainstream organizations have adopted this persona. Similarly, 84% of Disruptors directly identify as “software companies”, regardless of industry, compared to 40% among the masses.
    • Transformation is required: This point may seem obvious, but it certainly bears repeating. A whopping 92% of leaders have actively moved to break down traditional silos between Dev and Ops, with 51% of those organizations having already completed this work. By comparison, only 18% of mainstream organizations claim to have done the same, supported by 59% who at least share this more collaborative DevOps vision.
    • Commitment is critical: This may also seem like a no-brainer, but the numbers bear out this requirement. Some 77% of Disruptors have agile development methodologies in place, with 70% having “broadly” implemented DevOps activities. For mainstream organizations, these metrics rate at only 31% and 21%, respectively; leaders are going all-in and appreciating the benefits, and those that follow will need to do the same.
    • Experimentation is omnipresent: To garner the benefits of continuous delivery, embracing trial and error is a must. Among Disruptors, 93% recognized the need to “experiment and fail quickly on the road to success”, compared to only 58% of mainstream organizations. Some 95% of leaders have already shifted from fixed product cycles to continuous innovation, while only 56% of the mainstream have made this leap.
    • APIs are everywhere: Ubiquitous integration, both in-house and in connecting to third parties, has only increased emphasis on the API. Disruptors are resoundingly focused on API management, with 68% using APIs to integrate third party services into their applications, and 66% to offer external access to systems and data. Mainstream peers compare at only 26% and 24%, respectively, denoting a sizable API management gap.

    These are only a handful of the most dramatic and obvious areas where DevOps advocates have clearly re-architected their approach, separating themselves from those organizations that have yet to go all-in; it doesn’t stop there.

    As highlighted in the research, other areas of differentiation and dichotomy include cross-functional alignment, emphasis on digital channels, use of digital to drive core business efficiency, and even funding of digital investments.

    Beyond these key philosophical milestones that litter the pathway to DevOps success, the survey also highlights the vast array of strategic goals that these organizations have sought to accomplish. The results spotlight the massive range of significant business benefits that practitioners believe DevOps can deliver.

    Leading use cases include:

    • Improved customer satisfaction (49%)
    • Better workforce productivity (44%)
    • Improved customer experience (41%)
    • Improved competitive differentiation (38%)
    • Better agility / speed-to-market (37%)
    • Opening up of new markets (35%)
    • Creation of new business models (30%)
    • Creation of new routes to market (26%)

    If one were to take those outcomes out of context, unrelated to the impact of DevOps, it’s fair to say that they read like a generic list of leading business initiatives – further underlining the movement’s overall relevance.

    That’s not to say this is all accomplished easily. Report respondents detail hard won victories that required significant time and investment to achieve all this DevOps-driven advancement. After all, the notion of “disruption” doesn’t exactly evoke images of ease or comfort.

    Click here to read the report for yourself – there’s a lot more to be considered. Namely, how well your organization measures up in affecting digital transformation in its own right, and where it can take steps to move forward.

  • Native Integration of APM Driving DevOps Maturity

    Native Integration of APM Driving DevOps Maturity

    With applications performance monitoring established as one of the fundamental cornerstones of today’s DevOps methodologies, it’s easy to overlook the reality that effective instrumentation of such tools remains extremely difficult. Specifically, the biggest challenge remains creating performance metrics that address both IT and larger business initiatives.

    There’s no question that APM is one of the most widely adopted and deeply embedded elements in today’s DevOps world, serving as the critical source of measurement that enables more transparent and agile applications. Yet, despite the fact that performance monitoring solutions, and their respective providers, are considered to be among the most mature on the DevOps landscape, customers still struggle to optimize the systems to meet their specific requirements.

    However, as the market continues to advance, specifically via direct integration of widely-adopted APM tools with other strategic applications management systems, such as e-commerce platforms, organizations are finding a faster path to success, and bigger-picture DevOps benefits. One such partnership, announced in late August, saw market leader New Relic expanding integration with e-commerce platform provider Magento, owned by eBay.

    Jon Azrielant, director of marketing at online retailer Jewelry.com, a Magento customer, said that his company initially ran into significant obstacles in adapting New Relic’s APM capabilities to its processes, despite the comparative strength of the tools.

    “We had a really difficult time implementing New Relic initially, it was a painful process,” Azrielant said.

    While the APM tools provided Jewelry.com with the data necessary to generate vital alerts, along with more agile sysadmin and operations monitoring, the more silo-based, fragmented approach that it employed could not empower the retailer to circumvent traditional walls between apps support and service support teams, Azrielant said.

    Leveraging the extension of New Relic’s Software Analytics Cloud within Magento, the company has created more advanced instrumentation to inform critical business decisions. Among the most helpful of these measurements are those relating directly to customer experiences, while correlation of such metrics with Jewelry.com’s bottom line has also advanced executive decision-making, he said.

    “The critical result of this integration is more complete visibility across applications and business performance, importantly tied to the actual business ramifications,” said Doug McIver, director of product for Magento. “In the past, merchants had to do a lot of difficult integration to use these systems together, so obviously this is a hugely valuable capability for us to be able to offer, especially to customers with highly performant site requirements.”

    Moreover, Magento claims that it is growing its own customer base by tying APM more closely to its platform. Likewise, New Relic officials concede that to help advance crucial DevOps workflows, it must partner with such platforms to deliver APM data within the context of business requirements.

    “If we’re talking about how to win in world where we’re iterating the end users’ experience on a regular basis, we have to put performance metrics within context of the UI, and the business results; working from the same data sets and getting people on the same page is incredibly important,” said Abner Germanow, senior director of solutions marketing at New Relic.

    While pure performance stats are “near and dear” to New Relic for obvious reasons, the solutions provider is also attempting to advance a broader array of information that drives closer collaboration through new “iteration and experimentation”. New Relic is also focused on further developing the specific types of information most critical to key constituents such as e-commerce platform users. The need to create a common language for those DevOps efforts is amplified by the fact that development and operations officials are often unsure which metrics are critical to the business, and vice versa.

    “Instrumenting code has always been a challenge, and we’ve worked very hard to ease that pain across a lot of different platforms; there’s always another language where work can be done, and we look to our customers to help lead us into the next phase of instrumentation,” Germanow said. “We’re seeing a lot of pressure around the need for constant change and iteration in industries where it’s truly a matter of survival, such as e-commerce, publishing, digital governance and finance – places where the industry is behaving very differently now than it has over the last 50 years.”

    This work, and the subsequent APM integrations offered via platform providers such as Magento, have allowed end user organizations to remain focused on customer experience and sales, the ultimate goal of DevOps, said Jewelry.com’s Azrielant.

    “In terms of how this relates to the DevOps revolution, this is the advancement that is bringing everything together and allowing everyone to look at the same numbers, to share common APIs, to streamline reporting and event response, and define everything in uniform terms,” Azrielant said. “Ostensibly, the most basic tenet of DevOps centers around the unification of these disciplines, and you can’t support that without the unification of this information.”

  • Continuous test monitoring DevOps ”health-beat”

    Continuous test monitoring DevOps ”health-beat”

    In my prior blog Test Results Analysis at the Speed of DevOps I discussed ways that test analysis features and practices can be optimized to ensure that test results for each CI/CT cycle keep pace with accelerated continuous testing. Beyond optimizing continuous test analysis for each cycle, longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much. Some of the more difficult problems to detect and diagnose are those that only occur occasionally or under “corner case conditions”.

    Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles then there is no way to build sufficient sustainable confidence in the long term health of the DevOps pipeline or products produced by the pipeline. The combination of continuous test monitoring tools with continuous testing and analysis tools can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in the product for each release and its evolution over multiple releases. Below are some suggestions in a checklist format that have proven useful for continuous test monitoring for sustainable high performance DevOps environments.

    1. Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies, and performance degradation within the margin of a test but accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.
    2. Regression test product areas even though there were no expected changes: One of the practices in high performance DevOps environments is to skip tests for areas that are “known” not to have changed during individual CI/CT cycles. Unintended consequences of indirect changes may impact performance so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility features and upgrade/downgrade administration features.

    3. Select continuous test monitoring tools that collect and report trends: Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases, and functional tags can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.

    4. Use continuous test monitoring results to diagnose and resolve problems: Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

    The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable DevOps. At Spirent we think testing has a bright future in DevOps. You can hear more about our views on this topic by reviewing our joint webinar conducted together with test management product provider Zephyr at: Overcoming-DevOps Challenges

    What do you think of these suggestions and do you have others that should be mentioned?