devopsy.kinsta.cloud

Tag: continuous testing

From Laboratories to DevOps Factories for Continuous Testing
The prior blog Continuous Testing and Monitoring Essentials for SDN and NFV discussed the importance and suggested practices for testing and monitoring complex systems such as software defined networks and network functions that are virtualized . Whether the product being tested is a complex network system or not, with DevOps the laboratory is no longer just a place where engineers go to test their changes. In a DevOps environment, implemented according to best practices, the laboratory is becoming an on-line operation that is reliable, usable, remotely accessible and shareable by many types of remote users that expect 24/7/365 availability.
The old “Santa’s workshop” model of development engineering (elves) hand-crafting products in specialized local laboratories and then handing off their products to operations staff (reindeer) for infrequent delivery (on Santa’s sleigh once per year) is being replaced by a fully automated, orchestrated software DevOps laboratory infrastructure akin to a modern factory staffed by robots churning out products at a dizzy pace. Elves and reindeer need to rest between deliveries and simply can’t keep up with today’s continuous delivery expectations. Santa’s “lap-top” has become an app store where users can download products anytime no matter if they have been bad or good (but be good for goodness sake!).

If a modern DevOps factory-style laboratory is a priority on your Christmas list, here are some suggestions for tool requirements you may want to include on your letter to Santa.
1. Inventory tools to record and track resources:
  All of the systems used in tests such as test tools, systems under test, and interconnection systems need to be tracked in an on-line inventory database in which each resource and the critical attributes are visible and addressable by administration staff and users of the resources. The ability to discover and import new or changed resources is essential. Without these capabilities it is impossible to keep up with the continuous changes that occur as the DevOps continuous test environment evolves to meet ongoing requirements.
2. Tools to administer users:
  With a large number of users accessing lab resources features for administration of users and user groups are essential. Large scale DevOps environments require administration of hundreds or thousands of users with varying privileges for resource access according to project priorities and schedules. The ability for lab administrators to assign privileges to individual users or user-groups is essential to control access to some resources which may be restricted for security or other reasons.
3. Ability to create, and save test configurations:
  While tests may be created to work with a variety of configurations every test assumes something about the test configuration. In a test environment implemented according to best practices the configuration of the test, or at least an abstract version of the test configuration is specified and associated with each test. Configuration data is saved in a database and may be recalled by search attribute matches when a test needs to run.
4. Ability to reserve tests and test configurations:
  With a large number of lab users in large scale DevOps environments and limited lab resources conflict resolution features are needed. The ability assign real resources to abstract configurations provide the flexibility to use lab resources more efficiently. The ability to reserve test resources and test configurations in advance according to a calendar, or immediately when an event trigger fires are important options to minimize wasted time for both users who may have limited time windows or automated processes that have non-deterministic schedules.
5. Ability to orchestrate many automated tests in parallel:
  In large scale DevOps environments the number of tests that need to run in a given short DevOps cycle may vary from a few tests to thousands depending on the complexity and nature of the change that are integrated in the cycle. The tools need to be able to scale the tests and run on as many physical and virtual test resources as are available, to minimize the total time for test.
6. Test Environment Metrics:
  The entire DevOps pipeline depends on reliable, scalable and maintainable continuous testing. Built-in metrics are essential to monitor the environment’s reliability, scalability and maintainability.
The above is a partial list of suggestions for a state-of-the art DevOps continuous testing laboratory “factories” list. At Spirent we think continuous testing laboratory “factories” are critical for successful DevOps.

You can read more about our views at Spirent.com/solutions/LabManagement

What do you think of these suggestions and do you have others that should be mentioned?

In the meantime we wish Santa, Elves and Rudolf, reindeer friends and you happy holidays and continuous happiness for the New Year!
December 23, 2015
An obituary to a Managed Test Service

1st October 2020, Enterpirseborough , F500 Inc street

Managed Test Service, 66, died Friday, 3rd October, 2020 at the Delivery governance war room in the IT head quarters of F500 inc. following a terminal illness caused by a lifetime burden of 34 SLA’s, debt burden of 44 KPIs , 29 painful claw backs by procurement due to failed deliveries and overdose of turnaround specialists and consultants

MTS leaves 4 programs , 45 projects, 12 application centres and a number of small changes behind.Born and raised in the HQ , MTS was a lifelong resident of the borough.

In the early years, MTS would help do functional testing for the core system portfolio and eventually taking ownership of non-functional testing and user testing. After graduating from the delivery transformation high school , MTS attended the SLA University, graduating with a degree in cost optimisation and skill super-speciaisation. MTS eventually joined the IT Board, where he was the voice of quality(or so he believed)

His attention to detail and obsession with centralisation of anything that had Test in it’s name was legendary. He was proud and preached the virtues of , albeit bloated and inflexible, test service

However,he struggled with newer delivery approaches of Agile and DevOps which spread like cancer throughout the delivery organisation. Devops challenged the fundamental idea of skill super-specialisation and independent testing that MTS championed and built his empire on. DevOps challenged the way of living of MTS as unsustainable

DevOps valued people and interactions over process unlike MTS which obsessed about process

DevOps valued working software while MTS was worried about comprehensive documentation

MTS got caught up in time-wasting contract negotiations instead of allowing his team to focus on customer collaboration

MTS preferred a predictable plan while his delivery friends had already moved on with adaptable plans

A funeral service was held on Monday 6th October from the funeral Home of Procurement , with a private burial.

December 10, 2015
What is expected of software developers vs QA testers during development?

“What is expected of software developers vs QA testers during development?”

Michael Oubre – Agile evangelist, practitioner and trainer, and Marc Hornbeek – DevOps evangelist and thought leader, share their views on popular questions regarding Agile and DevOps from their respective viewpoints.

Michael: The thing we do in agile is to train the team to understand that they equally share the responsibility for all aspects of delivering completed, functioning code to customers, and that we deliberately try not to make any differentiation between the various skills possessed by the members of the team. As a result, we try to remove any reference to QA or PV or any other term that has the connotation of the old way of doing things – testing is done as part of the development and not a process that happens when everything is coded. Testing in an agile model is ALSO done incrementally, in-line with the development, which is what makes CI absolutely critical. Everyone on the team tests, it’s just that there are some tests, just like some coding activities, that require specialized, senior skills, so there are people who contribute to those testing activities more than others.

Marc: In best practice DevOps environments everyone on the team is responsible for quality. The specific task responsibilities for continuous testing may be allocated in different ways depending on the organization’s preferences but developers take at least equal responsibility for test results as QA team members. Whether the developer or tester is responsible to design and create tests for a code change is a team decision. For example a developer may take responsibility for white box tests because of familiarity with the code but a tester may be the one that creates functional test cases, and together the tester and developer may review all test cases and work together to ensure all tests and code changes reach a successful result within the development “pre-flight” environment prior to integration into the DevOps pipeline. In any case, no matter the allocation of test tasks, the code changes are not deemed ready for integration until the tests are ready and working with the code changes. Both the code and the tests for the code are submitted together into the DevOps integration branch. If this is not done there is risk that the code changes will not pass CI testing after integration, resulting in rollbacks and process waste.

Bottom line is, both Agile and DevOps do not prescribe a specific division of test task responsibilities for developers and testers. Both are equally responsible to ensure tests are conducted and successful results are achieved.

At Spirent we think Agile and DevOps work together to help teams develop and deliver quality software efficiently. More information about Spirent’s views can be found at the following links :
Does Agile Need continuous testing?
Spirent.com/solutions/devops

What do you think of these views and do you have other views?

Michael Oubre
is an expert in Agile practice, transformation and management and has spent more than 10 years as a practitioner, coach, trainer and evangelist. His long and eventful tech career has given him practical experience and insight into every aspect of building, deploying, supporting and managing software, systems and networks. https://www.linkedin.com/in/michaeloubre

November 19, 2015
Testing Nightmares: Real-Life Frightening QA Experiences

In the spirit of Halloween, this past month testing experts from around the globe answered the call to join in a “crowd-sourced” article highlighting their most frightening testing QA experiences. These are real-life examples of testing gone wrong—leading to all sorts of mayhem in some cases. We allowed these testers to hide their identity, and to keep their organizations anonymous to solicit the most behind-the-scenes responses.

In no particular order here are the top five Testing Nightmares, as told by the testers themselves:

The Importance Of Making Sure Your Hosting Company Gets Paid

By Ivar (from a medium sized company/ now acquired)

The company I was working for at the time had bet its existence on signing an enterprise contract that it had already been in negotiations for more than year with. We were building a digital asset management system that controlled the licensing and delivery of assets to corporate devices. (Think ‘private app store’.) I had written backup tooling that wasn’t ready for use but was included, unused, in the branch that was being deployed. Ten minutes after the final deployment (in preparation for client review!) all of our assets disappeared. Nothing new in the deployment was related to asset storage except my code. There was an eight-figure contract on the line and we had no backups. Fueled by the mother of adrenaline rushes, I reviewed my work and remained convinced that my code was not the cause of the loss. I was already into investigating other possible causes when another developer informed our team that after calling our asset host, he discovered that we had neglected to update our billing credit card. The host had turned off access to all of our assets ‘to get our attention’. They succeeded, but at oh what a price…

Not The Best Way To Start As The New Test Automation Engineer

By James Eisenhauer

This happened my first week as the new and only Quality Engineer at a health and wellness startup. I was on a mission to get test automation created and show them what I could do for them! My first test was to invite all the members of a test company to a new company wellness challenge. It just so happened the company President’s test account was a member of that test company. I scheduled my tests to run at 5 a.m. At 5:38 a.m. we all received an email from the President that read, “Emails every few minutes telling me that QA wants me to walk by the damn river. Make it stop!” “Walk by the River” was a copy of a real wellness challenge we had in our test system. Something in my auto-retry logic had gone horribly wrong. First impressions are a bitch!

Gaming Fail

By Abe Chiu

While working for a gaming app company, we pushed out a build with an engine update that reverted all of our users back to levels that were saved on their device. This as opposed to the one that was saved on our server side. (Example: If logged in to create an ID and was at level 50 on your device, that was the save that popped up even though on our server you had previously reached level 125). All of our users were affected. That’s not the worst part. Some of our users reverted back to level 1 when they were above level 100 and up. This all happened on a Friday and the build was pushed on Thursday night. On Friday we were only able to fix some issues with a small percentage of our user base but the ones that were reverted back to level 1, those were not able to be helped until the following Monday.

The White Screen of Death

By Anonymous

I work as a Software Engineer in test at a media company. This company has its own video player that is embedded either as a single standalone video or a playlist. The player was re-done completely, and I was responsible for testing it before it went out to production. I tested the player as a single video and found a number of issues so the release to production was pushed out. After the issues had been fixed, I tested it again as a video and then gave my “go-ahead” to release it to production. The problem was I had COMPLETELY FORGOTTEN to test the playlist! We had figured that everything looked good and since the release had already been pushed off once, we made a decision to release it to production at the end of the day, around 5:30 p.m. I double-checked it on production as a single video and it looked fine. An hour later, when I got home, we started receiving urgent messages and emails that the playlist wasn’t working!!! You could only watch the first video and then it became a blank white screen! Since the playlists are embedded so deeply, this was a major problem. At 6:30 pm that night only one engineer was around and online but he wasn’t sure how to revert it. Finally after a very stressful hour and a half, he figured out how to do it and I verified that everything was back and working again with the old player. I still can’t believe that I had forgotten to test the playlists, which is such an important part of the feature.

Global Communication Breakdown

By Joe Nolan

We were trying to create a mobile on-line conferencing feature to include in our education app, much like Google Hangouts. Testing had gone well in controlled environments. Then we pulled all of the teams together to have a large group call using and testing the feature. We immediately had reports of crashes and all sorts of issues. It was impossible to talk over the people trying to diagnose the problems, while at the same time others were randomly pushing buttons to try the feature. Even worse, participants were located around the world with different networks and connectivity issues. It was worse than herding cats!

About the Author/Sauce Labs

Sauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity, and reduce infrastructure costs for software teams of all sizes.

October 29, 2015
Continuous test and monitoring essentials for SDN and NFV
The prior blog Continuous Test Monitoring DevOps Healthbeat discussed the importance and suggested practices for testing and monitoring test trends. This need is amplified when the system that is being tested and monitored is a network of co-operating applications, infrastructures and devices such as networked components of next generation Software Defined Networks (SDN) and Network Function Virtualization (NFV) systems. In networked systems a problem with any network component such as end-user applications, network interface devices, network control plane, network data plane, load balancers, firewalls, network administration, orchestration, and network virtual infrastructures can seriously impact user and network services performance. The number, variety, topology and versions of components and their host environments present special challenges for DevOps continuous testing and monitoring.
In an SDN and NFV world all production topologies and network variations must be tested and results must be monitored or the results will be uncertain. Despite the complexity, or perhaps because of it, continuous testing and monitoring must be optimized to ensure that testing and results analysis for each CI/CT cycle keep pace with accelerated continuous testing and longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much.

Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles for all network nodes and all topologies then there is no way to build sufficient sustainable confidence in the long term health of the SDN network and NFV components. The combination of topology-aware continuous testing solutions together with continuous test monitoring tools that can aggregate results over the entire set of network resource can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in network components and services for each release and evolution over multiple releases.

Below are some suggestions in a checklist format that have proven useful for continuous testing and monitoring for sustainable high performance SDN and NFV systems and services.
1. Determine continuous testing and monitoring priorities: Some examples of problems that continuous testing and monitoring can help with include intermittent failures caused by the virtual network infrastructures, virtual network functions, network administration, orchestration and management, 3rd party network and end-user systems compatibility, network performance, network fault-tolerance features. The best practice for continuous testing and monitoring indicates that the problems of most concern to a specific network system or service will be tested and result trends will be monitored.
2. Regression test network systems and services even though there were no expected changes: Unintended consequences of indirect changes may impact performance so automated regression suites should audit all production network test topologies and version variations areas occasionally just to be sure. Typical examples that are often caught by this are features important to audit functions, system backwards compatibility features and upgrade/downgrade administration features.
3. Select continuous test and monitoring tools that collect and report trends:
  Tools that can correlate and report test results across multiple network nodes and topology variations can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.
4. Use continuous test monitoring results to diagnose and resolve problems:
  Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.
The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable SDN and NFV network systems and services. At Spirent and Zephyr we think continuous testing and monitoring is critical for successful DevOps. You can hear more about our views on this topic by reviewing our joint webinar at:
Overcoming DevOps Challenges

What do you think of these suggestions and do you have others that should be mentioned?

About The Authors ⁄ Marc Hornbeek & Sanjay Zalavadia

Marc Hornbeek is Sr. Solutions Architect of DevOps continuous test solutions at Spirent Communications, Infrastructure Test Optimization (ITO) BU. He recently managed DevOps at Spirent. He has performed as the primary architect of test automation tools and champion of test automation for firms ranging from start-ups to large multi-national companies. He published more than 30 articles and has been a speaker at numerous conferences and user forums primarily regarding topics related to continuous automated testing and DevOps.

Sanjay Zalavadia is responsible for driving custom success at Zephyr. This includes training, consulting, customer support and client management. Most recently, as the Associate Vice President for Patni Computers Telecoms IT Managed Services Practice, he established IT Operations teams supporting mobile content providers. Sanjay brings more than 15 years of leadership experience in IT and Technical Support Services teams across multiple geographies for both large and small companies. Sanjay has a graduate degree from the Manipal Institute of Technology in India
October 28, 2015
Does Agile need continuous integration?

Michael Oubre – Agile evangelist, practitioner and trainer, and Marc Hornbeek – DevOps continuous testing evangelist and thought leader, share their views on popular questions regarding Agile and DevOps from their respective viewpoints.

If you are doing Agile do you need to do Continuous Integration?

Michael: From an Agile perspective, let’s consider your company decided to take the plunge and switch its software development process to Agile, formed Agile teams, got them some training and set them up with a coach. The team have been scrumming for a couple of iterations but there’s not a lot of work getting finished, and many of the stories are having to be carried over from sprint to sprint to get the defects resolved.

You might be tempted to start worrying about the quality of the code being written, or whether you have the right ratio of test engineers, but it might be more useful to look at your build/test cycle. One of the big benefits of Agile is the reduction of the time between the code being written and the identification of defects – the shorter that time is, the lower the cost is to get it fixed. Agile removes the process barriers, but if you only rely on unit testing and infrequent integration testing, you won’t achieve the results that you hope for.

Agile teams will have a very difficult time of ever reaching high performance without the benefits of Continuous Integration providing the ability to deliver new code in an integrated, testable build to the Agile teams as close to “instantly” as you can manage.

Marc: From a DevOps perspective development teams that are “doing Agile” are the primary customers of Continuous Integration together with Continuous Testing.

Both Agile and DevOps emphasize the importance of incremental development processes rather than large, serial waterfall processes because incremental processes facilitate fast feedback and reduce process waste compared to long waiting periods between software creation and validation. In fact the DevOps “movement” was born out of a system test consultant’s frustration with the disconnect between some Agile development teams and separate operations teams. (Reference: http://itrevolution.com/the-history-of-devops/) . Today the words Continuous Integration and DevOps, together with Continuous Testing and Continuous Delivery are practically synonymous.
The mere fact that there are plenty of successful Agile examples that existed prior to DevOps or Continuous Integration became a word tells you that Agile can indeed be done without CI, but clearly CI is a great complement to Agile and supports the goals of Agile while helping to tie it to operations. So the strict answer the question “If you are doing Agile do you need to do Continuous Integration?” is, well, “no” but to leave it at that would be incomplete, so a better answer is “ no, but CI is definitely recommended to complement Agile”.

Bottom line is, from both Agile and DevOps perspectives continuous integration is really important to achieving performance goals. At Spirent we think Agile and DevOps work together to help teams develop and deliver quality software efficiently. More information about Spirent’s views can be found at the following link: Spirent.com/solutions/devops

What do you think of these views and do you have other views?

About Michael Oubre

Michael Oubre is an expert in Agile practice, transformation and management and has spent more than 10 years as a practitioner, coach, trainer and evangelist. His long and eventful tech career has given him practical experience and insight into every aspect of building, deploying, supporting and managing software, systems and networks.

October 26, 2015
Digital Disrupters – Building a DevOps Leadership Profile
While the advancement of DevOps methodology remains fairly nascent for many organizations, an ever-expanding class of leading practitioners are evolving a wide range of related best practices.

Beyond the well-worn concept of “unicorns” – the scant few organizations that stand as mythical examples of DevOps nirvana – this broader set of early adopters have begun to pull back the curtain on their efforts, lending further detail to the strategies that have bred this current model for success.

Last week, industry analysts Freeform Dynamics published valuable research that highlights the ongoing work of such DevOps innovators, dubbed as “Digital Disrupters” for their accomplishments in pursuing larger digital transformation. The report “Exploiting the Software Advantage” – sponsored by CA Technologies, outlines a critical set of initiatives crucial to navigating the DevOps landscape.

What are the specific hallmarks of those organizations leading the charge? Where are others struggling to catch-up? Many of the Freeform Dynamics survey results, based on 1442 respondents, reaffirm widespread DevOps observations, while other findings serve to shed new light, including:
- Everyone builds software: Perception is, in fact, reality and to deeply engrain digital transformation, Digital Disruptors are increasingly identifying with this mantra. While 95% of leaders have committed to becoming more app-centric, only 51% of mainstream organizations have adopted this persona. Similarly, 84% of Disruptors directly identify as “software companies”, regardless of industry, compared to 40% among the masses.
- Transformation is required: This point may seem obvious, but it certainly bears repeating. A whopping 92% of leaders have actively moved to break down traditional silos between Dev and Ops, with 51% of those organizations having already completed this work. By comparison, only 18% of mainstream organizations claim to have done the same, supported by 59% who at least share this more collaborative DevOps vision.
- Commitment is critical: This may also seem like a no-brainer, but the numbers bear out this requirement. Some 77% of Disruptors have agile development methodologies in place, with 70% having “broadly” implemented DevOps activities. For mainstream organizations, these metrics rate at only 31% and 21%, respectively; leaders are going all-in and appreciating the benefits, and those that follow will need to do the same.
- Experimentation is omnipresent: To garner the benefits of continuous delivery, embracing trial and error is a must. Among Disruptors, 93% recognized the need to “experiment and fail quickly on the road to success”, compared to only 58% of mainstream organizations. Some 95% of leaders have already shifted from fixed product cycles to continuous innovation, while only 56% of the mainstream have made this leap.
- APIs are everywhere: Ubiquitous integration, both in-house and in connecting to third parties, has only increased emphasis on the API. Disruptors are resoundingly focused on API management, with 68% using APIs to integrate third party services into their applications, and 66% to offer external access to systems and data. Mainstream peers compare at only 26% and 24%, respectively, denoting a sizable API management gap.
These are only a handful of the most dramatic and obvious areas where DevOps advocates have clearly re-architected their approach, separating themselves from those organizations that have yet to go all-in; it doesn’t stop there.

As highlighted in the research, other areas of differentiation and dichotomy include cross-functional alignment, emphasis on digital channels, use of digital to drive core business efficiency, and even funding of digital investments.

Beyond these key philosophical milestones that litter the pathway to DevOps success, the survey also highlights the vast array of strategic goals that these organizations have sought to accomplish. The results spotlight the massive range of significant business benefits that practitioners believe DevOps can deliver.

Leading use cases include:
- Improved customer satisfaction (49%)
- Better workforce productivity (44%)
- Improved customer experience (41%)
- Improved competitive differentiation (38%)
- Better agility / speed-to-market (37%)
- Opening up of new markets (35%)
- Creation of new business models (30%)
- Creation of new routes to market (26%)
If one were to take those outcomes out of context, unrelated to the impact of DevOps, it’s fair to say that they read like a generic list of leading business initiatives – further underlining the movement’s overall relevance.

That’s not to say this is all accomplished easily. Report respondents detail hard won victories that required significant time and investment to achieve all this DevOps-driven advancement. After all, the notion of “disruption” doesn’t exactly evoke images of ease or comfort.

Click here to read the report for yourself – there’s a lot more to be considered. Namely, how well your organization measures up in affecting digital transformation in its own right, and where it can take steps to move forward.
October 14, 2015
Native Integration of APM Driving DevOps Maturity

With applications performance monitoring established as one of the fundamental cornerstones of today’s DevOps methodologies, it’s easy to overlook the reality that effective instrumentation of such tools remains extremely difficult. Specifically, the biggest challenge remains creating performance metrics that address both IT and larger business initiatives.

There’s no question that APM is one of the most widely adopted and deeply embedded elements in today’s DevOps world, serving as the critical source of measurement that enables more transparent and agile applications. Yet, despite the fact that performance monitoring solutions, and their respective providers, are considered to be among the most mature on the DevOps landscape, customers still struggle to optimize the systems to meet their specific requirements.

However, as the market continues to advance, specifically via direct integration of widely-adopted APM tools with other strategic applications management systems, such as e-commerce platforms, organizations are finding a faster path to success, and bigger-picture DevOps benefits. One such partnership, announced in late August, saw market leader New Relic expanding integration with e-commerce platform provider Magento, owned by eBay.

Jon Azrielant, director of marketing at online retailer Jewelry.com, a Magento customer, said that his company initially ran into significant obstacles in adapting New Relic’s APM capabilities to its processes, despite the comparative strength of the tools.

“We had a really difficult time implementing New Relic initially, it was a painful process,” Azrielant said.

While the APM tools provided Jewelry.com with the data necessary to generate vital alerts, along with more agile sysadmin and operations monitoring, the more silo-based, fragmented approach that it employed could not empower the retailer to circumvent traditional walls between apps support and service support teams, Azrielant said.

Leveraging the extension of New Relic’s Software Analytics Cloud within Magento, the company has created more advanced instrumentation to inform critical business decisions. Among the most helpful of these measurements are those relating directly to customer experiences, while correlation of such metrics with Jewelry.com’s bottom line has also advanced executive decision-making, he said.

“The critical result of this integration is more complete visibility across applications and business performance, importantly tied to the actual business ramifications,” said Doug McIver, director of product for Magento. “In the past, merchants had to do a lot of difficult integration to use these systems together, so obviously this is a hugely valuable capability for us to be able to offer, especially to customers with highly performant site requirements.”

Moreover, Magento claims that it is growing its own customer base by tying APM more closely to its platform. Likewise, New Relic officials concede that to help advance crucial DevOps workflows, it must partner with such platforms to deliver APM data within the context of business requirements.

“If we’re talking about how to win in world where we’re iterating the end users’ experience on a regular basis, we have to put performance metrics within context of the UI, and the business results; working from the same data sets and getting people on the same page is incredibly important,” said Abner Germanow, senior director of solutions marketing at New Relic.

While pure performance stats are “near and dear” to New Relic for obvious reasons, the solutions provider is also attempting to advance a broader array of information that drives closer collaboration through new “iteration and experimentation”. New Relic is also focused on further developing the specific types of information most critical to key constituents such as e-commerce platform users. The need to create a common language for those DevOps efforts is amplified by the fact that development and operations officials are often unsure which metrics are critical to the business, and vice versa.

“Instrumenting code has always been a challenge, and we’ve worked very hard to ease that pain across a lot of different platforms; there’s always another language where work can be done, and we look to our customers to help lead us into the next phase of instrumentation,” Germanow said. “We’re seeing a lot of pressure around the need for constant change and iteration in industries where it’s truly a matter of survival, such as e-commerce, publishing, digital governance and finance – places where the industry is behaving very differently now than it has over the last 50 years.”

This work, and the subsequent APM integrations offered via platform providers such as Magento, have allowed end user organizations to remain focused on customer experience and sales, the ultimate goal of DevOps, said Jewelry.com’s Azrielant.

“In terms of how this relates to the DevOps revolution, this is the advancement that is bringing everything together and allowing everyone to look at the same numbers, to share common APIs, to streamline reporting and event response, and define everything in uniform terms,” Azrielant said. “Ostensibly, the most basic tenet of DevOps centers around the unification of these disciplines, and you can’t support that without the unification of this information.”

September 29, 2015
Continuous test monitoring DevOps ”health-beat”
In my prior blog Test Results Analysis at the Speed of DevOps I discussed ways that test analysis features and practices can be optimized to ensure that test results for each CI/CT cycle keep pace with accelerated continuous testing. Beyond optimizing continuous test analysis for each cycle, longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much. Some of the more difficult problems to detect and diagnose are those that only occur occasionally or under “corner case conditions”.

Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles then there is no way to build sufficient sustainable confidence in the long term health of the DevOps pipeline or products produced by the pipeline. The combination of continuous test monitoring tools with continuous testing and analysis tools can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in the product for each release and its evolution over multiple releases. Below are some suggestions in a checklist format that have proven useful for continuous test monitoring for sustainable high performance DevOps environments.
1. Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies, and performance degradation within the margin of a test but accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.
2. Regression test product areas even though there were no expected changes: One of the practices in high performance DevOps environments is to skip tests for areas that are “known” not to have changed during individual CI/CT cycles. Unintended consequences of indirect changes may impact performance so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility features and upgrade/downgrade administration features.
3. Select continuous test monitoring tools that collect and report trends: Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases, and functional tags can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.
4. Use continuous test monitoring results to diagnose and resolve problems: Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.
The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable DevOps. At Spirent we think testing has a bright future in DevOps. You can hear more about our views on this topic by reviewing our joint webinar conducted together with test management product provider Zephyr at: Overcoming-DevOps Challenges

What do you think of these suggestions and do you have others that should be mentioned?
September 11, 2015
Librato Seeks to Unlock DevOps Application Monitoring

As wider adoption of DevOps methodologies continues to foster demand for new tooling to help streamline evolving workflows, at least one vendor is attempting to address the growing need for related monitoring capabilities.

Librato, a San Francisco-based provider of Amazon AWS applications performance monitoring (APM) services, claims that it is finding favor among the rapidly expanding DevOps community driven by its ability to inject critical visibility into the engineering lifecycle.

Acquired by Austin, Texas-based SolarWinds in January for a lofty $40 million and renamed Librato SolarWinds Cloud, the company is touting its recent adoption by well-known providers such as Slack for monitoring of DevOps-centric applications performance and health metrics.

“At its core, DevOps is destroying conventional wisdom around ritualized processes and the need to perform them, mostly in the domain of scale,” said Dave Josephsen, Developer Evangelist at Librato. “By attempting to include monitoring as a key element of the development lifecycle, this has created a whole new range of challenges, and opportunities.”

By providing a centralized Web interface and open API to help create visibility into critical applications engineering requirements, Librato maintains that it is creating the tooling necessary to empower integrated development and deployment processes, such as continuous change control.

“There are a lot of disparate tools for monitoring individual engineering tasks, but there hasn’t been any practical way to bubble up that data for efficient monitoring,” Josephsen said. “Within the context of DevOps, change control is a ritual that we’ve abandoned as we’ve embraced test driven development and automated away the human element, so there’s a distinct need for monitoring of feedback and metrics.”

For example, cloud-based productivity messaging specialists Slack are using the Librato “StatsD” lightweight daemon process to stream business and app-health metrics regarding PHP applications into the monitoring platform. Slack also leverages Librato to chart data regarding connect, query and fetch latencies for its server clusters, and to monitor internal metrics about overall system health.

“Having those graphs side-by-side lets us clearly see and compare the effects of network latency on one of our new services, and that’s very cool,” Richard Crowley, director of operations at Slack, wrote in a recent testimonial posted to the Librato Solarwinds blog.

Industry experts agree that broader adoption of DevOps strategies will continue to drive demand for relevant monitoring tools, with solutions such as Librato seeking to deliver “modern stack architecture for looking at the modern stack.”

“I think that APM is always interesting because it’s always new,” said James Governor, industry analyst with RedMonk. “The pace with which we’re building new architectures to deliver services to end users is increasing all the time, and with every new wave of technology, we tend to get new and associated models of APM tooling.”

Librato is taking advantage of the need for a system that brings together the elements necessary for cloud native monitoring, and in an agentless manner different from other competing tools such as New Relic, the analyst noted.

However, whether or not the platform is able to break down traditional engineering siloes to enable more efficient DevOps management, as Librato officials claim, remains to be seen, he said.

“Each wave of technology has promised to do this, to remove the inherent finger pointing, and each has always made the same claim to break down organizational firewalls,” Governor said. “DevOps hasn’t boiled the ocean in this manner yet, but there are some unique opportunities; unlike previous waves, where automation was intended to replace humans, this is now about augmenting people, and we are seeing that teams are choosing new tools from which developers can truly derive value.”

August 10, 2015