devopsy.kinsta.cloud

Tag: testing

DevOps & Continuous Change

A remark by a colleague while waiting for the coffee machine to complete its cycle started my train of thought. “Should we have multiple minor releases or just do a few major ones in a year?”

In large organizations, due to many factors, the turnaround time for a single successful release is quite extensive; but if the conceptual change which we are talking about bringing whilst using the #DEVOPS methodology would not only reduce these long cycles but deliver quicker and quality software.

I have already discussed in my earlier article “DEVOPS: Getting our organizations aligned” on the traditional approach and how the combination of this two (DEV & OPS) has been proven beneficial in various organizations and how many are still hesitant in embarking on that path.

Continuous Change which encompasses both Continuous Integration and Delivery is paramount to achieving this objective. To attain the objective of delivering rapid and consistent value to the stakeholders regularly, we have to have Continuous Change happening.

I am sure we have already moved out from the Mainframe era to a more Distributed systems state. But unfortunately we have yet to let go of that culture where we pile up components on a single quasi Distributed Application, instead of creating smaller independent, yet interdependent systems.

So, back to the same question; shall we go with a few Dinosaur sized, labored releases or make multiple short and swifter runs? In my opinion, we should strive towards the leaner model. Easier said than done you say! Agree. If I have to, then I would go about it this way:

In an environment having a suite of Distributed Applications, start with Analyzing the components which make up the Major release, List out the changes (fixes and new features) which are planned, asses functional and technological impact, Identify dependencies, upstream/downstream connectivity’s, architectural deficiencies, list out the identified subset of components or group of components by functionality, which can go as an independent release.

I am resisting the urge to say ‘standalone’ as with the current set of complex intervened collection of applications, it would be an incorrect statement. It would rather be a subset of components which can be independently upgraded. Now the big task is to review and decide if this subset or group would add functional value to the overall suite of applications without the rest of components going in.

Once identified, the success of this release working as expected would be in the extensive testing (Unit/ Regression/Performance) of these components, and as we gain confidence in this split and deploy procedure, we can start with, prioritizing and scheduling critical must have features or bug fixes which also should be backward compatible with other components lagging behind. We can even come up with scenarios where these components are not just getting updated frequently, but would also have multiple independent tracks. Of course it goes without saying the testing and signoff process is strictly followed albeit in a shorter window.

Deployments to production have always been a source of heartburn to the OPS teams, we have often seen finger pointing and firefights to contain change fails. In order to maintain stability and maximize availability, stricter controls and reduced deployment windows are put in place. In promoting frequent and smaller component level changes, OPS will also be in control of what’s going in, and smaller component level releases can be rolled back quickly within the Green-Zone window if the change is not working as expected. Though I would categorize this rollback, as a process fail than a deployment fail, and should be reviewed very seriously with right mitigation plan and lessons learnt retrospection, Continuous learning after every successful or failed sprint ensures competent and confident releases.

Another major factor impacting multiple runs of Production releases is the long Green-Zone windows, by moving to the quicker and leaner deployments cycles, these requests can be substantially reduced, and can eventually aim to reach a stage where changes are deployed online without extensive downtime.

Similarly, frequent maintenance activities when there are no planned deployments, effectively has the same risk on environment unavailability. Analyzing and moving to a state where these activities are conducted online without a Green-Zone would alleviate the maintenance downtimes and ensure seamless availability, adding stakeholder value.

I have not touched on the Continuous Integration or the Automation of the deployment process for obvious reasons, we can’t achieve the objective of Continuous Change without having the Integration process in place where incremental changes to code are build into a package swiftly and seamlessly, even a sanity or basic testing might be integrated to ensure the quality of the builds. Same goes for automating the deployments, develop & integrate tools required for rapid deployments. We have seen instances where automating a single activity or revisiting a process has created substantial time savings.

While it’s easy to propose changes to the Release Process or advocate rapid deployment cycles, we have to ensure adequate checks and balances are in place, compliance adherence and audit trails are created.

Now to the most important measure of this article, Cost Savings!

Proponents of the Dinosaur Releases would argue that the common activities which entail a single release like the builds, Testing, Release Management, OPS availability or reduced GZ would cut costs and multiple runs to PROD would logically increase costs.

But it’s definitely the other way around, we wouldn’t need to feed the Dino anymore, it will be a cost worthy rapid delivery with optimized resources, a leaner process where DEV and OPS are working together to ensure quality value added deliverables are made available to business faster.. This cultural change and collaborative effort also creates new vistas of growth and provides a perfect platform to excel. The cost benefit analysis would eventually be in favor of Continuous Delivery.

Opportunity cost is another variable which has not been considered; an innovation or a new feature available to customers quickly with reduced time to market cycle will provide the much needed edge in these competitive times. I recall some conversations where a few must have’s were pushed out due to various reasons, frustrating the business.

DEVOPS was never meant to replace the traditional approach to software development; rather it is an efficient use of the existing resources within a collaborative model to rapidly deliver quality software products. A major cultural shift in our approach is required and to reach that goal we need to embrace the shades of grays, many a times over 50 (sorry, couldn’t resist) when it’s not truly black or white.

Thank you for the time.

Visit my blog http://askhurram.com/

Read On, Live On.

February 24, 2015
Functional Testing for IoT

The Internet of Things (IoT) is becoming a bit more than a fun three letter acronym. And while internet connected devices have been around for a while, turning it into a practice is an important evolution. It allows the establishment of best practices, better integration into standard software development, and thus more robust applications, sometimes too robust. And because software drives everything, and software quality is key. How can you ensure quality in all those connected devices?

(more…)

February 24, 2015
From QA to Continuous Testing

My prior blog “DevOps – It’s All About Continuous Testing” posits that QA testers don’t need to fear DevOps because continuous testing QA is a central part of DevOps infrastructures that are implemented according to best practices. That is not to say however that continuous testing is precisely the same as traditional QA testing or that QA test engineers can rest on their laurels in the new world of DevOps and continuous testing.

Traditional QA testing is a serial process that constitutes a distinct phase that follows the development phase and is conducted by an SQA test team “independent” of the development team. The idea was clearly described by Glenford Myers in his seminal books “Software Reliability – Principles and Practices” and “The Art of Software Testing” published in 1976 and 1979 respectively. According to Myers “4th Axiom [of testing] “It is impossible to test your own program”. Myers felt programmers shouldn’t do any testing, even unit testing. While the application of this axiom to unit testing was not followed by most practitioners in the software industry, the idea of independent software functional and system level testing has been a widely accepted practice for many years. The underlying rationale is that it is “impossible” for the same person (the developer) to objectively test his/her own code due to the inherent bias to want the code to work, while a good QA tester must have the opposite attitude to want to break it in order to uncover bugs in the code. If you follow this logic only “unusual” developers would be able to perform both roles successfully and therefore building a team based on this rare personal quality is essentially not practical.

Despite the obvious logic of “independent QA test” concept, many very complex software projects without an independent QA test team resulted in extremely high quality products. There are many high quality open source software products that had no independent test organization behind each release version. Meanwhile there are countless examples of software projects that had massive independent test resources that resulted in products that failed miserably. How can this be? Perhaps it is not “who” is doing the test that matters but rather it is the quality and frequency of the testing itself that is important.

Continuous testing, with its fundamental concept of “fail early and fail often”, is the new QA test method of choice. Sure, testing is still essential and QA test experts that are most skilled in the best practices of test design are valued because they will produce the best quality products. Years of software engineering experiences have demonstrated that developers can learn these skills if they are taught and even “normal” developers can do a great job of testing. Instead of QA testing “after development”, testing is conducted continuously as the software is developed, build-by-build, feature-by-feature. Obviously it is better to find bugs earlier rather than later. So why didn’t we always have continuous testing as part of development processes?

Several industry developments converged to make continuous testing the winning method for software engineering quality assurance. Numerous “modern” test methods such as agile testing, test driven development, behavior test methods, model based testing, and automated regression testing, to name a few, describe how quality software can be accomplished without the need for an independent QA test team. The advent of Agile development processes and DevOps infrastructures, in which development build-test cycles are accelerated to minutes or hours instead of days or weeks, has accelerated the development and adoption of automation and orchestration tools and methods. There is simply no other way to get a reasonable amount of testing done on each build-test cycle unless the tests themselves and the environment for testing are supported by orchestration tools and reliable regression tests are systematically automated as the product is developed.

Meanwhile virtualization technologies have enabled a wide variety of product test topologies to be stood up on demand at a speed compatible with the rapid build test cycles and yet not require expensive dedicated equipment configurations. It is this combination of improved test methods, Agile processes, continuous integration speed afforded by automation and orchestration tools, and low-cost test dynamically configurable infrastructures afforded by virtualization that has resulted in DevOps.

Continuous testing is “the new QA.”

At Spirent we think testing has a bright future in DevOps. You can read more about our views at spirent.com/solutions/devops

February 19, 2015
DevOps – It’s all about continuous testing !

I loved reading Steve Wilson’s blog post “Don’t Fear Testing Team. DevOps is Here!” https://devopsy.kinsta.cloud/blogs/dont-fear-testing-team-devops-is-here/ The idea that DevOps somehow reduces the need for testing certainly puts fear in the hearts of QA and test teams who may envision the demise of product quality, reduced test resources and even their very livelihood. I understand the sentiment of the idea about testing going away, but this is misleading.

Testing is important to DevOps. Prior to code integration to trunk, developers pre-flight test their code to make sure it doesn’t blow up the trunk. During the CI cycle and code commits from multiple developers integration testing is used to verify merges. During the continuous testing phase where nightly and weekend regression runs are automated for trunk and release branches it is testing that does the job of finding problems before the software release. During the development process the verdicts trends from testing are a primary measurement of progress. During the release deployment phase more testing verifies the release packages are ready for release. Rest assured test team, you are needed. What DevOps really does is bring testing into the mainstream of development processes, and avoids the problems created by having “big-bang” testing left to the end of the cycle, such as release delays, and quality issues.

Testing maturity is a key differentiator of best practices DevOps infrastructures. My experiences with DevOps infrastructures over the years indicate that testing maturity is a key differentiator of overall DevOps infrastructure maturity. Many organizations can automate their integrations, builds and delivery processes but still have trouble with the subtleness of test orchestration and automation. There is a vital role for testing architects and testing teams to offer their expertise for test design, test automation and test case development with DevOps. Whether the organization is using a test driven development methodology, behavior based test creation, model-based testing or whatever they use, testing is a vital part of the overall DevOps process — not only to ensure verify code changes work and integrate well — but to ensure the changes do not blow up the product. So clearly testing is still an integral part of product development and delivery.

At my company Spirent we think testing has a bright future in DevOps. You can read more about our views at http://www.spirent.com/solutions/devops

February 2, 2015
Crowdsourced Software Testing

As a developer or technical architect, there is no better feeling than taking pride in giving a quality product or service result to your customers. Now, with computer trends going rapidly towards cloud services and mobile devices, regular company IT development teams are finding their testing capabilities more and more limiting. They are faced with faster releases with more complex products. How do they keep up?

(more…)

January 27, 2015
You Asked for Quality, You Got Alienated QA

The role of Quality Assurance (QA) is creating a bit of a buzz these days, because even as teams and budgets are shrinking, the demand for better quality is increasing. The chilling repercussions of service disruptions caused by software flaws warrants interest even at the executive level: “Nobody’s madder than me about the fact that the website isn’t working as well as it should,” said President Obama when HealthCare.gov went down last year due to software defects.

(more…)

November 24, 2014
The Dark Side of Infrastructure as Code

We spend a lot of time extolling the virtues of infrastructure as code. All of it is true. Treating infrastructure as code can add great value in terms of its ability to promote consistent, predictable and repeatable results during the application deployment process. One of the ways in which infrastructure is treated as code is through the use of programmability.

Programmability is an integral component in software defined architectures and a part of the DevOps approach is automation through integration with traditionally development lifecycle focused tools and technologies. APIs, app templates and the ability to programmatically interact with the data path are enabling rapid provisioning of services in the network. That’s important to the business. Improving service velocity in the network is critical to achieving faster time to market and enabling the operational scale necessary to meet growing pressure on networks arising from the growth of applications driven by cloud, mobile and the Internet of Things.

Speeds and feeds of processes are now as important to gain a competitive advantage as speeds and feeds of packets.

Those who see DevOps and SDN as having a strategic impact on their organization view programmability as having even greater importance.

But that does not mean that treating infrastructure as code doesn’t have its dark side. Because adopting a programmatic approach to anything means you get both the benefits and the risks.

I happened to be cogitating on this topic when I stumbled across a post reminding us of the importance of code reviews. The content was so-so, but the premise was spot on: code reviews are a must for those embracing the notion of treating their infrastructure as code.

Why is that (or should it be) an imperative?

Consider some stats from InitialState, “a technology startup helping engineers capture and understand product data. Our cloud-based log visualization-as-a-service platform helps engineers make sense of the exploding number of logs being produced by “smart”, complex products and applications.”

According to their data, developers have to find and fix 100 bugs per 1000 lines of code. And it takes 30 times longer to fix one of those bugs than to write one line of code. Overall, developers end up spending 75% of their time (about 1500 hours per year) debugging code.

Now spend a moment thinking about what that means when you start treating infrastructure as code, writing scripts to automate provisioning and integrating with other systems to orchestrate deployment processes.

I won’t lie, code reviews can be tedious. But the first time someone points out an error, particularly one that could have resulted in a security breach or an undesirable service outage, the pain is more than made up for. In the long run, code reviews are beneficial to eliminating disruptive errors. A SmartBear survey in 2012 found that “70% of organizations indicated that code reviews are very important” and “Those that do code reviews clearly have a greater level of satisfaction in the overall quality of their software – they were twice as likely to indicate that they were very satisfied with their software quality.”

In other words, code reviews are an important part of the software development lifecycle (SDLC). Which means it should be a part of the software deployment lifecycle when it leverages programmability to treat infrastructure as code.

Therefore it is important in the early stages of DevOps (or SDN if you’re going that way) adoption that you put into place processes to check (and perhaps double check) the scripts, templates and integrations you’re using to automate application and service deployment along the entire operational chain – from the app to the network.

October 7, 2014
BlazeMeter Adds New Relic Insights to Drive Testing

BlazeMeter a performance / load testing as a self-service platform for mobile apps, websites, and APIs lets developers test mobile and API backend performance and view KPIs in a dashboard. BlazeMeter tools simplify testing, reducing the time to test while running many tests per day. Customers use BlazeMeter as a self-serve and on-demand service.

“All they need is a web browser to generate massive loads from various geographical locations and view the test results in real-time,” says Alon Girmonsky, Founder & CEO, BlazeMeter. BlazeMeter lets developers simulate any size and configuration of device (Android, iOS, etc) and network (AT&T, Verizon, etc) user groups from any country using their app or browsing their website in order to test the backend.

Developers can run simulations from minutes to hours in length, gaining real-time performance intel. Developers can test and fix the backend repeatedly until they are satisfied they have a version that is ready for release. Developers can then test continuously throughout the product lifecycle. Developers can script automated tests, test what-if scenarios, and examine user interactions with their backend technologies. BlazeMeter can scale simulations to satisfy very large customers who need to test for the kinds of loads that come with up to a million users.

Irdeto Perfects Multiscreen With BlazeMeter

Irdeto (http://www.irdeto.com/), a Naspers (http://www.naspers.com/) subsidiary and solutions provider for Pay-TV operators needed to performance test its Multiscreen mobile app, which enables TV viewing on a variety of devices and screens online. The Netherlands-based company had to examine broad simultaneous use of its app across many systems and complex, variegated usage patterns. Irdeto selected BlazeMeter to load test its Multiscreen technology backend while scaling user group size and changing test scenarios in real-time.

Irdeto must enable a seamless, low latency UX across all devices while enabling audiences to access media content on-demand on new mobile devices as they appear in the marketplace. To do this, Irdeto partnered with BlazeMeter for additional support to test the app against “real world” scenarios, simulated by JMeter scripts (Apache JMeter http://jmeter.apache.org/). The JMeter scripts had to be created and entered into the system, leaving a lapse in an otherwise automated process. Tested scenarios included surfing, buying, and running premium content and interacting with Google Wallet (https://www.google.com/wallet/). Irdeto tested ramp up using JMeter’s Ultimate Thread Group (http://jmeter-plugins.org/wiki/UltimateThreadGroup/).

Applying BlazeMeter and JMeter tools, Irdeto pinpointed and opened bottlenecks based on results from scaling tests to hundreds of thousands of users running a labyrinth of usage patterns. BlazeMeter enabled Irdeto to alter test schemes “on the fly” including adjusting weighting for one user activity over another. Irdeto was able to use BlazeMeter to drill down to less apparent bottlenecks and then resolve those as well.

Speaking The Language of DevOps

BlazeMeter is partnering with New Relic Insights (http://newrelic.com/insights) to add functionality to BlazeMeter’s testing and dashboard including the ability to query application data after testing using natural language questions, query data sets over time, audit test activity, and answer questions that relate to data coming from BlazeMeter and NewRelic combined. “New Relic Insights enables developers to create new dashboards that run in real-time. Insights can add five more dashboards to the 20 that BlazeMeter provides,” says Girmonsky.

“BlazeMeter and New Relic Insights speak the language of DevOps, the language of continuous development and improvement by automating the development process, removing bugs, and reducing the time to project completion,” says Girmonsky. Developers can use these tools to follow performance trending, to see whether and where performance is improving or degrading, and why.

The partnership helps operations, too. “The ops guy needs to see that all apps are up in production and not affecting each other or the overall user experience. These tools enable regression tests to do just that with every new software release,” says Girmonsky. The ops guy can use his New Relic Insights dashboard to build trends from continuous testing and improve response times as they discover issues using big data analytics.

According to Girmonsky, enterprises should expect ROI in time, cost, and person-hours savings using these tools. “We didn’t conduct these specific kinds of analyses, however, prior to the availability of these tools enterprises had to export data into Excel and do the calculations that way. Now, they can run automated calculations in minutes,” says Girmonsky. According to Girmonsky’s estimates, BlazeMeter offers the enterprise the equivalent of building its own infrastructure to do testing, which would require a commitment of about two professionals for a full year to build that out.

BlazeMeter alternatives include Flood.io (https://flood.io/), Load Impact (http://loadimpact.com), and HP LoadRunner (http://www8.hp.com/us/en/software-solutions/loadrunner-load-testing/). There are several other load testing platforms.

[*Irdeto data compiled from Irdeto / BlazeMeter customer reference. New Relic was not a participant in that case study.]

September 30, 2014
Testing in the Agile Age

The Agile Age
For a business to survive or, dare I say, thrive in the “Agile Age”, it must adopt agile methodologies. The term agile may ring some bells with those of you from the product lifecycle field. But when I say agile, I’m not referring to product lifecycle terms like Scrum or Kanban. I’m talking about agility throughout the organization, starting from the CEO, through marketing, sales, R&D and, of course, to IT operations.

To compete in today’s environment, you must act and react fast! If you don’t, your competition will beat you to it. Today, the barriers to compete are minimal and the only way to defend one’s stature is by innovating in short iterations.

Agility facilitates survival in a highly competitive business environment. And it dictates a different DNA. One that can move fast, innovate and work in ridiculously short iterations.

This new DNA demands an IT infrastructure that supports agility. What was so right in the waterfall-age is sometimes so wrong in the agile-age. While the right IT infrastructure can enhance agility, the wrong infrastructure will add friction and cause failures.

Five years ago, the vast majority of IT infrastructures were dominated by giants like HP, IBM, Compuware, BMC, etc. Today, you’re more likely to see agile, streamlined companies making waves thanks to their ease of use and excellent ROI for their customers. Companies like Heroku, New Relic and BlazeMeter are fast becoming the new household names in the technology of today and tomorrow.

Waterfall Vs. Agile Approaches

A product lifecycle is built out of iterations. As adopting agile methodologies mean you have dozens or hundreds of extremely short iterations every year, the waterfall development cycle is no longer adequate. A typical iteration starts with product marketing specifications, then goes through design, development, QA and finally deployment. During the Waterfall age, this process would take many months to complete. However, in the Agile age, a typicaliteration can take anything from a few days for a minor release to a few weeks for a major release.

Testing in the Agile Age

An organization that relies on web or mobile applications to deliver services, benefits and features needs to introduce agility into its product development lifecycle. In order to achieve agility, we need to build applications differently than we did in the past. Bugs, bad user experience and downtime come with a far greater cost than they used to.

The Agile Age is all About the Continuous Delivery Process

By moving from Waterfall to Agile, the entire testing process has changed significantly. There can no longer be manual handoffs between development and QA in order to perform testing. There is simply no time as iterations are so short. Testing needs to be automated and performed as part of the cycle and not as the next phase. Today, testing is part of the continuous integration and deployment processes.

What is Being Tested and by Whom?

Today, testing is conducted by the entire technical team in all phases of the development lifecycle. Testing starts at the development stage by the developers, continues through the continuous delivery process all the way to post production monitoring of the application’s service level. While testing is performed throughout the product lifecycle, it is carried out by different teams with various skill sets: Dev, DevOps, Performance Engineering, QA, Ops and others. Some of these teams have a comprehensive understanding in the field of performance testing and some have close to zero understanding and little patience to learn. As these teams can no longer do handoffs to the QA team, they must carry out testing on their own. They need to leverage their skill sets and utilize tools that will enable them to perform these tests.

A Fantastic Story: The Life of the Agile-Age Test Script

The Agile-Age test script can be used by developers manually for troubleshooting, unit-testing and module-testing. Very quickly, the test is automated and run as part of the continuous integration cycle after each daily build. Prior to production, the test script can be used to run a load test. Once the release is ready for deployment, the test script is attached to the continuous deployment process as part of a Chef or a Puppet recipe to be performed as part of an automated deployment. The same test can also be used to monitor the application service 24/7 post production.

Open Source Vs. Commercial

If, in the past, the testing market was dominated by commercial tools such as Loadrunner and QTP, today it is obvious that the use of open source tools for testing such as Apache JMeter and Selenium is becoming increasingly common. Working with a commercial tool binds you to a certain company – making you dependent on its support team and feature roadmap. Open source tools today are far more up-to-date than many of the commercial tools and they give you freedom. With open source tools, you can find one that you like, download it, view a couple of YouTube videos and get going without speaking to anyone. In the past, no CEO would ever be fired for choosing IBM, today it’s the same for choosing open source. Not only do open source tools tend to be more up-to-date than commercial tools, they are also easily integrated with other open source tools and users can share their experiences. Most importantly, they can be altered and customized per use case.

Conclusions

For the past four years, we’ve been witnessing an IT revolution. Organizations have a new DNA – which some have started calling ‘DevOps’. This new DNA cannot survive without an IT infrastructure that supports agility. The right IT infrastructure can enhance the agility that DevOps requires, whereas the wrong infrastructure will create friction and ultimately cause failures. Testing has become an integral part of the development lifecycle. It needs to isolate bugs and bottlenecks – but it absolutely cannot stop or slow the release cycle.

Cloud computing and open source tools have become a great facilitator for agility. They replace many of the old techniques, saving a significant amount of time while enabling users to achieve more with less. Apache JMeter, Selenium and AWS allow the Agile-Age professional to do ten times more with ten times less. In this agile age, an investment in testing might just be the smartest investment an organization can make.

August 27, 2014
Change your thinking about Change Management

Change Management, let’s face it, it’s usually a checklist item and a CYA tool. But in the world of DevOps where change is part of the culture and processes. Change Management needs to be more than a cost center, and really a way to improve what you are doing. And there is no excuse not to.

(more…)

August 5, 2014