Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.
The ability to fail and recover quickly and frequently roll out new features are DevOps benefits any development team can appreciate. But when the team is building a security application that must protect users in real time, the ability to update and upgrade the product quickly isn’t just a nice-to-have—it’s a requirement. Security applications require rapid development to respond to security issues and must be tested continuously to ensure quality.
By design, Cyxtera’s AppGate SDP is a highly technical product. A comprehensive software-defined perimeter solution, AppGate SDP is designed to secure any application, on any platform, in any location. The product dynamically creates a one-to-one network connection between the user and the resources they need. These sophisticated capabilities make development and testing a challenge.
Testing AppGate requires a complex infrastructure that includes the network, user desktops, the application itself and a variety of scenarios. To ensure product quality, everything must be tested, including all upgrade processes on all versions of clients.
Previously, the AppGate team used an iterative approach to development, but the cycles were long. In some cases, developers had to wait up to two weeks for feedback on their build, making it difficult to maintain quality.
In an effort to reduce the time to feedback, the AppGate team adopted a DevOps approach. By automating testing, we sought to eliminate the barrier between developers and test results. Developers are most productive when you give them instant feedback. They are more eager to change their code when they’re excited about a feature they’ve just developed and want to see it succeed.
In addition to automating tests, we removed the QA team and created a single team of 20 developers who are responsible for coding, testing and implementing the feedback from tests and from customers. This organizational change created some cultural challenges.
Our developers needed a different attitude, and QA needed a different skill set to go from manual testing to development. That’s not a change you can do overnight, and sometimes it’s not even possible if a QA engineer has no ambitions to become a developer.
DevOps offers many advantages but can be challenging for organizations to embrace. For us, it took about a year of coding outside the product before developers began coding in the product itself. You need to invest time at the beginning to automate procedures, and that’s not always feasible in a normal project because development is all about quick gains and getting the product to market rapidly.
After about a year, our AppGate SDP team started to see speed and quality improvements in an order of magnitudes. The team of 20 has performed last year more than 2.5 million automated end-to-end product tests and created 560,000 virtual machines for testing alone. Those numbers are only possible with automation. If a security issue or a bug is identified in the field, the DevOps team can address it immediately. The team’s agility empowers them to respond to inbound customer requests as well.
An unforeseen benefit of moving to a DevOps model is the additional creativity the team enjoys. Test automation requires strict rules and processes, but instead of being perceived as a restriction, developers believe the structure gives them more freedom. Developers can write anything they want as long as the tests, code reviews, security checks, etc., are passed. If the tests weren’t automated, we’d have procedures in place for sign-offs, etc., to make sure everything is done properly. Now developers feel like they have more freedom, and that drives creativity and productivity.
The new model also provides a safety net, allowing the team to be creative without fear of failure because they know they can recover quickly. They get instant feedback on product quality and feedback for rapid failure. It’s better to fail fast instead of over several weeks, when it’s much harder to fix.
The AppGate team has the luxury of using the product they build on a daily basis. When developers request access to an application or system, AppGate looks at the user and device profile to enable secure, context-based access. This serves as another layer of testing, and basically makes every developer and employee a test user. It brings a lot of knowledge to the team on how the overall product works (so it goes way beyond the code they have developed), providing a better knowledge base for future improvements.
Automating tests enables you to scale at an unseen level, but you’re always testing the same use case repeatedly, and that’s not realistic. Using our own product all the time provides a lot of exploratory testing feedback. We’re testing a variety of traffic patterns and different ways of using the product, and we have immediate feedback. This feedback, combined with rapid development, empowers us to deliver a high-quality product.
At the recent Accelerate San Francisco 2018 conference, Tricentis was joined on stage by big-name customers, global partners and well-known quality and DevOps evangelists. This was the first time a U.S. date for Accelerate was added alongside its annual stop in Vienna, Austria. And, while the number of attendees was impressive—especially when Google IO and Microsoft Build were held the same week—it was the undeniable sense of urgency running through the audience that stood out, to me, as the week’s biggest takeaway.
Having covered countless software and technology conferences over the years, it’s always somewhat eye-rolling when the hosting vendor or conference organizer uses their time on stage poorly. I often see this done in one of two ways. They either, a) relentlessly try to sell you on a current transformation or shift that no one really buys into, or, b) only explain that there is a shift—one that everyone agrees on—but then offer no actionable information for those who will be impacted by it.
Agile, DevOps, digital transformation, continuous integration/delivery—all of which came up throughout Accelerate—have been the focal points of software conferences for years. These shows have given development and operations teams a wealth of information, case studies, lessons learned, best practices and inspiring keynotes, and they should be commended for it. But, today’s CIOs are frantically looking for the culprit behind how they’ve invested so much to make these processes and methodologies possible—yet, are still struggling to get high-quality, software out the door faster. Many of these CIOs are realizing the glaring oversight that they, and these conferences, have made.
Accelerate San Francisco’s manual testers, automation engineers, exploratory testers, test managers, QA directors and more didn’t have to be sold on the idea that testing has an enormous opportunity to solve the challenges their CIOs are facing today. Those who truly respect the power of “good” testing (an the artistry behind it) have always known what testers were capable of providing to the business. However, testing has been hamstrung by legacy technologies that prevented moving at the same pace of their development and operations colleagues.
But, today, those constraints have been removed, thanks to innovative test automation technologies and testers and QA professionals may not even have to sell their “new” worth to management. More and more often, the C-suite is recognizing software testing as an untapped resource for innovation, and many of Accelerate’s attendees could clearly sense the attention and expectations from their leadership on the horizon.
Day 1 of Accelerate was largely product-focused and featured a number of “discovery” and “deep dive” sessions led by the product owners of some of Tricentis Tosca’s most popular functionality. Continuous Testing, Load Testing, API Testing & Service Virtualization, SAP Testing were all showcased and demoed to existing customers and those evaluating whether Tricentis could assist them on their own unique journeys. Day 1 wrapped up with a product road map discussion given to a standing room-only crowd, and led by Tricentis chief product officer, Dr. Gerd Weishaar.
Day 2 kicked off with a bang with what was dubbed, “The Great Debate in Software Development.” Electric Cloud’s Anders Wallgren, Accenture’s Jeff Wilkinson and Tricentis founder Wolfgang Platz sat down to discuss who owns quality today, why that responsibility might change, and how testers and SDETs can be prepared for the future. There were some really outstanding—and oftentimes, conflicting—points made by each panelist, including:
“Development is a skillset; quality engineering is a mindset.”
“Testing is always reactive; quality engineering is proactive.”
“Quality is a frequency and depth of measurement game.”
Once we get the debate’s recording online, I’ll share the link in a future blog, and would love to keep the conversation going with anyone who is interested. There was so much that was presented that’s worth discussing and debating further.
Dr. Nicole Forsgren took the stage immediately after the debate and introduced her new book, “Accelerate,” (how could we not have invited her to a conference with the same name?) that she co-authored with fellow DevOps household names Gene Kim and Jez Humble. I’m constantly looking for ways to get testing more on the minds of DevOps community (because it makes zero sense for it to be so absent), so I was thrilled to learn that continuous testing has made it into DORA’s next “State of DevOps” report. Their annual survey that this report is focused on should be released soon, so look out for that.
Tricentis CEO Sandeep Johri then took the stage to give his thoughts on Tricentis’ role within the current state of software testing. Johri pointed out that while software becomes the key to success in many digital transformation initiatives, it also becomes a serious problem when software cannot be released at the speed that business demands. “Who cares if you can develop faster if you can’t also ship faster?” said, Johri. “You cannot ‘ship and pray’ in the enterprise.”
Being able to ship faster—with a sound, risk-based assessment of each release candidate—requires continuous testing, said Johri. And, to CIOs who historically have viewed testing as a bottleneck to innovation, I can see where this could be a hard sell. “You know that thing that you hate? The thing that costs a fortune, and takes forever, and doesn’t provide you with an easily discernible ROI? Do that way more, or, better yet, do it all the time.”
But, as Johri remarked—and I’d never actually heard someone word it this way—”Testing isn’t a bottleneck. Scripting is.” This is 100 percent accurate, and, as CIOs in greater number begin to realize this, continuous testing will become not just better understood, but a must-have in every organization. It’s the manual effort around testing that slows you down, stalls innovation and adds increased costs around the requirement for testers or developers to have coding expertise. These costs and bottlenecks are what CIOs want to eliminate. As awareness of modern test automation technologies that remove these constraints increases, Johri predicts that the solutions that can’t offer the same level of agility at scale will continue to fade from the market.
The next keynote of the day was given by Tricentis founder Wolfgang Platz. While Platz gave due-credit to what I’m calling the “A.I.-lephant in the room,” it was the advice to the Accelerate attendees of how to navigate the current transformation going on in world around them that I found most inspiring. Platz shared the current strategic priorities for CIOs in 2018, and it was remarkable how clear it was to everyone in the crowd where they could assist with meeting each one of them.
We hear all the time about lofty aspirations such as, “Quality is everyone’s responsibility,” and, “DevOps establishes that every team is working toward the same goal.” But, when Platz ran through CIO priorities such as digital transformation, customer experience, data analytics, moving faster, improving culture, scaling agile development and freeing up budget, eyes didn’t roll. People didn’t groan or walk out. It was clear that the entire room understood exactly what they were capable of by trading the limitations of yesterday for the capabilities of tomorrow.
Today’s CIOs have never been under more pressure to avoid disruption, and my biggest takeaway from Accelerate was that the testing community understands that accomplishing this monumental task—in every software release—is only possible with their involvement.
Let’s see how many testers continue to navigate through change introduced by others, and how many take the opportunity to drive new change themselves.
As development moves ever faster, continuous testing can help further reduce the risk of introducing errors
In IT circles, “risk management” implies information security. But in the business as a whole, risk management is a function that spans the organization. The risk that an application will negatively impact corporate image, sales or employee morale is very real in the age of automating everything.
Once properly identified, these types of risks can be mitigated with continuous testing processes. The ability to set up a test to prove the login system works as expected keeps users able to get to the application.
Increasing use of agile and DevOps offers the opportunity to receive rapid feedback. Among other things, that includes business (as opposed to InfoSec) risk management. Requirements from the business side can clearly delineate items that create risk for the business unit and the requirements to reach desired outcomes. Continuous testing can turn those requirements into tests that validate the application is not contributing to business risks.
After all, there are enough risks in the business environment: competition, suppliers, coordination of marketing, legal, etc. Making application development less of an issue is good news for business owners and IT alike.
By translating business risk management requirements into automated tests early in the development process, stability and risk aversion are built into the application from (or nearly from) inception. This offers stability and provable correctness in the application development cycle.
As the pace of development continues to accelerate, there is a risk of negatively impacting the user experience. Whether that negative impact be through failures of critical parts of the system or degradation of performance, it harms the application and the business it supports. Continuous testing can help alleviate fears by testing for these scenarios and pointing developers at fixing them early.
Test automation tools are becoming more prevalent, and the next step is to include business requirements designed to mitigate business risks at every step in the process. It is simple to say and test “valid users must be able to log in,” but as development progresses, if the login prompt takes more than 10 seconds to return, that adds the “in a timely manner” business requirement that is then refined by IT to a number, and a new test is run. Only by having the inputs from the business at each stage can such refinements be identified and implemented.
Interested in this topic? Check out the ebook here to hear from experts on balancing speed of development with risks to the business.
And consider it. The purpose of any given piece of software is to serve the business. Speed that introduces unwarranted risk to the business cycle is not the best answer. Continuous testing, and attendant business owner involvement, can mitigate those risks.
Mindless testing, whether it is manually or automatically executed, can cover only a very narrow scope of possible bugs: the ones you’ve already thought of. Exploratory testing however, endeavors to think outside that scope. Thoughtful testers look at a problem from all angles, weighing factors such as feature usage, psychology, user stories and more. Their goal is to chart the uncharted territory of an application.
The faster our development cycles become, the more thoughtful our testing should be. It’s only through techniques such as exploratory testing that we’ll be able to get there.
Continuous testing is the process of executing automated tests to obtain rapid feedback on the business risks associated with a software release. Where does that leave exploratory testing? It’s not automated, but it’s certainly critical for determining whether a release candidate has an acceptable level of risk.
Test automation is perfect for repeatedly checking whether incremental application changes break your existing functionality. However, where test automation falls short is in helping you determine whether new functionality truly meets expectations. Does it address the business needs behind the user story? Does it do so in a way that’s easy to use, resource-efficient, reliable and consistent with the rest of your application?
Exploratory testing promotes the creative, critical testing required to answer these questions. Obviously, it doesn’t make sense to repeat the same exploratory tests continuously across and beyond a sprint, but exploratory testing can be a continuous part of each delivery cycle.
Exploratory Testing in Action
Here are a few ways teams embed exploratory testing throughout their process.
Perform Ad Hoc Exploratory Testing as Each User Story is Implemented
This is the exploratory testing equivalent of peer code review. When a developer completes a user story, they sit down with a tester. The tester starts testing while providing a running commentary on what they are doing and why. Next, the developer takes control, explaining how they would test the software given their knowledge of the implementation details and challenges. The developer gains a user- and business-focused perspective of the functionality, and the tester learns about the inherent technical risks.
Another tactic is to have the developer and a tester separately test the same feature simultaneously, then discuss their findings at the end of the session. Often, this turns testing into a competition, where each participant tries to uncover the most or “best” issues in the allotted time.
Align Exploratory Testing Sessions with Full Regression Testing
It’s simply not possible to perform exploratory testing or full regression testing on every code commit. That’s what smoke testing is for. Instead, many teams run full regression testing and session-based exploratory testing in parallel a few times per week, whenever they’ve implemented new functionality that a user could feasibly exercise.
For optimal results, these sessions should be lightly planned and tightly timeboxed, include diverse perspectives and really take the six thinking hats theory seriously.
Host Blitz Exploratory Sessions for Critical Functionality
The best way to uncover user experience issues before users do is to get a broad array of feedback prior to release. One way is to host “blitz” exploratory testing sessions. When you’re wrapping up work on critical new functionality, invite people from a variety of backgrounds and teams to participate in a short timeboxed session. Incentives can help drive participation, maximize results and make testing fun.
Using test automation to continuously check the integrity of existing functionality is certainly critical. However, if you’re not also making exploratory testing a continuous part of your process, how will you know if the new functionality meets expectations?
The goal of continuous testing is to understand whether a release candidate has an acceptable level of risk. Exploratory testing is perfectly suited for helping you answer that critical question.
Safety, security and controls are the foundations upon which 135-year-old ExxonMobil is built. As such, quality has become a key component in delivering software. The company is currently undergoing a shift from traditional project management organizations and support organizations to product teams, which has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.
Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”
To do this, ExxonMobil started with the basic idea—quality control manages risk—and then explored what is tracked today and whether quality is being measured. Another key activity was to remove the roadblock to continuous testing.
Ann Lewis from ExxonMobil discusses in this video how the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset, as well as the company’s view that continuous testing and deployment can only be accomplished through model-based testing and that managing quality reduces overall cost of ownership.
The full transcript of her discussion is below:
“At ExxonMobil, we have presence in six continents, so that means we get to deal with a lot of people. There’s over 72,000 employees plus another 20 percent contract staff, and that’s a problem. Guess what they want to do? Manual testing. Yeah, because they’re not gonna give it up. We have been on an adventure this last year that I’m gonna talk about and explain to you how we help train these 72,000 plus people that there’s a better way to do it.
“In addition to the numbers of people that we have, the upstream, I always … Well, someone referred to it about sticking a straw in the ground. Yeah, it was actually somebody from Tricentis. Yeah. I was just like, ‘There’s a little bit more to it than that.’ No, we do produce all of the product from under the ground. We get it out, and then we send it downstream to our refineries so they can do something with it like sell it. We also have a chemical company, that … I think right now are second or third to DuPont. We have a lot of product and a lot of things to do with it. That’s a problem, too. Because when it comes to the applications that test, that need to be tested, we have something like 5,000 applications that run our business all technologies. We’ll talk a little bit about that as we go through, too.
“The title is The Middle Child Rises. Why did I choose this? Because in the world of Dev and Ops, test is in the middle.
“We have a few other friends in the middle, and they are a little bit like a middle child, so we’ll talk about that. For today, your takeaways are that our organizational design has shifted. We’re going to explain why and how that happened. We will explain that Tosca is strategic for ExxonMobil and we will explain why that is.
“Tosca is an enabler to speedy delivery.
“You’ve heard about that this morning from many of our Tricentis friends, and it is true and I’m here to share that story. We now have within ExxonMobil, a quality philosophy, yeah, and it’s a lot more than just testing. Yes indeed, it is. We’re going to take a look at what we did and have a lot of fun stories in between.
“Being the middle child, you really didn’t get as much attention. How many are middle children, by the way? A few, okay. Do you have babies in here? Yeah, yeah, you kind of get away with everything, don’t you? The oldest, I have a favorite, I know. Oh, there’s another one out there of my fave, okay, cool.
“It’s very interesting in family relationships how these children get along. The oldest kind of sets the stage for everybody. That middle one comes along and it’s like, ‘Oh, it’s the second kid.’ The third one comes along and it’s like, ‘Oh, it’s the baby, it’s the baby,’ and the middle child just kind of sneaks through, okay? They don’t get quite as much attention. We’re going to talk about the middle child as they relate to testing.
“Typically, it’s the middle child that ended up with rules. That oldest child was all about, if you just tell them once, they usually would stop doing it. That’s the case I had. Then the second one was born, that turned out to be the baby, but I swear he’s got components of being the baby and the middle child. It was like, ‘Adam, we have rules in this household. Yes, we have rules.’ He was like, ‘I just don’t get the same respect as that older sister, Jessica.’
“Now within the corporate world, these middle children, you can think of them as being security and controls, data migration, data conversion. It’s like, ‘Oh, yeah, we forgot about you.’ Last but not least, testing. How many times has it been when suddenly that project shows up, ‘Oh, yeah, I think I might need to engage your team to help me organize my testing.’ Yes, yes, those of us that live in the testing world.
“The big deal right now, though, is that these middle children, our testing organizations, our quality assurance organizations, are becoming the key enabler, like we heard this morning, 50 percent of our problem is with testing. Oh, I could do that but I can’t get testers. I could do that but, ugh, we’ve got to do the testing. It takes so long.
“According to the World Quality Report, the reliance on manual testing is the top technical challenge in app development.
“How many of you knew about the World Quality Report? Yes? Yes? Good, good, good. We’re all the same kind of people, yeah. I like that.
“Businesses see test automation as a way to address one of their top challenges to delivering better software faster, by adopting Agile, DevOps and continuous delivery practices to shorten delivery cycles.” – Forrester
“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks.” – Gartner
“The need to support faster time to market with higher quality is driving the demand for effective functional test automation tools. This is why we’re here. Tricentis Tosca, it is a framework. Yay! It’s not scripting. Boo!
“Tricentis Tosca will address that top technical challenge of delivering software faster.
“The need to support faster time to market with higher quality. It’s higher quality in the form of risk coverage. That is what we do with Tricentis Tosca. That is what we do with Tricentis Tosca.
From a framework perspective, the last year and a half we have been on a mission to explain the difference on automation tools. We have gone from record and replay, to test automation from a framework perspective using scripting, such as with the Hewlett-Packard products and Selenium. It is programming, it takes developer time to do automation when you’re talking about some of these historical tools. We have now moved into model-based, and I hope everyone in here has been to at least some of the Tricentis’ demos on what model-based is. I’m not going to go into detail here, but I will tell you that it has taken our automation from a 40-hour event down to a 8 to 16-hour event. No joke.
“I have been able to take Tricentis Tosca and put it into non-developer, normal people’s hands.
“I actually sat down with one of the Tricentis’ architects and did automation myself. I said, ‘Oh my gosh, I haven’t touched a keyboard to do anything technical in so long,’ and I actually did it. That is key, that is significant. That has been the story that has been going on with ExxonMobil for the past year and a half.
“It has become strategic for us. It’s automated screen scanning, it’s lower maintenance, it’s targeted testing. I’m doing it in half of the time, if not less. It’s faster, it’s easier to support, it’s talking about prevention versus detection. We have created a road map.
“What you’re seeing in our colorful TFS diagrams on the top, yes, we are doing our monitoring in TFS. You’re seeing a migration and better put a re-engineering of our existing scripts into Tosca. Re-engineering because for the last two years that I’ve been in this job, people kept telling me, ‘Ann, did you know that those existing Hewlett-Packard scripts really don’t test anything?’ ‘Well, gee, I don’t think I knew that.’ ‘Well, gee willikers, let’s see if it’s really true.’ Lord, have mercy. Twenty percent of those Hewlett-Packard scripts that were driven by the business to automate didn’t test anything. Well okay, they tested something but it wasn’t worthwhile.
“All of these little blue dots on your right beyond the solid blue vertical line are things that don’t really matter because we didn’t do a risk analysis. When I talk about re-engineering, we’re not just taking what we had and push it into Tosca, which you can do, by the way, there’s a tool for that. It’s up to you to use it, I’m not going to say anything. All I’m telling you is all our experience has been it made no sense.
“What we have found is when we sit down with our subject matter experts and we had this really cool dialogue like, ‘What do you do all day long?’ ‘Oh, I use VA-01 and NM-16 all day long.’ I said, ‘Cool. And so what happens when one of them doesn’t work? Dead in the water. You’re in customer service, aren’t you?’ ‘Yeah.’ ‘How many sales? Millions of dollars of sales do you do every day?’ ‘Well … ‘ This is the user, I mean they are totally into explaining to us what they do. Through this dialogue, this interview, we find out that they’re going to cost us $1 million a day, an hour, I don’t know what the big number is but it’s a big number, when it comes to customer-facing transactions. Therefore, those transactions are the ones that we want to automate.
“Now if I find out, no offense to anybody in controllers, that at the end of the month they have a report that they need to generate, ‘I’ve got some time to fix that one, okay?’ ‘How often do you do this? Monthly? Okay, that’s a one. What happens if it doesn’t work?’ ‘Well, I just did last month’s report and … ‘ ‘That’s a one.’ Low priority, move on.
“We have been re-engineering all of our existing tests.
“This word crypt is an ugly word. I actually I know the person who named our existing automations. Critical Integrated Process Testing. We don’t say that word anymore, by the way.
“The middle child has risen. This is my team. When I joined the group in February of 2015, we had about 25 people. About 25 people. Within ExxonMobil, I think there might have been seven of us. Then, we had some offshore group that did all of those monthly executions of those Hewlett-Packard scripts.
“Today, I have Vulcar over automation, he’s based in Hamburg. I have Cathy in Houston for functional testing and Santa Juan for functional testing, basically split between the two regions. They’re what we call test leads. They do a lot of the coordination of our testing activities. I have Aaron in Houston for performance testing. I have Jay, he is Doctor Jay, he is the man that knows everything about testing and everybody loves Doctor Jay. We have Suzanne right there, one of my oldest children. I didn’t say that, did I? I meant oldest child. Anyway, the other thing that was really funny about these pictures, they kept saying, ‘Oh, I want mine to be skinny.’ I want mine to be really upfront. We had a whole dialog on how the pictures turned out. It was pretty fun. Guess what? There’s close to 150 people now. That’s a signal that over this past year and a half, two years of getting this team put together, the emphasis that ExxonMobil has put on quality. One of the ways to get to quality was through testing.
“We are everywhere. I’m going to go into a little bit more detail now about organization because it’s good to understand where you are located. I was talking about being on the six continents. We have now placed people in region, so that when a person has a question on, ‘I want to automate, should I automate?’ We’ve got people in those locations to help with that. You will see we are using architects.
“Within the Tricentis nomenclature, you have architects, automation specialists and automation engineers. It is a tiering of skillset.
“We have found that the architects become those go-to people, when it comes to, ‘Should I be automating this or should I not and how do I go about it?’ Yes, we are using some offshore resources in India. Yes, we have people in Auckland. Yes, Bangkok, and that’s actually one of our ExxonMobil business support centers in Bangkok. Then, you can see Houston, Vienna, New York and Curitiba, Brazil.
“Organizationally, what we have found is that when it comes to implementing automation and doing it in a … I’ll call it conscientious and disciplined way, you’ve got to have a structure. Don’t think that you can just, I’ll say, run into doing automation. There needs to be a structure upfront to help you manage this work, because once one customer comes, you’re going to get five more. The enthusiasm level that we have experienced is unbelievable. We’re almost on tilt. But through this concerted effort of having an organization to help support it through a governance and then program management. Again, you can see this partnership that we have with Tricentis. Highly recommended when it’s time to get started.
“Successful team composition. It’s typically one architect to one engineer to 10 specialists and then you ramp up or ramp down, based upon if you’re working on systems of engagement. The shiny coin fun stuff, mobile, gooey, or your systems of record. They’re a little bit of a worn coin, all right? I’m sure some of you have systems that are quite old. We have ones that are really old so our financial system is still in IMS in PO1. Yes, rock on! That’s what I grew up with. Yeah, give me some JCL, too.
“See, geek, we’re all geeks, I know. It’s like once you join the world of IT, you’re just like into it. Nobody really understands what you talk about, unless you’re with people that are like you. Uh-huh (affirmative), all in the family, totally all in the family.
“Okay. We also have an immersion program. We have, again, the process of getting somebody into you’re approved, we think you can handle the work, all right? Going through automation specialists and all the way up to graduation. Again, working with Tricentis, we have come up with again, a disciplined process. You need this when you get started because they will come in out of the woodwork, I want to automate. I talked to you two years ago about automating those processes at the refinery and you said that no machine could do what you could do. I don’t say that, but that’s what I’m thinking when they come to me two years later with, ‘I think I want to do automation.’ I’m like, ‘Cool, let’s get into it.’
“We have a quality manifesto. Yes, we do. We will create forward-looking, value for our customers and for our organizations through quality approaches. Quality engineering is a team responsibility. I’ve got a great team, but we’re just here. We’re the messengers, we’re the trainers, we’re the coaches. Everybody that we work to, we talk about quality.
“How did you get to work today? Did you think about a safe way to get to work today with all of the rain? Yes. That’s a quality decision right there. Little things that we teach when we’re starting to push the quality manifesto out. You have to understand business function risk. Again, what do you do all day? What happens if it doesn’t work? Oh, really bad things happen, or maybe not so bad and a month in close, we’ve got a whole month to worry about it. Yeah, we can always close the books this month of last months’ numbers. I actually have a degree in accounting, I can talk about them.
“Including quality checks throughout the delivery life cycle. Quality gates. That’s another item that we have built into our system delivery life cycle. Starting to talk about at the moment requirements are written down, let’s start talking about what you’re going to test, based upon those requirements. We have built these quality gates, these questions throughout system delivery life cycle.
“What has happened? Our organizational design has shifted.
“Tosca is strategic for ExxonMobil. Tosca is an enabler to speedy delivery.
“Test automation brings value. We’re actually connecting the automation into total cost of ownership, as a way to reduce application total cost of ownership. We have a quality philosophy now and we’re just pretty darn excited about what we did. It took the rise of the middle child in order to make it happen. Thank you so much.”
The continuous demand market for flawless applications is being addressed through advanced software development techniques along with continuous delivery. As such, many organizations adopted the shift-left approach and agile best practices—which suggests the motto, “test early, test often,”—through which they get to imbibe QA practices and testing early in the software development pipeline. That also includes continuous testing for many organizations.
Continuous Testing in DevOps
Continuous testing today is the most preferred practice by leading software companies. Continuous testing implies testing through the whole software development pipeline, from product design to product release, by integrating the complete product cycle. Even after the product release, companies test to troubleshoot customer complaints to further improve the product.
Continuous testing in DevOps involves encouraging the development engineers and operations to practice testing as well, and also makes them a part of the whole cycle instead of being a separate team. Continuous testing involves testing not just from an earlier stage but also through each and every single process. Its goal is to automate almost all the processes as possible.
A Templated Approach to Continuous Testing in DevOps
A perfect approach to continuous testing in DevOps should be a right balance between manual and automation testing, and to achieve it, organizations must practice virtualization to plan out the whole product development pipeline successfully. Many organizations are not yet ready to risk it all with continuous testing, as big errors are still a possibility if they don’t really know how to execute it. But, errors in the continuous testing method can be easily rectified as it is practiced through each process.
Errors are common when it comes to product development; hence, the right set of tools to automate testing can be critical to finding and rectifying errors, since manual testing can miss certain errors. Manual testing also takes a considerable amount of time, which is a huge drawback. A well-planned but carefully structured templated approach can be practiced by companies to make a full transformation toward continuous testing.
Continuous testing is achieved through imbibing multiple use cases over certain parameters to determine the strength of this methodology. Some of those parameters that would prove to be a sure-fire way of determining whether continuous testing is worth it are distribution, technology, scale and load, environments and security and release.
Checking the Parameters
The parameters to completely check a dimension such as continuous testing can be done to create readings based on elimination and weighted scoring to derive results from a use-case scenario. Let us see a brief application of running a hosted public-facing service through these parameters:
Distribution – The distribution areas and the geographical location of this product are considered.
Scale and Load – The number of test cases (3,500 automated cases in this scenario, by way of an example), the suites that are used based on the browsers, cloud test platform and the numbers of users are considered in this category.
Environments and Security – Varied device uses, cross-browser testing, secured cloud environment and approval of releases are considered in this parameter.
Release – The number and frequency of release and the technology stack are considered in this parameter.
The tools that are used for this scenario are Selenium web driver, Selenium GRID, Jenkins, Sauce Labs, Sauce On-Dand Plugin.
The process through which checking the parameters is done is by triggering automated test cases from Bitbucket automation repository using Jenkins during deployments, and the execution of automated scripts in Sauce Lab cloud, as parallel threads are done by interfacing Sauce Lab Plugin with Jenkins.
The control mechanisms first include testing through smoke, regression and integrated automated tests execution. Then, the test evidence for all of the test cases is captured. Later, as the tests are completed and the screenshots which are captured by automated tests are validated, UAT sign-off is done.
Some parameters for test automation framework are compared within Selenium and tests completed over various categories, their scores are compared and then they are passed onto the test environment. Some suite, regression suite and integration test suites are carried out during Dev and QA stages, later when the changes from these two environments are deployed. Smoke suite is carried out during pre-production, editorial and the production stages. Finally, the proven results of these stages are derived successfully.
The proven results of this use-case scenario through these stages are:
Parallel testing visibly reduced regression hours from 150 hours to 15 hours.
Enhanced test coverage matrix addressed varied device and OS fragmentation, as well as browser combinations.
Quality gatekeeper for code promotion to production.
Reduced regression issues in production.
Conclusion
The example of going through such a use-case scenario helps us better understand how continuous testing can be integrated gradually into a product development pipeline through careful observation, planning and deployment through such parameters.
Engrafting continuous testing in a DevOps eEnvironment needs careful execution to avoid errors and future defects. And with the right set of tools and a perfect execution, it can be achieved to successfully experience faster time to market and reduction in time and cost.
For more information on how a templated approach can be applied in different scenarios, watch this webinar.
It should be no surprise to anyone working in software today that delivery speed is the name of the game. Irrespective of business sectors or B2B/B2C focus, global competition in application development and delivery has flooded the market. Due to the increased number of options in the marketplace, users expect an excellent experience from your application or site—and if you cannot meet expectations, they’ll find it elsewhere.
A company’s ability to be highly responsive to (if not proactive about) customer’s needs is also valuable for adoption and branding. Established companies understand that delivering high quality in their products and services is core to brand value.
This shift isn’t always easy for testers. Moving to a faster pace of work can be unsettling when your job is all about assessing the perceived quality of the software. Testers like to be cautious and check everything twice. The current work environment has made testers more germane than ever in software and IT organizations.
Testers help build confidence in the quality of an application, and this perception of quality is what typically drives user adoption. If two applications have more or less the same functionality but one is faster, more reliable and has a better UI and design, you can guess which one will be more popular.
Setting the Foundation
To thrive in this new world, testers must learn to prioritize and focus on the most important needs of the team’s delivery goals versus testing every last mouse click. Testers will need to quickly incorporate feedback from multiple places, and work well with stakeholders across the development organization and the business. Efficiency is imperative, and that means increasing the use of tools whenever possible. The following framework to develop a world-class testing organization when your company is moving faster all the time can make the transition a little easier.
Understand business goals. While this may sound obvious, testers haven’t always had to think about the bigger picture. With more competition from everywhere, everyone on the development team should have a solid understanding of customer needs, business goals such as to drive new revenue or grow customer loyalty, and how the competition stacks up for the solution in development.
Define a testing strategy. This exercise begins with understanding the business goals as they relate to the product or project and, at a more granular level, the goals for each sprint. The testing strategy should include a description of the overall project, priorities for testing mapped out into phases, timelines, testing approach, tools and methods to use and how progress will be measured.
Develop a risk management plan. What will happen if those business goals are not achieved? For instance, the business stakeholder might say the project must go live in four months with new functionality. But is this an arbitrary goal or one that is embedded in an actual customer commitment? What will be the financial impact if the deadline slips? Which enhancements are critical and which are nice to have from the user perspective? Once those particulars are documented (or discussed), the testing team can create the appropriate plan, prioritize and determine what resources are needed to execute it.
Resources. Testing teams with mandates to move faster will need support in the form of new tools, training and even new hires. Software that testers (as well as others) use can include DevOps and CI/CD tools, ALM tools, reporting and analytics, UI and API testing tools, systems for recording and logging manual testing, tools for usability feedback and more. This can get overwhelming for startups and other companies that don’t have the budget to purchase new software. Fortunately, there are many credible open source tools that can fit the bill. This reality is also why development teams must understand well enough what customers really want so they can prioritize on which features must be perfect and which areas of the application could meet a lower bar of quality.
A Word of Caution
Testers are well-equipped to make all the changes required for the faster pace of modern software development. This shift has been underway for a few years now, and many testers are embracing the change and the elevation of their role on engineering and IT teams. However, as testing relies on more automated tools, there are a few things to consider:
Avoid tools mania. Testers, like many technical people, love to try out the newest toys, but it’s important to justify a tool’s adoption before acquiring it. With too many tools, there’s just too much noise and complexity. That will counteract any speed and efficiency gains that automation supplies.
Automation tools do not replace testing. It’s not necessary nor helpful to automate every testing task. Testers should view automation tools as enablers, not a replacement for the function. Automation is wonderful for reducing effort and giving testers time to work on value-added activities that can boost the overall user experience and usefulness of an application. Be sure to baseline test objectives before recommending an automation solution.
Institute process around automation. Ensure you have defined a proper process in sharing artifacts that are produced by your automation tools. Artifacts such as scripts must be versioned with proper comments and stored in a central repository instead of on individuals’ machines. Defining and enforcing good development practices such as check-in/check-out process, scripting style guides and object organization with automation will ensure a higher return on effort in the long run.
In a few years, this point may be moot. But there are still many testers coming into new roles from the waterfall mindset who may struggle in the transition.
Testers once lived simply in the silo of sluggish procedures. They knew exactly when they would be called upon for action and how they would complete a task. Now, though, testers must put on their business assurance hats and work side by side with developers, operations managers and others early in the process. They will have to learn new ways to test, such as with behavior-driven development and test-driven development. They need to get more technical, given the push toward test automation.
What hasn’t changed though is the tester’s quest for quality in software. Great testers will add the most value when they can be proactive about alerting others to brewing problems before they present themselves to the user.
According to Forrester, 64 percent of businesses are dissatisfied with the time it takes to release new features for customers. Brand loyalty has given way to customer loyalty; to drive it, organizations must focus on customers and their experiences using their products and services.As a result, organizations of all sizes need to make customer experience their highest priority.
To many business leaders, the speed of rolling out new updates is the most important part of satisfying customer expectations, maintaining customer loyalty and staying ahead of the competition. At the same time, user expectations for fast and reliable digital experiences across platforms continue to increase. As a result, development teams are pushing out service updates faster than ever before while trying to minimize any degradation of the experience as they move at a quickened pace.
But with heightened and persistent focus on rollout speed, it is easy to forget that the overall user digital experience is critical to maintaining customer loyalty. Businesses must integrate their focus on performance testing with their DevOps workflows to make sure speedy product rollouts don’t come at the expense of customer experience. So how should businesses ensure strong delivery processes to both improve customer experiences and drive efficiency in the application development and delivery pipeline?
Performance Starts with Constant Measurement in Unit Testing
DevOps, as does Agile, breaks down testing into smaller chunks. Each new rollout has an iterative impact on the application or website. Whether good or bad, the results stack up. Customers may demand certain new features, but user experience will suffer if each update that meets that demand slows down the application more and more over time.
Testing these units is very important and setting performance SLAs per unit per build is the first step to a performance culture. Organizations with a mature performance culture will turn back code if it doesn’t meet their performance SLA, keeping it from ever reaching QA.
This is Not an Either/Or — It is an And
DevOps teams are very good at rolling out rapid iterations of products and features, but without careful consideration, the pace of those updates can build momentum for digital experience failure. There are false positives in DevOps testing. With unit tests constantly passing, it’s easy to lose sight of the bigger picture. It’s not the fault of the DevOps engineers or QA; they are working within an established process that is designed to sniff out errors on a per-build basis. But what happens when you put it all together? Organizations start to forget about end-to-end testing and can see failures when the application is stressed by heavy demand, when it really matters during critical business events like Black Friday, live sporting/news events, product launches, or changes in financial markets. Teams implementing DevOps-style testing must remember to test all the “units” working together end to end. This needs to be part of the DevOps workflow but is often being forgotten.
What You Can’t See Can Hurt You
Testing for performance in the development lab can help mitigate some of these performance issues before they get out in the wild. But it’s overly optimistic to think that you’ll catch everything. Delivery teams need to constantly maintain visibility in the real-world use of their applications with real-time real user monitoring to alert them of any degradation in the experience during or just following a release. A quick response can help mitigate a larger detrimental impact to the business, and the fidelity of real-user data gives assurance there aren’t any blind spots so you can fail fast, and shorten time to resolution. The web is a wild place, and users have sneaky ways of finding their way into and through your app that you may not foresee when building your test plan. You can use this real user data that you’ve collected while monitoring to create more realistic test plans that will truly replicate real-world browsing behavior.
This data can also inform the development team about the impact of design decisions on performance in each new update. They can prioritize which parts of the site or app to fix based on what would be most impactful to the business and weigh that against plans to build out new features aimed to enhance functionality. The key factor is an understanding of the business impact of performance and using that to guide and prioritize development resources to achieve the best possible results.
Adopting a methodology to monitor, optimize and validate ongoing enhancements to your application can help with the continuous improvement of user experiences and give your development efforts a customer focus. DevOps teams have a responsibility to the user experience and should consider how it correlates to business impact. Faster time to market is important, but as important is making sure the customer experience continuously improves as well. Uniting cloud delivery with DevOps workflows and keeping performance in mind will put your business on the path to ensuring both faster rollouts and superior user experiences.
Demand for faster product development and quick market release created a need for increased collaboration between Development (Dev) and Operations (Ops) teams, which eventually led to “DevOps.” With this, transition to DevOps has become one of the key points of discussion across many IT firms. This article gives you an overview of the demand and trend for organizations to transition to DevOps.
Rise in IT innovation and burgeoning digital revolution have led to the demand for faster development and quicker releases to stand out in the IT competitive scenario. This transition eventually led to a new culture that saw collaboration between two key IT teams, development and operations teams, which the industry calls DevOps.
DevOps established a new style of IT functioning, easing different phases of software process from product development to testing and final market release, all in a rapid and reliable way.
Why DevOps Transition?
Rapid delivery of independent software components and continuous integration of the same with production systems forms the key principle behind DevOps functioning.
DevOps is often confused with Agile methodology. Though they share similarities in providing improved IT performance through rapid development, DevOps stands apart from Agile because of its efficacy in the software deployment phase, ensuring a constant communication between the two teams. Thus, DevOps is often termed as a successor to Agile.
A collaborative approach, key for information exchange across the line, is one of the major factors driving firms to undergo DevOps implementation.
Though DevOps is well-known as a merger of development and operations teams, an effective DevOps model extends beyond that to other key functional areas of organizations.
Breaking the old trend of limiting individual teams to their respective work domains, DevOps brought in a scope for knowledge-sharing among them, wherein developers can extend themselves to testing and deployment and operations team members can upgrade their skill set, learning new languages, architectures and tools.
Increased Demand for DevOps Resource
Transition to DevOps holds a key demand in the fast-growing IT market, wherein managing with the existing infrastructure or resource plays an almost equal role as coming up with something new.
Greenfield projects aside, DevOps deployment is most often seen in brownfield projects that encourage building on the existing resources.
With this situation in place, there is also a lot of demand for DevOps resources across firms. The “2017 Open Source Jobs Report” from the Linux Foundation shows 60 percent demand for DevOps human resources among more than 2,000 IT open source professionals and IT hiring managers.
This clearly shows the level of demand that transition to DevOps is holding in the IT industry.
“If there isn’t an existing DevOps role within your company, software developers should spend time sitting with the operations team or vice versa to understand how their processes work and how they can be improved through collaboration,” said Mike Durney, CEO of the job-hunting site Dice.
The report found:
Nearly 50 percent of hiring managers are keen about those DevOps resources with professional credentials such as relevant certifications
More than 47 percent of the companies are ready to fund certification programs for their existing personnel
42 percent of companies are keen on adding DevOps skills to their hiring portfolio
“DevOps Engineer” profile ranks second in Glassdoor’s Best 50 jobs in America
Despite its advantages, however, DevOps has a few challenges to overcome related to application integration, test automation, budget and skill set, among others.
As organizations embrace the age of Agile and digital transformation strategies, every brand is looking for a competitive advantage. To succeed, organizations need to focus on the user, offering services to users on their terms, on their devices and at their convenience. Even more, users expect these offerings to be visually appealing and fast. That’s a lot to keep track of!
When selecting a digital transformation strategy, there are two key trade-offs organizations need to understand. First, getting features to market faster and increasing presence on users’ devices, and second, maintaining high application quality. Acceleration can come in the form of adopting an agile process, i.e. highly independent dev teams who are responsible for one feature in the code, delivering incremental functionality from design to production. A proper quality methodology is as equally effective, ensuring a high-quality application at the end of each sprint. This can actually help the team accelerate.
Continuous integration (CI), continuous delivery (CD) and continuous testing (CT) are the top three methods by which many development teams adopt an agile strategy. While each serves a slightly different objective, when combined, they can significantly help the team achieve velocity and quality, two of the most important goals of any development team. Let’s dive deeper into these three different components to understand the critical role they each play.
Method 1: Continuous Integration
CI is the most critical component to making an effectively agile development team. To help understand the important role CI plays, the image below depicts a team that has not installed a CI process. As you’ll notice, this team does not share their code with the rest of the development team until after the 60-day development period is over. As a result, by waiting to share key performance components, such as code, this team has unnecessarily extended the post-sprint stabilization phase, where developers test and redo integration points. With speed being such a sensitive KPI in situations such as app development, prolonging this phase can make a significant difference in the success of a project. What’s more, it can cause frustration among developers and testers across the software development life cycle (SDLC).
For comparison, the image below depicts a team that has implemented a CI process. Using CI, the team integrated increments from the main tree continuously. In addition, by using test automation, they are able to ensure the integration actually works. By implementing a CI approach, the conclusion of each sprint is on time and within the defined quality expectation, resulting in a reduced stabilization phase. In some situations, it might be possible to get rid of this phase altogether. The ideal CI process yields a working product at the end of each sprint and each day.
Method 2: Continuous Delivery
CD is the practice is the practice of streamlining and automating all the processes leading up to deployment. There are a number of steps in CD, such as validating the quality of the build in the previous environment. When done manually, these steps can take significant effort and time; however, they can be automated using cloud technologies and proper orchestration.
Teams should ensure they have a monitoring dashboard for their production environment in place to eliminate performance bottlenecks and respond fast to issues. This will complete an efficient CD process.
Method 3: Continuous Testing
CT, which can also be referred to as continuous quality, is the practice of embedding and automating test activities into every “commit.” CT helps developers use their time more efficiently when trying to fix a bug for code that was written years ago. To fix the bug, developers should first remind themselves of which code it was, undo any code that was written on top of the original code, and then retest the new code—not a short process. Testing that takes place every commit, every few hours, nightly and weekly not only increases confidence in the application quality, but it also drives team efficiency.
Outlined below are various steps teams should take to achieve CT:
Confirm a stable test lab 24×7.
Arm the testing team with a suite of test and development tools that enable high productivity.
Leverage automation for only the high value and stable tests.
Properly size the platform and test coverage for all projects.
Enable fast feedback via frequent reporting and analytics.
Summary
One of the most popular misconceptions about adapting to today’s digital transformation and implementing an agile development process is that it hinders speed and quality. Incorporating the CI/CD/CT trifecta, in addition to an agile development methodology enables the velocity and quality needed for success in today’s rapidly transforming industry.
