devopsy.kinsta.cloud

Tag: continuous testing

  • DevOps Chat: The Kitty Hawk Venture & Continuous Testing w/ Aruna Ravichandran

    DevOps Chat: The Kitty Hawk Venture & Continuous Testing w/ Aruna Ravichandran

    A great new book on DevOps is coming out called “The Kitty Hawk Venture.” In the vein of “The Phoenix Project,” it is a novel about one fictional company’s journey specifically with continuous testing. Based on a real-life airline company, the book is great case study and learning experience around continuous testing and its importance to the entire CI/CD and DevOps experience.

    The book is written by Jeff Schaeffer, general manager of the Continuous Testing business unit at CA Technologies; Alex Martin, CTO of Continuous Testing at CA Technologies (and former CTO of Rally Software); and Aruna Ravichandran, vice president of  Gobal Marketing at CA Technologies and author of “DevOps for Digital Leaders.” Beyond the book, there is a companion ebook on continuous testing, as well as other related content.

    In this DevOps Chat, we speak with Aruna, who is a frequent guest on our chats, about the book, continuous testing and the overall DevOps market. It is an excellent chat and, as usual, Aruna’s passion for what she does and DevOps shines through.

    For those not familiar with our DevOps Chats, the streaming audio of our conversation is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel. devopsy.kinsta.cloud. You’re listening to another DevOps chat. This is gonna be a really great chat. I have one of my dear friends in the DevOps industry joining us. Aruna Ravichandran Ravichandran, VP of Marketing DevOps at CA technologies. But more than that, now a multi-time author as well as a code engineer and a bunch of other things. She’s had an amazing, amazing life and career. Aruna Ravichandran, welcome.

    Aruna Ravichandran: Thank you, Alan. I am very excited to be here with you today talking about my latest book.

    Shimel: Yup. So it’s a new one coming out, hopefully in July I guess is the current timeline. And it’s called, “The Kitty Hawk Venture.”

    Ravichandran: That is exactly right. So you know, I think we chatted about a year ago when I wrote my first DevOps book. It was called, “DevOps for Digital Leaders.” And what we decided to do based upon the substance of the book is to basically to take a leaf from the Phoenix Project and write this book in terms of a real life story about one of our customers. I can’t name who the customer is. And the book is actually called, “The Kitty Hawk Venture.”

    And it’s a novel about continuous testing in the world of DevOps in order to help customers as they’re going through their digital transformation journey. So it’s not written as a continuous testing handbook.

    But it’s basically a real life example which happened with one of our customers and tells the story of the journey they actually went through in terms of how they actually transformed their testing discipline, lines of automation down to continuous testing and how they actually addressed some of the bigger challenges they actually had internally across the board.

    Shimel: Got it. So Aruna Ravichandran, I promised you and I promise our audience, no spoilers on this. So I don’t want to go too much into the actual story of the book, let’s keep it at a high level and say that it is about continuous testing and it is based on a real life customer experience. Maybe an amalgamation of customers.

    But you know, Aruna Ravichandran, you mentioned your first book, “DevOps for Digital Leaders.” So this is then your second book on DevOps.

    Ravichandran: Yes, it is. But this time I’ve actually focused on one of the emerging disciplines within DevOps. the first book was primarily focused around DevOps as a discipline, some of the challenges customers actually face.

    Where do they actually start. Because you know, there is no hard and fast rule in terms of DevOps. you can start with continuous integration. You could start with continuous operations. Or you could start with continuous testing.

    The book was primarily focused around end to end DevOps and gives some amount of coaching to the end reader on where to start, what’s the maturity model, what are some of the metrics, how do you measure the substance of DevOps.

    It also talks about several different customer examples on the journey they actually went through. But and speaking from that, we saw that a whole bunch of our readers are interested in learning more about an emerging discipline around continuous testing. So this book is primarily focused on continuous testing as a discipline and why it’s a very important area within the world of DevOps.

    Shimel: Yup. Now, Aruna , though, you hear it in your voice when you’re speaking to our audience here – it comes through loud and clear. this whole area of DevOps – continuous testing specifically with the book. It’s something that you are passionate about. I get it, you’re a VP in the DevOps group. You should feel an affinity. But this seems to go beyond your title and your job. It’s truly something you have a passion for.

    The book is almost 200 pages from the early view I saw. You don’t write 200 pages, you know, blah, blah, blah. You’ve gotta have sort of a burning passion. What is the passion that’s driving you?

    Ravichandran: So the passion that drives me is my customers. As I talk to my customers in the DevOps space, I am very passionate about solving customer problems. And as I talk to my customers, especially customers who want to make an impact, who are on their digital transformation journey, what I learned was testing is a discipline which has existed for several different decades across the board, the very first time when this actual code was actually implemented.

    But what you – what I actually found is that people have testing practices. But it’s mostly around islands of automation. Very, very few customers actually are doing end to end testing, which I call continuous testing.

    And even though you have islands of automation, what I actually find is that testing still continues to be manual. I did a poll of a whole bunch of enterprise customers across the board, and I was shocked to find that testing still appears to be 70% manual across the board.

    And given all of the – this is my personal reason for why I am very passionate about continuous testing, because I think that DevOps has gained a tremendous amount of precedent, but if you don’t embed continuous testing as a discipline across the software development life cycle, you really cannot go through DevOps.

    So continuous delivery has a lot of what do you say affiliation with the market. More and more people are getting educated on how to basically go through the CICD pipeline so that it can go from code to production in minutes. But what they realize is that if you don’t have the testing discipline embedded across the SCLC, going through the pipeline doesn’t buy you anything.

    And so that’s why this is a very important topic which I am very passionate about.

    Shimel: I mean, Aruna Ravichandran, on some level it’s almost – and I think I’ve heard you say this, continuous testing is almost the missing link. Right? In the continuous delivery, in the software development life cycle delivery chain. Right?

    For a lot of – you know what I mean? Companies, they focus on configuration management, on CICD, even on security. But that continuous testing is – it’s right there. It’s a lynchpin. The other thing I wanted to just quickly mention about it is continuous testing isn’t just a pre-delivery —

    Shimel: It’s truly continuous.

    Ravichandran: You nailed it. I would 100 percent agree with you on everything you stated. It is the missing link within DevOps. continuous testing is a practice of testing across every activity in the software development life cycle with the hope that you can uncover and fix unexpected behavior as soon as it’s ingested.

    So it’s about embedding testing as a fundamental and ongoing aspect of every activity through the application life cycle, going all the way from requirements down into production, while facilitating the pre-exact mechanisms what happens in the world of DevOps to where you can actually fix the problem throughout the life cycle.

    Shimel: Let me return to “The Kitty Hawk (Venture),” here again if I can for a moment. You mentioned earlier you took a page out of Gene Kim’s Phoenix Project book in that this isn’t similar to other books you’ve written and that I’ve even reviewed. That are your typical sort of business textbooks, right? You follow an outline and a table of contents and it goes. “Kitty Hawk” is more like you say – a novel, I think is the right word for it.

    How does that make it different for the reader? Or the listener if it’s an audiobook? For someone reading the book.

    Ravichandran: Yup. So for example – this book “[The] Kitty Hawk Venture” is actually based on the main character in the book is Leigh Freemark and she’s a senior director for quality assurance at a company called Runway Air. So the vertical we actually chose was an airline company.

    The way this book is actually written is it talks about the journey of what they actually went through from a transformation journey. So they experienced a major glitch in terms of quality and which caused a substantial amount of negative reputation for Runway Air.

    So this book actually walks through Leigh’s journey, the journey of her own personal mission of making quality first as a big part of her agenda.

    And the way this book is written – we have so many technical books which basically come into the market on DevOps and on testing, so on and so forth. So we wanted to basically – we took a leaf from “The Phoenix Project,” and we also took another leaf from “The DevOps Handbook,” which is the second book which was recently announced by Gene Kim.

    So this is basically written as a novel which tells you the journey which Leigh Freemark went through with Runway Air, but at the end of each chapter we give guidance in technical terms, of what was the actual continuous testing journey Leigh actually went through.

    Right? So that’s why I said, it’s written as a novel but at the end of each chapter we also have like a handbook around the continuous testing discipline on what changes people actually need to make in order to fully embrace continuous testing as a discipline in the book.

    Shimel: And even beyond that, Aruna Ravichandran, from what I understand and correct me if I’m wrong, you guys are planning on coming out with sort of an eBook companion to “Kitty Hawk” on “The Definitive Guide for Continuous Testing.” Is that correct?

    Ravichandran: That is exactly right and so we just actually published it and it will be out very soon. It’s called “The Definitive Guide to Continuous Testing.” And here, we walk our users through 11 different disciplines on continuous testing.

    Ongoing all the way on small teams [sound drops] _____ to how basically they can go through the transformation journey on continuous testing. And at a very high level, I walk you through eleven disciplines and you will see that completely in the novel because that was the journey which Leigh Freeman actually went through in the book, is the first discipline how do you basically and it will – the – you know, all of your testers.

    It could be a developer who is actually doing unit testing or it could be someone who is doing API testing or it could be load testing. Or it could be reproduction testing. All testing in production, you need to be able to have your environment set up. How do you quickly virtualize your environment becomes a very important part of the mix.

    How do you basically break on your environment, how do you actually create your environments very, very quickly is one of the first disciplines. The second discipline talks about testing management right? So regardless of what testing you’re doing having the right data at the right time is very critical. So how quickly can you generate your data?

    And given the world of GDPR right now, you know, which basically went into a test in May, people have to be even more critical about generating test data. You want to make sure that your GDP are compliant. And so having the test data at the right time is critical but also being compliant with a whole bunch of mandates out there is even more critical. The third discipline talks about test automation. How do you basically fully orchestrate the application pipeline.

    So that if the scripts actually fail, you can orchestrate how you can actually achieve continuous testing. The fourth one is around pipeline orchestration. This is the backbone. Everything is tied to it. everything must be integrated with your entire automation stream. You need to understand how it works, how to interpret the results and how do you make quick decisions in order to find out which part of testing is actually a bottleneck?

    And this is what we actually call pipeline authorization. The fifth one focuses on the various different competences, right? If you are testing, the world is going more towards an API economy. So how do you actually bring API testing discipline so that you reduce the reliance on UI testing? Especially for business logic testing.

    The sixth one is around performance and load testing. Right? So you can’t think about performance and load testing toward the end of the testing life cycle. You need to think about shifting left performance and load testing so that you can enable development teams to actually think about this much more earlier on.

    The seventh one is the around the new emerging practice around security testing. So this is where everybody talks about DevSecOps, it’s all about shift left security. How do you enable developers to actually embed static code analysis as a part of their unit testing?

    How do you basically get them to think about security testing by making it easy for them? By having it integrated as a part of their ID environments, whether it’s intelliga or it’s xless or visual studios, so on and so forth. So security testing is the seventh one.

    The eighth one is accidents of test driven development and behavior-driven development. So you take the accidents criteria for each user and create that to ensure that those were actually met. Right? So this keeps testing focused within sprints and ensures developers develop what the business actually expect.

    The ninth one is around automated test generation. And this is the activity of manually designing and writing manual tests of automated tests which is the biggest bottleneck.

    They are the computing magazine data surveyed very recently and they asked, which is the biggest bottleneck across your software development life cycle? 63% of the practitioners actually said that testing was a bottleneck. And why was testing a bottleneck? Because it takes time.

    Think about it, you have actually created 200 test cases for your regression suite for a vertical feature you are going to launch. Let’s assume you’ve caught a major defect and you fix the vertical defect, it means that you have to go back and recreate all of your regression text suites.

    And if there was an automated way that you could take your requirements and as things changed in production you can recreate your test suite, that will be a tremendous amount of savings for anybody to actually not have to create the test cases manually which then eliminates the bottleneck which we actually have.

    The tenth one is around requirement engineering. You must include all of SDLC stakeholders into your continuous testing journey. This means finding a better way to communicate and collaborate right from the requirements phase. The earlier one includes requirements engineering into continuous testing initiatives, the smoother that continuous testing journey will be. Because team members will be working with the same understanding right from the beginning.

    And the last but not least – the eleventh discipline talks about feedback loops. This is the heart and soul of DevOps, it still continues to be the heart and soul of continuous testing. These are critical to ensure that you can have successful continuous testing.

    Shimel: Excellent. You know, Aruna Ravichandran, sitting here looking – I want you to talk about quality, I don’t know if we have time. Maybe we’ll do it on another show. But the 11 disciplines, you did a great job laying them out here. I’m just wondering if someone listening says to themselves, oh my goodness, where do I begin? This is daunting. There is eleven of them. What is your advice to someone just starting out, listening in here – do I do eleven at all at the same time? What do I start with? What – do I take the path of least resistance? How do I get started with this?

    Ravichandran: So I break it up into six different areas, so this is a question you get asked for DevOps and you get asked the same question around continuous testing. So if I have to give guidance on techniques on how to actually adopt – it’s also a cultural transformation too.

    So the first one is, you know, you need to figure out a way to improve the relationship between the tester and each developer. So keeping the team small and creating inter-team collaboration makes it very easy to actually report and actually share the results between the tester and developer.

    So that would be one important tactic which needs to be adopted from a cultural transformation perspective. Number two, is like I said, people actually have automation within testing, which is what we call test automation. But they really haven’t automated the whole thing. So I would say continue to make automation a priority.

    So figure out where people are spending a lot of their time – it could be around building a regression test suite, because they tend to be manual. So how do you automate? So keeping that automation mindset is important. Figuring out where you’re spending cycles of your time in terms of doing manual tasks is important. So making automation a priority is important.

    Map out within your SDLC and identify your automation opportunity. Number three is make it small – right? Like get small. Break it down into smaller increments so that it’s easier to test after every aspect of the SDLC. So break down increments after coding. Break it after design. This makes it easier to automate the test and it also makes it much more easy to deploy and integrate as a part of continuous delivery.

    The fourth one is you want to keep track of everything. Use metrics. Start-up criteria. Right? So your defect density rate is high, figure out how can you reduce the defect density rate. Continuous testing is all about immediately identifying if things are working or not. How do you basically rinse and repeat if it’s working, or how do you actually pivot and change if it’s actually not working.

    The fifth one is getting the right continuous testing tools. Tools play a big role. So we talked about also transformation, we talked about processes and practices, and also having the right tools is important. Figure out the tools that help you develop tests and analyze continuously.

    Pick the best of breed tools that work together in a way that incorporates very easily into your continuous delivery environment. Choose tools which have community based support, that have open source plans that everyone shares the challenges and solutions and have interesting and lead to interesting use cases.

    And last but not least, the number sixth one, is develop a system to display your results. So do deep dives into the results, because that shows you that if you have your code not working, where the gaps are. define your KPI and your acceptance criteria and make them quantifiable. Create dashboards to track those KPIs, including a baseline so that you can actually track with the sequence changes. So these are some of the disciplines I always share with my customers as well as my clients.

    Shimel: Excellent. And now with everyone listening on here. Aruna Ravichandran we’re almost out of time, one more question if you have a second.

    Ravichandran: Absolutely. I do.

    Shimel: So I don’t know if we’re letting the cat out of the bag. If you can’t talk about it, politely say so. But you’re about to roll out something we hear called continuous testing academy? Launching this summer?

    Ravichandran: Yup.

    Shimel: So what’s that about?

    Ravichandran: So the key thing – going back to the question you asked, this continuous testing is like an emerging discipline. People want to know what do I need to do, how do I need to start? How do I get there. And so last year I launched the – I saw that there was a tremendous amount of need in the market around JMeter.

    And so I launched JMeter academy in order to basically to provide input to the community so they could actually train themselves on JMeter. But this year, knowing that continuous testing is gaining tremendous amount of momentum, I’m going to launch a continuous testing academy.

    This is going to have a basic module, it will also have an advanced module. And people will actually have the opportunity to go through the entire module and get themselves certified. We’re also going to launch a certification and you will have the opportunity to become continuous testing certified by DevOps institute. So this is our way to give back to the community in order to basically get people understanding on how they need to evolve their continuous testing practice in order to make a bigger impact in the DevOps world.

    Shimel: Excellent, again, Aruna Ravichandran – great job. You are just, I mean, literally, what a dynamo. All over the place. Doing so many different projects. All with DevOps and continuous testing as a theme there. Congratulations and hats off to you. One of my heroes. Anyway, we’re about out of time. Thank you so much, Aruna Ravichandran, for giving us a preview. Once Kitty Hawk is available to the public, I’d love to have you back on and perhaps we can chat more about it, sound good?

    Ravichandran: Yep. That sounds awesome. As always, Alan, it was excellent talking to you. I love your energy, I love your passion. And looking forward to partnering with you more as we basically enable our customers and prospects to continue on marching further along in the world of DevOps.

    Shimel: Well stay tuned for some announcements, we’re working on it. Aruna Ravichandran, I will be in touch. Aruna Ravichandran Ravichandran, a VP product marketing, marketing DevOps, CA technology, multiple times author, and really DevOps enthusiast. Thank you for being our guest on DevOps chats today. We’ll speak to you soon. This is Alan Shimel for devopsy.kinsta.cloud. You’ve just listened to another DevOps chat. Have a great day everyone.

    — Alan Shimel

  • Tricentis Accelerate Conference’s Big Reveal Came from its Audience

    Tricentis Accelerate Conference’s Big Reveal Came from its Audience

    At the recent Accelerate San Francisco 2018 conference, Tricentis was joined on stage by big-name customers, global partners and well-known quality and DevOps evangelists. This was the first time a U.S. date for Accelerate was added alongside its annual stop in Vienna, Austria. And, while the number of attendees was impressive—especially when Google IO and Microsoft Build were held the same week—it was the undeniable sense of urgency running through the audience that stood out, to me, as the week’s biggest takeaway.

    Having covered countless software and technology conferences over the years, it’s always somewhat eye-rolling when the hosting vendor or conference organizer uses their time on stage poorly. I often see this done in one of two ways. They either, a) relentlessly try to sell you on a current transformation or shift that no one really buys into, or, b) only explain that there is a shift—one that everyone agrees on—but then offer no actionable information for those who will be impacted by it.

    Agile, DevOps, digital transformation, continuous integration/delivery—all of which came up throughout Accelerate—have been the focal points of software conferences for years. These shows have given development and operations teams a wealth of information, case studies, lessons learned, best practices and inspiring keynotes, and they should be commended for it. But, today’s CIOs are frantically looking for the culprit behind how they’ve invested so much to make these processes and methodologies possible—yet, are still struggling to get high-quality, software out the door faster. Many of these CIOs are realizing the glaring oversight that they, and these conferences, have made.

    Accelerate San Francisco’s manual testers, automation engineers, exploratory testers, test managers, QA directors and more didn’t have to be sold on the idea that testing has an enormous opportunity to solve the challenges their CIOs are facing today. Those who truly respect the power of “good” testing (an the artistry behind it) have always known what testers were capable of providing to the business. However, testing has been hamstrung by legacy technologies that prevented moving at the same pace of their development and operations colleagues.

    But, today, those constraints have been removed, thanks to innovative test automation technologies and testers and QA professionals may not even have to sell their “new” worth to management. More and more often, the C-suite is recognizing software testing as an untapped resource for innovation, and many of Accelerate’s attendees could clearly sense the attention and expectations from their leadership on the horizon.

    Day 1 of Accelerate was largely product-focused and featured a number of “discovery” and “deep dive” sessions led by the product owners of some of Tricentis Tosca’s most popular functionality. Continuous Testing, Load Testing, API Testing & Service Virtualization, SAP Testing were all showcased and demoed to existing customers and those evaluating whether Tricentis could assist them on their own unique journeys. Day 1 wrapped up with a product road map discussion given to a standing room-only crowd, and led by Tricentis chief product officer, Dr. Gerd Weishaar.

    Day 2 kicked off with a bang with what was dubbed, “The Great Debate in Software Development.” Electric Cloud’s Anders Wallgren, Accenture’s Jeff Wilkinson and Tricentis founder Wolfgang Platz sat down to discuss who owns quality today, why that responsibility might change, and how testers and SDETs can be prepared for the future. There were some really outstanding—and oftentimes, conflicting—points made by each panelist, including:

    “Development is a skillset; quality engineering is a mindset.”

    “Testing is always reactive; quality engineering is proactive.”

    “Quality is a frequency and depth of measurement game.”

    Once we get the debate’s recording online, I’ll share the link in a future blog, and would love to keep the conversation going with anyone who is interested. There was so much that was presented that’s worth discussing and debating further.

    Dr. Nicole Forsgren took the stage immediately after the debate and introduced her new book, “Accelerate,” (how could we not have invited her to a conference with the same name?) that she co-authored with fellow DevOps household names Gene Kim and Jez Humble. I’m constantly looking for ways to get testing more on the minds of DevOps community (because it makes zero sense for it to be so absent), so I was thrilled to learn that continuous testing has made it into DORA’s next “State of DevOps” report. Their annual survey that this report is focused on should be released soon, so look out for that.

    Tricentis CEO Sandeep Johri then took the stage to give his thoughts on Tricentis’ role within the current state of software testing. Johri pointed out that while software becomes the key to success in many digital transformation initiatives, it also becomes a serious problem when software cannot be released at the speed that business demands. “Who cares if you can develop faster if you can’t also ship faster?” said, Johri. “You cannot ‘ship and pray’ in the enterprise.”

    Being able to ship faster—with a sound, risk-based assessment of each release candidate—requires continuous testing, said Johri. And, to CIOs who historically have viewed testing as a bottleneck to innovation, I can see where this could be a hard sell. “You know that thing that you hate? The thing that costs a fortune, and takes forever, and doesn’t provide you with an easily discernible ROI? Do that way more, or, better yet, do it all the time.”

    But, as Johri remarked—and I’d never actually heard someone word it this way—”Testing isn’t a bottleneck. Scripting is.” This is 100 percent accurate, and, as CIOs in greater number begin to realize this, continuous testing will become not just better understood, but a must-have in every organization. It’s the manual effort around testing that slows you down, stalls innovation and adds increased costs around the requirement for testers or developers to have coding expertise. These costs and bottlenecks are what CIOs want to eliminate. As awareness of modern test automation technologies that remove these constraints increases, Johri predicts that the solutions that can’t offer the same level of agility at scale will continue to fade from the market.

    The next keynote of the day was given by Tricentis founder Wolfgang Platz. While Platz gave due-credit to what I’m calling the “A.I.-lephant in the room,” it was the advice to the Accelerate attendees of how to navigate the current transformation going on in world around them that I found most inspiring. Platz shared the current strategic priorities for CIOs in 2018, and it was remarkable how clear it was to everyone in the crowd where they could assist with meeting each one of them.

    We hear all the time about lofty aspirations such as, “Quality is everyone’s responsibility,” and, “DevOps establishes that every team is working toward the same goal.” But, when Platz ran through CIO priorities such as digital transformation, customer experience, data analytics, moving faster, improving culture, scaling agile development and freeing up budget, eyes didn’t roll. People didn’t groan or walk out. It was clear that the entire room understood exactly what they were capable of by trading the limitations of yesterday for the capabilities of tomorrow.

    Today’s CIOs have never been under more pressure to avoid disruption, and my biggest takeaway from Accelerate was that the testing community understands that accomplishing this monumental task—in every software release—is only possible with their involvement.

    Let’s see how many testers continue to navigate through change introduced by others, and how many take the opportunity to drive new change themselves.

    — Noel Wurst

  • Exploratory Testing: Expanding Testing Across the Delivery Cycle

    Exploratory Testing: Expanding Testing Across the Delivery Cycle

    Mindless testing, whether it is manually or automatically executed, can cover only a very narrow scope of possible bugs: the ones you’ve already thought of. Exploratory testing however, endeavors to think outside that scope. Thoughtful testers look at a problem from all angles, weighing factors such as feature usage, psychology, user stories and more. Their goal is to chart the uncharted territory of an application.

    The faster our development cycles become, the more thoughtful our testing should be. It’s only through techniques such as exploratory testing that we’ll be able to get there.

    Continuous testing is the process of executing automated tests to obtain rapid feedback on the business risks associated with a software release. Where does that leave exploratory testing? It’s not automated, but it’s certainly critical for determining whether a release candidate has an acceptable level of risk.

    Test automation is perfect for repeatedly checking whether incremental application changes break your existing functionality. However, where test automation falls short is in helping you determine whether new functionality truly meets expectations. Does it address the business needs behind the user story? Does it do so in a way that’s easy to use, resource-efficient, reliable and consistent with the rest of your application?

    Exploratory testing promotes the creative, critical testing required to answer these questions. Obviously, it doesn’t make sense to repeat the same exploratory tests continuously across and beyond a sprint, but exploratory testing can be a continuous part of each delivery cycle.

    Exploratory Testing in Action

    Here are a few ways teams embed exploratory testing throughout their process.

    Perform Ad Hoc Exploratory Testing as Each User Story is Implemented

    This is the exploratory testing equivalent of peer code review. When a developer completes a user story, they sit down with a tester. The tester starts testing while providing a running commentary on what they are doing and why. Next, the developer takes control, explaining how they would test the software given their knowledge of the implementation details and challenges. The developer gains a user- and business-focused perspective of the functionality, and the tester learns about the inherent technical risks.

    Another tactic is to have the developer and a tester separately test the same feature simultaneously, then discuss their findings at the end of the session. Often, this turns testing into a competition, where each participant tries to uncover the most or “best” issues in the allotted time.

    Align Exploratory Testing Sessions with Full Regression Testing

    It’s simply not possible to perform exploratory testing or full regression testing on every code commit. That’s what smoke testing is for. Instead, many teams run full regression testing and session-based exploratory testing in parallel a few times per week, whenever they’ve implemented new functionality that a user could feasibly exercise.

    For optimal results, these sessions should be lightly planned and tightly timeboxed, include diverse perspectives and really take the six thinking hats theory seriously.

    Host Blitz Exploratory Sessions for Critical Functionality

    The best way to uncover user experience issues before users do is to get a broad array of feedback prior to release. One way is to host “blitz” exploratory testing sessions. When you’re wrapping up work on critical new functionality, invite people from a variety of backgrounds and teams to participate in a short timeboxed session. Incentives can help drive participation, maximize results and make testing fun.

    Using test automation to continuously check the integrity of existing functionality is certainly critical. However, if you’re not also making exploratory testing a continuous part of your process, how will you know if the new functionality meets expectations?

    The goal of continuous testing is to understand whether a release candidate has an acceptable level of risk. Exploratory testing is perfectly suited for helping you answer that critical question.

    — Ingo Philipp

  • Continuous Testing Success at ExxonMobil

    Continuous Testing Success at ExxonMobil

    Safety, security and controls are the foundations upon which 135-year-old ExxonMobil is built. As such, quality has become a key component in delivering software. The company is currently undergoing a shift from traditional project management organizations and support organizations to product teams, which has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.

    Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”

    To do this, ExxonMobil started with the basic idea—quality control manages risk—and then explored what is tracked today and whether quality is being measured. Another key activity was to remove the roadblock to continuous testing.

    Ann Lewis from ExxonMobil discusses in this video how the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset, as well as the company’s view that continuous testing and deployment can only be accomplished through model-based testing and that managing quality reduces overall cost of ownership.

    The full transcript of her discussion is below:

    “At ExxonMobil, we have presence in six continents, so that means we get to deal with a lot of people. There’s over 72,000 employees plus another 20 percent contract staff, and that’s a problem. Guess what they want to do? Manual testing. Yeah, because they’re not gonna give it up. We have been on an adventure this last year that I’m gonna talk about and explain to you how we help train these 72,000 plus people that there’s a better way to do it.

    “In addition to the numbers of people that we have, the upstream, I always … Well, someone referred to it about sticking a straw in the ground. Yeah, it was actually somebody from Tricentis. Yeah. I was just like, ‘There’s a little bit more to it than that.’ No, we do produce all of the product from under the ground. We get it out, and then we send it downstream to our refineries so they can do something with it like sell it. We also have a chemical company, that … I think right now are second or third to DuPont. We have a lot of product and a lot of things to do with it. That’s a problem, too. Because when it comes to the applications that test, that need to be tested, we have something like 5,000 applications that run our business all technologies. We’ll talk a little bit about that as we go through, too.

    “The title is The Middle Child Rises. Why did I choose this? Because in the world of Dev and Ops, test is in the middle.

    “We have a few other friends in the middle, and they are a little bit like a middle child, so we’ll talk about that. For today, your takeaways are that our organizational design has shifted. We’re going to explain why and how that happened. We will explain that Tosca is strategic for ExxonMobil and we will explain why that is.

    “Tosca is an enabler to speedy delivery.

    “You’ve heard about that this morning from many of our Tricentis friends, and it is true and I’m here to share that story. We now have within ExxonMobil, a quality philosophy, yeah, and it’s a lot more than just testing. Yes indeed, it is. We’re going to take a look at what we did and have a lot of fun stories in between.

    “Being the middle child, you really didn’t get as much attention. How many are middle children, by the way? A few, okay. Do you have babies in here? Yeah, yeah, you kind of get away with everything, don’t you? The oldest, I have a favorite, I know. Oh, there’s another one out there of my fave, okay, cool.

    “It’s very interesting in family relationships how these children get along. The oldest kind of sets the stage for everybody. That middle one comes along and it’s like, ‘Oh, it’s the second kid.’ The third one comes along and it’s like, ‘Oh, it’s the baby, it’s the baby,’ and the middle child just kind of sneaks through, okay? They don’t get quite as much attention. We’re going to talk about the middle child as they relate to testing.

    “Typically, it’s the middle child that ended up with rules. That oldest child was all about, if you just tell them once, they usually would stop doing it. That’s the case I had. Then the second one was born, that turned out to be the baby, but I swear he’s got components of being the baby and the middle child. It was like, ‘Adam, we have rules in this household. Yes, we have rules.’ He was like, ‘I just don’t get the same respect as that older sister, Jessica.’

    “Now within the corporate world, these middle children, you can think of them as being security and controls, data migration, data conversion. It’s like, ‘Oh, yeah, we forgot about you.’ Last but not least, testing. How many times has it been when suddenly that project shows up, ‘Oh, yeah, I think I might need to engage your team to help me organize my testing.’ Yes, yes, those of us that live in the testing world.

    “The big deal right now, though, is that these middle children, our testing organizations, our quality assurance organizations, are becoming the key enabler, like we heard this morning, 50 percent of our problem is with testing. Oh, I could do that but I can’t get testers. I could do that but, ugh, we’ve got to do the testing. It takes so long.

    “According to the World Quality Report, the reliance on manual testing is the top technical challenge in app development.

    “How many of you knew about the World Quality Report? Yes? Yes? Good, good, good. We’re all the same kind of people, yeah. I like that.

    “Businesses see test automation as a way to address one of their top challenges to delivering better software faster, by adopting Agile, DevOps and continuous delivery practices to shorten delivery cycles.” – Forrester

    “By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks.” – Gartner

    “The need to support faster time to market with higher quality is driving the demand for effective functional test automation tools. This is why we’re here. Tricentis Tosca, it is a framework. Yay! It’s not scripting. Boo!

    “Tricentis Tosca will address that top technical challenge of delivering software faster.

    “The need to support faster time to market with higher quality. It’s higher quality in the form of risk coverage. That is what we do with Tricentis Tosca. That is what we do with Tricentis Tosca.

    From a framework perspective, the last year and a half we have been on a mission to explain the difference on automation tools. We have gone from record and replay, to test automation from a framework perspective using scripting, such as with the Hewlett-Packard products and Selenium. It is programming, it takes developer time to do automation when you’re talking about some of these historical tools. We have now moved into model-based, and I hope everyone in here has been to at least some of the Tricentis’ demos on what model-based is. I’m not going to go into detail here, but I will tell you that it has taken our automation from a 40-hour event down to a 8 to 16-hour event. No joke.

    “I have been able to take Tricentis Tosca and put it into non-developer, normal people’s hands.

    “I actually sat down with one of the Tricentis’ architects and did automation myself. I said, ‘Oh my gosh, I haven’t touched a keyboard to do anything technical in so long,’ and I actually did it. That is key, that is significant. That has been the story that has been going on with ExxonMobil for the past year and a half.

    “It has become strategic for us. It’s automated screen scanning, it’s lower maintenance, it’s targeted testing. I’m doing it in half of the time, if not less. It’s faster, it’s easier to support, it’s talking about prevention versus detection. We have created a road map.

    “What you’re seeing in our colorful TFS diagrams on the top, yes, we are doing our monitoring in TFS. You’re seeing a migration and better put a re-engineering of our existing scripts into Tosca. Re-engineering because for the last two years that I’ve been in this job, people kept telling me, ‘Ann, did you know that those existing Hewlett-Packard scripts really don’t test anything?’ ‘Well, gee, I don’t think I knew that.’ ‘Well, gee willikers, let’s see if it’s really true.’ Lord, have mercy. Twenty percent of those Hewlett-Packard scripts that were driven by the business to automate didn’t test anything. Well okay, they tested something but it wasn’t worthwhile.

    “All of these little blue dots on your right beyond the solid blue vertical line are things that don’t really matter because we didn’t do a risk analysis. When I talk about re-engineering, we’re not just taking what we had and push it into Tosca, which you can do, by the way, there’s a tool for that. It’s up to you to use it, I’m not going to say anything. All I’m telling you is all our experience has been it made no sense.

    “What we have found is when we sit down with our subject matter experts and we had this really cool dialogue like, ‘What do you do all day long?’ ‘Oh, I use VA-01 and NM-16 all day long.’ I said, ‘Cool. And so what happens when one of them doesn’t work? Dead in the water. You’re in customer service, aren’t you?’ ‘Yeah.’ ‘How many sales? Millions of dollars of sales do you do every day?’ ‘Well … ‘ This is the user, I mean they are totally into explaining to us what they do. Through this dialogue, this interview, we find out that they’re going to cost us $1 million a day, an hour, I don’t know what the big number is but it’s a big number, when it comes to customer-facing transactions. Therefore, those transactions are the ones that we want to automate.

    “Now if I find out, no offense to anybody in controllers, that at the end of the month they have a report that they need to generate, ‘I’ve got some time to fix that one, okay?’ ‘How often do you do this? Monthly? Okay, that’s a one. What happens if it doesn’t work?’ ‘Well, I just did last month’s report and … ‘ ‘That’s a one.’ Low priority, move on.

    “We have been re-engineering all of our existing tests.

    “This word crypt is an ugly word. I actually I know the person who named our existing automations. Critical Integrated Process Testing. We don’t say that word anymore, by the way.

    “The middle child has risen. This is my team. When I joined the group in February of 2015, we had about 25 people. About 25 people. Within ExxonMobil, I think there might have been seven of us. Then, we had some offshore group that did all of those monthly executions of those Hewlett-Packard scripts.

    “Today, I have Vulcar over automation, he’s based in Hamburg. I have Cathy in Houston for functional testing and Santa Juan for functional testing, basically split between the two regions. They’re what we call test leads. They do a lot of the coordination of our testing activities. I have Aaron in Houston for performance testing. I have Jay, he is Doctor Jay, he is the man that knows everything about testing and everybody loves Doctor Jay. We have Suzanne right there, one of my oldest children. I didn’t say that, did I? I meant oldest child. Anyway, the other thing that was really funny about these pictures, they kept saying, ‘Oh, I want mine to be skinny.’ I want mine to be really upfront. We had a whole dialog on how the pictures turned out. It was pretty fun. Guess what? There’s close to 150 people now. That’s a signal that over this past year and a half, two years of getting this team put together, the emphasis that ExxonMobil has put on quality. One of the ways to get to quality was through testing.

    “We are everywhere. I’m going to go into a little bit more detail now about organization because it’s good to understand where you are located. I was talking about being on the six continents. We have now placed people in region, so that when a person has a question on, ‘I want to automate, should I automate?’ We’ve got people in those locations to help with that. You will see we are using architects.

    “Within the Tricentis nomenclature, you have architects, automation specialists and automation engineers. It is a tiering of skillset.

    “We have found that the architects become those go-to people, when it comes to, ‘Should I be automating this or should I not and how do I go about it?’ Yes, we are using some offshore resources in India. Yes, we have people in Auckland. Yes, Bangkok, and that’s actually one of our ExxonMobil business support centers in Bangkok. Then, you can see Houston, Vienna, New York and Curitiba, Brazil.

    “Organizationally, what we have found is that when it comes to implementing automation and doing it in a … I’ll call it conscientious and disciplined way, you’ve got to have a structure. Don’t think that you can just, I’ll say, run into doing automation. There needs to be a structure upfront to help you manage this work, because once one customer comes, you’re going to get five more. The enthusiasm level that we have experienced is unbelievable. We’re almost on tilt. But through this concerted effort of having an organization to help support it through a governance and then program management. Again, you can see this partnership that we have with Tricentis. Highly recommended when it’s time to get started.

    “Successful team composition. It’s typically one architect to one engineer to 10 specialists and then you ramp up or ramp down, based upon if you’re working on systems of engagement. The shiny coin fun stuff, mobile, gooey, or your systems of record. They’re a little bit of a worn coin, all right? I’m sure some of you have systems that are quite old. We have ones that are really old so our financial system is still in IMS in PO1. Yes, rock on! That’s what I grew up with. Yeah, give me some JCL, too.

    “See, geek, we’re all geeks, I know. It’s like once you join the world of IT, you’re just like into it. Nobody really understands what you talk about, unless you’re with people that are like you. Uh-huh (affirmative), all in the family, totally all in the family.

    “Okay. We also have an immersion program. We have, again, the process of getting somebody into you’re approved, we think you can handle the work, all right? Going through automation specialists and all the way up to graduation. Again, working with Tricentis, we have come up with again, a disciplined process. You need this when you get started because they will come in out of the woodwork, I want to automate. I talked to you two years ago about automating those processes at the refinery and you said that no machine could do what you could do. I don’t say that, but that’s what I’m thinking when they come to me two years later with, ‘I think I want to do automation.’ I’m like, ‘Cool, let’s get into it.’

    “We have a quality manifesto. Yes, we do. We will create forward-looking, value for our customers and for our organizations through quality approaches. Quality engineering is a team responsibility. I’ve got a great team, but we’re just here. We’re the messengers, we’re the trainers, we’re the coaches. Everybody that we work to, we talk about quality.

    “How did you get to work today? Did you think about a safe way to get to work today with all of the rain? Yes. That’s a quality decision right there. Little things that we teach when we’re starting to push the quality manifesto out. You have to understand business function risk. Again, what do you do all day? What happens if it doesn’t work? Oh, really bad things happen, or maybe not so bad and a month in close, we’ve got a whole month to worry about it. Yeah, we can always close the books this month of last months’ numbers. I actually have a degree in accounting, I can talk about them.

    “Including quality checks throughout the delivery life cycle. Quality gates. That’s another item that we have built into our system delivery life cycle. Starting to talk about at the moment requirements are written down, let’s start talking about what you’re going to test, based upon those requirements. We have built these quality gates, these questions throughout system delivery life cycle.

    “What has happened? Our organizational design has shifted.

    “Tosca is strategic for ExxonMobil. Tosca is an enabler to speedy delivery.

    “Test automation brings value. We’re actually connecting the automation into total cost of ownership, as a way to reduce application total cost of ownership. We have a quality philosophy now and we’re just pretty darn excited about what we did. It took the rise of the middle child in order to make it happen. Thank you so much.”

    — Jessica Wanivenhaus

  • Attaining Continuous Testing in DevOps through a Templated Approach

    Attaining Continuous Testing in DevOps through a Templated Approach

    The continuous demand market for flawless applications is being addressed through advanced software development techniques along with continuous delivery. As such, many organizations adopted the shift-left approach and agile best practices—which suggests the motto, “test early, test often,”—through which they get to imbibe QA practices and testing early in the software development pipeline. That also includes continuous testing for many organizations.

    Continuous Testing in DevOps

    Continuous testing today is the most preferred practice by leading software companies. Continuous testing implies testing through the whole software development pipeline, from product design to product release, by integrating the complete product cycle. Even after the product release, companies test to troubleshoot customer complaints to further improve the product.

    Continuous testing in DevOps involves encouraging the development engineers and operations to practice testing as well, and also makes them a part of the whole cycle instead of being a separate team. Continuous testing involves testing not just from an earlier stage but also through each and every single process. Its goal is to automate almost all the processes as possible.

    A Templated Approach to Continuous Testing in DevOps

    A perfect approach to continuous testing in DevOps should be a right balance between manual and automation testing, and to achieve it, organizations must practice virtualization to plan out the whole product development pipeline successfully. Many organizations are not yet ready to risk it all with continuous testing, as big errors are still a possibility if they don’t really know how to execute it. But, errors in the continuous testing method can be easily rectified as it is practiced through each process.

    Errors are common when it comes to product development; hence, the right set of tools to automate testing can be critical to finding and rectifying errors, since manual testing can miss certain errors. Manual testing also takes a considerable amount of time, which is a huge drawback. A well-planned but carefully structured templated approach can be practiced by companies to make a full transformation toward continuous testing.

    Continuous testing is achieved through imbibing multiple use cases over certain parameters to determine the strength of this methodology. Some of those parameters that would prove to be a sure-fire way of determining whether continuous testing is worth it are distribution, technology, scale and load, environments and security and release.

    Checking the Parameters

    The parameters to completely check a dimension such as continuous testing can be done to create readings based on elimination and weighted scoring to derive results from a use-case scenario. Let us see a brief application of running a hosted public-facing service through these parameters:

    Distribution – The distribution areas and the geographical location of this product are considered.

    Scale and Load – The number of test cases (3,500 automated cases in this scenario, by way of an example), the suites that are used based on the browsers, cloud test platform and the numbers of users are considered in this category.

    Environments and Security – Varied device uses, cross-browser testing, secured cloud environment and approval of releases are considered in this parameter.

    Release – The number and frequency of release and the technology stack are considered in this parameter.

    The tools that are used for this scenario are Selenium web driver, Selenium GRID, Jenkins, Sauce Labs, Sauce On-Dand Plugin.

    The process through which checking the parameters is done is by triggering automated test cases from Bitbucket automation repository using Jenkins during deployments, and the execution of automated scripts in Sauce Lab cloud, as parallel threads are done by interfacing Sauce Lab Plugin with Jenkins.

    The control mechanisms first include testing through smoke, regression and integrated automated tests execution. Then, the test evidence for all of the test cases is captured. Later, as the tests are completed and the screenshots which are captured by automated tests are validated, UAT sign-off is done.

    Some parameters for test automation framework are compared within Selenium and tests completed over various categories, their scores are compared and then they are passed onto the test environment. Some suite, regression suite and integration test suites are carried out during Dev and QA stages, later when the changes from these two environments are deployed. Smoke suite is carried out during pre-production, editorial and the production stages. Finally, the proven results of these stages are derived successfully.

    The proven results of this use-case scenario through these stages are:

    • Parallel testing visibly reduced regression hours from 150 hours to 15 hours.
    • Enhanced test coverage matrix addressed varied device and OS fragmentation, as well as browser combinations.
    • Quality gatekeeper for code promotion to production.
    • Reduced regression issues in production.

    Conclusion

    The example of going through such a use-case scenario helps us better understand how continuous testing can be integrated gradually into a product development pipeline through careful observation, planning and deployment through such parameters.

    Engrafting continuous testing in a DevOps eEnvironment needs careful execution to avoid errors and future defects. And with the right set of tools and a perfect execution, it can be achieved to successfully experience faster time to market and reduction in time and cost.

    For more information on how a templated approach can be applied in different scenarios, watch this webinar.

    — Niranjani Ramesh

  • Agile Development: The 3 Methods You Need to Master

    Agile Development: The 3 Methods You Need to Master

    As organizations embrace the age of Agile and digital transformation strategies, every brand is looking for a competitive advantage. To succeed, organizations need to focus on the user, offering services to users on their terms, on their devices and at their convenience. Even more, users expect these offerings to be visually appealing and fast. That’s a lot to keep track of!

    When selecting a digital transformation strategy, there are two key trade-offs organizations need to understand. First, getting features to market faster and increasing presence on users’ devices, and second, maintaining high application quality.  Acceleration can come in the form of adopting an agile process, i.e. highly independent dev teams who are responsible for one feature in the code, delivering incremental functionality from design to production. A proper quality methodology is as equally effective, ensuring a high-quality application at the end of each sprint. This can actually help the team accelerate.

    Continuous integration (CI), continuous delivery (CD) and continuous testing (CT) are the top three methods by which many development teams adopt an agile strategy. While each serves a slightly different objective, when combined, they can significantly help the team achieve velocity and quality, two of the most important goals of any development team. Let’s dive deeper into these three different components to understand the critical role they each play.

    Method 1: Continuous Integration

    CI is the most critical component to making an effectively agile development team. To help understand the important role CI plays, the image below depicts a team that has not installed a CI process. As you’ll notice, this team does not share their code with the rest of the development team until after the 60-day development period is over. As a result, by waiting to share key performance components, such as code, this team has unnecessarily extended the post-sprint stabilization phase, where developers test and redo integration points. With speed being such a sensitive KPI in situations such as app development, prolonging this phase can make a significant difference in the success of a project. What’s more, it can cause frustration among developers and testers across the software development life cycle (SDLC).

    For comparison, the image below depicts a team that has implemented a CI process. Using CI, the team integrated increments from the main tree continuously. In addition, by using test automation, they are able to ensure the integration actually works. By implementing a CI approach, the conclusion of each sprint is on time and within the defined quality expectation, resulting in a reduced stabilization phase. In some situations, it might be possible to get rid of this phase altogether. The ideal CI process yields a working product at the end of each sprint and each day.

    Method 2: Continuous Delivery

    CD is the practice is the practice of streamlining and automating all the processes leading up to deployment. There are a number of steps in CD, such as validating the quality of the build in the previous environment. When done manually, these steps can take significant effort and time; however, they can be automated using cloud technologies and proper orchestration.

    Teams should ensure they have a monitoring dashboard for their production environment in place to eliminate performance bottlenecks and respond fast to issues. This will complete an efficient CD process.

    Method 3: Continuous Testing

    CT, which can also be referred to as continuous quality, is the practice of embedding and automating test activities into every “commit.” CT helps developers use their time more efficiently when trying to fix a bug for code that was written years ago. To fix the bug, developers should first remind themselves of which code it was, undo any code that was written on top of the original code, and then retest the new code—not a short process. Testing that takes place every commit, every few hours, nightly and weekly not only increases confidence in the application quality, but it also drives team efficiency.

    Outlined below are various steps teams should take to achieve CT:

    • Confirm a stable test lab 24×7.
    • Arm the testing team with a suite of test and development tools that enable high productivity.
    • Leverage automation for only the high value and stable tests.
    • Properly size the platform and test coverage for all projects.
    • Enable fast feedback via frequent reporting and analytics.

    Summary

    One of the most popular misconceptions about adapting to today’s digital transformation and implementing an agile development process is that it hinders speed and quality. Incorporating the CI/CD/CT trifecta, in addition to an agile development methodology enables the velocity and quality needed for success in today’s rapidly transforming industry.

    — Amir Rozenberg

  • Sauce Labs Adds Analytics to Continuous Testing Platform

    Sauce Labs Adds Analytics to Continuous Testing Platform

    Sauce Labs has added additional debugging and analytics capabilities to its testing platform to make it easier to manage continuous testing across multiple application development projects.

    Lubos Parobek, vice president of product at Sauce Labs, said DevOps teams need to prioritize which bugs to address first across multiple development projects occurring simultaneously. In fact, according to the latest annual survey of 1,091 technology professionals conducted by Dimensional Research on behalf of Sauce Labs, DevOps teams on average are now taking longer to address bugs. These days there are many more issues that require debugging across multiple development projects, which is overwhelming DevOps teams, who must then prioritize some debugging requirements over others.

    Specifically, the survey found only 21 percent of respondents report they fix bugs immediately, down from 23 percent in 2017.

    Analytics capabilities included in Sauce Labs’ rechristened Continuous Testing Cloud include the ability to view test results by browser and operating system, visibility into pass/fail and error trends and the ability to see how a single test behaves over time so DevOps teams can more easily identify tests inconsistent with overall test run time and pass/fail rates.

    Scheduled to be available in the second quarter, the Continuous Testing Cloud makes it possible to more easily reconcile advances in the rate at which applications are being developed and constrained testing and debugging tools, Parobek said.

    The Sauce Labs survey results also show progress being made in the realm of mobile application testing. A full 77 percent of respondents said they now use a mix of both real devices and simulators or emulators for mobile testing, up from 34 percent in 2015. That approach results in higher fidelity to the actual experience of an end user on each device, said Parobek.

    That shift, however, may also explain why there is more manual testing. More than half (56 percent) said they do as much or more automated as manual testing, a minor decrease from 59 percent (2017) and 60 percent (2016), the report found. Just under half (45 percent) said they do, however, expect to increase spending on test automation in 2018.

    Parobek said organizations are moving into what he describes as the third era of testing. The first two eras were manual testing followed by a shift to automated testing. The third era is marked by continuous testing that leverages automation to make it possible to test code as it’s being developed rather than after, when the application is complete.

    Naturally, each organization is not as far down the path toward continuous testing as every other. Parobek said for the most part, early adopters of DevOps practices employ continuous testing more widely than traditional enterprise IT organizations that are still in the early stages of embracing DevOps. The good news is the latter have an opportunity to leverage a much larger body of DevOps best practices than the organizations that originally pioneered the concepts.

    — Mike Vizard

  • Continuous Testing and the Broken Promise of Test Automation

    Continuous Testing and the Broken Promise of Test Automation

    The Broken Promise of Test Automation

    For more than two decades now, software testing tool vendors have been tempting enterprises with the promise of test automation. However, most companies have never been able to achieve the desired business results from their automation initiatives. Recent studies report that test automation rates average around 20 percent overall, and from 26 percent to 30 percent for agile adopters.

    We believe that several factors contribute to these dismal automation results:

    Legacy software testing platforms were designed for a different age

    The most commonly used software testing tools today are predicated on old technology, but enterprise architectures have continued to evolve over the years. Development no longer focuses on building client/server desktop applications on quarterly release cycles, with the luxury of month-long testing windows before each release.

    Almost everything has changed since test automation tools such as those from Mercury, HP, Micro Focus, Segue, Borland and IBM were developed. Retrofitting new functionality into fundamentally old platforms is not the same as engineering a solution that addresses these needs natively.

    Legacy script-based tests are cumbersome to maintain

    Scripts are cumbersome to maintain when developers are actively working on the application. The more frequently the application evolves, the more difficult it becomes to keep scripts in sync. Teams often reach the point where it’s faster to create new tests than update the existing ones. This leads to an even more unwieldy test suite that still (eventually) produces a frustrating number of false positives as the application inevitably changes. Exacerbating the maintenance challenge is the fact that scripts are as vulnerable to defects as code—and a defect in the script can cause false positives and/or interrupt test execution.

    The combination of false positives, script errors and bloated test suites creates a burden that few QA teams can overcome. It’s a Sisyphean effort—only the boulder keeps growing larger and heavier.

    Software architectures have changed

    Software architectures have changed dramatically, and the technology mix associated with modern enterprise applications has grown immensely. We’re trying to migrate away from mainframes and client/server as we shift toward cloud-native applications and microservices. This creates two distinct challenges:

    • Testing these technologies requires either a high degree of technical expertise/specialization or a high level of business abstraction that allows the tester to test without diving into the low-level technical details.
    • Different parts of the application are evolving at different speeds, creating a process cadence mismatch.

    The software development process has changed

    Although most enterprises today still have some waterfall processes, there’s an undeniable trend toward rapid iterations with smaller release scopes. We’ve shifted from quarterly releases to bi weekly or daily ones, with extraordinary outliers such as Amazon releasing new code to production every 11.6 seconds. This extreme compression of release cycles wreaks havoc on testing—especially when most testers must wait days or weeks to access suitable test environment and test data.

    The responsibility for quality has changed

    In response to the desire for faster release cycles, there’s been a push to “shift left” testing. The people creating the code are assuming more responsibility for quality because it’s become imperative for getting to “done” on time. However, for large enterprises working on complex applications, developer-led testing focuses primarily on a narrow subset of code and components. Developers typically lack the time and the access required to test realistic end-to-end business transactions. Although the onus for quality has shifted left, the legacy platforms, rooted in waterfall processes, have a distinct bias toward the right. This makes it difficult to blend both approaches.

    Open source testing tools have changed the industry

    The rise of open source software testing tools such as Selenium and SoapUI have had both positive and negative effects. Traditionally, open source testing tools are laser-focused on solving a very specific problem for a single user. For example, Selenium has become an extremely popular script-based testing tool for testing web interfaces. Yet, although Selenium offers speed and agility, it does not support end-to-end tests across packaged apps, APIs, databases, mobile interfaces, mainframes and other components. There’s no doubt that most of today’s enterprise applications feature a web UI that must be tested. However, in large enterprises, that web interface is just one of many elements of an end-to-end business process. The same limitation applies to SoapUI and API testing.

    So, Now What?

    Software testing must change. Today’s software testing challenges cannot be solved by yesterday’s ALM tools. With disruptive initiatives such as DevOps, continuous delivery and Agile expanding across all industry segments, software testing becomes the centerpiece for data-driven software release decisions. This next wave of software delivery lifecycle (SDLC) maturity requires organizations to revamp antiquated testing processes and tools. This means that organizations must have technologies that enable continuous testing, or innovative ideas will remain hostage to yesterday’s heavyweight testing tools.

    — Wayne Ariola

  • Adopting Continuous Testing: 4 Success Stories

    Adopting Continuous Testing: 4 Success Stories

    Much has been written about the benefits and best practices of continuous testing. However, it’s not always clear how to make these concepts a reality. Here are a few stellar examples of how several leading organizations were able to apply continuous testing to remove roadblocks and achieve their organization’s distinct quality goals.

    The Middle Child Rises – Testing Evolves to Quality Engineering

    Ann Lewis, ExxonMobil

    ExxonMobil’s philosophy is centered on safety, security and controls as the foundation of our 135-year-old company. Based upon our fundamental policy, quality has become a key component in delivering software. We are currently undergoing a shift from traditional project management organizations and support organizations to product teams. This shift has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.

    Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”

    To do this, we started with the basics—quality control manages risk—and then explored what is tracked today and whether quality is being measured. We also removed the roadblock to continuous testing. In the end, the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset; that continuous testing and deployment can only be accomplished through model-based testing; and that managing quality reduces overall cost of ownership.

    Continuous Testing: How Vantiv is Maturing

    Sandra Baker, Vantiv

    Previously, Vantiv shared our automation journey and how we got started. Now, will share how we are maturing as a QA organization and in the use of Tricentis Tosca.

    Our focus has shifted to how we manage and execute our 15,000+ tests. We have focused on unattended execution to achieve maximum benefits. The next step was to further advance in CI/CT/CD and our maturity. To do this, we have introduced new metrics and processes.

    Finally, we are expanding our capabilities to end-to-end testing, performance testing, user acceptance, service virtualization and big data testing.

    How A1 Telekom Austria Speeds its Delivery with Continuous Testing

    Remove featured image

    Christian Fischl, A1 Telekom Austria & Walter Pindhofer, Qualysoft

    The automated tests enable A1 Telekom Austria AG to deploy new features faster and with less risk. This leads to shortened release cycles and more agility. While it is still an ongoing transformation, it has already become a huge success story. This session is all about A1 Telekom Austria AG’s journey to continuous testing, starting with the first automation to the sometimes-tough steps toward automated system regression tests, and a look at the future plans.

    How Varian Achieved Quality @ Speed in Regulated Environments

    Manish Mathuria, Infostretch

    In the new world of digital, agile and DevOps, traditional approaches to software delivery no longer apply. Development and testing teams are under immense pressure to deliver quality products at speed. Varian Medical Systems, a leading manufacturer of medical devices and software for treating cancer, implemented sales and service management solutions using Salesforce, ServiceMax and SAP. To keep up with continuous quality demands in regulated environment, the company partnered with Tricentis and Infostretch to automate testing of key processes. This session highlights Infostretch‘s automation solution using Tricentis Tosca, nuances of testing SaaS models, and key test case, test data and environment management strategies.

    — Jessica Wanivenhaus

  • A Crash Course in Continuous Testing

    A Crash Course in Continuous Testing

    What better way to start a blog on continuous testing than with this “I Love Lucy” clip, and then an analogy:

    https://youtu.be/8NPzLBSBzPI

    Like Lucy and Ethel, many software testers have been scrambling to keep pace with accelerated processes—then along comes the supervisor saying, “You’re doing splendidly … Speed it up!”

    This ever-increasing pace is just one of four aspects of the “perfect storm” driving the need for continuous testing.

    What is Continuous Testing?

    Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.

    Why do we need to rethink our approach to testing?

    The expectations associated with test are changing–but legacy testing platforms aren’t keeping up. Legacy testing platforms take a “heavy” approach to testing. They rely on brittle scripts, deliver slow end-to-end regression test execution and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18 percent, on average–8 percent for enterprises. In a polling question, the audience confirmed that the results of test automation have been so-so, at best.

    Now, software testers are facing increasingly complex applications, delivered at dramatically accelerated paces—and they’re expected to deliver trustworthy go/no-go decisions at the new speed of modern business. More of the same won’t get us there. We need to transform the testing process as deliberately and markedly as we’ve transformed the development process.

    The Differences Between Test Automation and Continuous Testing

    There are three main differences between test automation and continuous testing:

    • Risk
    • Breadth
    • Time

    Risk: Exposing more and more innovative functionality to the user is now a competitive differentiator—but it also increases the number, variety and complexity of potential failure points. Most tests are designed to provide low-level details on whether user stories are correctly implementing the requirements—not high-level assessments of whether a release candidate is too risky to release. Would you instantly stop a release based on test results? If not, your tests aren’t properly aligned with business risks.

    Breadth: Just knowing that a unit test failed or a UI test passed doesn’t tell you whether the overall user experience is impacted by recent application changes. To protect the end user experience, you need tests that are broad enough to detect when an application change inadvertently impacts functionality which users have come to rely on.

    Time: If your organization has adopted DevOps and is performing continuous delivery, software may be released hourly—or even more frequently. In this case, feedback at each stage of the process can’t just be “fast”; it must be nearly instantaneous. If the business wants to minimize the risk of faulty software reaching an end user, you need some way to achieve the necessary level of risk coverage and testing breadth—fast.

    What Are the Key Points of Continuous Testing?

    • Continuous testing’s primary goal is assessing business risk coverage.
    • Continuous testing establishes a safety net that helps the team protect the user.
    • Continuous testing requires a stable test environment to be available on demand.
    • Continuous testing is seamlessly integrated into the software delivery pipeline and DevOps toolchain.
    • Continuous testing delivers actionable feedback appropriate for each stage of the delivery pipeline.

    You can find a more extensive list (14 points) at the What is Continuous Testing page.

    Where Can I Learn More About Continuous Testing?

    Watch the complete continuous testing webinar to get more details on the above topics and also learn:

    • How continuous testing impacts the organization
    • What practices testers need to know to be successful with continuous testing
    • How non-automated practices such as exploratory testing fit into continuous testing

    If you want to learn more, download the white paper, “Continuous Testing vs Test Automation: What Testers Need to Know” or visit the What is Continuous Testing page.

    — Cynthia Dunlop