devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • DevTest Scales Out in the Cloud for 2019

    DevTest Scales Out in the Cloud for 2019

    It’s 2019, which means a whole new year of advancements in all areas of technology. One topic that is near and dear to my “cloud” heart is the acceleration and maturation of development and test (DevTest) scenarios and solutions in the public and hybrid cloud space.
     
    I’ve been talking with folks and presenting on the movement of Dev/Test to the public and hybrid cloud for a couple of years now, and I have progressively learned a couple of things from those with which I have had great and sometimes very frank discussions. First, I learned that the definition of DevTest has expanded exponentially, with the movement to the different clouds. Second, I found that the terminology of “DevTest” just doesn’t fit a single threaded way of thinking any longer, such as “It’s just for developers.” Let’s take a closer look at these.
     
    Back when I was a young technician just starting  in the new world of IT, I was working primarily with programmers (they weren’t called developers or devs back then) who would sit in front of their green screens, writing huge applications with COBOL and RPG, or maybe writing shell scripts on a Unix system. They would test their code in different environments that were always on the same infrastructure that the production code was running (which we all know these days is wrong, right?). They didn’t have secondary hardware or OS installations to test against because it was just too expensive to accommodate and most of the code could not support that separation. 
     
    Fast forward the clock almost 30 years, and now the developers need to have logically separated and secure environments to provide accurate and meaningful testing of the code that is “born in the cloud” as well as those created on-premises. The next few years will bring a big push for developers to have their own self-service environments, extensive code portability and more secure isolation for their code. By utilizing container technologies, automation and rapid deploy processes for their code, more development groups will eventually become their own application development as a service. 
     
    DevTest solutions are also moving far beyond just testing code. Quality Assurance (QA) is already a substantial consumer of the cloud for their load, smoke and regression testing. I predict that there will be a huge advent across all roles within IT—and not just the ones of development and QA—that will take the plunge into the DevTest solutions. They will be running release scenarios on their rapidly and continuously updated OS versions, on all of the software-defined environments, while also adding software patch testing and issue mitigations. 
     
    Knowledge transfer, training and E-Learning also will increase the use of these landscapes over the coming years, with everything from purpose-built labs, globally connected classrooms and even enterprises expanding their hands-on learning management system (LMS) offerings to their employees and customers. 
     
    Finally, I would suggest that there will be an inflection point in the coming years in which the use of private, hybrid and public clouds for DevTest—and, eventually, production—will become transparent to the user to the point that, within the overall vision of what you need to accomplish, what cloud or combinations of clouds you utilize won’t matter any longer. Eventually, for infrastructure as a service (IaaS) at least, the cloud will go the way of hardware, which has largely become a commodity in the new evolution of the modern software-defined data center.
     
    Technology is moving incredibly fast these days, especially when compared to the days of old when I was coming through the ranks in IT. The DevTest solutions will continue to evolve, progress much faster and touch more people in technology—and even other professions—than ever before. Start building a solid DevTest design and implementation plan for whatever your business and IT goals may be, and invest in the solution for all of your IT functions. By embracing and utilizing what is available in the hybrid and public cloud space for DevTest, IT will be empowering more productivity and faster execution moving into 2019 and well beyond.

    — Mike Nelson

  • Split Launches Free Managed Feature-Flag Service

    Split Launches Free Managed Feature-Flag Service

    As part of an effort to attract more DevOps teams, Split is making available a free managed feature-flag service for up to five users within an organization.

    Based on the same commercial managed service Split provides today, company president Trevor Stuart said the free service provides those five users with unlimited feature-flagging and unlimited usage in the hopes that the number of DevOps teams employing feature-flagging will rise to the point where the organization will opt to pay to consume the paid managed service. The free managed service also includes user segmentation capabilities to provide control over how many end users might be exposed to any new feature.

    The paid managed service provides access to additional capabilities such as integrations, access and permissions, team collaboration and full-stack experimentation.

    Stuart noted many DevOps teams are today attempting to incorporate feature-flagging into their processes by implementing one of more than 40 different open source projects. However, many of them quickly discover that building and maintaining a feature-flag platform is a difficult undertaking. Many developers decide that building a feature-flagging platform is a challenge they want to take on because it provides a critical set of DevOps capabilities. But when those developers leave the organization, the DevOps team is suddenly left with a platform no one else in the organization knows how to support, he said.

    Feature flags, also known as feature toggles, provide a means for DevOps teams to test and add new capabilities to their applications without having to take the entire application offline. Feature-flagging also enables phased application rollouts that make it easier for DevOps teams to assess new functionality as it is added. Any problematic update can be turned off with a single click. The downside is they generally result in more branches of applications that need to be managed carefully within the context of a DevOps process. Because of that complexity, vendors such as Split have been making the case for employing platforms that enable DevOps teams to more easily incorporate feature-flagging within a DevOps process.

    Stuart said the ultimate goal is to provide a more controlled approach to experimentation. Instead of randomly testing new features on end users, feature-flagging makes it possible to more easily test features on specific end users, said Stuart.

    It may be a while longer before feature-flagging is adopted more pervasively. But as DevOps teams become increasingly more sophisticated, there’s little doubt feature-flagging won’t become more commonly employed. The simple truth is that uncontrolled application experiments on end users usually are not appreciated. Savvy DevOps teams are finding ways to test application functions in real-world scenarios with minimal disruption to the majority of end users.

    The only real issue now is determining how best to go about implementing feature-flagging at a time when most DevOps teams already are struggling to keep pace with their existing application development and deployment projects.

    — Mike Vizard

  • DevOps Video Chat: DevOps Testing with Ryan Lloyd of SmartBear

    DevOps Video Chat: DevOps Testing with Ryan Lloyd of SmartBear

    The testing landscape in DevOps is a big one, with lots of established players feeding the machine. SmartBear is one of those companies. The force behind Swagger, SoapUI and others is perhaps not as well-known as its brethren in the DevOps testing space, but it cuts a wide swath. And, with its recent acquisition of Zephyr, it’s about to get wider.

    In this DevOps Chat, we speak via video with Ryan Lloyd, VP of Products at SmartBear, about the Zephyr acquisition and the DevOps testing and test management space in general. As usual, the streaming video is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, Container Journal, Security Boulevard, and we’re here for a great interview, hopefully, with Ryan Lloyd, VP of products for testing and development at SmartBear. Ryan, welcome.

    Ryan Lloyd: Thank you. Great to be here.

    Shimel: Great to have you here. So, Ryan, before we jump into today’s news that we wanna discuss, I wanted to just quickly kinda do a level-set. SmartBear is a company I think most people have probably heard of it, but they may not be 100 percent sure of the full breadth of the SmartBear portfolio. So why don’t we talk a little bit about who is SmartBear and what are the kind of products and services you guys offer?

    Lloyd: Yeah, absolutely. So SmartBear software is a company with a lot of history and if you don’t recognize the company brand, it’s very likely that you recognize one or more of our product brands, so we are the company behind some of the key product brands like Swagger, SoapUI, CrossBrowserTesting, TestComplete, so quite a diverse portfolio of products focused on helping our customers who’re building software. Help them build tests and monitor their software applications.

    Shimel: Great. So, Ryan, big news out today is SmartBear had made another acquisition. You guys have acquired Zephyr. Is that correct?

    Lloyd: That’s right. Yeah, we’ve announced our intent to acquire Zephyr, a company in a similar market space as us.

    Shimel: So it’s a similar market space, but let’s get specific. What does Zephyr do?

    Lloyd: Great. So Zephyr is a collection of tools for testing and specifically focused on the area of test management, managing the process of testing.

    Shimel: Great. And you mentioned an intent to inquire – acquire, so I guess that means it hasn’t yet closed, but you’re both private companies and I would imagine it closes fairly quickly.

    Lloyd: That’s right. We expect in the next 30 days, subject to the usual regulatory approvals.

    Shimel: Yep. Fantastic. So, you know, Ryan, one of the things I read in the background around this one was that one of the big pluses here is Zephyr’s presence within the Atlassian, the Atlassian ecosystem, and SmartBear actually is already a big player in that Atlassian ecosystem. Well, aren’t they?

    Lloyd: Yeah, so we have a couple of products that have plug-ins to the Atlassian ecosystem. I mean, the reality is 60, 70 percent of our customers are using Atlassian’s tools. They’re pretty dominant in the area of software teams, for managing some part of the software development process, so it’s a very strategic and important ecosystem for SmartBear as a company. And, certainly, Zephyr improves our ability to reach that ecosystem of users and make sure that they’re aware of all of the tools and capabilities that SmartBear can offer.

    Shimel: Yep. And, Ryan, you know, 60 to 70 percent use Atlassian. I’m gonna imagine the overwhelming majority are using Jira and then maybe Jira Plus.

    Lloyd: Jira –

    Shimel: Correct?

    Lloyd: Certainly, Jira is probably the most prevalent. Confluence gets tagged on with Jira quite a lot. And then BitBucket, actually, is also becoming quite popular as well. BitBucket, GitHub, GitLab – they’re sort of three of the same, in terms of enterprise solutions for developers.

    Shimel: Got it. So I don’t wanna turn this into being about Atlassian and their products, but, obviously, they do have a big ecosystem. There are a lot of third parties, such as SmartBear and Zephyr, who move a lot of services and product via that marketplace. But let’s talk a little bit similar, but where’s the overlap? Or is there any overlap between what Zephyr and SmartBear’s offerings are about?

    Lloyd: Yeah, so SmartBear offers a broad portfolio of products and, certainly, a couple of the products we have, pre-acquisition, were in this test-management space. What Zephyr brings to the table, though, is a solution that is natively built around Jira and delivered through the marketplace. So what that means, to be delivered through the marketplace, is it’s a native Jira solution that Jira administrators and customers can install directly from within their Jira instance. So it provides a very seamless extension of their workflows, within Jira, to handle some of these test-management scenarios, which we think is really important to teams that are trying to consolidate their processes and have a unified approach to that.

    Shimel: Excellent. I guess one of the issues – and, with your title and your responsibilities around VP of products for test-and-dev, just in a number, how many products are there in test-and-dev now, within SmartBear and then if we add the Zephyr suite to the mix?

    Lloyd: Yeah, so we had five products in that portfolio. Today, Zephyr brings three additional products into that portfolio, so we’re now up to eight products in our test and development business, as well as a range of products in our other business unit, around API.

    Shimel: You know what? That’s a lot of products to be managing, huh?

    Lloyd: It is. I mean, SmartBear has grown a lot over the years, through both acquisition of companies and building new organic products and bringing them to market. It’s pretty exciting time for us. We’re, pre-acquisition, at about 450 employees globally, so it’s an exciting time to be part of this company.

    Shimel: Absolutely is, man. So, you know, another not necessarily unique but another kind of factor you have to play into the SmartBear – and I don’t know about Zephyr as much – is that some of your solutions have open source kind of versions out there that you guys help manage the open source communities around these, as well as the commercial versions of the products themselves. Correct?

    Lloyd: Yeah, that’s right. I mean, core to our business is really a couple of philosophies. One is we wanna be open and interoperable with all the tools that are part of the ecosystems that software teams are using. And so, in some cases, that means commercial opportunities to partner with companies like Atlassian; in other cases, it means we want to be a part of an existing open-source ecosystem or help build up and nurture an open-source ecosystem. So, certainly, when we brought SoapUI and Swagger into our product portfolios, that was all about really taking that existing organic ecosystem, where developers and testers are going through tools, and helping build and nurture and develop that ecosystem.

    Shimel: Excellent. So what kind of special challenges – you specifically, right? You own the products; you’re VP of products here. What challenges does that pose for you, having these open source kinda versions and tools and communities. I mean, you don’t wanna burn them, right? Because, ultimately, you kinda bite the hand that feeds you. So let’s talk about that. What does that – how does that manifest itself with you?

    Lloyd: Yeah, certainly. I think there’s a brand and a culture that exists around open-source products and I guess it’s our responsibility, as stewards of those brands, to really continue to nurture that community. And that means we’ve gotta make balanced decisions about where we choose to add features to the open source products, versus where we commercialize some of those features, so there’s a constant balance there and kind of evaluating and making sure we’re engaging with the community.

    And then, more broadly, beyond just our open-source products, I think one of the biggest challenges in the software tools space and market is the interoperability that’s required between tools these days. You know, developers and testers, they’re demanding practitioners. They want their tools to work together and there’s a huge diversity of tools out there in the market. So, as a product manager, that’s what keeps me up at night, is making sure all these tools are compatible and working to support these complex workflows.

    Shimel: Absolutely. I mean, one way of looking at it is, if there wasn’t such a big market, there wouldn’t be all these tools.

    Lloyd: That’s true.

    Shimel: Another way of looking at it is also that it presents a tremendous opportunity for consolidation and for one or a few products to emerge as true market leaders. And then you’ll see it kinda – you know, you’ll see the amount of products in the market kinda consolidate. You know the old saying, “If you’re not – you’re either leading, following, or get out of the way,” right? If you’re not top-three, you gotta ask yourself, “Do you wanna stay in that market?” But, you know, when we talk about the testing market – and, in DevOps, we talk about continuous testing – it certainly is a huge, huge market here.

    Lloyd: Yes, it is. And it’s one that’s continually evolving and shaping, right? So there’s traditional test tools that have been around for a decade or more, and then there’s sort of emerging products that are coming to market. And I think it’s how all these work together and evolve to support the change in workflows and, yeah, even just organizational changes within environments.

    Shimel: Yep. So another area of the testing space, Ryan, that I’m interested in personally is security and security testing. And it’s really funny when I – you know, 20 years ago, when I was into security, when I first got into security, info-sec, QA was here and security was there and never the twain shall meet, right? But now we’re starting to see security become synonymous with quality – that’s the “Q” in QA.

    Lloyd: Yeah.

    Shimel: And so security testing and other kinds of testings are starting to merge. What do you guys see with that at SmartBear?

    Lloyd: Yeah, I think what’s driving a lot of that is the Agile workflows that teams are adopting, so, as they increase the pace of trying to deliver and release software, they’re being forced to evaluate quality and security earlier in the life cycle and more from the context of the development team, rather than having a separate, isolated team that deals with that downstream or after the fact.

    Shimel: Correct.

    Lloyd: So I think that’s the driver, so definitely a lot of demand for your Agile workflows to support testing by design and security by design. From our perspective, I think that opens up some unique opportunities for adjacencies and areas that SmartBear can continue to expand to in the future.

    Shimel: Yeah. I mean, so, you know, what you’re describing is almost the classic “shift left.” Right? As we start shifting testing, security and security testing, left early into the development life cycle. But it’s also the reality of it’s been called the “modern software factory.” Right? It’s an assembly line and is a production line, for software, and where traditional QA, traditional testing, security testing, where they fit into that life cycle has shifted left.

    But also is – you know, where we used to think, oh, QA was going away and security people were gonna give up control of security. Not the case. They’ve become more important, right? But also highly automated where they can be and clearly defined as part of this factory, if you will.

    Lloyd: Yeah, it’s really just about reshaping the responsibilities and improving the collaboration across the whole unit or team that’s now responsible for delivering iterations of software.

    Shimel: So, Ryan, then – so that begs the question then, “What does that mean for you as a product manager?” Right? Where you used to sell to a QA engineer or security testing sold to a security person, but now your audience includes the guys using Jira and BitBucket and a bunch of other tools. It’s not just Atlassian, obviously. So you have new customers to serve and to service. Now how do –

    Lloyd: Yeah, that’s absolutely right. Yeah, so, over the last several years, we’ve been focusing on how to modernize our portfolio to support those sort of emerging needs, both the process change that’s going on with our customers and the persona change that’s happening in many instances. So the acquisition of Swagger is a great example of us providing tools that help developers earlier in the life cycle, so, where we had SoapUI to help with API testing from a QA perspective, Swagger aims to solve some of that further upstream by enabling more collaboration around the definition of those APIs to begin with. And, oh, by the way, making sure that interoperates with SoapUI, so you get leverage out of that API definition to generate some of those tests, automate it, and it triggers sort of a seamless workflow and thread.

    And so we’re continuing down this path of modernizing our portfolio. We acquired CrossBrowserTesting about two years ago, to help companies address some of the modern types of applications that they’re looking to test, integrated more deeply with Selenium, which has become hugely popular in sort of the developer organizations, to drive automated Web and mobile testing. And then, with Hiptest and Zephyr, again, it’s furthering our relevance for those software teams that are on that journey.

    Shimel: Excellent. Ryan, last kind of issue I – or not issue – it’s not an issue at all – but the last area I wanted to talk to you about was so you’ve got five. There’s three more coming in. And that’s just test-and-dev; you have an AP kind of silo as well within SmartBear. How do you – I mean, I’ve been in the boat not quite as many as you, but, when you have so many different products and then a company name, how do you brand? Right? Do you continue branding the individual products? Do you throw SmartBear in front of all of them, but some of ’em already have names that may be, in fact, bigger than SmartBear, as you said in the beginning? Again, as a product manager, you have to think about that a bit. You know, what’s the right answer there, if there is –

    Lloyd: Yeah, so I think SmartBear is going through a bit of a transformation, in that regard. You know, as I said and as you pointed out, we’ve had very strong product brands up to this point. We wanna continue to nurture those product brands, so no intent for those to go away because we love the high-velocity, try-buy model we have around our products, where developers and the practitioners can find the right tool for the job and get started.

    But we’ve gone through an exercise on how we can integrate our products in the right way, so like the Swagger, SoapUI example I provided, to create those sort of interesting workflows. And we’re building up the SmartBear Academy, our SmartBear community, our big user conference, to help elevate the SmartBear brand and create more awareness of the portfolio of products we have – how they work together, how they fit into the ecosystem. And I think we’re just at the start of that journey right now. We’ll see a lot more, as we launch into the year next year, around what we’re doing from a brand perspective to galvanize the company and all of the great products we have.

    Shimel: Excellent. Well, Ryan, we’re about out of time, but, first of all, congratulations on the Zephyr announcement and anticipated acquisition here. It’ll make for some good times. And continued success with SmartBear and everything you guys have going on.

    Lloyd: I appreciate it. All right. Great talking to you, Alan.

    Shimel: All right. Ryan Lloyd, VP products, test and development, for SmartBear, on SmartBear’s acquisition of Zephyr, here on devopsy.kinsta.cloud. And we’ll talk to you soon, everyone. Have a great day.

    — Alan Shimel

  • DevOps Chat: Crowdsourced Continuous Testing with Rainforest’s Derek Choy

    DevOps Chat: Crowdsourced Continuous Testing with Rainforest’s Derek Choy

    Scaling your testing is the scourge of many software teams. While test automation and continuous testing have made great strides, it still is difficult to automate every QA test. Also, engineering resources who can script these tests are difficult to find. Enter Rainforest.

    Rainforest has a unique crowdsourcing solution to continuous testing. It combines crowdsourcing talent with automation to deliver a high level of quality assurance on a continuous basis.

    In this DevOps Chat, we speak with CIO Derek Choy as he explains to us why Rainforest in some ways can be the Uber of QA. As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features a first-time guest on DevOps Chat. He’s Derek Choy, CEO at Rainforest, or Rainforest QA, as some may know them. Derek, welcome.

    Derek Choy: Thank you. Just a little correction: I’m CIO at Rainforest.

    Shimel: What did I call you?

    Choy: Responsible for primarily the technical side of the functions here.

    Shimel: Did I call you “CEO”?

    Choy: I think you did, but I just want to make a little correction there.

    Shimel: You know, it’s 2019; that’s my prediction.

    [Laughter]

    Choy: I don’t know that I would support that prediction, but thank you. [Laughs]

    Shimel: Well, I _____ – CIO’s enough. But, anyway, Derek, thanks. Yes, it’s Derek Choy, CIO at Rainforest. So, Derek, you know what? I didn’t make the mistake on purpose; it was an honest mistake. Good. That leads us – why don’t we talk a little bit about Rainforest for our audience who may not be familiar?

    Choy: Absolutely. So Rainforest is a testing platform. What we do is we help our customers with their software testing. You know, as you know, we’re moving – a lot of companies are moving towards a rapid development cycle, CI/CD, frequent releases and stuff. Testing becomes much of a challenge for a lot of our customers because now they not only have to do testing, but they have to do it fast. They have to have accurate results and stuff.

    So Rainforest is actually a crowd-based testing platform that allows our customers to simply input test cases in plain English, so no Selenium or any sort of programming language, and then we can actually execute them as if they are automated test. We’re fully integrated into any CI/CD tools, like Jenkins or CircleCI and so on, so you can actually trigger a Rainforest test. Typically, we can return results in 30 minutes or so, and you can actually test across many different platforms, like different browsers, different OS’s, and the same for mobile as well. And so that allows our customers a wide variety of options, in terms of where they want to run their plain-English test cases and when they want to run it – it’s on demand. And we typically return results in 30 minutes, fully integrated into an automated release cycle.

    Shimel: Great. Excellent. And, Derek, how did it come to be that you became the CIO of Rainforest? Let’s hear a little bit your own personal journey.

    Choy: Cool. Oh, thank you. So I have been always involved in the product engineering side of things. I started off my career earlier with Accenture. Spent a number of years at eBay. And then my last company at Aria Systems, building SAS-based billing system, also running their engineering and product function. So, coming to Rainforest, I brought my experience and knowledge in this space, to help build the engineering and the product organizations. Now here, besides product engineering, I’m also responsible for our professional services team and our client success team. So essentially kind of helping build out product and making sure that our customers are successful in using that.

    Shimel: Absolutely. So, Derek, you know what? Here we are in December. We’re looking at 2019.

    Choy: Mm-hmm.

    Shimel: I probably have spent the last three weeks – if I tell you I’ve spoken to dozens of vendors and C-level execs at large organizations, and, to me, the real story of 2019, as well as what’s emerged as the story of 2018, is around – you know, you wanna call it “digital transformation” and “DevOps,” _____ say great, but, really, down to these two things: companies are – they need scale in what they’re doing today, right?

    Choy: Yeah, absolutely.

    Shimel: When we’re talking about data and computation and transactions or whatever the metric is that’s driving your company, everybody wants to do it at huge scale, like unprecedented scale that, 20 years ago, you would’ve went – you know, I’m just reminded of Carl Sagan saying, “Billions and billions of stars,” _____.

    Choy: [Laughs] Yes, that’s right.

    Shimel: That’s the scale. And the only way to achieve that kind of scale is automation.

    Choy: Yes. Absolutely.

    Shimel: So these are two kind of twins, co-joined at the hip – scale and automation.You can’t have scale, I think – well, you can’t handle scale without automation.

    Choy: Absolutely. Yeah. And – sorry.

    Shimel: Go ahead. No, no, go ahead.

    Choy: I –

    Shimel: You know, to me, that’s it right there. That’s the crux of it.

    Choy: Exactly. And, Alan, I would say that I’ve never been more excited, right, than where we are right now. I think we are definitely at a very exciting time and, just to kinda set the context and maybe kinda recapping a little bit of what you just said, right, the world is very different now, even compared to five years ago. Right? A while ago, you would hear companies having a quarterly release. Some would even have an annual release, if you remember, especially the bigger companies, right? You know, release cycle was a lot longer, takes a lot more time to get things out to the market. Today, release cycles are typically maybe in a month, right? Maybe two weeks and, in some cases, days or even multiple times a day. Right?

    And exactly to you – and the interesting thing is that the world has got accustomed to that and has come to expect that, frankly. Right? Think about the apps that you’re using. If they come back to you and say, “Well, you know what? We update our apps once every six months,” or something, you’re like, “Wait, six months? Wow, that’s unheard of.” Right? “I was expecting a weekly update or at least monthly or something.” So the world has actually comes to expect that.

    Now the challenge, exactly as you pointed out, is how do companies deal with that? Right? How can we scale our organization through automation, among other things, right, to make sure that we can keep up with that pace? I think that that is the challenge with a lot of technical leaders, is we understand what the companies are looking for, understand what the consumers and our customers are looking for, right? But how do we do that? Right?

    And there’re different aspects of it, Alan. I think that there is definitely a development aspect of it, right? How do we scale development? How do we do rapid development, Agile, sprint? You know, all these things. Things have evolved a lot over the last, I would say, ten years, in terms of faster development. There’s a lot _____ –

    Shimel: On the development side?

    Choy: Exactly. And there continue to be more evolution on that, which is super exciting. Now one thing that hasn’t really, at least in my opinion, gone as far in this particular context is testing. Right? If you think about testing, we’ve had automated testing for a long time – 10, 20 years, maybe even longer, right? We’ve always had manual testing, right?

    Now, when you think about testing, that’s – which is a critical part of the release cycle. Of course, right? You cannot push things out without some sort of testing at least, right? There hasn’t really been a lot of innovations and evolutions in testing. We’re still trying to do automated testing; we’re still doing some manual testing; we’re still doing it kinda the same way, so to speak. There’re new automated testing language, but they all kinda do similar things, right? And they all have similar –

    Shimel: Absolutely. And, Derek, let me stop you there for a second.

    Choy: Yeah.

    Shimel: To me, here’s the real crux in the issue.

    Choy: Yeah.

    Shimel: We talked about that we’ve certainly seen, let’s call it, a “revolution” in development acceleration, rapid development over, let’s say, 10 years. I’ll say even 20 years, right? _____ _____ _____ around for about 20 years.

    Choy: Agreed.

    Shimel: Do you consider testing part of development? Or did DevOps come along and say, “Wait a second, it’s all one continuity here, with development, testing, operations, security – you wanna put security next to testing, whatever – but it’s one continuum”?

    Choy: Yeah. It’s definitely – I completely agree with that. It is definitely one continuum, meaning that they are all part of the overall workflow, right, to make automation possible, to make a fast release process possible. You cannot do it without any one of those components, right?

    Shimel: No. But they haven’t kept pace with the changes that we’re in – what we consider when we talk development.

    Choy: Yes.

    Shimel: Where many of us think, you know, of coding.

    Choy: Exactly.

    Shimel: That’s the issue right there.

    Choy: That’s the issue, exactly as you pointed out, right? All the components of that whole continuum hasn’t really kept pace, right, with the developments that we’ve made on coding, right? On the actual coding of the product. And I think that that’s starting to show or actually has been starting to show, in terms of how other components of that, testing in particular, has been frankly dragging down the overall speed of being able to push things out faster and so on, so forth. Right?

    If you talk to – and you do that all the time. If you talk to a lot of CIOs and CTOs and so on, frequently, you’ll hear challenges with testing. Right? And most people will tell you, “Yeah, I’m okay. I get it going,” but very few people –

    Shimel: “It’s good enough.”

    Choy: Yeah, it’s just good enough, but it’s not great, right?

    Shimel: No, it’s not.

    Choy: Yeah.

    Shimel: And, Derek, I’ll go one further with you. What I see is this, is that, when we talk about testing – and _____ _____ the same thing with security, right? Is we’ve got twin challenges.

    Choy: Yes.

    Shimel: I can’t do it fast enough, so I’m constantly behind, running as fast as I can, but I can’t – just not fast enough.

    Choy: Exactly.

    Shimel: And then, number two, the train won’t wait for me.

    Choy: Yeah. Exactly.

    Shimel: If I can’t finish my testing, “Well, you did it good enough.”

    Choy: Yeah. “Let’s go.”

    Shimel: “We’ve gotta release. We’ve gotta release this hour,” or, “We’ve gotta release this week.” You didn’t do – dirty testing? That’s okay; it’s probably secure anyway. And, boom, out the door it goes, right? And that’s the dilemma, the enigma of testing, is “I can’t do it fast enough, but, if I don’t do it fast enough, they don’t wait for me.”

    Choy: Exactly. Right. And there is a reason for that. Exactly, right? Like we talked about earlier, because customers actually expect releases, right, whether you’re ready or not, right? They are expecting releases in a very frequent cycle. Right? And so companies therefore cannot wait. Right? It is not acceptable to go back to the customer and say, “You know what? My release is delayed and therefore you have to wait another week or two.” In the past, that used to be the common thing to do, right? Nowadays, that’s not acceptable.

    And so then what happen, exactly like you said, is that people compromise, right? They’re like, “Oh, you know what? Testing is not done. Okay, well, you know what? It should be fine. Let’s just go.” Right? “Security testing is not complete, but we’ve done a portion,” right? “Okay, well, let’s just go.” Right?

    The downside, however, is that – and you’re seeing it frequently; some people are just not paying attention to it – is that there is a high price to pay for that, right? You’ve seen a lot of security loopholes out there, right? And, yes, the release is out, right, but then at what cost, right? If people found a loophole and there is a data breach and costs companies lots of moneys and reputations and things like that.

    And equally with software quality as well, right? Things don’t work. Even bigger companies, you’ve heard companies like Apple, for example, struggle with that. Google, right, struggle with that. And they release something and people like, “Wow, this is junk,” right? “And I don’t want to upgrade,” and you hear that all the time, right? And it tarnish your reputation.

    Now bigger companies have, obviously, a lot more buffer to kind of weather these kind of quality problems, right? You know, whether you like it or not, you’re gonna continue to use Apple, right? Or Android or whatever. But then smaller businesses may have less tolerance for that, right? And customers of those businesses, if they’re not happy with your quality, they’ll move somewhere else.

    So I think it’s very important for technical leaders to figure that out and not just kinda brush it aside and say, “You know what? It’s not important. If quality’s not good enough, let’s just go. It’s fine. We’ll find bugs in production; we’ll fix it.” Personally, I don’t think that that’s the right approach and I think that it’s gonna hurt businesses down the road.

    Shimel: So you know what I call that? So that’s the double-edged sword of MVP.

    Choy: Yeah.

    Shimel: Right? ‘Cause you talk to these people and they say, “Don’t worry. We’re just doing an MVP. It’s a minimally viable product. We know it’s not 100-percent tested. We know it’s not a 100-percent baked. We know it doesn’t have everything, but we’ve gotta get an MVP out there, god darn it.” Right? [Chuckles] And so –

    Choy: Yeah.

    Shimel: And we appreciate – look, I’ve been – I founded, cofounded several venture-backed companies. I’ve been on pushing the VP of engineering, saying, “Hey, we promised our customer we had a new release coming out before Thanksgiving, man, and it has to get out. It has to get out. I don’t care if it’s fully tested. Get it out.”

    Choy: Yeah.

    Shimel: It’s not – you know, MVPs were being chewed to death.

    Choy: Yeah. I – [laughs]

    Shimel: So that “minimum” – that “M” in MVP keeps getting bigger and bigger and bigger. Right?

    Choy: Exactly. Right, exactly.

    Shimel: And more and more critical.

    Choy: Exactly. I think we continue to push the boundary of MVP. You know, I think that – and that’s really the challenge, right? So, you being kind of the leader of the business, right, you’re not wrong because customers really need it, right? “In order to move my business forward, I really need the release. I really need to get that feature out. I really need to get ahead of the market.” That’s not wrong. Right? And I think that that’s the needs of the business; that’s the market; that’s what people expect today. The challenge that we technical leaders needs to solve – CTOs, CIO, VP of engineering, VP of product – what we need to solve is “How can we support the business in doing that but also maintain the quality?” Right?

    And so, as a leader, in my opinion, going back to the business and say, “You know what? I can’t actually make your delivery date. You will have to wait another two weeks, four weeks, six weeks,” or something – that’s gonna be a tough sell, right? But, at the same time, releasing it without quality assurance, without security and all these stuff, is also not the right approach.

    So I think what we leaders needs to figure out is “How can we do better, right, in QA? How can we do better in security?” All these other things that are probably less mature than development itself, coding itself, right, how can we do better in these areas so that we can also support the business in shipping this MVP? Whatever the definition of this MVP is, right? We can support a business in shipping the MVP because that’s very critical to the business but, at the same time, maintain the level of quality and the level of security and, among other things, right, that we are proud of, right? That we want to have, right?

    And I think that that’s the biggest thing that we need to figure out in 2018 and definitely going into 2019, right? How fo businesses figure that out? How do we implement processes and tools and technologies that will allow us to do that?

    Shimel: Yep. So, Warren – Warren? I’m sorry. Derek, fantastic discussion. Let’s bring it home a little bit to Rainforest.

    Choy: Sure.

    Shimel: What can you do to help?

    Choy: Yeah, so Rainforest has a very – takes a very different approach to QA, right? You know, so the traditional way is, obviously, automated test with Selenium and all these different Appium, all these different technologies, or we can go manual testing and perhaps building an AMI offshore – that’s very, very typical as well. Rainforest looks at it and say, “Well, neither one are the best approach,” right? And therefore we come up with a testing platform, you know, that is powered by the crowd. And we allow our customers to actually put plain-English test cases on our platform. We can execute these tests without actually building an AMI offshore, but, at the same time, you don’t have to worry about the downside of automation, having to hire QEs and spend a lot of time writing Selenium scripts and things like that.

    And so we offer a solution that, at least we think, solves problems on both sides, right, and also allows you to hook it up into your CI/CD process so that you can actually push this MVP out in an automated fashion, fully kinda connected with all the rest of the continuum, right, and be able to actually ship code faster but with higher quality.

    Shimel: So let me ask, so, at some level, though, you are reintroducing or relying on human element?

    Choy: Right. Correct.

    Shimel: So why don’t you have the same scale problems that others do?

    Choy: That is an excellent question. So I would say that there are two elements, right? So, when we look at the human elements, we look at it from a crowdsourcing basis, right? So think about Uber, for example, right? If I were to hire my own team of drivers, that would be a little bit challenging, but then, leveraging the crowd and scaling it based on demand, we’re able to actually solve a lot of the human aspect of the problem. We can have testers that come in whenever they like to come in and we have enough testers to support the test that our customers would like to do. Right?

    And now the challenge there and probably a followup question then is “Well, but what about the quality of these people? Do they know what they’re doing?” Just like a Uber driver, right? How do you know that that guy actually drives well, right? So we actually developed a bunch of AI algorithms that actually analyze the work that they do and also take in feedback from customers, based on their former – the previous tests that they have done. And a combination of that with a lot of data mining and AI analysis helps our customers determine whether that test result is correct or not. If it’s not correct, we will engage additional testers to make sure that the tests are consistent and results are correct.

    So we have refined that algorithm over the years and now we have achieved a very high level of accuracy that, based on our data, is showing us that the accuracy’s actually higher than having your own manual QA team. And so you get better results, without actually hiring an AMI, either onshore or offshore. You can actually run your test on demand – typically, you’ll get results in 30 minutes or so. Right? And it’s fully integrated into your CI/CD cycle. And the best part is you can do all these things and achieve all these benefits without having to hire QEs to write automated scripts and having to maintain it, and so on, so forth.

    Shimel: Excellent. So, Derek, I get the outsource. I get the army of people doing it. But, on the back end, are you doing some automated testing as well?

    Choy: Yes, we are. And so what we’re also doing is we’re developing additional AI algorithms to actually learn from these human executions. So what we do is we have lots of human executions that are happening as we speak. In our environment, we have 80 million, 90 millions of tests that have already conducted on our platform, so what we do is we actually do machine learning through these data and learn from previous executions and develop algorithms so that, instead of actually having to send it to a human tester, we can actually achieve it with a robot.

    Very similar to the approach that Uber is taking. Today, human drivers are driving people around, right? In the back end, they’re developing these self-driving cars so that, in the future, we don’t really need a human drivers anymore. Very, very similar approach, from a QA standpoint. Today, we’re sending testers, but then, in the future, we’ll have these robots that will execute these tests based on what they have learned from these human testers.

    And tomorrow is already here for us today because we’re already doing that and a number of our customers are already using these automated AI algorithms. So they just need to write their test cases in plain English and they’ll see the tests executed by robots automatically, without having to worry about building automations or anything like that. So that’s kind of our approach to solve the QA problem.

    Shimel: So Rainforest, the Uber of QA.

    Choy: I don’t know that marketing will approve that line, but –

    Shimel: No, I don’t think _____ _____ –

    Choy: – personally, but, yes. Yes.

    Shimel: It does remind me – you know, when – you’re out in San Francisco, we’ve all been in venture-backed world.

    Choy: That’s right.

    Shimel: There was a period there where every pitch was “We wanna be the Uber of,” right?

    Choy: Yeah. Exactly.

    Shimel: Uber of this or the Uber of that and _____ _____ to be disruption, or crowdsource in this case, but fascinating take on solving an age-old problem.

    Choy: Mm-hmm. Exactly.

    Shimel: Interesting to see. Just – Rainforest is, what, about two-and-a-half years, three years old now?

    Choy: Yeah, it’s just a little bit longer. We’ve been – our total existence, probably about six, seven years. We’ve kinda pivoted a little bit at the beginning, so our current product is about three to four years old.

    Shimel: That’s when I think I first became aware of it.

    Choy: That’s right.

    Shimel: And, look, testing and continuous testing is such a – I mean, it’s a hot area. There’s a lot of different solutions vying in the market, but this one is really kinda unique, so it’ll be interesting to watch how it happens.

    Choy: Yeah.

    Shimel: Derek, as I promised you before we started, the time goes really fast here. We’re way out of time, but, hey, man, I wanna thank you for being a guest on DevOps Chat. And maybe we’ll catch up with you again in the new year and find out how things are going.

    Choy: That will be awesome. Well, thank you, Alan, and have a wonderful holiday to you and everybody listening. And, hopefully, I’ll talk to you again soon.

    Shimel: Absolutely. Derek Choy, CIO still, not CEO, of Rainforest. Thanks for being our guest on DevOps Chat today. Have a great holiday and thanks very much.

    Choy: Thank you. Buh-bye.

    Shimel: Buh-bye. This is Alan Shimel for devopsy.kinsta.cloud and you’ve just listened to another DevOps Chat.

    — Alan Shimel

  • The 4 Keys to Effective Continuous Testing

    The 4 Keys to Effective Continuous Testing

    In the digital era, products no longer can win on capabilities alone. Speed to market and buyer experience now matter just as much to customers as the extent to which a product meets a pressing need or outperforms competitive offerings. Against this backdrop, software developers find themselves under increasing pressure, left with no choice but to develop apps that provide everything customers want, are delivered on time and are continuously updated, all while ensuring a user experience that’s never short of flawless. There’s simply no margin for error, and that’s why more developers than ever are turning to continuous testing.

    Unlike the traditional approach of testing performed at the end of the development cycle, where it can potentially cause production delays should issues or bugs be uncovered, continuous testing involves running tests throughout the entire software development process. That enables developers to identify and resolve existing issues immediately—and, more importantly, deliver continuously updated and always flawless applications to their customers.

    Fundamentally changing something you’ve always done a certain way is never easy, however, and continuous testing is no exception. There’s no single lever to pull, but rather a series of steps and best practices necessary to effectively implement it within your development team. Let’s walk through the four most critical of those best practices.

    Make a Cultural Commitment to Collaboration and Quality

    As with any transformational change, people come before processes and developing the right mindset comes before developing the right tactics. Effective continuous testing requires a teamwide, cultural commitment to collaboration and quality. The notion that developers own development and QA owns testing inevitably leads to the types of problems—namely, delayed releases and flawed user experiences—that businesses simply can’t afford in the digital era. Instead, developers and test engineers must have a mutual understanding of the baseline level of quality and efficiency they collectively intend to maintain throughout the development process and a shared understanding that testing is everyone’s responsibility.

    Apply a Broad Range of Tests, Anchored by Automation

    As mentioned earlier, continuous testing isn’t about pulling any single lever, but rather about applying a broad range of tests throughout the testing process, including unit testing, integration testing, automated end-to-end and regression testing and production testing. By continuously administering tests at various points throughout the delivery pipeline, you can identify and fix issues spanning functionality, performance and security, and do so before they morph into the kind of roadblocks that delay your release and cost your business money.

    Though you should strive for variety when it comes to the types of tests you implement, you should strive for consistency when it comes to the concept of automation. If you’re serious about succeeding with continuous testing, automation is a necessity. Automated testing, which involves writing a test script (typically in Selenium or Appium) that interacts with the functional elements on a web page without requiring human intervention, not only helps you deliver better scalability and reliability than manual testing, but it also frees your manual testers to focus on more value-add activities such as exploratory testing.

    Leverage Cloud for Breadth and Scalability

    Just setting up and maintaining the infrastructure, physical or virtual, necessary to test across the ever-expanding universe of browsers, operating systems and mobile platforms is both costly and complex. And that’s before you consider the need to scale your test environment to run tests in parallel. It’s a daunting challenge for even the most mature enterprise development team.

    Enter cloud. In many ways, continuous testing and the cloud were made for each other. By leveraging a cloud-based, cross-browser testing platform, development teams can offload all the infrastructure-related headaches they’d otherwise be forced to grapple with on a daily basis, test across virtually any browser/OS combination (leveraging both real devices and cloud-based simulators and emulators) and ensure they have the scale necessary to keep their test times to a minimum.

    Develop a Strategic Approach to Analytics

    Continuous testing, applied optimally, requires a strategic approach to analytics. Developers should actively look to integrate a layer of analytics within the testing process, such that insights can be applied across the product team. Smartly layered analytics can shed light on a number of functional issues, including which browsers and mobile platforms are most often experiencing failure and whether an application’s test results are consistent.

    Beyond just functional testing, teams also can leverage analytics to assess user experience and performance, examining, for example, how the user experience is impacted by heavier traffic loads and how overall application performance is impacted by new updates to the code. Having this added layer of analytic insight enables developers to better understand the entire quality picture and deliver improvements that can be applied both in real-time and to future tests.

    The Digital Age Demands Continuous Testing

    In the digital age, your ability to attract and retain customers is directly tied to your ability to build the best products, deliver them faster than your competitors and keep them continuously updated with all the latest features and function—and do it all within the context of an impeccable digital experience. It’s a tall order, but it’s one made considerably less daunting by the effective application of continuous testing.

    — Lubos Parobek

  • Reinvent Testing to Galvanize Digital Transformation

    Reinvent Testing to Galvanize Digital Transformation

    We’re well past software “eating the world.” According to Tricentis CEO Sandeep Johri, software has become irreversibly intertwined with the world. You can’t find a modern enterprise that doesn’t depend on software, or a business transaction that doesn’t require software at some point in the end-to-end process. In the new world of digital business, companies that deliver differentiating customer/user experiences through software have the clear competitive advantage. Those that cannot are living on borrowed time.

    In response, every CIO is now focused on digital transformation initiatives that ensure the company is disrupting, not disrupted. From an IT perspective, this requires faster delivery of innovative software and greater agility—the ability to pivot as soon as you identify a new opportunity or challenge.

    Everyone agrees that this is a priority. Developers are producing software faster than ever before, and Ops can now deliver applications in a matter of seconds. So, what’s holding us back? Testing.

    When Johri talks to CIOs about how their digital transformation is progressing, the same basic story always emerges: Organizations transform Dev and Ops, but then testing cannot keep pace … and they get stuck. The tools and processes architected for traditional months-long release schedules simply don’t fit modern delivery cadences, which require immediate quality feedback with each new build. It’s a sad but simple fact: If you have a slow testing process standing between highly accelerated development and operations processes, there’s just no way that you can achieve the desired delivery speed.

    Many companies have already recognized this—and they’ve galvanized their digital transformation initiatives by modernizing their testing. By re-examining and re-inventing software testing across their organization, they went beyond removing the “testing bottleneck”—they’re also driving the positive user experiences that can now make or break a business.

    Many of these distinguished quality leaders joined thousands of software testing professionals from around the world at Accelerate 2018 in Vienna, Austria. This year, the conference theme was “Digital Transformation Requires Continuous Testing.” To ensure that the conference thoroughly covered both the digital transformation and continuous testing sides of that topic—as well as the core intersection between them—we invited CIOs as well as quality leaders to share their perspectives. Here are some of the key insights that are most pertinent for executives leading digital transformation initiatives.

    Archaic Testing Approaches Can Cause a ‘Brain Drain’

    Merck sought out top development talent to advance their digital transformation initiatives. When those developers started quitting, (now retired) Merck CIO Clark Golestani wanted to know why. He learned that the No. 1 reason for these developers leaving the company was the company’s testing practices. Developers were “fed up” because the company was still using traditional testing practices within Agile development processes—and developers believed it was undermining their hard work and dedication.

    That’s the precise moment that Merck launched its testing transformation. Ultimately, the testing organization changed the way testing was done, dramatically reducing attrition and enabling faster delivery with superior levels of quality. According to Golestani: “If it weren’t for the testing organization driving that change, they would have been an inhibitor. We would have never been able to release the types of products we do, and the technology would never work the way that it needs to. It’s absolutely essential.”

    Agile, DevOps, Cloud and Testing are Inextricably Linked

    Agile, DevOps, cloud and testing transformation are all interlinked as part of a broader digital transformation. Andy Sturrock, CIO of one of the world’s largest energy companies, stated: “All of these things are self-reinforcing. You certainly don’t get the full benefit of one without doing the rest.”

    He explained the Ops team used to be the primary bottleneck to moving applications to production (just getting servers provisioned could take weeks or months). Now, the cloud relieves that bottleneck—and that places a lot more pressure on the “apps” team, which tends to emerge as the new bottleneck. “You can write code as fast as you want, but if you can’t test it, you’re not going to get into production quickly,” he said. “Testing has to become automated, and the role of the tester must evolve.”

    When Software Runs Your Business, Software Testing Becomes a Business Issue

    According to Golestani, nearly $1 billion US in revenue passes through Merck’s SAP instance each week. Consider that this instance is updated at least four times a year, and more than a thousand changes are made each time. Manual testing is simply not an option. No amount of manual testing can thoroughly de-risk the core business processes, given the frequency of updates, scope of changes and breadth of functionality that’s impacted.  Extreme test automation is the only way to achieve the required business risk coverage with the necessary speed.

    Due to the nature of Merck’s business, lives also are at stake. Merck ships billions of medicines and hundreds of millions of vaccines each day. The company’s continued focus on digitizing the business means that software runs everything from the shop floor to the way that they interface with customers and educate 4 million healthcare practitioners. As Golestani explained, “When the software driving all this fails, nothing ships, manufacturing shuts down, research shuts down … and people die.”

    It’s Not Easy, but Transforming Enterprise Testing Pays Off—Significantly

    Global 2000 organizations that want to implement continuous testing—the extreme test automation required for DevOps—need to overcome challenges such as:

    • Highly complex application architectures (e.g., processes that cross browsers + APIs + mobile + SAP + other packaged apps + custom Java/.NET apps + mainframes).
    • Different application types, planning cycles, development methodologies and tools existing in parallel.
    • Deeply ingrained quality processes across different groups and projects.
    • Scarce test automation resources.
    • “Bloated” test suites that delay the process while providing limited business value.

    There is no silver bullet. Transformations take time and they must involve all of the people, processes and technologies associated with your application delivery pipeline.

    Nevertheless, you can rest assured that your efforts will pay off. Consider just a few examples of how organizations that committed to transforming testing are now growing the business and enhancing the customer/user experience:

    • Duke Energy achieved a $9 million cost savings during the first year of its testing transformation—and it also reduced testing time by ~ 85 percent. This enables the company to more rapidly deliver software that transforms the customer experience, modernizes its energy grid and creates a smarter energy future.
    • SPAR accelerated time to market 3X, enabling the company to expand its online presence across Europe, deliver more customized shopping experiences and provide seamless “omnichannel” experiences across the customer journey.
    • EdgeVerve now releases 30 percent more new functionality in cycles that are 60 percent shorter with 50 percent fewer escaped defects. This enables the company to help its global banking clients onboard, sell, service and engage customers with flawless personalized experiences.
    • ExxonMobil saved $140 million in cost avoidance and increased productivity 3X. This enables the company’s IT department to take on more projects and complete them faster—which in turn helps the company boost sales and increase customer loyalty.
    • Vantiv (now Worldpay) transitioned from eight-week testing cycles to daily testing, which has been critical for enabling rapid growth and agility during the recent Vantiv-Worldpay merger.

    Continuous testing was once just an aspiration. Now, the drive toward digital transformation has made continuous testing a business imperative. Organizations such as these are paving the path forward, demonstrating that continuous testing can—and must—become a reality in enterprise organizations.

    — Wayne Ariola

  • The Benefits of Test-driven Development

    The Benefits of Test-driven Development

    It sounds like an oxymoron: Create test cases before coding. In the process of writing and using test cases before writing functionality, a developer can produce higher-quality software. That is the concept behind test-driven development (TDD), and it works.

    TDD Snapshot

    TDD has been around since at least 1999, and it is part of the emerging test-first approach to development associated with extreme programming. In 2003, American software developer Kent Beck “rediscovered” TDD and applied it as a way to create simple designs and inspire developer confidence. Fast forward to today’s agile development world, TDD is a software development process using a very short feedback loop in which developers do the following:

    1. Create a test that will fail immediately.
    2. Write the necessary code to pass the test as quickly as possible.
    3. Refactor the code written in step two, using the created test as a reference.

    TDD Benefits

    So why might a long-in-the-tooth software development approach have tangible benefits in an agile development world? Because put in its simplest terms, TDD is a way to develop highly usable software.

    Following the TDD process, a developer must focus on the test cases before actually coding anything. That meansa developer is thinking more about the use of the software and the design of user interfaces to achieve that. As a result, the developer is more interested in the interface than in the implementation—and that leads to more usable software.

    There are several other tangible benefits for taking a TDD approach, including:

    • Creating a metric for the code quality that can be easy and quick.
    • Allowing for a quick visualization to determine if the code base has any functional issues.
    • Living documentation of functionality with new code written.
    • Allowing for safe refactors of code, whether based on attempts to improve code quality or a changed requirement.

    That final point is worth dwelling on for a moment. The TDD approach requires that the evolving code base is cleaned up frequently so that new tests and code are easy to introduce. That usually means that code moves from its current place to where it more logically belongs in the software. That has the effect of reducing any extraneous duplication of code and enforces a strong discipline around objects, classes, modules, etc. In this way, the overall maintainability of the software increases incrementally.

    The improvement in readability and maintainability will pay huge dividends during the expected life cycle of the software. Following a TDD methodology requires developers to focus on writing smaller units of testable code. By following the TDD methodology, this leads to more modularized, flexible and extensible software.

    Where Can TDD Fit?

    A TDD approach is appropriate for both new greenfield software and legacy systems. For a development team that has to deal with existing legacy software, the key is to start small and begin with working on bug fixes. A good practice is that for each bug reported, create a test that addresses the broken bug and then fix the functionality. After several iterations of this, the development team has created a repeatable working test to address bug fixes. When applying this methodology to new software applications, make a point of understanding the testing tools used for the technology stack.

    For example, when working in an Angular application that generally uses the Jasmine testing framework for unit testing, and when using the Angular CLI for creation, unit tests are created alongside modules of code. Using a TDD methodology, the approach would be to:

    1. Determine a portion of the functionality to be created with this component.
    2. Create a unit test that will fail immediately targeting this portion of functionality.
    3. Run the test runner to confirm the failing test (it may be useful here to leave the test runner on to run after each source file save, which speeds the process).
    4. Write code in the Angular component that will make the written test pass.
    5. Make any refactoring changes for the Angular component after confirming passing, using the test as a guideline to ensure code refactoring does not break functionality.

    Measuring Testability with Code Coverage

    Another essential consideration when improving testability in code is to use a code coverage tool. Code coverage is a metric to show the percentage of code that has a unit test written for it. Angular applications use Istanbul to calculate code coverage through the application. Running a single run of code coverage in an existing project gives the following output:

    The output provided by Istanbul gives a metric on the overall test coverage and the areas of code that need improvement in testing. Code coverage can be useful in a few ways:

    • Provides an idea of the overall testability, allowing for a threshold to ensure the total software testability does not drop past a certain point.
    • Identifies areas in the code base with poor testing, making them opportunities for refactoring.

    However, as effective as code coverage can sound, it’s important to understand that it’s just a metric. Writing good unit tests is a matter of following what the code will do, and metrics such as these should not drive significant decisions.

    Considerations When Using TDD

    It’s important to note that TDD does not solve all problems. There are many different types of testing required to create a fully comprehensive testing strategy, including acceptance testing. In TDD, the focus is on a single unit of code at a time. A sophisticated software application may have many thousands of units of code and their corresponding tests. That’s why it’s critical to make sure that testing quality stays high when following the TDD methodology. Tests cannot become something bypassed in the pursuit of more functionality or expediency. Avoiding testing generates the risk of test creation becoming a hindrance for developers.

    For example, ignoring failing tests makes it difficult to determine the actual state of the application. It’s also important to have buy-in for a TDD approach form all the teams involved in the effort. Buy-in is particularly true on the business side. Time must be spent up front discussing the nature and benefits of a TDD approach, and the belief that using TDD will improve the resulting software. Otherwise, business management sees writing tests as an activity that doesn’t contribute to the bottom line.

    Conclusion

    TDD emphasizes the importance of an effective and sustainable testing approach. TDD also contributes directly to the overall quality of software. It’s a truism for small or large system development that often goes missing in the day-to-day hustle to get new functionality into production. Quality software gets built when there’s an acknowledgment that quality test code should receive the same amount of attention and resources as quality production code, as they are equally essential in development.

    — Dave Farinelli

  • The Art of Software Performance Testing

    The Art of Software Performance Testing

    When users turn away from applications or have critical feedback, it’s often because the application is slow, hangs/freezes or, worse, it isn’t available when needed. In this era of limited—or, zero—patience for app instability, delivery teams should ensure that their performance testing capability is sound. First though, let’s start with definitions. Despite popular opinion, performance testing is not the same as load testing. Load testing checks how systems function under a heavy number of concurrent users performing transactions over a certain period of time, while performance testing is much broader in that it checks how the system behaves and performs. Performance testing examines responsiveness, stability, scalability, reliability, speed and resource usage of your software and infrastructure.

    Performance testing doesn’t always get the attention it deserves. Testers have a tendency to focus on features and component functionality, even when verifying performance on application architectures designed leveraging APIs. As software development methodologies and application infrastructure evolve, so should performance testing. We observe that in Agile teams, performance testing typically executes at least one sprint behind other testing methods. Performance testers don’t start testing until the application’s features are deemed stable. They operate separately from the delivery team. Yet, performance testers should run tests as soon as new code is available, which in turn helps developers receive real-time feedback on issues that they then can fix immediately.

    To ensure a successful strategy in validating the performance of an application, here’s a look at three fundamental performance test categories:

    • Endurance: This involves running a transaction or scenario continuously for a period of time to detect memory leaks or anything else that causes a system to slow down.
    • Breakpoint: The goal here is to increase the load on a system until it reaches a threshold, after which it crashes. These tests are critical to understand at what point the system or underlying components will buckle.
    • Scalability: By increasing the number of concurrent users, instances or database size, you can measure how applications will handle growth.

    Performance Testing: Recommended Tools and Skills

    Performance testing is difficult without automated tools that can simulate scenarios and user activity so that you can test comprehensively. Many good tools are free or available as open source. Some of the popular ones include JMeter, Grinder and Locust.io.

    To be successful, performance testers need a DevOps mindset. Technical skills are a must, since understanding software code behaviors and interpretation of hardware and software utilization is necessary. An understanding of how to interpret the technical concerns observed from the business perspective is also a great skill to build. Tests should always answer a “what if” scenario and occur within the sprint. Hence, the ability to apply a continuous testing approach as part of the holistic testing strategy is a great skill to learn! Finally, learn how to provide real-time feedback to developers, as a form of hypothesis about changing code to support better performance tracking.

    Guidelines to Make Your Performance Testing a Success 

    These steps will  help structure your performance testing to achieve results for the business success of the delivered application:

    1. Map your tools and environment. You’ve got to understand the environment infrastructure that will host the application under test. The ability to quickly configure, build and tear down the environment to test is a crucial time-saver.
    2. Set acceptance criteria. Getting concise goals of application performance among all stakeholders will chart your path and allow you to gauge success. An example metric is page load times for your mobile application.
    3. Define KPIs. The acceptance criteria will dictate the KPI goals for performance tests. These may include percentile response times, throughputs, garbage collection performance, heap utilization and resource utilization.
    4. Set up test runs, execute and monitor. After completing the preparatory steps above, you can define test scenarios and create the test cases. Don’t forget to incorporate monitoring. This entails creating alerts in the testing environment to ensure that monitoring services are working as expected.
    5. Analyze, repair and assess. After each performance test cycle has completed you’ll want to assess the results against your acceptance criteria and fix any problems immediately. Over time, acceptance criteria will change and so should your test scenarios and anticipated outcomes.

    As with all testing activities, early involvement and collaboration with developers, analysts and business stakeholders will save time and headaches later. The more you know about the technology stack and upcoming application changes, anticipated maturity of the application and evolving business goals, the better prepared you can be for any testing need as it arises.

    In the evolving shift to a modern software development environment, performance testing has become central to building confidence in software. It’s time we give it our full attention.

    — Mush Honda

  • Collaboration: The Glue for Test Automation Maturity

    Collaboration: The Glue for Test Automation Maturity

    QA’s role in DevOps continues to evolve in positive ways. In a recent survey on test automation conducted by QASymphony and TechWell, nearly one-third of organizations reported that among organizations where testers and developers share testing responsibilities, about a third of test cases are automated. This survey and others show that on average, fewer than 20 percent of organizations have adopted test automation.

    What we can discern here is that when testers and developers work closely together, organizations can achieve greater levels of test automation maturity. This is important, because test automation ties in closely with the primary goal of DevOps, which is to develop high quality software, rapidly and reliably. There’s no way to reach velocity at scale without test automation maturity—and QA teams play a critical role in getting there.

    Let’s break this down a bit further. How does developer-tester collaboration contribute to test automation maturity, and what exactly do we mean by that?

    Once developers complete unit testing, their job is to communicate the findings of those first level tests and code verifications to the testing team. If they don’t, testers are somewhat in the dark when determining what other tests—whether functional, API or performance—to build from there. Developers have the first line of vision on problem areas in their builds. It’s not enough for testers to look at a dashboard to see that a code branch has passed a test. Test automation requires speed and accuracy, so the more detailed information testers receive upfront the more successful they will be in writing automated tests and in determining manual testing needs.

    Developers should also automate unit-level testing. DevOps requires more tests, more often; everyone should take advantage of the efficiencies that test automation brings.

    Making Collaboration Work

    How to exchange information is less important than just doing it. There are lots of options these days: Developers and testers can share information back and forth within DevOps code repositories and pipeline tools systems, such as GitHub and Jenkins, or by using collaboration tools such as Slack. Geographically-dispersed teams may prefer web conferencing systems to facilitate live discussions on the fly.

    Team members can sit side by side, doing what’s called paired development. It’s powerful for testers to watch the code as developers write it so they can ask questions during the process. That gives the tester more knowledge about the product and in turn, developers understand the tester’s objectives. The point here is that achieving a high level of test automation coverage isn’t easy; testers and developers are both learning new skills and processes. Through frequent exchange of information, they can make progress faster and always be on the same page.

    Clearly though, with such low average test automation adoption levels, we need to work on closing the collaboration gap.  Here’s how I see the barriers:

    • Developers are fiercely independent. While that’s not a surprise, it requires incentives and education to help developers understand the need to get out of their silos. Developers must adopt test automation and talk to the testers. If they don’t adapt, they won’t last—at least that’s what a CIO told me recently.
    • Testers and developers have been in opposition for too long. It’s time to get rid of this “us versus them” mentality. Developers sometimes think testers are just trying to break their code and criticize their work. Testers sometimes complain that developers could care less about writing buggy software. This is a cultural problem and leadership can help resolve it by promoting software quality as an organization-wide priority for which everyone shares responsibility.
    • Testers and developers don’t always speak the same language. The thinking behind how developer writes code is not the same as how the tester writes automated scripts. Developers don’t consider all the scenarios a tester might when they write code. As a result, they may not write clear object identifiers in the code which help testers write scripts.

    Overcoming these barriers comes back to collaboration. Developers and testers need to find ways to understand each other’s world view. Begin by obtaining a joint understanding of completion. If developers and testers can agree on the release criteria, meaning that testers are comfortable with the level of automation in place to support future releases, it avoids a huge backlog of tests to automate later. Backlogs lead to delays and frustrations all around. Using a common development language for coding and test automation is also helpful; otherwise, it’s more difficult for both sides to speak the same language when working together. In the same vein, testers should have access to all the tools that developers use: source code repositories, continuous integration, project management and so on.

    We live in a digital world and have endless instant communication options, but even on a software development team, there’s nothing that replaces the value of a face-to-face conversation. It’s going to take more than one tactic to make collaboration work in your organization. Keeping the lines of communication open should be a central goal to increase test automation coverage and streamline the DevOps release process.

    — Ryan Yackel

  • Continuous Testing in the ‘2018 Accelerate State of DevOps’ Report

    Continuous Testing in the ‘2018 Accelerate State of DevOps’ Report

    Here’s a peek into what continuous testing means to DevOps today

    The “2018 Accelerate State of DevOps” report is the brainchild of Dr. Nicole Forsgren, Gene Kim and Jez Humble at DORA (DevOps Research and Assessment). Based on five years of research, with more than 30,000 data points from thousands of companies, the project aims to understand precisely what practices enable teams to deliver better software faster.

    The 2018 report places an increased focus on testing and quality, including a new report section dedicated to continuous testing.  Here are three key report takeaways that are important for continuous testing:

    1. Testing must change—significantly—to keep pace with DevOps

    Consider this:

    • Elite performers deploy code on-demand, multiple times per day.
    • At elite performers, less than one hour elapses between code commit and production.

    The rate of change is accelerating—fast. Testing must somehow keep pace. Evolving testing for DevOps is critical for realizing DevOps’ full potential in terms of accelerated delivery, agility and all the derivative business outcomes outlined in the previous point.

    2. High rates of test automation (~90%) are essential for DevOps

    At elite performers, manual testing represents only 10 percent of the total effort. Even the 30 percent manual testing rate for low performers is significantly better than the industry average of 80 percent+ manual testing (including teams that have not yet adopted DevOps).

    The importance of test automation is underscored by the findings of a recent Forrester report that’s focused specifically on what quality practices separate DevOps leaders from DevOps laggards. That research found that DevOps leaders are:

    • Transforming software testing into continuous testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
    • Almost twice as likely to consider automating the software quality processto be a “critical business differentiator” (53 percent vs. 27 percent).
    • Significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

    3. Quality is correlated to DevOps success

    Elite performers have a 7X lower change failure rate than low performers. They also spend 2X less time fixing field-reported defects.  Exactly what DevOps practices are contributing to the impressive quality outcomes being achieved in the top-performing organizations? And how are different quality practices contributing to DevOps success?  This is certainly an area that’s ripe for further research and analysis.

    Want to learn more about this report’s findings—and what it all means for software testing?

    Join Nicole Forsgren, PhD, CEO and Chief Scientist at DORA and Wolfgang Platz, Founder and Chief Strategy Officer at Tricentis, for a webinar to explore “Continuous Testing’s Role in DevOps success.” The focus will be on helping IT leaders understand:

    • Why and how high-performing teams make Continuous Testing a linchpin of their DevOps process.
    • What specific strategies and practices enable high-performing teams to deliver faster and dedicate more resources to innovation.
    • Which quality-related technology and process investments have the greatest impact on DevOps success.

    — Cynthia Dunlop