devopsy.kinsta.cloud

Tag: testing

  • DevOps Video Chat: DevOps Testing with Ryan Lloyd of SmartBear

    DevOps Video Chat: DevOps Testing with Ryan Lloyd of SmartBear

    The testing landscape in DevOps is a big one, with lots of established players feeding the machine. SmartBear is one of those companies. The force behind Swagger, SoapUI and others is perhaps not as well-known as its brethren in the DevOps testing space, but it cuts a wide swath. And, with its recent acquisition of Zephyr, it’s about to get wider.

    In this DevOps Chat, we speak via video with Ryan Lloyd, VP of Products at SmartBear, about the Zephyr acquisition and the DevOps testing and test management space in general. As usual, the streaming video is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, Container Journal, Security Boulevard, and we’re here for a great interview, hopefully, with Ryan Lloyd, VP of products for testing and development at SmartBear. Ryan, welcome.

    Ryan Lloyd: Thank you. Great to be here.

    Shimel: Great to have you here. So, Ryan, before we jump into today’s news that we wanna discuss, I wanted to just quickly kinda do a level-set. SmartBear is a company I think most people have probably heard of it, but they may not be 100 percent sure of the full breadth of the SmartBear portfolio. So why don’t we talk a little bit about who is SmartBear and what are the kind of products and services you guys offer?

    Lloyd: Yeah, absolutely. So SmartBear software is a company with a lot of history and if you don’t recognize the company brand, it’s very likely that you recognize one or more of our product brands, so we are the company behind some of the key product brands like Swagger, SoapUI, CrossBrowserTesting, TestComplete, so quite a diverse portfolio of products focused on helping our customers who’re building software. Help them build tests and monitor their software applications.

    Shimel: Great. So, Ryan, big news out today is SmartBear had made another acquisition. You guys have acquired Zephyr. Is that correct?

    Lloyd: That’s right. Yeah, we’ve announced our intent to acquire Zephyr, a company in a similar market space as us.

    Shimel: So it’s a similar market space, but let’s get specific. What does Zephyr do?

    Lloyd: Great. So Zephyr is a collection of tools for testing and specifically focused on the area of test management, managing the process of testing.

    Shimel: Great. And you mentioned an intent to inquire – acquire, so I guess that means it hasn’t yet closed, but you’re both private companies and I would imagine it closes fairly quickly.

    Lloyd: That’s right. We expect in the next 30 days, subject to the usual regulatory approvals.

    Shimel: Yep. Fantastic. So, you know, Ryan, one of the things I read in the background around this one was that one of the big pluses here is Zephyr’s presence within the Atlassian, the Atlassian ecosystem, and SmartBear actually is already a big player in that Atlassian ecosystem. Well, aren’t they?

    Lloyd: Yeah, so we have a couple of products that have plug-ins to the Atlassian ecosystem. I mean, the reality is 60, 70 percent of our customers are using Atlassian’s tools. They’re pretty dominant in the area of software teams, for managing some part of the software development process, so it’s a very strategic and important ecosystem for SmartBear as a company. And, certainly, Zephyr improves our ability to reach that ecosystem of users and make sure that they’re aware of all of the tools and capabilities that SmartBear can offer.

    Shimel: Yep. And, Ryan, you know, 60 to 70 percent use Atlassian. I’m gonna imagine the overwhelming majority are using Jira and then maybe Jira Plus.

    Lloyd: Jira –

    Shimel: Correct?

    Lloyd: Certainly, Jira is probably the most prevalent. Confluence gets tagged on with Jira quite a lot. And then BitBucket, actually, is also becoming quite popular as well. BitBucket, GitHub, GitLab – they’re sort of three of the same, in terms of enterprise solutions for developers.

    Shimel: Got it. So I don’t wanna turn this into being about Atlassian and their products, but, obviously, they do have a big ecosystem. There are a lot of third parties, such as SmartBear and Zephyr, who move a lot of services and product via that marketplace. But let’s talk a little bit similar, but where’s the overlap? Or is there any overlap between what Zephyr and SmartBear’s offerings are about?

    Lloyd: Yeah, so SmartBear offers a broad portfolio of products and, certainly, a couple of the products we have, pre-acquisition, were in this test-management space. What Zephyr brings to the table, though, is a solution that is natively built around Jira and delivered through the marketplace. So what that means, to be delivered through the marketplace, is it’s a native Jira solution that Jira administrators and customers can install directly from within their Jira instance. So it provides a very seamless extension of their workflows, within Jira, to handle some of these test-management scenarios, which we think is really important to teams that are trying to consolidate their processes and have a unified approach to that.

    Shimel: Excellent. I guess one of the issues – and, with your title and your responsibilities around VP of products for test-and-dev, just in a number, how many products are there in test-and-dev now, within SmartBear and then if we add the Zephyr suite to the mix?

    Lloyd: Yeah, so we had five products in that portfolio. Today, Zephyr brings three additional products into that portfolio, so we’re now up to eight products in our test and development business, as well as a range of products in our other business unit, around API.

    Shimel: You know what? That’s a lot of products to be managing, huh?

    Lloyd: It is. I mean, SmartBear has grown a lot over the years, through both acquisition of companies and building new organic products and bringing them to market. It’s pretty exciting time for us. We’re, pre-acquisition, at about 450 employees globally, so it’s an exciting time to be part of this company.

    Shimel: Absolutely is, man. So, you know, another not necessarily unique but another kind of factor you have to play into the SmartBear – and I don’t know about Zephyr as much – is that some of your solutions have open source kind of versions out there that you guys help manage the open source communities around these, as well as the commercial versions of the products themselves. Correct?

    Lloyd: Yeah, that’s right. I mean, core to our business is really a couple of philosophies. One is we wanna be open and interoperable with all the tools that are part of the ecosystems that software teams are using. And so, in some cases, that means commercial opportunities to partner with companies like Atlassian; in other cases, it means we want to be a part of an existing open-source ecosystem or help build up and nurture an open-source ecosystem. So, certainly, when we brought SoapUI and Swagger into our product portfolios, that was all about really taking that existing organic ecosystem, where developers and testers are going through tools, and helping build and nurture and develop that ecosystem.

    Shimel: Excellent. So what kind of special challenges – you specifically, right? You own the products; you’re VP of products here. What challenges does that pose for you, having these open source kinda versions and tools and communities. I mean, you don’t wanna burn them, right? Because, ultimately, you kinda bite the hand that feeds you. So let’s talk about that. What does that – how does that manifest itself with you?

    Lloyd: Yeah, certainly. I think there’s a brand and a culture that exists around open-source products and I guess it’s our responsibility, as stewards of those brands, to really continue to nurture that community. And that means we’ve gotta make balanced decisions about where we choose to add features to the open source products, versus where we commercialize some of those features, so there’s a constant balance there and kind of evaluating and making sure we’re engaging with the community.

    And then, more broadly, beyond just our open-source products, I think one of the biggest challenges in the software tools space and market is the interoperability that’s required between tools these days. You know, developers and testers, they’re demanding practitioners. They want their tools to work together and there’s a huge diversity of tools out there in the market. So, as a product manager, that’s what keeps me up at night, is making sure all these tools are compatible and working to support these complex workflows.

    Shimel: Absolutely. I mean, one way of looking at it is, if there wasn’t such a big market, there wouldn’t be all these tools.

    Lloyd: That’s true.

    Shimel: Another way of looking at it is also that it presents a tremendous opportunity for consolidation and for one or a few products to emerge as true market leaders. And then you’ll see it kinda – you know, you’ll see the amount of products in the market kinda consolidate. You know the old saying, “If you’re not – you’re either leading, following, or get out of the way,” right? If you’re not top-three, you gotta ask yourself, “Do you wanna stay in that market?” But, you know, when we talk about the testing market – and, in DevOps, we talk about continuous testing – it certainly is a huge, huge market here.

    Lloyd: Yes, it is. And it’s one that’s continually evolving and shaping, right? So there’s traditional test tools that have been around for a decade or more, and then there’s sort of emerging products that are coming to market. And I think it’s how all these work together and evolve to support the change in workflows and, yeah, even just organizational changes within environments.

    Shimel: Yep. So another area of the testing space, Ryan, that I’m interested in personally is security and security testing. And it’s really funny when I – you know, 20 years ago, when I was into security, when I first got into security, info-sec, QA was here and security was there and never the twain shall meet, right? But now we’re starting to see security become synonymous with quality – that’s the “Q” in QA.

    Lloyd: Yeah.

    Shimel: And so security testing and other kinds of testings are starting to merge. What do you guys see with that at SmartBear?

    Lloyd: Yeah, I think what’s driving a lot of that is the Agile workflows that teams are adopting, so, as they increase the pace of trying to deliver and release software, they’re being forced to evaluate quality and security earlier in the life cycle and more from the context of the development team, rather than having a separate, isolated team that deals with that downstream or after the fact.

    Shimel: Correct.

    Lloyd: So I think that’s the driver, so definitely a lot of demand for your Agile workflows to support testing by design and security by design. From our perspective, I think that opens up some unique opportunities for adjacencies and areas that SmartBear can continue to expand to in the future.

    Shimel: Yeah. I mean, so, you know, what you’re describing is almost the classic “shift left.” Right? As we start shifting testing, security and security testing, left early into the development life cycle. But it’s also the reality of it’s been called the “modern software factory.” Right? It’s an assembly line and is a production line, for software, and where traditional QA, traditional testing, security testing, where they fit into that life cycle has shifted left.

    But also is – you know, where we used to think, oh, QA was going away and security people were gonna give up control of security. Not the case. They’ve become more important, right? But also highly automated where they can be and clearly defined as part of this factory, if you will.

    Lloyd: Yeah, it’s really just about reshaping the responsibilities and improving the collaboration across the whole unit or team that’s now responsible for delivering iterations of software.

    Shimel: So, Ryan, then – so that begs the question then, “What does that mean for you as a product manager?” Right? Where you used to sell to a QA engineer or security testing sold to a security person, but now your audience includes the guys using Jira and BitBucket and a bunch of other tools. It’s not just Atlassian, obviously. So you have new customers to serve and to service. Now how do –

    Lloyd: Yeah, that’s absolutely right. Yeah, so, over the last several years, we’ve been focusing on how to modernize our portfolio to support those sort of emerging needs, both the process change that’s going on with our customers and the persona change that’s happening in many instances. So the acquisition of Swagger is a great example of us providing tools that help developers earlier in the life cycle, so, where we had SoapUI to help with API testing from a QA perspective, Swagger aims to solve some of that further upstream by enabling more collaboration around the definition of those APIs to begin with. And, oh, by the way, making sure that interoperates with SoapUI, so you get leverage out of that API definition to generate some of those tests, automate it, and it triggers sort of a seamless workflow and thread.

    And so we’re continuing down this path of modernizing our portfolio. We acquired CrossBrowserTesting about two years ago, to help companies address some of the modern types of applications that they’re looking to test, integrated more deeply with Selenium, which has become hugely popular in sort of the developer organizations, to drive automated Web and mobile testing. And then, with Hiptest and Zephyr, again, it’s furthering our relevance for those software teams that are on that journey.

    Shimel: Excellent. Ryan, last kind of issue I – or not issue – it’s not an issue at all – but the last area I wanted to talk to you about was so you’ve got five. There’s three more coming in. And that’s just test-and-dev; you have an AP kind of silo as well within SmartBear. How do you – I mean, I’ve been in the boat not quite as many as you, but, when you have so many different products and then a company name, how do you brand? Right? Do you continue branding the individual products? Do you throw SmartBear in front of all of them, but some of ’em already have names that may be, in fact, bigger than SmartBear, as you said in the beginning? Again, as a product manager, you have to think about that a bit. You know, what’s the right answer there, if there is –

    Lloyd: Yeah, so I think SmartBear is going through a bit of a transformation, in that regard. You know, as I said and as you pointed out, we’ve had very strong product brands up to this point. We wanna continue to nurture those product brands, so no intent for those to go away because we love the high-velocity, try-buy model we have around our products, where developers and the practitioners can find the right tool for the job and get started.

    But we’ve gone through an exercise on how we can integrate our products in the right way, so like the Swagger, SoapUI example I provided, to create those sort of interesting workflows. And we’re building up the SmartBear Academy, our SmartBear community, our big user conference, to help elevate the SmartBear brand and create more awareness of the portfolio of products we have – how they work together, how they fit into the ecosystem. And I think we’re just at the start of that journey right now. We’ll see a lot more, as we launch into the year next year, around what we’re doing from a brand perspective to galvanize the company and all of the great products we have.

    Shimel: Excellent. Well, Ryan, we’re about out of time, but, first of all, congratulations on the Zephyr announcement and anticipated acquisition here. It’ll make for some good times. And continued success with SmartBear and everything you guys have going on.

    Lloyd: I appreciate it. All right. Great talking to you, Alan.

    Shimel: All right. Ryan Lloyd, VP products, test and development, for SmartBear, on SmartBear’s acquisition of Zephyr, here on devopsy.kinsta.cloud. And we’ll talk to you soon, everyone. Have a great day.

    — Alan Shimel

  • DevOps Chat: Crowdsourced Continuous Testing with Rainforest’s Derek Choy

    DevOps Chat: Crowdsourced Continuous Testing with Rainforest’s Derek Choy

    Scaling your testing is the scourge of many software teams. While test automation and continuous testing have made great strides, it still is difficult to automate every QA test. Also, engineering resources who can script these tests are difficult to find. Enter Rainforest.

    Rainforest has a unique crowdsourcing solution to continuous testing. It combines crowdsourcing talent with automation to deliver a high level of quality assurance on a continuous basis.

    In this DevOps Chat, we speak with CIO Derek Choy as he explains to us why Rainforest in some ways can be the Uber of QA. As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features a first-time guest on DevOps Chat. He’s Derek Choy, CEO at Rainforest, or Rainforest QA, as some may know them. Derek, welcome.

    Derek Choy: Thank you. Just a little correction: I’m CIO at Rainforest.

    Shimel: What did I call you?

    Choy: Responsible for primarily the technical side of the functions here.

    Shimel: Did I call you “CEO”?

    Choy: I think you did, but I just want to make a little correction there.

    Shimel: You know, it’s 2019; that’s my prediction.

    [Laughter]

    Choy: I don’t know that I would support that prediction, but thank you. [Laughs]

    Shimel: Well, I _____ – CIO’s enough. But, anyway, Derek, thanks. Yes, it’s Derek Choy, CIO at Rainforest. So, Derek, you know what? I didn’t make the mistake on purpose; it was an honest mistake. Good. That leads us – why don’t we talk a little bit about Rainforest for our audience who may not be familiar?

    Choy: Absolutely. So Rainforest is a testing platform. What we do is we help our customers with their software testing. You know, as you know, we’re moving – a lot of companies are moving towards a rapid development cycle, CI/CD, frequent releases and stuff. Testing becomes much of a challenge for a lot of our customers because now they not only have to do testing, but they have to do it fast. They have to have accurate results and stuff.

    So Rainforest is actually a crowd-based testing platform that allows our customers to simply input test cases in plain English, so no Selenium or any sort of programming language, and then we can actually execute them as if they are automated test. We’re fully integrated into any CI/CD tools, like Jenkins or CircleCI and so on, so you can actually trigger a Rainforest test. Typically, we can return results in 30 minutes or so, and you can actually test across many different platforms, like different browsers, different OS’s, and the same for mobile as well. And so that allows our customers a wide variety of options, in terms of where they want to run their plain-English test cases and when they want to run it – it’s on demand. And we typically return results in 30 minutes, fully integrated into an automated release cycle.

    Shimel: Great. Excellent. And, Derek, how did it come to be that you became the CIO of Rainforest? Let’s hear a little bit your own personal journey.

    Choy: Cool. Oh, thank you. So I have been always involved in the product engineering side of things. I started off my career earlier with Accenture. Spent a number of years at eBay. And then my last company at Aria Systems, building SAS-based billing system, also running their engineering and product function. So, coming to Rainforest, I brought my experience and knowledge in this space, to help build the engineering and the product organizations. Now here, besides product engineering, I’m also responsible for our professional services team and our client success team. So essentially kind of helping build out product and making sure that our customers are successful in using that.

    Shimel: Absolutely. So, Derek, you know what? Here we are in December. We’re looking at 2019.

    Choy: Mm-hmm.

    Shimel: I probably have spent the last three weeks – if I tell you I’ve spoken to dozens of vendors and C-level execs at large organizations, and, to me, the real story of 2019, as well as what’s emerged as the story of 2018, is around – you know, you wanna call it “digital transformation” and “DevOps,” _____ say great, but, really, down to these two things: companies are – they need scale in what they’re doing today, right?

    Choy: Yeah, absolutely.

    Shimel: When we’re talking about data and computation and transactions or whatever the metric is that’s driving your company, everybody wants to do it at huge scale, like unprecedented scale that, 20 years ago, you would’ve went – you know, I’m just reminded of Carl Sagan saying, “Billions and billions of stars,” _____.

    Choy: [Laughs] Yes, that’s right.

    Shimel: That’s the scale. And the only way to achieve that kind of scale is automation.

    Choy: Yes. Absolutely.

    Shimel: So these are two kind of twins, co-joined at the hip – scale and automation.You can’t have scale, I think – well, you can’t handle scale without automation.

    Choy: Absolutely. Yeah. And – sorry.

    Shimel: Go ahead. No, no, go ahead.

    Choy: I –

    Shimel: You know, to me, that’s it right there. That’s the crux of it.

    Choy: Exactly. And, Alan, I would say that I’ve never been more excited, right, than where we are right now. I think we are definitely at a very exciting time and, just to kinda set the context and maybe kinda recapping a little bit of what you just said, right, the world is very different now, even compared to five years ago. Right? A while ago, you would hear companies having a quarterly release. Some would even have an annual release, if you remember, especially the bigger companies, right? You know, release cycle was a lot longer, takes a lot more time to get things out to the market. Today, release cycles are typically maybe in a month, right? Maybe two weeks and, in some cases, days or even multiple times a day. Right?

    And exactly to you – and the interesting thing is that the world has got accustomed to that and has come to expect that, frankly. Right? Think about the apps that you’re using. If they come back to you and say, “Well, you know what? We update our apps once every six months,” or something, you’re like, “Wait, six months? Wow, that’s unheard of.” Right? “I was expecting a weekly update or at least monthly or something.” So the world has actually comes to expect that.

    Now the challenge, exactly as you pointed out, is how do companies deal with that? Right? How can we scale our organization through automation, among other things, right, to make sure that we can keep up with that pace? I think that that is the challenge with a lot of technical leaders, is we understand what the companies are looking for, understand what the consumers and our customers are looking for, right? But how do we do that? Right?

    And there’re different aspects of it, Alan. I think that there is definitely a development aspect of it, right? How do we scale development? How do we do rapid development, Agile, sprint? You know, all these things. Things have evolved a lot over the last, I would say, ten years, in terms of faster development. There’s a lot _____ –

    Shimel: On the development side?

    Choy: Exactly. And there continue to be more evolution on that, which is super exciting. Now one thing that hasn’t really, at least in my opinion, gone as far in this particular context is testing. Right? If you think about testing, we’ve had automated testing for a long time – 10, 20 years, maybe even longer, right? We’ve always had manual testing, right?

    Now, when you think about testing, that’s – which is a critical part of the release cycle. Of course, right? You cannot push things out without some sort of testing at least, right? There hasn’t really been a lot of innovations and evolutions in testing. We’re still trying to do automated testing; we’re still doing some manual testing; we’re still doing it kinda the same way, so to speak. There’re new automated testing language, but they all kinda do similar things, right? And they all have similar –

    Shimel: Absolutely. And, Derek, let me stop you there for a second.

    Choy: Yeah.

    Shimel: To me, here’s the real crux in the issue.

    Choy: Yeah.

    Shimel: We talked about that we’ve certainly seen, let’s call it, a “revolution” in development acceleration, rapid development over, let’s say, 10 years. I’ll say even 20 years, right? _____ _____ _____ around for about 20 years.

    Choy: Agreed.

    Shimel: Do you consider testing part of development? Or did DevOps come along and say, “Wait a second, it’s all one continuity here, with development, testing, operations, security – you wanna put security next to testing, whatever – but it’s one continuum”?

    Choy: Yeah. It’s definitely – I completely agree with that. It is definitely one continuum, meaning that they are all part of the overall workflow, right, to make automation possible, to make a fast release process possible. You cannot do it without any one of those components, right?

    Shimel: No. But they haven’t kept pace with the changes that we’re in – what we consider when we talk development.

    Choy: Yes.

    Shimel: Where many of us think, you know, of coding.

    Choy: Exactly.

    Shimel: That’s the issue right there.

    Choy: That’s the issue, exactly as you pointed out, right? All the components of that whole continuum hasn’t really kept pace, right, with the developments that we’ve made on coding, right? On the actual coding of the product. And I think that that’s starting to show or actually has been starting to show, in terms of how other components of that, testing in particular, has been frankly dragging down the overall speed of being able to push things out faster and so on, so forth. Right?

    If you talk to – and you do that all the time. If you talk to a lot of CIOs and CTOs and so on, frequently, you’ll hear challenges with testing. Right? And most people will tell you, “Yeah, I’m okay. I get it going,” but very few people –

    Shimel: “It’s good enough.”

    Choy: Yeah, it’s just good enough, but it’s not great, right?

    Shimel: No, it’s not.

    Choy: Yeah.

    Shimel: And, Derek, I’ll go one further with you. What I see is this, is that, when we talk about testing – and _____ _____ the same thing with security, right? Is we’ve got twin challenges.

    Choy: Yes.

    Shimel: I can’t do it fast enough, so I’m constantly behind, running as fast as I can, but I can’t – just not fast enough.

    Choy: Exactly.

    Shimel: And then, number two, the train won’t wait for me.

    Choy: Yeah. Exactly.

    Shimel: If I can’t finish my testing, “Well, you did it good enough.”

    Choy: Yeah. “Let’s go.”

    Shimel: “We’ve gotta release. We’ve gotta release this hour,” or, “We’ve gotta release this week.” You didn’t do – dirty testing? That’s okay; it’s probably secure anyway. And, boom, out the door it goes, right? And that’s the dilemma, the enigma of testing, is “I can’t do it fast enough, but, if I don’t do it fast enough, they don’t wait for me.”

    Choy: Exactly. Right. And there is a reason for that. Exactly, right? Like we talked about earlier, because customers actually expect releases, right, whether you’re ready or not, right? They are expecting releases in a very frequent cycle. Right? And so companies therefore cannot wait. Right? It is not acceptable to go back to the customer and say, “You know what? My release is delayed and therefore you have to wait another week or two.” In the past, that used to be the common thing to do, right? Nowadays, that’s not acceptable.

    And so then what happen, exactly like you said, is that people compromise, right? They’re like, “Oh, you know what? Testing is not done. Okay, well, you know what? It should be fine. Let’s just go.” Right? “Security testing is not complete, but we’ve done a portion,” right? “Okay, well, let’s just go.” Right?

    The downside, however, is that – and you’re seeing it frequently; some people are just not paying attention to it – is that there is a high price to pay for that, right? You’ve seen a lot of security loopholes out there, right? And, yes, the release is out, right, but then at what cost, right? If people found a loophole and there is a data breach and costs companies lots of moneys and reputations and things like that.

    And equally with software quality as well, right? Things don’t work. Even bigger companies, you’ve heard companies like Apple, for example, struggle with that. Google, right, struggle with that. And they release something and people like, “Wow, this is junk,” right? “And I don’t want to upgrade,” and you hear that all the time, right? And it tarnish your reputation.

    Now bigger companies have, obviously, a lot more buffer to kind of weather these kind of quality problems, right? You know, whether you like it or not, you’re gonna continue to use Apple, right? Or Android or whatever. But then smaller businesses may have less tolerance for that, right? And customers of those businesses, if they’re not happy with your quality, they’ll move somewhere else.

    So I think it’s very important for technical leaders to figure that out and not just kinda brush it aside and say, “You know what? It’s not important. If quality’s not good enough, let’s just go. It’s fine. We’ll find bugs in production; we’ll fix it.” Personally, I don’t think that that’s the right approach and I think that it’s gonna hurt businesses down the road.

    Shimel: So you know what I call that? So that’s the double-edged sword of MVP.

    Choy: Yeah.

    Shimel: Right? ‘Cause you talk to these people and they say, “Don’t worry. We’re just doing an MVP. It’s a minimally viable product. We know it’s not 100-percent tested. We know it’s not a 100-percent baked. We know it doesn’t have everything, but we’ve gotta get an MVP out there, god darn it.” Right? [Chuckles] And so –

    Choy: Yeah.

    Shimel: And we appreciate – look, I’ve been – I founded, cofounded several venture-backed companies. I’ve been on pushing the VP of engineering, saying, “Hey, we promised our customer we had a new release coming out before Thanksgiving, man, and it has to get out. It has to get out. I don’t care if it’s fully tested. Get it out.”

    Choy: Yeah.

    Shimel: It’s not – you know, MVPs were being chewed to death.

    Choy: Yeah. I – [laughs]

    Shimel: So that “minimum” – that “M” in MVP keeps getting bigger and bigger and bigger. Right?

    Choy: Exactly. Right, exactly.

    Shimel: And more and more critical.

    Choy: Exactly. I think we continue to push the boundary of MVP. You know, I think that – and that’s really the challenge, right? So, you being kind of the leader of the business, right, you’re not wrong because customers really need it, right? “In order to move my business forward, I really need the release. I really need to get that feature out. I really need to get ahead of the market.” That’s not wrong. Right? And I think that that’s the needs of the business; that’s the market; that’s what people expect today. The challenge that we technical leaders needs to solve – CTOs, CIO, VP of engineering, VP of product – what we need to solve is “How can we support the business in doing that but also maintain the quality?” Right?

    And so, as a leader, in my opinion, going back to the business and say, “You know what? I can’t actually make your delivery date. You will have to wait another two weeks, four weeks, six weeks,” or something – that’s gonna be a tough sell, right? But, at the same time, releasing it without quality assurance, without security and all these stuff, is also not the right approach.

    So I think what we leaders needs to figure out is “How can we do better, right, in QA? How can we do better in security?” All these other things that are probably less mature than development itself, coding itself, right, how can we do better in these areas so that we can also support the business in shipping this MVP? Whatever the definition of this MVP is, right? We can support a business in shipping the MVP because that’s very critical to the business but, at the same time, maintain the level of quality and the level of security and, among other things, right, that we are proud of, right? That we want to have, right?

    And I think that that’s the biggest thing that we need to figure out in 2018 and definitely going into 2019, right? How fo businesses figure that out? How do we implement processes and tools and technologies that will allow us to do that?

    Shimel: Yep. So, Warren – Warren? I’m sorry. Derek, fantastic discussion. Let’s bring it home a little bit to Rainforest.

    Choy: Sure.

    Shimel: What can you do to help?

    Choy: Yeah, so Rainforest has a very – takes a very different approach to QA, right? You know, so the traditional way is, obviously, automated test with Selenium and all these different Appium, all these different technologies, or we can go manual testing and perhaps building an AMI offshore – that’s very, very typical as well. Rainforest looks at it and say, “Well, neither one are the best approach,” right? And therefore we come up with a testing platform, you know, that is powered by the crowd. And we allow our customers to actually put plain-English test cases on our platform. We can execute these tests without actually building an AMI offshore, but, at the same time, you don’t have to worry about the downside of automation, having to hire QEs and spend a lot of time writing Selenium scripts and things like that.

    And so we offer a solution that, at least we think, solves problems on both sides, right, and also allows you to hook it up into your CI/CD process so that you can actually push this MVP out in an automated fashion, fully kinda connected with all the rest of the continuum, right, and be able to actually ship code faster but with higher quality.

    Shimel: So let me ask, so, at some level, though, you are reintroducing or relying on human element?

    Choy: Right. Correct.

    Shimel: So why don’t you have the same scale problems that others do?

    Choy: That is an excellent question. So I would say that there are two elements, right? So, when we look at the human elements, we look at it from a crowdsourcing basis, right? So think about Uber, for example, right? If I were to hire my own team of drivers, that would be a little bit challenging, but then, leveraging the crowd and scaling it based on demand, we’re able to actually solve a lot of the human aspect of the problem. We can have testers that come in whenever they like to come in and we have enough testers to support the test that our customers would like to do. Right?

    And now the challenge there and probably a followup question then is “Well, but what about the quality of these people? Do they know what they’re doing?” Just like a Uber driver, right? How do you know that that guy actually drives well, right? So we actually developed a bunch of AI algorithms that actually analyze the work that they do and also take in feedback from customers, based on their former – the previous tests that they have done. And a combination of that with a lot of data mining and AI analysis helps our customers determine whether that test result is correct or not. If it’s not correct, we will engage additional testers to make sure that the tests are consistent and results are correct.

    So we have refined that algorithm over the years and now we have achieved a very high level of accuracy that, based on our data, is showing us that the accuracy’s actually higher than having your own manual QA team. And so you get better results, without actually hiring an AMI, either onshore or offshore. You can actually run your test on demand – typically, you’ll get results in 30 minutes or so. Right? And it’s fully integrated into your CI/CD cycle. And the best part is you can do all these things and achieve all these benefits without having to hire QEs to write automated scripts and having to maintain it, and so on, so forth.

    Shimel: Excellent. So, Derek, I get the outsource. I get the army of people doing it. But, on the back end, are you doing some automated testing as well?

    Choy: Yes, we are. And so what we’re also doing is we’re developing additional AI algorithms to actually learn from these human executions. So what we do is we have lots of human executions that are happening as we speak. In our environment, we have 80 million, 90 millions of tests that have already conducted on our platform, so what we do is we actually do machine learning through these data and learn from previous executions and develop algorithms so that, instead of actually having to send it to a human tester, we can actually achieve it with a robot.

    Very similar to the approach that Uber is taking. Today, human drivers are driving people around, right? In the back end, they’re developing these self-driving cars so that, in the future, we don’t really need a human drivers anymore. Very, very similar approach, from a QA standpoint. Today, we’re sending testers, but then, in the future, we’ll have these robots that will execute these tests based on what they have learned from these human testers.

    And tomorrow is already here for us today because we’re already doing that and a number of our customers are already using these automated AI algorithms. So they just need to write their test cases in plain English and they’ll see the tests executed by robots automatically, without having to worry about building automations or anything like that. So that’s kind of our approach to solve the QA problem.

    Shimel: So Rainforest, the Uber of QA.

    Choy: I don’t know that marketing will approve that line, but –

    Shimel: No, I don’t think _____ _____ –

    Choy: – personally, but, yes. Yes.

    Shimel: It does remind me – you know, when – you’re out in San Francisco, we’ve all been in venture-backed world.

    Choy: That’s right.

    Shimel: There was a period there where every pitch was “We wanna be the Uber of,” right?

    Choy: Yeah. Exactly.

    Shimel: Uber of this or the Uber of that and _____ _____ to be disruption, or crowdsource in this case, but fascinating take on solving an age-old problem.

    Choy: Mm-hmm. Exactly.

    Shimel: Interesting to see. Just – Rainforest is, what, about two-and-a-half years, three years old now?

    Choy: Yeah, it’s just a little bit longer. We’ve been – our total existence, probably about six, seven years. We’ve kinda pivoted a little bit at the beginning, so our current product is about three to four years old.

    Shimel: That’s when I think I first became aware of it.

    Choy: That’s right.

    Shimel: And, look, testing and continuous testing is such a – I mean, it’s a hot area. There’s a lot of different solutions vying in the market, but this one is really kinda unique, so it’ll be interesting to watch how it happens.

    Choy: Yeah.

    Shimel: Derek, as I promised you before we started, the time goes really fast here. We’re way out of time, but, hey, man, I wanna thank you for being a guest on DevOps Chat. And maybe we’ll catch up with you again in the new year and find out how things are going.

    Choy: That will be awesome. Well, thank you, Alan, and have a wonderful holiday to you and everybody listening. And, hopefully, I’ll talk to you again soon.

    Shimel: Absolutely. Derek Choy, CIO still, not CEO, of Rainforest. Thanks for being our guest on DevOps Chat today. Have a great holiday and thanks very much.

    Choy: Thank you. Buh-bye.

    Shimel: Buh-bye. This is Alan Shimel for devopsy.kinsta.cloud and you’ve just listened to another DevOps Chat.

    — Alan Shimel

  • The 4 Keys to Effective Continuous Testing

    The 4 Keys to Effective Continuous Testing

    In the digital era, products no longer can win on capabilities alone. Speed to market and buyer experience now matter just as much to customers as the extent to which a product meets a pressing need or outperforms competitive offerings. Against this backdrop, software developers find themselves under increasing pressure, left with no choice but to develop apps that provide everything customers want, are delivered on time and are continuously updated, all while ensuring a user experience that’s never short of flawless. There’s simply no margin for error, and that’s why more developers than ever are turning to continuous testing.

    Unlike the traditional approach of testing performed at the end of the development cycle, where it can potentially cause production delays should issues or bugs be uncovered, continuous testing involves running tests throughout the entire software development process. That enables developers to identify and resolve existing issues immediately—and, more importantly, deliver continuously updated and always flawless applications to their customers.

    Fundamentally changing something you’ve always done a certain way is never easy, however, and continuous testing is no exception. There’s no single lever to pull, but rather a series of steps and best practices necessary to effectively implement it within your development team. Let’s walk through the four most critical of those best practices.

    Make a Cultural Commitment to Collaboration and Quality

    As with any transformational change, people come before processes and developing the right mindset comes before developing the right tactics. Effective continuous testing requires a teamwide, cultural commitment to collaboration and quality. The notion that developers own development and QA owns testing inevitably leads to the types of problems—namely, delayed releases and flawed user experiences—that businesses simply can’t afford in the digital era. Instead, developers and test engineers must have a mutual understanding of the baseline level of quality and efficiency they collectively intend to maintain throughout the development process and a shared understanding that testing is everyone’s responsibility.

    Apply a Broad Range of Tests, Anchored by Automation

    As mentioned earlier, continuous testing isn’t about pulling any single lever, but rather about applying a broad range of tests throughout the testing process, including unit testing, integration testing, automated end-to-end and regression testing and production testing. By continuously administering tests at various points throughout the delivery pipeline, you can identify and fix issues spanning functionality, performance and security, and do so before they morph into the kind of roadblocks that delay your release and cost your business money.

    Though you should strive for variety when it comes to the types of tests you implement, you should strive for consistency when it comes to the concept of automation. If you’re serious about succeeding with continuous testing, automation is a necessity. Automated testing, which involves writing a test script (typically in Selenium or Appium) that interacts with the functional elements on a web page without requiring human intervention, not only helps you deliver better scalability and reliability than manual testing, but it also frees your manual testers to focus on more value-add activities such as exploratory testing.

    Leverage Cloud for Breadth and Scalability

    Just setting up and maintaining the infrastructure, physical or virtual, necessary to test across the ever-expanding universe of browsers, operating systems and mobile platforms is both costly and complex. And that’s before you consider the need to scale your test environment to run tests in parallel. It’s a daunting challenge for even the most mature enterprise development team.

    Enter cloud. In many ways, continuous testing and the cloud were made for each other. By leveraging a cloud-based, cross-browser testing platform, development teams can offload all the infrastructure-related headaches they’d otherwise be forced to grapple with on a daily basis, test across virtually any browser/OS combination (leveraging both real devices and cloud-based simulators and emulators) and ensure they have the scale necessary to keep their test times to a minimum.

    Develop a Strategic Approach to Analytics

    Continuous testing, applied optimally, requires a strategic approach to analytics. Developers should actively look to integrate a layer of analytics within the testing process, such that insights can be applied across the product team. Smartly layered analytics can shed light on a number of functional issues, including which browsers and mobile platforms are most often experiencing failure and whether an application’s test results are consistent.

    Beyond just functional testing, teams also can leverage analytics to assess user experience and performance, examining, for example, how the user experience is impacted by heavier traffic loads and how overall application performance is impacted by new updates to the code. Having this added layer of analytic insight enables developers to better understand the entire quality picture and deliver improvements that can be applied both in real-time and to future tests.

    The Digital Age Demands Continuous Testing

    In the digital age, your ability to attract and retain customers is directly tied to your ability to build the best products, deliver them faster than your competitors and keep them continuously updated with all the latest features and function—and do it all within the context of an impeccable digital experience. It’s a tall order, but it’s one made considerably less daunting by the effective application of continuous testing.

    — Lubos Parobek

  • DevOps Chat: App Modernization and Service Virtualization with IBM’s Matt Tarnawsky

    DevOps Chat: App Modernization and Service Virtualization with IBM’s Matt Tarnawsky

    While everyone wants to focus on building a new app, the fact is there are many, many apps already out there. They need to be maintained, updated and modernized. How do you upgrade these legacy apps to new infrastructure, frameworks, languages and more? App modernization is a thing.

    In this DevOps Chat we speak with Matt Tarnawsky of IBM about app modernization and service virtualization, testing and more.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey everyone. Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. In this episode of DevOps Chat we’re going to spend time talking about app modernization, testing, service virtualization. We will see what else kind of gets dragged in within our time frame.

    Our guest, you know, who’s going to serve as the expert today, Matt Tarnawsky. Matt, hopefully I didn’t mess that up too bad. Matt is with IBM. And Matt, welcome to DevOps Chat.

    Matt Tarnawsky: Thank you, Alan. Good to be here.

    Shimel: Absolutely. Pleasure to have you here. So Matt, first of all, I did not mangle your name too badly, I hope.

    Tarnawsky: No, you got it pretty good.

    Shimel: Fantastic. So Matt, following our tradition here on DevOps Chat, we like our guests to introduce themselves. So why don’t you tell us a little bit about your current role at IBM and also a little bit about your own personal journey that led you to be here today.

    Tarnawsky: Sure thing. So I am the product manager for IBM’s Rational Test Workbench and Rational Test Virtualization Server, specializing in API testing and service specialization. In terms of how I got here, I have had an interesting journey, starting off with working with a lot of the circuit boards and SPGAs, working with those and then testing those as well.

    I ended up migrating across to working with a small company called Green Hat Software building, AGI testing and service specialization tools. We were acquired by IBM several years ago and I’ve been enjoying the journey ever since.

    Shimel: Good for you. That’s a great story met. And you know, it’s fairly – it’s more usual than unusual in the IT space when you see someone – when we look at the arc of our careers, five years from now who knows what we will be working on. It may not even exist or we even know about it today. So it’s kind of right par for the course.

    So Matt, as I mentioned earlier, I wanted to talk a little bit about app modernization. In my mind, anyway, it’s a loaded – that phrase is a loaded phrase. We hear it bandied out about a lot but when you talk to people – to some people they are talking about, “Well, I am modernizing my existing app. So I’m taking my legacy applications and modernizing them for new infrastructures, new delivery methods, new imports, what have you.” So really about modernizing your existing legacy applications.

    To other people, though, it’s more about modernizing the way you develop, deploy, and manage applications. So modern – application modernization. Of course, no matter how we are talking about it, testing plays a vital role in this. So why don’t we first do a little level set. You know, Matt, what you – when you hear app modernization, what does it mean to you?

    Tarnawsky: So to me, it encompasses a lot of things you mentioned _____ _____ _____ _____ people talk about is, how to do I _____ _____ my app to new markets, new interfaces. That can be things like mobile. It can be things like presenting apps _____ _____ _____. And that often does require _____ _____ _____ _____ their interfaces to actually bring them into the _____ away from our systems that they may have had in the past.

    But often, it also means they want to look at the faster methods of delivery, new development styles. These things often go hand in hand. You often can’t get those faster methods of development with rapid feedback and everything else without looking at, how do we change our entire development life cycle; trying to be stuck in a waterfall method doesn’t really get you that same acceleration that you can get when you start looking at, how do we automate everything and moved to DevOps.

    Shimel: Got it. Got it. Now you broke up there a little bit back on – and I’m hoping people listening will be able to catch all of that. But let’s – so we may fill in. But let’s talk a little bit about the role of testing and that framework of app modernization. Obviously, testing is important, but how so and in what context?

    Tarnawsky: So I actually see – you know, if we are taking older, monolithic applications and looking to migrate them towards micro-services, it’s a great opportunity for testing. You are putting things into small units that can be tested on their own. You get things like – let’s say you’re migrating to a _____  based _____ based service, you get the opportunity to describe it with _____ Swiger or _____ or open API. And you’ve got a contract there that you can then test against.This means that things now have the parameters, the what needs to be tested, how to test it, and it can be imported into many different test tools to help them start testing, but it’s smaller bits of functionality. So it’s a great opportunity to break things up and really look at how those can be worked with at a smaller level.

    At the same time though, it also introduces new opportunities for more complex interactions that we didn’t previously anticipate. So let’s just say you’ve taken that big monolith. You want to expose micro-services to the rest of the world or indeed to other teams in your organization well, suddenly the integration that you had relied on in the past might not be the same ones that you are looking at using in the future. You need to start looking at how we test not just those smaller levels of the micro-services, but also looking at how we test each of those integrations over time.

    One of my customers has a great analogy that they like to use in terms of the testing. They said when you’re testing a car, you don’t just test the car. You test little pieces of it. So you might start off by testing the spark plug. Then you might say we will integrate some pieces and see how those work together. We test the engine out. Then at some point in the future, we will put the engine in the car and we will test that as – you know taking it for a test drive, taking it for a spin, and make sure the entire end-to-end system works as well.

    Shimel: Excellent. Excellent. Actually, that’s a great analogy. Great analogy. So Matt, another phrase we hear, and we’ve all been hearing it, a lot of us, in IT for a long time now, is service virtualization. I think people have a better handle on what we mean when we talk about service virtualization. But again, let’s lay a foundation. When you talk about service virtualization, especially in terms of app modernization and testing, what does that mean to you?

    Tarnawsky: So service virtualization is where we take some sort of interface that we might have had that could be something silent rest. It could be something like Mqueue. It could be a legacy mainframe interface like Kicks. We provide some sort of mock that is going to handle the interactions with the interface. That can be coupled on the rest of the application. It starts to be providing us – I’m sorry.

    It starts providing us with the ability to look around and – I’m sorry about that. Got myself a little bit jumbled there.

    Shimel: No problem.

    Tarnawsky: Basically, it gives us the ability to remove restrictions from ourselves. So if we had legacy items in our environment that we might not have in the cloud yet, well actually, we can do is we can say, well, look, that mainframe connection, that SAP connection, whatever it might be that’s in our current on-prem environment, we can virtualize that, we can house something that sits in the cloud. And in fact, our developers can have an entire environment in the cloud that is no longer linked to those legacy systems.

    Shimel: Fair enough. So how does this all play out then, Matt? Right? So now we’ve got our little parts here and we’ve defined what we mean by this. In your world, where are you seeing sort of rubber meets the road, if you will, in terms of organizations deploying, employing, doing this kind of stuff? Make sense?

    Tarnawsky: So we see all sorts of things. We do see people looking at that – we want to do a migration. So we are going to record various pieces of environment and have them available to replay. Just very, very basic style of creating virtual services, but it can be effective. And indeed, they can do the same sort of thing for their API testing as well and just say, “Have we migrated this correctly? Does it still have the same functionality we had last week when we had it on-prem as we got on the cloud now?”

    That’s a very effective way to look at things. But equally, we see all sorts of interesting stories come up from customers, whether that’s looking to virtualize the way a third party – so we’ve seen customers look at things like Apple Pay or Experian or whatever, where they have to pay every time, they had that third-party service. Or indeed, that might just don’t have control over it and they want to simulate what happens if that third-party service responds or throws an error code or whatever.

    We see customers who do some really – one of my favorite stories comes from a customer who was looking at testing out their credit card issuing systems. Now, you can imagine that as you’re looking at issuing a credit card, at some point you’re going to print a credit card. Now, just imagine test automation for that sort of thing. Do you really want credit cards popping out of the machine? No you want to be able to test things effectively, but not have these physical dependencies.

    So what they did what they looked at the traffic that was going through that credit card machine, the issuing machine. They said, here we got a point where we can virtualize some things in our environment and remove this dependency that we currently have. We don’t want to print real credit cards. It’s a waste of time, waste of resources, and in fact, it slows that test automation down. So if we can virtualize that out, we can get faster feedback back to the development team. So that’s exactly what they did.

    Shimel: So Matt, we recently did a webinar over on devopsy.kinsta.cloud you are probably aware of it, modernizing testing as apps re-architect. Is this the kind of stuff we are talking about?

    Tarnawsky: Very much so. You want to make sure that your testing stays in pace with that re-architecting of the apps. A lot of that does come down to developers delivering fast from the cloud. And tests more and more need to be automated. That doesn’t remove the value of exploratory manual testing or anything like that. But we do see that need for testing to become more and more automated in the future as well, as this change happens.

    Shimel: Absolutely. You know, another thing, and I get this a lot, Matt, when I’m traveling around the world. Is this a sort of – not to use the – abuse the term, but first world problems, right? Is this where… In terms of the market, is it sophisticated, large enterprises who already have legacy infrastructures, a history of service virtualization and so forth? Or is this something that’s really kind of more grassroots that’s happening across the board, across the globe, as organizations are starting to move to app modernization and dealing with the realities of having to test around this stuff?

    Tarnawsky: So we see all sorts. Actually, the example I gave you earlier, the bank that was looking at the credit cards, was a fairly new bank here in the UK.

    Shimel: It was? Okay.

    Tarnawsky: They were – that they are very much based on looking at how they can do things in the cloud, trying to keep things as modern as they can. They are looking for all the latest techniques, all the newest tools that can help speed up their development and their time-to-market. So yes, we do have a number of customers using this stuff for legacy enterprise systems. They are looking to virtualize the way a mainframe, for example.

    But equally, we see other customers come to us and say, “I want to accelerate my development and my testing as fast as we can. What can you do to help us?”

    Shimel: Good.  Interesting. You know, Matt, I mentioned when we started that the time goes pretty quick. We are coming up on the end of our time frame here. But for people who maybe want to get more information on this, where can they – where can we dig? I mean, I should mention that the webinar I referenced earlier is available on devopsy.kinsta.cloud and YouTube, a recording of it along with the slides. So people who are interested there should absolutely head on over and get that. We will try to include the URL in our show notes for today’s podcast.

    But where else should they go, Matt? How do people get smart on this?

    Tarnawsky: So I would invite them to come on to IBM.com, searching for Rational Test Workbench or Rational Test Virtualization Server, depending on whether they are interested in test automation or service virtualization, as a starting point. And we’ve got loads of white papers. We’ve got customer stories. We’ve got all sorts of things you can look at for more information just to see how other people have been doing things.

    Shimel: Fantastic. Matt Tarnawsky, thanks for being our guest on DevOps Chat today. Pleasure to have you on. Hopefully we will have you on again soon.

    Tarnawsky: And thank you very much for having me.

    Shimel: All right Matt. Hey, this is Alan Shimel for devopsy.kinsta.cloud. You just listen to another DevOps Chat. Have a great day, everyone.

    — Alan Shimel

  • Reinvent Testing to Galvanize Digital Transformation

    Reinvent Testing to Galvanize Digital Transformation

    We’re well past software “eating the world.” According to Tricentis CEO Sandeep Johri, software has become irreversibly intertwined with the world. You can’t find a modern enterprise that doesn’t depend on software, or a business transaction that doesn’t require software at some point in the end-to-end process. In the new world of digital business, companies that deliver differentiating customer/user experiences through software have the clear competitive advantage. Those that cannot are living on borrowed time.

    In response, every CIO is now focused on digital transformation initiatives that ensure the company is disrupting, not disrupted. From an IT perspective, this requires faster delivery of innovative software and greater agility—the ability to pivot as soon as you identify a new opportunity or challenge.

    Everyone agrees that this is a priority. Developers are producing software faster than ever before, and Ops can now deliver applications in a matter of seconds. So, what’s holding us back? Testing.

    When Johri talks to CIOs about how their digital transformation is progressing, the same basic story always emerges: Organizations transform Dev and Ops, but then testing cannot keep pace … and they get stuck. The tools and processes architected for traditional months-long release schedules simply don’t fit modern delivery cadences, which require immediate quality feedback with each new build. It’s a sad but simple fact: If you have a slow testing process standing between highly accelerated development and operations processes, there’s just no way that you can achieve the desired delivery speed.

    Many companies have already recognized this—and they’ve galvanized their digital transformation initiatives by modernizing their testing. By re-examining and re-inventing software testing across their organization, they went beyond removing the “testing bottleneck”—they’re also driving the positive user experiences that can now make or break a business.

    Many of these distinguished quality leaders joined thousands of software testing professionals from around the world at Accelerate 2018 in Vienna, Austria. This year, the conference theme was “Digital Transformation Requires Continuous Testing.” To ensure that the conference thoroughly covered both the digital transformation and continuous testing sides of that topic—as well as the core intersection between them—we invited CIOs as well as quality leaders to share their perspectives. Here are some of the key insights that are most pertinent for executives leading digital transformation initiatives.

    Archaic Testing Approaches Can Cause a ‘Brain Drain’

    Merck sought out top development talent to advance their digital transformation initiatives. When those developers started quitting, (now retired) Merck CIO Clark Golestani wanted to know why. He learned that the No. 1 reason for these developers leaving the company was the company’s testing practices. Developers were “fed up” because the company was still using traditional testing practices within Agile development processes—and developers believed it was undermining their hard work and dedication.

    That’s the precise moment that Merck launched its testing transformation. Ultimately, the testing organization changed the way testing was done, dramatically reducing attrition and enabling faster delivery with superior levels of quality. According to Golestani: “If it weren’t for the testing organization driving that change, they would have been an inhibitor. We would have never been able to release the types of products we do, and the technology would never work the way that it needs to. It’s absolutely essential.”

    Agile, DevOps, Cloud and Testing are Inextricably Linked

    Agile, DevOps, cloud and testing transformation are all interlinked as part of a broader digital transformation. Andy Sturrock, CIO of one of the world’s largest energy companies, stated: “All of these things are self-reinforcing. You certainly don’t get the full benefit of one without doing the rest.”

    He explained the Ops team used to be the primary bottleneck to moving applications to production (just getting servers provisioned could take weeks or months). Now, the cloud relieves that bottleneck—and that places a lot more pressure on the “apps” team, which tends to emerge as the new bottleneck. “You can write code as fast as you want, but if you can’t test it, you’re not going to get into production quickly,” he said. “Testing has to become automated, and the role of the tester must evolve.”

    When Software Runs Your Business, Software Testing Becomes a Business Issue

    According to Golestani, nearly $1 billion US in revenue passes through Merck’s SAP instance each week. Consider that this instance is updated at least four times a year, and more than a thousand changes are made each time. Manual testing is simply not an option. No amount of manual testing can thoroughly de-risk the core business processes, given the frequency of updates, scope of changes and breadth of functionality that’s impacted.  Extreme test automation is the only way to achieve the required business risk coverage with the necessary speed.

    Due to the nature of Merck’s business, lives also are at stake. Merck ships billions of medicines and hundreds of millions of vaccines each day. The company’s continued focus on digitizing the business means that software runs everything from the shop floor to the way that they interface with customers and educate 4 million healthcare practitioners. As Golestani explained, “When the software driving all this fails, nothing ships, manufacturing shuts down, research shuts down … and people die.”

    It’s Not Easy, but Transforming Enterprise Testing Pays Off—Significantly

    Global 2000 organizations that want to implement continuous testing—the extreme test automation required for DevOps—need to overcome challenges such as:

    • Highly complex application architectures (e.g., processes that cross browsers + APIs + mobile + SAP + other packaged apps + custom Java/.NET apps + mainframes).
    • Different application types, planning cycles, development methodologies and tools existing in parallel.
    • Deeply ingrained quality processes across different groups and projects.
    • Scarce test automation resources.
    • “Bloated” test suites that delay the process while providing limited business value.

    There is no silver bullet. Transformations take time and they must involve all of the people, processes and technologies associated with your application delivery pipeline.

    Nevertheless, you can rest assured that your efforts will pay off. Consider just a few examples of how organizations that committed to transforming testing are now growing the business and enhancing the customer/user experience:

    • Duke Energy achieved a $9 million cost savings during the first year of its testing transformation—and it also reduced testing time by ~ 85 percent. This enables the company to more rapidly deliver software that transforms the customer experience, modernizes its energy grid and creates a smarter energy future.
    • SPAR accelerated time to market 3X, enabling the company to expand its online presence across Europe, deliver more customized shopping experiences and provide seamless “omnichannel” experiences across the customer journey.
    • EdgeVerve now releases 30 percent more new functionality in cycles that are 60 percent shorter with 50 percent fewer escaped defects. This enables the company to help its global banking clients onboard, sell, service and engage customers with flawless personalized experiences.
    • ExxonMobil saved $140 million in cost avoidance and increased productivity 3X. This enables the company’s IT department to take on more projects and complete them faster—which in turn helps the company boost sales and increase customer loyalty.
    • Vantiv (now Worldpay) transitioned from eight-week testing cycles to daily testing, which has been critical for enabling rapid growth and agility during the recent Vantiv-Worldpay merger.

    Continuous testing was once just an aspiration. Now, the drive toward digital transformation has made continuous testing a business imperative. Organizations such as these are paving the path forward, demonstrating that continuous testing can—and must—become a reality in enterprise organizations.

    — Wayne Ariola

  • SmartBear Reinvents Load Testing for Web Applications with LoadNinja

    SmartBear Releases Groundbreaking Technology that Reduces Script Creation Time by 50 Percent, Replaces Emulators with Real Browsers, and Arms Agile Teams with Metrics that Matter to Diagnose Problems at Ninja Speed

    SOMERVILLE, Mass. – Oct. 29, 2018 – SmartBear, the leader in software quality tools for teams, released LoadNinja, a cloud-based platform for engineers and performance professionals who load test web applications. LoadNinja delivers a rare combination of efficiency, accuracy, and actionable performance data, allowing teams to incorporate load testing easily in an agile and devops environment. Unlike traditional load testing platforms that are protocol based and require tedious programming and correlation of dynamic application aspects, LoadNinja allows the engineer to focus more on building applications that scale and less on building load testing scripts.

    “Application performance is critical to ensure the continued success of a digital business,” said Anand Sundaram, VP of Performance Testing and Monitoring at SmartBear. “Teams struggle to incorporate performance testing of their applications due to the unreasonable amount of time it takes to get the load testing scripts right using conventional tools. LoadNinja removes this manual effort, helps you generate the most realistic load using real browsers, and provides actionable performance data about the behavior of the web applications under conditions of stress.”

    Conventional protocol based tools have a steep learning curve and require tedious programming of recorded scripts, before they can be played back. LoadNinja, with its InstaPlay recorder, eliminates this step so recorded scripts can be played back immediately. The efficiency gains from this approach, especially when load tests are run frequently in an agile and devops setting, enable teams to focus their efforts solely on load testing and saves time previously spent in creating and massaging scripts in advance of load tests.

    Unlike the load generated by conventional tools that use vendor developed proprietary browser emulators, the TrueLoad technology within LoadNinja uses real browsers at scale for load tests, creating the most realistic and accurate representation of load on the infrastructure supporting the web application under test.

    The power of using real browsers for all of your load generation is also seen in the performance data returned by LoadNinja. Traditionally, load testing tools return request-response times in test results, which need to be further introspected and deciphered in order to be usable by developers and performance testers. LoadNinja obviates this additional step and directly provides actionable data in the form of browser based navigation timings that developers and performance testers are used to working with every day.

    LoadNinja also provides two additional innovations in the form of the VU Inspector and the VU Debugger. The VU Inspector allows you to visualize the degradation as your virtual users generate load against the application. When a virtual user encounters an issue, the VU Debugger can be used to connect and interact with the erring virtual user’s browser to gather additional diagnostic information about the problem.

    LoadNinja is being unveiled to customers at SmartBear Connect 2018, the company’s annual user conference, taking place on October 29-30, 2018 in Boston. The event brings together hundreds of testing, development, and technology enthusiasts from around the world to share ideas and get hands-on training on the company’s software quality products. Stay up-to-date on Twitter using #SBConnect. 

    To learn more about LoadNinja, visit: https://loadninja.com/

    About SmartBear Software Supporting more than six million software professionals and over 22,000 companies in 194 countries, SmartBear is the leader in software quality tools for teams. The company’s products help deliver the highest quality and best performing software possible while helping teams ship code at nearly impossible velocities. With products for API testing, UI testing, code review and performance monitoring across mobile, web and desktop applications, SmartBear equips every development, testing and operations team member with the tools to ensure quality at every stage of the software cycle. For more information, visit: http://smartbear.com, or for the SmartBear community, go to: LinkedIn, Twitter or Facebook.

  • DevOps Chat: Perforce Acquires Perfecto Mobile

    DevOps Chat: Perforce Acquires Perfecto Mobile

    Consolidation is in the air, as two well-known players in the DevOps space—Perforce and Perfecto Mobile—are joining forces. Perforce has announced its intent to acquire Perfecto Mobile for its mobile and web automation testing technology, which, when combined with Perforce’s existing portfolio, will better enable organizations to achieve continuous testing at scale. It’s a great combination and one I believe will be well-received in the industry.

    Perforce CEO Mark Ties and Perfecto Mobile COE Eran Yaniv spoke with me about the acquisition and its potential impact on the testing space as a whole, and Perforce’s plans for more acquisitions down the road.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hey, everyone. It’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat is breaking some big news. The news involves the merger – acquisitions and merger of Perforce and Perfecto Mobile, two pretty well-known brands in the DevOps space. News just came out this morning and we have with us the CEO of Perforce, Mark Ties, as well as the CEO of Perfecto, Eran Yaniv, and if I mispronounced anyone’s name I do apologize in advance. But gentlemen, welcome.

    Eran Yaniv: Thank you.

    Mark Ties: Thank you very much. Great to be here.

    Shimel: Thank you. So guys, you know, I was lucky enough to sell a company or two in my time, and then one of the companies I helped found we did about 30 acquisitions in 36 months so it was almost one a month. It was a little crazy. But, you know, a wise man much older, wiser, and wealthier than myself told me that any time you can, you know, found a company and someone buys it, whether it be a merger, acquisition, whatever you want to call it, but you find a suitor who’s willing to pay significant money, it is a – it is something to be proud of, right? It’s an accomplishment. So congratulations to you,  Eran. You know, any time you can CEO a company that has some sort of liquidity in that, it’s a good thing. So congratulations. And Mark –

    Yaniv: Thank you very much.

    Shimel: – on the other hand – thank – you’re welcome. Mark, on the other hand, any time you could, you know, take the helm of a company that’s out here and getting entrepreneurs to actually, you know, trust you with their baby, so to speak, and it shows a level of trust that you will do right by them and right by their customers, so congratulations to you as well.

    Ties: Thank you.

    Shimel: Let’s dive in a little bit though, Mark. What kind of drove this? What drove the acquisition here or the merger?

    Ties: Since we acquired the management team at Perforce, acquired Perforce a couple years ago, and we’ve been on a journey here to create what we believe will be a strong DevOps player in numerous assets of the space, and as of recently we’ve been very focused on the testing environment witnessed by our earlier-in-the-year acquisition of PRQA and Static Code Analysis and now Perfecto, so Eran and I have been chatting for a number of months here and we’re just, you know, absolutely excited, ecstatic, that we’ve gotten the opportunity here to partner with his cofounder Yoram and himself and the rest of the Perfecto team to bring their capabilities in web and mobile testing into the Perforce family, so that’s really what drove it, is our focus on testing and then their industry-leading capabilities.

    Shimel: Sure. Eran, what about you? You know, Perfecto Mobile – I mean I’ve had devopsy.kinsta.cloud now for four and a half years with publishing. Perfecto’s always been on the scene. We’ve had the pleasure of doing webinars and reports and conferences, sponsorships with Perfecto. What was the, you know, impetus to do this deal now?

    Yaniv: Well we were kind of discussing this internally for a while now, and when we look at the market we believe because the market is evolving more and more towards implementing DevOps-related processes when there is a requirement with which we call internally a gorilla in the market. The decline of CAs and, you know, IBMs and the HPs of the world is actually opening up – has actually opened up quite a few opportunities and we thought that – we think that the development of a very strong leader in the DevOps market is something that is required, that the market is actually requiring. And so we, you know, were looking around to see what companies are out there that would make sense for us to merge with or acquire or get acquired; something that would create something of a bigger size and a bigger voice in this market.

    And, you know, there were various options but we – you know, I really like this notion Perforce because I think nobody in the market really saw it coming, and for us it’s a very exciting opportunity because the offerings are complementing – even complementing status in terms of the install base, so the opportunity was there to actually make it happen and, you know, it aligns really nicely with what we were thinking about the market, so we went forward.

    Shimel: Absolutely. Well, you know, if I may add my own armchair quarterback –

    Yaniv: I’ve got to say one more – I’ve just got to say one more point that I think is very important, and that is the initial chemistry with the team at Perforce made it a very easy decision.

    Shimel: You know, Eran, on that point I will tell you just this culture is important in DevOps in doing mergers and deals like this. The chemistry of the teams is as important or more important sometimes than even the numbers in the books, right? Because the merger doesn’t end on the day of the merger; it only begins. And, you know, it’s important that you have that in there. If I can offer my two cents as an armchair quarterback, I think what we’re seeing here is also the result of a larger trend in the DevOps space.

    You spoke about CA, which is, you know, now getting acquired by Broadcom and there’s perhaps uncertainty there, the IBMs – you know, certainly while these companies played in the DevOps space they weren’t necessarily perceived leaders in DevOps, in the DevOps space and, you know, when you see a company like GitHub, which started out as a code repository, right – it did open source good – getting purchased for, you know, this $7.5 billion by Microsoft, it wasn’t just as a code repository, obviously. And, you know, just recently in the last two, three, four weeks companies like GitLab, you know, with $100 million raise, $1 billion-plus valuation, last week it was JFrog with a DevOps platform with $165 million raise, over $200 million raised – you would have to assume they’re also over $1 billion – it certainly seems like the time is ripe for size.

    Size matters in DevOps, right? Size and scale matter. And so, Mark – go ahead.

    Yaniv: I was going to say that we are seeing these things actually happening in the market. _____ is making all kinds of –

    Shimel: Sure.

    Yaniv: _____ primarily in the _____ area. The thing that I like about the Perforce Clearlake, if you will, in conversations with – and this is what I really like, is the fact that they are dead serious about what they are doing and their moves are very, very quick and they’re very nimble in their decisions and that’s something that makes a lot of sense and it also coincides with what DevOps is all about, right? So I think that’s a great addition.

    Shimel: Absolutely. Mark, let’s hear from you. You know, what’s Perforce’s take on this? Are you in violent agreement or maybe something else to it?

    Ties: I’m in violent agreement with the number of the comments that you’ve made. You know, the space – you know, the research that we’ve done around the space for the last three-plus years now is that this space and environment is going to consolidate, and as a result of that consolidation, you know, those players that can put together best-of-breed technology, that can bring the value that the customer’s ultimately desire or need will be – you know, they will be the players that will rise to the top, and we believe we’re doing that with our version control solution _____ Perforce with our _____ solution and now with web and mobile solutions with Perfecto along with the other technologies that we have. So I’m in violent agreement with that. You know, this won’t be the last acquisition that Perforce will do.

    I mean we’ve done five now; I could see us easily, you know, doing five additional acquisitions and continuing to grow this business. So that’s one point I’m in violent agreement with. The other point is that, you know, to your earlier point about the people and the culture and Eran’s as well, you know, it’s hard but it’s easy to buy a company. It’s harder to make that company successful –

    Shimel: Exactly.

    Ties: – during the acquisition, and I think all of the personalities on both sides of the fence with the Perfecto team and with our team are focused on one thing, which is creating a great company that has great products, phenomenal customer support, and then ultimately new customers. So it’s just – this is a great marriage and, you know, I look forward to continuing to drive it with my team with Eran and with Yoram and the entire Perfecto team.

    Shimel: Absolutely. On that point, let me ask what are the plans? Eran, will you be staying on? What will your role be? What about the rest of your executive team?

    Ties: Yeah so, you know, again, we’re in the days – we’re in the early days here, but the hope on my end is that Eran and team stay around and Eran will be leading this business as an independent segment, subsidiary, whatever you’d like to call it of the Perforce family reporting directly into me, so that is the plan. Now we do have teams obviously from both sides of the house working on the formal integration and how we’ll bring the companies together over the days, weeks, and months to come. We’ve just signed the agreement here earlier this week and we hope to close it by the end of the month.

    And, you know, we will take our time. We will be very diligent in what we’re doing here so that we do not disrupt either businesses, because right now both businesses have been very successful.

    Shimel: Excellent. Mark, I’ve got a question for you. You know, with each of the kind of deals and money raises that I mentioned earlier, I’ve spoken one-on-one with the CEOs, and one of the things I’ve asked them is what is their vision for how their company – you know, where does their company fit into the larger DevOps? You know, so for instance, GitLab wants the first true end-to-end DevOps product line. Shlomi Ben Haim at JFrog, they want a true DevOps platform that handles all artifacts, anything related to artifacts.

    Can you succinctly give us what is the vision – right? – for Perforce? What – you know, when you’ve done 5 more acquisitions, 10 more acquisitions, whatever it is, what’s the vision for what Perforce will become and what you’re striving for?

    Ties: Yes. So there are some very grandiose visions out there, and you’ve mentioned a couple of them. Ours is very simple; we want to be the enterprise player at scale. I mean we provide a very strong solution to enterprise customers that are looking for speed, scalability, security – to name a few of the attributes of the products that they’re purchasing and look to acquire as their providers. So that’s what we’re looking to do, win every inch. All right? We’re not gonna win every market vertical, but the verticals that we choose to go down our plans are to do nothing but dominate them, and so that’s why we’re looking for industry leaders like the Perfecto solution, like the Perforce solution and the others that we’ve acquired, to make sure that we continue to demonstrate the need that our enterprise customers have.

    Shimel: Fantastic.

    Yaniv: And this – and if I may, investing in the vision that Mark articulated around the enterprise aligns extremely well with what we were thinking back in a narrow – if you will, in a narrow space, which is around the continuous testing. But in the notion of addressing the enterprise and the specific ____ that we would be strong at, would make the vision, the joint vision, something that is really great, and that made it also very, very easy for us to _____.

    Shimel: Excellent, excellent. Well, gentlemen, let me wish you both success and congratulations. We’re about out of time here on this DevOps Chat, and I’m sure you’re both very busy. It is an exciting day. Maybe we can check back in a few months when the dust settles a little bit and see how things are going and where you’re headed from there, all right?

    Ties: Great. More than happy to –

    Yaniv: I’d be happy to.

    Ties: – continue the conversation. Thank you.

    Shimel: Okay. Mike Ties, CEO of Perforce, Eran Yaniv, CEO of Perfecto, Announcing the merger of their two companies, and as Perforce continues to scale in the DevOps space. Thanks for being this episode’s guest on DevOps chat, gentlemen. This is Alan Shimel for devopsy.kinsta.cloud and you’ve just listened to another DevOps Chat.

    — Alan Shimel

  • The Benefits of Test-driven Development

    The Benefits of Test-driven Development

    It sounds like an oxymoron: Create test cases before coding. In the process of writing and using test cases before writing functionality, a developer can produce higher-quality software. That is the concept behind test-driven development (TDD), and it works.

    TDD Snapshot

    TDD has been around since at least 1999, and it is part of the emerging test-first approach to development associated with extreme programming. In 2003, American software developer Kent Beck “rediscovered” TDD and applied it as a way to create simple designs and inspire developer confidence. Fast forward to today’s agile development world, TDD is a software development process using a very short feedback loop in which developers do the following:

    1. Create a test that will fail immediately.
    2. Write the necessary code to pass the test as quickly as possible.
    3. Refactor the code written in step two, using the created test as a reference.

    TDD Benefits

    So why might a long-in-the-tooth software development approach have tangible benefits in an agile development world? Because put in its simplest terms, TDD is a way to develop highly usable software.

    Following the TDD process, a developer must focus on the test cases before actually coding anything. That meansa developer is thinking more about the use of the software and the design of user interfaces to achieve that. As a result, the developer is more interested in the interface than in the implementation—and that leads to more usable software.

    There are several other tangible benefits for taking a TDD approach, including:

    • Creating a metric for the code quality that can be easy and quick.
    • Allowing for a quick visualization to determine if the code base has any functional issues.
    • Living documentation of functionality with new code written.
    • Allowing for safe refactors of code, whether based on attempts to improve code quality or a changed requirement.

    That final point is worth dwelling on for a moment. The TDD approach requires that the evolving code base is cleaned up frequently so that new tests and code are easy to introduce. That usually means that code moves from its current place to where it more logically belongs in the software. That has the effect of reducing any extraneous duplication of code and enforces a strong discipline around objects, classes, modules, etc. In this way, the overall maintainability of the software increases incrementally.

    The improvement in readability and maintainability will pay huge dividends during the expected life cycle of the software. Following a TDD methodology requires developers to focus on writing smaller units of testable code. By following the TDD methodology, this leads to more modularized, flexible and extensible software.

    Where Can TDD Fit?

    A TDD approach is appropriate for both new greenfield software and legacy systems. For a development team that has to deal with existing legacy software, the key is to start small and begin with working on bug fixes. A good practice is that for each bug reported, create a test that addresses the broken bug and then fix the functionality. After several iterations of this, the development team has created a repeatable working test to address bug fixes. When applying this methodology to new software applications, make a point of understanding the testing tools used for the technology stack.

    For example, when working in an Angular application that generally uses the Jasmine testing framework for unit testing, and when using the Angular CLI for creation, unit tests are created alongside modules of code. Using a TDD methodology, the approach would be to:

    1. Determine a portion of the functionality to be created with this component.
    2. Create a unit test that will fail immediately targeting this portion of functionality.
    3. Run the test runner to confirm the failing test (it may be useful here to leave the test runner on to run after each source file save, which speeds the process).
    4. Write code in the Angular component that will make the written test pass.
    5. Make any refactoring changes for the Angular component after confirming passing, using the test as a guideline to ensure code refactoring does not break functionality.

    Measuring Testability with Code Coverage

    Another essential consideration when improving testability in code is to use a code coverage tool. Code coverage is a metric to show the percentage of code that has a unit test written for it. Angular applications use Istanbul to calculate code coverage through the application. Running a single run of code coverage in an existing project gives the following output:

    The output provided by Istanbul gives a metric on the overall test coverage and the areas of code that need improvement in testing. Code coverage can be useful in a few ways:

    • Provides an idea of the overall testability, allowing for a threshold to ensure the total software testability does not drop past a certain point.
    • Identifies areas in the code base with poor testing, making them opportunities for refactoring.

    However, as effective as code coverage can sound, it’s important to understand that it’s just a metric. Writing good unit tests is a matter of following what the code will do, and metrics such as these should not drive significant decisions.

    Considerations When Using TDD

    It’s important to note that TDD does not solve all problems. There are many different types of testing required to create a fully comprehensive testing strategy, including acceptance testing. In TDD, the focus is on a single unit of code at a time. A sophisticated software application may have many thousands of units of code and their corresponding tests. That’s why it’s critical to make sure that testing quality stays high when following the TDD methodology. Tests cannot become something bypassed in the pursuit of more functionality or expediency. Avoiding testing generates the risk of test creation becoming a hindrance for developers.

    For example, ignoring failing tests makes it difficult to determine the actual state of the application. It’s also important to have buy-in for a TDD approach form all the teams involved in the effort. Buy-in is particularly true on the business side. Time must be spent up front discussing the nature and benefits of a TDD approach, and the belief that using TDD will improve the resulting software. Otherwise, business management sees writing tests as an activity that doesn’t contribute to the bottom line.

    Conclusion

    TDD emphasizes the importance of an effective and sustainable testing approach. TDD also contributes directly to the overall quality of software. It’s a truism for small or large system development that often goes missing in the day-to-day hustle to get new functionality into production. Quality software gets built when there’s an acknowledgment that quality test code should receive the same amount of attention and resources as quality production code, as they are equally essential in development.

    — Dave Farinelli

  • The Art of Software Performance Testing

    The Art of Software Performance Testing

    When users turn away from applications or have critical feedback, it’s often because the application is slow, hangs/freezes or, worse, it isn’t available when needed. In this era of limited—or, zero—patience for app instability, delivery teams should ensure that their performance testing capability is sound. First though, let’s start with definitions. Despite popular opinion, performance testing is not the same as load testing. Load testing checks how systems function under a heavy number of concurrent users performing transactions over a certain period of time, while performance testing is much broader in that it checks how the system behaves and performs. Performance testing examines responsiveness, stability, scalability, reliability, speed and resource usage of your software and infrastructure.

    Performance testing doesn’t always get the attention it deserves. Testers have a tendency to focus on features and component functionality, even when verifying performance on application architectures designed leveraging APIs. As software development methodologies and application infrastructure evolve, so should performance testing. We observe that in Agile teams, performance testing typically executes at least one sprint behind other testing methods. Performance testers don’t start testing until the application’s features are deemed stable. They operate separately from the delivery team. Yet, performance testers should run tests as soon as new code is available, which in turn helps developers receive real-time feedback on issues that they then can fix immediately.

    To ensure a successful strategy in validating the performance of an application, here’s a look at three fundamental performance test categories:

    • Endurance: This involves running a transaction or scenario continuously for a period of time to detect memory leaks or anything else that causes a system to slow down.
    • Breakpoint: The goal here is to increase the load on a system until it reaches a threshold, after which it crashes. These tests are critical to understand at what point the system or underlying components will buckle.
    • Scalability: By increasing the number of concurrent users, instances or database size, you can measure how applications will handle growth.

    Performance Testing: Recommended Tools and Skills

    Performance testing is difficult without automated tools that can simulate scenarios and user activity so that you can test comprehensively. Many good tools are free or available as open source. Some of the popular ones include JMeter, Grinder and Locust.io.

    To be successful, performance testers need a DevOps mindset. Technical skills are a must, since understanding software code behaviors and interpretation of hardware and software utilization is necessary. An understanding of how to interpret the technical concerns observed from the business perspective is also a great skill to build. Tests should always answer a “what if” scenario and occur within the sprint. Hence, the ability to apply a continuous testing approach as part of the holistic testing strategy is a great skill to learn! Finally, learn how to provide real-time feedback to developers, as a form of hypothesis about changing code to support better performance tracking.

    Guidelines to Make Your Performance Testing a Success 

    These steps will  help structure your performance testing to achieve results for the business success of the delivered application:

    1. Map your tools and environment. You’ve got to understand the environment infrastructure that will host the application under test. The ability to quickly configure, build and tear down the environment to test is a crucial time-saver.
    2. Set acceptance criteria. Getting concise goals of application performance among all stakeholders will chart your path and allow you to gauge success. An example metric is page load times for your mobile application.
    3. Define KPIs. The acceptance criteria will dictate the KPI goals for performance tests. These may include percentile response times, throughputs, garbage collection performance, heap utilization and resource utilization.
    4. Set up test runs, execute and monitor. After completing the preparatory steps above, you can define test scenarios and create the test cases. Don’t forget to incorporate monitoring. This entails creating alerts in the testing environment to ensure that monitoring services are working as expected.
    5. Analyze, repair and assess. After each performance test cycle has completed you’ll want to assess the results against your acceptance criteria and fix any problems immediately. Over time, acceptance criteria will change and so should your test scenarios and anticipated outcomes.

    As with all testing activities, early involvement and collaboration with developers, analysts and business stakeholders will save time and headaches later. The more you know about the technology stack and upcoming application changes, anticipated maturity of the application and evolving business goals, the better prepared you can be for any testing need as it arises.

    In the evolving shift to a modern software development environment, performance testing has become central to building confidence in software. It’s time we give it our full attention.

    — Mush Honda

  • Collaboration: The Glue for Test Automation Maturity

    Collaboration: The Glue for Test Automation Maturity

    QA’s role in DevOps continues to evolve in positive ways. In a recent survey on test automation conducted by QASymphony and TechWell, nearly one-third of organizations reported that among organizations where testers and developers share testing responsibilities, about a third of test cases are automated. This survey and others show that on average, fewer than 20 percent of organizations have adopted test automation.

    What we can discern here is that when testers and developers work closely together, organizations can achieve greater levels of test automation maturity. This is important, because test automation ties in closely with the primary goal of DevOps, which is to develop high quality software, rapidly and reliably. There’s no way to reach velocity at scale without test automation maturity—and QA teams play a critical role in getting there.

    Let’s break this down a bit further. How does developer-tester collaboration contribute to test automation maturity, and what exactly do we mean by that?

    Once developers complete unit testing, their job is to communicate the findings of those first level tests and code verifications to the testing team. If they don’t, testers are somewhat in the dark when determining what other tests—whether functional, API or performance—to build from there. Developers have the first line of vision on problem areas in their builds. It’s not enough for testers to look at a dashboard to see that a code branch has passed a test. Test automation requires speed and accuracy, so the more detailed information testers receive upfront the more successful they will be in writing automated tests and in determining manual testing needs.

    Developers should also automate unit-level testing. DevOps requires more tests, more often; everyone should take advantage of the efficiencies that test automation brings.

    Making Collaboration Work

    How to exchange information is less important than just doing it. There are lots of options these days: Developers and testers can share information back and forth within DevOps code repositories and pipeline tools systems, such as GitHub and Jenkins, or by using collaboration tools such as Slack. Geographically-dispersed teams may prefer web conferencing systems to facilitate live discussions on the fly.

    Team members can sit side by side, doing what’s called paired development. It’s powerful for testers to watch the code as developers write it so they can ask questions during the process. That gives the tester more knowledge about the product and in turn, developers understand the tester’s objectives. The point here is that achieving a high level of test automation coverage isn’t easy; testers and developers are both learning new skills and processes. Through frequent exchange of information, they can make progress faster and always be on the same page.

    Clearly though, with such low average test automation adoption levels, we need to work on closing the collaboration gap.  Here’s how I see the barriers:

    • Developers are fiercely independent. While that’s not a surprise, it requires incentives and education to help developers understand the need to get out of their silos. Developers must adopt test automation and talk to the testers. If they don’t adapt, they won’t last—at least that’s what a CIO told me recently.
    • Testers and developers have been in opposition for too long. It’s time to get rid of this “us versus them” mentality. Developers sometimes think testers are just trying to break their code and criticize their work. Testers sometimes complain that developers could care less about writing buggy software. This is a cultural problem and leadership can help resolve it by promoting software quality as an organization-wide priority for which everyone shares responsibility.
    • Testers and developers don’t always speak the same language. The thinking behind how developer writes code is not the same as how the tester writes automated scripts. Developers don’t consider all the scenarios a tester might when they write code. As a result, they may not write clear object identifiers in the code which help testers write scripts.

    Overcoming these barriers comes back to collaboration. Developers and testers need to find ways to understand each other’s world view. Begin by obtaining a joint understanding of completion. If developers and testers can agree on the release criteria, meaning that testers are comfortable with the level of automation in place to support future releases, it avoids a huge backlog of tests to automate later. Backlogs lead to delays and frustrations all around. Using a common development language for coding and test automation is also helpful; otherwise, it’s more difficult for both sides to speak the same language when working together. In the same vein, testers should have access to all the tools that developers use: source code repositories, continuous integration, project management and so on.

    We live in a digital world and have endless instant communication options, but even on a software development team, there’s nothing that replaces the value of a face-to-face conversation. It’s going to take more than one tactic to make collaboration work in your organization. Keeping the lines of communication open should be a central goal to increase test automation coverage and streamline the DevOps release process.

    — Ryan Yackel