Cloud-based testing platform company LambdaTest has launched KaneAI, an end-to-end software AI test agent. With AI software in (some would say dire, or at least thorough) need of deep-level testing to eradicate hallucinations, to identify bias and to ensure that wider ratification systems are in place including retrieval augmented generation, there is much to test.
Overused by technology vendors spanning the networking, system observability and development lifecycle tools space, the term “end-to-end” in this context is the company’s pledge to offer tools that work at the authoring phase through to debugging and ongoing management, all through natural language so that teams create and execute automated tests.
What is End-To-End Testing?
A function much-beloved of software quality engineering teams (well, the effective ones at least), end-to-end testing quite specifically breaks down into five core stages in the LambdaTest world, if not others.
Phase #1 is planning, where tests are added to the test manager function.
Phase #2 is creation, KaneAI works to generate test scenarios in natural language.
Phase #3 is execution and scheduling so that tests can be run across various cloud instances i.e. a browser testing cloud, a visual regression testing cloud, a real device cloud (LambdaTest works to test websites and apps on 5000+ real Android and iOS devices) and a HyperExecute Cloud (the company’s test orchestration tool that automatically groups and distributes tests intelligently across different testing environments and takes into account past executions).
Phase #4 is debugging, where developers can debug tests using natural language commands.
Phase #5 is reporting, where team members can analyze test reports.
LambdaTest claims KaneAI to be the software testing industry’s first generative AI-powered test studio. Since the tests use natural language, it also provides an opportunity for different personnel (often less technical) to participate in the test-case creation process and then integrate tests into DevOps workflows.
Beefing Up Brittleness
“Automation testing has long been a critical part of software development, but it often comes with significant challenges. Test automation frequently lags behind development sprints, with brittle tests requiring constant maintenance. Even low-code/no-code solutions have a learning curve and often start to break down at scale, making it difficult for test automation setup to keep pace with evolving testing needs,” said Asad Khan, co-founder and CEO at LambdaTest.
Combined with built-in features including intelligent test planning, AI-powered test healing and advanced conditional assertions, KaneAI supports complex workflows and ensures comprehensive test coverage.
Where the LambdaTest team says that low-code/no-code solutions can hit scalability limits, KaneAI provides two-way test editing features, that enable users to either author in code or natural language. In addition, it allows multi-language code export across all major frameworks along with unique Instruction-to-Code and Code-to-Instruction translation features. This enables effortless maintenance of tests for scale and ensures that even the most intricate testing needs are met.
Workflow Wonderhorse?
CEO Khan suggests that his firm’s toolset is a happy revolution for the debugging process with its AI-powered test observability and real-time root cause analysis capabilities. This helps large-scale distributed engineering teams minimize application downtime and accelerate release cycles, without compromising on software product quality.
Whether KaneAI is as “revolutionary” as LambdaTest claims or not, it appears to be a precision-engineered product with an expansive set of functions, one might almost say end-to-end functionality. KaneAI integrates seamlessly with existing workflows, supporting popular tools such as Jira, Slack, GitHub Actions and Microsoft Teams.
Stephen and Alan discuss the biggest testing challenges in 2022, along with the tools and methods needed to solve them. The video and a transcript of the conversation are below.
Recording: This is Digital Anarchist.
Alan Shimel: Hey everyone. Welcome to another Techstrong TV interview here. My guest for this segment is Stephen Feloney of BlazeMeter. I think it’s BlazeMeter by Perforce. Is that correct Steve?
Stephen Feloney: That’s correct.
Alan Shimel: Yep. And for those who aren’t familiar of course Blaze Meter was originally founded by a friend of mine Alon Girmonsky. Great story around one of the pioneers in open source testing and continuous testing. Blaze Meter was acquired I guess it had to be six years ago, maybe more, by at the time called CA or CA Technologies, Computer Associates. Then it always kind of operated kind of semi autonomously. And then I guess it was last year they found a great home with the folks at Perforce who have also been around a while. And Perforce is of course a leader and really expanded their capabilities from code repository to testing and so many other things. So it was a good match, Blaze Meter and Perforce. We’re happy to see it. Steven so I gave the Blaze Meter story but I can’t give the Stephen Feloney story. I need you to do it.
Stephen Feloney: Oh well we’ll need hours to give my story.
Alan Shimel: Give us the short version.
Stephen Feloney: So I’ve been in testing more or less for 20 some odd years. I came to CA must have been seven or eight years ago. And I was part of the team responsible for acquiring Blaze Meter into CA. And I always had plans for Blaze Meter. And as you mentioned it sort of ran autonomous inside of CA. And then CA was acquired by Broadcom and then once we were in Broadcom I was able to expand Blaze Meter to go beyond performance and we added a multitude of features, virtual services, API testing, API monitoring, grew all that. And now as you said we found a very good home with Perforce and that acquisition happened right at the end of October of last year.
But I’ve been doing product management work for, I don’t know, about 15 – 16 years almost exclusively in the testing space, a little bit in the mobile space and a little bit in the monitoring space. And what’s interesting is moving around going from testing to mobile development and mobile testing and then into monitoring I now have all of that with Perfecto and Blaze Meter coming together at Perforce. So I have the monitoring. I have the mobile. I have – it’s all coming together. So everything I’ve learned over the years I can make it all come together and work in one spot.
Alan Shimel: So give us your best Dr. Evil and tell us isn’t it great when something comes together like that?
Stephen Feloney: Yeah. It’s fantastic and I hope to make one billion dollars.
Alan Shimel: Absolutely. Good stuff Steve. So look. Let’s first of all we’ve had the pleasure of talking to several of the Blaze Meter folks since the acquisition by Perforce. And it really does seem to be a good match. But what I wanted to kind of focus in on today is maybe the state of continuous testing if you will and then a little bit of what specifically at Blaze Meter what’s new. What’s coming down the pike that we can be on the lookout for here.
Stephen Feloney: Sure. Sure. So Blaze Meter historically has been a performance testing solution built upon open source as you mentioned and we were, we play two different roles. We work with centers of excellence who are doing large, massive load tests. And then we work in a shift left world with agile teams. And we were noticing struggles when it came to testing with the agile teams. So when you’re looking at continuous testing, continuous testing is well, a continuum. So it starts off with the development and the agile groups, works along sometimes into a full end to end testing and then continues into production.
And so if we talk about that continuum we look down at the agile teams and we noticed that they were struggling. They weren’t testing as much as they should. And we interviewed dozens of different companies to try to understand what were their challenges with this. And there were a multitude of challenges. One is they didn’t like using multiple different tools to try to get all the testing done, whether it’s a functional test, a performance test, an API test, unit test, all different types of testing, all of these different tools. Some have to be downloaded onto our devices. Some are in the Cloud. They all have different UIs, reporting. That’s a challenge. And then another challenge was we don’t have time to get data. I mean think about it. When you’re testing you need data.
Alan Shimel: Yeah.
Stephen Feloney: But it wasn’t just data to run the test because why can’t you do that. Then I need data for the back end system and then I need data if I don’t have my full environments and I’m using mock services or virtual services I now need data for that. How do I get all that data in and make sure it’s all in sync? Because when I’m developing code I have a very limited time to develop that code and get it out the door and if I have to spend time trying to figure out get all this data and sync all this data together, that’s a challenge.
And then well with that challenge as I mentioned I don’t have my full environment. So now I need virtual services. Where do I get those? Who is developing those? And then I have to figure out what data did they put in? So the mock services, the virtual services as well as the data, two big problems that are hindering people from getting their testing done. And then we have other issues around how do I generation my test. I just want to develop. I don’t want to spend my time creating all these tests. So those are challenges there and then on the flip side if I’m trying to do continuous testing and I’m trying to continue the testing into production what tests do I set up.
How do I monitor? How do I test in production when I don’t know what I tested, when there’s not a full collaboration across the entire value stream? How do I know what to test in production? Who is creating those tests? Do you have your ops team and are they monitoring or testing the same things that I wanted to test or that I did test in preproduction? So those are quite a few of the challenges that people run into when they’re trying to do continuous testing and continuous testing at scale.
Alan Shimel: Absolutely and emphasis on that scale because really that’s where you start seeing the cracks. That’s where you start seeing – when you’re not at scale sometimes you could muscle it. Right? Or fake it even. But when you start hitting scale man, it’s like whatever your weakest points are they’re going to just come right to the forefront.
Stephen Feloney: That’s correct.
Alan Shimel: You’re doing to say oh my goodness. This doesn’t scale. Right?
Stephen Feloney: No. That’s right because you could have like one group, one team, one service that’s able to do something that’s special, unique and they could get that out the door in their own special way. But when you’re trying to scale that either that service or application has grown or you’re trying to scale it in an enterprise as you just said all the cracks show. The whole thing falls apart.
Alan Shimel: Agreed. All right. Let’s talk about what’s new at Blaze Meter then. What are we doing to help?
Stephen Feloney: Well surprisingly a lot of the challenges I just mentioned we’re able to solve just coincidental.
Alan Shimel: I’m shocked.
Stephen Feloney: I’m shocked as well. I mean I didn’t know this was coming. Yeah. So one of the things we actually just released on February 3rd was our Blaze Meter test data piece of Blaze Meter. And this combined the ability to whether it’s for performance testing, functional testing, API testing we can generate the data that you need. And that data that gets generated is not just to generate the data to drive a test. We’ll also generate the data if you’re using mock services or virtual services, generate the data for that as well and generate data for system undertests.
And the idea is all that stays in sync. And we’ll generate the data at the time you want to run the test. So now you don’t have to worry about managing, maintaining the data sets. You don’t have to worry about that. We will generate the data on the fly and to run the tests and to ensure that all that data is in sync. That takes away a lot of the false positives that you’ve seen in the past where things are failing. Oh but it turns out it was the data in the virtual service or the system under test wasn’t set up the right way. And so all of that goes away and you don’t have to put any thought into it.
Alan Shimel: Crazy. It’s crazy that no one has thought of this before or kind of implemented this before when it seems rather elementary at some level. Like of course if you want to do this. But I had a conversation with the last guest here on Tech Strong TV and we spoke about the issue is maturation issue. Right? Until your devops processes, until your CICD processes are at such a level of maturity. Right, you don’t even run into this problem because you have problems that pop up before it that kind of take your focus away.
Stephen Feloney: Correct. Yeah. So in the past what’s happened is when you were handing it off to a testing team the testing team was used to the problem and they had time to get the data. They had time to set up the data. They had a gold copy of the data sitting around that they would use. They were familiar with it. But as you’re saying the CICD process, the shift left process happened the data problem, the virtualization problem is completely exacerbated. ‘Cause they don’t have the time. They don’t have the ability. They don’t have the skill set. They don’t have the desire to do all that. They want to get their code that they developed and they want to get that out the door as quickly as possible. And testing is sometimes seen as a hindrance. It’s a necessary evil – going back to Dr. Evil. A necessary evil if you will.
And so anything that we can do to accelerate that and make it much easier for them that is the challenge. So yeah. You could say why wouldn’t someone have done this whole data thing. There are synthetic data generators. Right? There’s open source synthetic data generators. That’s fine. But that just generates the data in one spot. You still have to figure out how to take that, pull that out, put that here, put that there, put that there. And we’re taking care of all of that for you.
Alan Shimel: Got it. Very cool. Hey. For people wanting to get more information on that Stephen where can they go?
Stephen Feloney: They can go to blazemeter.com.
Alan Shimel: Just straight up right on the front page?
Stephen Feloney: Yeah. Just right on the front page. It’s our newest part of our release so you’ll see it there. You’ll see blogs. You’ll see help. We have multiple discussions on it. Yep.
Alan Shimel: Very cool. Hey. We’re coming up on time here. Anything else coming down the pike or you wanted to let our audience know about?
Stephen Feloney: Yeah. So there’s actually quite a few things that are coming down the pike. But one of the more interesting things that we are working on is one of the challenges that I had mentioned is generating tests. So they don’t want to spend the time to even generate the tests. And there’s two things that come with that. One is just generating your API tests. So we’re going to have coming soon an API test generator to help with that and that is combined with our well synthetic data gen. So those two are combined together to help generate that. The next thing that comes which is a natural evolution is you don’t just want positive tests. You need negative tests as well.
And so that data gen is going to auto generate negative tests for you whether it’s negative data driving a test, putting negative data into your system test or having negative things happen with those mock services or those virtual services. So you want to delay a virtual service. You want to knock a virtual service so it doesn’t respond or just provide different types of data sets to see how well and how robust your system is. And once you do that the next step is all right. So I have negative testing. Then you combine the positive and negative together and now you have chaos testing. And so those are coming down as well.
Alan Shimel: Cool. Love it. Any – I don’t want to hold your feet to the fire, make you say anything you’re not supposed to say but what’s timeframes on those if you can talk about them?
Stephen Feloney: Well I’ll say this. By the end of the year you’ll see this. I don’t want to give solid timeframes but this year those will be coming out the door.
Alan Shimel: Ok. I think that’s fair Stephen. Hey. I want to thank you for coming on and updating, getting us up to speed here.
Stephen Feloney: Yeah. Thank you very much.
Alan Shimel: Continued success. We’re hoping maybe at some point we’ll see Blaze Meter at a real live conference.
Stephen Feloney: Oh I’m hoping to get to real live conference. As much as I enjoy maybe not traveling as much. But the interpersonal interaction.
Alan Shimel: Yeah, no. We all miss it.
Stephen Feloney: Yeah. I mean it’s a needed. It’s a must.
Alan Shimel: Absolutely man. Hey. Steve Feloney. BlazeMeter powered by Perforce here on Techstrong TV. We’re going to take a break and we’ll be right back with another guest.
As the practice of DevOps evolves, so do the supporting tasks; hopefully in such a way that they introduce increased efficiency and automation to accelerate development and deployment pipelines. However, one specific process still remains a speed bump on the road to DevOps acceleration: the process—or, more specifically, the chore—of testing.
Naturally, testing is an undeniably important component of quality assurance. However, introducing efficiency into a process that dictates quality assurance can accelerate development and improve outcomes, especially when it comes to application testing.
“Application testing should never be done in a vacuum, or, worse, a silo,” said Sara Faatz director, developer relations at Progress Software. “The DevOps process can benefit from accelerating the testing process, introducing automation and embracing teamwork.”
Faatz makes some valid points; historically, application testing was treated as either a standalone or separate process, and often was performed by non-programmers following some type of script. “We discovered early on that creating synergy between testers and developers was an important concept for accelerating development, while fostering teamwork,” said Ramiro Millan, director, product development, Test Studio. “Addressing the trifecta of acceleration, automation and teamwork challenges actually requires adopting a different mindset when it comes to testing, especially when it comes to automation and communication.”
However, solving those problems requires adopting different procedures, as well as deploying tools to streamline processes and introduce automation. That said, there are ancillary challenges that also have to be addressed. For example, there is often a disconnect between the cultures of QA teams and development teams. Here, QA teams need to overhaul their processes and introduce more agile practices. In other words, the time is now to identify new practices and methods that allow QA teams to be creative and innovative while helping to test the software created by DevOps.
“Automation is becoming hyper-critical for testing. However, it is not just automation alone, but also the ability to run tests in the background and not hijack control of the console or the machine,” Millan said. “Automation should also bring with it the ability to concurrently run multiple tests in the background, which, in turn, accelerates the testing processes.”
That makes a great deal of sense. In many DevOps shops, QA has become a speed bump in the CI/CD pipeline, meaning that developers often have to wait for testing processes to complete before they can deploy anything. What’s more, developers are often disconnected from the status of testing.
“Communication between developers and testers is taking on renewed importance. Using tools that can hand off statuses between teams, while also enabling different members to participate in the processes, proves to accelerate execution times,” Faatz said. “The ultimate goal here is to bring forth faster delivery, less maintenance and more stability. I think stability is a big part of the equation. If we can automate those day-to-day, repetitive tasks, we garner that additional stability.”
Although testing is a broad subject, many of the lessons offered can have a beneficial impact on the overall process. Currently, many vendors are focused on subsets of the testing process, such as headless browser testing, PDF validation, UI/UX testing, backend integration and so on. This can be an indicator of just how complex the QA process can become if unified tools and automation are not introduced into the process.
“The trick, here, is to pick a tool that does not replace peoples’ processes, but transforms manual QA processes into something that can leverage automation,” Millan said. “Automation brings about consistency, and consistency brings forth improved results.”
The lessons learned indicate that testing must evolve to keep up with the DevOps model, and also suggest that DevOps is pushing QA teams to adopt new methods of testing and interaction. The only remaining question is how quickly QA can better integrate into DevOps, and eliminate the deployment speed bumps.
Shift-left testing is the key to deliver quality software at speed. Developers shift testing left to shorten long test cycles, prevent defects and mitigate risk earlier in the development cycle, rather than in later stages, to avoid rework and delays.
In this interview, IBM Z DevOps Testing Lead Nathan Cassata talks about shift-left testing, the benefits of shifting mainframe application testing to the left and how IBM Z DevOps testing solutions help accelerate enterprise cloud-native development.
Nathan is also one of the distinguished speakers presenting at Open Mainframe Virtual Summit. During his session, he will explain how to shift left testing for your traditional z/OS applications.
To learn more about how testing is integrated into a developer pipeline and how to create real unit tests, attend Nathan’s session at Open Mainframe Virtual Summit on Feb. 11. To check out the full agenda or to register for free, visit the Open Mainframe website.
DevOps testing, as an Agile methodology, is key to delivering testing and integration on a continuous basis, including continuous feedback, to enable better collaboration and quality.
For too long, testing has progressed by small incremental steps, creating a quality assurance (QA) winter. We think it’s time for a revolution, so we have written and published our manifesto for change. It’s a future where an intelligent test agent handles all testing needs so organizations’ resources are freed up, thus saving them time and driving revenue. This future means intelligent testing is a function everyone on a given team can contribute to—and, above all, allows for quality to be embedded in everything an organization produces.
How do we get there? Apply these perspectives and view testing strategically:
Testing Shouldn’t Be an Afterthought
QA should be a C-suite responsibility. No one laughs at the idea of a chief compliance officer, so maybe you should consider appointing a chief quality officer. Successful companies are the ones that adopt the mantra of test early, test often and test in depth. This means testing becomes a key strategic requirement in your organization. If testing needs more resources, make them available. If there is tension between your test and dev teams, try merging them. If you want to release new features frequently, look into continuous testing.
Testing Must Be Cloud-First
Up until now, testing has been done in-house using old servers, begged resources and outdated infrastructure. We believe testing should be run in the cloud, allowing you to leverage an almost infinite resource pool. Testing done like this will be faster, more efficient and cheaper.
Testing should not be the poor man of the software development process. Companies that engage with their testers produce the best and most reliable software. Those that treat testing as an afterthought risk bugs, production delays and, ultimately, their reputations.
Agile DevOps Requires Autonomous Testing
DevOps has become critical to the success of many companies. Without DevOps, many of the services end users take for granted would simply fall over. Social media sites and apps such as Facebook and LinkedIn would fail if they constantly went offline. If content delivery services such as SoundCloud, Netflix and Hulu kept dying mid-track, you’d soon cancel your subscription. People also expect messaging services to be 100% reliable.
DevOps straddles the border between developing software and running it in production. It is what you get when you apply the Agile approach to software development to deploying services in production.
Autonomous testing is the outcome of intelligent test automation. With autonomous testing, you now have an intelligent test agent (ITA) augmenting the work of your test and DevOps teams. This ITA is like the perfect regression tester—focused, tireless and driven, but still intelligent. Using this ITA means changes can be tested and deployed completely automatically. Without this, the job of an agile DevOps engineer would be infinitely more difficult.
Intelligent Automation Just Works
Classic test automation is dumb, and like all dumb systems, its growth and utility are limited. In this era of AI, test automation should be intelligent, too. There are few areas of software development where it is easier to apply AI—and even fewer where the rewards are so obvious and immediate.
Put simply, well-designed, intelligent automation just works. Intelligent automation means tests become less brittle, so no more time wasted on unnecessary maintenance. Productivity increases because test failures are automatically diagnosed and classified. New tests become easier to create and test flows are easy to reuse.
Mindless testing, whether it is manually or automatically executed, can cover only a very narrow scope of possible bugs: the ones you’ve already thought of. Exploratory testing however, endeavors to think outside that scope. Thoughtful testers look at a problem from all angles, weighing factors such as feature usage, psychology, user stories and more. Their goal is to chart the uncharted territory of an application.
The faster our development cycles become, the more thoughtful our testing should be. It’s only through techniques such as exploratory testing that we’ll be able to get there.
Continuous testing is the process of executing automated tests to obtain rapid feedback on the business risks associated with a software release. Where does that leave exploratory testing? It’s not automated, but it’s certainly critical for determining whether a release candidate has an acceptable level of risk.
Test automation is perfect for repeatedly checking whether incremental application changes break your existing functionality. However, where test automation falls short is in helping you determine whether new functionality truly meets expectations. Does it address the business needs behind the user story? Does it do so in a way that’s easy to use, resource-efficient, reliable and consistent with the rest of your application?
Exploratory testing promotes the creative, critical testing required to answer these questions. Obviously, it doesn’t make sense to repeat the same exploratory tests continuously across and beyond a sprint, but exploratory testing can be a continuous part of each delivery cycle.
Exploratory Testing in Action
Here are a few ways teams embed exploratory testing throughout their process.
Perform Ad Hoc Exploratory Testing as Each User Story is Implemented
This is the exploratory testing equivalent of peer code review. When a developer completes a user story, they sit down with a tester. The tester starts testing while providing a running commentary on what they are doing and why. Next, the developer takes control, explaining how they would test the software given their knowledge of the implementation details and challenges. The developer gains a user- and business-focused perspective of the functionality, and the tester learns about the inherent technical risks.
Another tactic is to have the developer and a tester separately test the same feature simultaneously, then discuss their findings at the end of the session. Often, this turns testing into a competition, where each participant tries to uncover the most or “best” issues in the allotted time.
Align Exploratory Testing Sessions with Full Regression Testing
It’s simply not possible to perform exploratory testing or full regression testing on every code commit. That’s what smoke testing is for. Instead, many teams run full regression testing and session-based exploratory testing in parallel a few times per week, whenever they’ve implemented new functionality that a user could feasibly exercise.
For optimal results, these sessions should be lightly planned and tightly timeboxed, include diverse perspectives and really take the six thinking hats theory seriously.
Host Blitz Exploratory Sessions for Critical Functionality
The best way to uncover user experience issues before users do is to get a broad array of feedback prior to release. One way is to host “blitz” exploratory testing sessions. When you’re wrapping up work on critical new functionality, invite people from a variety of backgrounds and teams to participate in a short timeboxed session. Incentives can help drive participation, maximize results and make testing fun.
Using test automation to continuously check the integrity of existing functionality is certainly critical. However, if you’re not also making exploratory testing a continuous part of your process, how will you know if the new functionality meets expectations?
The goal of continuous testing is to understand whether a release candidate has an acceptable level of risk. Exploratory testing is perfectly suited for helping you answer that critical question.
Safety, security and controls are the foundations upon which 135-year-old ExxonMobil is built. As such, quality has become a key component in delivering software. The company is currently undergoing a shift from traditional project management organizations and support organizations to product teams, which has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.
Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”
To do this, ExxonMobil started with the basic idea—quality control manages risk—and then explored what is tracked today and whether quality is being measured. Another key activity was to remove the roadblock to continuous testing.
Ann Lewis from ExxonMobil discusses in this video how the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset, as well as the company’s view that continuous testing and deployment can only be accomplished through model-based testing and that managing quality reduces overall cost of ownership.
The full transcript of her discussion is below:
“At ExxonMobil, we have presence in six continents, so that means we get to deal with a lot of people. There’s over 72,000 employees plus another 20 percent contract staff, and that’s a problem. Guess what they want to do? Manual testing. Yeah, because they’re not gonna give it up. We have been on an adventure this last year that I’m gonna talk about and explain to you how we help train these 72,000 plus people that there’s a better way to do it.
“In addition to the numbers of people that we have, the upstream, I always … Well, someone referred to it about sticking a straw in the ground. Yeah, it was actually somebody from Tricentis. Yeah. I was just like, ‘There’s a little bit more to it than that.’ No, we do produce all of the product from under the ground. We get it out, and then we send it downstream to our refineries so they can do something with it like sell it. We also have a chemical company, that … I think right now are second or third to DuPont. We have a lot of product and a lot of things to do with it. That’s a problem, too. Because when it comes to the applications that test, that need to be tested, we have something like 5,000 applications that run our business all technologies. We’ll talk a little bit about that as we go through, too.
“The title is The Middle Child Rises. Why did I choose this? Because in the world of Dev and Ops, test is in the middle.
“We have a few other friends in the middle, and they are a little bit like a middle child, so we’ll talk about that. For today, your takeaways are that our organizational design has shifted. We’re going to explain why and how that happened. We will explain that Tosca is strategic for ExxonMobil and we will explain why that is.
“Tosca is an enabler to speedy delivery.
“You’ve heard about that this morning from many of our Tricentis friends, and it is true and I’m here to share that story. We now have within ExxonMobil, a quality philosophy, yeah, and it’s a lot more than just testing. Yes indeed, it is. We’re going to take a look at what we did and have a lot of fun stories in between.
“Being the middle child, you really didn’t get as much attention. How many are middle children, by the way? A few, okay. Do you have babies in here? Yeah, yeah, you kind of get away with everything, don’t you? The oldest, I have a favorite, I know. Oh, there’s another one out there of my fave, okay, cool.
“It’s very interesting in family relationships how these children get along. The oldest kind of sets the stage for everybody. That middle one comes along and it’s like, ‘Oh, it’s the second kid.’ The third one comes along and it’s like, ‘Oh, it’s the baby, it’s the baby,’ and the middle child just kind of sneaks through, okay? They don’t get quite as much attention. We’re going to talk about the middle child as they relate to testing.
“Typically, it’s the middle child that ended up with rules. That oldest child was all about, if you just tell them once, they usually would stop doing it. That’s the case I had. Then the second one was born, that turned out to be the baby, but I swear he’s got components of being the baby and the middle child. It was like, ‘Adam, we have rules in this household. Yes, we have rules.’ He was like, ‘I just don’t get the same respect as that older sister, Jessica.’
“Now within the corporate world, these middle children, you can think of them as being security and controls, data migration, data conversion. It’s like, ‘Oh, yeah, we forgot about you.’ Last but not least, testing. How many times has it been when suddenly that project shows up, ‘Oh, yeah, I think I might need to engage your team to help me organize my testing.’ Yes, yes, those of us that live in the testing world.
“The big deal right now, though, is that these middle children, our testing organizations, our quality assurance organizations, are becoming the key enabler, like we heard this morning, 50 percent of our problem is with testing. Oh, I could do that but I can’t get testers. I could do that but, ugh, we’ve got to do the testing. It takes so long.
“According to the World Quality Report, the reliance on manual testing is the top technical challenge in app development.
“How many of you knew about the World Quality Report? Yes? Yes? Good, good, good. We’re all the same kind of people, yeah. I like that.
“Businesses see test automation as a way to address one of their top challenges to delivering better software faster, by adopting Agile, DevOps and continuous delivery practices to shorten delivery cycles.” – Forrester
“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks.” – Gartner
“The need to support faster time to market with higher quality is driving the demand for effective functional test automation tools. This is why we’re here. Tricentis Tosca, it is a framework. Yay! It’s not scripting. Boo!
“Tricentis Tosca will address that top technical challenge of delivering software faster.
“The need to support faster time to market with higher quality. It’s higher quality in the form of risk coverage. That is what we do with Tricentis Tosca. That is what we do with Tricentis Tosca.
From a framework perspective, the last year and a half we have been on a mission to explain the difference on automation tools. We have gone from record and replay, to test automation from a framework perspective using scripting, such as with the Hewlett-Packard products and Selenium. It is programming, it takes developer time to do automation when you’re talking about some of these historical tools. We have now moved into model-based, and I hope everyone in here has been to at least some of the Tricentis’ demos on what model-based is. I’m not going to go into detail here, but I will tell you that it has taken our automation from a 40-hour event down to a 8 to 16-hour event. No joke.
“I have been able to take Tricentis Tosca and put it into non-developer, normal people’s hands.
“I actually sat down with one of the Tricentis’ architects and did automation myself. I said, ‘Oh my gosh, I haven’t touched a keyboard to do anything technical in so long,’ and I actually did it. That is key, that is significant. That has been the story that has been going on with ExxonMobil for the past year and a half.
“It has become strategic for us. It’s automated screen scanning, it’s lower maintenance, it’s targeted testing. I’m doing it in half of the time, if not less. It’s faster, it’s easier to support, it’s talking about prevention versus detection. We have created a road map.
“What you’re seeing in our colorful TFS diagrams on the top, yes, we are doing our monitoring in TFS. You’re seeing a migration and better put a re-engineering of our existing scripts into Tosca. Re-engineering because for the last two years that I’ve been in this job, people kept telling me, ‘Ann, did you know that those existing Hewlett-Packard scripts really don’t test anything?’ ‘Well, gee, I don’t think I knew that.’ ‘Well, gee willikers, let’s see if it’s really true.’ Lord, have mercy. Twenty percent of those Hewlett-Packard scripts that were driven by the business to automate didn’t test anything. Well okay, they tested something but it wasn’t worthwhile.
“All of these little blue dots on your right beyond the solid blue vertical line are things that don’t really matter because we didn’t do a risk analysis. When I talk about re-engineering, we’re not just taking what we had and push it into Tosca, which you can do, by the way, there’s a tool for that. It’s up to you to use it, I’m not going to say anything. All I’m telling you is all our experience has been it made no sense.
“What we have found is when we sit down with our subject matter experts and we had this really cool dialogue like, ‘What do you do all day long?’ ‘Oh, I use VA-01 and NM-16 all day long.’ I said, ‘Cool. And so what happens when one of them doesn’t work? Dead in the water. You’re in customer service, aren’t you?’ ‘Yeah.’ ‘How many sales? Millions of dollars of sales do you do every day?’ ‘Well … ‘ This is the user, I mean they are totally into explaining to us what they do. Through this dialogue, this interview, we find out that they’re going to cost us $1 million a day, an hour, I don’t know what the big number is but it’s a big number, when it comes to customer-facing transactions. Therefore, those transactions are the ones that we want to automate.
“Now if I find out, no offense to anybody in controllers, that at the end of the month they have a report that they need to generate, ‘I’ve got some time to fix that one, okay?’ ‘How often do you do this? Monthly? Okay, that’s a one. What happens if it doesn’t work?’ ‘Well, I just did last month’s report and … ‘ ‘That’s a one.’ Low priority, move on.
“We have been re-engineering all of our existing tests.
“This word crypt is an ugly word. I actually I know the person who named our existing automations. Critical Integrated Process Testing. We don’t say that word anymore, by the way.
“The middle child has risen. This is my team. When I joined the group in February of 2015, we had about 25 people. About 25 people. Within ExxonMobil, I think there might have been seven of us. Then, we had some offshore group that did all of those monthly executions of those Hewlett-Packard scripts.
“Today, I have Vulcar over automation, he’s based in Hamburg. I have Cathy in Houston for functional testing and Santa Juan for functional testing, basically split between the two regions. They’re what we call test leads. They do a lot of the coordination of our testing activities. I have Aaron in Houston for performance testing. I have Jay, he is Doctor Jay, he is the man that knows everything about testing and everybody loves Doctor Jay. We have Suzanne right there, one of my oldest children. I didn’t say that, did I? I meant oldest child. Anyway, the other thing that was really funny about these pictures, they kept saying, ‘Oh, I want mine to be skinny.’ I want mine to be really upfront. We had a whole dialog on how the pictures turned out. It was pretty fun. Guess what? There’s close to 150 people now. That’s a signal that over this past year and a half, two years of getting this team put together, the emphasis that ExxonMobil has put on quality. One of the ways to get to quality was through testing.
“We are everywhere. I’m going to go into a little bit more detail now about organization because it’s good to understand where you are located. I was talking about being on the six continents. We have now placed people in region, so that when a person has a question on, ‘I want to automate, should I automate?’ We’ve got people in those locations to help with that. You will see we are using architects.
“Within the Tricentis nomenclature, you have architects, automation specialists and automation engineers. It is a tiering of skillset.
“We have found that the architects become those go-to people, when it comes to, ‘Should I be automating this or should I not and how do I go about it?’ Yes, we are using some offshore resources in India. Yes, we have people in Auckland. Yes, Bangkok, and that’s actually one of our ExxonMobil business support centers in Bangkok. Then, you can see Houston, Vienna, New York and Curitiba, Brazil.
“Organizationally, what we have found is that when it comes to implementing automation and doing it in a … I’ll call it conscientious and disciplined way, you’ve got to have a structure. Don’t think that you can just, I’ll say, run into doing automation. There needs to be a structure upfront to help you manage this work, because once one customer comes, you’re going to get five more. The enthusiasm level that we have experienced is unbelievable. We’re almost on tilt. But through this concerted effort of having an organization to help support it through a governance and then program management. Again, you can see this partnership that we have with Tricentis. Highly recommended when it’s time to get started.
“Successful team composition. It’s typically one architect to one engineer to 10 specialists and then you ramp up or ramp down, based upon if you’re working on systems of engagement. The shiny coin fun stuff, mobile, gooey, or your systems of record. They’re a little bit of a worn coin, all right? I’m sure some of you have systems that are quite old. We have ones that are really old so our financial system is still in IMS in PO1. Yes, rock on! That’s what I grew up with. Yeah, give me some JCL, too.
“See, geek, we’re all geeks, I know. It’s like once you join the world of IT, you’re just like into it. Nobody really understands what you talk about, unless you’re with people that are like you. Uh-huh (affirmative), all in the family, totally all in the family.
“Okay. We also have an immersion program. We have, again, the process of getting somebody into you’re approved, we think you can handle the work, all right? Going through automation specialists and all the way up to graduation. Again, working with Tricentis, we have come up with again, a disciplined process. You need this when you get started because they will come in out of the woodwork, I want to automate. I talked to you two years ago about automating those processes at the refinery and you said that no machine could do what you could do. I don’t say that, but that’s what I’m thinking when they come to me two years later with, ‘I think I want to do automation.’ I’m like, ‘Cool, let’s get into it.’
“We have a quality manifesto. Yes, we do. We will create forward-looking, value for our customers and for our organizations through quality approaches. Quality engineering is a team responsibility. I’ve got a great team, but we’re just here. We’re the messengers, we’re the trainers, we’re the coaches. Everybody that we work to, we talk about quality.
“How did you get to work today? Did you think about a safe way to get to work today with all of the rain? Yes. That’s a quality decision right there. Little things that we teach when we’re starting to push the quality manifesto out. You have to understand business function risk. Again, what do you do all day? What happens if it doesn’t work? Oh, really bad things happen, or maybe not so bad and a month in close, we’ve got a whole month to worry about it. Yeah, we can always close the books this month of last months’ numbers. I actually have a degree in accounting, I can talk about them.
“Including quality checks throughout the delivery life cycle. Quality gates. That’s another item that we have built into our system delivery life cycle. Starting to talk about at the moment requirements are written down, let’s start talking about what you’re going to test, based upon those requirements. We have built these quality gates, these questions throughout system delivery life cycle.
“What has happened? Our organizational design has shifted.
“Tosca is strategic for ExxonMobil. Tosca is an enabler to speedy delivery.
“Test automation brings value. We’re actually connecting the automation into total cost of ownership, as a way to reduce application total cost of ownership. We have a quality philosophy now and we’re just pretty darn excited about what we did. It took the rise of the middle child in order to make it happen. Thank you so much.”
Most of us use Postman during API development. The company claims 3 million users, and as one of them, that number doesn’t raise my eyebrows any. But if you are like me, you’ve figured out collections, how to back up your data and perhaps looked at tests, but not done much more. As a tool for API development, Postman is powerful and useful, but that means many of us don’t look very far beyond how quick and easy it is to set up an out-of-browser test environment to evaluate request/response.
We use collections to separate projects, but really only as a way to group calls together and keep a single copy of each call close to hand while the history list scrolls through multiple copies.
But for DevOps, there’s more. And they’ve just added to it again.
Packing in the Features
Groups can be used for grouping calls with tests, for example, and a command line tool can perform automated testing. If, like us, you use Postman to test as the API is developed, and use groups to keep those calls pulled together in one place, then testing from the source that already contains your entire API makes sense.
And as of this week, Postman also includes the server side. Postman Mock service allows you to mock up the API calls and specify a valid return value. That means client-side developers can use Postman Mock for testing their app/page, while server-side developers are using the original Post
man to develop the APIs that Mock emulates and testing is running tests from the real-world APIs that already are in Postman for development purposes.
Dev, DevOps, Testing and Ops: Something for Everyone
So a quick diagram shows that the entire development process has a bit of Postman in it if that’s your tool of choice:
While all assist with agile development, for DevOps, the pieces that are the most beneficial are Test combined with Groups and the new Mock service. As long as the Mock service is configured for a group of calls, test can call them and the logged output can reflect that the test succeeded but ran through the mock service. Once the service is completed and the group is directed to point at the server, testing will run tests against the real API. Using Newman, the command line tool, these tests then can be integrated with your build toolset.
The Mock service can take imports from SOAPui, if you are converting from SOAP calls or use SOAPui for REST and are moving to Postman. I’m told the Postman toolset also supports SOAP calls, but truth in writing, I’ve never even looked to see how that would be done. I’ve used both tools, but never on the same project.
There’s a Cost, but There’s Also Value
This next bit sounds a bit salesy, and I try not to do that. But for the price, there aren’t a lot of value-add tools out there that deliver this kind of value. Lori and I are talking about going Pro, as we both use Postman in projects. And considering we’re regular users, I figured if we didn’t know about most of these features, many of you wouldn’t, either.
Mock is part of Postman’s Pro package, along with sharable team collections, and API documentation tools. That means it will cost you a little bit of money, but if you’re on a team and have been cobbling together a way to share collections, team collections and associated collaboration tools are worth the price—particularly since built-in auditing will show all of the changes made to Postman collection items. That means fixing a change-based error on the Postman side is easier.
As of this writing, the cost for Postman Pro was $8 per user per month. My guess is sharing groups (or worse, each team member re-creating/maintaining them) costs more than that per month in many cases. Consider that documentation can be generated and shared to a private URL and the fact that Mock service enables parallel development, and $8 sure seems like a steal. Finally, Pro offers API monitoring, so performance and uptime can be tracked to better adjust to the realities of production. There’s more (such as integrations with some great Dev/Ops/DevOps tools), but I’ll stop.
Summary
Design, Development (both client and server, in parallel), Test, Monitoring … as long as you can get it deployed, Postman can help with the rest. Postman Pro has a reasonable price for the functionality it offers, and the introduction of the mock service makes agile development, tight timelines and consistent “start to deployment” testing easier. I’d venture to say it is worth taking a look at. And if you’re developing APIs without it, pop out and get the free version, you’ll quickly find why so many of us use it.
You can find out more about the various editions of postman on Postman’s Product page.
When I got started as a professional software developer, I thought awesomeness in development was measured by the lines of code (LOC) per day and how quickly code got to QA for testing. Coding a few thousand lines overnight gave one instant bragging rights, regardless of how well the code worked (or didn’t) in the morning. The more senior guys always acted like they knew better, suggesting that it’s not about the quantity but the quality. But I was too young and arrogant to listen. And, of course, I had to learn it the hard way. My seemingly amazing overnight code would show cracks once under the microscope of the QA team.
True today as it was then, most mistakes are tiny and easy to fix, but even small problems require a tedious set of steps from when a QA engineer finds the mistakes to when the developer fixes them and sends them back to QA for verification. Some bugs are the size of the Mariana trench and require weeks of work and thousands of additional LOC. And, even new code, once resubjected to QA, can generate bugs, therefore repeating the cycle without an end in sight. Of course, by this time projects are late and developers are tired, frustrated and burnt out.
After this happened to me too many times, I realized that maybe I was wrong. Maybe the experienced guys were on to something with this quality thing. They never seemed too tired, and their projects were in much better shape. I decided that for my next feature, delivering quality instead of LOC would be my priority.
I’m sure you think you know what’s coming next. I’m going tell you that I tested my stuff from that day on and it worked perfectly—no more boring bug-fixing. You think I’m going to tell you, you should do it too, right? Well, no. I actually spent a lot of time in my second attempt instrumenting my code better so that I could debug it faster as the need arose. I also spent a lot of time writing my own test tools because I didn’t have one. But I kept moving forward and learned the hard way. Here is what I took away from those moments:
Develop More, Bug-Fix Less
There are two common reasons for developers’ involvement in a project. Developing a new feature or enhancement is about creativity, as it gives one the opportunity to design and implement new things. And then there’s the challenge of bug-fixing, which is all about maintaining existing functionality within older, already-made things. As a developer needs to do both, often at the same time, minimizing bug-fixing time is important to have more time in which to actually be creative and develop new functionality. And the best way to minimize bug-fixing is to test code more, as it’s developed, to avoid the death spiral I describe above. Good initial quality will mean QA will find fewer bugs and it is very likely that the criticality of the bugs found will be greatly reduced. As such, fixes no longer would be a struggle on short notice; shortcuts that generate more bugs don’t have to be used; and developers can focus on new features as opposed to fixing bugs. In all, the reasons are to develop more, and bug-fix less to be more productive.
Be More Reliable
Bugs also are nasty in that they tend to appear really, really late in the game. The Murphy’s Law on bugs should be something like, “The hardest bug is found one hour before the release is pushed to production.” This typically happens because everyone is busy peeling the onion, finding and fixing bug after bug until they get the “system” stable enough for QA to crush it with those large-scale system test cases that expose that one nasty bug. It is an artifact of waterfall development but also happens in agile development, as no agile team can afford running long duration tests in a sprint. So what happens then? Who ends up in crunch mode? The developer. To avoid this, developers need to find and fix the easier bugs sooner, as they write the code. That way, the onion is half peeled once it reaches the QA team. The nasty bug will be found much sooner—if the developer did not find it and fix it already. In short, developers can be more reliable by avoiding last minute bug-fixing. And we all know, reliability in execution is fundamental for fostering trust with management. A trusted developer is the one that gets to work on important, meaningful projects.
Be More Flexible
While in an ideal world the design of a feature is not changed frequently, in real life the exact opposite is true. At some point, even regularly, developers will have to modify the code as the scope of the features keeps changing. Massive code changes such as refactoring, if not done right, can send developers right back to a long, teeth-grinding, bug-fixing period. Avoiding it by testing early, as they change the code, offers a significant advantage—flexibility. Developers would be able to accommodate required changes more quickly.
Agile Development is Critical
Let’s do a quick recap: By simply testing code as it is written, developers can spend less time fixing bugs and more time developing new features, to become more productive, more reliable and trustworthy and even more flexible in implementing changes. This takes us to our final reason for early testing: Agile development is no longer nice to have, but business-critical. All of the attributes described above are quintessential for succeeding in an agile team. Developers are expected to be flexible, reliable and build a lot in short time periods. The days of trial and error, of learning the hard way, are gone. Today, we need to go from design to development to testing to going live in days, not months. Quality is what differentiates products. Developers need to do all of this to be successful.
About the Author/Gabriel Chiriacescu
Gabriel Chiriacescu is the director of Software Development at Ixia, with more than 13 years of experience in embedded software development. He is currently leading the product management effort for the new Ixia Developer solution, an agile test tool designed specifically for developers.
At Ixia, Gabriel has led multiple engineering teams responsible for developing market-leading test solutions in a variety of technological areas, including wireless, broadband access, data center, triple play, authentication and security.
A native of Bucharest, Romania, Gabriel holds a BS in computer science from the Polytechnic University of Bucharest. After living several years in Los Angeles, Gabriel is now based in North Carolina and enjoys discussing CICD, SOA design patterns and the evolution of software development.
Gabriel Chiriacescu is the director of Software Development at Ixia, with more than 13 years of experience in embedded software development. He is currently leading the product management effort for the new Ixia Developer solution, an agile test tool designed specifically for developers.