Digitization presents many new opportunities for businesses to create value. At the same time, it forces IT to rethink the way it operates. Modern CIOs need an operating model that understands the blurring lines between IT and business and is designed for agility and speed. The product-centric IT operating model is rapidly emerging as the answer.
What does the model entail? In the product IT operating model, product lines are funded based on their strategic value to the organization, from the top-down, and are supported by dedicated management and delivery resources. The entire product team is measured on a common set of goals, which include customer experience, time-to-market, efficiency improvements, quality of software and availability and reliability of production environments.
5 Dimensions of Product IT Operating Model
Adopting a product IT operating model requires organizations to take an integrated approach toward product-based development and embark on a journey of continual improvements. While scaling agile DevOps that aligns business outcomes with engineering effort, enterprises must focus on these five key dimensions:
1. Organizational Change Management (OCM): Successfully transitioning from the traditional IT operating model to a product-based development approach requires organizationwide change management. An experienced IT service provider, can prove to be a key ally in ensuring a smooth transition.
2. Culture: Culture is an intrinsic characteristic of any organization, and instituting small but effective cultural changes coupled with OCM is the right approach to transition, rather than a “big bang” change. That’s also what some of the well-known enterprises across industries are doing.
a. Teams and Collaboration: The alignment of large teams across multiple organizational groups delivers to a common plan on the pace of change and helps to continuously embed innovation and experimentation for improved services.
3. Structure: Product-focused and highly talented teams across design, architecture, DevOps and operations are key to enhancing business capability. By measuring the entire team on same set of goals, it is easy to enhance customer experience, efficiency and quality of software while creating a stable production environment.
Structure also means aligning traditional application portfolios to specific business outcomes and bringing together all the capabilities and resources required for an organization to deliver on its business outcomes.
4. Technology: If business models, organizational structures, people and ways of working are the gears that run the enterprise IT engine, technology is the grease that enables this engine to run smoothly, without friction. Two key aspects of technology include:
a. Application Programming Interfaces (API): API-based architecture provides an extensible framework of building blocks, which in turn can be used to build powerful applications or software products. Easy maintenance is a hallmark of API-based architecture, as it can be quickly migrated further to any other target architecture.
b. Cloud, Microservices and Containers: Modern organizations need on-demand scalability. Cloud and microservices enable just that without disturbing ongoing business operations. However, it is important to first understand an organization’s product vision and road map, business priorities, competition, pace of market transition in their area of business and industry trends before choosing cloud or container-based infrastructure.
5. Process and Automation:
a. Agile Delivery Model: This is a critical prerequisite to building a minimal viable product (MVP), ensuring rapid delivery of critical enhancements or delivering a suite of product features in appropriate increments. Industry-standard agile practices such as LeSS and SAFe should be used, as they allow complete flexibility to tweak the model on the fly.
b. DevOps and DevSecOps: A DevOps culture is fundamental to adopting a product IT operating model. Continuous integration and delivery (CI/CD) require DevOps to be fully interwoven into the software development life cycle (SDLC), along with the right tools and governance mechanisms. This ensures high-quality products are delivered within short deployment lead times, along with the ability to fix production issues with low mean time to restore (MTTR) and ensure minimal to no change failure rate. DevOps also enables new-age practices such as automated testing, self-service access to infrastructure and automated performance management.
DevSecOps goes hand-in-hand with DevOps. It ensures security aspects are built into the products across the delivery pipeline, so the products are not only fault tolerant but also fool-proof, to a large extent. An experienced DevOps team builds and integrates security tools, solutions and processes to safeguard the organization.
c. Test Automation and Test-Driven Development (TDD): By integrating test automation into the development process, the product IT operating model allows organizations to create high-quality and quickly deployable products. Additionally, TDD enables governance to be integrated into the delivery pipeline by developing quality code in line with test cases written to describe user requirements more precisely. Tests are fully automated to ensure quality of code.
d. Service Health: With KPIs focused on the financial health of engagement, it’s easy to set a budget based on business capabilities and product levels to meet business goals effectively. The KPI framework encompasses key KPIs such as business, financial, learning and innovation and employee engagement outcomes, while staying flexible enough for different contracting models.
Delivering Value in a Changing Business Environment
Digitization mandates that IT deliver on three important aspects: Improve time to market, optimize costs and increase quality. Adopting a product IT operating model helps IT deliver on the mandate by enabling a fail-fast way to deploy software quickly and cost-effectively, without sacrificing quality. However, given the extent of change involved in the shift, it is not something that can be achieved overnight. It must be underpinned by a journey of cultural change and continuous learning, using tools such as design thinking-driven, persona-based change journeys, alignment workshops and targeted campaigns and communication. Partnering with an expert system integrator can help organizations jump-start the process and ensure on-track transition with minimal disruption to business.
If you have been investing in test automation, but it’s not getting you where you think it should, it could be that you approached it superficially.
One of the most common issues is that the test part of test automation is rather straightforward to automate, especially with today’s abundance of great automation tools. But jumping straight into automated testing leaves you with a big heap of automation scripts, and often not the best ROI. Understanding why requires looking into the automation process.
When an engineer builds an automated test, they do it in a test environment that they (or someone in the organization) set up. Usually, this is the same way manual testers get access to test environments.
Before the engineers start automating, they have a working copy of the product they are testing, with relevant data on it. They have a good idea of the test they want to write. Everything looks promising.
They write the test and give it a few runs. Everything is looking nice and stable. Now, to provide the ROI we were hoping for, the test must run a million times. Or, at least, a few thousand times, right?
To reap the benefits of automation, it needs to happen frequently enough. However, the bottleneck is often discovered after creating a bunch of tests. Because the tests must run on a similar test environment to the one used to create them, with the new versions of the product or with any introduced changes that are being tested, you need access to hundreds of environments. This requirement makes it difficult to test effectively.
Below are three test environment approaches that inhibit test automation.
Manually Setting Up an Environment Each Time to Run Automated Tests
For tests that don’t need to run as frequently, such as security testing and performance testing, it’s common to set up an environment for automated tests every time the test needs to be executed. As a result, testers must wait for the environment to be ready, perhaps do some additional configurations and then run a suite of tests. And, if they’re trying to optimize utilization, the environment is left to run overnight or weekend.
The problem with this approach is it takes the “auto” out of automated testing. When automation depends on a manual step, the test is only as efficient as this manual step.
If the manual step isn’t very efficient, automation may not be helping at all. I’ve seen companies take so long to get an environment to work that running automated tests on it didn’t make any difference from a time perspective.
Setting Up a Static Environment That Automation Always Runs On
When setting up an environment takes a long time, it’s natural to assume it’s worth leaving an environment running so test automation can run on it. The preparations were made, the infrastructure is all in place, the product is configured and the data is ready, so let the automation party begin.
While complex, static test environments may seem ideal, but the party doesn’t last long. First, since it usually takes some time to put the environment together (same as the previous case), it often becomes high-maintenance. It can be scary to make changes that could ruin things, and you can end up needing someone to manage change full-time. Much like chocolate cake in the office kitchen, an environment that mimics production with good data on it tends to attract everyone. Soon, these nice environments become shared assets and tests start conflicting for the same resources. The natural next step is to set up another static environment. This loop usually ends with an angry manager saying something about cutting CAPEX.
Automating the Environment and Cutting Ourselves Some Slack
With the rise of infrastructure as code (IaC) and immutable infrastructure, there are more ways to spin up a test environment. Resourceful test teams will try to automate the environment creation or use built-in test tool or CI system capabilities to run tests on a public cloud instance or a container, if the product they test allows that. This is cheap and useful—and could work for some simple tests. The challenge usually starts with everything this approach doesn’t cover. It can be difficult to use IaC to spin up a test environment and handle all aspects of it—from security and isolation to dependencies, data, third-party integration and more. This is where this approach is often blocked and sends us back to the previous two approaches for a big part of the testing load—which again makes test automation investment less fruitful.
The Right Way to Set Up Test Environments: Test Automation on Dynamic Environments
The test automation world needs a game changer. You can make your car faster, but if the road is full of potholes, you won’t be able to get that fast or far. Test automation is similarly limited by the pace of environment creation, with its infrastructure, application, data and all.
This challenge becomes critical when adopting DevOps and testing stops being a stage in the pipeline, instead becoming an overall pipeline service. QA is no longer about being the gate between development to operations. It’s about overseeing the quality of the product during its entire life cycle. This means automated tests should always be able to run, and the environment they run on must be dynamic.
The solution to this challenge is environments as a service (EaaS). EaaS is all about packaging the environment, including the application, infrastructure and data, as an entity that can be provided as a service to a business process. Once defined, it can be provided on-demand for manual or automated users. Since it’s a well-defined entity, it also can be decommissioned when the process finishes using it. Playing an important role in making infrastructure horizontal, EaaS is also the key for paving the road for brilliant test automation.
If you think you should get more from your test automation initiative, it may be a good time to make EaaS part of your automation strategy.
While all developers strive for bug-free code, it’s particularly crucial in a blockchain deployment where sensitive data or other confidential info is being exchanged, such as in health care or finance. However, some businesses have learned that lesson the hard way. Cryptocurrency exchange Binance recently revealed a devastating security breach that resulted in a loss of more than $40 million. While this is a large sum, a security breach could be even more costly for businesses that lose private messages, confidential contracts and more.
Rebuilding (and regaining customer trust) is not where organizations should be spending their time. A better way forward is for company leadership to mandate a blockchain test automation strategy from the very beginning.
While most software testers are familiar with testing web applications, they may not be sure how to approach blockchain testing. The good news is, some of the same focus areas of web testing apply to blockchain testing, too (functional, performance and security). However, blockchain involves a greater number of tests, effort and focus areas, such as infrastructure orchestration, staging and simulating scenarios in a distributed environment, which requires an even greater level of expertise.
Because of these additions, fixing a bug in a blockchain app requires several extra steps and, as a result, it is more time-consuming and costly than fixing bugs in other apps. For example, to reproduce bugs or verify code, developers should perform these steps multiple times:
Form a new endpoint.
Deploy new code to the new endpoint.
Migrate the current data to the new back end.
Suspend the old back end.
Update all front ends to the current version.
After that, verifying the bug-fix performed by testers or continuous integration (CI) tools will require these steps again. This complexity makes it difficult to release bug-free blockchain applications, yet doing so puts organizations and their customers at risk of financial harm. Due to the high-risk nature of deploying blockchain applications, it’s critical that testing is included in the managerial vision and strategy.
To release blockchain apps with bug-free code, businesses should follow these guidelines:
Integrate Test Automation Into Your Blockchain Testing Strategy
Test automation can relieve testers from performing thousands of orchestrating tests manually, which is both difficult and inefficient. A good system architecture should be scalable, maintainable and testable, so teams can test components independently along with mocked components. If a test requires a different environment or infrastructure, infrastructure as code (IaC, such as Puppet, Chef, Ansible and AWS is invaluable in setting up appropriate test environments. Another crucial part of the strategy is to provide the infrastructure with declarative definitions and allow developers to implement codes that support test automation.
Run Unit and Integration Tests to Cover Both Back End and Front End Code
Blockchain applications contain at least a back end (a set of smart contracts or chain codes running on a blockchain) and a front end (where users can interact with data stored in the blockchain). Performing unit tests will cover both back end and front end code, and, because of its critical role to blockchain apps, its primary focus should be on the chain codes. Integration testing, however, should cover the various types of oracles and front end applications. Performing additional tests will eliminate any holes that can threaten the back end by verifying the integration between the UI and chain codes.
Perform Tests in a Local Infrastructure
Though public blockchains (such as Bitcoin or Ethereum) have several test-nets, it’s optimal to perform the tests in a local infrastructure. Local infrastructures are lightweight, faster and more stable for unit and integration tests. For example, in a Ganache local infrastructure, it’s possible to run more than 1,000 unit tests on five Ethereum smart contracts with approximately 3,000 deployments in an hour or less.
Continuously Update Security System Test Suites
Blockchain is a multi-party infrastructure and in some cases, it is a public network. As a result, these applications will be visible to many stakeholders and may even be open-sourced to the world. This level of transparency means that hackers are able to study the programs and plan attacks based on a particular system. Continuously updating security system test suites will ensure applications are protected against all common security vulnerabilities.
Minimize Continuous Testing Pipelines for Each Phase of the Development Process
Layering integration and unit tests in a local infrastructure with continuous updates to the security system test suites creates a continuous testing pipeline that is essential to high-quality blockchain code. Continuously verifying code changes in multiple environments, such as local blockchain and test-net, ensures high-quality code is delivered to the production blockchain. These tests, however, can last hours or days for each code change, so it’s important for teams to minimize and create lean pipelines that focus on test stages, test types and test environments, and develop multiple pipelines for code at each phase of the development process.
Build Customized Tests Suites to Eliminate Prospective Cheating Holes
Test teams also need to customize test suites in the pipelines to cover both functional and security testing. Covering security cases is important, but unit test modules should also exercise smart-contract-unit potential security threats, such as overflow and re-entrancy, to eliminate as many prospective cheating holes as possible.
Due to the nature of decentralization and orchestration of underlying infrastructure, developing bug-free code for blockchain applications presents many more challenges than other applications. By taking best practices from traditional software testing, especially test automation, and customizing them for blockchain development, businesses can achieve bug-free blockchain coding and protect their most vulnerable applications.
In addition, SmartBear announced it has integrated Zephyr with the rest of the company’s SoapUI Pro, TestComplete, CrossBrowserTesting and LoadNinja testing tools.
SmartBear committed to integrating its entire portfolio of tools with other Atlassian offerings as well, including the Bitbucket continuous integration/continuous development platform and Confluence content management software.
In general, SmartBear views efforts to modernize applications using DevOps practices as a primary driver of increased demand for tools to test application programming interfaces (APIs). A recent global survey of 3,000 API developers, architects, testers and product leaders conducted by SmartBear finds nearly three-quarters of respondents believe that adoption of microservice architectures will drive the biggest growth in API adoption in the next two years.
The biggest challenge faced by well over half the respondents (58%) is finding a way to standardize API development. In pursuit of that goal, the survey finds 69% of survey respondents are now developing APIs that comply with the OpenAPI Specification (OAS). That specification defines a language-agnostic interface to RESTful APIs that enables both humans and computers to discover and understand the capabilities of the service without access to source code or documentation, or having to inspect network traffic.
The SmartBear survey also finds 76% of respondents say they already have a formal API monitoring process in place and that 76% of organizations are developing both internal and public-facing APIs.
Daniel Giordano, associate director of product marketing for SmartBear, said these days there’s a lot more focus on trying to make it easier to discover APIs whenever possible by setting up API catalogs. Those catalogs are also now making it easier for organizations to reuse APIs rather than develop new ones for each microservice that gets developed, he added.
In fact, Giordano noted that many organizations now define the APIs they are going to depend on to integrate their applications long before they begin to write application code. Those APIs are then reviewed by an architecture team to make sure they meet performance requirements while still being flexible enough to evolve consistently with the other APIs the organization needs to maintain, he said.
Today, organizations make use of many more private APIs than they do public ones. APIs first came into their own as a means to expose applications to enable external applications to consume data from another application as part of the so-called API Economy. Since then, the number of private APIs that organizations rely on to integrate internal applications has exploded. In fact, building and deploying those APIs has become an integral part of any DevOps process.
Naturally, the relationship between API management and DevOps processes is still evolving. But as DevOps processes within IT organizations continue to mature it’s only a matter of time before they are applied equally to both the applications being developed and the APIs relied on to integrate them.
Continuous integration (CI) calls for code to be always working. Continuous delivery (CD) and continuous deployment mean that we must be able to roll out a working configuration to production at any time. As DevOps becomes the norm, bugs in production are now a developer’s personal problem. Developers (and their managers) want to know immediately if code changes they are working on can break something.
What is Continuous Testing?
The only guaranteed way to prove code and configurations are working is to test them. That’s why we need continuous testing. In organizations making this transition, performance engineers and test automation experts shift from minding a backlog of test-building work, to facilitating a democratization of testing. Everybody builds their code and everybody tests their code, continuously.
Continuous testing is becoming the “new normal.” In this modern and agile testing approach, developers are equipped with low-friction testing capabilities and tools and are expected to unit test any new or updated code. With testing shifted all the way left to developers in pre-production, and all the way right to post production monitoring, organizations are just seconds away from proving code is ready and working to acceptable service levels.
Although continuous testing is becoming a more standard practice, it certainly presents unique challenges for execution.
Lack of clear requirements can result in wasted time and delays. Behavior-driven development (BDD), acceptance test-driven development (ATDD) and model-based testing using tools should be used so that all requirements are clearly documented and communicated.
Test cases need to be clearly defined and test scripts created ahead of time, to enable continuous testing of code at all stages of production.
Optimize Testing Processes and Test Coverage
Save time, money and resources by testing only what you need to test. Visual models allow various paths to be explored and optimized so that only your tests provide maximum coverage, without unnecessary testing.
It is also recommended to import test cases, remove duplicates and distribute and allocate optimized test cases in existing tools.
Shift-Left Your Testing (and Shift It Right as Well)
What do we mean by shift-left testing? Tests should be run earlier in the development cycle. Developers test as they go, with test automation including functional testing, performance testing, monitoring and security testing. This requires developer-friendly tools for easy adoption and use.
By shift-right testing, we continue to test through production and post-production, including continuous monitoring. Gartner recently released a report on the shift-right testing, calling it a “must” for adopting DevOps practices.
Provide Complete Test Environments
The ability to provide virtualized test environments is critical to achieving continuous testing. Eliminate blocks and reduce wait times by providing complete test environments on demand, with dev-friendly tools (as code, CI/CD integrations, supported open source). These environments should include test data on demand to ensure that teams can perform comprehensive tests with production-like data. These environments should be ephemeral; ready on demand, decommissioned after use.
Get the Right Test Data
The inability to get robust test data causes significant delays in many application release cycles. To accurately test new functionality, test data should be as close as possible to what the application will encounter in production. If the test data lacks certain real-world characteristics (i.e. actual fields, data specifications, negative scenarios, etc.), the tests are unlikely to find many potential issues or break the application where there are weak points. Understanding data models in order to extract the right data is a special skill in and of itself.
Production data is clearly the best data to use for testing, but data privacy regulations and team silos often restrict its availability. The answer to this dilemma is a robust test data management tool that lets you copy production data and mask it to protect sensitive information, while maintaining the characteristics that make production data desirable for testing.
Remember, continuous delivery and continuous testing are processes that can take time to implement successfully, but teamwork, continuous feedback based on data and the right tool set will help you on your journey. You can read more about implementing a continuous testing policy here.
This is the sixth in a series of articles highlighting speakers at SauceCon 2019. SauceCon brings together the global community of Sauce Labs users and continuous testing experts, to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 23-25 in Austin, Texas.
Richard Bradshaw is known for running the Ministry of Testing, speaking at events all over the world and for being the “Friendly Tester”—a nickname given to him by a developer.
However, one thing toward which Richard isn’t 100 percent friendly is the Test Automation Pyramid. He says it can and should bring about conversations, but thinks it lacks too much to function as a solid automation strategy, which is how many people use it. At SauceCon 2019, Richard will present some other options in his talk, “Pyramids Are Ancient – Let’s Talk Automation Strategy.”
Richard hopes that people will take away the fact that to truly be successful with automation, you need to fit automation into your overall testing strategy. During his talk, he will review the steps that he takes when conducting an automation audit, steps attendees will then be able to immediately take back to work and begin applying. “I also hope that observing me critique existing models will motivate them to do the same to some of the models used in their current workplace, and identify ways to improve their approach,” he said.
Richard is well-qualified to help testers improve in this area. As the BossBoss (aka CEO) of the Ministry of Testing, his primary role is to ensure they stick to their mission of “co-creating better testers.” Richard and his team offer numerous ways to learn and share testing knowledge. They run the TestBash conference, manage an online learning platform called The Dojo and host meetups all around the world.
When he is not working or thinking about work (a rarity!), Richard loves cooking up a storm, hiking and traveling.
You can hear from Richard and more than 30 other expert speakers at SauceCon 2019, the third-annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.
This is the fifth in a series of articles highlighting speakers at SauceCon 2019. SauceCon brings together the global community of Sauce Labs users and continuous testing experts to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 23-25 in Austin, Texas.
SauceCon 2019 speaker Wim Selles is passionate about front-end test automation and his enthusiasm is infectious.
He started manual testing back in 2007, then began dabbling in web development to challenge himself. He soon fell in love with front-end web development, deciding that if he truly understood how code works, he would become a better tester.
While employed at Tele2, a large telecom company in the Netherlands, Wim realized the need for an image compare tool for native apps, but he couldn’t find exactly what he needed. So he decided to roll up his sleeves and build an open source module himself.
Wim’s SauceCon presentation, “Building Your Own Native App Visual Testing Module with Appium,” will focus on exactly how he did this. He will share the things he learned about Appium in the process, as well as how he managed to create a cross-platform solution to answer this need.
Wim wants to make people as enthusiastic about test automation as he is. As a solutions architect at Sauce Labs, he helps customers successfully implement the Sauce Labs Continuous Testing Cloud. He conducts trainings, shares best practices and travels the world speaking (and learning!) about coding and testing.
In addition to his passion for test automation, Wim enjoys contributing to the open source community and spending time with his family and two young daughters.
You can hear from Wim and more than 30 other expert speakers at SauceCon 2019, the third annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.
Mindless testing, whether it is manually or automatically executed, can cover only a very narrow scope of possible bugs: the ones you’ve already thought of. Exploratory testing however, endeavors to think outside that scope. Thoughtful testers look at a problem from all angles, weighing factors such as feature usage, psychology, user stories and more. Their goal is to chart the uncharted territory of an application.
The faster our development cycles become, the more thoughtful our testing should be. It’s only through techniques such as exploratory testing that we’ll be able to get there.
Continuous testing is the process of executing automated tests to obtain rapid feedback on the business risks associated with a software release. Where does that leave exploratory testing? It’s not automated, but it’s certainly critical for determining whether a release candidate has an acceptable level of risk.
Test automation is perfect for repeatedly checking whether incremental application changes break your existing functionality. However, where test automation falls short is in helping you determine whether new functionality truly meets expectations. Does it address the business needs behind the user story? Does it do so in a way that’s easy to use, resource-efficient, reliable and consistent with the rest of your application?
Exploratory testing promotes the creative, critical testing required to answer these questions. Obviously, it doesn’t make sense to repeat the same exploratory tests continuously across and beyond a sprint, but exploratory testing can be a continuous part of each delivery cycle.
Exploratory Testing in Action
Here are a few ways teams embed exploratory testing throughout their process.
Perform Ad Hoc Exploratory Testing as Each User Story is Implemented
This is the exploratory testing equivalent of peer code review. When a developer completes a user story, they sit down with a tester. The tester starts testing while providing a running commentary on what they are doing and why. Next, the developer takes control, explaining how they would test the software given their knowledge of the implementation details and challenges. The developer gains a user- and business-focused perspective of the functionality, and the tester learns about the inherent technical risks.
Another tactic is to have the developer and a tester separately test the same feature simultaneously, then discuss their findings at the end of the session. Often, this turns testing into a competition, where each participant tries to uncover the most or “best” issues in the allotted time.
Align Exploratory Testing Sessions with Full Regression Testing
It’s simply not possible to perform exploratory testing or full regression testing on every code commit. That’s what smoke testing is for. Instead, many teams run full regression testing and session-based exploratory testing in parallel a few times per week, whenever they’ve implemented new functionality that a user could feasibly exercise.
For optimal results, these sessions should be lightly planned and tightly timeboxed, include diverse perspectives and really take the six thinking hats theory seriously.
Host Blitz Exploratory Sessions for Critical Functionality
The best way to uncover user experience issues before users do is to get a broad array of feedback prior to release. One way is to host “blitz” exploratory testing sessions. When you’re wrapping up work on critical new functionality, invite people from a variety of backgrounds and teams to participate in a short timeboxed session. Incentives can help drive participation, maximize results and make testing fun.
Using test automation to continuously check the integrity of existing functionality is certainly critical. However, if you’re not also making exploratory testing a continuous part of your process, how will you know if the new functionality meets expectations?
The goal of continuous testing is to understand whether a release candidate has an acceptable level of risk. Exploratory testing is perfectly suited for helping you answer that critical question.
Safety, security and controls are the foundations upon which 135-year-old ExxonMobil is built. As such, quality has become a key component in delivering software. The company is currently undergoing a shift from traditional project management organizations and support organizations to product teams, which has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.
Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”
To do this, ExxonMobil started with the basic idea—quality control manages risk—and then explored what is tracked today and whether quality is being measured. Another key activity was to remove the roadblock to continuous testing.
Ann Lewis from ExxonMobil discusses in this video how the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset, as well as the company’s view that continuous testing and deployment can only be accomplished through model-based testing and that managing quality reduces overall cost of ownership.
The full transcript of her discussion is below:
“At ExxonMobil, we have presence in six continents, so that means we get to deal with a lot of people. There’s over 72,000 employees plus another 20 percent contract staff, and that’s a problem. Guess what they want to do? Manual testing. Yeah, because they’re not gonna give it up. We have been on an adventure this last year that I’m gonna talk about and explain to you how we help train these 72,000 plus people that there’s a better way to do it.
“In addition to the numbers of people that we have, the upstream, I always … Well, someone referred to it about sticking a straw in the ground. Yeah, it was actually somebody from Tricentis. Yeah. I was just like, ‘There’s a little bit more to it than that.’ No, we do produce all of the product from under the ground. We get it out, and then we send it downstream to our refineries so they can do something with it like sell it. We also have a chemical company, that … I think right now are second or third to DuPont. We have a lot of product and a lot of things to do with it. That’s a problem, too. Because when it comes to the applications that test, that need to be tested, we have something like 5,000 applications that run our business all technologies. We’ll talk a little bit about that as we go through, too.
“The title is The Middle Child Rises. Why did I choose this? Because in the world of Dev and Ops, test is in the middle.
“We have a few other friends in the middle, and they are a little bit like a middle child, so we’ll talk about that. For today, your takeaways are that our organizational design has shifted. We’re going to explain why and how that happened. We will explain that Tosca is strategic for ExxonMobil and we will explain why that is.
“Tosca is an enabler to speedy delivery.
“You’ve heard about that this morning from many of our Tricentis friends, and it is true and I’m here to share that story. We now have within ExxonMobil, a quality philosophy, yeah, and it’s a lot more than just testing. Yes indeed, it is. We’re going to take a look at what we did and have a lot of fun stories in between.
“Being the middle child, you really didn’t get as much attention. How many are middle children, by the way? A few, okay. Do you have babies in here? Yeah, yeah, you kind of get away with everything, don’t you? The oldest, I have a favorite, I know. Oh, there’s another one out there of my fave, okay, cool.
“It’s very interesting in family relationships how these children get along. The oldest kind of sets the stage for everybody. That middle one comes along and it’s like, ‘Oh, it’s the second kid.’ The third one comes along and it’s like, ‘Oh, it’s the baby, it’s the baby,’ and the middle child just kind of sneaks through, okay? They don’t get quite as much attention. We’re going to talk about the middle child as they relate to testing.
“Typically, it’s the middle child that ended up with rules. That oldest child was all about, if you just tell them once, they usually would stop doing it. That’s the case I had. Then the second one was born, that turned out to be the baby, but I swear he’s got components of being the baby and the middle child. It was like, ‘Adam, we have rules in this household. Yes, we have rules.’ He was like, ‘I just don’t get the same respect as that older sister, Jessica.’
“Now within the corporate world, these middle children, you can think of them as being security and controls, data migration, data conversion. It’s like, ‘Oh, yeah, we forgot about you.’ Last but not least, testing. How many times has it been when suddenly that project shows up, ‘Oh, yeah, I think I might need to engage your team to help me organize my testing.’ Yes, yes, those of us that live in the testing world.
“The big deal right now, though, is that these middle children, our testing organizations, our quality assurance organizations, are becoming the key enabler, like we heard this morning, 50 percent of our problem is with testing. Oh, I could do that but I can’t get testers. I could do that but, ugh, we’ve got to do the testing. It takes so long.
“According to the World Quality Report, the reliance on manual testing is the top technical challenge in app development.
“How many of you knew about the World Quality Report? Yes? Yes? Good, good, good. We’re all the same kind of people, yeah. I like that.
“Businesses see test automation as a way to address one of their top challenges to delivering better software faster, by adopting Agile, DevOps and continuous delivery practices to shorten delivery cycles.” – Forrester
“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks.” – Gartner
“The need to support faster time to market with higher quality is driving the demand for effective functional test automation tools. This is why we’re here. Tricentis Tosca, it is a framework. Yay! It’s not scripting. Boo!
“Tricentis Tosca will address that top technical challenge of delivering software faster.
“The need to support faster time to market with higher quality. It’s higher quality in the form of risk coverage. That is what we do with Tricentis Tosca. That is what we do with Tricentis Tosca.
From a framework perspective, the last year and a half we have been on a mission to explain the difference on automation tools. We have gone from record and replay, to test automation from a framework perspective using scripting, such as with the Hewlett-Packard products and Selenium. It is programming, it takes developer time to do automation when you’re talking about some of these historical tools. We have now moved into model-based, and I hope everyone in here has been to at least some of the Tricentis’ demos on what model-based is. I’m not going to go into detail here, but I will tell you that it has taken our automation from a 40-hour event down to a 8 to 16-hour event. No joke.
“I have been able to take Tricentis Tosca and put it into non-developer, normal people’s hands.
“I actually sat down with one of the Tricentis’ architects and did automation myself. I said, ‘Oh my gosh, I haven’t touched a keyboard to do anything technical in so long,’ and I actually did it. That is key, that is significant. That has been the story that has been going on with ExxonMobil for the past year and a half.
“It has become strategic for us. It’s automated screen scanning, it’s lower maintenance, it’s targeted testing. I’m doing it in half of the time, if not less. It’s faster, it’s easier to support, it’s talking about prevention versus detection. We have created a road map.
“What you’re seeing in our colorful TFS diagrams on the top, yes, we are doing our monitoring in TFS. You’re seeing a migration and better put a re-engineering of our existing scripts into Tosca. Re-engineering because for the last two years that I’ve been in this job, people kept telling me, ‘Ann, did you know that those existing Hewlett-Packard scripts really don’t test anything?’ ‘Well, gee, I don’t think I knew that.’ ‘Well, gee willikers, let’s see if it’s really true.’ Lord, have mercy. Twenty percent of those Hewlett-Packard scripts that were driven by the business to automate didn’t test anything. Well okay, they tested something but it wasn’t worthwhile.
“All of these little blue dots on your right beyond the solid blue vertical line are things that don’t really matter because we didn’t do a risk analysis. When I talk about re-engineering, we’re not just taking what we had and push it into Tosca, which you can do, by the way, there’s a tool for that. It’s up to you to use it, I’m not going to say anything. All I’m telling you is all our experience has been it made no sense.
“What we have found is when we sit down with our subject matter experts and we had this really cool dialogue like, ‘What do you do all day long?’ ‘Oh, I use VA-01 and NM-16 all day long.’ I said, ‘Cool. And so what happens when one of them doesn’t work? Dead in the water. You’re in customer service, aren’t you?’ ‘Yeah.’ ‘How many sales? Millions of dollars of sales do you do every day?’ ‘Well … ‘ This is the user, I mean they are totally into explaining to us what they do. Through this dialogue, this interview, we find out that they’re going to cost us $1 million a day, an hour, I don’t know what the big number is but it’s a big number, when it comes to customer-facing transactions. Therefore, those transactions are the ones that we want to automate.
“Now if I find out, no offense to anybody in controllers, that at the end of the month they have a report that they need to generate, ‘I’ve got some time to fix that one, okay?’ ‘How often do you do this? Monthly? Okay, that’s a one. What happens if it doesn’t work?’ ‘Well, I just did last month’s report and … ‘ ‘That’s a one.’ Low priority, move on.
“We have been re-engineering all of our existing tests.
“This word crypt is an ugly word. I actually I know the person who named our existing automations. Critical Integrated Process Testing. We don’t say that word anymore, by the way.
“The middle child has risen. This is my team. When I joined the group in February of 2015, we had about 25 people. About 25 people. Within ExxonMobil, I think there might have been seven of us. Then, we had some offshore group that did all of those monthly executions of those Hewlett-Packard scripts.
“Today, I have Vulcar over automation, he’s based in Hamburg. I have Cathy in Houston for functional testing and Santa Juan for functional testing, basically split between the two regions. They’re what we call test leads. They do a lot of the coordination of our testing activities. I have Aaron in Houston for performance testing. I have Jay, he is Doctor Jay, he is the man that knows everything about testing and everybody loves Doctor Jay. We have Suzanne right there, one of my oldest children. I didn’t say that, did I? I meant oldest child. Anyway, the other thing that was really funny about these pictures, they kept saying, ‘Oh, I want mine to be skinny.’ I want mine to be really upfront. We had a whole dialog on how the pictures turned out. It was pretty fun. Guess what? There’s close to 150 people now. That’s a signal that over this past year and a half, two years of getting this team put together, the emphasis that ExxonMobil has put on quality. One of the ways to get to quality was through testing.
“We are everywhere. I’m going to go into a little bit more detail now about organization because it’s good to understand where you are located. I was talking about being on the six continents. We have now placed people in region, so that when a person has a question on, ‘I want to automate, should I automate?’ We’ve got people in those locations to help with that. You will see we are using architects.
“Within the Tricentis nomenclature, you have architects, automation specialists and automation engineers. It is a tiering of skillset.
“We have found that the architects become those go-to people, when it comes to, ‘Should I be automating this or should I not and how do I go about it?’ Yes, we are using some offshore resources in India. Yes, we have people in Auckland. Yes, Bangkok, and that’s actually one of our ExxonMobil business support centers in Bangkok. Then, you can see Houston, Vienna, New York and Curitiba, Brazil.
“Organizationally, what we have found is that when it comes to implementing automation and doing it in a … I’ll call it conscientious and disciplined way, you’ve got to have a structure. Don’t think that you can just, I’ll say, run into doing automation. There needs to be a structure upfront to help you manage this work, because once one customer comes, you’re going to get five more. The enthusiasm level that we have experienced is unbelievable. We’re almost on tilt. But through this concerted effort of having an organization to help support it through a governance and then program management. Again, you can see this partnership that we have with Tricentis. Highly recommended when it’s time to get started.
“Successful team composition. It’s typically one architect to one engineer to 10 specialists and then you ramp up or ramp down, based upon if you’re working on systems of engagement. The shiny coin fun stuff, mobile, gooey, or your systems of record. They’re a little bit of a worn coin, all right? I’m sure some of you have systems that are quite old. We have ones that are really old so our financial system is still in IMS in PO1. Yes, rock on! That’s what I grew up with. Yeah, give me some JCL, too.
“See, geek, we’re all geeks, I know. It’s like once you join the world of IT, you’re just like into it. Nobody really understands what you talk about, unless you’re with people that are like you. Uh-huh (affirmative), all in the family, totally all in the family.
“Okay. We also have an immersion program. We have, again, the process of getting somebody into you’re approved, we think you can handle the work, all right? Going through automation specialists and all the way up to graduation. Again, working with Tricentis, we have come up with again, a disciplined process. You need this when you get started because they will come in out of the woodwork, I want to automate. I talked to you two years ago about automating those processes at the refinery and you said that no machine could do what you could do. I don’t say that, but that’s what I’m thinking when they come to me two years later with, ‘I think I want to do automation.’ I’m like, ‘Cool, let’s get into it.’
“We have a quality manifesto. Yes, we do. We will create forward-looking, value for our customers and for our organizations through quality approaches. Quality engineering is a team responsibility. I’ve got a great team, but we’re just here. We’re the messengers, we’re the trainers, we’re the coaches. Everybody that we work to, we talk about quality.
“How did you get to work today? Did you think about a safe way to get to work today with all of the rain? Yes. That’s a quality decision right there. Little things that we teach when we’re starting to push the quality manifesto out. You have to understand business function risk. Again, what do you do all day? What happens if it doesn’t work? Oh, really bad things happen, or maybe not so bad and a month in close, we’ve got a whole month to worry about it. Yeah, we can always close the books this month of last months’ numbers. I actually have a degree in accounting, I can talk about them.
“Including quality checks throughout the delivery life cycle. Quality gates. That’s another item that we have built into our system delivery life cycle. Starting to talk about at the moment requirements are written down, let’s start talking about what you’re going to test, based upon those requirements. We have built these quality gates, these questions throughout system delivery life cycle.
“What has happened? Our organizational design has shifted.
“Tosca is strategic for ExxonMobil. Tosca is an enabler to speedy delivery.
“Test automation brings value. We’re actually connecting the automation into total cost of ownership, as a way to reduce application total cost of ownership. We have a quality philosophy now and we’re just pretty darn excited about what we did. It took the rise of the middle child in order to make it happen. Thank you so much.”
For more than two decades now, software testing tool vendors have been tempting enterprises with the promise of test automation. However, most companies have never been able to achieve the desired business results from their automation initiatives. Recent studies report that test automation rates average around 20 percent overall, and from 26 percent to 30 percent for agile adopters.
We believe that several factors contribute to these dismal automation results:
Legacy software testing platforms were designed for a different age
The most commonly used software testing tools today are predicated on old technology, but enterprise architectures have continued to evolve over the years. Development no longer focuses on building client/server desktop applications on quarterly release cycles, with the luxury of month-long testing windows before each release.
Almost everything has changed since test automation tools such as those from Mercury, HP, Micro Focus, Segue, Borland and IBM were developed. Retrofitting new functionality into fundamentally old platforms is not the same as engineering a solution that addresses these needs natively.
Legacy script-based tests are cumbersome to maintain
Scripts are cumbersome to maintain when developers are actively working on the application. The more frequently the application evolves, the more difficult it becomes to keep scripts in sync. Teams often reach the point where it’s faster to create new tests than update the existing ones. This leads to an even more unwieldy test suite that still (eventually) produces a frustrating number of false positives as the application inevitably changes. Exacerbating the maintenance challenge is the fact that scripts are as vulnerable to defects as code—and a defect in the script can cause false positives and/or interrupt test execution.
The combination of false positives, script errors and bloated test suites creates a burden that few QA teams can overcome. It’s a Sisyphean effort—only the boulder keeps growing larger and heavier.
Software architectures have changed
Software architectures have changed dramatically, and the technology mix associated with modern enterprise applications has grown immensely. We’re trying to migrate away from mainframes and client/server as we shift toward cloud-native applications and microservices. This creates two distinct challenges:
Testing these technologies requires either a high degree of technical expertise/specialization or a high level of business abstraction that allows the tester to test without diving into the low-level technical details.
Different parts of the application are evolving at different speeds, creating a process cadence mismatch.
The software development process has changed
Although most enterprises today still have some waterfall processes, there’s an undeniable trend toward rapid iterations with smaller release scopes. We’ve shifted from quarterly releases to bi weekly or daily ones, with extraordinary outliers such as Amazon releasing new code to production every 11.6 seconds. This extreme compression of release cycles wreaks havoc on testing—especially when most testers must wait days or weeks to access suitable test environment and test data.
The responsibility for quality has changed
In response to the desire for faster release cycles, there’s been a push to “shift left” testing. The people creating the code are assuming more responsibility for quality because it’s become imperative for getting to “done” on time. However, for large enterprises working on complex applications, developer-led testing focuses primarily on a narrow subset of code and components. Developers typically lack the time and the access required to test realistic end-to-end business transactions. Although the onus for quality has shifted left, the legacy platforms, rooted in waterfall processes, have a distinct bias toward the right. This makes it difficult to blend both approaches.
Open source testing tools have changed the industry
The rise of open source software testing tools such as Selenium and SoapUI have had both positive and negative effects. Traditionally, open source testing tools are laser-focused on solving a very specific problem for a single user. For example, Selenium has become an extremely popular script-based testing tool for testing web interfaces. Yet, although Selenium offers speed and agility, it does not support end-to-end tests across packaged apps, APIs, databases, mobile interfaces, mainframes and other components. There’s no doubt that most of today’s enterprise applications feature a web UI that must be tested. However, in large enterprises, that web interface is just one of many elements of an end-to-end business process. The same limitation applies to SoapUI and API testing.
So, Now What?
Software testing must change. Today’s software testing challenges cannot be solved by yesterday’s ALM tools. With disruptive initiatives such as DevOps, continuous delivery and Agile expanding across all industry segments, software testing becomes the centerpiece for data-driven software release decisions. This next wave of software delivery lifecycle (SDLC) maturity requires organizations to revamp antiquated testing processes and tools. This means that organizations must have technologies that enable continuous testing, or innovative ideas will remain hostage to yesterday’s heavyweight testing tools.
He started manual testing back in 2007, then began dabbling in web development to challenge himself. He soon fell in love with front-end web development, deciding that if he truly understood how code works, he would become a better tester.
