devopsy.kinsta.cloud

Tag: testing

  • 4 Best Practices for Entertainment App Testing

    4 Best Practices for Entertainment App Testing

    Entertainment apps represent a broad category of software, ranging from mobile games, to social media platforms, to streaming video apps and beyond.

    However, all entertainment apps share a few key traits in common:

    They typically host dynamic content that is unique for each user. This makes entertainment apps different from apps where most or all of what each user sees is the same.

    They host rich multimedia content including, in many cases, streaming audio and video.

    They are often created by vendors operating in highly competitive markets, where a poor user experience can send users flocking to competitors’ apps. There is no shortage of apps out there for gaming, video streaming and so on, and if your app fails to perform, your users can easily convert to a competing solution.

    For these reasons, teams responsible for testing entertainment apps must develop a testing strategy tailored to the unique challenges of these applications. Traditional testing tools and methodologies aren’t enough for running tests that support an excellent end-user experience for entertainment software.

    With that reality in mind, let’s take a look at four best practices to follow when testing entertainment apps. These strategies go a long way toward ensuring that entertainment apps deliver the end-user experience that is necessary for maximizing user engagement and retaining users, no matter how complex or dynamic the apps happen to be.

    Testing for Video and Audio Syncing

    An easy – and all too common – way to shred the user experience in an entertainment app is to stream videos where the visual and audio streams go out of sync with each other. Videos need to be perfectly in sync with audio to keep users happy.

    Of course, the challenge that testers face is that most testing frameworks don’t offer an easy means of evaluating whether visual and audio content are in sync with each other, especially when the content is streamed. Frameworks that let you evaluate when content initially loads or when it updates aren’t enough for analyzing sync between visual and audio streams. As a result, teams have traditionally resorted to testing for sync manually, or not testing it at all.

    Fortunately, there’s a better way. Using AI, you can run tests that automatically identify lag between video and audio. In turn, you can detect problems that may cause streams to go out of sync, then fix them before your users experience sync issues.

    Minimize Test Latency

    When you have an entertainment app where content changes in real-time, your tests need to be able to execute and display results in real-time, too. But that’s difficult to do if network latency issues prevent data from moving between your test cloud and your testing tools quickly enough to provide real-time results.

    There are two ways to solve this challenge. First, you could run tests on on-premises devices in order to avoid network latency problems. Alternatively, if you want to take advantage of a test cloud while still achieving real-time results, you can optimize network traffic in a way that minimizes latency and allows test data to flow between your test cloud and your testing tools in a few dozen milliseconds–as opposed to the hundreds of milliseconds traditionally required for packets to flow from tools to the device cloud, and then back again.

    Test Across Geographies

    The way a user based in, say, Barcelona experiences an entertainment app might be quite different from the way one located in New York or Tokyo experiences the same app. The reason why is that the network hops over which data needs to travel, as well as the efficiency of the carrier network that delivers the data, which can have a major impact on how quickly content loads and is streamed. For media-rich entertainment apps, latency and bandwidth limitations can be particularly problematic.

    For this reason, you should be testing your entertainment apps from the perspective of users spread across the world. Doing so allows you to represent multiple geographies and carrier networks in your tests, ensuring that your app performs properly not just from the location where your testing team is located, but from all of the locations where your actual users reside.

    Test for Multiple Screen Resolutions

    Variations in screen resolution can have a significant impact on how content is rendered. Attempting to display content at a screen resolution that software developers didn’t plan for can result in pixelated images, distorted images or images that aren’t fully visible.

    That’s why you should be testing how content on your entertainment app is rendered under different screen resolutions. Testing for just one or a handful of resolutions isn’t enough.

    You can perform these tests manually, by reviewing content rendering by hand. A better approach, however, is to take advantage of AI tools that can automatically compare renderings of the same content under different screen resolutions to detect inconsistencies. That’s a much faster, more efficient, and more scalable way to ensure your content appears as users expect it to.

    Conclusion

    Entertainment apps are a unique category of software, and they require a unique testing strategy. Testing teams need to test for factors that aren’t important for most other types of apps, such as evaluating whether video and audio streams are in sync. They also need to minimize latency during testing and test across multiple geographic locations and screen resolutions. Practices like these are the only means of ensuring that entertainment apps deliver the user experience necessary to succeed in highly competitive niches.

  • Software Quality is the Heartbeat of the Best Organizations

    Software Quality is the Heartbeat of the Best Organizations

    JPMorgan recently announced it was hiring 2,000 engineers, despite the gloom in global economic markets. Is this not an odd risk for an organization to take, given the demand for (and cost of) software developers today? What’s happened?
    JPMorgan’s hiring drive is no mystery. Virtually every company in the world is leveraging software, with many transforming themselves into software organizations to drive new efficiencies, widen profit margins and create new markets. Software has been disruptive, so much so that the current era has been dubbed the “third industrial revolution.” Consumers are benefiting from switching providers at the click of a few buttons and most consume services through software. Bad software experiences today mean rapid customer churn and competitive disadvantage.

    Quality Should be the Heartbeat of any Organization

    In spite of quality’s importance, testing is still perceived as a secondary discipline and a check-the-box exercise. This reflects practices developed for assembly lines during the original industrial revolution. A product gets assembled and testers verify its quality as it leaves the production line. Testing’s goal in this context was to identify problems before they reach end-users. This same approach has been applied widely to software development and is still used by “waterfall” organizations.

    Yet, software development has been revolutionized by new methodologies and practices. Software today is constructed at incredible speed. Notice I’m using the word “constructed” and not “built”—the best developers leverage libraries or APIs to assemble software using existing components. They then add a few pieces of custom business logic to define the intellectual property of an application. Many organizations don’t even code software, instead customizing off-the-shelf software with vast configurations to modify functionality.

    The assembly-line approach to software development is no longer appropriate. When testing is left late, defects are found before a release or even in production. Release cycles are longer and products poorer quality. Yet, many organizations still stick religiously to this approach, with testing teams operating in distinct silos.

    The Role of the Tester is Evolving

    The best organizations in the world are changing how software is built. Testing is no longer a check-the-box exercise at the end of the assembly line. It is transforming into a discipline that revolves around evangelizing and driving quality throughout an organization.

    Quality should be baked in as early as possible, touching every artifact produced. Ideally, this should occur from the start, as a requirement is formulated, all the way through to the software being shipped. The idea of siloed teams working on different pieces is outdated. For quality to thrive, teams must collaborate.

    Modeling offers one approach to driving quality and collaboration throughout an organization. Its journey starts with a requirement. These are typically formulated by business analysts or product owners. Having been a product owner, I found that this usually involves lots of scribbles on paper before communicating to stakeholders on a whiteboard. The preliminary requirement then gets reviewed and refined.

    At some point, the requirement will be converted into walls of text in a project management tool. Modeling intervenes at this stage, adding visualizations that make requirements more consumable, less ambiguous and easier to understand.

    The word “modeling” may seem a scary concept, but it need be no more than a flowchart that encapsulates a requirement. The benefit of modeling is not only that it provides a digital collaboration asset; models can also generate coverage-focused tests before a line of code is written. Modeling creates requirements that are quality-driven and quality-driving.

    The Journey of Quality-Driven Requirements

    • Models Provide Data Upfront

    Developers today require rich data to develop, while testing is only as good as the available test data. Yet, many organizations still feed slices of production data into developers and test cases.
    This approach is problematic for several reasons. First, it often risks legislative non-compliance. Second, production data features large volumes of low-variant data. Data may not then be available when developing unusual-but-important scenarios. Imagine an overdrawn account with several beneficiaries: What should happen when a transfer is attempted?

    Modeling defines data explicitly in requirements. It allows engineers to understand a requirement explicitly, including the relevant data. From the modeled definitions, correct data can then be masked and subset, or generated synthetically. Data can further be delivered on-demand and at scale using database virtualization.

    • Models Provide Understanding

    Once the requirement and necessary data are available, code can be developed. Using text-based requirements, misinterpretations often lead to incorrect functionality. Visual models instead align closely with business logic, and we often see developers enriching models with additional context so that models align with the actual software implementation.

    • Models Generate In-Sprint Tests

    Once the code is developed, the same tests which were created earlier from the requirements can be executed against new functionality. This facilitates reuse and means the test cases are ready to be executed immediately.

    Some tests might further be selected for automation. In this instance, automation can be overlaid onto the requirements model, reusing code to generate scripts. These scripts then become part of the standard regression pack as new features are added to the product.

    A Change Request Comes In

    With iterative development, change requests are common. This typically creates havoc for quality, since it is difficult to understand the impact of a change, and time-consuming to update artifacts like tests and data.

    Traceability is key to overcoming this challenge. Models can connect to external DevOps tools to provide end-to-end traceability, from a requirement, all the way to code and tests. The impact of a change can then be identified directly. Models can be updated, automatically synchronizing test data, test cases, and automation to bring everything up to date.

    Testing for the Fourth Industrial Revolution

    Software is playing a vital role in virtually every organization, as already recognized by organizations thriving today. Baking quality in early is the only way to ensure quality software. The role of testing must change on a large scale. Models offer a way to achieve a “shift left” and collaborative approach to quality while testing earlier and more rigorously by automating and optimizing test and data creation.

  • Finding Value in AI-Augmented DevOps

    Finding Value in AI-Augmented DevOps

    While artificial intelligence (AI) and machine learning (ML) are emerging technologies, we know they can help an organization parse large data sets and glean actionable insights. But do AI-infused processes actually make a difference in organizations that employ DevOps? The answer is explored in a recent survey by Tricentis. Working in partnership with Techstrong Group, Tricentis surveyed more than 2,600 DevOps practitioners across the globe about their AI-augmented DevOps processes. The results pointed to a bright future in DevOps.

    In this survey, the majority of respondents cited they’re already getting value from AI-powered DevOps processes. A third of survey respondents found AI-augmented DevOps extremely useful today and nearly half found it very useful. Although the technology is still relatively young, a 2021 report by IT infrastructure automation company Puppet Labs found that 83% of organizations are implementing DevOps processes. This rapid adoption speaks to the promise of the technology.

    Interestingly, nearly two-thirds of respondents cited that testing is the area within DevOps where AI will have the most impact. Testing is a major pain point for DevOps organizations because a scaled DevOps environment calls for complex testing scenarios and large amounts of data, and many organizations struggle to scale their automation to the level DevOps requires. Without automation, manual testing can significantly delay releases. In AI-infused DevOps processes, AI can help accelerate the authoring of tests, help testers understand where the risks are, and repair broken tests.

    AI’s Greatest Potential is in Functional Testing

    We typically think of software testing as either functional, where the program enables users to properly execute tasks, or non-functional, where the application is secure, fast and stable. Respondents were asked which disciplines of testing stood to gain most from AI and 65% of them indicated functional testing. Unit testing and UI testing were the sub-categories most often selected.

    The complexity of the functional testing process is where AI shines. Functional testing is open-ended and many permutations are required to ensure proper test coverage. AI steps in and manages those permutations, parsing large amounts of data and providing insights. Additionally, AI can help wrangle production signals that aren’t discovered in a test environment.

    Throughout the survey, respondents specifically noted AI’s promise in regard to UI testing. UI testing is perhaps the most time-intensive and manual type of testing in most organizations. With the many variations of the user experience paths that exist, AI can help mimic the actions of a real end-user, adjusting for application nuances or variations in experience. AI and ML can also boost your build process, increasing the efficiency of repetitive tasks. The ability to quickly identify and resolve issues can improve the effectiveness of your automation.

    While survey respondents see the most potential in testing, they acknowledge that AI-powered DevOps can address other business challenges. More than 40% claim that these processes can reduce the skills gap and empower junior staff to perform complex tasks, improve the customer experience, cut costs, boost innovation and increase efficiency among developers. Nearly 40% also believe that AI-powered DevOps delivers leadership the insights they need for continuous improvement.

    Automation is the Key to Scaling DevOps

    It’s worth noting that among the DevOps organizations that see benefits from AI and ML, most of them are mature organizations; they employ DevOps workflow pipelines, toolchains, automation and cloud technologies, and they’ve automated more than half of their testing. Only 21% of practitioners who are just dipping their toes into DevOps have achieved this level of automation. And even in mature organizations, testing is still the main bottleneck; only 40% of them have automated more than half of their testing.

    The key to scaling DevOps practices is developing test automation that can keep up with an increasing volume and speed of releases. But the road to DevOps isn’t always easy. Practitioners cited obstacles like a lack of tech skills (44%), insufficient budget (25%) and tool selection (19%). Organizations that want to implement DevOps need to rely on external vendors and internal resources to address these challenges and effectively scale.

    Looking to Adapt More DevOps Practices?

    So where does an organization that wants to implement DevOps start? With the big questions. It’s crucial to identify the specific business problem that you’d like to solve. What’s the gain you’re looking for? Where in the software life cycle do you expect to find a benefit? AI and ML can add business value, but they don’t solve every problem across the life cycle.

    After an organization has identified its DevOps goals, its focus should shift to automation. To reap the benefits of AI-powered testing, an organization should automate repetitive functional testing, especially unit and regression testing. Most organizations start by implementing continuous integration and continuous delivery. Once an organization has taken these steps, it will start to amass a large cache of results and artifacts generated from running the test cases, which AI can mine to increase test stability and identify recurring problems. The more you automate, the more you’ll get out of it and the faster you’ll accelerate your delivery.

    Organizations just starting out with DevOps should also understand the distinction between artificial intelligence and ML. Most people refer to ML as AI, but they’re not interchangeable, and most organizations haven’t advanced to the use of AI. A true AI is broadly powerful, while ML technologies are narrow and solve a specific domain problem that a model was trained for.

    When you’re evaluating potential vendors, there are a number of key questions to ask, including:

    • What specific AI technology is the vendor using? Is it an off-the-shelf library or something they built themselves?

    • How are they using AI? Which algorithms are in play and what approach do they take?

    • How do they train their models? Different types of patterns emerge in data sets depending on the industry, the frequency with which data is updated and the application. The data population that the vendor is training on must look like your production data, or the model could take you in the wrong direction.

    • How do they ensure that bias doesn’t come into play? If they train models in a different domain or vertical, for example, you could be an outlier for that vendor.

    • How do they handle failure? How do they monitor for miscategorization and misapplication of the AI in a way that lets you trust the data?

    • Will your data be used to train and improve your learning models? If so, does that raise any privacy concerns? Expectations around data privacy should be explicit and leave no questions.

    • Are there other customers in your vertical with a similar business profile that are getting the benefits you’re looking for? Can the vendor provide examples?

    What Does AI-Infused DevOps Look Like in Practice?

    Consider a financial services organization that has more than 20 applications to manage. With so many applications, it’s necessary to conduct significant UI testing and the tests must be stable. One of the biggest challenges to UI testing lies in locating various web elements; most people start with manual script-based automation, but it can be challenging and time-consuming to create and maintain. Employing AI and smart locators to identify these elements can save time and drastically cut down on UI testing maintenance.

    Survey respondents were asked how they’re currently using AI to augment their testing processes. Thirty-seven percent say they’re accelerating the creation of their automated test cases, while 44% focus their testing on the areas of highest risk. Almost half are able to reduce test case maintenance with self-healing and 43% use AI to identify the root cause of failed tests. Thirty-one percent claim that AI offers insights into test process improvements and 34% use AI to help identify tests to run based on changes within the application.

    Where is AI-Powered DevOps Going Next?

    Going forward, AI will become more and more embedded in testing processes. Great progress with off-the-shelf vendors is being made, and their products are a lot more accessible than they used to be. As the tooling continues to mature, these products will reduce the need for specialized skills. Internal initiatives like data lakes or data warehouses will persist, but there will be less need to build large data science teams to get the advantages of AI.

  • Playwright: A Modern, Open Source Approach to End-To-End Testing

    Playwright: A Modern, Open Source Approach to End-To-End Testing

    I was excited when I started writing my first end-to-end tests years ago. The idea was promising; create an automated test suite that spins up a browser and mimics your user’s behavior. Test your website end-to-end. Frontend, backend, your database—you name it. All these automated tests would cover everything! Unfortunately, it wasn’t all roses and sunshine. The implementation was more complicated than expected and the results were not as stable as I had hoped. On one hand, I expected that learning a new testing framework would take time. But on the other hand, investing time and money just to end up with flaky tests and countless false positive alerts was not acceptable.

    Luckily, testing frameworks evolved, and new libraries make end-to-end testing more approachable and, most importantly, less flaky. One of the more recent solutions is Playwright. Let me tell you more about it!

    Write Your Test Code in Four Different Languages

    While Playwright’s default choice is Node.js, it offers support for three other languages: Java, .NET and Python. You’re in good hands if your team uses one of these languages because no one wants to learn a new language just to write some tests, right?

    Cross-Browser Tests on the Back of the Chrome DevTools Protocol

    Contrary to other solutions, Playwright doesn’t use the WebDriver protocol. It leverages the Chrome DevTools protocol to communicate with Chromium browsers (Chrome/Edge) directly. This approach allows for more direct and quicker communication. Your tests will be more powerful and less flaky.

    But Playwright doesn’t stop at Chromium browsers. The team behind the project understood that cross-browser tests are essential for an end-to-end testing solution. They’re heavily invested in providing a seamless experience for Safari and Firefox, as well, and even Android WebView compatibility is in the works.

    Testing your sites in Chrome, Edge, Firefox and Safari is only a configuration matter. And this saves time and headaches!

    Fast Tests Due to Auto-Waiting

    It’s not only about automating multiple browsers, though. If your tests are hard to write because you have to place countless “sleep” statements everywhere, your test suite will take hours to complete and become a burden.

    To avoid unnecessary waits, Playwright comes with “auto-waiting.” The idea is simple: Instead of figuring out when a button is clickable by yourself, Playwright performs actionability tests for you.

    Let’s look at an example:

    “`
    await page.click(‘[data-test=cta]’)
    “`

    There’s nothing fancy about clicking a button, right? Except in Playwright, there is. This innocent “click” call includes additional convenience functionality. When you call “click,” the framework checks and waits for a matching DOM element to be:

    – attached to the DOM
    – visible
    – stable (it’s not animating or moving)
    – able to receive events (not hidden or obscured by other elements)
    – enabled

    All these checks guarantee that your target element is ready to roll when you want to interact with it and, more importantly, it’s clicked as quickly as possible. Don’t worry about DOM elements. Just interact with them and let Playwright figure out the rest—no more wait statements!

    Auto-waiting is built into all of Playwright’s interaction methods so that you don’t slow down your test suite. Your tests will be easier to write and run as quickly as possible.

    But that’s not all! While I use Playwright via the command line, it also comes with handy tools to make end-to-end testing easier.

    Valuable Tools to Create, Debug and Analyze

    Playwright provides additional commands and visual tools to make the writing of end-to-end tests as accessible as possible.

    Record Scripts With Codegen

    To give you a head start with your first test, use Playwright’s test generator.

    “`
    npx playwright codegen playwright.dev
    “`

    Only a single CLI command is necessary to spin up two new windows: An interactive browser and the Playwright Inspector. The fun part is that every interaction in the browser window will be recorded and shown in the inspector.

    If you want to create a Playwright test that navigates somewhere, clicks a button and fills out a form, these tools are perfect for creating an initial setup.

    Playwright library

    Replay the generated script, adjust your selectors and assertions, and you’re good to go!

    Congratulations! You just created your first end-to-end test, and I bet it didn’t take you more than a few minutes to get there.

    Debug Your Scripts While Running Them

    When your test suite grows, you’ll find yourself in situations where you must debug complex scripts. For these cases, Playwright provides a handy debug view.

    Enable debug mode by explicitly calling `test –debug` or defining the environment variable `PWDEBUG=1`.

    “`
    npx playwright test –debug
    PWDEBUG=1 npm run test
    “`

    Playwright then opens a browser and the inspector walks you through your test case, similar to your editor’s debugger. Additionally, it highlights the touched elements with a little red circle so you can see what was clicked or filled.

    Playwright debugger

    This approach is perfect for debugging tests during local development. But what about debugging failed tests that run in CI or on a different machine? There’s a solution for that, too!

    Analyze and Debug Previously Failed Tests

    In an ideal scenario, you’ll run your tests automatically on commit or deployment in your CI/CD pipeline. But debugging failed tests from a different environment can be a massive challenge.

    Luckily, the Playwright framework allows you to enable a trace mode. Trace mode records all script actions while screenshotting and keeping the website’s state.

    Trace view

    Then, if your tests fail on your remote servers, use the provided Playwright tooling to load a generated `trace.zip` file and inspect the test to find out what failed. Method calls, a timeline and screenshots will be available—it’s time travel debugging at your service!

    A Refreshing Approach to Synthetic Monitoring and Testing

    These are only a few things that make Playwright my testing framework of choice today and the team behind it develops fast and ships new releases every month.

    If you want to learn more about the Playwright framework or to chat with the maintainers, there’s a very active community Slack channel with many friendly people.

    So, do you still avoid end-to-end tests because they used to be slow and flaky? I hope that I could convince you to give it another try. Because there’s nothing better than knowing that you didn’t break things while developing and shipping software.

  • Plutora Releases Test Environment QuickStart Bundle – Techstrong TV

    Plutora Releases Test Environment QuickStart Bundle – Techstrong TV

    Bob talks about Plutora’s new “Test Environment QuickStart Bundle”. It provides an easy, immediate fix to the Achilles’ heel of software development: environments that cause delays, produce bugs, and waste time and effort. The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Welcome to another Tech Strong TV segment. I’m really happy to be joined today – I don’t know if he’s actually been on Tech Strong TV with us, but I’ve spoken to him many times; it’s my friend Bob Davis from Plutora.

    Hey, Bob, how are you? 

    Bob Davis: I am great, Alan. It’s so good to be here. I haven’t had the pleasure of being on the Tech Strong TV spotlight and it’s great to be here.

    Shimel: Fantastic to have you. So, Bob, our audience has been fed a steady diet of Plutora for years already, right? So most of them know Plutora, one of the leaders in the value stream management space, a DevOps pioneer leader. But I don’t know if they really know you. I’d like to introduce them to Bob Davis. 

    So why don’t we start there? Give them a little bit of your history, Bob.

    Davis: Sure thing. Alan, I’ve got a few years under my belt, and I started in the semiconductor business back at Intel as a chip designer. And I ultimately said, “Gee, you know, I think I understand why I want to build things bigger, better, faster and cheaper, but I don’t understand the business.” So I went and got an MBA and I moved into more of a product marketing orientation, and from there became a serial entrepreneur. 

    This is, Plutora is my tenth startup. I never joined another large company after I joined Intel, but I found my way into some through acquisitions and so forth: Novell, CA. Always on the software side, always on the, you know, typically the infrastructure, you know, a lot of enterprise IT, networking, compression, security, a lot of that, and ultimately back up in recovery and continuous data protection and those kinds of things, which ultimately led me to really understanding, wanting to understand more about the developer side. 

    And so about five or six years ago I found my way into Plutora and I became chief marketing officer and away I went. Along the way I became a pilot, I play golf, and love the water. So I’ve got a lot of things. And six grandchildren, so good stuff.

    Shimel: Six children. God bless you, man. Whoo. You know, I’ll tell you off camera; I don’t even want to get into it now.

    But anyway, that’s great, man. It’s great to have you here. And it’s interesting, ’cause you know what, a lot of people will hear “Bob Davis, CMO Plutora” and say, “Ah, he’s a marketing guy.” Well, yeah, but you’re a marketing guy who started off designing chips at Intel when a lot of people watching this were playing computer games, not even computer games, Show & Tells or whatever the heck they were. Right?

    Davis: Yeah. Well, I’ve always been on the product side and I’ve always been more on the “What’s the next idea?” In fact, the best example of that was right around 2000 – you know, we started Kaseya. 

    Shimel: Oh, I remember that.

    Davis: So I was the original founder of Kaseya and started that in my kitchen and it’s …

    Shimel: Really?

    Davis: Yeah. To being what it is today. And there wasn’t even a managed service provider market at the time.

    Shimel: No, I remember that early on. Because I had started a company called StillSecure and in 2005 we pivoted into MSSP. ‘Cause I just thought MSSP was the way to go. It was too hard for too many companies; they needed an MSSP. Actually Kaseya bought friends of mine, Carl and Billy Austin, they’re serial entrepreneurs as well. They were acquired by Kaseya. And then of course Kaseya was acquired, what was it, a year or two ago maybe? 

    Davis: Yeah, Inside Venture Partners bought Kaseya in 2013, which is when I left actually.

    Shimel: Right. Oh, okay.

    Davis: That’s when the new management came in. And we sold the company and kind of went off on our own journeys. I went to a company called Atlantis, which we were doing hyper-converged storage, and then ultimately Dalibor Siroky, CEO of Plutora, found me and brought me over to Plutora in 2017. 

    Shimel: Very cool. So, Bob, we were talking off camera a bit. We’re at an interesting sort of boundary layer here again. I think many of us are let’s call it cautiously optimistic that we are going to emerge from our kind of COVID cocoons and venture out again en masse. 

    Davis: Yes.

    Shimel: Many of us are already.

    Davis: Yes. 

    Shimel: The interesting thing is the world we left two years ago is a different place today than it was two years ago. We’ve learned a lot of things, we’ve experienced a lot of things. Digital transformation – no pun intended – has transformed many a business – many a business model. For us in the tech industry, for our audience, you know, the way they shop for products, try products, use products, measure flow in terms of value stream and stuff like that, it’s really changed in two years. They’re different. You can’t put your foot in the same water twice, right? And so companies like Plutora – someone once said that, I’m pretty sure. 

    But companies like Plutora, they’ve had a change. You’ve had to adapt. And you guys have. You have that sort of a whole new program you’re coming out in market with. Let’s talk about it.

    Davis: You bet. You know, in the beginning, right in the beginning of the pandemic, early pandemic, early 2020 we saw as the world was trying to find its axis again we saw an incredible increase in people wanting to learn. So they spent the first part of that 2020 just in this learning mode. And value stream management was one of the things that people were really focused on learning. And the notion of being able to say “Wow, if I could really see a pipeline, a value stream from the beginning to the end and understand if I’m getting better.” And that was really the question that the CIOs were asking, “I’ve spent millions of dollars on these tools. Am I getting better?”

    The point I’m making is that everybody was trying to understand how they could tell if they were getting better. And all these flow things became almost an MBA project or a PhD-type project to really understand all that’s possible. And that was awesome and we got a lot of interesting traction from that. 

    What we found, however, is that that transformation is a daunting transformation and the world changed around everybody. We went from distributed teams to distributed teammates. So now that autonomous team was distributed in and of itself, and the communication and collaboration became very difficult to achieve. So processes, organizations were changing. So many things changing all at once that people were really slowly doing like they’d be happy to go to an event, sit on a virtual show. What do I need to understand to get better?

    That’s not what’s happening today. Some of it, sure. Some of it still happens. But people are moving to more execution. They want to get value. They want to start seeing the value of that journey and they want to see the value in real-time. They don’t want to see it a year from now; they want to see it right now, this year. So we have seen our customers struggle with saying “How can I get started in value stream management? I may not even want to go all the way to the end of the value stream management journey, depending upon my specific tastes.”

    So we’ve developed an entire program around doing just that. How can we get people into a value stream process mentality, into a value stream platform, and do so quickly so that they can value right away. And so we have this program called the Quick Start Program and we released in January the first of several Quick Start products. This one is on environment management, which is a core component of value stream platform that Plutora sells. And in so doing we’re able to provide people solutions to a very complex test environment problem in three or four weeks, whereas today it can take 30 days just to configure and make an environment available on some of the prospects that we’re talking to.

    So in the time it takes to get an existing manually configured test environment ready for a test event you can have a full test environment management program and go from manual management of test environments to an automated process very quickly. Now I see value. Now I can see value in terms of the efficiency of utilization of expense and test environments. I can see better quality. I can see saving tons of money. 

    We find that people with average levels of releases, well UMA did a study and they came out and said the average number of releases in an enterprise environment was 76 a year. That was in a survey. They did a survey and they came back and said that was the average. There was certainly a distribution, but that was the average. And if you’re manually managing your test environments you’re losing money. They determined based on all the issues of managing these things almost $1.5 million a year you’re wasting. 

    And so we can save you that with a product like this. So imagine I’ve got a bottleneck in my DevOps tool chain, the Achilles’ heel of my tool chain is waiting for test environments as I shift left and try to test earlier. I solved that problem, I removed the wait states there, I become more efficient, I save money, and I can do that in about three to four weeks worth of setup. And now I’m rolling. Wow, I’m getting value. Now I go to the next problem and so forth. And I get quick starts, I can come into the platform pretty soon. I’ve got a full value stream management platform down the road. I’ve gotten value every step of the way and I never had to really think about the prospect of doing the whole thing. It’s like one step at a time; every journey is one step. So it’s like let’s give people the ability to get value in that same way, one step at a time. 

    And that’s really, it’s been met with tremendous success already. We’ve seen a lot of responses, a lot of people wanting more information, and our SDR team is very busy fielding calls on it. So it’s been a real successful launch so far.

    Shimel: That’s great. You know, Bob, I think part of the reason for its success is there’s two kind of trends you’re riding there. First of all it’s this, I call it “deliver delight,” but I mean that’s just two words, right? It goes to when I was an entrepreneur founder of companies we were hiring enterprise salespeople who sold to the C-levels and all that stuff. In today’s world we hire DevRel evangelists, and the idea is get the software into the hands of the people who are going to use it, deliver delight, and if it works they’ll tell their bosses and so forth to buy it. So it’s a different sales model. But it’s become dominant, especially in DevOps cloud native, you know, the kind of markets we play in. So this really is kind of playing into that. 

    Davis: Totally.

    Shimel: It’s delivering delight. Because synonymous for delivering delight is delivering value. 

    Davis: Yes. Well, that’s the point, right? Now it’s the whole point of value stream management in the first place. You know, it’s interesting; it’s really an agile approach to delivering the value stream management solution. There’s a parallel there. You know, saying “I’m going to buy a value stream management platform and implement it” is a little like deciding I’m going to build a waterfall process. ‘Cause it’s everything at once, this is what I want to do.

    Shimel: No, it’s like trying to eat an elephant.

    Davis: Exactly. 

    Shimel: You’ve got to eat it one bite at a time. [Laughs] That’s the only way you can eat it.

    Davis: Right. Well, you’re crossing – you know, by the very definition value stream management embraces both, you know, everything from the ops person on the far right to the strategic planning office on the far left, and so you’ve got lots of people, you’ve got lots of moving parts. It’s hard. We actually have found hard when we market the product we get the interest from the pain point, I’m having trouble. In this case I’m having trouble managing my environment. If you’re selling the entire platform that environ manager, in order to take advantage of that feature, has to sell the platform to the planning person, to the ops person, ’cause they’re all going to get affected. 

    There’s value there, but they’re like, “Show me.” Now we’re giving them a tool they can do that. It’s an on ramp into the platform, a smaller bite of the apple, as you said, an agile approach to delivering value stream management and value at the pain point level. And so we’re doing everything around it to make it super-clear if you have this pain, this is what it looks like in our product when you solve it, and you can get there by the end of the month. Does that sound reasonable? And it’s way cheaper.

    And so that’s the other thing, these platforms can be expensive. But what if I can buy something that’s in my sign-off authority, so it’s more approachable to me? The whole notion of S-a-a-S is all about product life growth and let me get something in my hands for free and try it out and then I can grow from there. That’s the beauty of what so many people have enjoyed with S-a-a-S-style software products. 

    Value stream management is hard in that sense, ’cause it’s complicated. There’s integrations into the tool chain, there’s process changes as people evolve from more legacy processing to DevOps tool chains and DevSecOps tool chains and so forth. There’s that. There’s organizational change, budget-holder changes, people moving from silos and project-oriented things to product-oriented things. So we’ve run into prospects that get so frustrated, they want the solution but there’s a lot going on in the organization. We’re answering that question.

    You can each have had a success, show the value, and go from there. It’s a great on-ramp. It’s a freemium approach almost in a complicated environment and its value stream management, and we think it’s really going to be awesome when people can have multiple. We’ll do probably deployment, we’ll have a quick start for master deployment plans. You know, Jeff was on in November talking to you about our progressive release management and some of the new stuff that we did with the Insights dashboard. That’s another screaming for of Quick Start, “Hey, wouldn’t it be cool if you could get this up and running very quickly?” 

    So all of those things is where we’re headed. We started with environment management and it was made available on the 18th of January and it’s already in the market, so we’re excited about it.

    Shimel: So no pun intended, but where can we get a quick start on this Quick Start? [Laughs] Where can we go, Bob? 

    Davis: Well, we have, there are online demos. You know, there are all kinds of spots on our website. We have a spot where you can say “Talk to me and you can sign up for that and have it by the end of the month.”

    Shimel: So just go to Plutra.com and check out the stuff for Quick Starts. 

    Davis: Yep. There’s a banner right on the top of the home page that will take you right to the information on it.

    Shimel: Love it. Good stuff.

    Davis: Yep. It’s all right there.

    Shimel: Hey, Bob, we’re about of time. I want to thank you for coming on and telling us about the Plutora Quick Start program. You know, the first one, as you mentioned, came out the 18th. Any timeframe on subsequent quick starts?

    Davis: We’re going to bring out the next one in the beginning of the next quarter, so right around April. 

    Shimel: Excellent, man.

    Davis: So you heard it here first, Alan. 

    Shimel: Well, that’s why we’re here. Thank you for letting us know. We’ll be watching. Good luck with this one. Maybe we’ll see you show up on one of the Plutora webinar workshops that you guys do with us so often. Love to have that.

    Davis: We’re going to do that.

    Shimel: Cool stuff. All right. Bob Davis, CMO Plutora here on Tech Strong TV. We’re going to take a break right now. We’ll be right back with our next guest. 

  • Techstrong TV: Scratching the Surface of Testing Through AI

    Techstrong TV: Scratching the Surface of Testing Through AI

    Mark, Torsten and Alan discuss the findings of the “Disrupting the Economics of Software Testing Through AI” report. The report identifies critical factors that hinder software engineering and DevOps teams including the escalating costs of quality control, and growing complexity associated with the increasing release velocity, number of smart devices, operating systems, and programming languages. The video and a transcript of the conversation are below.

    Announcer:                 This is Digital Anarchist.

     

    Interviewer:                Hey, everyone. Welcome to another segment here on Techstrong TV. I am really happy to tell you, introduce our guests for this next segment. First of all, he’s a returning guest with us. He’s from our friends at Applitools. It’s Mark Lambert. Hey, Mark. How are you, man?

     

    Interviewee 1:             Fantastic. Thank you, Alan. Great to have you – for you to have me on. Yeah, great to talk to you.

     

    Interviewer:                Absolutely. And then joining Mark and I today is Torsten Volk. Torsten is – I just drew a blank – EMA, right? Enterprise Management Associates?

     

    Interviewee 2:             Yes, exactly.

     

    Interviewer:                Yep. And we were talking off camera. Torsten is actually a friend of our friend, Andi Mann, who he knew or was his predecessor at EMA 11 years ago.

     

    Interviewee 2:             Yeah.

     

    Interviewer:                And, Torsten, welcome to Techstrong TV. It’s great to have you on.

     

    Interviewee 2:             Thank you very much. It’s awesome to be here.

     

    Interviewer:                Thank you. So, guys, look, Applitools, leader in visual testing. We’ve had, beyond Mark we’ve had many Applitools people on over the last almost two years now of doing Techstrong TV, so I think our audience is familiar. But just in case people aren’t, Mark, how about a little quick Applitools backgrounder?

     

    Interviewee 1:             Sure, yeah. So Applitools is known for inventing Visual AI, so helping software development teams accelerate their releases by mimicking the human eye and brain and how we as humans analyze computer screens, highlighting differences and issues within your UI so you can do not only visual testing, but also fundamentally change and modernize your functional test automation. So it’s really the glue that plugs right into your existing practices, so Selenium, Cypress, we support over 50 SDKs. There’s a free account as well, so you can get in there and get going, kind of feel the water, so to speak. But what we’re seeing is that it’s helping leading brands accelerate their digital experiences, and really it ties directly into the research obviously that Torsten’s been doing around AI and how that can help organizations disrupt the economics of software testing.

     

    Interviewer:                Love it. And, Torsten, look, I know EMA from before 2010 even, but for those folks who maybe, you know, a lot of our audience are technical people. They may not necessarily follow the analyst beat. Why don’t you give them a little EMA background?

     

    Interviewee 2:             My boss describes it always as the industry and analyst firm for enterprise grade solutions, and in the beginning – I’ve been here for a while. In the beginning I was always a little bit wondering, you know, what does that really mean? And what it at the end of the day means is we have all of those emerging technologies coming up, especially in the area of artificial intelligence and cloud native applications and intelligent automation, and what we are looking at is really how can we leverage those capabilities within an enterprise setting. So once the dust has settled and we’ve all gorged ourselves, you know, by consuming the leading edge of all of those tools, the question then is how do I expand that footprint of those tools across the enterprise so that the enterprise as a whole is more competitive in the end. So basically projecting those constraints of all of those cool new products and technologies into the enterprise, that is really what EMA is all about. And my specific practice is focused, as you can see in the background here, as one of my research graphics is focused on looking at machine learning, deep learning, reinforcement learning, and how those capabilities can enhance the enterprise.

     

    Interviewer:                I get it. Great use of the background, Torsten. Thank you. All right. So, Mark, we’ve got all that out of the way. Let’s jump into what we really want to talk on today, and I’m gonna let you kick it off if it’s okay.

     

    Interviewee 1:             Yeah, I’d love to. Thank you. So yeah, so recent report from EMA, and Torsten is the author and I’ll kind of throw the ball to him very quickly, obviously, but really talking about how artificial intelligence and machine learning is revolutionizing the software testing space, and is really important right now because there’s so much buzz around AI, machine learning, and kind of where it comes in. You want to understand where does the rubber meet the road? What is impactful today? What’s impactful for the future? And then what’s, what are the drivers that are getting us there? So I mean, Torsten, I’ll turn over to you, and maybe kind of like talk about some of the capabilities that you talk about in the report and the challenges that they, that you’ve seen come into play.

     

    Interviewee 2:             We started that report by looking at the key pain points, like we start almost all of our reports. Right? What is not sustainable anymore in a specific discipline, and in this case it’s test engineering. And we have pressure factors. One of them is that we, the business needs us to release software a lot more often, and in a really granular manner where somebody’s asking for a feature capability and the software development team is expected to release that thing very quickly, ideally in an ongoing manner. So that is one part, and the second part is that the end users are very, have become very much more diverse than they even were in the recent past, where we have multiple charts where we show the escalation of the number of devices with different screens, different processors, a different network connection, different interfaces for your fingers and for voice control. And all of those things make testing much, much more difficult, and one of my favorite parts always is the little war stories that you get told. Right?

    So in the beginning when we look at al the material that we have, we talk to a lot of developers and test engineers and DevOps guys, and my war stories or their war stories always started with, “Oh, yeah, regression testing, Torsten. That is really not doable anymore. This is kind of a dreamland that you are living in as an analyst if you think that we can regression test all of those releases. It’s just too much, and it would take forever to be able to do that.” So if we can’t regression test, we are running a significant risk, Mark, that something goes awry. Right? We always are sure that in theory things should work, but in real life there are combinations of technologies, of code functions, that they then don’t work. And that’s where we start with artificial intelligence and this whole pair of eyes, Mark, that you are describing, right, where you say, “Oh, yeah, my test engineers don’t have to be that pair of eyes that check through dozens of boxes and just say yes, yes, yes, yes, yes everywhere, and then also overlook some things.

    But that pair of eyes is something more. It’s something that has a certain degree of – when I say understanding, I think I’m gonna embarrass myself, but it has a certain degree of ability to comprehend – maybe that’s the same word – but to pick up things of a similar character and to categorize and cluster things in a manner that makes sense for human beings then to review, Mark. Right? And that is –

     

    Interviewee 1:             Yeah.

     

    Interviewee 2:             – where a tool like Applitools is very interesting, and also unique in the marketplace, right, where it’s a pair of eyes that you guys are building.

     

    Interviewee 1:             Yeah, and I think that talks – I mean your research touches on several different areas where AI comes in, and visual inspection, which is obviously where Applitools fits, is really a highly impactful and the most impactful today of those techniques because it does so many different things. So first of all, obviously there’s the streamlining of the manual review effort. Right? So I’ve got this explosion of devices. How do I check them all? Well, because of the high level of accuracy of Applitools Visual AI, and it’s been trained on over a billion images and has got four nines of accuracy, so what that means is as a human I don’t have to review tens or hundreds of screenshots. I’m just focused in on a very small subset. But that’s just the first part of it, because then you layer on top of that and this explosion of release velocity, increasing device combinations, and as well as the, just the complexity of the application and so many things out of your control, you need help in reviewing the volumes of data and really saying, okay, all of these things kind of look like the same.

    So and then bringing them together. I talk about basically sorting your visual review into different stacks of paper, so for example I might have 76 screens that have a visual defect, but I can break those down into two categories of 40 and 36. And those types of techniques go beyond just the basic pixel dipping approach, which is the one that would be a traditional visual testing technology, and really the ability to categorize that you talk about, Torsten, is the key driver that, or the key foundation that all of these things get built on top of. So it’s a really powerful technique, and as I said earlier plugs right into existing, so you don’t have to rip and replace. That’s one of the things that a lot of AI technology today is say, okay, it’s great, but you have to build it all new in this track. The thing that is really powerful with Applitools is we plug right into your existing frameworks, open source as well as third-party commercial tools, and then we elevate the process with the dashboard to really enable cross-team collaboration. So it’s not just your testers or your test automation or developers that are engaged. Your domain experts, even other parts of the team like UI, UX design, for example, get to be involved in that collaboration process.

     

    Interviewee 2:             Yes, and we see often that communication is key, and it’s not happening in testing where we have those more and more decentralized product teams that are doing their own thing with their own toolkits and pipelines, but they’re not really benefiting from what other teams are finding. And so coming back to that horrible list approach, right, where you have, you check those boxes or you don’t check them. Right? So it’s a very monotonous affair as a human being to review this, but then once you’ve reviewed it usually it comes back to you again and again, and then it comes back and hits other people in that organization because there is no communication and there’s no unified platform to retain that knowledge. Right? And with reinforcement learning, you can teach the system what it needs to come back to a human with, and then what it shouldn’t. So if one team makes decisions, all the other teams are benefiting from that and it’s less and less of this toil type of task where you just also as a human tend to make mistakes and it’s just not scalable. Right? So that whole scalability aspect is an interesting one that deep learning and reinforcement learning brings us there, Mark.

     

    Interviewee 1:             Yeah, you hit on the key things. Right? You’re scaling to meet the current demands of release velocity, shorter release cycles. But the other thing I always think about when I think about AI machine learning and how it can help us from a software quality perspective, it’s an assistive technology so you’re reducing or removing mundane and repetitive tasks, which in themselves are error prone. Humans are great at creative work. We’re terrible at boring, monotonous things  because we miss stuff. Whereas leveraging AI, and especially Applitools Visual AI, it’s highlighting those things. So that same example I talked about, 76 screens, you’re gonna miss stuff. If you’re looking at them all and you’re like, oh, it’s the same. It’s the same. It’s the same. But the Visual AI will identify key differences between the 76 images to really kind of like focus and improve the efforts.

     

    Interviewee 2:             Yeah. And that gets even when you look a little bit into the future, Mark. I know that you can’t share roadmap items, but that whole idea of having a Visual AI can go even beyond just user interface testing, right, where you can do sanity checks in certain areas, where you say, “Oh, yeah, no, that number can never be a billion because of what I know about the data in the background.” Right? And a lot of the things that come through that we continuously see coming through when you release and then you don’t want to be the one in the test team to hold off a, to hold back a release in the middle of the night.

    It’s funny that when I was young and had a ton of hair still, that was what happened, and it’s still happening today just as much, right, where people think, “Oh, no. I shouldn’t be the one who says that. I think it’s all fine. Let’s sign off on that.” Right? So the release data kind of getting mushy and mushy and mushier in those areas, and that ability to say, “Oh, yeah, I had somebody look over it, and something horrible had happened,” some compliance violation or some crazy numbers in the shopping cart that can never add up. Can be even simple things where it’s, oh, this cost a billion dollars, or it’s euros instead of dollars but we are in a certain country. Those are things that you can catch more and more.

     

    Interviewer:                Fair. Guys, I feel like you didn’t let me, I didn’t get much of any words in here –

     

    Interviewee 1:             We feel the same way.

     

    Interviewer:                – but it’s okay. You know, I would – but I try to be a good host and let the guests talk, and you guys had it. Right? This is something, Torsten, you’re obviously very, very passionate and involved in and, Mark, it’s kind of, it’s your life. Right? And so I let it go. However, we are just about out of time, believe it or not, and we went on here for 15-plus minutes on this. But –

     

    Interviewee 1:             Can I just let people know where they can get the report? Would that –

     

    Interviewer:                That’s what I was going to ask you.

     

    Interviewee 1:             Sorry, I just wanted to make – yeah. So –

     

    Interviewer:                Go ahead.

     

    Interviewee 1:             – yeah, go to applitools.com/ema and you’ll be able to download the full report and access all the information there. So yeah, thank you, Alan.

     

    Interviewer:                Thank you, Mark. So it’s applitools.com/ema.

     

    Interviewee 1:             Yes.

     

    Interviewer:                Which also happens to be where Torsten works, at EMA, Enterprise Management Associates. Guys, it sounds like it was a really worthwhile research project here with some really good results and really good findings and insights. Look, this whole, there’s a reason Applitools has been as successful as it has. The whole idea of AI and testing is like peanut butter and chocolate in some way, and we can – but we barely scratched the surface in a lot of ways, right? There’s so much more yet to be done, and looking forward to hearing more about it in the future.

     

    Interviewee 1:             Yeah.

     

    Interviewer:                All right.

     

    Interviewee 1:             Thank you, Alan. Look forward to talking again soon.

     

    Interviewee 2:             Thank you.

     

    Interviewer:                All righty. Excellent. Torsten Volk, EMA, of course Mark Lambert from Applitools here on Techstrong TV. We’re gonna take a break and we’ll be right back with our next guest.

     

    [End of Audio]

  • How Automated Testing Closes the Development and Delivery Gap

    How Automated Testing Closes the Development and Delivery Gap

    For the past two years, companies have leaned into digital transformation projects to keep their digital offerings at pace with their in-person operations. Rather than become a stopgap, this digital approach became the new standard for the modern employee and customer experience. It also creates friction between a company’s development team and its user base. How long should customers wait for new or updated applications in our increasingly fast-paced business environment? Companies need time for development and testing to deliver quality products, but will the demands of their users force them to cut corners?

    Consider each stage of a customer’s journey. Visiting a website or app and putting something into the shopping cart is only the beginning. Behind the scenes, systems that monitor availability, sourcing, loyalty and promotions come into play. Following checkout and payment, there are systems for picking the product, packing it and delivering it with timely notifications along the way.

    Businesses find themselves in a bind when they have to decide which of those areas to cut back on. There’s really no good answer—enter the continuous delivery mindset. We like to say there is no continuous delivery without continuous testing, but those two areas are often at odds with each other.

    The good news is that there is a way to close the gap between customer demand and necessary development time: Test automation. Here are three ways that test automation can help companies satisfy both the needs of their business and their customers.

    Relieving the Testing Bottleneck

    Moving fast is becoming a competitive advantage for many companies. That speed-to-market opportunity has been heightened as cloud-based, low-code/no-code apps like Salesforce, MS Dynamics, Workday and Oracle have become an industry standard. Today, while plenty of development activities may be handled by low-code/no-code platforms, the test automation of these apps remains a high-code, high-maintenance activity.

    To keep from becoming a bottleneck, teams should plan automation and testing strategies early in the process. This is where continuous development includes continuous testing. Suppose companies can make test automation accessible to more people by removing the need for specialized knowledge and daunting technical constraints. In that case, they will be able to streamline ongoing development cycles.

    Ensuring Higher-Quality Output; Aligning Development and Testing

    One of the keys to putting out a higher-quality product is by adopting in-sprint testing, which aligns development and testing efforts. Each feature is tested and automated as soon as it is implemented so that teams can more easily catch errors early rather than at the end of a development phase.

    The automation also elevates the area of regression testing, as teams make sure new changes haven’t broken existing functionality. As automation moves into the mainstream, a wider range of collaborators become more involved in testing each additional feature.

    Creating a Better User Experience

    Historically, testing has been extremely focused on function and is very technical while user experience efforts remain focused on the stylistic and usability end of the spectrum. But creating satisfying end-to-end user experiences is a critical part of any application adoption, so the two sides should work toward a common goal.

    That common link lies in a company’s business processes. Each process is the starting point for building and designing a test scenario. From start to finish, everyone understands the user’s journey and works together toward making that path as effortless as possible.

    Conclusion

    Today’s modern customers don’t expect a measurable difference in experience between online and in-person. Expectations are at an all-time high, leading companies to push updates to their websites and apps more quickly and more frequently. As companies increase their dependence on software, the way to test that software is evolving to keep pace with development speeds.

    Automated testing can simplify and speed up the testing phase, catching many of the basic errors and leaving testers to focus on more complex functionality.

     

  • Dynatrace Adds Synthetic Tests to Validate App Experiences

    Dynatrace Adds Synthetic Tests to Validate App Experiences

    Dynatrace today extended the application release management capabilities it provides to include synthetic tests for validating and assuring user experiences.

    Saif Gunja, director of product marketing for Dynatrace, said the user experience validation and user experience assurance (UXA) capability spans everything from application availability and performance to actual engagement with specific features based on the service level objectives (SLOs) defined by a DevOps team.

    Dynatrace views application release management as an extension of its observability platform, said Gunja. While application release management has historically been viewed as an extension of a continuous delivery (CD) platform, Gunja said that observability platforms—such as the Dynatrace platform, which is infused with a Davis artificial intelligence (AI) engine—provide the visibility required to better automate the process.

    In fact, as DevOps continues to evolve, it’s apparent that CD and continuous integration (CI) are becoming more loosely coupled as DevOps processes mature, said Gunja. Most organizations have yet to implement CD simply because it’s been too difficult to achieve. Each platform that applications have been deployed is unique. Dynatrace provides the foundation for automating application release management via the agent software it makes available to instrument applications, said Gunja.

    It’s not clear whether DevOps teams are reevaluating their approach to application delivery, but Gunja noted that much of the interest in the Dynatrace approach is coming from organizations that have already installed the Jenkins CI/CD platform. The number of extensions that have been built and that need to be supported has created a level of technical debt that is unsustainable for many DevOps teams, he noted.

    Many IT leaders concluded they would rather allocate more resources to building applications than maintaining DevOps platforms, added Gunja.

    Most DevOps teams are committed to ruthlessly automating as many IT processes as possible. Observability has always been a core DevOps tenet in pursuit of that goal. Applications that are not instrumented can be programmatically managed. The issue is that providing observability isn’t enough—DevOps teams need to be able to take actions across the entire software development life cycle based on the data collected by the observability platform, said Gunja.

    Obviously, it’s still early days as far as observability is concerned. While application performance management (APM) platforms have been employed for decades, it’s only recently that observability platforms capable of correlating application and IT infrastructure data have emerged. Thanks to the rise of open source agent software, it’s also becoming less costly to instrument IT environments. The challenge, of course, is, as always, deploying and maintaining all that agent software.

    Regardless of the approach to observability and DevOps, it’s clear that IT teams will soon have a lot more visibility into their IT environments. The days when IT teams determined the root cause of an IT issue by process of elimination are finally coming to an end. Observability, however, is not an end in itself, but instead is the end of the beginning of a previous DevOps era.

  • What Is User Acceptance Testing and Why Is it so Important?

    What Is User Acceptance Testing and Why Is it so Important?

    Anyone involved in software development knows how important it is to ensure a positive end-user experience. One way to do that is through the use of user acceptance testing (UAT). 

    UAT is one of the types of testing that you can execute as part of your overall quality assurance process. 

    Source

    It’s essential in helping you determine if your product is market-ready and if it can perform effectively when used by real end-users. 

    UAT helps you avoid implementing software, iterations, upgrades, or customizations that are filled with pesky bugs that can destroy the user experience and render your software unusable (not to mention the damage done to your business’s reputation). 

    When the stakes are so high, it’s clear to see why businesses need a process such as this. In this article, we take a look at user acceptance testing, why it’s so important, and six ways brands can use it to their benefit.

    What is User Acceptance Testing (UAT)?

    User acceptance testing (sometimes known as end-user testing, user acceptability testing or beta testing) is the last step in the entire testing process of any software.

    In product management terminology, acceptance testing takes place as beta testing whereby a small group of early adopters test out the platform and provide user feedback to let you know if the solution meets user expectations and business requirements. 

    It’s done after the system, functionality and all other tests are completed and before the product is launched. 

    The image below shows some of the quality assurance (QA) tests that precede user acceptance testing. 

    Source 

    As you can see from the image, as development work moves forward, UAT progresses from unit testing to integration testing, then system testing and, finally, acceptance testing.

    This final stage is arguably the most crucial element of software development because it ensures a positive end-user experience. 

    As its name implies, it helps you determine if your platform will be accepted by end-users. 

    Testers (end users) work from a checklist of expected outcomes from the test platform and then record what happens when they attempt to execute the different tasks, such as:

    • Logging into the user’s account
    • Placing an item into a shopping cart
    • Checking out 
    • Buying multiple items that qualify for free shipping

    … and so on. 

    If the expected outcomes are achieved when each task is executed, then the software is approved for release. If not, it’s sent back to the developers. 

    This testing process requires collaboration between different stakeholders including project sponsors, business owners, analysts, QA teams and developers. 

    Why is UAT Important to Software Development?

    Although there are many ways projects can benefit from UAT (some of which we’ll look at below), this process is often overlooked by developers. 

    Many see it as a time-consuming and unnecessary process, which is unfortunate because acceptance testing can actually help improve software quality as well as the overall acceptance rate of any newly implemented software. 

    Think about it: You would never drive an untested car, would you? Similarly, businesses should not take risks by implementing software that has not been tested. 

    Considering the time and money you stand to save by ensuring that your software works in accordance with a predetermined test plan, it’s clear that this is something developers need to take more seriously. 

    In fact, far from being the time drain that many developers think it is, UAT can actually gain you a lot of time. How? By preventing unnecessary waste. 

    And far from being a one-off when you come to the end of the project, user acceptance testing extends beyond business application projects because of factors like additional phases, hotfixes, releases/further modifications or customizations and upgrades. 

    Take a look: 

    • Additional Phases: If you choose to roll out at this stage or upgrade an existing application in a phased approach, each phase can have different types of users, different companies (for parent companies, acquisitions, mergers, etc.), additional locations, etc., and you’ll need to plan and conduct separate additional acceptance tests for each individual phase.
    • Hot Fixes: Bug reporting or developer testing during regular day-to-day operations can lead to hotfixes. However, these bug fixes will have to be tested by end-users to make sure that they actually solve the problems originally reported.
    • Further Modifications: Software creators regularly release updates, including limited distribution releases (LTR) and general distribution releases (GDR). LTR releases usually fix issues in a specific problem area, whereas GDR releases are typically heavily tested before release. Both need to be tested to ensure that they fit users’ real-life scenarios so you can achieve your business objectives.
    • Upgrades: Everyone wants to offer their customers the best and most advanced experience. That’s why, for the most part, brands update software regularly to follow industry trends and keep up with the latest innovative technology. Regardless of the upgrade approach you take, whether it’s a little or a lot of customization, it’s crucial that your plan is completed according to business requirements. This will help ensure that the project’s end result is in line with the original business requirements.
    • Privacy Policy Updates: Many end-users may enjoy the overall functionality of the software, but they may desire increased integrations with other platforms or the ability to use a different payment method than that which is currently accepted. While these upgrades and modifications may be easily completed, you may need to alter the information you input into privacy policy generators to ensure that your users legally consent to any new use of their data or access to it by third parties.

    Now that you understand what UAT is and why it’s so important, let’s look at some of the benefits of user acceptance testing to give you a better idea of all the different ways you can leverage it in your own business. 

    Benefits of User Acceptance Testing

    At its core, user acceptance testing will be pivotal in the successful launch and/or maintenance of any type of online platform, whether it’s a simple mobile app or something more complex, such as an eCommerce website. 

    If the test object doesn’t work as desired, the faulty business functions are documented and the item is sent back to developers for repair. 

    Going through this final check ensures that your platform is well-built and performs as expected and that users accept the finished product. 

    Listed below is a summary of the top benefits of UAT: 

    • Benefit: The UAT process allows you to demonstrate that the software’s required functions are operating in a way that’s suited to real-world use and circumstances. As such, acceptance testing focuses on software performance and user behavior to ensure that interactions between the test software’s functionality and the end user’s thought process are aligned. 
    • Benefit: Furthermore, you’ll be getting a much better piece of software because it will be tested and verified by the same people who will be working with it every day. You’ll get optimal opportunities to quickly identify and repair any broken features. So, it’s easy to see how the time you invest in these beta testing cycles will lead to significant improvements in the quality of your software.
    • Benefit: Not only that, but the quality review provided by UAT is also a great way for stakeholders to secure a healthy return on investment (ROI) out of projects. Also, it keeps maintenance costs low and it’s cheaper and faster to fix usability/functionality issues early on.
    • Benefit: You’ll also be able to ensure that your software integrates with third-party platforms including various website builders, email marketing software, project management tools, communication apps and everything else you or your end-users might need to effectively use the software. If your software is supposed to allow for social media sharing, for example, you want to ensure that end-users are able to easily share without going through multiple steps that decrease an otherwise positive experience with the software.

    As you can see, user acceptance testing is vital to the successful deployment of new business applications and software enhancements, as well as to ensure organizational efficiency and success.

    The process helps you to mitigate risk and boost the ROI of your project. But most importantly, UAT ensures that your customers don’t end up being acceptance testers by default, which would have a negative impact effect on your business.

    6 Ways to Use UAT

    Here are six ways you can use this process to achieve better outcomes.

    1. Use user acceptance testing for internal development.
    2. Use the testing process to review new features.
    3. Invite users to take part in the tests to help them become more invested in the brand and become brand ambassadors.
    4. Use beta tests as a premium feature to convert subscribers to higher tier packages, increasing sales for the company.
    5. Share information about UAT and its results on social media platforms, email and other channels as part of your digital marketing campaigns.

    These are just a few of the different ways your organization can benefit from UAT far beyond the initial goal of the testing process. 

    And remember, it’s all about the users. 

    UAT isn’t just another testing technique. This test puts the user at the center of the discussion to answer questions such as:

    • Are users able to use the software without error(s)?
    • Does the solution meet promised specs? Does it behave in an expected way?
    • Does the software continue to work when users take unexpected actions?

    …and so on. 

    Whether the platform is constructed in-house or by a third-party company, all solutions must go through user acceptance testing because your users are the ultimate audience who will pay for your solutions. For this reason, it’s worth taking the time to ensure that your software solution meets both user expectations and business requirements.

    Planning and Managing the UAT Process

    The key to a successful UAT testing process is to understand the needs of your target users and their expected behavior. This way, you’ll be able to develop defined acceptance criteria so you can come up with efficient tests to complete the testing process fast. 

    Also, since the goal is to assess the market readiness of the product, it’s important to record all evidence and track progress with the dev team as well as report to stakeholders so everyone is on the same page. 

    With that said, here are five steps to help you plan and manage the user testing process: 

    Step One: Planning. The user acceptance testing strategy mainly includes a time frame outlined during the planning step.

    Step Two: Designing Test Cases. Test cases cover the core functional frameworks of the platform in real-world usage. When designing test cases, you need to do the following: 

    • Determine the needs of target users and stakeholders 
    • Design test cases according to the user journey and product specifications 
    • Summarize test runbook, acceptance criteria, etc. 
    • Prepare test data, e.g. registration, three days in use, one week in use, etc. 

    Step Three: Setting Up Your UAT Team. The acceptance testing team is comprised of real-world end-users or users who are not part of the development or testing phases to ensure objectivity. You should never use developers to test their own software as they may bring bias (albeit unintentionally) into the testing process. 

    Step Four: Executing and Documenting Test Cases. The UAT team executes the test cases, together with relevant random exploratory tests, after which all bugs are logged into the bug tracker along with steps to reproduce.

    Step Five: Bug Fixing. Once you’ve executed the test cases, you need to keep track of the tickets and user feedback. The development team can account for all the bugs found by the testers and start fixing bugs depending on severity. 

    Step Six: Confirm Fixes and Get Stakeholders to Sign Off. Once all UAT tests have been run and the results are in hand, the acceptance decision can be made. If all users are satisfied, then it’s a go. If not, it’s a no-go. 

    Last But Not Least

    UAT is the last stage in the software development life cycle. But, it’s also one of the most important things you can do to ensure the success of your software. Just keep in mind that for this process to be effective, it should be viewed as validation and not verification. Use your test procedures to validate functionality in both smaller scope (as a way to test functionality) and in a larger context (to simulate real-world processes) so you can get an end-user vision of your product which will ultimately help you increase the robustness and usability of your software.

    Are you ready to use UAT to get feedback on your software so you can increase end-user happiness? Share your thoughts below!

  • This is Not Just a Test: DevOps and the Need to Automate

    This is Not Just a Test: DevOps and the Need to Automate

    Creating software is as much about the process as it is about the product. In recent years, we’ve seen the speed of innovation reach new heights due to more efficient development and quality control. DevOps has further accelerated development by combining software and operations in unified, self-improving teams. With automation taking care of rote tasks, developers can now tackle more difficult issues, resulting in even greater software quality and faster development. The need for speed to market is vital; a month or two can make or break a product launch. Then again, if a product isn’t tested thoroughly in the rush to get to market, flaws can slip through and an organization’s revenue and reputation can take a serious hit. In the U.S., software failures in businesses’ operations systems cost a total of almost $1.6 trillion in 2019. The problem is that it can be a bottleneck and create major delays in the release cycle. In fact, A recent survey of 4,300 developers revealed that it is the top reason that releases are delayed.

    In the past, the waterfall approach was common. Tests usually took place after code was written; carried out by testers who had no interaction with the original developers. As a result, bugs that could have taken moments to correct during development required days or even weeks to fix afterward. Even worse, the complexity of the finished software and its many dependencies meant that fixing a localized bug could introduce additional problems that needed to be fixed after another round of testing, further delaying the release.

    But even in a modern DevOps environment, testers and developers are still speaking entirely different languages. By and large, testers do not code and software developers are not expert testers. The two teams typically converse through multiple logs, event viewers and other operating system tools. It creates a great deal of friction and slows down the process.

    Shift Left 

    DevOps represented a big improvement over waterfall methods but, even so, it’s far from a set-in-stone process. It, too, has evolved and right now, shift left testing has moved into the spotlight.

    In this approach, the guiding philosophy is to test early and often, beginning very early in the development cycle and continuously throughout every stage whenever there’s an opportunity. As a result, bugs are caught early when the relevant code is still fresh in the developers’ minds, so it’s easier to fix. In the end, businesses are able to release higher quality software much faster and more frequently.

    Environmental Tests

    When it comes to the DevOps process, it’s not always clear where and how to automate the process. The following looks at how that can be incorporated into DevOps based on four different environments in a continuous delivery (CD) pipeline.

    These environments include local development (usually a coder’s personal device), a continuous integration (CI) or development server, test servers for user interface (UI) testing and the production environment itself. Each has stages with requirements that must be met for code to advance.

    Whenever any new software features or capabilities are entertained, though, developers and the QA team should begin by answering the question, “How can this be tested?”

    Moving Through the Stages

    Once code is created, it must be able to be built without error, passing all unit tests and local verification.

    This stage is particularly important because the development environment echoes the requirements of production; if it works here it’ll likely also work for the customer.

    At this stage, running early regression and developer verification tests ensures the build is ready while providing feedback about the reliability of the environment. It’s useful to have individual test cases as part of more than one schedule. This enables developers to test minor parts of the software without undergoing regression tests of all product features. These tests should be automated and triggered as part of the same process.

    Developers should have the ability to perform basic tests on their own code. After all, the sooner a flaw is identified, the easier it is to eradicate. Just as testers are not experts in coding, neither are developers expert testers. To shift left and effectively integrate tests at all stages of the development life cycle, it must be automated. This will allow developers to test larger quantities of code earlier and more frequently.

    Once through this stage, the software can advance to the test environment.

    This is where a variety of product tests take place, including those for function, load and performance. Passing this stage entails completing all automated functional UI tests, verification of visual appearance, an inspection of server logs and performance evaluation.

    This is where most regression tests are conducted and automation is crucial for scheduled tests to be triggered.

    Next, the product moves into the production stage.

    Tests still occur in this stage to pinpoint any remaining bugs. Automated tests are then run against the production environment to closely mirror interactions with real end users. At this point, it’s highly unlikely there will be major issues but, if there are, running tests at earlier stages could very well be the culprit.

    Throughout the process, ideally, testers will use a visual, no-code tool to create these automated tests. In this way, testers are able to more rapidly and intuitively implement test automation without having to tap into programming resources. Additionally, the integrated documentation of the actions taken during automation will be interactively connected to visual documentation of the test results. The upshot is that testers and developers gain a shared language for both building test automation and troubleshooting the issues it identifies.

    This is Not Just a Test

    Effective, scalable tests are crucial to delivering the reliable, high-quality software that today’s businesses rely on for day-to-day operations. This is why DevOps and development teams need to integrate tests at all stages of software development. Shift left speeds up the creation of better quality software, which provides a significant market advantage. After all, it’s not just about software quality; it can show which companies are best prepared to meet market demands.