devopsy.kinsta.cloud

Tag: continuous testing

  • Applying AI/ML to Continuous Testing

    Applying AI/ML to Continuous Testing

    Artificial intelligence (AI) and machine learning (ML) can play a transformative role across the software development lifecycle, with a special focus on enhancing continuous testing (CT). CT is especially critical in the context of continuous integration/continuous deployment (CI/CD) pipelines, where the need for speed and efficiency must be balanced with the demands for quality and security. AI/ML contributes by automating complex tasks, predicting potential issues before they occur and providing actionable insights, thereby reducing manual efforts and enabling more strategic use of human resources.
    Moreover, the application of AI/ML extends beyond mere automation of tests. It encompasses the capability to learn from data, adapt to new information, and improve over time. This ability is invaluable for identifying patterns, anticipating vulnerabilities and optimizing testing strategies. In quality assurance, AI-driven tools can predict areas most likely to fail and tailor testing efforts accordingly. In security, ML algorithms can detect anomalies that signify potential threats, while in operations, AI can enhance feedback mechanisms, leading to more resilient and responsive systems.

    AI/ML for Continuous Testing

    Here’s how AI technologies can be applied to reduce bottlenecks associated with testing activities across various testing activities:

    1. Requirements Analysis
    • Explanation: Ensures test scenarios align with business requirements and user needs.
    • Bottleneck: Misinterpretation or incomplete analysis can lead to inadequate test coverage.
    • AI/ML Solution: NLP can automate the extraction and interpretation of requirements, ensuring comprehensive and accurate test coverage.

    2. Test Strategy
    • Explanation: Outlines the testing approach, objectives, and resources.
    • Bottleneck: An unclear strategy may lead to inefficient testing efforts and resource allocation.
    • AI/ML Solution: AI can analyze historical data to suggest the most effective test strategies and predict resource needs.

    3. Test Plans
    • Explanation: Detailed documents guiding the testing process, timelines, and responsibilities.
    • Bottleneck: Inflexible plans can struggle to adapt to project changes, causing delays.
    • AI/ML Solution: Machine learning algorithms can suggest adjustments to test plans based on ongoing project developments and past outcomes.

    4. Test Cases
    • Explanation: Specific conditions under which a test is executed.
    • Bottleneck: Time-consuming development and significant effort to maintain.
    • AI/ML Solution: AI can automate the generation of test cases from requirements documents, improving efficiency and coverage.

    5. Test Scripts
    • Explanation: Automated scripts that execute test cases.
    • Bottleneck: Script development and maintenance can be resource-intensive.
    • AI/ML Solution: AI can generate and update test scripts based on changes in the application or test cases, reducing maintenance effort.

    6. Test Data
    • Explanation: Data sets used during testing to simulate real-world scenarios.
    • Bottleneck: Creating, managing, and maintaining accurate test data is challenging.
    • AI/ML Solution: AI can automate the generation and management of test data, ensuring relevance and variety.

    7. Test Environment
    • Explanation: The setup where testing is conducted mirrors production environments as closely as possible.
    • Bottleneck: Configuration and maintenance of test environments are complex.
    • AI/ML Solution: AI can predict and configure optimal test environments based on test requirements, reducing setup time.

    8. Coordination with Dependent Systems
    • Explanation: Ensuring the system under test interacts correctly with databases and other applications.
    • Bottleneck: Dependency management can cause delays.
    • AI/ML Solution: AI can automate the detection and resolution of integration issues, enhancing coordination efficiency.

    9. Test Environment Setup
    • Explanation: Configuring infrastructure and tools necessary for testing.
    • Bottleneck: Setup complexity and resource contention lead to delays.
    • AI/ML Solution: AI algorithms can optimize environment setup, automatically adjusting resources as needed.

    10. Test Campaign Setup
    • Explanation: Organizing and scheduling a series of test executions.
    • Bottleneck: Requires careful planning and can be hindered by resource limitations.
    • AI/ML Solution: AI can assist in scheduling and prioritizing test campaigns based on risk and impact analysis.

    11. Test Execution
    • Explanation: The process of running test cases and scripts, both automated and manual.
    • Bottleneck: Time-consuming, particularly for manual tests.
    • AI/ML Solution: AI can prioritize test execution and identify flaky tests, streamlining the process.

    12. Test Verdict Reporting
    • Explanation: Determining and reporting the outcome of test executions.
    • Bottleneck: Manual verdict determination can be slow.
    • AI/ML Solution: AI can automatically interpret test outcomes, speeding up reporting.

    13. Logging of Data
    • Explanation: Recording data relevant to the test for further analysis.
    • Bottleneck: Extensive data collection can overwhelm resources.
    • AI/ML Solution: AI can intelligently filter and log pertinent data, reducing noise.

    14. Test Result Analysis
    • Explanation: Analyzing test outcomes to identify defects and issues.
    • Bottleneck: Requires significant time and expertise.
    • AI/ML Solution: ML algorithms can quickly identify patterns and anomalies in test results, highlighting potential issues.

    15. Test Result Reporting
    • Explanation: Communicating findings to stakeholders.
    • Bottleneck: Compiling reports is time-intensive.
    • AI/ML Solution: Automated reporting tools powered by AI can generate insightful and comprehensive reports quickly.

    16. Waiting for Resource to Fix Failed Tests
    • Explanation: Downtime while awaiting fixes for identified issues.
    • Bottleneck: Halts testing progress.
    • AI/ML Solution: AI can predict which areas might fail and propose potential fixes

    Challenges

    Applying AI/ML to software testing activities offers numerous advantages but also introduces several challenges. Addressing these problems requires a combination of technical solutions, process adjustments, and cultural changes.

    1. Lack of Intuitive Understanding of the Application Being Tested
    • Problem: AI/ML models may not fully grasp the application’s context or the nuances of its functionalities, leading to less effective test scenarios.
    • Solution: Enhance AI models with richer contextual data and incorporate feedback loops where testers can refine and adjust AI-generated test cases. Employing techniques like reinforcement learning can also help AI models better understand application contexts over time.

    2. Repeatability and Consistency Between Test Sessions
    • Problem: AI-driven tests may generate different outputs for the same input over different sessions, complicating test consistency and traceability.
    • Solution: Implement versioning for AI models and their training data, ensuring consistency across test sessions. Use deterministic approaches in conjunction with AI to maintain a core of stable, repeatable tests.

    3. Lack of Understanding of the Tests Generated
    • Problem: Testers may find it challenging to understand or trust the rationale behind AI-generated test cases, impacting their ability to evaluate test outcomes effectively.
    • Solution: Incorporate explanations into AI/ML models to provide insights into their decision-making processes. Foster a culture of trust and understanding through education and transparency regarding how AI models operate.

    4. Test Coverage
    • Problem: There’s a risk that AI/ML might not adequately cover all testing scenarios, potentially missing critical defects.
    • Solution: Combine AI/ML with traditional testing methods to ensure comprehensive coverage. Regularly review and adjust the criteria used by AI/ML models to generate test cases, ensuring they align with evolving application features and risks.

    5. Compatibility with Different Test Tools
    • Problem: AI/ML models might not seamlessly integrate with existing testing tools and frameworks, limiting their utility.
    • Solution: Develop or use AI/ML solutions with extensive API support and integration capabilities. Work with tool vendors or contribute to open source projects to enhance compatibility.

    6. Acceptance by Teams
    • Problem: Testers and developers may be skeptical or resistant to AI-driven testing due to concerns about job displacement or mistrust in AI’s effectiveness.
    • Solution: Educate and involve teams in the development and implementation of AI/ML testing strategies. Demonstrate the value of AI/ML in augmenting their roles rather than replacing them, focusing on AI as a tool to tackle mundane tasks and allowing them to focus on more complex and rewarding work.

    7. Data Quality and Availability
    • Problem: AI/ML models require large amounts of high-quality data for training. Inadequate or poor-quality data can lead to ineffective testing.
    • Solution: Invest in data curation and generation strategies such as synthetic data creation to ensure models are well-trained.

    8. Continuous Learning and Adaptation
    • Problem: AI/ML models may become outdated as applications evolve.
    • Solution: Establish continuous learning mechanisms where models are regularly updated with new data and feedback, ensuring they remain relevant and effective.

    9. Ethical and Bias Considerations
    • Problem: AI/ML testing models might inherit or amplify biases present in their training data, leading to unfair or discriminatory outcomes.
    • Solution: Implement ethical guidelines and bias detection methodologies for AI/ML model development and use. Regularly audit models for biases and correct them as needed.

    By addressing these challenges with thoughtful strategies, organizations can maximize the benefits of AI/ML in testing activities while mitigating potential drawbacks, leading to more efficient, effective and trustworthy processes.

    Summary

    This blog explained use cases of artificial intelligence (AI) and machine learning (ML) within the realm of continuous testing of the software development life cycle. By harnessing the power of AI/ML, we’ve seen how organizations can significantly enhance their development processes, making them more efficient, secure and responsive to user needs. The exploration of specific AI/ML applications across various activities has provided a blueprint for automating and optimizing tasks that traditionally required significant manual effort.

  • Tabnine Extends Generative AI Testing Platform by Embracing RAG

    Tabnine Extends Generative AI Testing Platform by Embracing RAG

    Tabnine today revealed that its namesake generative artificial intelligence (AI) platform for creating test code can now surface more accurate and personalized recommendations based on specific code and engineering patterns.

    At the same time, Tabnine also announced that Tabnine Chat, a tool for interacting with the large language models (LLMs) at the core of the Tabine platform using a natural language interface, is now generally available.

    Data that resides in a codebase or an integrated developer environment (IDE) can now be used to extend the large language models (LLMs) on which the Tabnine platform is based using retrieval augmented generation (RAG) techniques. Previously, the Tabnine platform would only generate test code based on the data that Tabnine had exposed to its LLMs.

    Tabnine president Peter Guagenti said RAG makes it possible to provide additional contextual awareness for the test code generated along with documentation and explanations of the test code created. The overall goal is to leverage generative AI to make it easier to shift more responsibility for application testing further left toward developers, he noted.

    That approach enables developers to create and run more routine tests earlier in the software development life cycle, which should provide application testing teams with more time to run more complex tests before applications are deployed, added Guagenti. Rather than being a job killer, generative AI will eliminate rote tasks that most members of the DevOps team don’t really enjoy doing, he noted.

    Tabnine uses a mix of LLMs it developed with Google to apply generative AI to application testing. Unlike general-purpose LLMs, the ones used by Tabnine were trained using a narrower base of testing code that the company curated using tests made freely available on the internet.

    It’s now only a matter of time before generative AI is applied across the entire software development life cycle (SDLC). It’s not clear how automated testing will become in the age of AI, but the overall quality of applications should steadily improve as more tests are run. The goal is to make it possible to conduct more tests without slowing down the overall pace of application development.

    There continues to be as much irrational exuberance about AI as fear and loathing. Still, DevOps teams that committed early on to ruthless automation will naturally be at the forefront of adoption. The immediate challenge is assessing what functions will be automated and the impact those changes will have on the way DevOps teams are currently structured. Ultimately, the goal is to eliminate as many of the bottlenecks that conspire to slow application development and deployment while simultaneously improving the quality of the application experience.

    It may be a while before generative AI is pervasively applied across DevOps workflows, but one of the first places it will undoubtedly manifest is in test automation. The challenge and the opportunity now is determining how best to apply it in a way that augments DevOps teams that arguably have never had enough time to properly test code in the first place.

  • Harnessing Generative AI for Feature Management Testing

    Harnessing Generative AI for Feature Management Testing

    When it comes to DevOps, the emerging integration of generative AI into feature management testing marks a significant evolution. We’re going to take a no-nonsense look at how this technology is revolutionizing the way we create testing environments. 

    Using artificial intelligence to generate tests allows us to mimic a vast array of user scenarios and environments, which means we can develop and deploy features that aren’t just good but great—robust, efficient and ready for whatever the real world throws at them. 

    Embracing Generative AI in DevOps Testing

    Artificial intelligence is dramatically transforming the world around us, impacting a wide variety of industries at breakneck speeds. A stunning 83% of companies assert that AI is a critical component of their business strategies, particularly when it comes to software development.

    Generative AI has quickly become a transformative tool when it comes to DevOps, particularly when it comes to feature management testing, by leveraging machine learning algorithms to generate data and test scenarios that closely mimic real-world conditions.

    Traditionally, DevOps teams have relied mostly on manual and a couple of automated testing methods to validate their features. However, these methods often fail to accurately simulate the wide variety of complex and unpredictable user interactions and environmental variables. 

    By creating nuanced and diverse test cases, generative AI can fill this gap, enabling a more thorough evaluation of features under various conditions. Even now, on a smaller scale, WordPress site owners are using predictive analysis and even simple comparisons to properly orchestrate the rollout of new features–with over 60,000 plugins to choose from, generative AI has the potential to prevent lag, which can often be the result of a single shoddy plugin. 

    Simply put, AI is actively improving how features are implemented and tested across various platforms by intelligently crafting diverse test scenarios. 

    Enhancing Test Coverage and Efficiency

    The significant expansion in test coverage is one of the key benefits of integrating generative AI into DevOps testing. Traditional testing methods are typically limited by the scenarios that testers can foresee and manually script.

    In contrast, AI-driven testing tools can generate a multitude of unique, unexpected test scenarios, capturing edge cases that might otherwise go unnoticed, which is crucial in identifying potential bugs and vulnerabilities early in the development cycle, saving time and resources in the long run. 

    Additionally, AI can automate both the creation and implementation of these tests. This simplifies the testing workflow, allowing human testers to dedicate their full attention to more intricate and challenging tasks.

    Simulating Real-World Scenarios

    Another key advantage of generative AI in DevOps is its ability to simulate real-world user behavior and environments. 

    AI models can create realistic and diverse user scenarios by analyzing historical data, user patterns, and environmental variables. This type of functionality is a huge asset in feature management, where understanding how new features will perform in the real world is paramount. 

    To use a couple of examples, an AI model can generate test cases that mimic peak usage times, varying network conditions, or different user demographics. Furthermore, companies like Netflix, Google and IBM are already actively using generative AI to simulate real-world scenarios for various applications, such as content delivery optimization, predictive maintenance, and software testing. 

    Accelerating Feature Development in DevOps

    The integration of generative AI into feature testing also accelerates the feature development and deployment process. 

    A well-constructed prompt chain can set up an environment where DevOps teams can effectively compare different products/services, especially when it comes to more sensitive software, such as cloud management platforms and even find the best ways to escape vendor lock-in without actually wasting time manually comparing and testing everything. 

    Applying generative AI can also drastically cut down on the time it takes to validate a new feature by generating and running through thousands of test scenarios in quick succession. In DevOps environments, where rapid iteration and continuous deployment are key, this is essential. 

    AI-driven testing can also provide developers with immediate feedback, allowing for quick adjustments and improvements. This ensures that features are developed faster while meeting the highest quality standards before proper deployment.

    Potential Challenges and Considerations

    While the potential of generative AI in DevOps testing is immense, there are challenges and considerations that teams must navigate and work to overcome to successfully apply this powerful new tool. 

    One key issue is the quality and relevance of the data used to train AI models; since inaccurate or biased training data can lead to suboptimal or even harmful outputs, carefully selecting and curating data is a crucial step​​​​.

    Incorporating generative AI into established DevOps processes can also present challenges in terms of complexity and resource demands. Organizations may face significant challenges in modifying their existing pipelines and processes to accommodate AI technologies, which often require substantial investments in hardware, software and skilled personnel.

    Lastly, the need to continuously improve and update AI models cannot be overstated. As new technologies and security threats emerge, these models must be regularly refined and adapted to remain effective and relevant, which will require ongoing investment in AI systems and a commitment to staying abreast of evolving technologies and methodologies in what is a rapidly evolving and advancing field.

    Future Directions of Generative AI

    Looking forward, the role of generative AI in DevOps is poised to grow even more prominent, and as AI models become more sophisticated and data-rich, their ability to generate realistic and complex test scenarios will only improve. 

    But let’s look beyond just creating testing environments and isolated testing environments: A well-executed, AI-aided process can reduce costs during SAP staff augmentation and can also streamline processes in other high-level enterprise systems such as Oracle ERP and Salesforce CRM. Overall, ERP software is a perfect candidate for DevOps feature management, as it allows devs to use generative AI for everything from writing code to custom tests. 

    We can also expect to see further integration of AI in other aspects of DevOps, such as monitoring and performance optimization. The ultimate goal is to create a seamless, AI-enhanced pipeline that tests and deploys features efficiently and continuously learns and improves from real-world feedback.

    To better illustrate this transformative impact, some key examples of how generative AI will further impact DevOps in the future include the following: 

    • AI-Augmented Development: Generative will assist developers with code suggestions, bug fixes, and even making architectural decisions, thereby streamlining the development workflow​.
    • Improved Infrastructure Management and Automation: The use of predictive analytics, driven by generative AI, will play a crucial role in anticipating infrastructure requirements, facilitating preemptive scaling, and efficient distribution of resources.
    • Adaptive Problem Resolution and Incident Handling: Generative AI is set to revolutionize the way operational problems are identified, diagnosed, and resolved swiftly, greatly boosting the dependability and steadiness of DevOps practices. This is evident in platforms such as Google Cloud’s operations suite, where such AI capabilities are already making a significant impact. 
    • Generative AI in Proactive DevOps: This emerging trend focuses on leveraging generative AI to anticipate potential issues before they manifest, which enables preemptive maintenance strategies and is aimed at reducing system downtime.
    • AI-Driven Testing and QA: AI-powered testing will become more sophisticated, with capabilities such as automated test case generation, test data creation, and advanced anomaly detection​.

    Harnessing Generative AI in DevOps

    Integrating generative AI into DevOps is a groundbreaking development poised to revolutionize how we approach software development, testing, and infrastructure management. 

    This innovative technology offers immense potential to automate complex tasks, enhance efficiency and improve the quality and reliability of software products. However, as with any emerging technology, its adoption comes with its own set of challenges and considerations.

    Enterprises aiming to leverage generative AI within their DevOps practices must address challenges associated with data integrity, intricate integration processes, adherence to privacy laws and the necessity for ongoing enhancement of their AI models.

    Despite these challenges, the benefits of adopting generative AI in DevOps are quite substantial, ranging from increased efficiency and cost savings to enhanced scalability and improved overall software quality.

  • Kobiton Open Sources Test Script Generation Tool

    Kobiton Open Sources Test Script Generation Tool

    Kobiton today announced it is open sourcing a script generation tool for the Appium automation framework that automates the running of test scripts.

    The overall goal is to make the tool, which makes it simpler to create more testing scripts, more widely available and, ultimately, lead to higher-quality applications being deployed.

    Kobiton CTO Frank Moyer said the tool makes it possible to generate tests from a test case that has been captured in a Node.js format. Rather than having to hire hundreds of testers to create those scripts manually, it’s now possible to automate those tasks using an open source tool, he added.

    On average, Kobiton claims the tool it developed can write a 30-step script in 10 minutes, whereas a manual approach could take as long as four hours to construct a similar test.

    Once created, those test scripts can automatically be executed using the Appium framework. The output from that tool can also be used on other testing platforms, including Sauce Labs, BrowserStack and LambdaTest or integrated with low-code testing platforms from Perfecto and Tricentis.

    It’s no secret that the quality of any given application tends to vary widely, not just between organizations but also within them. Most of those issues can be traced back to a lack of testing that usually occurs because there isn’t enough time. As deadlines near, one of the first places organizations look to make up for delays is the time available for testing. The hope is that whatever issues arise will be addressed via the next update. Of course, when the update falls behind schedule, the testing process is again cut back.

    Ideally, application development projects would not be allowed to go forward without adequate testing, but given business pressures, they often do. The only way to limit the impact of application development delays is to reduce the time required to create and run tests. The more automated that process becomes, the less time there will be spent on manually running routine tests. In fact, hopefully, there might even be enough time to run more complex tests to uncover issues that might adversely impact the user experience in ways no one might have otherwise considered.

    One way or another, the quality of the applications being deployed needs to improve if for no other reason than end users have options. Most end users will never give a mobile application a second chance if the initial experience doesn’t meet expectations. At the same time, application security regulations are only going to become more stringent. The tolerance for known vulnerabilities discovered after an application has been deployed is falling. Many organizations will find themselves subject to fines being levied simply because there was not enough adequate testing baked into the software supply chain.

    In fact, application security should, in the final analysis, be an element of a large quality assurance process that begins with adequate testing.

  • SmartBear Acquires Reflect to Gain Generative AI-Based Testing Tool

    SmartBear Acquires Reflect to Gain Generative AI-Based Testing Tool

    SmartBear this week revealed it has acquired Reflect, a provider of a no-code testing platform for web applications that leverages generative artificial intelligence to create and execute tests.

    Madhup Mishra, senior vice president of product marketing at SmartBear, said the platform Reflect created will initially be incorporated into the company’s existing Test Hub platform before Reflect’s generative AI capabilities are added to other platforms.

    Reflect provides access to a natural language interface to create tests using multiple large language models (LLMs) that it is designed to invoke. It can also understand the intent of a test to understand what elements to test regardless of whether, for example, a button has been moved from one part of a user interface to another, said Mishra. Test step definitions, once approved, can also be automatically executed using scripts generated by the platform.

    SmartBear has no plans to build its own LLMs, said Mishra. Rather, the company is focusing its efforts on providing the tools and prompt engineering techniques needed to effectively operationalize them, he added.

    Reflect is the tenth acquisition SmartBear has made as part of an effort to provide lightweight hubs to address testing, the building of application programming interfaces (APIs) and analysis of application performance and user experience. Last year, the company acquired Stoplight to gain API governance capabilities.

    Rather than building a single integrated platform, the company is focused on providing access to lightweight hubs that are simpler to invoke, deploy and maintain, said Mishra. The overall goal is to meet IT teams where they are versus requiring them to adopt any entirely new monolithic platform that requires organizations to rip and replace every tool they already have in place, he said.

    There is little doubt at this point that generative AI will have a profound impact on application testing in a way that should ultimately improve the quality of the applications. As the time required to create tests drops, more tests will be run. Today, it’s all too common for tests not to be conducted as thoroughly as they should be simply because either a developer lacked the expertise to create one or, with a delivery deadline looming, they simply ran out of time.

    Naturally, the rise of generative AI will also change how testing processes are managed. It’s not clear how far left generative AI will push responsibility for testing applications, but as more tests are created and run, they will need to be integrated into DevOps workflows.

    Of course, testing is only one element of a DevOps workflow that is about to be transformed by generative AI. DevOps teams should already be identifying manual tasks that can be automated using generative AI as part of an effort to further automate workflows that, despite commitments to automation, still require too much time to execute and manage. Once identified, DevOps teams can then get a head start on redefining roles and responsibilities as generative AI is increasingly operationalized across those workflows.

  • Sauce Labs Adds Visual Testing Tool to Platform

    Sauce Labs Adds Visual Testing Tool to Platform

    Sauce Labs has added native visual regression testing capabilities to its testing platform to enable DevOps teams to streamline workflows.

    Marcus Merrell, vice president of technology strategy for Sauce Labs, said Sauce Visual provides the ability to test user interfaces without requiring DevOps teams to acquire, maintain and integrate a separate tool.

    The overall goal is to make it possible to test functions and user interfaces in parallel by adding a single line of code to a DevOps workflow, he added. Updates to the visual presentation of a page are automatically highlighted, with any visual inconsistencies automatically surfaced. In comparison, a standalone visual testing tool increases the time it takes to execute a test suite, noted Merrell.

    Sauce Visual natively uses the same tools as the rest of the Sauce test automation platform, and can be integrated into any continuous integration/continuous delivery (CI/CD) workflow. At launch, Sauce Visual is compatible with Java/JUnit, Java/TestNG, WebDriverIO, Selenium, Appium (web) and Cypress testing tools with support for Playwright, Puppeteer, TestCafe, Espresso, and XCUITest forthcoming.

    The Sauce Labs approach also serves to reduce the total cost of application testing at a time when the number of applications organizations are building and deploying only continues to increase exponentially, said Merrell.

    In general, organizations are attempting to strike a balance between shifting responsibility for testing too far left toward developers that typically lack testing expertise and the need to verify application experiences are delivered as expected. Many organizations have shifted more responsibility for testing toward developers in the hope of improving code quality and reducing the number of vulnerabilities that might find their way into a production environment. The challenge is those efforts have also increased the level of cognitive load developers need to carry at the expense of having more time to write application code.

    Of course, in time, artificial intelligence (AI) tools will make it easier to generate test scripts, but there is still going to be a need for testers who were not involved in the writing of code to validate it, noted Merrell. The difference is those testers will not necessarily need to be rocket scientists as the process of creating tests becomes more automated, he added.

    Testing user interfaces is especially problematic for developers because many of them don’t have a lot of expertise in the business functions that the application they are building enables. In addition, there are often minute changes to user interfaces that developers are not always going to discern.

    As always, one of the first things that tends to be reduced whenever a developer falls behind schedule is testing. In a recent Sauce Labs survey found more than two-thirds (67%) admitted they pushed code into a production environment without testing, with more than a quarter (28%) acknowledging they do so on a regular basis.

    The assumption is that if any issues do arise, they can be addressed via the next update to the application. End users, however, are less tolerant of that approach when, in the age of digital services, an alternative application is readily available.

  • Coexisting With AI: The Future of Software Testing

    Coexisting With AI: The Future of Software Testing

    If 2023 was the year of artificial intelligence (AI), then 2024 is going to be the year of human coexistence with the technology. Since the release of Open AI’s ChatGPT in November 2022, there has been a steady stream of competing large language models (LLMs) and integrated applications for specific tasks, including content, image processing and code production. It’s no longer a question of if AI will be adopted; we have moved on to the question of how best to bring this technology into our daily lives. These are my predictions for the software quality assurance testing industry for 2024.

    Automated testing will become a necessity, not a choice.

    Developers will lean heavily on AI-powered copilot tools, producing more code faster. That means huge increases in the risk profile of every software release. In 2024, testers will respond by embracing AI-powered testing tools to keep up with developers using AI-powered tools and not become the bottleneck in the software development life cycle (SDLC).

    The role of the tester will increase and evolve.

    While AI is helping software engineers and test automation engineers produce more code faster, it still requires the highly skilled eye of an experienced engineer to determine how good and usable the code or test is. In 2024, there will be a high demand for skilled workers with specific domain knowledge who can parse through the AI-generated output and determine if it’s coherent and useful within the specific business application. Although this is necessary for developers and testers to start trusting what the AI generates, they should be wary of spending inefficient amounts of time constructing AI prompts, as this can ultimately lead to decreased levels of performance. For instance, a developer could easily spend most of their time validating the AI-generated output instead of testing the release that will be deployed to users.

    Being able to distinguish between when to rely on AI and when to forego AI’s help will be key to streamlining the workflow. Eventually, we’re going to start seeing AI-powered testing tools for non-coders that focus on achieving repeatability, dependability and scalability so that testers can truly use AI as their primary testing tool and ultimately boost their productivity.

    The rise of protected, offline LLMs and the manual tester.

    As enterprise companies show signs they don’t trust public LLMs (e.g., ChatGPT, Bard, etc.) with their data and intellectual property (IP), they’re starting to build and deploy their own private LLMs behind secured firewalls. Fine-tuning those LLMs with domain-specific data (e.g., banking, health care, etc.) will require a great volume of testing. This promises a resurgence in the role of the manual tester as they will have an increasingly important role to play in that process since they possess deep domain knowledge that is increasingly scarce across enterprises.

    As we stand on the brink of 2024, it is evident that the synergy between artificial intelligence and human expertise will be the cornerstone of software quality engineering. Human testers must learn to harness the power of AI while contributing the irreplaceable nuance of human judgment. The year ahead promises to be one where human ingenuity collaborates with AI’s efficiency to ensure that the software we rely on is not only functional but also reliable and secure. There will likely be a concerted effort to refine these collaborations, ensuring that AI serves as a reliable support rather than an overpromised solution.