devopsy.kinsta.cloud

Tag: software testing

AI Is Forcing a Rethink of Software Testing and QA

SmartBear vice president of AI and architecture Fitz Nowlan explains why maintaining software integrity in the age of artificial intelligence requires organizations to double down on testing and quality assurance. He discusses how AI-driven development increases risk and why stronger QA practices are essential for reliable software delivery.

Nowlan argues that AI-driven development increases risk in a few ways. More code gets produced, more quickly, which means more surface area for defects and more opportunity for issues to slip into production. He also notes that relying on traditional, manually authored testing approaches can’t keep up with the churn created by AI-generated code. As a result, organizations need to “double down” on testing and quality assurance, and they need to modernize how QA is done.

A central theme of the conversation is the rise of AI-powered QA. Nowlan explains why the same wave of automation transforming software creation must also transform test management, test execution, and even test authoring. He points to functional specifications—requirements and definitions of correctness—as the source of truth that can guide both what gets built and what gets tested, especially as roles blend across development, QA, DevOps, and product management.

The discussion also tackles a common concern: can you use AI to test AI-written code without “letting the fox guard the henhouse”? Nowlan says the key isn’t necessarily a different model, but a different structure—separating the frameworks and prompts used for QA from those used for building features, and expanding validation beyond just code to include operational signals like latency and system behavior.

Finally, Nowlan looks ahead to “vibe coding” and citizen development, where prompts and specs matter as much as source code. His message: speed is great, but reliability still wins—and in an AI-driven world, stronger QA practices are the only way to maintain software integrity.

January 22, 2026
Survey Sees AI Being Applied to Improve Software Quality Testing

A global survey of 1,775 IT and business executives published today finds 71% are working for organizations that have integrated some form of artificial intelligence and generative AI capability into their operation, with just over a third (34%) specifically using AI to improve quality assurance.

Conducted by Sogeti, a unit of Capgemini in collaboration with OpenText and the market research firm Coleman Parkes, another 34% have developed roadmaps for improving quality engineering following successful AI pilots. A total of 19% are still conducting pilots.

Challenges hindering generative AI adoption include data breach concerns (58%), tool integrations (55%), the level of effort required (53%) hallucinations (47%) and unforeseen costs (43%). Issues that organizations are working through include a lack of clear AI strategy (56%), lack of skills (53%) and a lack of a well-defined approach to testing (50%). In the area of test automation specifically, 29% of respondents said their organizations have successfully embraced generative AI, while another 42% have conducted a few experiments.

The top three benefits of applying AI to testing are faster automation (72%), easier integrations (68%) and reducing testing resources/efforts (62%), while the top use cases cited are test reporting (56%), defect analysis (56%) and knowledge management (54%), test data generation (52%) and test automation script conversions (50%).

Overall, the survey finds that on average 44% of executives work for organizations that have adopted test automation. The biggest obstacles to adopting test automation are legacy IT architectures (64%), complex tooling (62%), a lack of strategy (57%) and lack of an orchestration framework (53%).

Tal Levi-Joseph, vice president of software engineering, research and development and product management for OpenText, said that while AI progress is being made, most DevOps teams are still a long way from fully realizing the benefits. Skeptical software engineering teams are still evaluating the levels of risk that AI might engender, she noted.

The DevOps ecosystem is highly fragmented, so organizations will continue to find it challenging to aggregate the type of data required to effectively train AI models, said Levi-Joseph.

Eventually, however, AI will drive more organizations to embrace integrated DevOps platforms that provide a better foundation for embedding AI within DevOps workflows, she added. That level of integration will be achieved mainly via a series of mergers and acquisitions that are already occurring, or alliances that more loosely couple tools and platforms from multiple vendors, said Levi-Joseph.

That’s crucial, because for AI to be effectively implemented there needs to be a feedback loop between the applications deployed in a production environment and the controls applied to improve quality, she noted.

Much like DevOps itself, each organization will need to determine for itself to what degree using AI to automate workflows makes the most sense. Clearly, there is an opportunity to reduce the current level of toil that over time conspired to burn out members of a DevOps team.

Longer term, there may even come a day when the quality of the code being generated by AI tools is equal, regardless of what level of software engineering expertise an organization has accrued, noted Lev-Joseph.

In the meantime, the most important thing when it comes to AI at this stage is to simply gain some experience on how to best apply it, both now and as it continues to evolve.

October 22, 2024
The Promise and Perils of Generative AI in Software Testing
The advent of generative AI (GenAI) is ushering in a new era in software testing. Acting as an AI test engineer, GenAI uses a test recorder designed to perform equivalent work to that of a human test automation engineer. It interprets plain-language instructions to autonomously generate test automation code. This capability democratizes testing, allowing individuals without coding expertise to directly interact with testing frameworks. It simplifies the process of test automation by enabling users to effortlessly generate code by recording their interactions with a website, with no manual coding required. GenAI-driven testers seamlessly integrate into CI/CD pipelines, autonomously detecting bugs and alerting teams about potential issues.

Transforming QA with AI

The journey from human automation tester to AI test automation engineer is transformative. Traditionally, transitioning to test automation required significant time and resources, including learning to code and understanding automation frameworks. AI removes these barriers and accelerates development cycles, dramatically reducing time-to-market and improving accuracy, all while decreasing the level of admin tasks for software testers.

AI-powered tools can interpret test scenarios written in plain language, automatically generate the necessary code for test automation, and execute tests across various platforms and languages. This dramatically reduces the enablement time, allowing QA professionals to focus on strategic tasks instead of coding complexities. This also addresses the severe shortage in test automation, helping companies achieve quality across their digital offerings in no time, for any number of scenarios in parallel, and in a matter of minutes. It creates test automation for multilingual websites and apps, as well as for any screen size, including computers, tablets and mobiles.

The productivity gains from AI in testing are substantial. While traditional test automation might be limited to a single platform or language and the capacity of one person, AI-enhanced testing breaks these limitations. Testers can now create and execute tests on any platform (web, mobile, desktop) using multiple languages, with the capacity of multiple testers. This not only amplifies testing capabilities but also introduces a new level of flexibility and efficiency.

The AI Advantage: Multilingual and 24/7 Testing

Upskilling QA teams with AI offers the significant advantages of multilingual testing and 24/7 operation. In today’s global market, software products often need to cater to diverse users, necessitating testing in multiple languages. AI facilitates this process without requiring testers to know each language, thereby expanding the reach and usability of software products.

The ability to test around the clock ensures that testing does not become a bottleneck in fast-paced development. Additionally, AI’s autonomous maintenance capabilities further reduce the time and effort needed for updating test cases, ensuring tests remain relevant and effective.

Navigating the Challenges

While GenAI offers significant advantages, it also presents operational and security challenges that organizations must address:
- Hallucinations: AI may generate inaccurate or fabricated outputs during testing, leading to incorrect results and potentially overlooking critical issues.
- Bias: AI systems can inherit biases from their training data, impacting testing outcomes and resulting in unfair testing scenarios and overlooked edge cases.
- Data Privacy: The risk of mishandling or leaking sensitive data used during testing raises significant privacy concerns.
- Lack of Transparency: The ‘black box’ nature of many AI systems makes it difficult to trace decision-making processes, hindering debugging and trust in the system.
- Security Vulnerabilities: GenAI systems are susceptible to adversarial attacks that could exploit system weaknesses, potentially compromising the entire testing process.
- Inconsistent Outputs: AI might produce erratic or irrelevant results, affecting test reliability and making it challenging to maintain consistent testing standards.
Strategies for Risk Mitigation

To harness the power of GenAI while mitigating these risks, organizations can implement several strategies:
- Human-in-the-Loop (HITL) Supervision: Incorporating human oversight ensures that AI-generated outputs are rigorously validated for accuracy and reliability. Human supervisors can review and approve AI-generated test cases, ensuring they meet necessary standards before implementation.
- Restricting AI Autonomy: Limiting the AI’s creative freedom prevents the system from making unwarranted assumptions or actions. Setting clear boundaries and guidelines ensures that the AI operates within acceptable parameters, maintaining a predictable and reliable testing process.
- Requiring Reasoning for Actions: Enforcing a policy where AI must explain its decisions promotes transparency and builds trust in AI-generated results. By demanding reasoning for each action, developers can gain valuable insights into the AI’s thought process and make informed adjustments.
- Secure Data Management Practices: Implementing robust data management policies safeguards sensitive information from being misused during AI training. Encryption, anonymization and access controls are critical measures in protecting data privacy.
- Utilizing Diverse Training Data: Training AI on a wide-ranging dataset minimizes biases and enhances the AI system’s robustness. Diverse data exposure helps the AI generalize better and reduce the risk of biased outcomes. Regularly updating the training data to reflect current and comprehensive scenarios ensures that the AI remains effective and fair.
Harness the Power of GenAI-Driven Testing Practices

As GenAI becomes increasingly integrated into software development life cycles, understanding its capabilities and limitations is paramount. By effectively managing these dynamics, development teams can leverage GenAI’s potential to enhance their testing practices while ensuring the integrity of their software products. With careful consideration of the outlined challenges and mitigation strategies, organizations can harness the full power of GenAI to drive innovation in software testing and deliver high-quality software products.
October 2, 2024
Challenges in ETL Testing and How to Overcome Them

The linchpin in consolidating data from multiple sources into a unified repository is the Extract, Transform and Load (ETL) processes, which are foundational to data integration. Despite their critical role, ETL processes are prone to challenges, especially during the testing phase, when ensuring data quality, integrity and performance becomes paramount.

ETL testing is essential for identifying and rectifying errors, inconsistencies and inefficiencies before data is finalized for analytics and decision-making. This post delves into the common hurdles faced during ETL testing and provides actionable strategies to navigate these challenges effectively, ensuring a seamless, reliable data integration process.

Understanding ETL Testing

ETL testing is a crucial component of the data integration process, designed to validate the transformation, extraction and loading of data from diverse sources into a designated target database or data warehouse. Its primary objective is to ensure that the data transferred through the ETL pipeline is accurate, consistent and comprehensive. This involves verifying that all records are correctly extracted from source systems, transformations align with business rules and logic, and the final load into the target system precisely reflects the intended outcome without any data loss or corruption.

ETL testing is pivotal in maintaining data quality, which is fundamental for reliable analytics and business intelligence. It encompasses several vital activities, including data completeness checks, data transformation validation and end-to-end data flow verification, all aimed at identifying and mitigating data anomalies and ensuring the integrity and usability of critical business data.

Common ETL Testing Challenges

ETL testing faces several challenges that can complicate the data integration process, impacting the accuracy and reliability of business intelligence and analytics. These challenges include:

• Complex transformation logic: Validating the business logic that transforms data can be intricate, especially when dealing with complex rules and multiple data sources. Ensuring that all transformations are correctly applied requires a detailed understanding of the data and the business processes it supports.

• Data volume and scalability: With the exponential growth of data, testing ETL processes for scalability and performance under large volumes becomes a daunting task. Testers must ensure that the ETL process can handle the current data load and scale for future growth.

• Data quality issues: Inherent data quality issues such as missing values, duplicates and inconsistencies across source systems pose significant challenges. Identifying and addressing these issues during the ETL process is crucial to maintaining the integrity of the data warehouse.

• Integration with multiple data sources: The ETL process often involves integrating data from disparate sources, each with its format and standards. Ensuring seamless integration and consistent data representation across all sources requires meticulous planning and testing.

• Performance and optimization: Testing for performance bottlenecks and optimizing the ETL process for speed and efficiency is critical, especially for real-time data processing needs. Identifying the optimal balance between performance and resource utilization is a crucial challenge.

Navigating these challenges successfully demands a strategic approach to ETL testing, encompassing thorough planning, advanced testing techniques and a deep understanding of the data and the business context it serves.

Strategies to Overcome ETL Testing Challenges

Overcoming the challenges in ETL testing demands a comprehensive strategy that ensures data integrity, performance, and scalability. Here are several effective strategies:

• Automated testing tools: Automating the ETL testing process can significantly enhance efficiency and accuracy. Tools that support automated data validation and comparison can quickly identify discrepancies and errors, reducing manual effort and the likelihood of oversight.

• Data profiling and quality tools: Early use of data profiling tools helps identify data quality issues such as inconsistencies, duplicates and outliers. Incorporating data quality tools to cleanse and standardize data before it enters the ETL pipeline ensures a cleaner transformation process and higher quality outputs.

• A modular testing approach: Breaking the ETL process into smaller, manageable modules or components allows for more focused and efficient testing. This approach enables testers to isolate and resolve issues more effectively, ensuring that each part of the process functions correctly before proceeding to the next.

• Performance benchmarking: Establishing performance benchmarks for the ETL process helps identify bottlenecks and inefficiencies. Conducting load and stress testing under varying data volumes and scenarios ensures the ETL process can handle real-world demands.

• Continuous integration and testing: Implementing a continuous integration (CI) pipeline for ETL processes enables early detection of errors and integration issues. This approach allows for constant testing and validation of data as it moves through the ETL pipeline, facilitating immediate feedback and quicker problem resolution.

• Human collaboration: Encouraging collaboration between data engineers, testers and business analysts ensures a thorough understanding of the business logic and data quality requirements. This collaborative approach helps design more effective test cases and better understand the transformation logic.

By adopting these strategies, organizations can address the common challenges faced during ETL testing, leading to more reliable data integration efforts and, ultimately, more accurate and actionable business insights.

Best Practices for Effective ETL Testing

To ensure effective ETL testing, adopting a set of best practices is essential:

• Develop a comprehensive test plan: Start with a detailed test plan that outlines testing strategies, objectives, and criteria for each phase of the ETL process. This plan should cover everything from data source verification to transformation logic and load testing in the target database.

• Use realistic test data: Use a representative sample of production data covering various scenarios, including edge cases and data anomalies. This approach helps uncover potential issues affecting the data’s integrity and usability.

• Automate where possible: Leveraging automation for repetitive and data-intensive testing tasks increases efficiency and accuracy. Automated regression testing is precious for ongoing ETL processes undergoing updates or changes.
• Prioritize data quality: Incorporate data quality checks at each stage of the ETL process. This includes validating data completeness, accuracy, and consistency to ensure that the data meets the predefined standards.

• Foster team collaboration: Encourage open collaboration and communication between the development, testing, and business analyst teams. A unified approach ensures a shared understanding of the ETL objectives and enhances the overall quality of the testing process.

Adopting these best practices can markedly enhance the outcomes of ETL testing, resulting in more reliable, accurate and high-quality data for business intelligence and analytics.

Final Words

Integrating effective ETL testing into the data integration process is crucial for ensuring high-quality, reliable data outputs. Organizations can enhance their ETL testing efforts by addressing common challenges with strategic solutions and adhering to best practices, paving the way for insightful analytics and informed business decisions.

June 6, 2024
Tricentis Adds Additional AI Copilots to Test Automation Platform

Tricentis this week made generally available a copilot add-on for its Tricentis Tosca test automation platform. The add-on simplifies finding, understanding, and optimizing tests via a chat interface.

Mav Turner, chief product and strategy officer for Tricentis, said the company is adding copilots to each of its product offerings using large language models (LLMs) developed by Open AI. Previously, Tricentis launched Tricentis Testim Copilot; the company plans to add a Tricentis Copilot for Tricentis qTest shortly .

Over time, those AI assistants will be able to collaborate on tasks with each other and eventually with AI assistants provided by other providers of other platforms, including SAP, Turner added.

In the meantime, DevOps teams can use Tosca Copilot to find unused test cases, duplicate tests, unlinked assets, specific executions, and tests linked to application elements. Turner noted that the summarizations enabled by Tosca Copilot should make it simpler for DevOps teams to onboard additional testers.

DevOps teams can also make changes or modifications to any combination of those tests via a generative AI prompt rather than having to directly master the existing Tosca Query Language. That capability should also significantly reduce the time and effort currently required to troubleshoot test defects, said Turner.

Overall, Tricentis reports that time spent on complex testing activities can be cut in half by, for example, reducing the number of repetitive testing tasks previously required. In fact, Tricentis is already seeing a 16% to 43% reduction in test failure rates with Tricentis AI tools and up to a 50% increase in test case generation.

The goal is to not necessarily to increase the number of tests being run as much as it is to make sure the right tests are being conducted at the right time to improve application quality, said Turner.

In will, however, also become more feasible for any member of a DevOps team — including application developers — to iteratively generate a test. That’s better than having to wait on a dedicated testing team to create a test. Generative AI should also make it easier for DevOps teams to reuse tests, run tests faster, generate fewer errors, reduce costs and increase overall productivity.

It’s not clear yet to what degree generative AI will democratize application testing, but as tests become easier to create there should be more time to run a wider range of tests that might, for example, address cybersecurity issues. Most of the tests that are run today typically surface common programming mistakes and there is still a tendency to skip testing any time an application development projects starts to fall behind schedule.

However, as testing becomes faster in the age of AI there should be more time to run a wider range of tests. The challenge is finding the best way to orchestrate those tests across a portfolio of applications that is only going to become larger as AI tools make it simpler for developers to write code faster.

June 5, 2024
Applying AI/ML to Continuous Testing

Artificial intelligence (AI) and machine learning (ML) can play a transformative role across the software development lifecycle, with a special focus on enhancing continuous testing (CT). CT is especially critical in the context of continuous integration/continuous deployment (CI/CD) pipelines, where the need for speed and efficiency must be balanced with the demands for quality and security. AI/ML contributes by automating complex tasks, predicting potential issues before they occur and providing actionable insights, thereby reducing manual efforts and enabling more strategic use of human resources.
Moreover, the application of AI/ML extends beyond mere automation of tests. It encompasses the capability to learn from data, adapt to new information, and improve over time. This ability is invaluable for identifying patterns, anticipating vulnerabilities and optimizing testing strategies. In quality assurance, AI-driven tools can predict areas most likely to fail and tailor testing efforts accordingly. In security, ML algorithms can detect anomalies that signify potential threats, while in operations, AI can enhance feedback mechanisms, leading to more resilient and responsive systems.

AI/ML for Continuous Testing

Here’s how AI technologies can be applied to reduce bottlenecks associated with testing activities across various testing activities:

1. Requirements Analysis
• Explanation: Ensures test scenarios align with business requirements and user needs.
• Bottleneck: Misinterpretation or incomplete analysis can lead to inadequate test coverage.
• AI/ML Solution: NLP can automate the extraction and interpretation of requirements, ensuring comprehensive and accurate test coverage.

2. Test Strategy
• Explanation: Outlines the testing approach, objectives, and resources.
• Bottleneck: An unclear strategy may lead to inefficient testing efforts and resource allocation.
• AI/ML Solution: AI can analyze historical data to suggest the most effective test strategies and predict resource needs.

3. Test Plans
• Explanation: Detailed documents guiding the testing process, timelines, and responsibilities.
• Bottleneck: Inflexible plans can struggle to adapt to project changes, causing delays.
• AI/ML Solution: Machine learning algorithms can suggest adjustments to test plans based on ongoing project developments and past outcomes.

4. Test Cases
• Explanation: Specific conditions under which a test is executed.
• Bottleneck: Time-consuming development and significant effort to maintain.
• AI/ML Solution: AI can automate the generation of test cases from requirements documents, improving efficiency and coverage.

5. Test Scripts
• Explanation: Automated scripts that execute test cases.
• Bottleneck: Script development and maintenance can be resource-intensive.
• AI/ML Solution: AI can generate and update test scripts based on changes in the application or test cases, reducing maintenance effort.

6. Test Data
• Explanation: Data sets used during testing to simulate real-world scenarios.
• Bottleneck: Creating, managing, and maintaining accurate test data is challenging.
• AI/ML Solution: AI can automate the generation and management of test data, ensuring relevance and variety.

7. Test Environment
• Explanation: The setup where testing is conducted mirrors production environments as closely as possible.
• Bottleneck: Configuration and maintenance of test environments are complex.
• AI/ML Solution: AI can predict and configure optimal test environments based on test requirements, reducing setup time.

8. Coordination with Dependent Systems
• Explanation: Ensuring the system under test interacts correctly with databases and other applications.
• Bottleneck: Dependency management can cause delays.
• AI/ML Solution: AI can automate the detection and resolution of integration issues, enhancing coordination efficiency.

9. Test Environment Setup
• Explanation: Configuring infrastructure and tools necessary for testing.
• Bottleneck: Setup complexity and resource contention lead to delays.
• AI/ML Solution: AI algorithms can optimize environment setup, automatically adjusting resources as needed.

10. Test Campaign Setup
• Explanation: Organizing and scheduling a series of test executions.
• Bottleneck: Requires careful planning and can be hindered by resource limitations.
• AI/ML Solution: AI can assist in scheduling and prioritizing test campaigns based on risk and impact analysis.

11. Test Execution
• Explanation: The process of running test cases and scripts, both automated and manual.
• Bottleneck: Time-consuming, particularly for manual tests.
• AI/ML Solution: AI can prioritize test execution and identify flaky tests, streamlining the process.

12. Test Verdict Reporting
• Explanation: Determining and reporting the outcome of test executions.
• Bottleneck: Manual verdict determination can be slow.
• AI/ML Solution: AI can automatically interpret test outcomes, speeding up reporting.

13. Logging of Data
• Explanation: Recording data relevant to the test for further analysis.
• Bottleneck: Extensive data collection can overwhelm resources.
• AI/ML Solution: AI can intelligently filter and log pertinent data, reducing noise.

14. Test Result Analysis
• Explanation: Analyzing test outcomes to identify defects and issues.
• Bottleneck: Requires significant time and expertise.
• AI/ML Solution: ML algorithms can quickly identify patterns and anomalies in test results, highlighting potential issues.

15. Test Result Reporting
• Explanation: Communicating findings to stakeholders.
• Bottleneck: Compiling reports is time-intensive.
• AI/ML Solution: Automated reporting tools powered by AI can generate insightful and comprehensive reports quickly.

16. Waiting for Resource to Fix Failed Tests
• Explanation: Downtime while awaiting fixes for identified issues.
• Bottleneck: Halts testing progress.
• AI/ML Solution: AI can predict which areas might fail and propose potential fixes

Challenges

Applying AI/ML to software testing activities offers numerous advantages but also introduces several challenges. Addressing these problems requires a combination of technical solutions, process adjustments, and cultural changes.

1. Lack of Intuitive Understanding of the Application Being Tested
• Problem: AI/ML models may not fully grasp the application’s context or the nuances of its functionalities, leading to less effective test scenarios.
• Solution: Enhance AI models with richer contextual data and incorporate feedback loops where testers can refine and adjust AI-generated test cases. Employing techniques like reinforcement learning can also help AI models better understand application contexts over time.

2. Repeatability and Consistency Between Test Sessions
• Problem: AI-driven tests may generate different outputs for the same input over different sessions, complicating test consistency and traceability.
• Solution: Implement versioning for AI models and their training data, ensuring consistency across test sessions. Use deterministic approaches in conjunction with AI to maintain a core of stable, repeatable tests.

3. Lack of Understanding of the Tests Generated
• Problem: Testers may find it challenging to understand or trust the rationale behind AI-generated test cases, impacting their ability to evaluate test outcomes effectively.
• Solution: Incorporate explanations into AI/ML models to provide insights into their decision-making processes. Foster a culture of trust and understanding through education and transparency regarding how AI models operate.

4. Test Coverage
• Problem: There’s a risk that AI/ML might not adequately cover all testing scenarios, potentially missing critical defects.
• Solution: Combine AI/ML with traditional testing methods to ensure comprehensive coverage. Regularly review and adjust the criteria used by AI/ML models to generate test cases, ensuring they align with evolving application features and risks.

5. Compatibility with Different Test Tools
• Problem: AI/ML models might not seamlessly integrate with existing testing tools and frameworks, limiting their utility.
• Solution: Develop or use AI/ML solutions with extensive API support and integration capabilities. Work with tool vendors or contribute to open source projects to enhance compatibility.

6. Acceptance by Teams
• Problem: Testers and developers may be skeptical or resistant to AI-driven testing due to concerns about job displacement or mistrust in AI’s effectiveness.
• Solution: Educate and involve teams in the development and implementation of AI/ML testing strategies. Demonstrate the value of AI/ML in augmenting their roles rather than replacing them, focusing on AI as a tool to tackle mundane tasks and allowing them to focus on more complex and rewarding work.

7. Data Quality and Availability
• Problem: AI/ML models require large amounts of high-quality data for training. Inadequate or poor-quality data can lead to ineffective testing.
• Solution: Invest in data curation and generation strategies such as synthetic data creation to ensure models are well-trained.

8. Continuous Learning and Adaptation
• Problem: AI/ML models may become outdated as applications evolve.
• Solution: Establish continuous learning mechanisms where models are regularly updated with new data and feedback, ensuring they remain relevant and effective.

9. Ethical and Bias Considerations
• Problem: AI/ML testing models might inherit or amplify biases present in their training data, leading to unfair or discriminatory outcomes.
• Solution: Implement ethical guidelines and bias detection methodologies for AI/ML model development and use. Regularly audit models for biases and correct them as needed.

By addressing these challenges with thoughtful strategies, organizations can maximize the benefits of AI/ML in testing activities while mitigating potential drawbacks, leading to more efficient, effective and trustworthy processes.

Summary

This blog explained use cases of artificial intelligence (AI) and machine learning (ML) within the realm of continuous testing of the software development life cycle. By harnessing the power of AI/ML, we’ve seen how organizations can significantly enhance their development processes, making them more efficient, secure and responsive to user needs. The exploration of specific AI/ML applications across various activities has provided a blueprint for automating and optimizing tasks that traditionally required significant manual effort.

March 12, 2024
Cycode Discloses GitHub Actions Vulnerability in Google Bazel Project

Cycode today revealed it discovered a command injection vulnerability in the way the GitHub Actions continuous integration/continuous delivery (CI/CD) platform was used to update the open source Bazel project that many organizations employ to build and test software.

The vulnerability, now remediated, could have been used to create a backdoor through which malicious code could have been inserted into a codebase that is managed by Google.

Cycode researchers shared their discovery with Google, which has since updated the way it employs GitHub Actions to manage the Bazel codebase. Specifically, Google updated the base permissions to embrace least privilege principles to resolve the vulnerability discovered by Cycode.

The vulnerability was discovered using RAVEN, an open source tool for scanning CI/CD platforms for vulnerabilities that Cycode makes available.

Elad Pticha, senior security researcher for Cycode, said the vulnerability is the latest example of an instance of an open repository for building software that has been shown to be potentially vulnerable to an attack on a software supply chain that could theoretically impact any number of downstream applications and projects, including Kubernetes, Angular and platforms and services provided by Uber and LinkedIn.

A Custom Actions capability that enables application development teams to create their own workflows presents any cybercriminal attempting to compromise of software supply chain with a tempting target, noted Pticha. A few changes to lines of code in a workflow can translate into thousands or even millions of lines of code, many of which an organization may never detect. Out of 3.4 million workflows examined by Cycode, nearly all (99%) make use of one or more Custom Action.

Cycode has previously identified dozens of vulnerabilities in open source projects by mainly focusing on vulnerabilities within workflows. This latest research shows those vulnerabilities extend to indirect dependencies, such as Custom Actions, noted Pticha.

Each DevOps team should review its own workflows for similar vulnerabilities at a time when more cybercriminals than ever are intent on compromising software supply chains. While many organizations have embraced DevSecOps workflows to improve application security, there is a natural tendency to overlook vulnerabilities that might exist in the workflows used to build and deploy software. The challenge is that DevSecOps workflows are still immature. A recent survey of 500 U.S. CISOs, AppSec directors and DevSecOps team members conducted by Cycode found 78% admitted application security attack surfaces are unmanageable, with 90% acknowledging relationships between their security and development teams need to improve.

It’s not apparent if the vulnerability discovered by Cycode was actually ever exploited, but as most cybersecurity professionals know, if it can be imagined, someone has tried it. As such, any organization that uses Bazel might be well-advised to conduct a code review.

In the meantime, it’s already apparent that regulations pertaining to securing software supply chains are only going to become more stringent in the months ahead. In the wake of the executive order from the Biden administration requiring federal agencies to lock down their software supply chains, it’s now only a matter of time before similar requirements are baked into legislation and various other regulations are implemented. The challenge and the opportunity now is to get ahead of this issue before a mandate with a hard deadline that can’t be ignored winds up disrupting the entire software development life cycle.

February 1, 2024
Coexisting With AI: The Future of Software Testing

If 2023 was the year of artificial intelligence (AI), then 2024 is going to be the year of human coexistence with the technology. Since the release of Open AI’s ChatGPT in November 2022, there has been a steady stream of competing large language models (LLMs) and integrated applications for specific tasks, including content, image processing and code production. It’s no longer a question of if AI will be adopted; we have moved on to the question of how best to bring this technology into our daily lives. These are my predictions for the software quality assurance testing industry for 2024.

Automated testing will become a necessity, not a choice.

Developers will lean heavily on AI-powered copilot tools, producing more code faster. That means huge increases in the risk profile of every software release. In 2024, testers will respond by embracing AI-powered testing tools to keep up with developers using AI-powered tools and not become the bottleneck in the software development life cycle (SDLC).

The role of the tester will increase and evolve.

While AI is helping software engineers and test automation engineers produce more code faster, it still requires the highly skilled eye of an experienced engineer to determine how good and usable the code or test is. In 2024, there will be a high demand for skilled workers with specific domain knowledge who can parse through the AI-generated output and determine if it’s coherent and useful within the specific business application. Although this is necessary for developers and testers to start trusting what the AI generates, they should be wary of spending inefficient amounts of time constructing AI prompts, as this can ultimately lead to decreased levels of performance. For instance, a developer could easily spend most of their time validating the AI-generated output instead of testing the release that will be deployed to users.

Being able to distinguish between when to rely on AI and when to forego AI’s help will be key to streamlining the workflow. Eventually, we’re going to start seeing AI-powered testing tools for non-coders that focus on achieving repeatability, dependability and scalability so that testers can truly use AI as their primary testing tool and ultimately boost their productivity.

The rise of protected, offline LLMs and the manual tester.

As enterprise companies show signs they don’t trust public LLMs (e.g., ChatGPT, Bard, etc.) with their data and intellectual property (IP), they’re starting to build and deploy their own private LLMs behind secured firewalls. Fine-tuning those LLMs with domain-specific data (e.g., banking, health care, etc.) will require a great volume of testing. This promises a resurgence in the role of the manual tester as they will have an increasingly important role to play in that process since they possess deep domain knowledge that is increasingly scarce across enterprises.

As we stand on the brink of 2024, it is evident that the synergy between artificial intelligence and human expertise will be the cornerstone of software quality engineering. Human testers must learn to harness the power of AI while contributing the irreplaceable nuance of human judgment. The year ahead promises to be one where human ingenuity collaborates with AI’s efficiency to ensure that the software we rely on is not only functional but also reliable and secure. There will likely be a concerted effort to refine these collaborations, ensuring that AI serves as a reliable support rather than an overpromised solution.

December 6, 2023
4 Mobile Software Testing Use Cases Where Device Clouds Don’t Work

A device cloud, which provides on-demand access to a variety of different types of mobile devices, may seem like the most efficient and effective way to run mobile tests–and it is, in many scenarios.

But depending on which types of mobile tests you need to run or how you need to run them, they aren’t always the best solution. If you blindly turn to device clouds by default to meet your test infrastructure needs for all use cases, you could end up shooting yourself in the foot.

To prove the point, I’d like to walk through four mobile testing use cases where you either fundamentally can’t use a device cloud or where the disadvantages of cloud-based mobile testing outweigh any benefits of the strategy.

What is a Device Cloud?

Before diving into a discussion of mobile testing scenarios where device clouds don’t make sense, let me make clear what I mean by “device cloud.”

I’m referring to a cloud-based mobile testing environment where developers and QA engineers can run software tests on remote devices. In other words, a device cloud is a “cloud” of mobile devices you can access on demand for executing tests. Device clouds are an alternative to running tests on mobile devices that you have to set up locally and manage yourself.

They have become a popular solution for mobile testing because they offer several benefits compared to testing with local devices:

● Testers can easily test applications across hundreds or thousands of different mobile devices without having to purchase and set up each device themselves.
● You typically pay only for the time during which your tests run, making device clouds a more cost-effective solution in many cases than testing on devices that you have to acquire locally.
● Because most providers add new types of mobile devices to their clouds on a regular basis, testing teams always have access to up-to-date mobile hardware.

A variety of vendors (including Kobiton, where I work) offer device clouds that deliver these benefits.

Why Device Clouds Don’t Always Work for Mobile Software Testing

Device clouds are great for use cases where teams need to test mobile apps across many mobile devices and don’t face special requirements. However, depending on your needs and priorities, they may not make sense for your testing routine.

Here are four scenarios where you can’t or shouldn’t use a device cloud for mobile testing.

1. Testing for infrastructure disruptions

Devices hosted in a device cloud operate under best-case conditions. They are fully charged, professionally managed devices that are connected to very stable, high-speed networks.

That’s good in the respect that it means the devices can run tests reliably. But it’s bad for teams who want to understand how their mobile software performs under less-than-ideal circumstances. If you want to know how Bluetooth interference or a spotty cellular connection could impact mobile app performance, for example, you won’t be able to simulate those conditions using a device cloud.

You can simulate these conditions by deploying devices locally in what QA professionals call an isolation chamber, where testers can introduce “chaos” like weak network connections to evaluate how the software responds.

2. Testing mobile apps with strict security requirements

Testing software on a device cloud requires you to upload your applications, along with any data that you need to feed into them for testing purposes, onto mobile devices owned by a third party.

In most cases, this isn’t a big deal. Providers secure their infrastructure and network connections to protect sensitive data.

But if you’re dealing with exceptionally rigid security requirements–as you might in highly regulated industries or in situations where you’re testing apps that manage very sensitive data–a device cloud might create too many security risks. You’re better off in that case testing on-premises, where you have full control over data and software.

3. Minimizing test execution time

Mobile software tests that take place on remote devices hosted in the cloud almost always take longer to complete than tests that run locally. The main reason why is that cloud-based tests require data to move over the Internet, which is slower than local network connections.

To be clear, we’re talking about delays of perhaps only 200 or 300 milliseconds. In most cases, that’s not a big deal. But if you require ultra-fast test execution, which is beneficial in scenarios where you need to run a large number of tests and you don’t want them to delay your software delivery pipeline, cloud-based testing might slow you down too much.

4. Testing on obscure devices

Major device clouds cover most of today’s most popular types of mobile devices, which makes them a great solution if you need to test on devices that are widely used by consumers.

But if you need to ensure that your apps run on more obscure devices–such as specialized IoT devices that you might use in your company’s facilities–you’re less likely to find them in a device cloud. You’ll need to acquire the specialized hardware and run tests on it locally.

Conclusion

Device clouds are a great way to meet garden-variety mobile testing requirements. However, not every mobile testing use case is simple enough to support using a device cloud. Sometimes, tests require a level of security, performance and control that only local mobile test environments can deliver.

October 12, 2023
Generative AI and Software Testing Tools to Manage Technical Debt

It is easy to take a pessimistic view of generative AI and large language models (LLMs), especially when a growing list of AI experts and public figures have voiced concern about their potential to lead to dire consequences—even to human extinction. But we must also consider how these same technologies might mitigate these risks. There is no doubt that we are living in an accelerated cycle of technological progress and repercussions. However, the intelligence, speed and agility of AI—and particularly machine learning (ML)—may be just what we need to roll with the punches.

As the speed of development outpaces our ability to mitigate potential risks posed by the systems we build, we are set on a path to accruing massive technical debt—the ever-increasing costs associated with modernizing code later rather than sooner. Too often, organizations in the private and public sectors are reluctant to pay off this debt, choosing an “if it ain’t broke, don’t fix it” mentality toward software maintenance and security. The problem is that as technology outgrows and outpaces our critical infrastructure systems, the potential for catastrophe escalates.

Machine Learning: A Tool for the New “Space Race”

Automated testing has proven superior in regression testing, especially in visually recognizing potential issues at the pixel level. If, for example, an important element of your interface is missing from the screen after an update, a visual test script can identify that faster than a human can. AI (and in particular, machine learning) generates test plans that provide more comprehensive coverage of the system under test and more practical coverage.

Traditional testing methods usually begin with a broad and hypothetical analysis of user interactions. Test engineers begin by casting a wide net to cover the many possible user engagement scenarios they imagine. In contrast, AI-assisted systems learn from actual user interaction, building script libraries that focus on known points of weakness and vulnerability. It also does this at speed and scale. This technology is going to be essential to software test engineers as they inevitably find themselves unable to keep up with the abundance of code generated from LLMs using traditional testing methods.

Technical Debt is Inevitable but not Unsolvable

Every new iteration of software has the potential to introduce unforeseen problems ranging from glitchy bugs to catastrophic failures. The risks of releasing new software vary immensely between applications: It’s one thing if the latest release of your music streaming app crashes and quite another if the system that runs the regional power grid fails. One will result in unhappy customers; the other, the potential loss of life.

The most critical systems (e.g., financial, transportation, safety, medical) are more likely to get frozen in time. Once a stable iteration is released, companies are generally not keen to make changes unless absolutely necessary—beginning the cycle of technical debt accumulation that grows until that debt is paid off by modernizing the code.

Deciding when to pay off technical debt has always been a matter of weighing pros, cons, costs and benefits. But sometimes, such as in the case of an epic failure, that decision is forced. Both the risk of breaking the system and the cost of modernization compel developers to leave well enough alone. Meanwhile, the longer a system remains unchanged, the more isolated, outdated, and obsolete it becomes, setting the stage for failure.

By introducing generative AI technology to the development stack along with other tools for authoring, compiling and testing software, developers will soon find themselves driving in the fast lane. With the security of knowing that code can be produced rapidly and released safely, owners of aging legacy systems will be more likely to modernize software sooner, helping them to pay off technical debt more quickly. That, in turn, could lead to increasingly rapid software development and release cycles, more work for developers, and more innovation. Such a workflow could very well become the embodiment of accelerating change, taking us from an “if it ain’t broke, don’t fix it” mentality to one where we anticipate the need for new code and avoid future catastrophic failures, or for that matter, human extinction.

September 12, 2023