devopsy.kinsta.cloud

Category: AI

Copado Applies Generative AI to Salesforce Application Testing

Copado today made generally available a generative artificial intelligence (AI) tool to automate testing of applications built for Salesforce software-as-a-service (SaaS) application environments.

Announced at a Salesforce World Tour event in New York, Test Copilot is based on an instance of the ChatGPT generative AI platform developed by OpenAI. Dubbed CopadoGPT, that platform is being embedded across the company’s DevOps platform to accelerate the building and deployment of custom applications on Salesforce’s various platforms. Test Copilot can also translate a recorded user session into a functional test script. Existing test scripts that need to be modified can also be pasted into the prompt.

David Brooks, senior vice president of evangelism for Copado, said Test Copilot will make it simpler to create tests that ultimately improve the quality of the applications being deployed. Test Copilot via a prompt can also convert existing tests, Selenium tests, or results from Copado Explorer into a reusable, maintainable robotic test. Test Copilot can also create test cases for application programming interfaces (APIs) to supplement user interface (UI) and end-to-end testing processes.

Test Copilot is an extension of Copado Explorer, a test automation platform the company launched last month. Its goal is to simplify testing custom applications more thoroughly, said Brooks. The “Holy Grail” is to reduce the technical debt any IT organization accumulates by ensuring that issues are resolved before applications are deployed, he said.

Eventually, the cost of iterating updates to applications will drop to the point where organizations can continuously update applications, both before and after deployment, noted Brooks.

It’s not clear what impact generative AI will have on testing and DevOps workflows. However, the level of toil most software engineering teams encounter is about to be sharply reduced. The challenge is to find the best way to operationalize generative AI platforms that are probabilistic. Not every prompt is going to generate the exact same output, so embedding generative AI across a team of software engineers working on the same application remains challenging.

Generative AI will also make it easier for stakeholders outside of the DevOps teams to participate in application testing. In effect generative AI will make it simpler to push testing responsibilities even further left beyond developers, noted Brooks.

In the meantime, however, DevOps teams should be identifying tasks that AI tools will soon automate. The way applications are built and deployed is about to fundamentally change, especially as more advanced reasoning capabilities are embedded into LLMs. In time, DevOps teams will find themselves invoking LLMs that have been trained to automate a range of complex tasks. Exactly how those advances will impact the way DevOps teams are structured remains to be seen but certainly, software engineering will never been the same again.

Photo credit: Joe William on Unsplash

April 25, 2024
AIOps Success Requires Synthetic Internet Telemetry Data

Any form of artificial intelligence (AI) is only as good as the data used to train it. If organizations expect to effectively apply AI to IT operations (ITOps) they need to collect as much telemetry data as possible.

IT teams frequently discover that their AIOps platform was been trained on a narrow base of telemetry data. It might have been collected from, for example, a DevOps platform that lacks complete visibility into the distributed computing environment where their application runs. In the absence of the synthetic telemetry data collected from an Internet performance monitoring (IPM) platform to collect that telemetry data, it’s simply not likely the machine learning algorithms that are at the core of any AIOps platform are going to surface the best recommendations for optimizing application experiences.

The challenge is the probabilistic nature of AI. The relevancy of the recommendations that surface is determined by the quality of the data that has been exposed to the AI model. Real user data, for example, may be sparse or non-existent. Of course, if telemetry data was never shared with the AI model in the first place, it’s extremely unlikely that the AI’s recommendations will to better application experiences.

Gaining Visibility Using Synthetic Data

IT teams need to be certain the data used to train the AI model reflects the production environments where applications are deployed. Otherwise, no matter how advanced an AI model may be, garbage in still equals garbage out. Before any IT team adopts an AIOps platform, it needs to know the data provenance of the underlying AI model. If the pool of AI training data is limited, the recommendations generated will be, too. IT teams are not going to put their faith in AIOps platforms that are advising them to take specific actions based on partial or incomplete data. Nor should they.

Instead, the teams will assume that every output needs to be verified before the next step in a process is allowed to proceed. After all, the only thing worse than being wrong when it comes to IT and AI is to be wrong at catastrophic scale. Of course, continuing to manage IT sequentially arguably defeats the purpose of investing in an AIOps platform that is supposed to manage tasks in parallel.

Given the dependency modern applications have on Internet services, any effort to apply AI to IT management that doesn’t include synthetic Internet telemetry data will lead to a suboptimal outcome. By including this type of telemetry, the insights being surfaced to DevOps teams will enable them to ensure key performance indicators (KPIs) are attained and maintained.

Multiple AI Models

There’s not likely to be one AI model to rule them all. In many cases, networking, security, and other IT service management (ITSM) platforms will have already applied AI to the telemetry data they collect in real time. The output from those AI models will then be shared with AIOps platforms to automate a series of tasks on an end-to-end basis that previously would require IT teams to orchestrate workflows across multiple islands of automation.

DevOps teams, as a result, need to evaluate the efficacy of what will soon be a network of AI models. There are so many, each of which is or will be designed to automate specific tasks, such as analyzing Internet traffic to identify the source of bottlenecks that might only intermittently impact an application. Armed with insights, it then becomes possible for the AIOps platform to consistently generate useful recommendations that DevOps teams can trust. Then they can let the tools automatically apply the suggestion to how, for example, Internet traffic should be rerouted to maintain service level objectives (SLOs).

Realizing the Promise of AI

As AIOps platforms improve, they can substantially reduce much of the toil that DevOps teams regularly encounter. Teams can spend weeks trying to determine the root cause of an issue that, once discovered, might take a few minutes to fix. The challenge is that the source of the issue often has little to do with anything the DevOps team has immediate control over, as is the case when, for example, latency created by an Internet service adversely impacts application performance. However, those insights should enable DevOps teams to send support requests that better pinpoint the exact source of an Internet service issue that their provider should be able to resolve faster. Just as importantly, the DevOps team can move onto issues that they have more direct control over.

Much of the stress that any DevOps team experiences stems from not knowing the real cause of issue that despite their best efforts continues to generate a series of ongoing alerts. AIOps promises to reduce that stress by making it simpler to first correlate causation and, secondly, automate remediation. That promise, however, will never be fully realized if the data being relied on to train the AI model doesn’t provide enough of the picture required to make a truly informed decision.

April 22, 2024
Tricentis Taps Generative AI to Automate Application Testing

Tricentis this week added a generative artificial intelligence (AI) capability, dubbed Tricentis Copilot, to its application testing automation platform. It aims to reduce the amount of code that DevOps teams need to manually create.

Tricentus Copilot is based on an instance of the large language model (LLM) created by Open AI which has been deployed on the Microsoft Azure cloud. It’s the first iteration in what will eventually become multiple AI assistants is Tricentis Testim Copilot, which makes it possible to use natural language to describe a test that is then automatically generated in JavaScript. Additional Tricentis Copilot solutions for the Tricentis Tosca and Tricentis qTest platforms will be added later this year.

Mav Turner, chief product and strategy officer for Tricentis, said the user interfaces that the company created for each platform will also continue to evolve, to make it simpler to launch the appropriate prompts during any workflow.

The goal is to make it easier to create tests to improve application quality, said Turner. For example, organizations that have been using the beta editions of the Tricentis Copilot have already seen a 20% to 50% increase in the number of tests created, while reducing test failure rates by 16% to 43% so far, he added.

There’s no doubt that generative AI makes it substantially easier to create tests in ways that should enable DevOps teams to run more tests across the entire software development life cycle (SDLC). Rather than always having to wait for dedicated testing teams to write the code required, it will be increasingly feasible for any member of a DevOps team to generate a test, including application developers, who should be testing code as it is iteratively created.

Generative AI should also make it easier for DevOps teams to reuse tests once created and understand how they are being run using the summarization capabilities enabled by generative AI platforms that will also be able to provide recommendations to improve testing code. Collectively, those capabilities will make it possible to run tests faster, have fewer errors, reduce costs and increase productivity.

Those capabilities can all be provided in a way that doesn’t result in any of the testing code being used to later train any update to the core LLM provided by OpenAI, noted Turner. Longer term, Tricentis will continue to research which LLMs might be best applied to any given use case in a way that is transparent to a DevOps team that might not be interested in which LLM is being employed to automate a task. Currently, however, it doesn’t make much economic sense for Tricentis to build its own LLM, said Turner.

It’s not clear yet to what degree generative AI will democratize application testing, but as they become easier to create there should be more time to run more complex tests that might, for example, address cybersecurity issues. Most of the tests that are run today typically surface common programming mistakes. However, as testing becomes faster in the age of AI there should be more time to run a wider range of tests to ensure the best application experiences possible.

April 20, 2024
GitLab Adds AI Chat Interface to Increase DevOps Productivity

GitLab this week announced the general availability of GitLab Duo Chat, a natural language interface through which DevOps teams can invoke generative artificial intelligence (AI) to generate code, create tests and access summarizations of code. It is available as an add-on to the GitLab Duo Pro AI framework, which the company is embedding across its DevOps platforms. GitLab Duo Chat can be invoked via the GitLab user interface or GitLab’s web integrated development environment (IDE) in addition to third-party IDEs, including VS Code and the IDE provided by JetBrains.

Over the last few years, GitLab has been working with Google to embed a range of AI capabilities into its core platform. With the addition of GitLab Duo Chat, these are now more accessible.

As part of that effort, GitLab is also now adding privacy controls for GitLab Premium and Ultimate customers. These enable organizations to control sensitive data at the project, group, and subgroup levels to address security and compliance concerns.

David DeSanto, chief product officer for GitLab, said as AI capabilities become more accessible, the pace at which applications can be built and deployed is about to significantly accelerate. The most immediate impact should be a reduction in the number of instances where builds fail to deploy because the quality of the code used to construct them has improved, he added.

AI should make it easier for subject matter experts who have little software engineering expertise to participate in the application development process, noted DeSanto.

The immediate challenge DevOps teams will encounter is to determine how best to employ the prompts to generate code and tests within the context of their existing DevOps workflows. Over time, however, as the reasoning engines embedded within the large language models (LLMs) that are at the core genAI platforms improve, it will become feasible to automate a series of actions involving multiple tasks, DeSanto noted.

In the meantime, DevOps teams should start determining which manual tasks might be automated using a generative AI platform. In many organizations, there are still numerous bottlenecks in DevOps workflows that conspire to slow down the pace of application development.

AI platforms are unlikely to eliminate the need for application developers and software engineers any time soon. However, many scripts that DevOps teams have historically written themselves will soon be written and tested by machines. DevOps teams will soon find themselves supervising the development of more applications that, in addition to being constructed faster than ever, are less troublesome to update and maintain without having to add a small army of software engineers. The tradeoff is the overall size of the codebases that DevOps teams will be expected to manage is about to substantially increase, noted DeSanto.

In an era where organizations are more dependent than ever on software, the demand for applications already exceeds the ability of DevOps teams to address. AI should enable DevOps team to build and deploy applications at a level of scale that only a couple of years ago would have seemed unimaginable.

Photo credit: Ondrej Supitar on Unsplash

April 19, 2024
The Role of AI in Securing Software and Data Supply Chains

Supply-chain vulnerabilities loom large on the cybersecurity landscape, with threats and attacks such as SolarWinds, 3CX, Log4Shell and now XZ Utils underscoring the potentially devastating impact of these security breaches. The latter examples of Open Source Software (OSS) attacks are a growing attack vector. Capterra’s research reveals that software supply-chain attacks impacted almost two-thirds (61%) of all U.S. businesses from April 2022 to April 2023.

Expect attacks on the open source software supply chain to accelerate, with attackers automating attacks in common open source software projects and package managers. Many CISOs and DevSecOps teams are unprepared to implement controls in their existing build systems to mitigate these threats. In 2024, DevSecOps teams will migrate away from shift-left security models in favor of “shifting down” by using AI to automate security out of the developers’ workflows.

Here are the factors fueling the increase in software supply-chain attacks and the role of AI in helping developers work more efficiently while creating more secure code.

Attacks on the Open Source Software Supply Chain Will Accelerate

Open source libraries and languages are the foundation of over 90% of the world’s software. In a U.S. survey of nearly 300 IT and IT security professionals, 94% reported their companies use open source software, and 57% employ multiple open source platforms. Exactly half of the respondents say the threat level is “high” or “extreme,” while another 41% view it as “moderate.” At the time of writing, details of the backdoor implanted in the XZ library and several other OSS packages have just been published. The ubiquity of open source worldwide is one key factor fueling the rise of supply chain attacks.

Data Governance and the Data Supply Chain Will Become Critical Issues

Security pros must also consider how security vulnerabilities extend to their data supply chains. Although organizations typically integrate externally developed software through their software supply chains, their data supply chains often need clearer mechanisms for understanding or contextualizing data. In contrast to software’s structured systems or functions, data is unstructured or semi-structured and faces a wide array of regulatory standards.

Many companies are building AI or ML systems on top of enormous data pools with heterogeneous sources. ML models on model zoos are published with minimal understanding of the code and content used to produce the models. Software engineers need to handle these models and data just as carefully as they do the code going into the software they’re creating, with attention to its provenance.

DevSecOps teams must assess the liabilities of using data, especially when building LLMs to train AI tools. That demands careful data management within models to prevent the accidental transmission of sensitive data to third parties like OpenAI.

Organizations should adopt strict policies outlining the approved usage of AI-generated code, and when incorporating third-party platforms for AI, conduct a thorough due diligence assessment ensuring that their data will not be used for AI/ML model training and fine-tuning

AI Security Automation Will Help Organizations Transition From ‘Shift-Left’ to ‘Shift-Down’

The industry adopted the shift-left concept a decade ago to address security flaws early in the software development lifecycle and to enhance developer workflows. Defenders of systems have long been at a disadvantage. AI has the potential to level the playing field. As DevSecOps teams navigate the intricacies of data governance, they must also assess the impact of the evolving shift-left paradigm on their organizations’ security postures.

Companies will begin moving beyond shift-left to embrace AI to fully automate security processes and remove them from the developer’s workflow. This is called “shifting-down,” because it pushes security into automated and lower-level functions in the tech stack instead of burdening developers with complicated and often difficult decisions.

GitLab’s Global DevSecOps Report: The State of AI in Software Development found that developers only spend 25% of their time on code generation. AI can elevate their output by optimizing the remaining 75% of their workload. That’s one way to leverage AI’s capacity to solve specific technical issues and improve the efficiency and productivity of the entire software development life cycle.

Expect to look back on 2024 as the year that the escalating threats OSS ecosystems adversely affecting global software supply chains, catalyzed substantial changes in cybersecurity strategies, including a heightened dependence on AI to safeguard digital infrastructures. The cybersecurity landscape is already transforming, with a growing focus on mitigating supply chain vulnerabilities, enforcing data governance and incorporating AI into security measures. This transformation promises to steer DevSecOps teams toward software development processes with efficiency and security at the forefront.

April 19, 2024
Exploring Low/No-Code Platforms, GenAI, Copilots and Code Generators

The emergence of low/no-code platforms is challenging traditional notions of coding expertise. Gone are the days when coding was an exclusive skill set reserved for a knowledgeable few. Low/no-code platforms have democratized software development. They empower individuals from non-IT or technical backgrounds to translate their business ideas into applications without the need to master complex programming languages.

These low/no-code platforms, with their user-friendly drag-and-drop interfaces and pre-built components, have simplified application development. Imagine building a smart home application with just a few clicks and drags — low/no-code platforms are making it possible.

Add generative artificial intelligence (Gen AI) into the mix, and the new development practices change the field remarkably.

The Impact of Gen AI on Low-Code Platforms

Expect low-code platforms to evolve further, becoming more intelligent, adaptable and seamlessly integrated into the software development lifecycle. There will be a shift towards platforms that use Gen AI to understand user intent, automate routine tasks and generate complex code snippets.

Gartner predicts that, by 2026, over 80% of enterprises will use GenAI APIs and models or deploy GenAI-enabled applications in production environments, up from less than 5% in early 2023.

Combining low-code and Gen AI will empower developers by automating routine tasks and allowing them to focus on high-level logic and creativity. Gen AI acts as a multiplier that can accelerate and optimize developmental operations.
McKinsey estimates that generative AI could add $2.6 trillion to $4.4 trillion annually to the global economy by 2040.

Another Tool: AI Copilots and Code Generators

Another promising application for AI large language models (LLMs) is AI-powered assistants. Enter copilots and code generators — AI assistants that take the hassle out of coding.

While low/no-code platforms have transformed application development, copilots and code generators add an extra layer of efficiency.

AI copilots use a chat interface to simplify coding through a chat interface, where developers articulate their code requirements in natural language prompts, and the copilot generates the code accordingly. These assistants provide real-time assistance, offering contextual suggestions and code snippets, and they can anticipate potential bugs. Moreover, they can explain code blocks, generate unit tests and propose bug fixes, which enhances the coding process.
AI copilots are like trusted assistants, ready to aid you with specific coding tasks or generate complex code – and they do it swiftly.

The Big Question

As a result of low/no-code platforms gaining traction, the skill sets required in the tech industry will transform. Gone are the days when proficiency in multiple programming languages was a must-have.

With low/no-code tools taking center stage, does this mean coding is becoming obsolete?

A big no.

While basic programming knowledge remains valuable, it’s no longer the sole requirement for building innovative applications. With low/no-code platforms democratizing application development, individuals from diverse backgrounds can leverage technology to create impactful solutions.

Do GenAI, copilots, and code generators render low/no-code platforms obsolete?

Again, no.

While GenAI, copilots, and code generators excel at certain specific tasks, low/no-code platforms offer a holistic solution, combining visual interface design, database integration and more. They’re the Swiss army knives of development, catering to a wide range of needs beyond just code generation.

Gen AI, copilots and code generators are not here to replace low/no-code platforms or traditional coding practices. Instead, they are positioned to complement and enhance them. The fusion of these technologies holds great promise in simplifying software development, boosting efficiency, and unlocking new avenues for innovation.

The Final Word

With better code suggestions and automated testing capabilities, low/no-code platforms deliver high-quality solutions that meet the demands of today’s tech-savvy users.

With the thrilling combination of low/no-code platforms and GenAI, the future of software development is promising. While these advancements may disrupt traditional coding, they also present opportunities for innovation and creativity.

It’s time for us to embrace this change and build a brighter tomorrow, one line of code — or lack thereof — at a time.

April 18, 2024
Survey Surfaces Widespread Reliance on Generative AI Among Developers

A survey of 885 developers conducted by Docker, Inc. finds reliance on artificial intelligence (AI) to help build applications is rapidly becoming pervasive.

Nearly two-thirds of respondents (64%) are using AI for coding (33%), documentation (29%), conducting research (28%), writing tests (23%) and troubleshooting/debugging (21%). The most widely used generative AI platforms are ChatGPT (46%), GitHub Copilot (30%), and Google Gemini (19%).

Overall, 65% of the developers surveyed said they view AI positively because it simplifies processes (61%) and enables them to focus more on crucial tasks (55%). Less than a quarter (23%) view AI as a threat to their job.

However, nearly half (45%) said there is too much emphasis on AI. Nevertheless, 40% of respondents identified generative AI as the most important trend in software development, followed closely by AI assistants for software engineering (38%).

Dr. Julia Wilson, user experience researcher for Docker, Inc. noted that more senior and full-stack developers tended to view GenAI as most important, whereas more junior developers with less than five years of experience viewed AI assistants for software engineering as being more important.

It’s still early days when it comes to applying AI to software engineering but it’s clear there are plenty of use cases. Most of that usage currently benefits developers. However, as reasoning engines embedded within large language models (LLMs) become more advanced, it’s now only a matter of time before generative AI platforms can orchestrate a series of tasks that previously would require DevOps teams to write scripts to programmatically perform. Instead, the generative AI platform will create the required scripts required. Those capabilities will serve to eliminate a lot of the toil that ultimately conspires to burn out members of a DevOps team, noted Wilson.

It remains to be seen, however, what impact generative AI will have on how DevOps teams are structured. In theory, an existing DevOps team augmented by AI should be able to build, update and manage a lot more applications. In many cases, the need for entry-level developers might be sharply reduced or, conversely, it may turn out that the junior developers can manage tasks that previously required a senior developer. At the same time, AI may ultimately make DevOps more accessible to a wider range of organizations that previously lacked the resources required to hire and retain software engineers.

The one certain thing is AI is not going back into the proverbial bottle. In fact, the next generation of generative AI platforms will be trained on a narrow set of code that has been vetted for quality and security. That approach should reduce the number of instances where a generative AI platform “hallucinates” in a way that produces code that simply doesn’t work or is rife with known vulnerabilities. General-purpose LLMs such as ChatGPT were trained using code examples pulled from across the Web so the current level of the code being generated is inconsistent.

As a result, one short term concern DevOps teams should have is whether developers become too dependent on generative AI platforms that, by definition, are probabilistic rather than deterministic. In fact, it’s often challenging for a generative AI platform to consistently surface the same code when responding to identical prompts.

As is the case with any major innovation, the challenge is making sure that AI is used both responsibly and safely anytime it is being used to generate code.

Photo credit: Thomas Lefebvre on Unsplash

April 16, 2024
Your AI Might be Lying to You

Simple tests can demonstrate how an AI can respond to and modify responses — a useful exercise for developers and DevOps who want to use these tools for activities like code or script generation.

While I was playing with generative AI for a hobby project – as most of us have – I discovered that in a few cases, it was stubbornly against following my explicit instructions. It wasn’t anything controversial. It was just that I was using a system that used math one way, and the AI was insistent in presenting results of the same category from a different subsystem. Even when told the steps to generate valid results, the AI tool got hung up on the name of the subsystem and “substituted” an invalid subsystem of the same name. It would say “Oh yeah, got you. Here: “ and then present what I’d just told it was factually incorrect. So I started pushing the edges and found a large number of instances where this type of confusion occurred.

In my hobby, I can take the time and calculate the bit I need separately, then insert it. In a professional setting, that is less than optimal. I mean, if we cut a dozen hours off a project and have to invest an hour fixing bits like this, it is still a massive win. The problem is that we have to know about the issue in order to fix it. But the system – where most of these issues are concerned – blindly does it, and doesn’t tell you. That’s so even where it knows it is not obeying your instructions.

Sometimes it’s okay to agree with the forced decisions of the AI’s designers. But an employee wouldn’t have a job long if, when told, “Do this task in exactly this manner” came back and said, “Yeah, your instructions didn’t fit my world-view, so I did something else; look at this great solution!” AI only gets a little more patience as long as it allows you to correct it quickly. And so far in our tests (except for the hobbyist issue above), we could convince it to do what we said, not what the AI designers said was “better.”

So here is the minimum test you need to run on any code generator to determine if it is doing what you ask, or what its AI designers programmed it to do.

Ask your AI to generate a simple dynamic website, in your language of choice. In one iteration, ask it to optimize for lines of code. Most will not; they know better, they think. Then after code generation, ask the tool if this is the fewest lines of code that solve the problem in that language. It will then give you the answer you actually asked for. Now repeat this exercise for readability. Use specific languages, frameworks, and libraries you might have as standards.

For simple problems, generative AI does generate usable code. I see AI in coding as being like an extension of libraries: a quicker way to solve common problems. But we still need to be cautious. When one major generative AI tool was asked to generate that simple webpage, it gave us four source files and said this was “an” answer. I had asked for the most efficient answer, optimized for minimum lines of code. As you might guess, we laughed at four source files to throw up a webpage that accepts a name and spits out “Hello [[name]]”. (This was our simple, dynamic test case). When I asked it, “Is this the fewest lines of code?” the tool promptly presented me with the answer you would expect. It was a single file, and then it scolded me that this was not the most readable. I never once mentioned readability to this tool, I mentioned optimized for minimum lines of code. But on the third iteration, it gave me the optimal answer for the dynamic example, and on the second the optimum answer for a simple webpage to display “hello world” non-interactively.

As I alluded to above, getting bad answers is not a deal breaker with AI, but you should be aware of the weaknesses in your chosen tool, and which tool you choose should certainly be informed about what level of manipulation the designers are engaged in, so staff can be on the lookout for suboptimal solutions (suboptimal in terms of organizational needs, which is the only “optimal solution” that matters).

So run that test along with your more functional tests, and keep kicking rear. If we can use generative AI for development and testing streamlining, we absolutely should, but we need to remember that it is just another automation tool, and sometimes less trustworthy than other tools.

April 8, 2024
Tabnine Extends Gen AI Platform for Writing Code to Multiple LLMs

Tabnine today revealed that it is giving DevOps teams the ability to switch between multiple large language models (LLMs) when they use its generative artificial intelligence (AI) platform to write code.

Application development teams can continue to use the LLM that Tabnine originally developed for its Tabnine Chat tool for generating code using a natural language interface. Alternatively, they can invoke either an instance of open source Mistral LLM jointly developed with Mistral AI or version 3.5 or 4.0 of the ChatGPT LLM (created by Open AI).

Tabnine president and chief marketing officer Peter Guagenti said the company will continue to add support for other LLMs, given the pace at which multiple LLMs advancements continues to accelerate.

As part of that effort, Tabnine will surface insights into LLM behaviors and their performance, privacy and protection behaviors, so development teams can identify which LLM is best for a specific use case, Guagenti added

Tabnine created an AI agent optimized for software development that optimizes prompts, provides contextual awareness of local and global code bases, and fine tunes LLM output.

There is, of course, no shortage of platforms for generating code, but the LLMs that Tabnine created are trained using a narrower base of code to reduce errors and hallucinations. However, development teams sometimes prefer to use LLMs that can support more parameters. Of course, the more parameters an LLM has, the more expensive it becomes, as the IT infrastructure resource requirement increases.

Regardless of the LLM type, the secret sauce is how they are invoked rather than in the LLMs that are rapidly becoming commodity services, noted Guagenti. Last month, for example, Tabnine made it possible for application development teams to better personalize output by exposing an LLM to both Tabnine’s code base and development environments.

It’s now only a matter of time before generative AI is applied across the entire software development life cycle (SDLC). In the short term, though, the pace at which code is being written should accelerate. The quality of that code will vary depending on the ways those LLMs were trained. A general-purpose LLM such as ChatGPT typically was trained using code examples of varying quality collected from across the Web. As such, that LLM is more likely to convincingly present code that doesn’t actually work, might run inefficiently, or contain vulnerabilities that could be exploited by cybercriminals.

The expectation is the output from LLMs that were been trained using vetted code will provide more reliable results. The challenge and the opportunity then becomes determining how best to apply AI to DevOps workflows before those teams are overwhelmed by the volume of code being generated by machines.

It may still be a while before AI is pervasively applied across the entire SDLC but the genie is out of the proverbial bottle. Eventually the overall code quality that makes it into production environments will improve steadily. In the meantime, DevOps teams should put guardrails in place to make sure the code AI is generating in the short term actually meets their requirements.

April 2, 2024
Survey Sees AI Playing Larger Role in Test Automation

A survey of 1,028 senior IT professionals and application developers in the United Kingdom, U.S., Germany and Singapore finds nearly half (47%) still use manual testing for mobile applications. More than a third (38%) noted that their organization could save between half and three-quarters (51-75%) of funding allocated to testing per year by fully automating mobile application testing.

Conducted by the market research firm Censuswide on behalf of Tricentis, a provider of a test automation platform, the survey also found that just under half of respondents (49%) reported their organization is already applying artificial intelligence (AI) to application testing. Another 21% plan to apply AI in the next six months. Almost three-quarters (74%) said their organization’s general sentiment toward AI in mobile software development testing is positive.

Respondents also expected AI to improve mobile application quality (37%), increase productivity (36%) enhance end-user experience (34%) and to reduce the number of vulnerabilities and/or bugs (32%) in their applications.

Mav Turner, chief product and strategy officer at Tricentis, said that rather than continuing to rely on expensive testing services, more organizations are looking to reduce costs by automating a wider range of testing workflows within the context of a DevOps workflow. In fact, the survey finds just under a third of respondents (32%) expect their organizations will invest more than $1 million to further automate testing.

As modernization of testing processes continues, the return on investment in mobile application development will only continue to improve, noted Turner.

The survey identifies the primary benefits of investing in test automation, which are increased productivity (32%), increased revenue (31%), increased user retention (28%) and increased compatibility across devices (28%).

The primary obstacles organizations encounter as they strive to achieve that goal are competing priorities (28%), lack of technology and talent (25%), time (25%) and cost (24%), the survey finds.

A full 87% said the quality of mobile applications is critical to their business, with 90% noting that poor mobile application quality costs their business as much as $2.49 million in lost revenue per year. However, just over a quarter (27%) said they believe their organization’s current mobile application development and testing strategy exceeds expectations.

As part of efforts to automate testing, it’s not clear just how far responsibility for testing is shifting left toward developers, but there is no doubt many more are being exposed to the results earlier in the software development life cycle. The goal is to provide continuous testing capabilities without overly disrupting code writing. It’s also becoming simpler for developers to leverage generative AI tools to provide summaries of tests along with recommendations to improve code. The end result should be higher-quality applications that require less troubleshooting once deployed in a production environment.

In the meantime, as organizations continue to become more dependent on mobile applications to drive a wide range of digital transformation initiatives, the cost of testing will continue to pale in comparison to the potential amount of lost revenue that might occur when end users either don’t invoke a specific feature or capability or, more commonly, simply abandon the application altogether.

March 29, 2024