devopsy.kinsta.cloud

Tag: GitHub Copilot

Coding at the Speed of AI: Innovation, Vulnerability, and the GenAI Paradox
Generative AI (GenAI) is reshaping how software is built. Tools like GitHub, Copilot, ChatGPT and Replit Ghostwriter have rapidly become indispensable in the modern development toolkit, promising increased throughput, reduced toil and faster time-to-market. They suggest code snippets, automate documentation, predict bugs, and even guide architectural decisions. We’re entering an era where developers don’t just write code; they collaborate with machines that do it for them.

But this speed comes at a cost: A rising wave of exploitable vulnerabilities baked into AI-generated code.

A paradox is emerging. The same tools enabling rapid innovation also reintroduce legacy vulnerabilities, spread insecure patterns, and inadvertently create fertile ground for attackers. As development cycles shorten, attackers are moving faster, too, using AI to scan for flaws and weaponize zero-days in record time.

This is not a call to abandon GenAI. Instead, it’s a call to reset expectations, policies and developer training to ensure that GenAI remains a co-pilot, not an autopilot, on the road to secure software.

The Rise of AI-Accelerated Development

GenAI has fundamentally altered the software development lifecycle. From automating repetitive tasks to suggesting code and generating documentation, these solutions can assist developers in their attempts to work faster and focus on high-value problem-solving. Organizations adopting GenAI-powered workflows often see productivity boosts and reduced burnout, particularly for junior developers who benefit from real-time guidance.

GenAI is becoming as essential as version control or CI/CD pipelines for many. But this normalization is where the risk begins to grow.

Blind Spots in the Code

AI-generated code often lacks context, particularly around security. These tools train on massive public datasets that include flawed, outdated, or insecure examples. As a result, GenAI may reproduce known bad practices without warning or the developer noticing.

Some of the most common vulnerabilities now appearing in AI-assisted code include:
- Cross-Site Scripting (XSS)
- Cross-Site Request Forgery (CSRF)
- Insecure deserialization
- Hardcoded credentials
- Open redirects
In some cases, GenAI tools have even reproduced critical, previously patched vulnerabilities. For example, code resembling the famous Log4Shell vulnerability (CVE-2021-44228) has surfaced in AI-generated outputs. While rare or accidental, their reappearance suggests a bigger issue in which GenAI lacks the “judgment” to avoid what it doesn’t understand.

And this is just the beginning. As attackers also leverage GenAI to chain vulnerabilities or create polymorphic malware, the time between a zero-day’s discovery and its exploitation continues to shrink.

The Illusion of Trust

A particularly dangerous assumption among developers is that AI-suggested code is “safe” because it’s syntactically correct or comes from a trusted tool. But code that compiles isn’t necessarily secure.

Because GenAI outputs often appear polished, developers may skip critical steps, code reviews, security audits, or proper documentation checks. This false sense of security is especially risky for junior developers or fast-moving teams under delivery pressure.

Even worse are AI hallucinations, known as syntactically valid outputs but semantically nonsensical. When this happens in a chatbot, it’s amusing. In production software, it’s a backdoor waiting to be exploited.

Best Practices for a GenAI World

Security teams and developer leads should resist the urge to treat GenAI as a plug-and-play solution. Instead, organizations must design policies and workflows that integrate GenAI safely and responsibly.

Recommended practices include:
- Always verify GenAI-generated code with linters, static analyzers, and security scanning tools.
- Cross-reference suggestions with official documentation or vetted code libraries.
- Avoid copy-pasting GenAI output into production environments without manual review.
- Incorporate secure coding training that explicitly addresses GenAI workflows.
- Include GenAI in your DevSecOps pipeline, with checkpoints for security and compliance.
Regulations are Catching Up

Recent policy changes highlight the urgency of responsible AI usage. The EU AI Act and U.S. Executive Order 14110 emphasizes human oversight and risk mitigation in AI-generated outputs, particularly in critical infrastructure and software systems.

These policies make it clear. AI cannot be a black box. Developers and security leaders must be able to explain, audit, and validate what AI creates. The burden is even greater for companies operating globally. Failure to implement oversight may lead to vulnerabilities and regulatory violations.

While AI is powerful, the human developer is still the ultimate arbiter of quality and safety. That means organizations must invest in training that teaches developers how to work with AI, not just beside it.

This includes:
- Understanding how GenAI models are trained—and their limitations.
- Spotting common security issues in AI-suggested code
- Practicing defensive programming and critical evaluation of suggestions
- Embedding security champions within development teams to coach and review GenAI usage.
We’re not trying to slow progress — we’re trying to steer it. Innovation without security is just acceleration toward risk.

Final Thought: GenAI is a Tool, Not a Teammate

The conversation around GenAI and security can’t wait. Ransomware actors and cybercriminals are adding AI to their tactics and arsenals. And many developers trust these tools without questioning their outcomes, unaware they can import the same vulnerabilities attackers are actively scanning.

It’s time to shift our mindset. GenAI isn’t a teammate; it’s a tool. Like any tool, it can be misused, misconfigured, or misunderstood.

Security remains a priority in software development lifecycles. Therefore, integrating oversight, education and security checkpoints into AI-assisted workflows can ensure that GenAI’s promise doesn’t come at the cost of safety.
August 28, 2025
GitHub Brings Together Security, Developers to Fix Code Flaws

GitHub is linking developers with security pros to reduce the number of vulnerabilities that may be hiding in code that already is in workflows.

The highly popular Microsoft-owned code repository this week said its security campaigns, which were released in public preview in October 2024, are now generally available to GitHub Advanced Security and GitHub Code Security developers.

The AI-powered tool is the latest step by GitHub to help developers address security issues that can accumulate in fast-paced CI/CD development processes – what’s called the security debt. GitHub in 2023 built Copilot Autofix into pull requests, which the company said allowed programming teams to address vulnerabilities and other security issues 60% faster, which reduced the critical mean time to remediation (MTTR) compared to manual efforts.

“Autofix helps you catch vulnerabilities before they ever make it into production, so you spend less time fixing bugs and more time coding,” James Fletcher, a project manager at GitHub, wrote in a blog post. “But what about the vulnerabilities already lurking in your existing code? Every unresolved security finding adds to your security debt – a growing risk you can’t afford to ignore.”

Fletcher wrote that development teams typically address only 10% of the security issues in their code, leaving 90% of them unresolved.

The Problem with Vulnerabilities

Cloud security firm Wiz, in its State of Code Security 2025 report, found that while code vulnerabilities remain a high security risk, many companies don’t secure their CI/CD pipelines or repositories. The company – which Google is buying for $32 billion – said that 80% of GitHub workflows have insecure permissions, which can allow bad actors to make unauthorized modifications.

That’s where GitHub’s security campaigns come in.

“Security campaigns bridge this gap by bringing security experts and developers together, streamlining the vulnerability remediation process right within your workflow, and at scale,” Fletcher wrote.

AI at the Core

The security campaigns rely on GitHub Copilot Autofix to create code suggestions for up to 1,000 of the alerts that code scans create. Security teams can use the suggestions to triage and prioritize the alerts while developers that that information and address the issues with Autofix. Because the recommendations are delivered to developers in GitHub, they can be incorporated into the process like any other feature of the work.

Included in the campaigns are notifications to developers to let them know which alert they or their team is responsible for, as well as a manager appointed to oversee the process.

In addition, developers can select a group of alerts that are closely related, which lets developers use what they learn by addressing one alert to fix others. Also, developers can use the REST API to more easily interact with campaigns at scale.

Addressing a Critical Need

It’s a step that can’t be missed in programming, he wrote.

“Triaging and prioritizing security problems already present in a codebase has to happen as part of the normal software development lifecycle,” Fletcher wrote. “Unfortunately, when product teams are under pressure to ship faster, they often don’t have enough time to dig through their security alerts to decide which ones to address first. Luckily, in most software organizations, there is already a group of people who are experts in understanding these risks: the security team.”

GitHub found through the early work with security campaigns by some companies that 55% of the alerts included in them were fixed, compared with the 10% that were fixed in situations outside of the process.

“This shows that when alerts are included in a campaign, you can spend more time fixing the security debt, since the prioritization of which alerts to work on has already been taken care of by your security team,” Fletcher wrote.

April 11, 2025
Arm Extension for GitHub Copilot: Accelerating Migration to Arm Architecture
In today’s cloud-centric world, efficiency isn’t just a goal – it’s a necessity. As organizations seek more sustainable and cost-effective computing solutions, Arm-based servers have emerged as a compelling alternative. However, migration challenges have historically slowed adoption. That’s changing with Arm‘s newly announced extension for GitHub Copilot, designed to simplify the journey to Arm architecture for millions of developers.

AI-Assisted Development Meets Arm Expertise

GitHub Copilot has transformed development workflows for over a million paid users across 20,000+ enterprises. These developers spend an average of 3.4 days per week using the tool, with Accenture reporting a 55% increase in coding speed and an 85% boost in developer confidence.

The Arm extension builds on this foundation by integrating architecture-specific knowledge directly into the development experience. By simply invoking `@arm` within GitHub Copilot Chat, developers gain immediate access to:
- Curated code examples tailored for Arm architecture
- Best practices and migration strategies
- Performance optimization techniques leveraging Arm-specific capabilities
- Direct links to Arm’s learning resources and documentation
This context-aware assistance significantly shifts from generic AI coding suggestions to architecture-specific guidance to address real-world migration challenges.

Complete CI/CD Pipeline on Arm

Perhaps most significant for DevOps teams is the introduction of a complete CI/CD pipeline featuring GitHub Actions, Arm Runners and the Arm extension, which is fully available on Arm for free to developers everywhere.

This integration creates a seamless development workflow that spans:
1. Initial coding with architecture-specific assistance from the Arm extension
2. Building and testing through GitHub Actions
3. Deployment using Arm Runners optimized for performance
The recent announcement of free GitHub Runners for Arm further reduces barriers to entry, enabling teams to experience the performance and efficiency benefits of Arm architecture without additional infrastructure costs.

Real-World Impact

Early adopters have already reported significant benefits. Rinat Shagisultanov, VP of Technology at InfoMagnus, describes the extension as “game-changing for developers,” highlighting how it helps teams leverage “Arm intrinsics and architectural features to make their software more performant and efficient.”

Its deep integration with Arm’s architectural knowledge distinguishes the extension from generic AI assistants. Jeremy Sinclair, Software & AppSec Architect and Microsoft MVP, notes: “What blew me away wasn’t just that the Arm extension could help with code – it was how it gave real context about the Arm architecture. The difference between receiving generic, unhelpful guidance and getting legitimate, architecture-specific guidance from Arm is critical to efficiency.”

Mitch Ashley, VP and Practice Lead, DevOps and Application Development at The Futurum Group, states, “Software development innovation is about reducing friction and helping development teams embrace new capabilities and efficiencies. Arm is a vital part platform for applications and software in the device, at the edge, and at the core of the cloud. Bringing together the Arm and GitHub ecosystems directly to developers’ copilots and IDEs shortens the pipelines for the development, testing and deployment on Arm technologies.”

Beyond Server Migration

While the initial focus is on servers and cloud environments, Arm has outlined plans to extend support across a diverse range of applications:
- Generative AI: Simplifying the creation and deployment of AI applications on Arm architecture
- Mobile and Gaming: Optimizing applications for performance on Arm-based platforms
- IoT and Software-Defined Vehicles: Creating seamless workflows for cloud-to-edge and cloud-to-car deployment
The extension will also help developers adopt advanced Arm architecture features like BF16 and I8MM, along with the latest Armv9 capabilities available in Azure Cobalt 100 GitHub Runners – including SVE2, which significantly accelerates machine learning performance.

MLOps Integration

For organizations building AI/ML pipelines, the combination of GitHub Actions, native Arm Runners, and the Arm extension creates compelling opportunities. Developers can now build end-to-end MLOps workflows that automate each step in the machine learning lifecycle while running natively on Arm-based platforms.

This integration is especially valuable for cloud-to-edge optimization scenarios, where consistent architecture across development and deployment environments can dramatically improve performance and reduce operational complexity.

The Developer Experience Advantage

GitHub’s Chief Product Officer, Mario Rodriguez, emphasizes the transformative potential of integrating these ecosystems: “We’re integrating two amazing ecosystems… melding them together to enable scenarios that were impossible before. Sometimes, I like to think about these things as “one plus one equals three.”

This collaboration has already generated impressive results, with over 11 million jobs executed and more than 250 million minutes of compute time utilized through the partnership. Rodriguez notes, “That’s a quarter billion minutes already in the platform through this partnership. And in my opinion, we’re just getting started.”

Looking Forward

As Arm continues enhancing the extension, future updates will support a broader range of devices and use cases. The company remains committed to being the trusted platform for developers, working with ecosystem partners to ensure seamless access to the latest Arm platform capabilities across every segment of compute.

For DevOps teams considering migration to Arm architecture, the extension represents an immediate productivity boost and a strategic investment in future-ready infrastructure. By reducing the knowledge gap and streamlining migration workflows, Arm and GitHub make high-performance, energy-efficient computing more accessible.
February 20, 2025
Getting More Out of GitHub With Advanced Product Requirements
There are plenty of challenges throughout the software development lifecycle, but one that has proven to be a make-it-or-break-it for projects is requirements management.

Requirements engineering is more than simply listing desired features. It is a comprehensive approach to capturing, organizing and evolving project specifications throughout development. Yet this segment of the development process is often an afterthought.

Poor requirements infiltrate countless projects, and many fail before they even begin. At Crowdbotics, we are intently focused on addressing this issue and devising solutions that create robust requirements, giving organizations the best chance at success. However, solving the requirements problem is only part of the equation; the other is the rapidly evolving development process itself.

This is where modern artificial intelligence (AI) tools are starting to make a game-changing difference.

In just the last few years, tools like GitHub Copilot, Devon, Poolside and others have transformed the way developers approach coding. Yet, while immensely useful, they often operate in isolation. Multi-agent systems — where multiple AI models or tools work in collaboration — have demonstrated significant potential. This led us to ask: What would happen if we brought these advanced tools together in an agentic architecture? Specifically, could we combine our requirements management solutions with top code-generation tools to create something greater than the sum of its parts?

The answer is yes.

What AI Collaboration Looks Like Today

The recent advancement of AI has been staggering, but it is primarily driven by the ‘more data and more compute’ mantra. While this approach has yielded powerful models with emergent capabilities, it becomes increasingly evident that there is a ceiling to what can be achieved by simply scaling these inputs. Chain-of-thought reasoning and retrieval-augmented generation (RAG) have proven valuable, yet they have foreshadowed an even more promising path: Collaboration.

Sharing knowledge and tasks between AI models is taking hold as a way to boost performance without the need for additional data or compute. Despite the potential, however, there has previously been little exploration of how commercially available AI tools could work together as agents in real-world scenarios.

To know more, we conducted an experiment to test how well these tools could collaborate in the DevOps space.

What is important to note here is that while multi-agent configurations have been extensively used in academia and in ‘black box’ systems like OpenAI, there is very little research on commercially available products working together. We believe that this demonstration of commercially available models improving in multi-agent configurations is indicative of what is to come soon.

Experimenting With Multi-Agent Architectures

In our study, we extended our business requirements model to GitHub Copilot, specifically leveraging its code suggestion features. The idea was simple: If Copilot had a more contextual understanding of the business requirements, could it offer better suggestions? To test this, we set up a simple study with three groups:
- Group 1: No AI tooling
- Group 2: GitHub Copilot only
- Group 3: GitHub Copilot + Crowdbotics
Each group then worked on the same roughly four-hour programming task, and we measured the data.

The results were convincing: Those who used GitHub Copilot in tandem with Crowdbotics’ requirements tools saw a 25% increase in task success rates. Additionally, the code suggestion accuracy of GitHub Copilot improved by nearly 14%. This improvement required no additional data, no extra compute and no further training — just the shared context between the two models. These results provided empirical evidence that using commercially available models in a multi-agent configuration can improve performance without additional retraining or increased compute.

AI Unlocks the Power of Collaboration

The study results underscore the untapped potential of AI collaboration. Simply sharing knowledge between two commercially available tools led to significant performance gains. And this was not just an academic exercise — Crowdbotics is available today as a GitHub Copilot extension. Developers can plug it into their integrated development environments (IDEs) to integrate improved PRD context and experience the benefits firsthand.

Perhaps even more importantly, the findings point to a larger truth: The future of AI is collaborative. The siloed nature of many current AI tools and systems limits the industry, however, by fostering interoperability and multi-agent systems, we unlock new efficiencies and capabilities that were previously out of reach.

I am personally excited about this vision. The path to a better and more productive future involves more models working together, combining their strengths to overcome individual limitations. As AI evolves, we are committed to exploring and advancing this collaborative paradigm, building tools that enable a more connected, efficient and innovative software development ecosystem.

Ultimately, the journey has just begun, but the potential for a brighter future for AI and DevOps is already evident.
January 6, 2025
DataStax Aims to Reduce Friction by Extending Support for GitHub Copilot

DataStax this week at the GitHub Universe 2024 conference revealed it has extended its support for the GitHub Copilot generative artificial intelligence (AI) tool to now include the ability to write data to its Astra databases-as-service (DBaaS) platform based on the open-source Cassandra database.

Previously, DatsStax had provided developers using GitHub Copilot with read-only access to the database. However, it is now extending that capability to make it possible for application developers to interact with vector, tabular and streaming data right from their integrated development environment (IDE).

In addition, application developers can also now directly invoke DataStax Langflow, a visual tool for invoking open-source Langchain software to build retrieval-augmented generation (RAG) workflows for generative AI applications, that DataStax gained earlier this year with the acquisition of Logspace.

Greg Stachnick, vice president of product management for DataStax, said the integrations with GitHub Copilot make it simpler for application developers to, for example, use natural language to configure a database or generate Langchain application programming interface (API) calls from within their IDE. In effect, that integration makes it simpler for organizations to integrate DevOps and DataOps workflows, regardless of who on the IT team is managing a database.

Application developers can now use natural language to, for example, troubleshoot database queries on their own versus always having to rely on the expertise of a database administrator (DBA), noted Stachnick.

As organizations look to operationalize AI, many are discovering they need to integrate DevOps and DataOps workflows more closely than ever. The challenge is that DataOps workflows have largely been managed by separate teams that often have a separate culture. DataStax and GitHub are now moving to leverage generative AI tools to reduce the level of friction that many developers would have previously encountered, said Stachnick.

That’s especially critical as application developers look to invoke vector data within the context of RAG workflow that needs to be deeply embedded into an application, he noted. The integrations with GitHub Copilot will make it simpler to automatically generate the scaffolding for building those applications, noted Stachnick.

Less clear, is the degree to which that ability to more easily meld DevOps and DataOps workflows might drive further convergence across the way many IT teams are currently organized. At the very least, however, pressure to eliminate the friction that exists across many of the silos that exist within IT teams is only going to increase in the age of AI. One of the reasons that many application developers tend to opt for a document database, rather than a Cassandra key/value store database, is that the latter has been too challenging for them to configure in the absence of a tool such as GitHub Copilot.

The pace at which organizations are looking to build AI applications will naturally vary widely, however, going forward the bulk of new applications being built will embed one or more AI models. The goal now is to make it simpler for application developers and the DevOps teams that support them to build those applications at scale as rapidly as possible.

November 1, 2024
Crowdbotics Integrates AI Platform for Requirements Docs with GitHub Copilot

Crowdbotics today made available an extension for GitHub Copilot that makes it simpler to generate code using higher-quality requirements documents.

Unveiled at the GitHub Universe 2024 conference, this extension provides a simpler method to invoke an AI platform that Crowdbotics created to convert plain language descriptions of applications into a requirements document. Application developers can then use those documents to create code using the GitHub Copilot generative artificial intelligence (AI) tool that is based on large language models (LLMs) originally developed by OpenAI.

Cory Hymel, vice president of research and innovation for Crowdbotics, said the overall goal is to increase the amount of AI-generated code that is actually accepted by application developers. An ongoing GUMDROP study of that code being conducted by Crowdbotics, GitHub and Microsoft finds the Crowdbotics platform improved the code suggestion acceptance rate for GitHub Copilot by 14%, representing a 51% relative improvement. Additionally, the study found that developers using this multi-model configuration were 25% more likely to succeed at feature development than developers who did not use any kind of AI tool.

Arguably, the biggest single challenge application development teams regularly encounter are requirements documents that in one way or another wind up being seriously flawed. Developers then often find they need to rework large segments of code simply because the original requirements document contained errors or did not account for some piece of critical functionality.

Crowdbotics has developed a platform that leverages AI to ensure higher quality requirements documents that can then be used to define technical requirements and implementation recommendations are produced. That capability then makes it easier to, for example, determine how best to construct the front- and back-end of applications, including business logic, data schemas and third-party integrations.
That’s critical because 70% of application development efforts fail because of bad requirements documentation, said Hymel. All the investment in agile development methodologies and best DevOps practices is for naught in the requirements documentation is flawed, he added.

AI tools are already widely employed by many DevOps teams. A Techstrong Research survey found a third (33%) are working for organizations that make use of artificial intelligence (AI) to build software, while another 42% are considering it. Only 6% said they have no plans to use AI. However, only 9% have fully integrated AI into their DevOps pipelines. Another 22% have partially achieved that goal, while 14% are doing so only for new projects. A total of 28% said they expect to integrate AI into their workflows in the next 12 months.

Of course, the level of increased productivity that might be achieved depends on where AI is implemented across those workflows. Generating more code faster doesn’t necessarily mean applications will be deployed any faster. A recent DORA report suggests that in some instances the rate at which software is being deployed has slowed since the rise of AI.

Clearly, the DevOps challenge and the opportunity now is to determine where, when and how to apply AI in a way that delivers the most value to the organization.

October 29, 2024
Microsoft Infuses AI into DevOps Workflows

Microsoft this week added a bevy of tools to its portfolio that infuses generative artificial intelligence (AI) into DevOps workflows.

Unveiled at the Microsoft Build 2024 conference, those additions include GitHub Copilot for Azure which enables DevOps teams to build, troubleshoot and deploy applications on the Microsoft Azure cloud using a natural language interface.

GitHub Copilot for Azure is one of the multiple extensions that Microsoft and third-party partners are providing to, for example, customize tools from Docker, Inc. and Sentry. Microsoft is also embedding its Copilot into Microsoft Teams and Microsoft 365 development tools.

At the same time, Microsoft is previewing a Copilot for Azure that DevOps teams can use to orchestrate application deployments on Azure.

In addition, Microsoft Visual Studio 17.10 embeds GitHub Copilot directly into the integrated development environment (IDE) to provide diagnostic and code review capabilities. Microsoft also previewed updates to Azure Developer Command Line Interface and AI Toolkit for Microsoft Visual Studio Code that enable DevOps teams to integrate copilot sample repositories into DevOps workflows that can be extended to address large language model operations (LLMOps).

Microsoft CEO Satya Nadella told conference attendees that Microsoft is now applying the AI capabilities it provides to write code to manage IT infrastructure and IT operations. Microsoft is redefining software development as part of an effort to one day enable anyone to go from idea to code in an instant, he added.

Paul Nashawaty, practice lead for application development and modernization at the Futurum Group, noted Microsoft in effect is streamlining development processes in a way that promises to make developers 50% more efficient as they build AI applications.

As part of that effort, Microsoft is also making available reference architectures and implementation guidance for building and deploying applications infused with AI models on Azure.

Microsoft is also previewing a Microsoft Azure Compute Fleet service that simplifies the provisioning of compute resources across different types of virtual machines, including spot instances, running in multiple availability zones depending on cost, capacity and performance requirements. IT teams will be able to deploy and manage up to 10,000 virtual machines with a single call to an application programming interface (API) to manage resources that can now dynamically scale as needed.

There is also now an additional instance of an Azure ND MI300X v5 VM based on graphical processor units (GPUs) from AMD and Microsoft is also previewing a Cobalt 100 Arm-based virtual machine based on its custom silicon. Microsoft has also made available a migration tool for shifting instances of Linux operating systems to Azure.

Microsoft is also adding support for a sidecar capability through which DevOps teams can embed logging, monitoring and caching capabilities without having to change application code. DevOps teams can now also use Git repositories to manage the development of applications created using the Microsoft Power framework.

In addition, Microsoft is also adding support for Pulumi as another option for provisioning cloud infrastructure as code.

Finally, Microsoft also made an Azure API Center generally available to discover, manage and govern APIs and make it possible to import Azure OpenAI endpoints as APIs into the Azure API Management service that now also supports OData and gRPC APIs.

Organizations Increasingly Operationalize AI

It’s not clear to what degree DevOps teams are infusing AI into workflows. However, as the amount of code being written using generative AI tools increases it’s only a matter of time before they will need to revisit existing pipelines. Legacy continuous integration/continuous delivery (CI/CD) platforms were not designed for the AI era.

In addition, it’s only a matter of time before machine learning operations (MLOps) workflows are merged with DevOps workflows to accelerate the building and deployment of AI applications.

The only thing that remains to be seen is whether those changes will be made this year, or next, as organizations increasingly operationalize AI.

May 22, 2024
Survey Surfaces Widespread Reliance on Generative AI Among Developers

A survey of 885 developers conducted by Docker, Inc. finds reliance on artificial intelligence (AI) to help build applications is rapidly becoming pervasive.

Nearly two-thirds of respondents (64%) are using AI for coding (33%), documentation (29%), conducting research (28%), writing tests (23%) and troubleshooting/debugging (21%). The most widely used generative AI platforms are ChatGPT (46%), GitHub Copilot (30%), and Google Gemini (19%).

Overall, 65% of the developers surveyed said they view AI positively because it simplifies processes (61%) and enables them to focus more on crucial tasks (55%). Less than a quarter (23%) view AI as a threat to their job.

However, nearly half (45%) said there is too much emphasis on AI. Nevertheless, 40% of respondents identified generative AI as the most important trend in software development, followed closely by AI assistants for software engineering (38%).

Dr. Julia Wilson, user experience researcher for Docker, Inc. noted that more senior and full-stack developers tended to view GenAI as most important, whereas more junior developers with less than five years of experience viewed AI assistants for software engineering as being more important.

It’s still early days when it comes to applying AI to software engineering but it’s clear there are plenty of use cases. Most of that usage currently benefits developers. However, as reasoning engines embedded within large language models (LLMs) become more advanced, it’s now only a matter of time before generative AI platforms can orchestrate a series of tasks that previously would require DevOps teams to write scripts to programmatically perform. Instead, the generative AI platform will create the required scripts required. Those capabilities will serve to eliminate a lot of the toil that ultimately conspires to burn out members of a DevOps team, noted Wilson.

It remains to be seen, however, what impact generative AI will have on how DevOps teams are structured. In theory, an existing DevOps team augmented by AI should be able to build, update and manage a lot more applications. In many cases, the need for entry-level developers might be sharply reduced or, conversely, it may turn out that the junior developers can manage tasks that previously required a senior developer. At the same time, AI may ultimately make DevOps more accessible to a wider range of organizations that previously lacked the resources required to hire and retain software engineers.

The one certain thing is AI is not going back into the proverbial bottle. In fact, the next generation of generative AI platforms will be trained on a narrow set of code that has been vetted for quality and security. That approach should reduce the number of instances where a generative AI platform “hallucinates” in a way that produces code that simply doesn’t work or is rife with known vulnerabilities. General-purpose LLMs such as ChatGPT were trained using code examples pulled from across the Web so the current level of the code being generated is inconsistent.

As a result, one short term concern DevOps teams should have is whether developers become too dependent on generative AI platforms that, by definition, are probabilistic rather than deterministic. In fact, it’s often challenging for a generative AI platform to consistently surface the same code when responding to identical prompts.

As is the case with any major innovation, the challenge is making sure that AI is used both responsibly and safely anytime it is being used to generate code.

Photo credit: Thomas Lefebvre on Unsplash

April 16, 2024
Best of 2023: Measuring GitHub Copilot’s Impact on Engineering Productivity
As we close out 2023, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2023.

Most software delivery teams are considering adopting AI in some form to help engineers accelerate their value delivery and increase delivery effectiveness. GitHub Copilot is one of the first examples of AI-powered engineering assistance. It is self-styled as ‘your AI pair programmer’ and can autocomplete lines of code.

Early adopters report ‘productivity improvements’ of up to 20% using GitHub Copilot. Still, it is not cheap (at $10 per user per month), so how can you build a business case for GitHub Copilot?

Most organizations build a business case around the productivity gains and associated value delivery GitHub Copilot can bring. But what methodology (and metrics) should you use to accurately assess the impact on productivity and value delivery of a tool like GitHub Copilot?

GitHub Copilot: First-Generation AI Assistance for Software Engineers

Copilot is clearly the start of a long and accelerating journey as AI is applied to many areas of the SDLC. It uses the OpenAI Codex to suggest code and entire functions in real-time from your editor.

Other AI tools arrive almost daily and can help software engineers in many time-saving and efficiency-enhancing ways. Here are just a few examples:
- Grit.io will help manage technical debt
- Mintlify provides automated documentation for developer
- Code AI helps translate (some) languages, debug, navigate code and act as a pair programmer
- And tools like AdrenalineAI use AI to improve understanding of your codebase.
So, when cost control is the order of the day, you may wonder how you can accurately quantify the impact of tools like these and justify the added expense.

A Methodology for Measuring the Impact of a Tool Like GitHub Copilot

To robustly measure the impact of GitHub Copilot (and similar AI engineering-enhancement tools), the methodology must be:
- Quantitative—based on hard, measurable data
- Holistic—considering all benefits and potential impacts across the end-to-end SDLC (software delivery lifecycle)
- Balanced—inclusive of subjective survey data alongside software delivery data.
This requires a metrics scorecard that fully captures the benefits and potential costs of GitHub Copilot. The metrics reflect the SPACE framework for measuring developer productivity, emphasizing the key areas GitHub Copilot will likely impact.

These metrics can be tracked over time for a representative group of GitHub users to see the ‘before and after’ effect. We suggest that a representative sample would include engineers of different seniority and activity–and the time period for analysis would be at least three sprint cycles (e.g., six weeks or more).

Key Metrics to Quantify the Impact of GitHub Copilot

An end-to-end software delivery analytics platform provides a single pane of glass to measure the real impact of a tool like GitHub Copilot.

It surfaces a range of engineering and software delivery metrics to capture the impact of GitHub Copilot on five key variables that determine ‘productivity’:
1. Velocity and throughput–Measures of team ‘output’
2. Time to value–Time taken to deliver an increment of software
3. Quality
4. Dependability–A key benefit if teams more reliably deliver against their plans
5. Developer satisfaction-Impact on accelerating repetitive/less interesting tasks
These metrics can be tracked over time for a GitHub Copilot control group versus non-users.

1. Velocity and Throughput Metrics

Throughput is a core measure of ‘output’ over time for Scrum and Kanban teams, and can be calculated in tickets, story points, pull requests, builds or value points. A delivery analytics tool should calculate throughput per engineer for users and non-users of GitHub CoPilot. This can be expressed as a percentage increase.

Sprint Velocity considers the rate of work achieved within a sprint and how it varies over time. It can be calculated in tickets or story points. Advanced analytics tools will also show you the amount of work carried over by sprint to see an even better underlying measure of delivery. This would be a key metric when considering the impact of GitHub Copilot.

2. Time to Value

Cycle Time is a core agile software delivery metric that tracks an organization’s ability to deliver software early and often. It calculates the time taken to deliver an increment of software from dev start to deployment. The shorter the cycle time, the shorter the feedback loops and the quicker the organization will receive new features and respond to customer needs. This is a vital KPI when assessing technology delivery efficiency.

Code Cycle Time typically accounts for 20-30% of overall cycle time. It calculates the average time taken from a pull request (PR) opening until it is merged/closed. The bulk of this time is usually spent during the approval process.

In theory, GitHub CoPilot enables quicker, easier development. Therefore, developers should have greater availability to review each other’s PRs. If code quality is improved, then the outcome of the reviews should result in fewer changes requested and an approval time.

3. Quality

Escaped Defects is a simple but effective measure of overall software delivery quality. It can be tracked in numerous ways, but most involve tracking defects by criticality/priority.

Any analysis of delivery efficiency pre-/post-implementation of GitHub Copilot should consider Escaped Defect rates, as it would be a poor trade-off to increase velocity and ‘productivity’ at the expense of quality.

Build Failure Rate identifies the percentage of builds that fail and the overall risk this poses to a team working productively. Notable changes to the failure rate after implementing GitHub Copilot indicate that code quality may be impacted.

4. Dependability

Sprint Target Completion tracks the percentage of the sprint goals achieved each cycle. ‘Scrum teams’ and ‘sprints’ are the basic building blocks of Scrum Agile software delivery. If scrum teams consistently deliver their sprint goals, Agile software delivery becomes relatively dependable, enabling the prediction of delivery outcomes across multiple teams and longer time periods.

Scrum team predictability is, therefore, a critical success criterion in Agile software delivery. If GitHub Copilot can improve the likelihood of a team delivering their tickets faster and with fewer bugs, then this is a major contributor to the overall improvement in effectiveness.

5. Developer Satisfaction

eNPS tracks employee satisfaction and loyalty within teams and organizations. Anecdotal reports suggest that developers find that GitHub Copilot makes the more tedious aspects of coding less taxing and positively impacts wellbeing. An employee NPS makes this straightforward to validate and quantify.

Although an important factor of productivity measurement, it shouldn’t be viewed in isolation from the other metrics when quantifying overall developer productivity.

The above are some examples of relevant metrics to consider when analyzing the impact of GitHub Copilot on delivery productivity. The key is to take a balanced set of metrics that holistically considers software delivery a complex process.

Combining the Balanced Scorecard of Metrics to Create a Business Case for GitHub Copilot

Typically, we would combine data from the ‘balanced scorecard’ of metrics discussed above using simple weightings to create an overall productivity impact assessment (PIA) of GitHub Copilot. See the below table:

GitHub CoPilot Productivity Impact Assessment – Example Template

The weighted average productivity improvement calculated in the PIA can then be applied to the estimated cost of the delivery capability (headcount x fully loaded staff costs). This provides a productivity improvement monetary calculation based on resource costs. It excludes the potentially (larger) benefits of delivering more value to customers earlier, which is not a benefit that is easily or necessarily calculated.

Productivity Improvements From Using GitHub Copilot – The Empirical Data

There is a distinct lack of independent data in this regard.

GitHub’s own survey of 2,000 developers showed that 88% of developers claimed ‘to be more productive’ when using the tool, while a task test undertaken by 95 developers saw the group that used GitHub Copilot was 55% faster and had a 7% higher rate of completing the task (see below).

GitHub’s Own Survey Data – The Impact of GitHub Copilot on Users (2022)

Our own analyses showed improvements using a PIA (as shown above) of about 5%. However, this is bound to improve further as AI technology rapidly improves.
January 3, 2024
Google’s Shiny New AI Dev Environment — the ‘Experimental’ Project IDX

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Google’s full-stack, browser based development environment in the cloud. It’s not just a Copilot clone, but aims to help you “get an app from zero to production.” (more…)

August 10, 2023