Tag: LLMs

Sourcegraph Launches Cody Generative AI Tool to Write Code

Sourcegraph today made generally available an open source Cody tool that leverages generative artificial intelligence (AI) tools to write and fix code.

Version 1.0 of Cody invokes StarCoder, an open source large language model (LLM) for creating code made available by Hugging Face and a chat tool that makes use of dense-sparse vector retrieval system for code and documentation to invoke GPT-4 Turbo, Claude 2, GPT-3.5 Turbo, Claude Instant and Mixtral-8x7B LLMs.

Designed to be deployed as an extension of VS Code, Cody is available in a free edition as well as a professional edition, dubbed Cody Pro, that Sourcegraph supports. Cody Pro will be free until February 2024, after which it will cost $9 per month. There is also a version for the JetBrains editor available in beta, along with an experimental Neovim offering.

Cody is also integrated with the code search tools provided by Sourcegraph to provide development teams with additional context that, depending on the use case, delivers a completion acceptance rate of 30% or higher for code.

In addition to enabling developers to write code faster, Coy also simultaneously invokes the graph technology Sourcegraph created to provide additional context to a prompt that makes it simpler to surface relevant code by parsing code and producing a graph that captures the semantic understanding of code. Cody, for example, can search the codebase for files and documents that can be used to suggest a rough plan of attack for implementing a new feature or resolving issues that surfaced in a bug report.

Cody can also create, save and share custom commands to make it simpler to reuse the prompts created to write code.
Future versions of Cody will be more tightly integrated with the Sourcegraph universal code graph to provide additional context. In addition, Cody will soon leverage deeper graph context to make autocomplete suggestions based on symbols defined elsewhere.

Sourcegraph CEO Quinn Slack said the combination of graph technologies and LLMs reduces latency while helping to ensure the code generated by the LLM doesn’t produce hallucinations such as type errors and imaginary function names.

In addition, Sourcegraph is committed to applying AI and graph technologies beyond the codebase itself to provide deeper insights into the environment code will eventually be deployed in, added Slack.

It’s not clear how much developers are relying on generative AI tools such as Cody and GitHub Copilot to write code faster, but it’s clear the pace at which software can be written is accelerating. That increased pace of application development will eventually require DevOps to revisit workflows that were not designed to manage codebases that, in the months ahead, will significantly increase in size.

Each developer will most likely have a personal preference for one generative AI tool versus another, depending on the quality of the code generated. The issue is that no matter how good any of those tools are, however, the code generated will still need to be reviewed by a human before it can be deployed in any production environment.

December 14, 2023
CircleCI Extends CI/CD Reach to AI Models

CircleCI this week revealed it is extending the reach of its continuous integration/continuous delivery (CI/CD) platform to make it simpler to incorporate artificial intelligence (AI) models into DevOps workflows.

In addition to providing access to the latest generation of graphical processor units (GPUs) from NVIDIA via the Amazon Web Services (AWS) cloud, Circle CI has added inbound webhooks to access AI model curation services from providers such as Hugging Face and integrations with LangSmith, a debugging tool for generative AI applications and the Amazon SageMaker service for building AI applications.

CircleCI CEO Jim Rose said while there is clearly a lot of enthusiasm for incorporating AI models into applications, the processes being used are still immature, especially in terms of automating workflows that include testing of probabilistic AI models.

Most AI models are built by small teams of data scientists that create a software artifact that needs to be integrated within a DevOps workflow just like any other artifact, noted Rose. The challenge is that most data science teams have not yet defined a set of workflows for automating the delivery of those artifacts as part of a larger DevOps workflow, he added.

DevOps teams will also need to make adjustments to a version control-centric approach to managing applications to trigger pipelines to pull AI software artifacts that exist outside of traditional software repositories. For example, the inbound webhooks provided by CircleCI now make it possible to automatically create a pipeline whenever an AI model residing on Hugging Face changes.

It’s still early days as far as the deployment of AI models in production environments is concerned, but there is no doubt generative AI will have a major impact on how software is developed. AI models are a different class of software artifacts that are retrained instead of being updated, a process that occurs intermittently. As such, DevOps teams need to keep track of each time an AI model is being retrained to ensure applications are updated.

At the same time, generative AI will also increase the pace at which other software artifacts are being created and deployed. Many of the manual tasks that today slow down the rate at which applications are built and deployed will be eliminated. That doesn’t mean there will be no need for software engineers, but it does mean the role they play in developing and deploying software is about to rapidly evolve. DevOps teams need to evaluate both how generative AI will impact the tasks they manage as well as the way the overall software development life cycle (SDLC) process needs to evolve.

Each organization, as always, will need to decide for itself how best to achieve those goals depending on the use cases for AI, but the changes that generative AI will bring about are now all but inevitable. The longer it takes to adjust, the harder it will become to overcome the cultural and technical challenges that will be encountered along the way.

November 30, 2023
New Relic Adds Ability to Monitor AI Models to APM Platform

New Relic today added an ability to monitor artificial intelligence (AI) models to its application performance management (APM) platform.

Peter Pezaris, senior vice president for strategy and experience for New Relic, said as next-generation applications are built and deployed, it’s apparent most of them will incorporate multiple AI models. New Relic is extending its APM platform to make it simpler to monitor the behavior of those AI models within the context of an application, he added.

To achieve that goal, New Relic has added more than 50 integrations with frameworks and AI models to troubleshoot, compare and optimize different prompts and responses to address performance, cost, security and quality issues such as hallucinations, bias, toxicity and fairness.

For example, response tracing for large language models (LLMs) can be applied using New Relic agent software to collect telemetry data that can be used to compare how different AI models are performing and responding to queries. Those results will provide immediate full visibility into the models, applications and infrastructure being used to provide complete visibility across the entire AI stack, said Pezaris.

That capability is going to prove crucial as developer use a mix of proprietary, open source and custom large language models (LLMs) alongside a range of other types of AI models to build and deploy applications, he added.

Organizations are likely to find themselves managing hundreds of AI models that either they or a third party developed. The challenge, as always, is bringing order to a potentially chaotic process that, in addition to wasting resources, represents a significant risk to the business given the potential for regulatory fines to be levied, noted Pezaris.

Each organization will need to determine for itself how best to construct workflows spanning data scientists, application developers, software engineers, cybersecurity teams and compliance specialists. Before too long, organizations will find themselves managing hundreds of AI models that might be integrated into thousands of applications. New Relic is essentially making a case for extending an existing APM platform to address that challenge rather than requiring organizations to acquire, deploy and maintain additional platforms.

Eventually, in addition to updating AI models, IT teams will find they are being regularly replaced as advances continue to be made at a fast and furious rate. Data science teams are now making AI models based on significantly larger parameters that make previous generations of models obsolete before they can even be deployed in production environments.

As a result, operationalizing AI is going to present DevOps teams with major challenges as they look to both tune application performance and ensure the results being generated are accurate and consistent. That latter issue is especially critical in enterprise application environments where the results generated by an AI model can’t vary from one query to the next.
It’s still early days in terms of how AI will be applied to applications, but as AI models join the pantheon of artifacts DevOps teams need to manage, application development and deployment are about to become much more complex to manage.

November 14, 2023
Digital.ai Update Extends Scope and Reach of DevSecOps Platform

Digital.ai this week made generally available a Denali update to its DevSecOps platform that promises to make it simpler to integrate custom artificial intelligence (AI) models with the AI models developed by the company.

At the same time, the company is adding self-guided workflows and templates to generate tests and implement DevSecOps best practices along with integrations with Terraform by Hashicorp, Azure Bicep, Azure Key Vault and AWS Secrets Manager.

Finally, Digital.ai is adding an ARM Protection feature to better secure iOS applications without requiring embedded bitcode or integrations into the build system. DevOps teams can, via a single command, protect compiled applications locally with support for obfuscation, run-time active protections and application monitoring without uploading them to a third-party service.

Greg Ellis, general manager for application security for Digital.ai, said the overall goal is to make it simpler for software engineering teams to invoke capabilities that have been embedded within the company’s DevSecOps platform.

In the case of AI models, that also means instead of requiring DevOps teams to only use AI models developed by Digital.ai, the company is moving to make it simpler for DevOps teams that adopt its platform to incorporate custom AI models as they see fit as part of an ongoing effort to democratize intelligence at scale, he noted.

In general, it’s already apparent organizations will be employing heterogeneous approaches to incorporating AI models into DevOps workflows, said Ellis. The challenge now is moving beyond experimenting with AI to embedding them within DevOps workflows, he added.

It’s already clear developers are using generative AI to develop code at increasingly faster rates. The challenge now is to manage that accelerated pace of development when many organizations are already struggling to manage existing DevOps workflows at scale. Hopefully, AI technologies will also one day help software engineers find ways to manage that volume of code moving across their DevOps pipelines.

In the meantime, organizations will also need to better define where the machine learning operations (MLOps) workflows that data scientists use to build AI models end and where DevOps workflows that will be used to embed AI models into applications begin. As is often the case when it comes to emerging technologies, cultural issues are just as challenging as the implementation hurdles that need to be overcome.

At this point, like it or not, the generative AI genie is out of the proverbial bottle. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated. Less clear is to what degree AI may drive organizations that have already embraced DevOps to adopt an alternative platform, but savvy DevOps teams are, at the very least, starting to map out which processes are about to be automated so they can have more time to focus on issues that add more value to the business.

October 27, 2023
Google De-Recruits 100s of Recruiters ¦ ARM Valued at $45½B in IPO

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Google fires hundreds of recruiters, and ARM gets a sky-high valuation. (more…)

September 14, 2023
IBM LLM AI: COBOL to Java ASAP ¦ ARM IPO is GO!

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Translating legacy COBOL code to a slightly more modern language, and Arm will go public (again) next month. (more…)

August 23, 2023
AI ‘is Getting Worse’ ¦ AI ‘Will Lose India Jobs’ (Probably Isn’t ¦ Probably Won’t)

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week, a conundrum: On the one hand, researchers say ChatGPT is losing the plot; and on the other, we hear outsourced coding jobs in India will be replaced by AI. (more…)

July 20, 2023
Forking RHEL! Oracle and SUSE Join the Fight ¦ Silverman Sues AI Firms

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: SUSE announces a Red Hat Enterprise Linux fork after Oracle said a similar thing, and Sarah Silverman says OpenAI and Meta have stolen her words. (more…)

July 12, 2023

Are LLMs Leading DevOps Into a Tech Debt Trap?

Everyone thought AI would free us from toil. Unfortunately, that’s not true. What it will do, however, is lead to an explosion of technical debt. Some of the most alluring uses of LLMs have the potential to lead to some of the worst outcomes for enterprises and their DevOps and platform engineering teams.

SK Ventures and Adrian Cockcroft raised the alarm bells in their warning posts about the tipping points LLM introduced into work and society. And in my own Gluecon 2023 presentation, I challenged just how powerful and disruptive the new generation of AI/ML will be for ITOps and DevOps efforts.

Working Faster is not Working Smarter

There is no doubt that LLMs are powerful and will change how we think about expertise and knowledge work. While they are not truly innovating, their ability to draw on such a vast pool of human expertise makes it difficult to distinguish from true creative work. Since they are unbounded, they will cheerfully use the knowledge to churn out terabytes of functionally correct but bespoke code.

It’s easy to imagine a future where LLMs crank out DevOps scripts 10x faster. We will be supercharging our ability to produce complex, untested automation at a pace never seen before! On the surface, this seems like a huge productivity boost because we (mistakenly) see our job as focused on producing scripts instead of working systems.

But we already have an overabundance of duplicated and difficult-to-support automation. This ever-expanding surface of technical debt is one of the major reasons that ITOps teams are mired in complexity and are forever underwater. Generating even more scripts, even if they were better written, does not reverse the trend.

Can AI/ML Avoid This Technical Debt Trap?

It depends on how we use the expertise in the models. Instead of asking it to generate new code, we could ask it to interpret and modify existing code. For the first time, we have tools to take down the “not invented here” barriers we’ve created because of the high cognitive load of understanding code. If we can help people work more effectively with existing code, then we can actually converge and reuse our systems. By helping us expand and operate within our working systems base, LLMs could actually help us maintain less code.

Imagine if the teams in your organization were invested in collaborating around shared systems! We haven’t done this well today because it takes significantly more time and effort. Today, LLMs have thrown out those calculations.

Taking this just one more step, we can see how improved reuse paves the way for reduction of the number of architectural patterns. If we improve our collaboration and investment in sharing code, then there is increased ROI in making shared patterns and platforms work. I see that as a tremendous opportunity for LLMs to improve operations in a meaningful way.

It comes down to whether we ignore complexity by replacing it or embrace complexity by understanding it. I see this as a choice between independence or collaboration, as illustrated by the table below.

LLM Driving Independence	LLM Driving Collaboration
Create new code	Interpret existing code
Replace existing code	Maintain existing code
Design new architectures	Be a guide and mentor
Build new stacks	Evaluate alternatives
Bypass documentation	Improve documentation

Can AI Help us Collaborate?

We keep taking the same hardware and software components and then building unique systems. That variation makes it impossible to leverage each other’s work. At its heart, the reusability problem lies in the difficulty of understanding systems, and that makes it nearly impossible for us to work together, collaborate, reuse and accelerate.

Breaking this cycle has been our mission at RackN for nearly a decade. We’ve seen tremendous ROI from availability, reliability and speed when customers are able to leverage standardized infrastructure pipelines. We’ve seen firsthand that it is possible to guide teams into collaboration with sufficient guidance, mentoring and training. Collaboration creates 10x outcomes even without AI, but it takes building expertise, deliberate effort and leadership discipline to implement.

Why does collaboration create 10x ROI? It works because true innovation actually comes from understanding and building on top of systems rather than rebuilding them from scratch over and over again. The key is being able to leverage LLMs to better understand existing automation and platforms instead of creating yet another bespoke stack.

Enterprises Should use LLMs to Drive Collaboration

I believe that the most effective enterprises will use these tools to elevate their IT leaders and architects to focus on collaboration and reuse. They must actually direct them to produce less bespoke automation; instead, they can use expertise in these tools to find and maintain existing systems. LLM presents options, but collaboration requires new thinking and strong leadership.

July 12, 2023

PagerDuty Signals Commitment to Adding Generative AI Capabilities

PagerDuty has signaled its intention to add generative artificial intelligence (AI) capabilities to its cloud platform for managing IT operations via integrations with large language model (LLM) providers that it will disclose at a future date.

These capabilities will make it possible to invoke natural language via PagerDuty Operations Cloud to automatically generate status updates, create drafts of incident postmortem reports and even co-author workflows in any programming language to automate remediation of an issue.

Damon Edwards, senior director of product for PagerDuty, said as AI continues to evolve, the goal will be to minimize the need for human involvement as much as possible. That doesn’t mean there won’t be a need for humans to be involved, but it does mean that many rote tasks associated with managing an IT incident can be automated, he added.

For example, the reports that are run to determine the root cause every time there is an incident can be automated, noted Edwards. In addition, summarizations of status reports or postmortems after incidents were resolved can be made readily available to any member of an IT team on demand, he added.

Those capabilities will also make it easier to onboard new members to an incident management team as required, said Edwards. Today, it’s difficult to add new members to a team after an incident has unfolded because it takes time to bring them up to speed, he noted.

In addition, generative AI also makes it easier to capture a lot of the tribal knowledge within an IT organization that would otherwise disappear when staff either leave the company or shift roles, said Edwards.

The generative AI additions to the PagerDuty platform extend an artificial intelligence for IT operations (AIOps) capability that PagerDuty made available this spring. The PagerDuty AIOps platform leverages the data model embedded in its incident management software to reduce the amount of time required for an AI platform to learn how an IT environment operates and begin surfacing valid recommendations to optimize workflows.

It’s not clear at what pace AI will be adopted within enterprise IT organizations, but it is rapidly becoming pervasive. Each DevOps team will need to decide the degree to which they can rely on AI to manage IT processes, but as IT processes become more complex, there is a clear need for AI to navigate all the dependencies that exist in an IT environment. As more routine tasks are automated, however, there will also inevitably be a realignment of roles and responsibilities within an IT organization. The overall goal is not so much to eliminate IT staff as much as it is to enable them to manage IT environments at a much higher level of scale.

In the meantime, DevOps teams need to plan for not only what is possible using generative AI today but also tomorrow. After all, there’s not a lot of point in training someone to take on a new task if that task is automated a few months from now.

June 6, 2023