devopsy.kinsta.cloud

Tag: analysis

Sonar Surfaces Multiple Caveats When Relying on LLMs to Write Code

An analysis of the code created by large language models (LLMs) finds that in addition to having a tendency toward creating messy code they also can also introduce security weaknesses and vulnerabilities into the output generated.

Sonar today published a report based on its proprietary analysis framework for assessing LLM-generated code spanning more than 4,400 Java programming assignments. The LLMs evaluated included Anthropic’s Claude Sonnet 4 and 3.7, OpenAI’s GPT-4o, Meta’s Llama-3.2-vision:90b and OpenCoder-8B.

Each model has its own “coding personality” and possesses a strong ability to generate syntactically correct code and boilerplate for common frameworks and functions. For example, Claude Sonnet 4’s success rate of 95.57% on HumanEval demonstrates a very high capability to produce valid, executable code. The models possess a strong foundational understanding of common algorithms and data structures and can create viable solutions for well-defined problems. Additionally, the models are highly effective at translating code concepts and snippets from one programming language to another, which makes them a powerful tool for developers who work with different technology stacks, according to the report.

However, critical flaws such as hard-coded credentials and path-traversal injections were common across all models. While the exact prevalence varies between models, all evaluated LLMs produced a high percentage of vulnerabilities with high severity ratings. For Llama-3.2-vision:90b, over 70% of its vulnerabilities are considered ‘blocker’ level of severity; for GPT-4o, it’s 62.5%; and for Claude Sonnet 4, it is nearly 60%, according to the report.

In fact, the study finds that improved functional performance was often accompanied by much higher levels of risk. While Claude Sonnet 4 improved its performance benchmark pass rate by 6.3% over Claude 3.7 Sonnet, meaning it solved problems more correctly, this performance gain came at a price: The percentage of high-severity bugs rose by 93%.

All models tested also showed a bias toward messy code, with more than 90% of the issues found being so-called “code smells” issues that are indicators of poor structure and low maintainability. Those smells, which include dead and redundant code, increase the long-term technical debt that might be accrued.

Prasenjit Sarkar, solutions marketing manager for Sonar, said the report makes it clear that DevOps teams need to review the code generated by LLMs but also realize that each LLM is essentially an opinionated resource that has been programmed to behave in a specific manner. As such, each developer should take into account some of the inherent biases that have been embedded into each model, he added.

The verbosity of the code generated can also be challenging to debug and fix, noted Sarkar. Many developers often find it challenging to understand code that they did not write. In fact, they may need a separate LLM and an associated set of AI agents to review code created by an LLM.

It’s not clear to what degree DevOps teams have adopted AI coding tools and just how much they trust the output generated. There is no doubt that developers will be more productive, but some of those gains will come at a cost tomorrow that organizations may not realize they are incurring today.

August 13, 2025
Black Duck Analysis Surfaces Raft of Open Source Software Vulnerabilities in Code Bases

An analysis of 965 commercial codebases across 16 industries conducted in 2024 finds 86% of commercial codebases evaluated contained open-source software vulnerabilities, with 81% of them known to be high- or critical risks.

Conducted by Black Duck Software, a provider of code scanning tools, the analysis, more troubling still, finds 90% of audited codebases included open-source components that are more than four years out-of-date.

In total, the analysis finds the number of open-source files in an average application has also tripled in the last four years, from more than 5,300 in 2020 to more than 16,000 in 2024.

Mike McGuire, a senior manager at Black Duck, said while progress has been made in improving software supply chains, the sheer volume of open-source code with known vulnerabilities being added to code bases is becoming overwhelming.

The most frequently found source of vulnerabilities in code bases can be traced back to jQuery, a widely used Javascript library, accounting for eight of the top ten high-risk vulnerabilities. In all, 43% of the applications Black Duck scanned contained some version of jQuery, most of which are outdated.

The issue, of course, is that application developers only have so much time to devote to upgrading and patching software components, and what time is made available is generally allocated to the simplest issues to resolve rather than the most critical.

Most application developers are also hesitant to update applications for fear that one dependency issue or another will cause the application to fail.

The issue is that as organizations become more dependent than ever on software the cost of breach can exceed the cost of any downtime that might be incurred by an application being offline. The only way to address that issue is to proactively test code long before it is added to a software build, said McGuire.

That approach will also make it simple to identify software licensing conflicts that are becoming more common, he added. The Black Duck analysis finds that 56% of the audited codebases contain license conflicts: Transitive dependencies caused nearly 30% of those licensing conflicts. The analysis also finds that 33% of codebases contained open-source code with no license or a customized license. Only 77% of dependencies could be identified via package manager scanning, suggesting that the remainder were introduced to applications by other means, including AI coding assistants, noted McGuire.

At this stage, it’s not especially feasible for most organizations to reduce their dependency on open-source software. Therefore, the only practical option is to more proactively scan that code for vulnerabilities and licensing conflicts that could have a range of potential issues that could impact intellectual property.

Regardless of approach, the one certain thing is cybercriminals count on organizational inertia to continue to provide them with plenty of opportunities to exploit code bases that are riddled with known vulnerabilities. The only wonder is how long it’s taken for them to exploit those vulnerabilities, in a way that will ultimately wreak more havoc than any update to a more secure version of a software component could ever match.

February 26, 2025
Things To Do During Slow Times
We all know it happens in North America; once the U.S. Thanksgiving holiday hits, things start to slow down in IT—from vendor salespeople busy with family to coworkers out of the office in larger numbers. Since all regions and most religions have holidays clustered around the new year, the worldwide slowdown goes on for a good month and a half, sometimes—and in some locales—even longer.

And how we experience that slow time varies pretty widely. For some, it’s time to be completely off, like those coworkers. For others, it is time to finish those tasks that have been simmering on the back burner. For still more, it is a time of frustration as the things they’ve been working on grind to a halt. Back in the network computing days, we would do large, complex technical evaluations of the gear the vendors sold to enterprises. I covered mostly storage and servers with some security at the end, but my lab had more security, some software and some networking. We needed vendors to participate, and participation mostly meant traveling to our labs for any physical product, to set it up and let us test it. Needless to say, this slowed to a (sometimes frustrating) crawl as people reduced travel closer to the new year. I’ve felt that same frustration as an architect—when we couldn’t get vendors to respond and you really can’t blame them for doing the best they can in a time of seriously reduced staffing.

So, what can we do that doesn’t tie us to others who may or may not be available?

You know the list specific to you far better than I do, but here are some general suggestions to ponder. Most of these I have done during slowdowns.
- Learn that new language or framework. We all have one that the org is interested in, but in which we are weaker. Pick it up in those quiet hours; you can slap on headphones and focus with fewer interruptions.
- Same with new tools like test generators, DLP, GitOps or whatever your org is considering implementing. Heads down; figure it out in a test bed, and become at least versed and, at best, an SME.
- Clean up all that cruft you know you left behind over the past year. We’ve all got dead-end branches, stray text files, outdated bash scripts … Dynamic environments spew extraneous artifacts more than historically was the case. Take that time to go clean it up. I had a large script a couple of years ago that spun up a bunch of containers and ran tests. When I went to clean up, there were three different versions of the script and several orphaned containers still running. Good to get taken care of.
- Research, research, research. Whether it is something like reading a product’s site, checking crowdsourcing tools like G2 or looking at what some of the better/smarter analysts out there have to say, pick the tool you think would benefit the org, validate that it will and maybe even start documenting the use case and ROI justification (if you use ROI for consideration of new products; benefit analysis if not).
- Catch up. While generally the least fun use of this extra time on the job, it is often the most practical. We all have a ton of work, and sometimes the peace of mind that cranking out backlog offers is worth not doing the other things on this list.
- Take the time off. Seriously. Most of us (though no longer all of us) got into IT because we loved the technology and the promise. Walk away, take a breath and don’t even think about it for a week or two. Then, come back; I bet a bit of that wonder will return, making the job more fun and you more adaptable. It works if you can manage it. As others have said for many years, walking away and rediscovering a hobby can be invigorating, too.
Don’t sweat it, and don’t totally waste it. Figure out the best use of that time and move on it. There are a few spaces that simply do not slow down for holidays, and for those people, this blog is unlikely to be useful now, but at some point, slowdowns will arrive. I was a team lead at a tax software company. We slammed the holidays prepping for tax season, then got short weeks all summer. The mental checkout of teammates was similar to the end of year but delayed until early May. So we did all of this then.

But no matter what or when slowdowns happen, you’ve knocked out another year of keeping the org running. So, celebrate in some way; you deserve it.
December 13, 2023
Product Evaluation Discussions

Instead of doing a predictions blog like 99% of the punditry, I figured I would do an awareness blog to wrap up 2022. Every year, you deal with a wide variety of people trying to tell you about products that may (or may not) be useful to your IT efforts. Some of the products even make it to the evaluation phase and while a few get installed, a smaller number get used.

One of the many ways to minimize your time investment when talking about the bewildering array of new DevOps technologies and products is to know where the information you are consuming comes from. Here is my start at a list of information sources, where the information they impart comes from and how to get the most out of these sources.

Marketing: The marketing team at any organization exists to spread the word about the product’s good points. Most marketing professionals only admit weakness in the underlying product if pressed. This is expected–if the sole purpose of your job is to tell the world why the product is needed, blathering on and on about the weak bits is not going to work out. But we all know there are no perfect products, even before you account for all of the environmental differences we’re installing into. So listen – they will present the product in the best light possible and have useful knowledge, particularly early in the evaluation process.

Sales: The sales team wants to sell products, but they also are invested in happy customers. A good sales rep in high-tech is building relationships because organizations can create an income stream if the product’s value is realized on their site. This makes them practical. Their goal is definitely to represent the product and company positively, but in a light that admits more weaknesses and offers workarounds for the deal breakers.

Sales Engineers: A lot of the sales team’s practicality comes from the team’s engineer(s). They’re the ones that have to help you get things installed and running, so they have a vested interest in the organization knowing ahead of time what will require work and what the risks are. Given a choice, talk to the SE alone before signing a contract–not because sales lies, but because the SE is there specifically to note issues that might crop up and how to get around them.

Crowd Evaluations: Two things to remember about crowd rating sites–the participants are volunteers on the best sites, paid contributors on others. For paid contributors, well, the motivation is pretty clear. For many voluntary participants, the motivation is that they either love or loathe the product. Most of these sites allow vendors to respond to commentary, and some even allow vendors to have evals removed. No one offers a service where glowing vendor reviews can be corrected … keep that in mind. These sites are useful, and a few of them are mostly people that want to help. So find the good ones (I’m not advertising anything here, so no links–sorry) and use them.

Analysts: Nearly all of the analysts I know–be they GigaOm analysts or a competitor–have experience in the field. They have done it, and most of them continue to keep their hands in the mix somehow. While I have not written (or maintained) a line of code in months, I do attempt to do so regularly. At the individual analyst level, the thing to watch is how long it’s been and how current they are. Some do pure analysis and haven’t touched the toys in a decade or more, and for C-level analysis, this is great … but for implementation recommendations, best to find a different source.

Analyst Firms: This is where the real issues in analysis creep in. As an enterprise architect, I dealt with an analyst firm that–at the time, at least–would say whatever we wanted them to say. Totally useless in evaluations, but it worked well when convincing our technical advisory committee to approve the right approach. Each firm comes at the problems with its own lens, and being aware of a given firm’s slant is a good step to understanding how to use them–or if you should rely on them at all.

Press: The age of pure-play press is largely dead, leaving a lot of “Paid for publication” out there. While there is still non-sponsored content being developed (one of the few rules Techstrong Group has for my writing of this blog is that I should avoid mentioning products; for example, making sponsorship they may pursue more about their vendor relationships and how many of you are reading my drivel), the majority of what is published as “articles” is “marketing” and should be treated as such. Reading this content can be useful for early education but not so much for evaluations.

Internal Champions: Not the mandatory sponsor most projects require, but the person inside that is driving the org to look at a solution. These people are generally key. They know the environment, they know the product, they have ideas on how the product could help the environment, and they have a history in the organization. Whether their ideas about how various products help the company have panned out with other tools, for example. Of course the usual caveats apply… Security people still think you should be doing more to lock out bad actors – that is what they’re paid for, but that influences their evaluation of a new security product and how it fits in the org. Awareness of that type of vested interest can help filter internal champion input.

There are others, or course, but I’m keeping this blog-length, and most–like MSPs or hosting providers–can be viewed as an extension of one of these (MSPs and hosting providers generally act like extensions of their chosen product’s sales and marketing, for example).

And keep rocking it. All these people can help you understand how to make the environment better or your role easier. They can’t do it for you; that is your skill. Listen to what they all say and then act in the organization’s best interests. When I’m not a pundit, I tend to use marketing/crowd rating to get a feel for the space, then analysts and SEs to tell me about the space and specific products in it respectively. But figure out what works for your org and be aware of what viewpoints the various actors bring to your learning.

January 4, 2023
Claim to Fame

February 24, 2021
Iambic Logic

November 11, 2020
Embedding Predictive Analytics

For an enterprise with a deep DevOps maturity model, the initial “wow” factor of automation has worn off. At this point, the sexier concepts involve proving continuous improvement. As an example, our first baseline of metrics might have taken the pre-DevOps timings of a typical build and deployment of a given application. We can measure this time against our post-implementation of DevOps automation and show a significant savings in time. This is opportunity time, meaning an employee must be ready to turn around the decision cycle faster to take advantage of the time saved (not be playing solitaire, for example). But once DevOps automation is old hat, the difference between our initial post-DevOps typical build and deployment time and our current one still using DevOps may not be all that significant. But is there more hidden there than meets the eye?

Establishing Context & Baselines

At this point context may be a more meaningful measurement. As an example, if my first version of DevOps automation for a deployment contained 35 steps and my current one contains 160 steps it would stand to reason I am “doing more” now in my deployments than I used to do. These additional steps may involve executing more tests than I originally thought to do. They may be interacting with monitoring systems (turning them on and off as appropriate), which reduces “noise” in the alerts my monitoring teams have to respond to. In short, over time, my deployment processes may become “longer” because I am increasing the capabilities or the sophistication of what I do during these deployments. While from a pure timing perspective this may make them longer, it also makes them more powerful and, so, is worth measuring. To know the delta, however, I must have retained older versions of automation to compare against, not just throw away the old every time I update with new steps.

So let us assume the context of my automation is now fairly stable. I do the same things, more or less, the same way every time I deploy my application. At this point, measuring the functionality of what I do becomes more meaningful. As an example, my deployment automation may complete four main functional tasks to complete, I deploy the code (the app), I deploy the dataset, I deploy the config files and, finally, I run a series of tests on the above. While I may not ever incorporate too many new kinds of things to do in this automation, understanding how long each function takes to complete is relevant to proving continuous improvement. For instance, if my code takes roughly the same amount of time each deployment, as does my config files, but all of the sudden my dataset begins to grow in each deployment (from normal usage), my overall deployments may be getting slower. This is not due to hardware inefficiency or network inefficiency; it is simply due to a larger dataset that keeps growing.

If on the other hand, it is my test set that is getting progressively longer to execute, that is an indicator of test script inefficiency (given same code, same config and same dataset). If the number of tests being executed has not increased, but the length of time it takes to run them has, there likely is inefficiency in “how” the testing is being constructed or executed. Understanding the context for what is going on in a deployment or similar kinds of automation must occur before introducing predictive analytics can occur.

Historical Trends Extended

Predictively evaluating automation is nothing more than taking the data from a historical perspective and extending the trend lines into the immediate future. If for example, my deployment automation is getting four seconds longer each month for the last 12 months, I can reasonably infer the trend will continue, and I can predict the length of time it takes to execute a deployment in the coming months. Understanding “why” this occurs requires a contextual investigation, but predicting its occurrence does not. This may seem inconsequential as we talk about the length of deployment processes of a given app, particularly if my portfolio of apps is small. But if the trend holds true for a portfolio of thousands of applications, the impacts become more consequential. Or if the phenomenon is directly connected to a particular cloud provider and not seen elsewhere, again, the consequences become meaningful.

Where having “ground-up” predictive analytics (build and/or deployment automation) becomes much more useful is in the construction of release events. Often, release managers are asked to supply an estimated time duration for a given event to factor into the construction of a new release. If a vendor were to integrate more predictive analytical tooling into the release orchestration (RO) toolsets, “random” or “best guess” timing estimates could be replaced by extending historical data into the timing estimation fields. The RO tooling could provide a default timing estimation based on examining historical trends for a period defined by the user. Estimation defaults could always be replaced by a user (assuming discreet knowledge of other impacting events), but at least the new overall lengths of releases then could become significantly more accurate.

This has special significance for manual tasks. Let’s say our database admin team traditionally takes 40 minutes to perform a given release event update, but now is running significantly longer than that. Having the RO software aware of the historical trends could alert the release manager, as well as the database team manager, during the event to check out why this is occurring. For manual tasks in particular it may be hard for anyone besides a predictive analytics module to determine what timings “should” be, as compared with what they are in a given release.

Impacts to the Bottom Line

The real questions are, Why should a customer care about this kind of feature set? Why should we be looking at the vendors of our tooling to incorporate forward-thinking architectures such as predictive analytics into the tooling they sell us? Because the efficiency it can bring to our DevOps set of services lowers the costs of administration and increases the speed at which we can deliver innovation in general. Over time, trending combined with context can statistically prove continuous improvement of our DevOps services, which is the promise we made to the executive teams that sponsored and made the investment in DevOps to begin with.

Metrics analysis can occur (usually with a dedicated investment on our part) without a predictive view of the impacts. But this changes the perspective of our analysis to hindsight thinking instead of forward thinking. In essence, we then are simply “reacting” to what has occurred instead of tailoring our environment to alter the picture of future predicted events to be better than the software predicts it to be. We start tweaking the variables to instantly impact future predictions, instead of only examining them in a post mortem.

This perspective is yet another cultural catalyst for change your business needs. When I talk about changing how an organization “thinks” about innovation, this is one of the key objectives. We take an enterprise from knowing little about the costs of innovation, to having an extensive knowledge, to having a forward view of what is—or what should be, possible. All of this is possible despite the loss of waterfall controls and in the midst of an agile construction model. All it takes is the integration of forward-thinking architectures into the products we select. If our vendor does not offer them, we begin to enlighten our vendors on the value they would bring to us in cost reduction and to them in market capture. If you need help in these areas, feel free to contact me, I am looking at the moment 🙂 .

To continue the conversation, feel free to contact me.

June 1, 2016
Why DevOps for the database must include three way analysis

Deployment automation is critical to practicing effective DevOps. Without deployment automation, releases still require a lot of manual steps and processes. Processes and steps that aren’t reliably repeatable, are prone to human errors, and can’t be handled consistently with high frequency.

However, the software best practices can’t simply be grafted onto database deployments. Unlike software code, a database isn’t just a collection of files. You can’t simply copy a database among your different development, testing, and production environments. Deploying a database also presents a greater business risk since it holds your most valued asset – your data.

In order to capitalize on DevOp automation advantages for databases and to promote database changes safely and quickly, all the particular components of a database; its tables, functions, and content, need to be protected.

Any successful DevOps database automation solution must be capable of intelligently dealing with conflicts and merges of code and cross updates from other teams. The solution must be able to ignore wrong code overrides and give the user ultimate control over which changes and merges are pushed out.

In order to achieve these goals a DevOps for the database solution must be able to conduct a three-way analysis on the source (that is, the object after the development work is done), the original (or “baseline”) state, and the target (or “Production”) environment, identifying the differences among these three states. Team members, gaining full knowledge of the complete scope of the project from the analysis, can review the results and options the solution presents to resolve any discrepancies. The user selects what action, if any, should be taken on the database. The solution then automatically generates the scripts to execute the chosen actions.

In this way, the solution uses automated three-way analysis and automatic script generation to eliminate both the potential for human error in manual script writing, and the potential for technological error in automated object overrides.

Reaping the Rewards
Implementing a solution that encompasses all these automation features to meet the particular challenges of database deployments, would enable a company to practice a DevOps approach to effective database automation and realize the following benefits:

· Reduced deployment costs through streamlining of development process management

· Minimizing deployment risks by enforcing change policy and best practices, and configuration management and consistency

· Improving responsiveness to customers and market needs with speedier roll-outs of task-based development updates

· Enhancing cross-team communication with clear audit trails and transparency regarding work done on objects and the state of different environments

Can you afford to be without it?

About The Author ⁄ Yaniv Yehuda, CTO, DBmaestro

Yaniv is the Co-Founder and CTO of DBmaestro, the leading provider of DevOps for Database solutions which enable control of database development and deployment. Yaniv is also the Co-Founder and the head of development for the Extreme group, a leading IT services solutions provider group.

July 23, 2015
Documenting DevOps: Agile, Automation and Continuous Documentation
Chris Riley former GigaOm analyst, O’Reilly author, devopsy.kinsta.cloud featured writer and founder of Fixate I/O has written an outstanding report on the key role documentation must play in the modern software development lifecyle. Agile or DevOps, automated or not, we live in a continuous IT era. So why shouldn’t documentation be continuous as well? Riley’s analysis includes key findings including:
- Even the definition of documentation is subjective. Despite several decades of implementation, the methods, drivers, and even language of documentation are nearly limitless. Standards such as ITSM help stabilize the look, feel, and process of documentation, but it does not help an organization understand how to use that documentation and integrate it into development.
- The modern software delivery pipeline introduces new complexities in administration. New tooling in the DevOps space has improved application efficiency and quality, but in some cases, this has also reduced the manageability of the environment. This creates new challenges to knowing what you have, how it functions, and how to extend it into the future.
- As the delivery pipeline moves to a continuous stream of releases, classic documentation processes become impossible. Documentation must happen automatically within the process, or not at all.
- In order for documentation to be useful, it needs to be actionable.
Download the complete report on continuous documentation for an in depth analysis, conclusions and guidance:
Documenting DevOps
Name
This field is for validation purposes and should be left unchanged.
Name*
First Last
Company
Email*
May 13, 2015