devopsy.kinsta.cloud

Category: IT as Code

Survey Sees Spike in Untested Code Leading to a DevOps Crisis

A survey of 500 developers conducted by the market research firm OnePoll on behalf of Sauce Labs found more than two-thirds (67%) admitted they pushed code into a production environment without testing, with more than a quarter (28%) acknowledging they do so on a regular basis.

More troubling still, 60% also admitted using untested code generated by ChatGPT, with more than a quarter (26%) doing so regularly.

More than two-thirds of developers have also merged their own pull requests without a review, with 28% confessing they do so often or very often.

Finally, the survey found three-quarters of developers admitted to circumventing security protocols, with 39% of developers doing so routinely. A full 70% also acknowledged using a coworker’s credentials to circumvent restrictions for access to data and/or internal systems, with 41% doing so regularly.

Jason Baum, director for community for Sauce Labs, said that while the survey makes it clear there are a lot of instances where best practices are being ignored, much of this behavior can also be attributed to the amount of extra work developers now routinely take on. The survey found more than three-quarters of developers (77%) have, for example, assumed more responsibility for testing in the last year.

The issue organizations need to evaluate is how much of this is due to developers being lazy versus looking for shortcuts in an era where too many tasks have been shifted left to developers and which they are not able to absorb, said Baum. Many organizations might have better outcomes if more tasks were automated within the context of a DevOps workflow to reduce the amount of cognitive load being placed on developers, he added.

The simple fact of the matter is there is a fundamental skills gap, said Baum. The number of so-called full stack developers capable of managing the entire software development life cycle is relatively small, he noted.

In the meantime, the amount of code generated is steadily increasing as developers take advantage of generative AI platforms to improve productivity. The challenge is the large language models (LLMs) being used to create that code were trained using examples of code pulled from across the web. In addition to having known vulnerabilities, it is of varying quality. In fact, those LLM platforms will frequently generate different snippets of code that vary in quality at different times.

DevOps teams that are responsible for managing the overall codebase are now finding their pipelines are becoming overwhelmed. That’s becoming especially problematic at a time when governments around the world are crafting legislation that will hold organizations more accountable for the security of the applications they build and deploy.

Eventually, AI will also be used to increasingly automate DevOps workflows and should enable software engineers to keep pace with the accelerated rate at which code is being written. In the longer term, LLMs that have been trained using vetted code will also generate higher quality code than a general-purpose LLM such as ChatGPT. In the meantime, however, massive amounts of untested code will likely create a series of downstream cascading events that will come back to haunt DevOps teams in the months and years ahead.

October 20, 2023
Survey Surfaces Benefits of Applying AI to FinOps

A survey of 200 enterprise IT decision-makers published this week found organizations that have infused artificial intelligence (AI) into financial operations (FinOps) workflows to reduce IT costs are 53% more likely to report cost savings of more than 20%.

Conducted by the market research firm Foundry on behalf of Tangoe, a provider of tools for managing IT and telecommunications expenses, the survey found organizations that embraced FinOps without any AI capabilities averaged less than 10% in cost savings.

The top three drivers for adopting FinOps/cloud cost management programs are the need to increase cloud resource production and performance (70%), reduce budgets (60%) and rising costs (58%), and simpler overall program management (50%), the survey found. Major benefits included productivity savings (46%), cost savings (43%) and reduced security risks (43%). Nearly two-thirds of respondents cited service utilization and right-sizing of services as another reason to embrace FinOps.

FinOps describes a methodology for embedding programmatic controls within DevOps workflows to reduce costs. In the face of increased economic headwinds, IT leaders are looking to reduce cloud computing costs, but it’s turning out to be more challenging than many of them anticipated. Cloud infrastructure is typically provisioned by developers using infrastructure-as-code (IaC) tools with little to no supervision. The reason for this is developers have long argued that waiting for an IT team to provision cloud infrastructure took too long. Developers would be more productive if they just provisioned cloud infrastructure themselves.

However, after ten years of cloud computing, it’s become apparent there are a lot of wasted cloud infrastructure resources. Developers who don’t pay the monthly bills for cloud services tend to view available infrastructure resources as essentially infinite. It’s usually not until someone from the finance department starts raising cost concerns that developers even become aware there might be an issue.

The challenge is that adopting FinOps best practices is not quite as easy as it might seem. In fact, more than half (54%) of survey respondents cited challenges in building the right process and human support systems for FinOps into workflows that have been in place for years.

Chris Ortbals, chief product officer for Tangoe, said the simplest path to FinOps is to rely on a software-as-a-service (SaaS) platform designed from the ground up to leverage AI to help IT teams manage cloud computing and telecommunications expenses both before and after applications are deployed. Each DevOps team will ultimately need to determine how much they will implement metrics to foster more efficient consumption of cloud computing resources. The more aware of those costs DevOps teams are, the more likely that better decisions about what types of workloads should be run where and, just as importantly in the age of the cloud, at what time, given all the pricing options provided.

Developers, of course, tend to jealously guard their prerogatives. Convincing them to give up their ability to provision cloud infrastructure on demand is going to be a challenge, at least until someone makes it plain how much all those cloud instances wind up costing the organization each and every month.

October 19, 2023
Atlassian Makes Compass IDP Generally Available

Atlassian this week made generally available a Compass developer experience platform through which DevOps teams can employ a software-as-a-service (SaaS) platform to centralize the management of the environments used to create code.

Taylor Pechacek, head of product for Compass at Atlassian, said Compass provides an instance of an internal developer platform (IDP) that is simpler for DevOps teams to set up and maintain.

At its core, Compass provides access to a software component catalog to track, search and discover the component and approved infrastructure that can be used to build and deploy applications.

In addition, templates enable developers to self-service environment setup using a set of policies that provide guardrails to ensure developers follow best practices for provisioning cloud resources and setting up pipelines.

Finally, Atlassian is making available scorecards based on the DevOps Research and Assessment (DORA) metrics defined by Google.

Compass is designed to be integrated with a wide range of continuous integration/continuous delivery (CI/CD) platforms and third-party observability, testing, collaboration and source code management tools, said Pechacek. The goal is to minimize context switching as much as possible for developers accessing the Compass platform, he added.

That approach also provides a level of consistency that both fosters collaboration and makes it simpler to onboard new members to a development team, noted Pechacek.

While IDPs as a concept have been around for some time, they are now central to any effort to embrace platform engineering methodologies to implement best DevOps practices at scale. The issue platform engineering teams need to come to terms with is determining whether they want to build and maintain an IDP versus invoking one provided as a SaaS application.

Regardless of approach, platform engineering teams are trying to strike a delicate balance between centralizing the management of development environments and the ability of developers to innovate by adding new tools to their development environment. Organizations clearly want developers to become more productive by spending less time managing development environments, but many of those same developers tend to jealously guard their prerogatives when it comes to the tools they prefer.

The challenge, of course, is that spending less time managing a development environment doesn’t necessarily mean more code will be written. All it does is afford developers the opportunity to write more code. The issue is that the writing of quality code only really occurs when developers are inspired, but at the very least, configuring a development environment shouldn’t stand in the way of creativity.

It’s not clear how many organizations are embracing platform engineering methodologies, but it’s apparent that more supervision is being brought to how applications are built and deployed within the context of a set of DevOps best practices. The challenge and the opportunity now is to make sure developers buy into, rather than resist, those efforts.

October 18, 2023
GitGuardian Adds Tool for Discovering Secrets in Public Repositories

GitGuardian today added a tool that makes it possible for DevOps teams to search GitHub repositories to determine if application secrets have inadvertently found their way into other applications.

Vikram Ghosh, chief growth officer for GitGuardian, said a HasMySecretLeaked toolset makes it possible to determine if developers have copied and pasted application secrets from one repository into another without permission.

The toolset is based on a private database that houses more than 20 million records of hashed secrets that have been found in public sources, including GitHub.com. Users can query the database by submitting a hashed version of their secret in the search console. The GitGuardian toolset will then search for perfect matches without revealing any other secrets or their locations, said Ghosh.

Alternatively, DevOps teams that have adopted the GitGuardian platform to protect secrets can use a command-line interface (CLI) exposed via a ggshield tool to launch queries. That tool also includes plug-ins for pulling secrets from tools such as HashiCorp Vault and AWS Secrets Manager and staging them in local environments to surface potential leaks. Eventually, HsMySecretLeaked will be extended to add support for additional repositories, noted Ghosh.

Collectively, those capabilities make it possible for organizations to audit how application secrets are being secured, he added.

The GitGuardian platform scans every public commit on GitHub for leaks, spanning API keys, database assignments and developer secrets. In 2022, GitGuardian found 10 million secrets that had been exposed.

Secrets management is, of course, getting more attention as more organizations review their software supply chain processes in the wake of a series of high-profile security breaches. It’s still early days as far as adopting DevSecOps best practices to secure those software supply chains, but it’s clear more responsibility for application security is being shifted left toward developers and the DevOps teams that support them.

The challenge is that many developers still hard-code secrets in plain text into applications to create shortcuts as software is developed. Unfortunately, many of them forget to remove those secrets before an application is deployed in a production environment. Once an application is deployed in a production environment, it may be months before anyone discovers there is an issue—if it’s discovered at all. In the meantime, cybercriminals have become more adept at using scanning tools to discover those secrets.

DevOps teams can reduce the number of applications that might have this issue by making it easier to discover secrets within applications before they are deployed. In the meantime, it’s only a matter of time before more stringent regulations require organizations to revisit the security of software supply chains. Countries around the world are debating legislation that would hold organizations much more accountable for the security of the applications they build and deploy.

Hopefully, DevOps teams will implement DevSecOps best practices to get in front of those issues before any fines are levied. But the challenge, as always, is ensuring application security without adversely impacting the rate at which software is currently being built.

October 17, 2023
HashiCorp Streamlines Workflows to Improve Productivity

HashiCorp today launched a series of tools to address everything from streamlining the way infrastructure is managed and applications are delivered to how secrets are kept.

Announced at the HashiConf Global conference, these latest offerings extend the Terraform and Vault portfolio to enable DevOps teams to programmatically automate DevSecOps workflows.

Specific capabilities added to HashiCorp Terraform and Terraform Cloud include a test framework to, for example, reduce misconfigurations along with, in beta, tools for generating tests, tools for launching tests across integrated Terraform modules and a private preview of tools for provisioning infrastructure at scale. In addition, the company is generally making available an enhanced editor for finding and resolving errors in code created using Visual Studio Code.

In terms of costs, Hashicorp is also making generally available an ephemeral workspaces capability to limit the amount of infrastructure resources consumed by Terraform itself.

At the same time, HashiCorp has extended Waypoint on the HashiCorp Cloud Platform (HCP) to define golden paths and workflows for application delivery using templates and add-on modules that make it simpler to identify dependencies. Waypoint was previously announced and is still in beta.

Finally, Hashicorp is adding an alpha edition of HCP Vault Radar, a tool to scan, identify and remediate secrets inadvertently stored in software, in addition to making HCP Vault Secrets generally available and adding a secrets synchronization capability, available in beta for Vault Enterprise and HashiCorp Vault 1.15. HCP Vault Radar is based on technology that HashiCorp gained with the acquisition of BluBracket. HCP Vault Radar will be available in beta in early 2024.

Chris Van Wesep, senior director for product marketing for HashiCorp, said, in general, the company is committed to making it easier to securely manage IT infrastructure and applications at scale across multiple IT environments.

While infrastructure-as-code (IaC) tools such as Terraform have been a boon to productivity, the developers that typically employ these tools have little to no cybersecurity expertise so it’s not uncommon for mistakes to be made. The issue organizations are struggling with today is cybercriminals have become more adept at exploiting those vulnerabilities.

At the same time, secrets management has become a significant issue because many developers will store secrets in plain text to streamline workflows while building an application. Unfortunately, many of these secrets wind up being exposed in a production environment simply because no one remembered to remove them.

It’s not clear to what degree organizations are centralizing the provisioning of IT infrastructure to improve cybersecurity. As forthcoming regulations increasingly require them to lock down their software supply chains, however, it’s only a matter of time before most organizations that build software will need to revisit their DevSecOps workflows.

In the meantime, the hope is those goals can be achieved without adversely impacting developer productivity. The level of cognitive load on developers today is already too high, so the more processes are automated, the less time developers should have to spend managing infrastructure. The challenge, as always, is achieving that goal in a way developers will actually embrace.

October 11, 2023
Microsoft kills Python 3.7 ¦ … and VBScript ¦ Exascaling ARM on Jupiter

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: VS Code drops support for Python 3.7, Windows drops VBScript, and Europe plans the fastest ARM supercomputer. (more…)

October 11, 2023
Pulumi Adds Internal Developer Portal to Portfolio

Pulumi today added a portal to its portfolio, dubbed Pulumi for Platform Teams, to centrally manage environments through which developers can self-service their own requirements within the context of a set of pre-defined guardrails that ensure compliance and improve security.

In addition, the company also announced general availability of the previously launched Pulumi Deployments orchestration tool for deploying workloads.

Pulumi CEO Joe Duffy said both offerings are part of an ongoing effort to extend the company’s portfolio of offerings beyond infrastructure-as-code (IaC) tools.

Pulumi for Platform Teams is designed to be deployed via the Pulumi Cloud platform for configuring infrastructure, and provides DevOps teams with a service catalog that can be integrated with DevOps workflows via a REST application programming interface (API). That approach makes it possible to leverage hundreds of existing Pulumi CrossGuard policies for automating compliance. In addition, there is a policy-as-code engine to author policies that automatically correct configuration violations.

Pulumi for Platform Teams also includes a plug-in for Backstage, an open source project for building internal developer platforms (IDPs) originally developed by Spotify that is now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

With the launch of Pulumi for Platform Teams, the company is providing the building blocks that will enable DevOps teams to build and maintain an IDP that complies with multiple regulatory requirements, noted Duffy.

IDPs are at the core of many efforts to embrace a platform engineering methodology that promises to enable organizations to manage DevOps workflows at scale. The overall goal is to reduce the amount of time developers spend on maintaining environments to enable them to spend more time writing code while at the same time reducing the total cost of DevOps by eliminating redundant tools and processes.

Regardless of approach, given the current overall state of the economy, there is more pressure than ever to improve developer productivity. The goal is to enable developers to self-service their own requirements as much as possible within the context of a standard set of “golden path” workflows. When a developer requests a new environment, the IDP uses calls to application programming interfaces (APIs) to set up a namespace, updates configurations, pulls any images that might be needed and connects to eternal resources such as databases. The IDP should provide developers with enough flexibility to employ new tools when required without unnecessarily adding tools and processes that are redundant to capabilities that already exist.

As every IT leader knows, that’s a fine line to walk, so they would be well-advised to embrace platform engineering in phases. The most critical asset any DevOps team has is culture. Major changes to existing workflows that developers have not bought into are likely to do more to increase friction and strife than to reduce them. When it comes to change management, slow and steady usually wins the race. After all, the worst thing that can happen after embracing platform engineering is for developer productivity to decline rather than improve.

October 11, 2023
Pulumi Previews Tool to Integrate Secrets and Infrastructure Management

Pulumi today previewed a tool that enables DevOps teams to unify the management of environments, secrets and configuration (ESC).

ESC is designed to enable developers to define reusable environments that combine secrets from multiple sources, including Pulumi IaC, AWS KMS, Azure Key Vault, Google Cloud KMS, OpenID Connect (OIDC) Relying Parties, 1Password and HashiCorp Vault. ESC works with any cloud execution context or tool, including Pulumi, Terraform, Cloudflare Workers, GitHub Actions or Docker.

Pulumi CEO Joe Duffy said multiple environments can be defined and composed together to make it simpler to provision cloud environments at scale and eliminate copy-and-paste errors. That approach streamlines a complex process that is fraught with opportunities for mistakes to be made, including inadvertently leaving secrets exposed in an application after it has been deployed in a production environment, he added.

ESC can manage both static and dynamic short-lived secrets, now considered a best practice for secrets management, in a way that can easily be audited, noted Duffy.

ESC can be run independently of Pulumi’s existing infrastructure-as-code (IaC) tool, or it can be integrated with the company’s identity and role-based access control (RBAC) tools. ESC includes deep integration with any SAML IdP, including Azure AD, Microsoft Entra ID, Okta and Google Workspace.

One of the primary reasons cybercriminals target cloud computing environments is because the developers that provision these services have little to no cybersecurity expertise. As a result, the probability a cloud service will be misconfigured is fairly high. As organizations add additional cloud services, it becomes more probable that mistakes will be made as developers programmatically invoke services in an unfamiliar IT environment.

Pulumi ESC is available for free as a public preview today with the intent to eventually offer multiple tiered versions, including a free offering and others with advanced Enterprise and Business Critical capabilities. The ESC client SDKs, CLI and plugins are all open source.

In general, Pulumi has been committed to an open source strategy that clearly defines what capabilities are going to be freely available versus those that can only be found in the other available tiers of service, said Duffy. One of the issues that too many open source software companies encounter is that they can’t generate enough revenue from subscriptions and services to sustain themselves, he noted. Pulumi, in contrast, has been squarely focused on providing additional capabilities that can only be found in its commercial offerings and provide enough value to warrant licensing, said Duffy.

It’s not clear whether organizations are moving to change the way they provision cloud infrastructure and manage secrets. As the amount of scrutiny of software supply chain management increases, it’s now only a matter of time before those processes are, at the very least, reviewed. The challenge, of course, is that when it comes to how those tasks are managed today, there is a lot of cultural inertia to overcome before development teams adapt to a new approach to provisioning infrastructure and managing application secrets.

October 10, 2023
Latest DORA Report Highlights Importance of DevOps Culture

A 2023 Accelerate State of DevOps Report published today by the DevOps Research and Assessment (DORA) team at Google Cloud found teams with generative cultures composed of people who felt included on their team have 30% higher organizational performance than organizations without a generative culture.

The report was based on a global survey of 36,000 IT professionals and tracked deployment frequency, change lead time, culture, change failure rates and failed deployment recovery times.

Overall, the survey found that while just under a third of respondents were successful in terms of their ability to deploy on demand (31%) and keep remediation efforts after deployments to 5% (33%), only 18% had lead times for changes of less than one day, while only 17% had a failed deployment recovery time of less than an hour.

This is an area where DevOps teams appear to need the most amount of help; nearly two-thirds (64%) of software releases or updates required some level of remediation after deployment, according to the report.

Nathen Harvey, developer advocate for DORA and Google Cloud, said the one area most underappreciated when it comes to improving performance is documentation. High-quality documentation leads to 25% higher team performance, the survey found.

DevOps teams that rely on the public cloud also benefited from a 22% increase in infrastructure flexibility, resulting in 30% higher organizational performance, noted Harvey.

The survey also noted that, as far as reliance on artificial intelligence (AI) to automate DevOps workflows is concerned, it’s still early. But most respondents are, at the very least, experimenting with one or more use cases.

In general, the management of DevOps will soon be very different as more AI capabilities are leveraged to eliminate bottlenecks and reduce toil. The immediate challenge is developers are benefiting more from AI than software engineers, which is resulting in more code starting to simultaneously move through DevOps pipelines. There is little doubt that in the months ahead there will be increased reliance on AI to manage pipelines at greater scale. In the meantime, it will be increasingly difficult to manage codebases that are steadily getting larger as developers become more productive.

At the same time, more organizations are embracing platform engineering as a methodology for centralizing the management of DevOps workflows at levels of scale that previously were difficult to achieve.

Hopefully, the combination of platform engineering and AI will result in less overall burnout among DevOps teams. In the short term, the burnout issue may become more challenging to address as the amount of code being created increases.

In the longer term, the results of the DORA survey may be substantially different next year as AI and platform engineering are employed more widely. The current survey results are not substantially different than in previous years, in the sense that DevOps teams are still challenged by many of the same issues.

Of course, as important as it might be to improve those metrics, the most important thing any DevOps team should focus on is developer productivity which, despite the existence of DevOps pipelines, remains relatively low.

October 5, 2023
Something is Amiss in the Open Source World
You don’t have to be a psychic to grok that the winds of change are blowing hard in the open source world. For the last few months, I’ve seen hints and heard whispers about a new direction for what I believe is the most passionate and enthusiastic community in tech. I think we are about to sail off into unchartered territory—and it will be interesting to see what the repercussions are.

Of course, this will all play out against the backdrop of continued layoffs and belt-tightening in the broader tech sector. Before I detail what I think are the telltale signals of change, let me state that I am and have always been a big supporter of open source software. I firmly believe we live in a golden age for open source software, which I call the “Foundation era of open source.” At a time when 90% plus of organizations use open source software, many of the leading open source projects are under the domain of not-for-profit foundations (NFPs) such as the Linux Foundation, the Apache Foundation, etc. This allows multiple commercial companies who might otherwise compete with each other to work in coop-etition. This has, in many ways, enabled this golden age of open source. Prior to the Foundation era of open source, there were two prior dominant models. Initially, there was what I call the “Cathedral and Bazaar era,” named after the great book by Eric Raymond. This was a time of idealistic notions of “pure” open source, unsullied by commercial aspirations. To me, Dr. Richard Stallman embodied that time; if you were to make an analogy to history, it was akin to the 1960s era in the U.S.

From that era, let’s zoom to sometime in the early 2000s when we moved to what I call the “Big Brother era” of open source. In this era, open source projects were owned and managed by one vendor, for all intents and purposes. The project could have a vibrant community, but ultimately, it was the managing vendor or “Big Brother” who reaped most of the rewards. This made it difficult for multiple tool vendors to support the same project and stifled progress.

The dawn of the foundation era of open source changed all that, and we have seen transformational growth as a result. But nothing lasts forever—and I think the king of the hill may be on the way down. Why do I say this? Here are several events/trends I’ve noticed recently that lead me to believe this. Some of these things I was told in confidence, so I cannot reveal names to protect the innocent.
1. The move away from “pure” open source licenses. Over the last few months, several pillars of the open source world have moved away from accepted open source licensing. Red Hat has done so with Red Hat Enterprise Linux (RHEL) and now Hashi has, as well. Now, to some, this change is subtle and it’s not a big deal. Only paying customers get access to source code. The overwhelming majority of users never look at the source code anyway. Nevertheless, this is a huge red flag for the community—the accepted licenses and a core tenant of open source, that source code is available to everyone, are no longer being used.
2. Diminishing corporate financial support for open source communities and foundations. I guess this was to be expected in hard economic times. The path to profits from supporting open source foundations, projects and communities can sometimes be more of a dotted line than direct. And a dotted line doesn’t usually work when the bean counters are involved. In looking to cut expenses, these are low-hanging fruit.
3. Closing foundations. This is a natural result of the economic strife. Foundations may operate as not-for-profits, but without operating funds, foundations can’t operate. Generally, foundations get a lot of mileage from volunteers. However, volunteers can only go so far before hard costs are involved. Also, volunteers are able to volunteer because their corporate employers empower them to do so. When their employers pull back the funding for those activities, the volunteers have to do what they can to earn a living. I have heard of at least one NFP foundation forced to cease or curtail operations due to corporate sponsor funds drying up.
4. Orphaned projects. Again, this is a direct result of number two, above. With the foundations closing and supporting funds drying up, I know of at least two bellwether open source projects that are in danger of becoming orphaned. Both of these projects boast huge community numbers and dominate their markets. The fact is though, that supporting a major project costs hundreds of thousands of dollars. While millions of community members may use a project and support it, they don’t contribute dollars. Dollars that pay for hosting, infrastructure and other hard costs. Coughing up a few hundred grand to support a project is a bridge too far for some companies in the current economic climate.
What does this mean for open source software and you? Well, in the near term, if you are just a consumer of open source software, probably not much will change day-to-day. You may have to cough up some money if you want access to source code or support from the community. Bigger picture, though, is the winds of change are blowing. I think we are going to see more companies taking a Hashi-type approach to protecting their source code. This will result in more forks of open source projects. However, my experience is that most open source forks don’t succeed (with some notable exceptions—like Jenkins forked from Hudson). Companies who are building businesses around open source models will be the most at risk. I think an open core business model may be the best bet for an open source vendor. I think the future of NFPs as the stewards of open source may become a bit cloudy (no pun intended). What will take its place or fill the void? That remains to be seen.

For the record I was never a fan of that old Irish proverb, “May you live in interesting times.” This kind of upheaval is a little too interesting for me. But one thing I have learned in 30 years in tech (and more than that in life) is that change is constant. You can embrace it and thrive or shrivel and die.
October 5, 2023