Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Google complains to the FTC about Microsoft market abuses, and new graduates don’t know how to office. (more…)
Noname Security today made generally available an update to a tool for testing application programming interface (API) security that promises to make it easier for DevOps teams to ensure APIs are secure.
Filip Verloy, a field CTO for Noname Security, said Active Testing V2 is purposely designed to make it simpler to integrate API testing within a larger DevSecOps workflow.
Version 2 of Active Testing, for example, provides additional integrations with continuous integration/continuous deployment (CI/CD) platforms along with automated API discovery tools.
Other capabilities included in the latest release are tools to automate API response inspection, troubleshooting, classification and vulnerability discovery.
Finally, Noname Security has also added support for GraphQL-based APIs alongside REST APIs.
In total, there are now more than 160 API security tests that can be employed on an API testing tool that Noname Security developed to enable it to build Active Testing, said Verloy. In effect, Active Testing is an extension of that core API testing tool, he noted.
The biggest issue DevSecOps teams encounter when testing API security is that most legacy tools are not designed to provide any visibility into the business logic used to construct APIs. As such, there is a need for API security tools that can be easily integrated within DevSecOps workflows to discover issues before APIs are exposed in a production environment, he noted.
While a lot of progress has been made toward securing software supply chains, APIs have generally not received enough attention. Developers typically create APIs, but they don’t typically assume responsibility for managing and securing them. As the number of APIs within organizations continues to proliferate, there is a growing need to centralize API management and security.
Historically, developers have assumed that cybersecurity teams were making sure APIs are secure. Cybersecurity teams, conversely, assumed that the developers that created APIs would be responsible for securing them. DevSecOps best practices make it possible to now extend responsibility for security further left toward developers in a way that enables cybersecurity teams to define the policies and controls that should be used, noted Verloy.
It may a while before every organization masters the all the nuances of DevSecOps, but as pending legislation that requires organizations to better secure software supply chains becomes law it’s only a matter of time before organizations revisit API security—if they have not done so already.
In the meantime, cybercriminals have already discovered that insecure APIs provide a convenient mechanism for exfiltrating data that they can then sell or hold for ransom. The number of rogue and zombie APIs that are unknown or that have been forgotten has become a major cybersecurity issue. Ideally, DevSecOps teams would not only test the security of APIs before they are deployed but would also address a significant amount of technical API security debt that has already accumulated.
One way or another, API security is going to become a significant issue in the weeks and months ahead. The issue now is how to address it before it gets any worse.
At its HasuraCon 2023 conference today, Hasura launched a Data Delivery Network (DDN) based on a set of GraphQL application programming interfaces (APIs) that makes it simpler to integrate distributed sources of data.
In addition, Hasura is adding Hasura Schema Registry to streamline API governance using graph technology, a MongoDB connector, a Native Data Collector (NDC) for REST APIs and support for the Open Domain Data Specification (OpenDDS), an open framework for modeling APIs originally developed for the travel industry.
Finally, Hasura is adding support for Native Queries and Logical Models capabilities that convert SQL requests into API calls.
The Hasura DDN leverages the GraphQL API to enable sub-second integrations across a network that, in concept, is similar to how a content delivery network (CDN) is constructed.
Hasura CEO Tanmai Gopal said the goal is to enable sub-second integration of data at a global scale across a network with more than 100 regions. Developers can integrate their APIs in less than a second regardless of the number of data models connected. Metadata changes can also be used to create builds that can be independently tested and applied to projects to enable API version control and collaboration among development teams.
The Hasura DDN is accessed via Hasura Cloud, an instance of a data integration platform that can be deployed on multiple clouds, including Microsoft Azure, and will soon be made available for private clouds managed by internal IT teams.
GraphQL is gaining traction as an alternative to REST APIs for integrating data, but it’s more complex to implement and manage. The Hasura DDN extends the reach of a Hasura Cloud platform that provides a simpler method for adding GraphQL capabilities to an IT environment in a way the also conserves network bandwidth, noted Gopal.
GraphQL is a query language for APIs and a runtime for fulfilling queries that provides a complete and digestible description of the data in an API. It can also be used to pull data from multiple sources because it uses a schema that defines the different types of data available. REST APIs, in contrast, can only fetch data from a single source. As such, many organizations that are trying to drive adoption of reusable digital services are relying on GraphQL.
It’s not always clear who within an organization is assuming responsibility for managing the APIs that developers create, but as the number of APIs increases, DevOps teams, in many cases, are moving to centralize API management. In fact, the more data an organization creates, the greater the corresponding use of APIs to programmatically access that data becomes.
Hasura is betting that as organizations reach a critical mass of API integration requirements, it will make more economic sense to rely on a purpose-built data integration platform accessed as a cloud service versus relying on a set of tools and platforms they need to configure and manage. Regardless of approach, the one thing that is certain is the number and types of APIs that need to be managed are rapidly exceeding the ability of IT teams to keep pace.
System Initiative (SI) today made available in private beta a platform that creates simulations of IT environments. The platform leverages a built-in framework to generate the code needed to update an application environment.
Fresh from raising $18 million in funding, SI CEO Adam Jacob said the ultimate goal is to provide a level of abstraction for DevOps environment that eliminates much of the complexity that creates bottlenecks in DevOps workflows.
For all intents and purposes, DevOps is fundamentally broken; there are too many manual processes that need to be performed, he added. The SI platform represents an ambitious effort to reimagine how DevOps processes are constructed and maintained, said Jacob.
At the core of that effort are models of IT environments created by the SI platform that are accessed via visual interface. The automation framework built into the platform then uses the relationships between models to infer configuration dynamically and then automatically generate TypeScript code that a DevOps team can apply to automate a task.
Feedback loops are also provided in real time to provide insight into the viability of a configuration and eliminate the need for no state files to track and no ‘plan’ versus ‘apply’ stages.
As more organizations realize how dependent they are on software to drive digital processes, many of them are looking to improve the productivity of developers and the DevOps teams that support them. After a wave of initial progress, many DevOps teams find themselves supporting DevOps workflows using custom code that varies widely in quality. SI is making a case for a digital twin platform that generates a set of code that can be applied more consistently, rather than relying on a DevOps engineer to write code.
Ultimately, the goal is to launch a new DevOps era to eliminate most of the silos that slow down application development and deployment, noted Jacob. In fact, one of the reasons few organizations today fully automated DevOps across the application development and deployment process is that it’s simply too difficult to achieve using legacy tools and DevOps platforms, he added.
A recent survey conducted by the Continuous Delivery Foundation (CDF), for example, found only 22% of respondents are using CI and CD approaches to automate all building, testing and deployment of code in a production environment. Instead, many organizations are using a combination of DevOps and waterfall processes to manage application development and deployment.
It’s not clear how many organizations are willing to fundamentally change the way they have implemented DevOps. But at a time when more organizations are moving to centralize the management of DevOps, many of them are evaluating their options, at the very least. The issue, of course, is that multiple DevOps teams within the same organization have committed to disparate platforms. Enticing them to centralize on a common platform will require not just a compelling technical advantage but also a significant amount of political capital.
Cycloid today added a Bootstrap Stacks capability that provides DevOps teams with the ability to build Git templates or use ones created by the company to define cloud computing environments.
The capability is accessed via a portal added to the Cycloid platform for automating software delivery. It also makes it simpler for DevOps teams to reuse templates that are compatible with Terraform or Kubernetes environments as needed.
In the months ahead, Cycloid plans to release Headers, a metadata tool that automatically populates project parameters in display pages and a project management tool to provide a unified view for each development environment, including continuous integration/continuous delivery (CI/CD) pipelines, documentation, automation, logs, cloud costs and carbon emissions.
Cycloid founder Benjamin Brial said the templates provide a layer of abstraction that makes it simpler to implement GitOps workflows. As a best practice, GitOps enables DevOps teams to unify the management of code to provision IT infrastructure and software delivery. However, providing access to a Git repository alone is insufficient, said Brial. DevOps teams need access to a set of templates that automate the process and eliminate what is otherwise a complex set of tasks, he noted.
Easily discovered templates also eliminate organizations’ tendency to create redundant automations simply because a development team was unable to find an existing framework that another team already created, added Brial.
That approach also enables DevOps teams to more easily embrace platform engineering principles to centralize the management of DevOps using a Bootstrap Service catalog embedded within the Cycloid platform, said Brial. That capability makes it possible to enforce GitOps policies in a way the developers will have to respect because the automation frameworks are readily available, he added.
It’s not clear whether organizations are embracing platform engineering. Historically, adoption of DevOps workflows in most organizations has tended to flow from the bottom up. As a result, it’s fairly common for multiple application development teams within an organization to have adopted multiple DevOps platforms to manage workflows. Advocates for platform engineering are making a case for eliminating many of those redundant platforms in favor of a corporate standard.
At the core of any approach is the assumption that centralization will reduce costs by eliminating redundancy. Issues can arise, however, when centralization results in loss of flexibility. Many of the teams that adopt DevOps want to be able to add and replace tools as they see fit. If there is a corporate standard that narrows their tool options, it’s just a matter of time before those teams start to look for ways around the corporate standard.
Each organization will need to strike a balance between flexibility and the need to bring some order to DevOps workflows. In the meantime, however, the pace at which multiple software development initiatives are being launched is likely to force a platform engineering conversation that, in many cases, many organizations have postponed longer than anyone cares to admit.
Harness today added a raft of generative artificial intelligence (AI) capabilities to its continuous integration/continuous delivery (CI/CD) platform as part of a larger effort to reduce the level of manual toil that today conspires to create DevOps bottlenecks.
The added generative AI capabilities include an AI Development Assistant (AIDA), which analyzes log files and correlates error messages with known issues. It also includes the ability to use natural language to define policies for governing cloud assets and costs and an ability to automatically identify security vulnerabilities and generate code fixes.
Other generative AI capabilities that will be added to AIDA over the next three to six months include automated code reviews along with tools for creating pipelines and launching chaos engineering experiments.
Harness CEO Jyoti Bansal said these are the first of a series of generative AI capabilities that will be applied across the entire software development life cycle (SDLC) using large language models (LLMs) that the company has trained using publicly available sources. AIDA, at its core, provides a tool for creating custom prompts for launching queries against LLMs and a vector database that Harness has incorporated into its platform.
That approach means no customer data was used to create the model and none of the data used will find its way into a generative AI platform that is available to anyone, said Bansal. As a result, organizations don’t need to be concerned they light run afoul of licensing issues because proprietary code was inadvertently included in an LLM, he added.
Most of the existing generative AI capabilities that have been brought to market thus far are designed primarily to improve developer productivity, noted Bansal. Harness is now bringing the benefit of generative AI to DevOps teams that today rely on too many manual processes to manage the automations they build on behalf of application development teams, he added.
Harness is making AIDA available at no extra charge at a time when many DevOps teams will soon be evaluating the AI capabilities provided to them. Most DevOps teams are not going to want to manage workflows without being able to rely on AI to reduce toil and eliminate bottlenecks. Overall, Harness is forecasting generative AI will drive a 30% to 50% increase in DevOps productivity. In addition, Harness is forecasting there will be a 50% to 75% reduction in the time spent remediating vulnerabilities.
Bansal said that as that process occurs, Harness is betting more organizations will move to replace legacy CI/CD platform with more modern alternatives.
Previously, Harness has invested in machine learning algorithms to automate a range of DevOps tasks. Generative AI, in combination with machine learning, will transform DevOps workflows in ways that will ultimately improve both the rate at which applications can be built and their overall quality.
It’s not clear whether organizations might be willing to replace CI/CD platforms that they have been customizing for decades, but at the very least, that option is now more widely considered than ever.
Quali today revealed it added governance and cloud cost management tools to its Torque platform for automating IT infrastructure management.
David Williams, senior vice president for market strategy at Quali, said the added governance tools will make it possible to enforce security, compliance and consumption policies as IT environments are launched using the automation blueprints that Torque enables IT teams to create.
At the same time, IT teams can now also define consumption policies based on cloud runtime or regions employed within a cloud computing environment. In addition to taking advantage of dashboards to visualize consumption of cloud resources, IT teams can also use Torque to identify cloud waste including detached, inactive and overprovisioned instances. They can pause, right-size or tear down environments to reduce costs and eliminate the possibility of a billing surprise at the end of the month, noted Williams.
Torque also enables IT teams to detect drift and then update live cloud environments to remediate any that fall out of scope, he added.
Quali has been making a case for an environment-as-a-service approach to managing IT. That approach is based on a set of triggers for controlling infrastructure via a software-as-a-service (SaaS) platform that can be invoked either via an application programming interface (API) or a graphical user interface (GUI).
In addition to providing a set of infrastructure provisioning guardrails that prevent mistakes from being made, Torque makes it possible to automatically deploy complete application environments on top of infrastructure provisioned as code using blueprints rather than relying on low-level Terraform scripts. The goal is to increase developer productivity by enabling IT teams to create self-service portals that developers can use to spin up environments.
That approach is becoming more appealing as many organizations begin to adopt platform engineering as a method to centralize the management of DevOps workflows, noted Williams. In fact, as part of that effort, many organizations are starting to unify the management of cloud and on-premises IT environments, he added.
While many developers routinely provision IT infrastructure using tools such as Terraform, many of them lack the cybersecurity expertise required to ensure cloud computing environments are configured properly. Developers also tend not to remember to de-provision cloud resources when no longer needed; over time, those factors contribute to unnecessarily increased cloud costs.
Ultimately, the goal is to find ways to centrally manage IT environments to reduce costs and improve cybersecurity without inhibiting the pace at which developers are building innovative applications, said Williams.
At the same time, as application environments become more complex, it is becoming even more challenging to provision and manage IT environments at a time when many organizations are reluctant to hire additional IT staff, he noted.
One way or another, IT environments will become more automated. The issue is determining how best to organize the IT teams managing those environments as tasks that once required a lot of manual effort are increasingly automated.
Observability is maturing throughout the software industry. Observability is often defined as merging metrics, logs and traces to get a fuller picture of an application’s state. This can help pinpoint disruptions or security incidents and offer a clearer path to remediation. Observability is touted to advance the application monitoring practices of yesteryear, but are investments paying off?
According to recent findings, many organizations are using observability tools and seeing a positive outcome. A full 73% of organizations reported using observability tools for over a year. And impressively, 86% of observability leaders said the ROI on their observability tools has exceeded their expectations. The practice appears to be paying dividends for organizations that have gone “all-in” with mature observability processes.
The State of Observability 2023 report released by Splunk demonstrated how observability leaders are reaping a positive return from observability tools. According to the report, observability is helping to meet performance expectations, avoid downtime and enhance resilience in general. However, not all organizations are on the observability maturity bandwagon. Below, we’ll examine some of the key trends from the report and consider the benefits of refining your observability efforts.
Now that observability is more commonplace, dedicated roles are emerging to support these initiatives. In fact, 87% of respondents now employ specialists who work exclusively on observability projects. Site reliability engineers likely make up a portion of these experts, as observability data can be used to advance efforts around meeting service-level objectives (SLOs).
But although observability might be more common than a few years ago, not every observability practice is mature or standardized. Just 40% have a formal approach to resilience that has been institutionalized organization-wide. Because of this, only 43% are confident in their team’s ability to meet its application reliability and performance objectives. It should be noted that this number increases significantly for those with more mature observability adoption, which we’ll expand on below.
Furthermore, not all applications are cloud-native. Organizations reported operating and maintaining, on average, 165 internally developed business applications, half of which are hosted in the public cloud and half of which are on-premises. Due to the persistence of legacy modalities, observability tools must continue to support a hybrid architecture if they are to be comprehensive and useful.
And although we’ve previously covered how many organizations are drowning in a sea of observability data, observability leaders now say their ROI on observability tools far exceeded their expectations. Perhaps this indicates that, with more open standards, analysis centralization and fewer false positives, teams are finding ways to avoid observabiliy tooling sprawl.
Lastly, the report unveiled desires to loop AI and ML into the observability pipeline. In fact, 66% of observability teams already use AI/ML, while 26% are in the process of deploying the technology. And 91% said AIOps is an important enabler of their observability goals. Certainly, AI is becoming a pervasive instrument to accelerate many areas, including observability.
An impressive 64% reported that their investment into observability has exceeded their expectations. So, what are some of the top benefits of observability? Well, the report found the topmost benefits to be areas like decreasing problem detection and resolution times, identifying “true positive” incidents, and enhancing application security.
Observability is being deployed as a tool to evolve traditional security efforts, as it uncovers a deeper understanding of application behaviors. This can help security teams correlate errors to specific diagnoses and spot potential threats in a quick manner. As such, 59% said observability helps them uncover security issues thanks to the intelligence capabilities of observability solutions. And 55% say they can uncover more security vulnerabilities thanks to the visibility afforded by observability.
Observability can also be used to improve visibility into hybrid architectures. A full 86% said it’s important to have flexible observability solutions that cover hybrid architectures, though half identify this as an area for improvement. Although we like to think that cloud-native development is the status quo, many enterprises still oversee a mixture of on-premises, private cloud and multi-cloud environments. A unified, agnostic observability layer is thus essential to peek into all these disparate environments to grapple with disparate data outputs.
Service interruptions can have negative consequences for organizations, including lower customer satisfaction, loss of revenue and loss of customers. In fact, 76% of respondents reported downtime can cost up to $500,000 per hour.
Thankfully, observability has emerged to help improve overall resilience and avoid these negative repercussions. But, to truly reap the benefits, you must have a mature practice in place. As defined by the report, “observability leaders” have a more formal approach to resilience and have standardized this across their organization. And 89% of those leaders are completely confident in their ability to meet their application availability and performance requirements. The catch-22 with observability is that you must invest more energy into formalization and knowing when the effort is worthwhile is a moving target.
A solid observability practice is supported by standardized metrics, logs and traces across the organization. Furthermore, to make observability behave correctly, you will need flexible tooling that can leverage data from multiple sources, such as the network, infrastructure, servers and databases.
The State of Observability 2023 report surveyed 1,750 observability practitioners from around the globe, making it the most extensive study on observability to date. For more insights, you can pick up a copy of the report behind an email gate here.
DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.
Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.
Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.
The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.
We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:
Indeed.com
SAP
San Francisco, California
DevOps Engineer
$141,378 to $162,000
Colgate-Palmolive
Piscataway, New Jersey
DevOps Lead
$112,000 to $164,800
CareerBuilder.com
Northrup Grumman
Dulles, Virginia
Principal DevOps Engineer
$104,600 to $157,000
SimplyHired
Crate & Barrel
Northbrook, Illinois
DevOps Engineer
$99,100 to $125,000
Dice.com
Tata Elxsi Ltd.
Philadelphia, Pennsylvania
DevOps Engineer
$80,000 to $100,000
At its virtual PulumiUP event, Pulumi this week announced it embedded an ability to convert instances of Terraform into its infrastructure-as-code (IaC) platform via a simple command line interface (CLI) command. In addition, Pulumi added a Review Stacks capability that automatically creates a dedicated cloud environment to review code every time a pull request is opened.
Pulumi is also making available a beta release of Pulumi Azure Native Provider 2.0, which decreases the size of the software development kit (SDK) by 60%. This is part of an effort to improve deployment times and reduce resource consumption by, for example, removing support for deprecated application programming interfaces (APIs).
Finally, Pulumi has added support for property names and values to Resource Search for Pulumi Cloud, a search and analytics tool for cloud computing environments. DevOps teams can now additionally search for instances of, for example, versions of databases or specific availability zone as a result.
Aaron Kao, vice president of marketing for Pulumi, said previously the company made available a separate tool for converting snippets of Terraform code. For the first time, Pulumi is also making available a converter plugin in the Pulumi engine which allows conversion tools from other IaC tools to be used.
Review Stacks, meanwhile, makes it simpler for DevOps teams to pick up an unfamiliar codebase, make changes to both application and infrastructure code and share a live environment to review that code.
Pulumi has been making a case for an alternative to Terraform that centralizes the management and provisioning of infrastructure using a set of embedded guardrails. One of the major issues enterprise IT organizations are encountering these days is the number of misconfigurations created when developers use tools like Terraform to provision infrastructure on their own. Those misconfigurations often lead to, for example, ports being left open and cybercriminals exfiltrating data.
That approach is gaining traction among enterprise customers that are looking to centralize the management of DevOps processes, said Kao.
In the wake of a series of high-profile breaches, more organizations are now reviewing the security of their software supply chains, including how infrastructure is provisioned. Many of those organizations will need to revisit those processes as pending legislation that requires more stringent requirements for building software eventually become law, noted Kao.
At the same time, many organizations are also looking to improve the productivity of DevOps teams during an economic downturn that limits the number of full-time employees they might be able to hire. The pace at which software is being developed has not slowed, so there is a greater urgency to do more with less.
It’s not clear how quickly organizations are centralizing the management of DevOps workflows, but with the rise of platform engineering, there is clearly more interest in defining consistent backend processes that multiple DevOps teams are currently managing individually. The challenge is finding a way to achieve that goal without impeding the pace of innovation enabled by embracing DevOps in the first place.