Madelyn Olson, an Amazon Web Services (AWS) software engineer and a Valkey maintainer, announced at the Open Source Summit in Seattle the availability of the first release, Valkey 7.2.5 GA. This release is based on the last open source version of Redis, 7.2.4. That’s good news, but it’s not big news. That’s what we expect from forks.
Rather, the important news is the immediate support for Valkey.
AWS, Ericsson, Google Cloud, Oracle and Verizon had all said they were supporting the fork. Now Alibaba Cloud, Aiven, Heroku and Percona are backing Valkey as well. I’ve seen many forks take off and overcome their parent projects – LibreOffice over OpenOffice comes to mind — but I’ve never seen one where so many Fortune 500 businesses spun on their heels to drop one project for another.
Olson said in her keynote speech, “These companies are planning to continue or even increase their contributions. Their support is needed for the long-term health and viability of the project. The community is thankful for their support, which will help keep the project open and keep our developers moving at the pace that our users demand.”
The Valkey fork’s maintainers have the technical chops needed to fulfill those business needs. As Olson pointed out, its technical leadership committee has 26 years of experience and over 1,000 commits to the open source project. Indeed, Tencent’s Binbin Zhu, a committee member, alone contributed almost a quarter of the open source Redis commits.
This response is not what Redis had in mind.
At the time of the fork, Redis CEO Rowan Trollope didn’t seem too worried. “The major cloud service providers have all benefited commercially from the Redis open source project, so it’s not surprising that they are launching a fork within a foundation,” he wrote. And he added, “Innovation has been and always will be the differentiating factor between the success of Redis and any alternative solution.”
With so many top maintainers now at Valkey, that assertion may be hard to pull off. As Heroku’s CTO Gail Frederick said during the conference, “With this group of experienced contributors and broad industry backing, Valkey is continuing the open source model that’s been transformative for developers everywhere.”
It appears that many of those customers are now on their way to Valkey.
Kyle Davis, an AWS senior developer advocate and former Redis employee, said, “AWS is committed to supporting open source Valkey for the long term. We are adding Valkey support to our ElastiCache and MemoryDB managed database services, which are built on open source Redis.”
Karan Singh, Oracle’s Director of Data and AI Product Management, wrote that “Oracle Cloud Infrastructure (OCI) Cache will continue supporting its current Redis 7.0-based service because the license changes don’t impact it.” However, Oracle is dropping the name Redis from its offering and is “working on integrating Valkey support into the OCI Cache service to ensure our unique in-memory database offerings on OCI continue to benefit from open source advancements, enhancing value for our customers.”
Redis’s move attracted a lot of snarkiness. For example, Corey Quinn, chief cloud economist at The Duckbill Group, an AWS analyst firm, tweeted, “Good morning to everyone except the thieves at Redis. You didn’t build it; you stole it from the community. If you want it from the ones who built it, it’s called Valkey.”
What we’re seeing here is not open source fans copping an attitude. It’s companies and developers alike making a major move away from Redis.
Supply-chain vulnerabilities loom large on the cybersecurity landscape, with threats and attacks such as SolarWinds, 3CX, Log4Shell and now XZ Utils underscoring the potentially devastating impact of these security breaches. The latter examples of Open Source Software (OSS) attacks are a growing attack vector. Capterra’s research reveals that software supply-chain attacks impacted almost two-thirds (61%) of all U.S. businesses from April 2022 to April 2023.
Expect attacks on the open source software supply chain to accelerate, with attackers automating attacks in common open source software projects and package managers. Many CISOs and DevSecOps teams are unprepared to implement controls in their existing build systems to mitigate these threats. In 2024, DevSecOps teams will migrate away from shift-left security models in favor of “shifting down” by using AI to automate security out of the developers’ workflows.
Here are the factors fueling the increase in software supply-chain attacks and the role of AI in helping developers work more efficiently while creating more secure code.
Attacks on the Open Source Software Supply Chain Will Accelerate
Open source libraries and languages are the foundation of over 90% of the world’s software. In a U.S. survey of nearly 300 IT and IT security professionals, 94% reported their companies use open source software, and 57% employ multiple open source platforms. Exactly half of the respondents say the threat level is “high” or “extreme,” while another 41% view it as “moderate.” At the time of writing, details of the backdoor implanted in the XZ library and several other OSS packages have just been published. The ubiquity of open source worldwide is one key factor fueling the rise of supply chain attacks.
Data Governance and the Data Supply Chain Will Become Critical Issues
Security pros must also consider how security vulnerabilities extend to their data supply chains. Although organizations typically integrate externally developed software through their software supply chains, their data supply chains often need clearer mechanisms for understanding or contextualizing data. In contrast to software’s structured systems or functions, data is unstructured or semi-structured and faces a wide array of regulatory standards.
Many companies are building AI or ML systems on top of enormous data pools with heterogeneous sources. ML models on model zoos are published with minimal understanding of the code and content used to produce the models. Software engineers need to handle these models and data just as carefully as they do the code going into the software they’re creating, with attention to its provenance.
DevSecOps teams must assess the liabilities of using data, especially when building LLMs to train AI tools. That demands careful data management within models to prevent the accidental transmission of sensitive data to third parties like OpenAI.
Organizations should adopt strict policies outlining the approved usage of AI-generated code, and when incorporating third-party platforms for AI, conduct a thorough due diligence assessment ensuring that their data will not be used for AI/ML model training and fine-tuning
AI Security Automation Will Help Organizations Transition From ‘Shift-Left’ to ‘Shift-Down’
The industry adopted the shift-left concept a decade ago to address security flaws early in the software development lifecycle and to enhance developer workflows. Defenders of systems have long been at a disadvantage. AI has the potential to level the playing field. As DevSecOps teams navigate the intricacies of data governance, they must also assess the impact of the evolving shift-left paradigm on their organizations’ security postures.
Companies will begin moving beyond shift-left to embrace AI to fully automate security processes and remove them from the developer’s workflow. This is called “shifting-down,” because it pushes security into automated and lower-level functions in the tech stack instead of burdening developers with complicated and often difficult decisions.
GitLab’s Global DevSecOps Report: The State of AI in Software Development found that developers only spend 25% of their time on code generation. AI can elevate their output by optimizing the remaining 75% of their workload. That’s one way to leverage AI’s capacity to solve specific technical issues and improve the efficiency and productivity of the entire software development life cycle.
Expect to look back on 2024 as the year that the escalating threats OSS ecosystems adversely affecting global software supply chains, catalyzed substantial changes in cybersecurity strategies, including a heightened dependence on AI to safeguard digital infrastructures. The cybersecurity landscape is already transforming, with a growing focus on mitigating supply chain vulnerabilities, enforcing data governance and incorporating AI into security measures. This transformation promises to steer DevSecOps teams toward software development processes with efficiency and security at the forefront.
Linux dodged a bullet. If the XZ exploit had gone undiscovered for only a few more weeks, millions of Linux systems would have been compromised with a backdoor.
Just as with XZ Utils, a few people wanted OpenJS to designate them as new project maintainers despite having little prior involvement. They “implored OpenJS to take action to update one of its popular JavaScript projects to address any critical vulnerabilities, yet cited no specifics,” wrote Robin Bender Ginn, OpenJS executive director, and Omkhar Arasaratnam, Open SSF general manager.
Well, that was never going to happen.
Jia Tan, the mysterious hacker who became a top XZ programmer and maintainer, before inserting a backdoor in the code, had spent years establishing his project credibility before making his move.
Still, warned Ginn and Arasaratnam, we’ll see more of these kinds of social engineering attacks. In such instances, the friendly attacker seeks to exploit “the sense of duty that maintainers have with their project and community in order to manipulate them.”
Specifically, these security leaders warn against pull requests containing blobs as artifacts. For example, they point out that the XZ backdoor included a cleverly crafted file in the test suite that wasn’t human-readable.
Dubious source code may also be intentionally obfuscated or difficult to understand. For example, the XZ issue started with a relatively innocuous replacement of safe_fprintf() with fprintf() to see if anyone would notice. No one did.
The question now is: Are such attacks already ongoing? Or, far worse, have they already been made, and they weren’t spotted? Chris Hughes, chief security advisor at Endor Labs, told me, he “suspects that many of these are already underway and may have already been successful but haven’t been exposed or identified yet.”
How? Hughes explained, “Most open source projects are incredibly underfunded and run by a single or small group of maintainers, so utilizing social engineering attacks on them isn’t surprising, and given the pressures maintainers are under, they will likely welcome the help. If done well by the attackers, it may be difficult for the maintainers to determine which involvement is from those interested in collaborating and contributing to projects versus those with malicious intent.”
He’s not wrong.
As Jim Zemlin, the Linux Foundation‘s executive director, said at the Open Source Summit North America in Seattle, we’ve been making progress with security. But “trust is something we need to talk more about,” he added. We need to answer the question: “Who writes all this software? This is different from a security question. Security is ‘Is there a vulnerability?’ Who is more of a trust question? Do we trust this identity? Enough to give them a maintainer role?”
Bad news, folks. Zemlin admitted we don’t have a good solution to this problem.
Achieving full observability across enterprise applications is challenging. While third-party services like New Relic or Datadog make end-to-end observability easier, as your application grows in complexity, so does the telemetry data, leading to increased costs. Migrating to an open source stack gives you control over telemetry data and reduces observability costs, despite challenges with existing service provider commitments.
Migrating to open source solutions is straightforward with simple architecture and tech stacks. However, transitioning the entire observability stack to open source demands meticulous planning, tools/frameworks assessments, testing, risk analysis, and cross-team communication, especially given the number of microservices, heavy cloud reliance, and diverse languages and frameworks to take into account.
A cloud observability platform usually absorbs 20%-30% of overall infrastructure spending, but in some cases, it can reach 50%-60%. If your observability expenses exceed 50%, then transitioning to open source solutions would be a sensible option to mitigate tech infrastructure costs.
For the past two years, I have been working on migrating the observability platform from a proprietary solution provider to an open source stack. Let’s dive into the key steps necessary for such a migration.
Finalize the key telemetry data and systems
Managed observability platforms offer comprehensive insights into system health, necessitating a significant volume of high-quality telemetry data. By default, the agents tasked with capturing and transmitting telemetry data are configured to gather as much information as possible, facilitating the creation of extensive dashboards and reports. However, this extensive data collection contributes to the overall cost.
Select which systems you actually need to monitor. A typical enterprise application includes databases, caching databases, container orchestrators, and numerous cloud services. Engineers often monitor more services than necessary, which leads to unnecessary complexities and costs. It’s often sufficient to streamline the monitoring scope to essential systems.
In my experience, the quantity of telemetry data required is significantly lower than what third-party services typically retrieve. You can easily divide those data collections into two buckets: must-have and nice-to-have. The “must-have” data aligns with the well-established four golden signals:
Latency: the amount of time it takes to serve a request
Traffic: the volume of requests that a system is currently handling
The error rate: the number of requests that fail or return unexpected responses
Resource saturation: the percentage of available resources being consumed
These fundamental signals serve as the foundation for key dashboards and alerts. Observability platforms built on these signals cover 80% of typical use cases.
Understanding each signal’s metadata requirements is crucial. The extent of metadata captured directly impacts the observability platform’s complexity and cost. For example, when monitoring service-to-service latency, consider the necessity for service IP addresses, instance IDs, or header information.
Select the relevant stack
Managed observability platforms such as New Relic and Datadog provide comprehensive monitoring for various enterprise components. However, when you migrate to an open source stack, you must assess and integrate the tool stacks to meet diverse monitoring requirements.
One important aspect to consider is scaling: how to manage the huge amount of data generated every minute across all those systems. Focus on two fronts: selecting the stack for storing and processing telemetry data (logs, metrics and traces) and devising methods for capturing and pushing telemetry data from diverse systems.
Telemetry signal stack selections
Stack for logs
You require a stack that can efficiently and economically process and store the substantial volume of logs generated by the system. In the past, I used the ELK stack for storing and searching logs, but it is a generic solution and not tailored specifically for logs.
I recommend Grafana’s Loki for its effective management of large log volumes and LogQL, a language akin to PromQL. If you are familiar with PromQL, navigating through extensive log data with LogQL is easy and intuitive.
Stack for Metrics
Prometheus is a popular database for storing time-series metrics data. However, it has scaling-related limitations; it cannot scale horizontally. Alternatives like Thanos, Grafana’s Mimir, and VictoriaMetrics offer better out-of-the-box solutions for horizontal scaling.
I conducted some research and ultimately chose Grafana’s Mimir to store metrics in a similar migration project. This decision was based on Mimir’s remote storage capability, as well as its scalable and highly available architecture.
Stack of Tracing
Distributed tracing is a must in a microservices architecture, for easily identifying bottlenecks in latency and for resolving performance-related bugs. Grafana’s Tempo could be a good option for several reasons:
It can seamlessly integrate as a backend for Grafana’s dashboard. Consolidating dashboards in Grafana helps avoid navigating between multiple applications. Switching platforms just for tracing while using Grafana for other aspects of observability would not be practical.
Tempo provides a similar experience to TraceQL when interacting with the database, streamlining the learning process and significantly reducing the learning curve.
Tempo is compatible with popular open source tracing protocols such as Zipkin and Jaeger. If any team already uses these protocols, transitioning to Grafana’s Tempo would be smoother.
Capturing and pushing telemetry data
You have to install an agent on the system to capture and push the telemetry data. Those steps vary based on the system being monitored. Typical enterprise application setups involve monitoring cloud integrations, process monitoring, infrastructure hosts, Kubernetes clusters and application monitoring. Among the possibilities:
Cloud integrations: This includes monitoring for cloud services such as SQS, SNS, EMR and EC2.
Process monitoring: This involves monitoring processes running on bare-metal machines. With the advent of dockerization and Kubernetes, there is now a standardized way of starting the application. In the past, there was no fixed mechanism. For example, to run a Java application, you could use a Java application, the ‘java -jar’ command, Tomcat, or OS systemctl. In the case of node, it could be npm or PM2. Each team or service may have had its own way of starting the process.
Infrastructure host: This entails monitoring the machine itself for metrics such as CPU usage, memory, disk I/O and Network IO or checking if the machine is offline.
Kubernetes monitoring: You regularly need to monitor a Kubernetes cluster, such as instances where Kubernetes is unable to schedule a pod due to insufficient resources.
Application monitoring: This monitoring focuses on overseeing the services the team created. Each service may differ in its development approach and choice of tech stack; but from an observability perspective, they usually are treated the same.
Browser and mobile monitoring: These metrics ensure optimal performance and user experience across different platforms. Browser monitoring includes tracking page load times, rendering performance, JavaScript errors and resource usage. For mobile, it includes monitoring app crashes, latency, battery usage, network requests and device-specific metrics.
Before the advent of OpenTelemetry, there was no standardized method for monitoring applications. OpenTelemetry emerged from the merger of two earlier projects, OpenTracing and OpenCensus. It stands as a vendor- and tool-agnostic framework to instrument applications or systems irrespective of language, infrastructure or runtime environment. OpenTelemetry represents a significant community effort, and its popularity and stability are steadily growing.
Validate observability stack on application architecture
Applications vary in architecture and development stages. To address this diversity, you need multiple proofs-of-concept (POCs) across different systems, such as infrastructure, cloud service and backend and frontend applications.
Migrating an observability platform from one tech stack to another is simple when the application architecture is a monolith. However, when the underlying architecture is microservices, it becomes difficult.
While the microservices architecture offers flexibility in choosing different tech stacks for building services, devising a standard solution for capturing telemetry data becomes challenging due to the diverse range of tech stacks involved.
Conducting POCs for various tech combinations is necessary to build a self-service tool that individual teams can follow to migrate their services easily. For instance, if your service uses Java 11, Spring Boot 3.x.x and PostgreSQL, these POCs can provide standardized steps to enable application monitoring.
Migrate core components
Setting up the observability platform involves configuring the stack for metrics, logs and traces, as well as installing the necessary agents. One advantage of migrating the observability platform is that there’s no need to transfer old telemetry data. During the testing phase, you will have an overlapping period to accumulate enough useful data to create specific dashboards and alerts.
Migrating alerts and dashboards to the new system is essential, even when data migration is not necessary. This is because changes in the underlying techniques for capturing metrics may modify the names of the metrics and subsequently change all query expressions used in alerts and dashboards. Some metrics may not be available in certain scenarios, especially derived metrics that can be recreated from underlying metrics.
While migrating query expressions manually is an option, a manual process is error-prone and time-consuming for a large number of alerts. I had experienced a similar situation with alert numbers ranging from 100 to 400.
To streamline the migration, we developed a Node.js script that programmatically converted New Relic’s query expressions to Prometheus expressions. The script performed the following high-level steps:
Connected to the New Relic API server and fetched all the alerts configured for cloud integration
Converted New Relic alert expressions into PromQL alert expressions
Wrote all the Prometheus alert expressions into the YAML file
Implementing this technique allowed us to successfully migrate all alerts and dashboards in four or five days, a process that would normally demand a month’s manual labor. Additionally, scripting significantly minimized the risk of human error.
Test the migration
Verify each alert and dashboard. Run both old and new platforms during testing. Parallel operation permits thorough testing of the metrics.
However, running both systems concurrently may lead to a performance hit due to services sending telemetry data to two locations. Additionally, there may be a slight increase in costs until the complete transition.
Running both systems in parallel provides an ideal setup to ensure the migrated dashboards and alerts are functioning correctly.
Testing alerts that trigger during this process is straightforward, but evaluating those that have not yet triggered poses a challenge. In one project, I reduced the threshold value of each alert and tested its functionalities. Approximately 90% of the alerts generated through custom scripts worked seamlessly, with only 10% requiring some manual tweaking.
Migrate other related components
Other systems depend on the data generated by the observability platform. For example:
Notification systems like email, Slack channel, etc., that alert in case of an incident
Incident management tools like PagerDuty provide a streamlined way to handle incidents.
These systems rely on the alert’s payload to work properly. Their assets should be updated, as a change in the observability platform leads to a change in the alert payload. Update the templates of notification systems to ensure they seamlessly integrate with the new alert payload.
Similarly, incident management tools like PagerDuty require modifications to routing rules, escalation policies, and scheduling. Fortunately, readymade open-source migration tools are available to facilitate easy migration from PagerDuty to Grafana’s OnCall. However, in other solutions, you may not have access to out-of-the-box migration tools. In such cases, manual migration or script writing may be required.
Timeframe for the Migration
Planning a migration involves finalizing key telemetry data and systems, selecting relevant stacks. and validating the observability stack on the application architecture. It typically takes around three weeks.
The project duration depends on factors such as the number of teams involved and the quantity of active services. For instance, a migration to open source with 100 microservices and involvement from 10 different teams may take four months to complete.
Migration Complete!
Transitioning your observability platform to an open-source stack offers a promising avenue for cost reduction and increased control over telemetry data. However, accomplishing this migration demands meticulous planning and execution, encompassing essential steps like feature prioritization, stack selection, POCs, core component migration, testing, and the migration of related systems. Despite the challenges posed by diverse architectures and technologies, a systematic approach, comprehensive assessment, and collaboration among teams can facilitate a smooth migration process and guarantee its success.
Long-time Java developers may recall Codehaus, which combined a Java open source project hosting service and community. Now, inspired by its legacy, a new group of developers has established the Commonhaus Foundation.
At its core, Commonhaus embodies a vision of providing a neutral harbor for established open source libraries and frameworks seeking a home. The foundation’s mission is to ensure a nurturing environment where projects can flourish unabated, transcending generations of technology evolution.
What sets Commonhaus apart is its emphasis on offering guidance and support instead of imposing mandates. Instead of setting rules for how an organization must be set up and run, it empowers projects to navigate in their own way.
Its key principles include succession planning, which addresses the always tricky question of the “bus factor.” That is, it addresses how an open source project can keep going if one or more of its leaders die in a bus accident. The foundation addresses this by providing an intricate web of leadership transitions and contributor changes with grace, thereby safeguarding their influence and functionality over time.
Erin Schnabel, a founding member and chairperson of the Commonhaus Foundation, articulated the rationale behind this initiative. “The open source ecosystem has undergone profound transformations in recent years,” Schnabel remarked. “While existing foundations have laid the groundwork for numerous successful projects, the advent of fiscal hosting platforms and self-governance models has unveiled a clear need for a fresh approach.”
“Traditional governance structures may not align with the unique needs of projects that have cultivated their own practices and infrastructure,” Schnabel continued. “These projects often face hurdles when considering merging into existing foundations, leaving crucial needs unaddressed. Commonhaus seeks to bridge this gap, ensuring that projects of all stripes receive the support they need to thrive.”
Trisha Gee, a distinguished software engineer and Java Champion, echoed Schnabel’s sentiments. “Commonhaus fills a crucial void in the support ecosystem for open source projects,” Gee asserted. “Its focus on community and continuity is paramount for ensuring sustainability and mitigating maintainer burnout. Successful projects shouldn’t have to compromise their autonomy to access foundation support.”
The roster of projects joining Commonhaus at its inception reads like a Who’s Who of the open source world, including Hibernate, Jackson, OpenRewrite, JBang, JReleaser and Morphia. What these projects have in common is a desire for a stable, nurturing home without having to shift existing governance structures.
Gavin King, Red Hat distinguished engineer and parent of Hibernate, offered another reason to join Commonhaus. “We’ve never really overcame the perception that this is a project dominated by one company,” King said. “That’s something we want to finally change. Red Hat’s leadership and sponsorship have been a source of stability, continuity, and user confidence, and we don’t wish to undermine that. But what we would like to do now is open up space for other groups and companies also to feel included in the project.”
The Foundation extends an open invitation to developers, contributors, projects and sponsors to join its cause. Through sponsorship on platforms like GitHub, individuals, and organizations can not only provide financial support but also shape the strategic direction of open source development.
As the Commonhaus Foundation sets sail on its maiden voyage, it aims to chart a course toward a future in which open source projects can continue to thrive unfettered by the specter of obsolescence.
Hashicorp did more than claim OpenTofu had misappropriated some of its code. Its law firm, Wilson Sonsini, issued a cease-and-desist order against OpenTofu. This read in part, “This is a cease and desist demand to the supporters of the OpenTofu project. Specifically, OpenTofu has repeatedly taken code HashiCorp provided only under the Business Software License (BSL) and used it in a manner that violates those license terms and HashiCorp’s intellectual property rights. In at least some instances, OpenTofu has incorrectly re-labeled HashiCorp’s code to make it appear as if it was made available by HashiCorp originally under a different license.”
The order continued, “Therefore, we demand that OpenTofu … cease and desist from further violations of HashiCorp’s BSL license and infringement of HashiCorp’s copyrights. If OpenTofu does not comply, we reserve all rights, including the right to send DMCA takedown notices to Github or any other third-party hosting or source code repository provider, and the right to initiate litigation to stop further violations.”
OpenTofu replied, “The OpenTofu team vehemently disagrees with any suggestion that it misappropriated, mis-sourced, or otherwise misused HashiCorp’s BSL code. All such statements have zero basis in facts.” In addition, it said, HashiCorp’s claims of copyright infringement are completely unsubstantiated.
Where’d the Code Come From?
As for the code in question, OpenTofu claims it can clearly be shown to have been copied from older code under the Mozilla Public License (MPL) 2.0. “HashiCorp seems to have copied the same code itself when they implemented their version of this feature. All of this is easily visible in our detailed SCO analysis, as well as their own comments.”
In a detailed source code origination (SCO) examination of the problematic source code, OpenTofu stated that HashiCorp was mistaken. “We believe that this is just a case of a misunderstanding where the code came from.” OpenTofu maintains the code was originally licensed under the MPL, not the BSL. If so, then OpenTofu was perfectly within its right to use the code in its codebase.
OpenTofu’s attorney said, “Both the OpenTofu files to which you refer and HashiCorp’s Terraform files to which you compare them are both derived (at least to some degree) from the pre-fork MPL-2.0 files – code that was made publicly available under the MPL.” Therefore, “To my client’s knowledge, none of the Terraform code subject to the BUSL has been improperly copied, incorrectly sourced, or used for any purpose.”
The attorney went on to say that HashiCorp should have known this. “[W]hile you complain that a comparison of HashiCorp Terraform code to OpenTofu files show ‘substantial similarity’ between the two, the diff files you attach to your Letter actually show the compared files to be quite dissimilar.”
The OpenTofu lawyer continued, “Going forward, we would be open to establishing developer liaison contacts between OpenTofu and Terraform, who would be available to review and address any intellectual property concerns with submitted contributions to OpenTofu or Terraform.”
Laferra concluded, “In the future, if you should have any concerns or questions about how source code in OpenTofu is developed, we would ask that you contact us first. Immediately issuing DMCA takedown notices and igniting salacious negative press articles is not the most helpful path to resolving concerns like this.”
So, what’s next? Having covered more than my fair share of intellectual property legal conflicts, I don’t expect HashiCorp to back down. I fear that this issue will be resolved in the courts, not in Git repositories.
The Eclipse Foundation and other open source organizations are working towards implementing the European Union’s Cyber Resilience Act’s software development security requirements.
The Eclipse Foundation is spearheading the effort to create a unified framework for secure software development. The foundations and allies are doing this via a new working group, established under the Eclipse Foundation Specification Process.
The collaboration is spurred by more than regulatory compliance. In an era where open source software is pivotal to modern society, the imperative for safety, reliability and security in software has never been more critical. As Arpit Joshipura, the Linux Foundation‘s senior VP of networking, said at the Open Source Summit Europe in Bilbao, Spain, last year, “We must look at the end goal. The end goal for all of us is the same. We want to secure software, and we want to secure open source software.”
This process won’t be easy. It will be a highly technical standardization journey. The project will start with the current security policies and procedures of the open source foundations involved. The final result will be process specifications, which will be made freely available under a liberal specification copyright license and a royalty-free patent license.
Under the CRA, the legal person—which is not the same as a flesh-and-blood person reading this story—responsible for these new policies and their implementation will be known as the “Open Source Software Steward.” This will be a heck of a job.
Leaving aside the code and technical issues, the open source groups must work with traditional standards organizations. Historically, neither kind of group has worked well with one other. To make matters worse, standard organization governance models don’t even have a way of dealing with open source groups. Indeed, for the fruits of this effort to make it into regulation, it will require going through the formal standardization processes of at least one of the European Standards Organizations.
This is going to be so much fun!
Adding insult to injury, setting technology standards usually takes years, and the CRA requires developers to have something in place by 2027. Good luck with that!
“There is an enormous amount of work that will need to be done over the next three years to implement the CRA,” Eclipse Foundation executive director Mike Milinkovich said. “It’s the first law anywhere regulating the software industry as a whole. The implications of this go far beyond the open source community and will impact startups and small enterprises as well as global industry players.”
Despite the challenges, the initiative represents a crucial step forward. The working group is optimistic about laying the groundwork for cybersecurity standards that can serve both the open source and proprietary software realms.
So, what is this code in question? Neither Hashicorp nor Terraform is saying, quite yet. However, it appears to be a “removed block” automation process that enables Terraform to manage resource deletion.
Matt Asay, MongoDB‘s VP of Developer Relations, revealed this accusation. He accused OpenTofu of lifting the “removed block” code, which was released under the BSL a few months after the OpenTofu fork was created. “The tell? OpenTofu took this BUSL-licensed HashiCorp code, removed the headers, and tried to instead relicense it under the Mozilla Public License (MPL 2.0),” Asay wrote.
Oh, it’s on now.
Even before news of the legally threatening message emerged, prominent open source leaders attacked Asay’s claim. Bryan Cantrill, the hyper-cloud Oxide Computer co-founder and CTO, claimed, “This is an extraordinarily serious accusation – and the cited files frankly don’t substantiate it.”
Cantrill is far from alone in thinking that there is insufficient evidence to justify the accusation that OpenTofu has stolen any code. Until we have harder evidence, the issue will remain a case of “He said, she said.”
Dan Lorenc, CEO and co-founder of Chainguard, a software supply chain security company, warns this could be the start of something big and ugly in software development circles. “I don’t have any philosophical or ethical issues with source-available licenses, but their proponents are going to have to address these cease-and-desist threats,” he wrote. “Developers working in OSS (OSI/FSF approved licenses) are going to be prevented from engaging with or even looking at source-available projects completely to prevent the risk of lawsuits. It’s just too risky to pretend otherwise.”
Lorenc adds, “There’s no gradual transition here, it’s going to be a hard fork in communities if these lawsuit threats keep continuing. And I know which community I’d bet on sticking around.”
This week at the KubeCon + CloudNativeCon Europe conference, the maintainers of the open source OpenTelemetry observability agent software project being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) revealed they have added profiling capabilities to enable DevOps teams to identify the root cause of issues down to a specific line of code.
Austin Parker, director of open source for Honeycomb and a maintainer of OpenTelemetry, said while several providers of IT observability platforms have in the past year added code profiling capabilities, it makes more sense to provide this as a capability surfaced by OpenTelemetry in a way that any IT platform can easily consume.
The overall goal is to reduce the amount of time it takes to troubleshoot IT issues by identifying not just the code at issue but also the development team responsible for it, noted Parker.
OpenTelemetry defines a format that makes it simpler to centrally manage the collection metrics, logs and traces using open source software. In general, OpenTelemetry continues to gain traction as DevOps teams look to rationalize the amount of commercial agent software they need to deploy, update and manage.
In effect, OpenTelemetry is aggregating agent software that, in terms of observability, provides no differentiated value, said Parker. That’s critical because as application environments become more complex, it’s all but impossible to manage them without being able to broadly apply the agent software required to instrument them.
Ultimately, as OpenTelemetry becomes more widely employed, it makes it significantly easier for developers to understand how their application is performing, noted Parker. Over time, armed with those insights, developers will build and deliver higher-quality applications, he added.
It’s still early days as far as adoption of OpenTelemetry is concerned, but it’s clearly well on its way to becoming a de facto standard for enabling observability. The number of developers who have access to not just metrics and logs but also traces should steadily increase as the cost of instrumenting code declines.
Of course, finding a way to collect that data inexpensively will also be crucial for applying artificial intelligence (AI) to DevOps workflows. After all, before an AI model can be trained, it needs access to verified data.
In the meantime, DevOps teams might want to consider at what rate they can replace existing commercial agents deployed across the enterprise. Much of that agent software is more mature than OpenTelemetry, but over time, the economic arguments for using open source agent software become compelling. Many of the providers of observability platforms have already made support for OpenTelemerty available by default.
Regardless of approach, the need for observability has always been apparent. The issue is that there is a world of difference between monitoring a pre-defined set of metrics and being able to launch queries to identify the root cause of an issue that metrics alone are not going to be able to surface.
eBPF is a revolutionary technology originating from the Linux kernel. It is used to safely and efficiently extend the kernel’s capabilities without requiring changing the kernel source code or loading kernel modules/extensions.
Today, eBPF is used extensively to:
Provide high-performance networking and security in modern data centers and cloud-native environments
Extract fine-grained security observability data at low overhead
Provide insights for performance troubleshooting
Provide preventive application and container runtime security enforcement
eBPF-Enriched Context
eBPF is the base data layer that is needed in runtime cloud security. eBPF gives you information at a high level on Layers 3, 4 and 7, allowing you to see all network communication and correlate it with processes and specific applications that are communicating on those ports.
To fully leverage eBPF, you need to pair this raw data with additional cloud infrastructure context to gain a full understanding of which resources and services are in use.
For example, by leveraging eBPF, you can determine if you have 10 different applications all communicating on port 443, whereas a traditional network monitoring tool would tell you that you have a certain amount of traffic on this port but wouldn’t be able to segment this traffic and show how much of that traffic is coming from each individual application.
Knowing which communication is coming from an individual application increases visibility. It allows you to see what your applications are doing in real-time and identify signs of compromise or decreased performance.
Increasing Observability With eBPF
eBPF provides users with intrinsic, comprehensive insight into communication flows across the entire system. The applications for enhancing security observability with eBPF are vast, ranging from enabling highly detailed process visibility to facilitating end-to-end observation of processes and flows.
When dealing with Layer 4 to Layer 7 traffic involving multiple protocols, eBPF emerges as an optimal solution. Through eBPF, TLS traffic traversing pods, nodes and cloud platforms can now be uniformly instrumented across layers.
In other words, with eBPF, you can observe your application and your node and gain a vast amount of raw data, which, with the right tools, you can correlate to understand your entire network topology.
Beyond eBPF Raw Data to Context and Correlation: Layer 7 and Beyond
Specifically for observability, you can use eBPF to apply filters directly in the kernel, which reduces the overhead and allows you to ingest significantly more raw data. eBPF’s architecture allows you to do all of this, observe network traffic and Layers 3, 4 and 7 without compromising your security or performance.
With eBPF, you are able to understand the execution events, system call events and all of the network ingress and egress. But it’s not always enough.
You will need to correlate it with Cloud APIs, Kubernetes identities, pods and namespaces to understand the actual identities and relationships between the individual workloads, all the way down to the application layer. By doing so, you are able to understand application-level identity and use this context to understand your application behavior as well as your most critical risks and threats.
Leveraging eBPF for Threat Detection and Response
Security and networking observability are inherently interconnected. eBPF helps you to gain insights such as abrupt shifts in behaviors, spikes in activity within specific processes indicating potential exploitation or attacks, service-to-service communication trends and more. These projects enable users to examine network transactions, identifying the pods, processes and system calls involved.
With eBPF, you can see OS-level runtime threats. eBPF gives you the ability to listen to system calls but also to go one step further and block or terminate any process, PID (process identifier), system call or network packet. This allows for a more granular approach to automated threat detection and response without damaging your system or hurting your infrastructure.
You can also use eBPF to implement prevention rules, meaning you can instruct eBPF to automatically block any malicious processes or pre-designated system calls or network packets, going beyond traditional uses of eBPF for monitoring and security, to become a more proactive, preventative tool for securing your infrastructure and applications.
Real-World Benefits of eBPF
eBPF presents a number of real-world benefits, going beyond traditional monitoring, observability and security to help with overall alert noise reduction and improved forensic analysis.
How eBPF Contributes to Noise Reduction
By itself, eBPF does not reduce noise. In fact, it increases because of the influx of raw data from every system call. However, with the right tools, you can leverage this raw data and correlate it with additional context to understand your infrastructure and application behavior. This effectively reduces the alert noise that you would otherwise receive because you can view more detailed insights into resource behavior.
By filtering and aggregating this data, you receive more insights about applications, overall giving you less noise because you’re focusing on what actually matters.
By having this eBPF raw data at runtime, you can apply intelligent filters to filter through the most important data and use it accordingly, resulting in better prioritization, more focused efforts and increased operational efficiency.
Forensic Analysis
Another important real-world use case for eBPF is conducting forensic analysis. By leveraging eBPF’s raw runtime data and applying intelligent filters, you can quickly pinpoint problem origins and conduct forensic analysis at the application level.
By leveraging eBPF’s raw data from Layer 7, you gain a granular understanding of applications and the process-level communication within an application, allowing you to foresee the potential blast radius if an application were compromised in terms of who it could interact with and what damage could occur.
Conclusion
In summary, eBPF is a powerful technology that can be effectively leveraged for monitoring, observability and security purposes. However, when eBPF’s raw data is combined with context from cloud APIs, it can be leveraged even further with intelligent filters to provide unparalleled cloud infrastructure and application context, allowing you to understand application-layer identity, provide real-time security, pinpoint problem origins and stop threats with surgical, active response.
