devopsy.kinsta.cloud

Tag: open source

Lineaje Allies With Persistent Systems on Open Source Software Remediation Service

Lineaje this week revealed it has partnered with Persistent Systems to provide a managed service for remediating open source vulnerabilities that leverages artificial intelligence (AI) technologies.

Persistent is extending an existing open source software maintenance service by incorporating a Lineaje platform that makes use of machine learning algorithms to analyze software bill of materials (SBOMs) to provide recommendations for fixing code and surface remediations such as updates that should be applied.

Lineage CEO Javed Hasan said Persistent Systems is augmenting that capability with a dedicated team of specialists that IT organizations can contract to maintain open source code along with previous investments the managed service provider has made in large language models (LLMs) that add generative AI capabilities.

Options for the services range from simply identifying issues that an internal IT team should address to a white glove offering through which Persistent Systems ensures all open source components are continuously updated.

In the wake of an executive order issued by the Biden administration that requires federal agencies to better secure their software supply chains, more attention is being focused on the inherent cybersecurity risks associated with employing open source software to build applications. Many open source projects are led by a handful of maintainers that often lack the time and skills required to ensure the components they employ have not, for example, been compromised by malware that might find its way into downstream applications.

Those maintainers are not being paid, so cybersecurity patches to open source software might not be readily made available, and even when they are, many organizations lack the skills and resources required to apply them in a timely manner. The majority of known vulnerabilities within application environments are still unpatched. Many IT teams are also reluctant to upgrade components for fear of breaking applications that have dependencies that are often difficult to track.

It’s not known how much of the open source software that many developers routinely incorporate into their applications might contain vulnerabilities that could be exploited, but the level of technical debt that has accrued over the last several decades is considerable. While only a small percentage of those vulnerabilities have been exploited, cybercriminals are becoming more adept at discovering them.

Hopefully, more contributions to open source projects will improve the overall state of application security, but in the meantime, it’s clear organizations will need to rely more on AI to drive DevSecOps workflows as ever-increasing amount of open source software continues to be relied on to build applications, said Hasan.

The challenge, of course, is identifying which vulnerabilities, if exploited, will have the biggest impact on the organization. After all, at this point, it’s not possible to defend every application so the focus needs to be on prioritizing remediation efforts based on their actual severity rather than a general rating.

In the meantime, DevSecOps teams would be well-advised to have incident response plans in place for remediating breaches involving both known and zero-day vulnerabilities that, in addition to being all but inevitable, can now be exploited within minutes of being discovered.

November 22, 2023
Canonical Automates Provisioning of Cloud Environments Anywhere

Canonical today made generally available an open source framework, dubbed MicroCloud, that leverages Linux containers and the Snap software packaging and deployment tools the company developed to automatically orchestrate the provisioning of cloud computing environments using a single command.

MicroCloud automatically creates a functional cloud computing environment in minutes that can be deployed anywhere, including edge computing environments, using policies defined by an internal IT team to ensure the appropriate guardrails are in place. All components can also be updated over the air in a way that preserves data, and if any errors occur, rollbacks are automated.

Miona Aleksic, product manager of MicroCloud, said the platform enables organizations to democratize DevOps workflows in a way that becomes accessible to the average IT administrator.

In effect, Canonical is making cloud computing more accessible in a way that is reminiscent of its creation of the Ubuntu distribution of Linux that is easier to deploy and install, she added.

As IT environments become more distributed, programmatically spinning up infrastructure resources has become more challenging. MicroCloud enables IT teams to provision resources more easily at the point where data is being created, analyzed and consumed, noted Aleksic.

It should also improve cybersecurity because many of the cybersecurity issues that organizations encounter today can be traced back to infrastructure misconfigurations, she added.

It’s not clear how automated the provisioning of IT environments is about to become, but the demand for applications today far exceeds the ability of many DevOps teams to keep pace. Much of the toil that leads to burnout among DevOps teams stems from the manual tasks that persist within workflows. In theory, the more low-level tasks—that don’t add much in the way of differentiated value to the business—can be automated, the more time there should be to address more complex issues.

That’s critical at a time when many organizations still struggle to find and retain DevOps engineers. In addition, the simpler it becomes to automate the provisioning of infrastructure, the more accessible DevOps best practices become for small-to-medium sized businesses (SMBs) that historically couldn’t afford to hire software engineers.

One way or another, DevOps workflows will need to become more automated. There is naturally a lot of focus on artificial intelligence (AI), but while that approach holds a lot of promise, there are clearly advances to be made using a range of existing automation frameworks. The challenge organizations have encountered is that writing the playbooks required to automate processes requires a level of expertise they generally lack.

Platforms such as MicroCloud present IT teams with an alternative approach that should reduce the cognitive load developers encounter whenever they want to provision IT infrastructure. Arguably, the one reason developers provision infrastructure themselves is that previously it simply took too long for IT teams to do it on their behalf.

It may be too early to determine whether or not tools such as Terraform are required to provision infrastructure, but it’s clear that most developers would rather spend their time writing code instead of managing infrastructure.

November 14, 2023
How Open Source Software Powers High-Impact Climate Science

According to the NOAA National Centers for Environmental Information (NCEI), the past nine years have been the nine warmest since record-keeping began in 1880, and 2023 is on track to set another record. Making the hard decisions involved in addressing this global crisis requires analyzing and understanding enormous quantities of data using sophisticated computational methods. Luckily, innovations from the open source software community are helping revolutionize climate science, letting researchers quickly derive insights from data.

Delivering Critical Collaboration

Modern science relies heavily on complex computational and numerical analysis techniques. However, the massive amount of measurement and simulation data available in fields like climate science makes traditional approaches impractical. With petabyte-scale data, researchers can no longer expect to download or obtain physical copies of entire datasets, nor can their local workstations or laptops handle datasets of that size once they arrive. Moving the scientists to the data is also impractical since researchers all over the world need to make use of the same data.

Instead, scientists and engineers from Pangeo.io and other global collaborations have developed a robust set of open-source tools for working efficiently with large, remote datasets, bypassing restrictions from proprietary software that’s limited to small datasets installed locally. Instead of monolithic commercial applications or inflexible Fortran codebases, modern researchers use flexible and lightweight approaches for distributing computation across cloud computing resources shared between many collaborators. These innovations are critical for climate science since further delay could lead to catastrophic challenges in combating global warming.

Deriving Faster, Flexible Data for Greater Output at Scale

Python’s scientific ecosystem, in particular, has gained prominence and is top of mind when considering open-source platforms best suited to execute climate solutions. Incoming graduate students more often know Python than any other programming language now that Python has taken over as the world’s most popular programming language. Python’s open-source ecosystem includes a wide variety of packages, each focusing on a specific area of functionality. These components evolve quickly and regularly introduce new features while maintaining compatibility through standard protocols. Unlike a monolithic, centrally designed approach, Python’s open ecosystem provides more flexibility when scaling data and solutions.

In high-impact fields like climate science, the goal is to enable scientists and researchers to concentrate on their domain-specific questions rather than becoming entangled in the intricacies of data manipulation and resource management. The numerical Python stack of open-source tools, combined with the ability to access available data in the cloud on a massive scale, facilitates this division of labor. Scientists can focus on writing software for the analysis parts specific to their expertise, meaning faster progress toward climate solutions and products.

With modern cloud-computing approaches, it makes more sense to move computation to the data rather than vice versa (since code is much smaller and more portable than data). With a platform like Jupyter running on a remote cluster and remote Python kernels as the execution environment, scientists can leverage the same infrastructure that hosts massive datasets in cloud object stores. Improvements in software technology and data formats also make it simple to access specific chunks of very large files, such as the temperature records associated with one particular city out of the entire world’s data. Accessing data becomes fast, cost-effective and scalable, bypassing traditional workflows involving repeatedly downloading large datasets consisting primarily of data not needed for a particular analysis.

For climate science specifically, this flexible remote-computing approach also means easier collaboration between developers and scientists to create innovations that could move climate studies and solutions forward. Distributed computing technologies such as Dask, Ray and Spark enable parallel and out-of-core computing, allowing for processing larger datasets at higher speeds. Several remote Jupyter-based services now exist, enabling even the main Python process (the “kernel”) to run remotely while utilizing the browser as an interactive development platform. Because all these tools are open, researchers can apply emerging technologies such as machine learning (ML) and artificial intelligence (AI) techniques as soon as they are developed without waiting for software and hardware manufacturers to catch up.

Creating Data Visualizations and Dashboarding

For humans to understand and validate the data and the steps involved in analysis requires visualization at every stage, which is difficult when the data is too large to transfer locally for display. New open-source tools like Datashader and VegaFusion offer a server-side rendering of even the largest datasets, efficiently rendering distributed datasets into a viewable image and then transferring only that image to a scientist’s local browser for display. Matplotlib, HoloViews, Altair, Plotly and hvPlot in Python, along with Makie.jl for Julia, now allow server-side rendering to be embedded into fully interactive plots in a web browser, letting researchers explore datasets and identify interesting features or detect issues with data quality or analysis steps. The Panel and Dash dashboarding tools in Python allow these plots to be combined into complete web applications for researchers to share their interactive analyses, all without any proprietary tools or restrictions (see this USGS wind turbine application as an example).

In a field with such a dire need for fast innovation, open-source technology is crucial to providing the creativity and speed required for climate scientists to fight global warming. The collaborative and transparent delivery of innovation is what the scientific community is increasingly counting on.

Dr. Martin Durant, staff software engineer at Anaconda, contributed to this article.

November 9, 2023
Generative AI: Some Parts of the Future Are Cool

I’ve been around long enough that I normally roll my eyes when ‘the new shiny thing’ shows up. If I start gushing about something, I truly see it as game-changing in a market space that is full of things that simply are not. Perhaps an example is in order. For years after college, my hobby was compilers and linkers. Until my Master’s program had a class in compilers and the newer variants of Lex and YACC showed me that those skills are archaic. Lots of fun, and I’ve kept my source to a 100% original MS-DOS linker, thinking one day I’ll update it to generate modern executables. But I won’t because the other thing that reduced the appeal of compiler and linker skills is open source. I can download code that, by virtue of being exercised by like a billion people, is inarguably better than my efforts were. Better to put my attention elsewhere. Open source was indeed a game changer. Today, when I see there is a new language du jour, my very first question is, “Okay, but why?” the answer is rarely, “To make developers’ lives easier.”

The same is true with generative AI. Just as open source has spawned a never-ending collection of failed or misdirected OSS projects, AI is starting to create buzz about dozens of “that’s not really useful” use cases. But amongst all those are the “This changes everything” use cases, and for IT–be they Devops or devOps, generative AI for code creation is huge. This will almost immediately become a mandatory job skill, and you should start learning how to get generative AI to spit out the code you need today. It will increase your productivity, help your company, and it will increase your career prospects, helping you.

Generative AI Goes Further

Two examples. In completely different projects that aren’t even for companies in the same space, Lori and I both used generative AI to develop systems faster. I know very little about Lori’s work other than that it was to generate Python that did the grunt work and took care of the repetitive bits she hates. My work was to generate bash scripts for things I don’t generally do in Linux. Any honest assessment agrees that the power of Linux is masked by the insane level of obscure commands. While researching online has helped us get over the “What the heck does that do?” for decades, generative AI goes one step further. It allows you to tell it what you want to do, then give it whatever additional information it might need to tweak the results, and the script is done.

We all use APIs, libraries, SOs and snippets pretty much daily; this is literally just an extension of that usage. Most of us would search online for examples of what we needed to do, then copy and modify them. This really is literally, “Give me an example of that. Good, now customize it to my needs.”

There are risks with any significant change, and generative AI for script development is no different, but the largest risk identified – that entry-level people will not learn and grow – is blown out of proportion. Given an example to read, the curious and those who are going to be responsible for supporting the system will go over it with a fine-toothed comb. The learning will be more reading than doing at that point, but in the end, the script still has to execute. Troubleshooting won’t perform itself, and people will continue to strive to understand — both because they need to when supporting systems, and because curiosity is indeed the hallmark of IT staff.

You all are the backbone that keeps the organization online and employees productive. We are way past the time when 90% of companies would fail without IT. Keep rocking it, but give your org that competitive edge. Take any tool that will increase your productivity and help your career in the process.

November 8, 2023
Something is Amiss in the Open Source World
You don’t have to be a psychic to grok that the winds of change are blowing hard in the open source world. For the last few months, I’ve seen hints and heard whispers about a new direction for what I believe is the most passionate and enthusiastic community in tech. I think we are about to sail off into unchartered territory—and it will be interesting to see what the repercussions are.

Of course, this will all play out against the backdrop of continued layoffs and belt-tightening in the broader tech sector. Before I detail what I think are the telltale signals of change, let me state that I am and have always been a big supporter of open source software. I firmly believe we live in a golden age for open source software, which I call the “Foundation era of open source.” At a time when 90% plus of organizations use open source software, many of the leading open source projects are under the domain of not-for-profit foundations (NFPs) such as the Linux Foundation, the Apache Foundation, etc. This allows multiple commercial companies who might otherwise compete with each other to work in coop-etition. This has, in many ways, enabled this golden age of open source. Prior to the Foundation era of open source, there were two prior dominant models. Initially, there was what I call the “Cathedral and Bazaar era,” named after the great book by Eric Raymond. This was a time of idealistic notions of “pure” open source, unsullied by commercial aspirations. To me, Dr. Richard Stallman embodied that time; if you were to make an analogy to history, it was akin to the 1960s era in the U.S.

From that era, let’s zoom to sometime in the early 2000s when we moved to what I call the “Big Brother era” of open source. In this era, open source projects were owned and managed by one vendor, for all intents and purposes. The project could have a vibrant community, but ultimately, it was the managing vendor or “Big Brother” who reaped most of the rewards. This made it difficult for multiple tool vendors to support the same project and stifled progress.

The dawn of the foundation era of open source changed all that, and we have seen transformational growth as a result. But nothing lasts forever—and I think the king of the hill may be on the way down. Why do I say this? Here are several events/trends I’ve noticed recently that lead me to believe this. Some of these things I was told in confidence, so I cannot reveal names to protect the innocent.
1. The move away from “pure” open source licenses. Over the last few months, several pillars of the open source world have moved away from accepted open source licensing. Red Hat has done so with Red Hat Enterprise Linux (RHEL) and now Hashi has, as well. Now, to some, this change is subtle and it’s not a big deal. Only paying customers get access to source code. The overwhelming majority of users never look at the source code anyway. Nevertheless, this is a huge red flag for the community—the accepted licenses and a core tenant of open source, that source code is available to everyone, are no longer being used.
2. Diminishing corporate financial support for open source communities and foundations. I guess this was to be expected in hard economic times. The path to profits from supporting open source foundations, projects and communities can sometimes be more of a dotted line than direct. And a dotted line doesn’t usually work when the bean counters are involved. In looking to cut expenses, these are low-hanging fruit.
3. Closing foundations. This is a natural result of the economic strife. Foundations may operate as not-for-profits, but without operating funds, foundations can’t operate. Generally, foundations get a lot of mileage from volunteers. However, volunteers can only go so far before hard costs are involved. Also, volunteers are able to volunteer because their corporate employers empower them to do so. When their employers pull back the funding for those activities, the volunteers have to do what they can to earn a living. I have heard of at least one NFP foundation forced to cease or curtail operations due to corporate sponsor funds drying up.
4. Orphaned projects. Again, this is a direct result of number two, above. With the foundations closing and supporting funds drying up, I know of at least two bellwether open source projects that are in danger of becoming orphaned. Both of these projects boast huge community numbers and dominate their markets. The fact is though, that supporting a major project costs hundreds of thousands of dollars. While millions of community members may use a project and support it, they don’t contribute dollars. Dollars that pay for hosting, infrastructure and other hard costs. Coughing up a few hundred grand to support a project is a bridge too far for some companies in the current economic climate.
What does this mean for open source software and you? Well, in the near term, if you are just a consumer of open source software, probably not much will change day-to-day. You may have to cough up some money if you want access to source code or support from the community. Bigger picture, though, is the winds of change are blowing. I think we are going to see more companies taking a Hashi-type approach to protecting their source code. This will result in more forks of open source projects. However, my experience is that most open source forks don’t succeed (with some notable exceptions—like Jenkins forked from Hudson). Companies who are building businesses around open source models will be the most at risk. I think an open core business model may be the best bet for an open source vendor. I think the future of NFPs as the stewards of open source may become a bit cloudy (no pun intended). What will take its place or fill the void? That remains to be seen.

For the record I was never a fan of that old Irish proverb, “May you live in interesting times.” This kind of upheaval is a little too interesting for me. But one thing I have learned in 30 years in tech (and more than that in life) is that change is constant. You can embrace it and thrive or shrivel and die.
October 5, 2023
GraphQL API Visibility Ensures Enterprises Can Escape Lock-In

While GraphQL is still early in its enterprise adoption curve, the runtime query language (and replacement for REST APIs) has already built an impressively robust ecosystem for jumpstarting major enterprise-grade deployments. Among the appeals of GraphQL are its open source implementations and breadth of available server types, which combine to offer enterprises flexibility and the benefits of best-fit solutions.

However, operating GraphQL in production, maintaining GraphQL APIs at enterprise scale and achieving the requisite GraphQL visibility and security require complementary tools and services. This is where enterprises can fall prey to vendor lock-in and lose the flexibility to change and control their own deployments if they aren’t careful.

Enterprises need vendor partners to manage their GraphQL deployments effectively, and vendors are glad to offer comprehensive SaaS solutions. But enterprises must understand that a vendor can seek to capture their long-term business by coercion—straitjacketing customers by hosting their entire GraphQL infrastructure and GraphQL gateway and making it, well, not exactly easy to let go. They can unleash particularly nefarious practices, using proprietary features to force customers to migrate to certain technologies or purchase expensive upgrades to keep their crucial GraphQL deployments afloat. Enterprises must be wary of allowing this vendor lock-in to happen for their own good.

Knowledge is essential to maintaining the balance of power in vendor relationships. Enterprises must understand their GraphQL graph and key measures to correctly assess current vendors and maintain agility and control over deployments by not entrusting their entire fate to any one partner.

These are the specific areas where enterprises should seek out intelligence and reliable metrics in order to choose their own future with GraphQL:

Understand Your Overall Graph (and Subgraph) Utilization

Establishing an accurate overview of your graph utilization—and harnessing actionable insights—are crucial, foundational steps that lay the groundwork for deeper analysis and better decision-making. You need to know what graph and subgraph objects and operations your most important API users heavily engage with and which objects and operations are underused or not used at all. You need visibility into who is using GraphQL APIs and how. Getting to actionable intelligence should also tell you where errors are most likely to occur and what’s causing them, as well as where major performance bottlenecks pop up during heavy load events.

With this overview available, the next steps call for building a depth of knowledge in each of these five areas:

1) Understand areas of heavy graph and subgraph usage

Knowing exactly where heavy usage occurs in your graph puts you in a position to upgrade or migrate your GraphQL API server or gateway on your terms while mitigating outage and error risks. Specifically, you want to know which operations in your graph are executed most and how often, which graph objects and fields are consumed most often, and which subgraphs support those most heavily burdened areas in your supergraph. This knowledge and some careful planning help to put successful GraphQL server or gateway changes within your power and also reduce risk when introducing changes to areas of the graph under heavy consumption.

2) Understand where your graph is underused

GraphQL migration and modernization efforts are simpler—and carry less risk—when you first remove any unused functionality and reduce your scope as much as possible. To do so, you need visibility into your graph’s least executed operations, objects and fields, and subgraphs, as well as any operations with zero recent usage. Then, flag low-use graph areas you can deprecate, replace or just remove to simplify your deployment before making changes.

3) Understand your GraphQL API users and usage

You want a seamless experience for your GraphQL API users throughout any changes you make, especially for the users and graph-usage patterns most crucial to your business. That means knowing the frequent users of the areas of your graph under the most heavy usage. You also want to recognize and bolster any GraphQL operations that your most important users frequently call, in addition to the objects and fields those users consume.

4) Understand graph errors and potential error hotspots

The right actionable intelligence will prepare enterprise teams to anticipate and address graph errors. This requires visibility into where, when, why, and how often graph errors occur and error hotspots (for example, high P95 latencies indicative of potential timeouts). Insights into which users feel the brunt of graph errors are also valuable in shaping mitigation strategies.

5) Understand graph performance bottlenecks

Bottlenecks caused by your GraphQL server or gateway itself may call for a shift in your graph’s underpinnings. Bottlenecks can also arise from costly queries of substantial depth and height, especially when these queries see intensive use.

To foresee and address bottlenecks, actionable insights include the average and P95 latencies of your graph’s most-called operations and the heights and depths of your graph queries with the worst performance metrics. You also want to know the number of times that bottlenecked operations get called each day, the subgraphs at the root of bottlenecks, the latency added by supergraph query planning, and which users are impacted the most by bottlenecks.

Graph Knowledge = The Power to Avoid Vendor Lock-In

Implementing the GraphQL API visibility and actionable intelligence strategies covered here help ensure that an enterprise has the means to modernize or migrate as necessary to preserve its strategic independence. Those unfortunate organizations already in the grip of vendor lock-in may need to bide their time but should still use these same techniques to improve and decouple their fundamental GraphQL services from their vendor-controlled GraphQL technology. There’s no acceptable reason why organizations relying on free and open source GraphQL servers and gateways should find themselves stuck in proprietary software arrangements. Thankfully, the right knowledge can deliver enterprises a more flexible future as they modernize API development with GraphQL.

October 4, 2023
Harness Launches Open Source Gitness Platform

At its unscripted conference today, Harness unveiled a streamlined open source git platform, dubbed Gitness, that is significantly lighter weight than the platforms DevOps teams traditionally rely on to build increasingly complex applications.

At the same time, the company is adding four additional modules to its continuous integration/continuous delivery (CI/CD) platform to improve developer productivity and make it simple to build and deploy secure applications.

Finally, Harness is launching an Engineering Excellence Collective, a consortium of 300 senior engineering leaders. The collective is developing a maturity model based on 11 pillars for achieving software engineering excellence; a FinOps Excellence community initiative was also announced to advance best practices for controlling cloud computing costs.

Harness CEO Jyoti Bansal said the need for a modern open source Git platform—as an alternative to legacy platforms developed a decade ago—has never been more apparent. Gitness provides all the collaborations, performance, analytics and security capabilities DevOps teams require in a streamlined platform that reduces overall bloat by seamlessly integrating, for example, code hosting and automated pipelines, he added.

Other core capabilities include automated status checks and mandatory reviews, branch protection rules, user access management tools and a semantic search engine that has been infused with generative artificial intelligence (AI) capabilities that invoke multiple large language models (LLMs).

Collectively, the approach reduces the amount of time spent when attempting to integrate and manage disparate tools on legacy platforms that inevitably lead to the creation of additional bottlenecks, said Bansal. Rather than bolting on additional tools, the time has come to simplify software engineering by migrating to a platform specifically designed to address modern software engineering requirements, he added.

Designed to be installed using Docker contains in a matter of minutes, DevOps teams can also take advantage of tools to automatically migrate their existing repositories and pipelines from platforms such as GitHub and GitLab.

Gitness is also at the core of one of the four additional modules that Harness is adding to its portfolio. The Harness Repository is a premium tool, available in beta, that provides governance and compliance-as-code capabilities embedded within the repository.

The three other modules being added are a Harness Internal Developer Portal based on open source Backstage software, Harness Infrastructure as Code Management that makes it simpler to implement GitOps workflows at scale and Harness Software Supply Chain Assurance to provide visibility into open source software components and manage the creation of software bills of materials (SBOMs).

It’s not clear whether DevOps teams are ready to rip and replace existing platforms, but the pressure to improve developer productivity has never been higher. Organizations increasingly realize that developers are spending the bulk of their time managing development environments tied to legacy platforms that create bottlenecks and are becoming more costly to manage and sustain.

Harness, in effect, is now calling for the establishment of an alternative approach based on open source software that promises to streamline software engineering workflows. As applications incorporate microservices, security controls and AI models, they are only going to become more challenging to build and deploy at scale.

September 21, 2023
Innersourcing Open Source Principles in the Enterprise
Software development has experienced a seismic shift in recent years, with open source becoming the heartbeat of innovation. A full 90% of enterprise IT leaders say their organization is using open source, according to a Red Hat report. Leading this transformation is GitHub, the ubiquitous platform that not only jumpstarted the open source movement but has also been quietly transforming how enterprises approach development at large.

But it’s not just tech companies trumpeting the open source creed—academics are on board, too. A paper recently published by the University of Georgia found a hard-to-dismiss correlation between open source innovation and overall business success. “We find a significantly positive association between GitHub activity and future realized net income, suggesting that open innovation is a strong predictor of future earnings,” according to the report.

I recently met with Chris Reddington, senior manager of developer advocacy at GitHub, to dive into how and why open source is becoming essential for enterprise innovation. According to Reddington, the once-dominant proprietary information mindset is shifting, with open source cultural principles now permeating most companies, a practice also known as innersource. Put simply, open source is now cool in the enterprise.

Bridging the Open Source-Enterprise Divide

GitHub’s close ties to the open source space have allowed them to illuminate the parallels between the open source and enterprise worlds. Reddington noted that, nowadays, open source and enterprise aren’t really separate communities but rather intertwined facets of the same software ecosystem. This movement is represented by the open source program office (OSPO) becoming more commonplace in large organizations.

For an example of innersource in practice, take Porsche, who recently announced the Porsche FOSS Movement to guide internal common values and development principles around open source. We’ve also seen similar large-scale commitments to open source values at Walmart, Spotify, Facebook and others.

Open source clearly isn’t just for startups—large companies are increasingly seeing an advantage in both using open source software and sharing their own open source components. The growing investment and maturation of OSS initiatives are positive strides, as it’s important to the stability of the free open source software (FOSS) ecosystem as a whole that those using software contribute back to its maintenance, especially to core packages the world depends upon.

The Power of Innersource

The study from the University of Georgia brings to light how enterprises increasingly recognize the need to work in sync with their developers and customers to remain competitive. This is a testament to the evolving nature of development—an industry that’s embracing a “team sport” mentality instead of a siloed approach, explained Reddington. And as enterprises align with open source principles, they are discovering the power of innersource, he said. “Innersource is a development methodology where engineers build proprietary software using best practices from large-scale open source projects, like Kubernetes or Microsoft’s Visual Studio Code,” GitHub explained.

Reddington elaborated on two key elements of this approach:
- Breaking down silos: The traditional organization-centric structure of projects is being replaced by a more transparent and collaborative approach. Innersource encourages transparency and asynchronous communication, tearing down barriers that previously inhibited efficient collaboration.
- Engaging responsibly: Reddington emphasized the significance of enterprises giving back to the open source community. Whether through sponsorships, allocating time for engineers to contribute, or open sourcing their projects, companies can foster symbiotic relationships where they benefit from and contribute to the open source realm.
Essentially, by adopting innersource principles, enterprises foster an environment that promotes collaboration and knowledge sharing, thus enabling innovation to flourish across teams and projects.

Balancing Open Innovation and Proprietary Software

One of the most intriguing aspects of this transformation is finding the equilibrium between open innovation and proprietary software. Reddington acknowledged the necessity for businesses to generate profits, which inherently stems from delivering value.

While reinventing the wheel may be tempting, Reddington highlighted the importance of recognizing existing solutions and incorporating them into ongoing projects. This pragmatic approach maximizes efficiency and enables teams to focus on the unique, value-generating aspects of their work.

Organizations should also consider what software of their own could be open sourced. From a business perspective, this most likely won’t be the “crown jewels,” said Reddington, but ancillary components that others could benefit from. For example, had it not been for Google open sourcing Kubernetes, Lyft open sourcing Envoy Proxy or Facebook sharing GraphQL, we would not have these standard components for all to use.

Open Source Movements: Past, Present, and Future

Over the last decade, we’ve seen significant, more institutional maturation around open source projects. Reddington pointed to the Cloud Native Computing Foundation (CNCF) as a standout. He specifically noted Kubernetes-based Event Driven Autoscaler (KEDA) as a cool project to help scale Kubernetes containers based on the number of events needing to be processed.

He also underscored the surge of interest in generative AI, evidenced by projects like GitHub Copilot. It will be curious to see how AI assistants incorporate open source at their core, such as open foundational models and community-supported datasets.

Potential Setbacks To The Open Source and Innersource Movement

In the dynamic realm of enterprise-scale software development, nurturing innovation can be a complex endeavor. Innersource principles are one way to change the attitude toward development, making it more efficient, collaborative and aligned to the current and future developer experience expectations.

Yet, this isn’t to say we haven’t had some setbacks recently that could threaten the progress the OSS community has built. For example, HashiCorp recently abandoned its open source license around Terraform in favor of a business source license (BSL). There is also an increasing number of supply chain-related attacks that threaten common open source libraries.

Gazing Into the Future

As Reddington aptly summarized, we’re still in the early days of this transformative journey. The developer community’s inclination for experimentation will undoubtedly drive the solutions to the most pressing challenges in open source. And with enterprise adoption on the rise, the future promises more cohesive collaboration, broader adoption of innersource principles and continued innovation driven by open source values.

Interestingly, we’re seeing more synergy between open source and enterprise domains, which used to feel like separate realms entirely. And it will be interesting to see how businesses continue to balance transparency and proprietary value into the future. No matter what, the lines between them are blurring, enabling companies to harness the power of innersource for enhanced collaboration and innovation.
September 21, 2023
Summit Highlights Open Source Software Security Progress

More than a year after the Open Source Security Foundation (OpenSSF) summit kicked off an initiative to better secure open source software-based software supply chains, a lot of progress has been made—but much work remains to be done.

The OpenSSF this week hosted a Secure Open Source Software (SOSS) Summit 2023 event in Washington, D.C., during which it made available a Secure Open Source Software Vision Brief 2023. The brief describes the various efforts being made to improve open source software security, including, for example, providing maintainers of open source software projects with free DevSecOps tools.

This week, the Cybersecurity and Infrastructure Security Agency (CISA) also published an Open Source Software Security Roadmap that defines its role in providing more visibility into how open source is being used and the associated risks.

However, it’s not clear how much maintainers of these projects are embracing DevSecOps workflows. Many projects are led by a handful of maintainers that often lack the time and skills required to ensure the components they employ have not, for example, been compromised by malware that might find its way into downstream applications.

Sonatype CTO Brian Fox said the focus of the summit’s participants will be on education and securing public code repositories. While there is a lot of focus on maintainers, he added that there is not enough attention to all the organizations that have not updated open source components within a software build with known vulnerabilities. Many IT teams are reluctant to upgrade components for fear of breaking applications with complex dependencies, but the amount of code that is potentially insecure is massive. “There’s a huge mountain to climb,” he said.

The challenge is determining which code to upgrade first in a way that will have the most impact in terms of improving application security, added Fox.

Shawn Ahmed, chief product officer for CloudBees, also noted that organizations should evaluate the DevSecOps processes being applied by maintainers of open source software projects. CloudBees, for example, reviews the plugins provided by the open source community for the Jenkins continuous integration/continuous delivery (CI/CD) platform for vulnerabilities every month, he noted.

It’s also critical for organizations that consume open source software to contribute code to improve the security of open source software, added Ahmed. Too many organizations are focused on open source software security without devoting resources that would help resolve the issue. he added. “We need to make sure open source software is not a vulnerability threat,” said Ahmed.

Moran Ashkenazi, chief strategy officer (CSO) and vice president of security engineering for JFrog, said on the plus side, there is a lot more awareness of open source software security issues since the disclosure of a zero-day vulnerability in the Log4j shell tool widely used to manage logs in Java applications. But added that this issue is not going to be resolved overnight. “That was a wakeup call,” she said.

Tapabrata Pal, vice president of architecture for Fidelity Investments, noted that the fundamental problem is that cybercriminals have become more adept at injecting malware into software supply chains, so no one knows whether their application environments might be compromised. “It’s a serious issue,” he said.

It’s not known whether the open source software that many developers routinely incorporate into their applications might contain vulnerabilities that could be exploited, but the level of technical debt that has accrued over the last several decades is considerable. The issue is finding a way to address it without becoming overwhelmed by the enormity of the task.

September 14, 2023
Latest eBPF Advances Are Harbingers of Major Changes to IT

The open source community advancing the extended Berkeley Packet Filter (eBPF) gathered at a virtual eBPF Summit today that featured a demonstration of eBPF being used in a Windows environment. eBPF enables applications to run within a sandbox in the Linux microkernel.

Microsoft has signaled its intent to employ eBPF within Windows like the latest version of Linux already does today. That capability makes it possible to, for example, run networking, security, observability and storage software closer to the kernel to dramatically improve performance versus running that software in user space.

Liz Rice, chief open source officer for Isovalent, a provider of networking software based on eBPF, said that while it’s clear eBPF will have a major impact on IT, the transition to eBPF won’t be a single event. Instead, it will be more of an evolutionary process that will occur as organizations upgrade their operating systems and the various tools and applications they employ.

The eBPF Foundation, an arm of the Linux Foundation, was formed to expand the use of eBPF across all operating systems. Members of the eBPF Foundation include Facebook, Google, Isovalent, Microsoft and Netflix.

In effect, eBPF changes the way operating systems are designed. It bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that were historically completely independent. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput at a time when brute-force cyberattacks are overwhelming existing cybersecurity tools.

Similarly, networking, storage and observability tools should be able to run at much higher levels of scale. That’s especially critical for DevOps teams embracing observability platforms that need to dynamically process massive amounts of data in near-real-time.

Less clear is the degree to which eBPF might one day drive convergence across distinct categories once multiple tools and applications run at the microkernel level.

IT organizations would be well-advised to ask their vendors when they plan to support eBPF as they plan their upgrade cycles. It’s not likely IT teams are going to want to upgrade every tool and application that can benefit from eBPF all at once. In fact, updates to these applications that don’t support eBPF might even be postponed in the knowledge that a much larger and arguably more critical update is on the immediate horizon.

In the meantime, as the number of organizations running the latest versions of Linux continues to increase, more hands-on experience with eBPF will be gained. IT teams may not need to concern themselves with what is occurring in the microkernel of the operating systems, but they should at least understand how eBPF ultimately helps reduce the total cost of running IT and how it makes the Linux microkernel more programmable. Those IT teams, however, will soon discover how stark a difference in performance there will be.

September 13, 2023