devopsy.kinsta.cloud

Tag: linux

JFrog Aims to Push DevOps to the Edge

At its swampUP event, JFrog launched a JFrog Connect platform for updating, managing, monitoring and securing remote Linux and internet of things (IoT) devices.

The JFrog Connect platform is based on a lightweight agent JFrog gained last year with the acquisition of Upswift and that has now been integrated with the JFrog Artifactory repository. DevOps teams can now use a low-code tool to build flow logic that is integrated with a Git repository in addition to triggering a rollback to ensure a device returns to its previous known software state when necessary.

DevOps teams will also be able to connect to a wide range of Linux-based IoT devices from anywhere in the world and then manage and monitor them in real-time by name, description or tags.

JFrog CTO Yoav Landman said in the near future, DevOps teams will find themselves building and deploying applications that will not only be deployed on fleets of endpoints at the network edge, but also will be continuously updated.

Historically, those endpoints have been managed mainly by operations technology (OT) teams, but it will soon be up to DevOps teams to automate the management of those platforms down to an individual container using the lightweight agent created by Upswift, he said.

The number of application workloads being deployed on edge computing platforms is rising sharply as organizations seek to process and analyze data closer to the point where it is being created and consumed. In fact, that shift is at the core of digital business transformation initiatives that depend on edge computing platforms and devices running, for example, IoT applications that process data in near-real-time.

The challenge today is in many cases organizations lack the ability to consistently deploy software on edge computing platforms, which results in the creation of a hodgepodge of homegrown tools to manually deploy software on edge computing platforms. There is a clear need to raise the DevOps bar to address edge computing requirements, said Landman.

It’s too early to say just how much software will be deployed on the edge, but there may come a day when there is more software at the edge than there is in the cloud. In the meantime, the line between edge computing and cloud computing continues to blur. Many more DevOps teams will soon find themselves managing thousands of distributed endpoints from a central cloud.

In the meantime, DevOps teams should expect to find themselves working more closely with OT teams that generally have a significantly different culture. Many of those OT teams are just now coming to terms with the implications of connecting endpoints to the internet. The need to continuously update the software on those platforms is not typically the first issue that comes to mind for them. Today much of the software running on edge computing platforms is rarely updated. In the future, however, that software is likely to be updated at a rate that many OT teams are not ready to envision much less manage.

May 26, 2022
Libbpf Vs. BCC for BPF Development

If you’re into Linux development, you’ve probably heard BPF mentioned over the last few years. BPF stands for Berkeley Packet Filter, and the technology has a large number of use cases. BPF apps can get deep access into an operating system and enable you to perform tasks such as high-performance load balancing, DDoS mitigation and more simply and easily.

Libbpf and BCC-Tools are both sets of tools to help with BPF development. However, both of these have their own strengths and weaknesses. In this guide, you will learn in detail about the two tool collections and understand when to use them.

What is BCC?

BCC stands for BPF Compiler Collection and is one of the oldest ways to develop BPF applications. It helps you embed your BPF code into your user-space program in the form of a plain string. When the user-space program is executed by the kernel, BCC invokes its embedded Clang/LLVM, pulls in system-wide kernel headers and compiles the program on the spot.

Since BCC compiles BPF programs on the host machine, it ensures that the memory layout your BPF program expects is precisely the same as that of the target host.

BPF programs are designed to be injected directly into a kernel, so BCC tools seem to be the perfect solution for developing BPF applications. However, they have proven to be bulky in the modern context due to their heavy reliance on the Clang/LLVM combination, which is resource-intensive. This has led to the need for a better, more modern solution to developing BPF apps.

What is Libbpf?

Libbpf is one of the hottest new BPF tools on the market. It is usually coupled with BPF CO-RE (which stands for compile once, run everywhere). BPF CO-RE enables you to generate binaries that run on multiple kernel versions.

The idea behind Libbpf is to make BPF development as similar to other forms of development as possible. With BPF CO-RE, Libbpf does this by compiling BPF programs into small binaries that can be deployed to multiple deployment hosts. Libbpf does the setup work like loading and verifying programs, creating maps, attaching to hooks, etc., which enables developers to focus on more critical tasks at hand, such as program performance and correctness.

Libbpf aims to eliminate the overheads associated with BPF app development and deployment by reducing the dependency on system-wide kernel headers as well as Clang/LLVM libraries for compilation on runtime.

Libbpf Vs. BCC: Key Takeaways

Now that you understand how each of these tools works, let’s compare some key benchmarks to learn when to use them.

Dependency Management

BCC relies on kernel header packages that need to be installed on the target host machines. While it is not a problem in most cases, it can get difficult to set up and maintain if you are working on multiple machines.

Libbpf, on the other hand, eliminates this issue by relying on a vmheader file. This file includes multiple kernel types which helps you to remove dependency from system-wide kernel headers.

Compilation Style

For programs written with BCC, compilation occurs at runtime. To facilitate this, such programs require Clang/LLVM be present on the host machine. This adds to the code footprint of your programs.

Furthermore, the Clang/LLVM libraries are resource-intensive. Complete libraries need to be available and run at compile time, even for small programs. This can upset a perfectly balanced BPF workload on the host machine.

On the other hand, Libbpf enables you to generate binaries that are compiled once and can be run anywhere. You do not need any system-wide dependencies to be present on the target machine for running such apps. Hence, it reduces the overall application size as well as resource consumption on runtime.

Error Detection

For apps developed with BCC tools, errors are detected only at runtime since that is when the programs are compiled. This leads to a rather sluggish development experience compared to apps developed using Libbpf. Since Libbpf enables compilation while developing, finding and fixing bugs is easier.

Final Thoughts

In this guide, we walked you through what Libbpf and BCC tools are and compared them based on some important aspects of application development and delivery.

All in all, Libbpf appears to best BCC tools in most aspects. BCC tools offer fast prototyping and experimentation possibilities, but when it comes to production deployments, BCC tools turn out to be a rather expensive choice compared to Libbpf.

You can learn more about why BCC became the popular alternative and how it was eventually replaced by Libbpf here. Even if you currently use BCC tools for your apps, you can easily migrate to Libbpf by following a few simple steps. And, in most cases, it is better to opt for Libbpf than BCC tools for a better development experience and application performance.

May 3, 2022
Apple Outage Outrage | Linux Random Redo | Okta Hacked (or Not)

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Why Apple services were down, Linux gets a huge RNG overhaul, and we wonder if Okta was hacked again. (more…)

March 22, 2022
Why the Linux Kernel is Still Important

Last year, the Linux kernel turned 30. If you’re someone who’s been immersed in the Linux world since Y2K like I am, it may feel a bit surreal that so much time has passed since the kernel’s inception.

As a training architect at A Cloud Guru (ACG), I teach courses about all things Linux and specialize in hands-on lab-based learning. Before joining ACG, I worked as a Unix systems engineer at GE and IBM as well as a technical account manager and customer advocate for Red Hat. I’m hugely passionate about Linux, just as so many other engineers and enthusiasts are, not just because of its importance to our careers but its impact overall.

In its 30 years of existence, the Linux kernel has had a massive impact on the modern computing landscape—revolutionizing what’s possible for operating systems and allowing countless tinkerers to get their hands dirty in the process. Linux has also become the foundation of paradigm-shifting innovations over the years due to its ever-evolving nature.

Here are some of the key reasons why the kernel is so valuable and why it will continue to be a major player in the computing landscape for years to come.

My Own Linux Journey

My first exposure to Linux was in college in 1999. I took an introductory Unix Shell Scripting class and Fedora was installed on the lab servers because it was free and easily scalable. I view Linux as my entry point into customizable operating systems. After learning basic Unix commands on Linux, I ultimately went on to pursue a career supporting multiple commercial Unix vendors.

As Linux made its way into larger enterprise-level companies, I quickly returned to working with Linux distributions. Something I’ve always loved about Linux is the capacity to fine-tune your system to support the applications and distros that work best for your projects. Linux improves the functionality of whatever applications you’re running. The Linux kernel has literally changed how the world processes information, which is why I’m so invested in the software.

Now, it’s my job to share my knowledge and passion for Linux with other technologists. At ACG, I develop courses to help aspiring Linux experts learn how to optimize their systems. Additionally, I contribute to Linux this Month, an ACG-hosted web series that provides monthly updates from the global Linux community.

The fact that I am able to build a career around teaching Linux and staying up-to-date with Linux news shows the vast use cases and applications of the kernel. What makes Linux so unique and evergreen is its open source nature—Linux innovations are only limited by the creativity of the technologists who use and adapt it.

The Open Source Effect

When Linux first arrived, it was mostly a hobby for enthusiastic engineers and computer science students who could contribute by developing code. The steep learning curve associated with fitting Linux to your machine was a barrier to entry for more novice programmers.

Over time, this has changed considerably. Online forums, workshops and classes have made Linux more accessible to the average internet user. The free sharing of ideas has come to epitomize the open-source community, and for software engineers, Linux is at the heart of this community. This democratization of Linux has had incredibly positive impacts on the computing world.

Now, Linux is everywhere. Enterprise-level companies use Linux distributions to process the biggest production workloads in the world. It has replaced proprietary commercial Unix operating systems in very large companies with better stability and less downtime. Because Linux systems can be as small or as large as you want, it’s also now being used in our homes for smart and mobile devices as well.

The open source nature of Linux is incredibly beneficial for these enterprise-level companies. Linux distributors leverage the contributions of the entire open source community. This wide range of contributors produces a more stable product with more features but also ensures that the OS keeps growing and solving real-world problems that are beneficial to a wide range of users.

Linux Kernel Runs the Cloud

Potentially the most impactful outcome of the kernel is its importance to the infrastructure of modern cloud computing. Linux’s scalability has paved the way for supercomputers and server farms to function efficiently while requiring relatively lightweight computing resources. In fact, Linux supports about 90% of the public cloud workload.

Without Linux, the cloud as we know it would not exist. This is, in part, because Linux has become so ubiquitous—its use cases are nearly limitless. Because it has been time-tested, many engineers and IT professionals have a solid grasp of Linux fundamentals, making it an attractive choice for enterprise companies dealing with the cloud.

Cloud-based software and products are increasingly becoming the norm in the engineering world. Unsurprisingly, major cloud providers such as AWS, Azure and Google Cloud are all supported by Linux as well. Linux is unique because it is a shape-shifter that can conform to the needs of any given engineering environment, and it’s incredibly stable because of the army of contributors that fortify weak points in the software.

Linux Forever

The concept of Linux is just as important as the Linux product. The concept allows a free and open source operating system to be refined, reinforced and replicated across an endless web of contributors. Thirty years is a long time for software to be relevant, especially with the ever-shortening tech product cycles. Because it was designed with the intent to be changed and updated by an open source community, Linux has no foreseeable expiration date.

Had Linux not achieved the prominence it has today, we would see more commercial Unix vendors attempting to solve some of the problems that Linux addresses, but none would address them all. Additionally, customers would have to choose which OS to invest in based on which addresses some of their use cases, but none would be as beneficial as the Linux OS.

Linux is always growing and will become even more popular within the next few years. As more people become familiar with Linux and learn to use it, I see major potential for growth in the mobile computing space, within personal computers, and across small and large companies. In fact, we are already seeing it filter into home gaming systems and Raspberry Pi projects. With Linux, the sky’s the limit!

March 17, 2022
Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the third in our series of the Best of 2021.

Linux does, occasionally, raise security concerns. While many users see it as the most secure, robust and versatile operating system available — that’s this writer’s opinion, as well — security precautions still have to be taken.

A recent, widely publicized case illustrated this point; Linux creator himself, Linus Torvalds, warned against the use of the Linux 5.12 release. He described a “nasty bug,” and wrote that the situation is a “mess,” due to the use of swap files when adding Linux updates. This nasty bug, in fact, had the potential to destroy entire root directories.

Some of the main takeaways following this “mess” include: tread very carefully when installing early Linux releases, especially those that involve swapping files instead of partitions, and especially, despite Linux’s well-known security advantages, avoid becoming complacent, because Linux security is not always foolproof.

Hence, while the “state of Linux security today is quite good, and has evolved in a positive way with more visibility and security features built, like many operating systems, you must install, configure and manage it with security in mind; that is how cybercriminals take advantage, [via] the human touch,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic, a provider of privileged access management (PAM) solutions.

A Patch for Nastiness

As Torvalds noted a few weeks ago, “most people don’t use a swap file, but a separate swap partition and the bug in question really only happens when you have a regular file system, and put a file on as a swap.”

“The bad news is that the reason we support swap files in the first place is that they do end up having some flexibility advantages, and so some people do use them for that reason. If so, do not use [release candidate] RC1,” Torvalds wrote. “Thus, the renaming of the tag.”

After issuing the warning, Torvalds released a patch that he says prevents the bug from destroying swap file systems. However, it may have already been too late for early adopters of release 5.12. Ubuntu, a leading Linux distro, can swap files by default.

“It is nasty bug if you are still using swap files,” Carson said. “If you do still use swap files, then you could be impacted, resulting in potential data loss or a corrupted system.”

DevOps teams – or anyone else running Linux and installing patches, whether on multi-servers or on individual workstations – still need, of course, to follow strict best practices. “Like any operating system, security depends entirely on how you use, configure or manage the operating system,” Carson said. “Each new Linux update tries to improve security; however, to get the value, you must enable and configure it correctly.”

Linux Goodness

The fact that Torvalds was so forthcoming about the bug, as well as the level of transparency that the Linux kernel offers, also demonstrates one of the many reasons Linux remains popular. Given that the Linux kernel, in one variety or another, is used “not only in about 50% of the internet servers of the world, but also in a substantial part of all our smartphones, it is good to see this level of transparency at ‘root level,” said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), which providers cybersecurity and compliance software.

“The security of Linux is based on its transparency; the ability to review the code of a distribution,” says Schrader. “Quite often forgotten is that transparency also involves talking about the mistakes, the errors, those nasty bugs.”

Citing National Institute of Standards and Technology (NIST) vulnerability database statistics, Schrader described how, compared to the Windows family of desktop and server operating systems, for example, the Linux kernel shows better results for overall vulnerabilities. The number of vulnerabilities have also declined over the past four years, while Microsoft’s operating systems do not display the same trend, according to NIST’s national vulnerability database.

Since Linux’s famous kernel is open source and transparent, it is possible to extrapolate that there are a greater number of potential vulnerability watchdogs compared to those monitoring vulnerabilities in closed systems. Some may argue that Microsoft has been, at times, less successful at detecting vulnerabilities and issuing much-needed patches.

However, Linux users still must remain vigilant.

“Still, for any of the Linux distributions, anyone using the early release candidates — RC1 in particular — should make sure that their own development or build process is undergoing change control, so that no mishaps will transfer the nasty bug into a production environment,” said Schrader.

December 21, 2021
AWS Outage Outrage | Rusty Linux | ARM Latest

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Amazon Web Services falls on its face, Linux’s move to Rust takes the next step, and the FTC stabs another fatal wound in the horrible Arm/Nvidia deal.

AWS us-east-1 Breaks the Internet

First up this week: Amazon Web Services’ outage on Tuesday sent scads of DevOps teams scrambling. But what was the cause?

Analysis: DNS DDoS?

While we still don’t officially know, there is some chatter to speculate on. Could it be that AWS’s DNS infrastructure is vulnerable to DNS reflection attacks?

Hello. It’s Dave Lee: From angry Adele fans to broken robot vacuums

It started early on Tuesday morning. Robot vacuums ceased sucking, Wi-Fi cameras stopped watching and eager Tinder daters were left unable to “swipe right.”
…
Within Amazon itself … the unthinkable occurred: grounded delivery drivers were unable to load packages and deliver to customers’ doorsteps, just as the peak Christmas season begins to step up. … Drivers at multiple facilities across the country were sent home … some dreading whatever workload may await.
…
An outage at Amazon Web Services … had rippled through the online economy, crippling services used by millions of people. … An “impairment of several network devices” in … US-EAST-1 was the “root cause” of the disruption, Amazon said.

I was wondering what DevOps’ers learned? Enter Max Rozen:

AWS took approximately an hour to update their status page to indicate anything was happening. … While our infrastructure is hosted across 20 AWS regions, a single point of failure was hosted in us-east-1.
…
What we learned: … I learned the saying, “Friends don’t let friends deploy in us-east-1.” As a result, we’ll be moving some things over to us-east-2 to start with. Secondly, we’ll be adding redundancy and the ability to failover to another AWS region.

Getting inside the business, here are Eugene Kim and Katherine Long, to go over everything:

Even inside AWS … information on the outage remained sketchy [and] rumors spread among staff. One AWS employee speculated that the outage was caused by an “orchestrated DNS attack,” while another employee downplayed those concerns.
…
According to a screenshot of an internal AWS communique … ”firewalls are being overwhelmed.” … A separate internal note said … the network issue “specifically impacted” Amazon’s internal DNS servers.

Linux Officially Gets Rust Code

The Rust language has taken the next step toward being part of Linux kernel development. Some “drivers and other non-core kernel programs” written in Rust can now be PR’ed. But there’s still no plan to go further.

Analysis: Still needs lubrication

Rust is much better than C for writing memory-safe code, and coding provable implementations. However, bootstrapping the toolchain is still stupidly hard.

Steven J. Vaughan-Nichols: Linux is getting more Rust in it by the day

It wasn’t that long ago that the very idea that another language besides C would be used in the Linux kernel would have been laughed at. [But] this has been coming for several years now. … Why? Because it’s much safer than C, especially at handling memory errors.
…
The next “patch series [adds] support for Rust as a second language to the Linux kernel.” … While he’s encouraging a slow but steady approach to introducing Rust into Linux, [Linus Torvalds] has also said that using Rust interfaces for drivers and other non-core kernel programs makes sense. [But] no one’s going to rewrite the kernel’s 25 million lines of C in Rust.
…
The Rust code proposed for the kernel now relies on the stable Rust compiler rather than the beta compilers. … As Linux kernel and lead Rust on Linux, developer Miguel Ojeda, put it, “By upgrading the compiler, we have been able to take off the list a few unstable features we were using.”

How is it “safer than C”? ssokolow shares this fascinating insight:

Rust tries to encourage that through good API design which allows it to be proven at compile time that the invariants will be upheld, thus avoiding the need to choose between safety and performance. My favourite example … is “the typestate pattern” which lets you teach the Rust compiler to prove correct traversal of any state machine.
…
A more mundane example would be the iterator APIs: No need for bounds checking if the iterator looks up the length once at the beginning and the APIs prevent you from wandering out of each entry it hands you. … Rust programs tend to run faster than their C and C++ counterparts because the developers feel comfortable introducing and maintaining more aggressive optimizations.

“And then a miracle occurs,” quips @dave_universetf:

I’m intrigued [but] I can’t help but feel sorry for the people doing bootstrapping work. C is hard enough to bootstrap, but Rust is a whole new level of pain.
…
Rust is literally not source bootstrappable. [The compiler] that works right now is Rust 1.39. So then you have to compile 18 more Rust compilers to get up to the current version.

Nvidia’s ARM Woes Continue

As I’ve said before, this deal really shouldn’t go through. Ownership of the ARM ISA and the core designs would be a huge strategic stick to beat Nvidia’s competition with. ARM chips are an increasing fixture in the datacenter—especially those that value “performance per Watt.”

Analysis: Stick a fork in it

Nvidia’s attempt to buy Arm from SoftBank is suffering death by a thousand cuts. And the latest wound comes from the FTC’s Higonokami knife.

Paul Thurrott: FTC Sues to Block Sale of Arm to Nvidia

The U.S. Federal Trade Commission … sued to block Nvidia’s $40 billion acquisition of … Arm, citing competitive concerns. … Nvidia announced its intention to purchase Arm from Softbank for about $40 billion, over one year ago, in September 2020. But the deal fell under immediate criticism from the major industry players that rely on Arm’s … designs, and the UK government announced … it would “intervene” for national security reasons.
…
Now, it may not happen at all. “We will continue to work to demonstrate that this transaction will benefit the industry and promote competition,” Nvidia said.

Explain like I’m 5? Here’s swillden:

ARM has a non-trivial and increasing share in the data center. … ARM is much, much bigger than phones … Arm is an important competitor to Intel and AMD. … Arm licensing IP to Qualcom, nVidia, Mediatek, Unisoc, Samsung, Apple, Google, etc., has been great for many areas of technology.
…
Arm acting as a common … designer for a bunch of different chip vendors has been extremely good for … progress in CPU/SoC design. … It would be really unfortunate if a lot of the industry were to shift back to x86, eliminating all of the competition on hardware features that we get from the many ARM chipmakers.

But what of SoftBank? Ian King runs the numbers:

Wall Street analysts are all but calling the deal dead. … According to Citibank analysts, the chances of the Nvidia purchase closing are now just 5%.
…
The company faces the prospect of losing a jackpot of roughly $74 billion. … Other suitors are unlikely to match such an offer. … So SoftBank’s best alternative may well be an initial public offering for the Arm business—but that is likely to value the company at … $19.8 billion—only about 27% the value of the … Nvidia deal.

The Moral of the Story: It’s no secret that the both of us are running out of time

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk. (It’s so typical of me to talk about myself—I’m sorry.)

Image: Ləman Şirinzadə (via Unsplash)

December 9, 2021
What 30 Years of Linux Taught the Software Industry
Linux has become the largest collaborative development project in the history of computing over the last 30 years. Reflecting on what made this possible and how its open source philosophy finally imposed itself in the industry can offer software vendors valuable lessons from this amazing success story.

The web may not have reached full adulthood yet, but it has already crafted its own mythology.

August 25, 1991: Linus Torvalds, a 21-year-old university student from Finland, writes a post to a Usenet group: “Hello everybody out there using minix — I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386 (486) AT clones […]”. A few weeks later, the project, which will eventually be known as Linux, is published for the first time.

This is the starting point of an epic that few could have foreseen.

Fast-forward 30 years and the Linux kernel isn’t only running on most of the web servers and smartphones around the globe, but it also supports virtually all of the much more recent cloud infrastructure. Without open source programs like Linux, cloud computing wouldn’t have happened.

Among the major factors that propelled Linux to success is security. Today, the largest software companies in the world are taking open source security to new levels, but the Linux project was one of the first to emphasize this.

How Linux Became the Backbone of the Modern IT World

Brief History

Open source predates the Linux project by many years and is arguably as old as software itself. Yet, it is the success of the latter that propelled this movement in the 1990s. When it was first submitted for contribution in 1991 by Torvalds, the Linux kernel was the GNU project’s ‘missing link’ to a completely free software operating system, which could be distributed and even sold without restrictions. In the following years, and as the project started to incorporate proprietary licensed components and grow in popularity, a clarification on the meaning of “free software” became necessary.

This led to the coining of the term “open source” as we use it today, thanks in part to Eric Raymond’s seminal paper The Cathedral and the Bazaar, a “reflective analysis of the hacker community and free software principles.” Open source was chosen to qualify software in which the source code is available to the general public for use or modification from its original design, depending on the terms of the license. People may then download, modify and publish their version of source code (fork) back to the community.

Open source projects started gaining traction in the late nineties thanks to the popularity of software like Apache HTTP Server, MySQL and PHP to run the first dynamic websites on the internet.

Facts and Figures

Today, not only is Linux powering most of the digital era, but open source has become the leading model for how we build and ship software. Though most people don’t realize it, much of the technology we rely on every day runs on free and open source software (FOSS). Phones, cars, planes and even many cutting-edge artificial intelligence programs use open source software. According to the Linux Foundation, 96.3% of the world’s top one million servers run on Linux and 95% of all cloud infrastructure operates on it. Other infrastructure also relies on open source: 70% of global mobile subscribers use devices running on networks built using ONAP (Open Network Automation Platform).

Linux adoption is very high in professional IT, where it’s become a de facto standard, especially with the advent of the cloud era. In fact, 83.1% of developers said Linux is the platform they prefer to work on. This success is due, in large part, to the community that contributed to its source code since its creation: More than 15,000 developers from more than 1,500 companies. Linux went on to become, arguably, the biggest success story of the free software movement, proving that open source could lead to the creation of software as powerful as any sold by a corporation.

The Linux Foundation, a non-profit technology consortium founded in 2000 to support the collaborative development of Linux and OS software projects, is itself a big success. It now has more than 100 projects under its umbrella, spread across technology sectors like artificial intelligence, autonomous vehicles, networking and security. Several subset foundations have also emerged over the years, including the Cloud Foundry Foundation, the influential Cloud Native Computing Foundation, and the recently announced Open Source Security Foundation. The Foundation estimates the total shared value created from the collective contributions of its community to a whopping $54.1 billion.

All these achievements may not have been possible without the embrace of open source by the enterprise world, which may represent its biggest win.

Enterprise Adoption

Companies began to realize that many open source projects were easier and cheaper to implement than asking their developers to build the basic pieces of an internet business over and over again from scratch.

Twenty years ago, most businesses ran atop proprietary software from Microsoft, Oracle and IBM, and the idea of collaborating on big software projects might have sounded laughable to them. Today, these companies, along with relative newcomers such as Google, Facebook and Amazon, are not only employing thousands of full-time contributors to work on open source projects like Linux, they also regularly choose to open source some of their state-of-the-art projects; from Google Brain’s machine learning platform TensorFlow and container orchestration platform Kubernetes to Facebook’s React.

There’s no question that open source software created a new wave of business opportunities. As more companies took an interest in open source projects, they realized they didn’t necessarily have the in-house expertise to manage those projects themselves and turned to startups and larger companies for help.

Even Microsoft, which famously warred against the very concept of Linux for nearly a decade, made a strategic shift to embrace open source in the 2010s, led by CEO Satya Nadella. The IT giant finally joined the Linux Foundation in 2016 and acquired GitHub, the largest host for open source projects, two years later. It has since become one of the biggest sponsors of open source projects.

As a consequence, the stakes have been raised for open source software, which is the engine powering the shift toward the cloud for virtually every company. In this context, security is becoming a topic of the utmost importance, and the commitment to secure the open source ecosystem is growing fast.

Setting a Standard for Security and Trust

Open Source Security Issues

Following the OSS adoption boom, the sustainability, stability and security of these software packages is now a major concern for every company that uses them.

The Census II report on structural and security complexities in the modern-day supply chain “where open source is pervasive but not always understood” revealed two concerning trends that could make FOSS more vulnerable to security breaches. First, the report said it is common to see popular packages published under individual developers’ accounts, raising the issue of security and reliability. Second, it is very common to see outdated versions of open source programs in use, meaning they contain fewer security patches.

The 2021 OSSRA report agrees: “98% of the codebases audited over the past year contain at least one open source component, with open source comprising 75% of the code overall.” The report also noted that 85% of the audited codebases contained components “more than four years out of date”.

This highlights the mounting security risk posed by “unmanaged” open source: “84% of audited codebases containing open source components with known security vulnerabilities, up from 75% the previous year. Similarly, 60% of the codebases contained high-risk vulnerabilities, compared to 49% just 12 months prior.” Not only is the security posture affected, but there are also compliance issues that can arise from unsupervised integration of open source content because licenses can be conflicting or even absent.

Because large corporations are now a big part of the open source ecosystem, their sponsorship is a welcome source of financing for many people whose work had been done for free until now, yet it may not be enough. The open source community is well-known for its commitment to independence, its sense of belonging and its self-sufficiency, and expecting contributors to voluntarily address security issues is unlikely to succeed.

This is where the experience of building Linux over 30 years and coordinating the work of thousands of individual contributors may be an example to follow.

Linux Foundations

In Linux kernel development, security is taken very seriously. Because it is an underlying layer for so many public and private software ‘bricks’ in the digital world means that any mistake can cost millions to businesses, if not lives. Since the beginning, it has adopted a decentralized development approach with a large number of contributors collaborating continuously. Therefore, it has consolidated a strong peer-reviewing process as the community development effort grew and expanded.

The last stable release at the time of writing is 5.14, released on August 29th, 2021, only a few days before the 30th birthday of the project. The most important features in the release are security-related: One is intended to help mitigate processor-level vulnerabilities like Spectre and Meltdown and the other concerns system memory protection, which is a primary attack surface to exploit. Each Linux kernel release sees close to 100 new fixes per week committed by individuals and professionals from the likes of Intel, AMD, IBM, Oracle and Samsung.

With such broad adoption and long history, the Linux project has reached a level of maturity that few, if any, other FOSS projects have seen. The review process and release model have built confidence for numerous downstream vendors. Although the world is not perfect and it is arguably difficult for them to keep up with such a high rate of change, they can at least benefit from strong security enforcement mechanisms and they can adapt their security posture in concordance to their “risk appetite”: Vendors are able to do the calculus of determining how old a kernel they can tolerate exposing users to.

Pushing the Boundaries of Open Source Security

Heartbleed and the Fragility of OS Security

In April 2014, a major security incident affecting the OpenSSL cryptography library; disclosed as “Heartbleed.” The developer who introduced the bug acknowledged that, though he was working on the project with a handful of other engineers:

“I am responsible for the error because I wrote the code and missed the necessary validation by an oversight. Unfortunately, this mistake also slipped through the review process and therefore made its way into the released version.” OpenSSL, an open source project, is widely used to implement the Transport Layer Security (TLS) protocol. In other words, it’s a fundamental piece used to secure a large part of the web.

Open source was seen as fundamentally secure for a long time because the more people examine a line of code, the better the chances of spotting any weakness. Additionally, this model prevents “security by obscurity,” whereby the bulk of the protection comes from people not knowing how the security software works—which can result in the whole edifice tumbling down if that confidential information is released or discovered externally.

This incident was a major turning point for a large share of the biggest web corporations: They realized that many open source technologies underpinning their core operations could not be “assumed to be secure” anymore. Any human error could have huge implications; therefore, a specific effort had to be made to improve the security in this specific space.

A New Era for Open Source

As we advance in an era where open source is omnipresent in codebases, tooling, networks and infrastructure and is even in fields other than software, security awareness is starting to take hold. But it needs a lot more work.

A big part of the challenge, to begin with, is for the industry to understand the scope of the problem.

Google just announced that it will be committing “$100 million to support third-party foundations that manage open source security priorities and help fix vulnerabilities.”

The Secure Open Source (SOS) pilot program, run by the Linux Foundation, will reward developers for enhancing the security of critical open source projects that we all depend on.

In doing so, Google leads the way in enlarging the financial sponsorship of big players like companies and governments — which are increasingly sponsoring open source both directly and indirectly. However, they also recommend that organizations “understand the impact they have on the future of the FOSS ecosystem and follow a few guiding principles.”

What could these principles look like?

A Roadmap to Safely Use and Contribute to Open Source

The Linux Foundation proposed a specific Trust and Security Initiative which describes a collection of eight best practices (with three degrees of maturity) open source teams should use to secure the software they produce as well as by a larger audience to “raise the collective security bar.” Here they are:
- Clarifying the roles and responsibilities, and making sure everyone is aware of their security responsibilities across the organization.
- Setting up a security policy for everyone; in other words, a clear north star for all members of the organization.
- ‘Know your contributors’ is defined as a set of practices to make risk-based decisions on who to trust and fight offensive cyberwarfare techniques, such as the poisoning of upstream code.
- Locking down the software supply chain: This has become a preferred target as adversaries clearly understood that they can have a bigger and more effective impact with less effort than targeting individual systems.
- Provide technical security guidance to narrow potential solutions down to the more appropriate ones in terms of security.
- Deploy security playbooks to define how to do specific security processes, specifically incident response and vulnerability management processes, like creating roles and responsibilities or publishing security policies. This may feel formal, antiquated and old-school but having pre-defined playbooks means that teams can focus on shipping software and not learning how to do security, especially at the least convenient and most stressful time.
- Develop security testing techniques with automated testing strongly recommended since it scales better, has less friction and less cost to the teams and aligns well to modern continuous delivery pipelines.
However, the authors of the guide are aware that some major challenges are still facing the industry and, as such, need to be addressed. They mention:
- The lack of open source security testing tools
- The fact that open source package distribution is broken
- The fact that the CVE format for vulnerability disclosure is also broken
- The lack of a standard for a security build certificate which would allow any consumer to transparently verify that a product or component complies with the announced specifications
“The types of verification can and should include the use of automated security tools like SAST, DAST and SCA, as well as verification of security processes like the presence of security readmes in repos and that security response emails are valid.”

A scheme like this could have a significant and lasting effect on the security quality of open source software and the internet at large.

The Linux project, born 30 years ago, is present in all layers of the modern software stack today. It is used by all the largest server clusters powering the modern web and any business going digital will use it at some point. This unparalleled longevity and success have demonstrated that the open source model was compatible with the requirements of enterprise-grade services and economically viable. Now that open source is all the rage in the software industry, a consensus and action plan on how to ensure the sustainability of this ecosystem becomes urgent. The top priority for businesses that depend on it is to adopt strong application security guidelines, like the ones promoted by the Linux Foundation, which have proven their value.

One last note on the nature of open source: As businesses are now much more bound by the common use of open source components to build upon, they should not fall into the “tragedy of the commons” trap. This would mean waiting until others take action; for instance, to improve the global software security landscape. This might be one of the biggest challenges confronting our highly collaborative industry.
October 28, 2021
Foundation Proposes Advancing eBPF Adoption Across Multiple OSes

The eBPF Foundation, which proposes to advance adoption of an approach that enables sandbox programs to run faster at the kernel level, is being launched today as an arm of the Linux Foundation.

The technology known as eBPF was originally developed for Linux. The eBPF Foundation is now committed to expanding use of eBPF across all operating systems. Members of the eBPF Foundation include Facebook, Google, Isovalent, Microsoft and Netflix.

Thomas Graf, CTO and co-founder of Isovalent, a provider of networking and security tools that leverage eBPF, said IT organizations should expect to see the technology become part of Windows, along with other operating systems, as the core technology is tested and validated across a much wider range of platforms.

In effect, eBPF changes the way operating systems are designed. It bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that, historically, were completely independent of one another. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput to improve overall scale at a time when the volume of cybersecurity attacks being simultaneously launched continues to increase.

Currently, eBPF is widely employed by web-scale companies such as cloud service providers. Facebook is using it as the primary software-defined load balancer in its data centers while Google is using open source Cilium networking software within its managed Kubernetes offerings.

However, adoption within on-premises IT environments that have deployed Linux has been more limited simply because the number of vendors that have optimized networking, security and storage offerings to take advantage of eBPF is still fairly limited.

Sysdig, for example, recently launched made available an instance of the open source Falco container security platform available that employs eBPF. Tigera has also made available an instance of its container networking platform that leverages eBPF at the kernel level of Linux.

Ultimately, Graf notes the biggest benefit of eBPF is efficiency. The total cost of processing for security, networking and storage platforms should decline as more vendors take advantage of a capability that will one day be widely employed across multiple operating systems.

In the meantime, IT organizations would be well-advised to ask their vendors when they plan to support eBPF. Cloud service providers generally require it to deliver their own managed services more efficiently, so any vendor that hopes to sell a platform to those providers will need to support eBPF to drive performance at scale. The issue then becomes determining which platforms that have embraced the technology are also available in on-premises IT environments looking for similar benefits.

Regardless of approach, it’s clear there will soon be a reason to upgrade not just networking, storage and security platforms but also instances of operating systems that don’t yet support eBPF. DevOps teams would be well-advised to plan accordingly because the number of platforms that will ultimately benefit from eBPF spans the entire enterprise.

August 12, 2021
Shoreline Platform Automates Repetitive IT Tasks

Shoreline emerged from stealth today to make available a namesake incident automation platform that eliminates the need to manually perform the same repetitive tasks.

Anurag Gupta, Shoreline CEO, said the platform is based on domain-specific language, dubbed Op, that provides a simple pipe delimited syntax to integrate real-time resources and metrics in a way that allows DevOps teams to invoke an orchestration engine to automate the execution of a task from the Linux command prompt. Op employs a syntax that is familiar to any IT professionals that currently use shell commands and scripting tools to resolve IT issues, said Gupta.

That same Shoreline tool, however, can also be employed to create remediation loops that check for issues, collect diagnostics and apply repairs automatically in the background, he added.

Shoreline achieves that goal by first collecting metrics on a per-second basis via data collected from open source Prometheus monitoring software. Armed with that data, it then becomes possible to distribute queries to both collect data in real-time and run local loops in parallel to automate repairs, Gupta said.

The platform employs the JAX and XLA frameworks for machine learning algorithms to reduce the amount of CPU resources required to compute metrics and create alarms. Wavelets are also employed to compress the input metric stream. Finally, queries are portioned across historical data stored in the S3 cloud storage service from Amazon Web Services (AWS) and real-time data collected from each node.

All Shoreline components are designed to be fault-tolerant, added Gupta. Automation can still run locally to Shoreline backends, even when connectivity is lost. All processes have supervisors to restart when failing and reestablish connectivity.

Finally, Shoreline provides controls to limit the scope of a manual operator action and automate executions in a given time period to help prevent any errors from cascading through an entire IT environment. That approach makes it possible to safely automate tasks in a way that makes it less likely outages will occur because, for example, a misconfiguration winds up being propagated across every system, noted Gupta.

While IT environments are arguably more complex than ever, Gupta said it’s the repetitive tasks that take up so much of developers’ time. In the absence of any way to automate those tasks, an IT team will, over time, become increasingly burned out, which Gupta noted inevitably leads to mistakes being made.

At the same time, Gupta noted the Shoreline platform makes it simpler to onboard new members of an IT team because much of the tribal knowledge that once had to be learned on the job has been captured within an automation platform.

The arguments for automating IT have, of course, been growing louder and more frequent for nearly two decades. Most platforms that promise to automate IT have required a fair amount of expertise to implement and maintain. The Shoreline approach, in contrast, starts with a Linux prompt that many IT teams already routinely use to execute a range of tasks. The difference now is simply the level of scale at which those tasks can be performed with a minimum of programming skills.

July 27, 2021
Rocky Linux Emerges as a CentOS Replacement

Rocky Linux was created to offer the community an alternative following Red Hat’s recent decision to shift its focus away from CentOS—the open source version of Red Hat Enterprise Linux (RHEL). However, the Linux distro’s popularly can also be attributed to how its creator sought to offer a superior alternative based on lessons learned during CentOS’ development—and less as a perceived backlash to Red Hat’s controversial move, said Gregory Kurtzer, Rocky Linux’s creator, who is also founder and CEO of Ctrl IQ.

“As one can probably imagine, there were a lot of lessons learned from the founding of CentOS—we did a lot of things correctly, but we made some mistakes, as well, that set into motion some of the difficulties the project went through,” said Kurtzer. “I was not—nor am I now—upset at Red Hat or anyone else. It’s important to understand that companies have a legal responsibility to do what they need to do. But I will say, this is why open source communities should not be controlled by a single company, especially if that company has a competing product to the open source project.”

In addition to applying experience gained from the CentOS project to help “recreate the initial goals of CentOS in a new project,” Kurtzer said he was prompted to create Rocky Linux following Red Hat’s decision to end-of-life (EOL) CentOS. The move “affected my company, our customers, as well as the entire industry,” said Kurtzer.

“To be blunt, my knowledge—in terms of doing it myself is severely dated. Luckily, after I made the announcement that I was considering recreating CentOS, thousands of people reached out to me wanting to be part of the project and contribute,” said Kurtzer. “From that, our development team and open source community was born.”

Additionally, CentOS Stream offers value to the community by allowing “us and others” to take part in developing the upstream Enterprise Linux standard, said Kurtzer, who added that his team will continue to debug Rocky Linux so that solutions are properly integrated into the Enterprise Linux Standard.

“The necessity was absolutely clear, so I founded Rocky Linux and the Rocky Enterprise Software Foundation (RESF),” Kurtzer explained. “Together with our contributors, sponsors and partners, we all have united for a common goal: to create an open source community that will provide us all with an enterprise operating system to stand the test of time, attrition and commercial agendas.”

Interest in the development of a CentOS alternative was strong when Kurtzer first floated the idea of the project to the community. Since its release in June, the total number of ISO downloads is 56,000, according to Rocky Linux Tier0 mirror data.

“We have zero visibility as to any other ISO downloads from any of our other mirrors, but if I were to guess, I’d say it would be about two to three times more, in total,” said Kurtzer. “When we all get a moment to breathe, it will be fun to see how many people are using Rocky Linux, and take in all the feedback the community has provided,” he said.

With this team of developers, Kurtzer said it took about seven months to build the general availability (GA) release. “We didn’t just jump right into building the operating system because the goal was to create an infrastructure where many contributors and developers can take part without putting the security of the product at risk. Building that infrastructure took about four months,” Kurtzer said. “Once the infrastructure was completed, we spent about two months building the first-release beta candidate and then spent the following month ensuring that we did everything correctly.”

July 13, 2021