devopsy.kinsta.cloud

Tag: kernel

  • Latest eBPF Advances Are Harbingers of Major Changes to IT

    Latest eBPF Advances Are Harbingers of Major Changes to IT

    The open source community advancing the extended Berkeley Packet Filter (eBPF) gathered at a virtual eBPF Summit today that featured a demonstration of eBPF being used in a Windows environment. eBPF enables applications to run within a sandbox in the Linux microkernel.

    Microsoft has signaled its intent to employ eBPF within Windows like the latest version of Linux already does today. That capability makes it possible to, for example, run networking, security, observability and storage software closer to the kernel to dramatically improve performance versus running that software in user space.

    Liz Rice, chief open source officer for Isovalent, a provider of networking software based on eBPF, said that while it’s clear eBPF will have a major impact on IT, the transition to eBPF won’t be a single event. Instead, it will be more of an evolutionary process that will occur as organizations upgrade their operating systems and the various tools and applications they employ.

    The eBPF Foundation, an arm of the Linux Foundation, was formed to expand the use of eBPF across all operating systems. Members of the eBPF Foundation include Facebook, Google, Isovalent, Microsoft and Netflix.

    In effect, eBPF changes the way operating systems are designed. It bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that were historically completely independent. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput at a time when brute-force cyberattacks are overwhelming existing cybersecurity tools.

    Similarly, networking, storage and observability tools should be able to run at much higher levels of scale. That’s especially critical for DevOps teams embracing observability platforms that need to dynamically process massive amounts of data in near-real-time.

    Less clear is the degree to which eBPF might one day drive convergence across distinct categories once multiple tools and applications run at the microkernel level.

    IT organizations would be well-advised to ask their vendors when they plan to support eBPF as they plan their upgrade cycles. It’s not likely IT teams are going to want to upgrade every tool and application that can benefit from eBPF all at once. In fact, updates to these applications that don’t support eBPF might even be postponed in the knowledge that a much larger and arguably more critical update is on the immediate horizon.

    In the meantime, as the number of organizations running the latest versions of Linux continues to increase, more hands-on experience with eBPF will be gained. IT teams may not need to concern themselves with what is occurring in the microkernel of the operating systems, but they should at least understand how eBPF ultimately helps reduce the total cost of running IT and how it makes the Linux microkernel more programmable. Those IT teams, however, will soon discover how stark a difference in performance there will be.

  • Libbpf Vs. BCC for BPF Development

    Libbpf Vs. BCC for BPF Development

    If you’re into Linux development, you’ve probably heard BPF mentioned over the last few years. BPF stands for Berkeley Packet Filter, and the technology has a large number of use cases. BPF apps can get deep access into an operating system and enable you to perform tasks such as high-performance load balancing, DDoS mitigation and more simply and easily.

    Libbpf and BCC-Tools are both sets of tools to help with BPF development. However, both of these have their own strengths and weaknesses. In this guide, you will learn in detail about the two tool collections and understand when to use them.

    What is BCC?

    BCC stands for BPF Compiler Collection and is one of the oldest ways to develop BPF applications. It helps you embed your BPF code into your user-space program in the form of a plain string. When the user-space program is executed by the kernel, BCC invokes its embedded Clang/LLVM, pulls in system-wide kernel headers and compiles the program on the spot. 

    Since BCC compiles BPF programs on the host machine, it ensures that the memory layout your BPF program expects is precisely the same as that of the target host.

    BPF programs are designed to be injected directly into a kernel, so BCC tools seem to be the perfect solution for developing BPF applications. However, they have proven to be bulky in the modern context due to their heavy reliance on the Clang/LLVM combination, which is resource-intensive. This has led to the need for a better, more modern solution to developing BPF apps.

    What is Libbpf?

    Libbpf is one of the hottest new BPF tools on the market. It is usually coupled with BPF CO-RE (which stands for compile once, run everywhere). BPF CO-RE enables you to generate binaries that run on multiple kernel versions.

    The idea behind Libbpf is to make BPF development as similar to other forms of development as possible. With BPF CO-RE, Libbpf does this by compiling BPF programs into small binaries that can be deployed to multiple deployment hosts. Libbpf does the setup work like loading and verifying programs, creating maps, attaching to hooks, etc., which enables developers to focus on more critical tasks at hand, such as program performance and correctness.

    Libbpf aims to eliminate the overheads associated with BPF app development and deployment by reducing the dependency on system-wide kernel headers as well as Clang/LLVM libraries for compilation on runtime.

    Libbpf Vs. BCC: Key Takeaways

    Now that you understand how each of these tools works, let’s compare some key benchmarks to learn when to use them.

    Dependency Management

    BCC relies on kernel header packages that need to be installed on the target host machines. While it is not a problem in most cases, it can get difficult to set up and maintain if you are working on multiple machines. 

    Libbpf, on the other hand, eliminates this issue by relying on a vmheader file. This file includes multiple kernel types which helps you to remove dependency from system-wide kernel headers.

    Compilation Style

    For programs written with BCC, compilation occurs at runtime. To facilitate this, such programs require Clang/LLVM be present on the host machine. This adds to the code footprint of your programs.

    Furthermore, the Clang/LLVM libraries are resource-intensive. Complete libraries need to be available and run at compile time, even for small programs. This can upset a perfectly balanced BPF workload on the host machine.

    On the other hand, Libbpf enables you to generate binaries that are compiled once and can be run anywhere. You do not need any system-wide dependencies to be present on the target machine for running such apps. Hence, it reduces the overall application size as well as resource consumption on runtime.

    Error Detection

    For apps developed with BCC tools, errors are detected only at runtime since that is when the programs are compiled. This leads to a rather sluggish development experience compared to apps developed using Libbpf. Since Libbpf enables compilation while developing, finding and fixing bugs is easier.

    Final Thoughts

    In this guide, we walked you through what Libbpf and BCC tools are and compared them based on some important aspects of application development and delivery. 

    All in all, Libbpf appears to best BCC tools in most aspects. BCC tools offer fast prototyping and experimentation possibilities, but when it comes to production deployments, BCC tools turn out to be a rather expensive choice compared to Libbpf. 

    You can learn more about why BCC became the popular alternative and how it was eventually replaced by Libbpf here. Even if you currently use BCC tools for your apps, you can easily migrate to Libbpf by following a few simple steps. And, in most cases, it is better to opt for Libbpf than BCC tools for a better development experience and application performance.

  • Why the Linux Kernel is Still Important

    Why the Linux Kernel is Still Important

    Last year, the Linux kernel turned 30. If you’re someone who’s been immersed in the Linux world since Y2K like I am, it may feel a bit surreal that so much time has passed since the kernel’s inception.

    As a training architect at A Cloud Guru (ACG), I teach courses about all things Linux and specialize in hands-on lab-based learning. Before joining ACG, I worked as a Unix systems engineer at GE and IBM as well as a technical account manager and customer advocate for Red Hat. I’m hugely passionate about Linux, just as so many other engineers and enthusiasts are, not just because of its importance to our careers but its impact overall.

    In its 30 years of existence, the Linux kernel has had a massive impact on the modern computing landscape—revolutionizing what’s possible for operating systems and allowing countless tinkerers to get their hands dirty in the process. Linux has also become the foundation of paradigm-shifting innovations over the years due to its ever-evolving nature.

    Here are some of the key reasons why the kernel is so valuable and why it will continue to be a major player in the computing landscape for years to come.

    My Own Linux Journey

    My first exposure to Linux was in college in 1999. I took an introductory Unix Shell Scripting class and Fedora was installed on the lab servers because it was free and easily scalable. I view Linux as my entry point into customizable operating systems. After learning basic Unix commands on Linux, I ultimately went on to pursue a career supporting multiple commercial Unix vendors.

    As Linux made its way into larger enterprise-level companies, I quickly returned to working with Linux distributions. Something I’ve always loved about Linux is the capacity to fine-tune your system to support the applications and distros that work best for your projects. Linux improves the functionality of whatever applications you’re running. The Linux kernel has literally changed how the world processes information, which is why I’m so invested in the software.

    Now, it’s my job to share my knowledge and passion for Linux with other technologists. At ACG, I develop courses to help aspiring Linux experts learn how to optimize their systems. Additionally, I contribute to Linux this Month, an ACG-hosted web series that provides monthly updates from the global Linux community.

    The fact that I am able to build a career around teaching Linux and staying up-to-date with Linux news shows the vast use cases and applications of the kernel. What makes Linux so unique and evergreen is its open source nature—Linux innovations are only limited by the creativity of the technologists who use and adapt it.

    The Open Source Effect

    When Linux first arrived, it was mostly a hobby for enthusiastic engineers and computer science students who could contribute by developing code. The steep learning curve associated with fitting Linux to your machine was a barrier to entry for more novice programmers.

    Over time, this has changed considerably. Online forums, workshops and classes have made Linux more accessible to the average internet user. The free sharing of ideas has come to epitomize the open-source community, and for software engineers, Linux is at the heart of this community. This democratization of Linux has had incredibly positive impacts on the computing world.

    Now, Linux is everywhere. Enterprise-level companies use Linux distributions to process the biggest production workloads in the world. It has replaced proprietary commercial Unix operating systems in very large companies with better stability and less downtime. Because Linux systems can be as small or as large as you want, it’s also now being used in our homes for smart and mobile devices as well.

    The open source nature of Linux is incredibly beneficial for these enterprise-level companies. Linux distributors leverage the contributions of the entire open source community. This wide range of contributors produces a more stable product with more features but also ensures that the OS keeps growing and solving real-world problems that are beneficial to a wide range of users.

    Linux Kernel Runs the Cloud

    Potentially the most impactful outcome of the kernel is its importance to the infrastructure of modern cloud computing. Linux’s scalability has paved the way for supercomputers and server farms to function efficiently while requiring relatively lightweight computing resources. In fact, Linux supports about 90% of the public cloud workload.

    Without Linux, the cloud as we know it would not exist. This is, in part, because Linux has become so ubiquitous—its use cases are nearly limitless. Because it has been time-tested, many engineers and IT professionals have a solid grasp of Linux fundamentals, making it an attractive choice for enterprise companies dealing with the cloud.

    Cloud-based software and products are increasingly becoming the norm in the engineering world. Unsurprisingly, major cloud providers such as AWS, Azure and Google Cloud are all supported by Linux as well. Linux is unique because it is a shape-shifter that can conform to the needs of any given engineering environment, and it’s incredibly stable because of the army of contributors that fortify weak points in the software.

    Linux Forever

    The concept of Linux is just as important as the Linux product. The concept allows a free and open source operating system to be refined, reinforced and replicated across an endless web of contributors. Thirty years is a long time for software to be relevant, especially with the ever-shortening tech product cycles. Because it was designed with the intent to be changed and updated by an open source community, Linux has no foreseeable expiration date.

    Had Linux not achieved the prominence it has today, we would see more commercial Unix vendors attempting to solve some of the problems that Linux addresses, but none would address them all. Additionally, customers would have to choose which OS to invest in based on which addresses some of their use cases, but none would be as beneficial as the Linux OS.

    Linux is always growing and will become even more popular within the next few years. As more people become familiar with Linux and learn to use it, I see major potential for growth in the mobile computing space, within personal computers, and across small and large companies. In fact, we are already seeing it filter into home gaming systems and Raspberry Pi projects. With Linux, the sky’s the limit!