Author: Amiram Shachar

  • Leveraging eBPF for DevSecOps

    Leveraging eBPF for DevSecOps

    eBPF is a revolutionary technology originating from the Linux kernel. It is used to safely and efficiently extend the kernel’s capabilities without requiring changing the kernel source code or loading kernel modules/extensions.

    Today, eBPF is used extensively to:

    • Provide high-performance networking and security in modern data centers and cloud-native environments
    • Extract fine-grained security observability data at low overhead
    • Provide insights for performance troubleshooting
    • Provide preventive application and container runtime security enforcement

    eBPF-Enriched Context

    eBPF is the base data layer that is needed in runtime cloud security. eBPF gives you information at a high level on Layers 3, 4 and 7, allowing you to see all network communication and correlate it with processes and specific applications that are communicating on those ports.

    To fully leverage eBPF, you need to pair this raw data with additional cloud infrastructure context to gain a full understanding of which resources and services are in use.

    For example, by leveraging eBPF, you can determine if you have 10 different applications all communicating on port 443, whereas a traditional network monitoring tool would tell you that you have a certain amount of traffic on this port but wouldn’t be able to segment this traffic and show how much of that traffic is coming from each individual application.

    Knowing which communication is coming from an individual application increases visibility. It allows you to see what your applications are doing in real-time and identify signs of compromise or decreased performance.

    Increasing Observability With eBPF

    eBPF provides users with intrinsic, comprehensive insight into communication flows across the entire system. The applications for enhancing security observability with eBPF are vast, ranging from enabling highly detailed process visibility to facilitating end-to-end observation of processes and flows.

    When dealing with Layer 4 to Layer 7 traffic involving multiple protocols, eBPF emerges as an optimal solution. Through eBPF, TLS traffic traversing pods, nodes and cloud platforms can now be uniformly instrumented across layers.

    In other words, with eBPF, you can observe your application and your node and gain a vast amount of raw data, which, with the right tools, you can correlate to understand your entire network topology.

    Beyond eBPF Raw Data to Context and Correlation: Layer 7 and Beyond

    Specifically for observability, you can use eBPF to apply filters directly in the kernel, which reduces the overhead and allows you to ingest significantly more raw data. eBPF’s architecture allows you to do all of this, observe network traffic and Layers 3, 4 and 7 without compromising your security or performance.

    With eBPF, you are able to understand the execution events, system call events and all of the network ingress and egress. But it’s not always enough.

    You will need to correlate it with Cloud APIs, Kubernetes identities, pods and namespaces to understand the actual identities and relationships between the individual workloads, all the way down to the application layer. By doing so, you are able to understand application-level identity and use this context to understand your application behavior as well as your most critical risks and threats.

    Leveraging eBPF for Threat Detection and Response

    Security and networking observability are inherently interconnected. eBPF helps you to gain insights such as abrupt shifts in behaviors, spikes in activity within specific processes indicating potential exploitation or attacks, service-to-service communication trends and more. These projects enable users to examine network transactions, identifying the pods, processes and system calls involved.

    With eBPF, you can see OS-level runtime threats. eBPF gives you the ability to listen to system calls but also to go one step further and block or terminate any process, PID (process identifier), system call or network packet. This allows for a more granular approach to automated threat detection and response without damaging your system or hurting your infrastructure.

    You can also use eBPF to implement prevention rules, meaning you can instruct eBPF to automatically block any malicious processes or pre-designated system calls or network packets, going beyond traditional uses of eBPF for monitoring and security, to become a more proactive, preventative tool for securing your infrastructure and applications.

    Real-World Benefits of eBPF

    eBPF presents a number of real-world benefits, going beyond traditional monitoring, observability and security to help with overall alert noise reduction and improved forensic analysis.

    How eBPF Contributes to Noise Reduction

    By itself, eBPF does not reduce noise. In fact, it increases because of the influx of raw data from every system call. However, with the right tools, you can leverage this raw data and correlate it with additional context to understand your infrastructure and application behavior. This effectively reduces the alert noise that you would otherwise receive because you can view more detailed insights into resource behavior.

    By filtering and aggregating this data, you receive more insights about applications, overall giving you less noise because you’re focusing on what actually matters.

    By having this eBPF raw data at runtime, you can apply intelligent filters to filter through the most important data and use it accordingly, resulting in better prioritization, more focused efforts and increased operational efficiency.

    Forensic Analysis

    Another important real-world use case for eBPF is conducting forensic analysis. By leveraging eBPF’s raw runtime data and applying intelligent filters, you can quickly pinpoint problem origins and conduct forensic analysis at the application level.

    By leveraging eBPF’s raw data from Layer 7, you gain a granular understanding of applications and the process-level communication within an application, allowing you to foresee the potential blast radius if an application were compromised in terms of who it could interact with and what damage could occur.

    Conclusion

    In summary, eBPF is a powerful technology that can be effectively leveraged for monitoring, observability and security purposes. However, when eBPF’s raw data is combined with context from cloud APIs, it can be leveraged even further with intelligent filters to provide unparalleled cloud infrastructure and application context, allowing you to understand application-layer identity, provide real-time security, pinpoint problem origins and stop threats with surgical, active response.


    To hear more about cloud-native topics, join the Cloud Native Computing Foundation, Techstrong Group and the entire cloud-native community in Paris, France at KubeCon+CloudNativeCon EU 2024 – March 19-22, 2024.

  • Spot Instances on Black Friday: A Case Study

    Spot Instances on Black Friday: A Case Study

    Businesses made the right choice using spot instances for Black Friday

    Black Friday is the unofficial kickoff to the holiday shopping season, when retailers announce exciting deals to attract as many customers as they can into their stores. With online shopping surging in recent years, e-commerce and AdTech businesses are also participating in Black Friday to increase their online presence and revenues. Today, millions of customers have the choice of shopping online or in brick-and-mortar stores. While these stores have their challenges managing massive amounts of shoppers, e-commerce and AdTech businesses are dealing with their own set of challenges when it comes to handling shopper traffic.

    Source: Business Insider

    During peak shopping days such as Black Friday, businesses use significantly more capacity to process online shopper requests while constantly trying to provide them optimal performance. In theory, as more companies utilize resources from a cloud provider, they may experience a capacity shortage when trying to scale. When a busy online shopping day finally approaches, this theory can become a reality as online retailers scale to meet their shoppers’ demands and cloud providers struggle to accommodate capacity. However, what many cloud users may not know is that about 50 percent of the major cloud vendors’ infrastructure is mostly idle.

    To be clear, this is not because AWS, Microsoft or Google are struggling to sell their capacity; it’s because they have to be ready for rapid increases in usage—just like the one businesses experience during Black Friday.

    The fear of insufficient capacity also has made users apprehensive to use cloud providers’ excess capacity such as spot instances to run their workloads, since many customers are using more cloud resources and are eating all of the cloud inventory. Businesses simply do not want to take a chance trying to compete for excess capacity and opt for something more predictable, such as on-demand instances. The fact that cloud providers are capable of handling the most aggressive computing demands has created a situation in which businesses that chose to make intelligent use of excess capacity during Black Friday saw great returns on their investments.

    Events such as Black Friday, national elections and the Super Bowl are pretty extreme stress tests for any consumer-facing business. We thought it would interesting to go through the data and see how the AWS spot market behaved during Black Friday, and see what we can learn from it. In this blog post, I will go over the spot market statistics and lessons learned from 2018’s Black Friday.

    Spot Markets and Statistics

    Spot markets are defined as the supply and demand for a specific instance type at a specific spot price in an availability zone. Spotinst Elastigroup ranks spot markets for availability and costs from current and historical data in a process called Spot Market Scoring. Spot Market Scoring enables us to predict an instance interruption in a market approximately 15 minutes before it happens (sometimes even more than an hour). This advance notification on upcoming interruptions is what triggers the spot instance replacement and ensures the continuous availability of compute workloads. Let’s see how the spot markets behaved before and after Black Friday.

    Now that you have a better understanding of the spot market, let’s delve into the statistics from Black Friday. In the image below, we can see the number of spot interruptions for each Friday three months prior to Black Friday in the us-west-1 and us-east-1 region. The us-east-1 region had significantly more spot interruptions during this period.

    The graph below shows the number of spot interruptions that occurred a week before and after Black Friday in us-east-1 and us-west-1 across all instance types. We can see that us-west-1 was more reliable for spot instances than us-east-1. Also, us-east-1 continued to experience more outages the week after Black Friday.

    We then took a look at popular instance types and sizes used before, on and after Black Friday to see how many interruptions there were. In the image below, we can see that c4.4xlarge had the highest number of interruptions and incrementally improved after.

    The section below shows metrics for Back Friday in comparison across different availability zones in us-east-1 and us-west-2:

    1. Total Spot Interruptions this Black Friday in the US: 462
    2. Breaking down the Interruptions by Availability Zones, some had more Spot interruptions than others
    Availability Zone us-east-1 us-west-2
    AZ 1 29 1
    AZ 2 3 7
    AZ 3 24 3
    1. Average Spot Uptime: 2675 Minutes (1.8 days)

    What Spot Interruptions and Autoscaling Events Look like

    Let’s examine what a spot market interruption looks like and how Elastigroup handles the situation. In the data below, you can see an actual event log from one of our customers.We can see that early on Black Friday, the day started with Elastigroup automatically replacing terminated or soon-to-be-terminated instances. A little later, as shoppers began scavenging the internet for deals, we see that the instances were automatically scaled up to meet demand. Elastigroup scaled down the instances the following day because demand was not as high as on Black Friday. With this type of autoscaling automation in Elastigroup, administrators can spend more time worrying about application development and other tasks.

    Takeaways

    1. DevOps Automation is Key – Elastigroup uses predictive algorithms to identify and drain instances that are about to be terminated. Prior to termination, Elastigroup will launch a new instance and replace it seamlessly. Elastigroup will also make sure to distribute your instances across different types to optimize costs and longevity.
    2. Spot Instance Availability – Having a stable uptime is crucial for peak days such as Black Friday. With millions of shoppers visiting e-commerce sites, an outage can result in revenue loss. On Black Friday, the small number of spot instances that experienced interruptions had an average uptime of 1.8 days. With that uptime, organizations can rely on spot instances managed by Elastigroup to reduce costs while not sacrificing service availability during peak times.
    3. Spot Market – Spots allow businesses to make better decisions without sacrificing costs and availability. Elastigroup reliably leverages the spot market to optimize your underlying infrastructure for cost, without compromising availability. Per the statistics above, we can see that some availability zones had more spot terminations than others. AZ 1 in us-east-1 had the highest number of spot interruptions.

    — Amiram Shachar