devopsy.kinsta.cloud

Tag: K8s

Want to Stop Kernel Panics? Use eBPF

Writing kernel-level code is hard. It is one of those pieces of software that brings the size-to-complexity ratio to its absolute max, with a relatively compact codebase being the foundation for running any other software on every type of platform. And it is not just understanding how the kernel works which is hard; it is what happens when the code doesn’t work. There isn’t a single kernel developer in the world who hasn’t crashed their system on several occasions during development. And like in any software being written, no matter how good your testing process is, bugs will sometimes slip into production. If you are wondering how bad a kernel bug could be — July 19, 2024 provided a painful example.

The Appeal of Writing Kernel-Level Code

The July 19 IT outage happened because of a kernel bug in Windows, originating in a faulty update by a popular security vendor. You might ask yourself, how is it possible that a security vendor, which is not directly responsible for writing the Windows kernel, has created a code that crashes it? Why are they writing kernel code to begin with, and why hasn’t Microsoft prevented this in some manner?

The short answer is — yes, writing kernel code is hard; yes, you shouldn’t do it in 99.9% of the cases. But there are a lot of things you can only do from the kernel level. Security agents are a good example — high privileges are needed to detect threats and prevent them as they manifest. Observability and networking tools are another example, where routing traffic or observing process memory is not available from outside the kernel.

And so, companies turned to writing kernel modules, aka drivers. These are custom pieces of software that extend the kernel, allowing vendors to insert their own logic into the system, which executes as one with the operating system. And that’s where the trouble begins, as Microsoft — or any other vendor for that matter — can’t be responsible for testing third-party code. Or can they?

Rethinking the Ecosystem

The root issue with kernel modules is the fact that they are given the same amount of trust as the actual kernel, even though they are a non-critical extension of the operating system. Could there be a way to provide all the benefits of extending the kernel, while assuring safety for critical systems?

Enter eBPF — a game changer in writing kernel-level code. eBPF is a way to allow developers to write code that will execute in the kernel in a controlled manner; it has all the benefits of kernel modules, without the risks. It is getting widely adopted by security, networking and observability practitioners.

eBPF is so groundbreaking that it is not even just about kernel modules. User-level tools such as network sniffers or APM libraries are also beginning to be outperformed by eBPF equivalents. It is one of those things that make you question years of previous technologies and solutions. It is mind-blowing.

Are Kernel Panics a Thing of the Past?

eBPF is revolutionary, but it will not solve every kernel-level problem. For example, some security tools require certain actions that the eBPF sandbox can’t provide. And this means we will still be seeing a lot of unsafe kernel code in the foreseeable future. But the word is out there, and every developer writing kernel code should proactively make this change. The landscape is changing and it will be safer and more efficient. Trust me on this one.

August 15, 2024
Are We Delivering?

I had an interesting conversation today where I was told, “We do great with CI but terrible at the CD part.” My immediate reaction was to point at all the great CD activity going on in vendor space right now. But as I stopped to ponder, I realized that we’re doing all of that great CD activity to play catch-up.

DevSecOps has us covered in spades—the toolsets allow for scanning and protection all through the process, from the first line of code to application retirement. I would argue that after being left out in the cold, DevSecOps has done a better job than DevOps at the right side of the equation.

In DevOps, the various “cloud-native” deployment models (I use scare quotes because they’re not really cloud-native. If you dig into them, nearly every one of them includes Kubernetes in their definition of cloud) are starting to do the job admirably, but in shops starting to automate the entirety of the process, the CI part has been automated and code delivery accelerated for a good long while.

In short, adoption seems to prove, once again, that the benefits of rapid development are more important than the benefits of offering a stable, repeatable deployment environment. This makes one wonder what, exactly, we should be doing to convince IT and business leaders in orgs smaller than web monsters that security and repeatability both demand the implementation of capable continuous delivery.

I think we failed the audience when we allowed pundits to blather on and on about deploying thousands of times a second when most applications in most orgs need to deliver at a significantly slower pace. And not only that, but want to deliver at that slower pace, given the “Don’t fix what isn’t broken” rule. Even if the environment is stable, rolling out for a single minor bug fix is possible but it’s not the best use of time—both computer and human.

But we are catching up. We have things like GitOps and Kubernetes automation catching up with the rate of development; security is already there and test automation is ready to find issues before deployment. It is simply implementation that is behind and, as time goes on, more orgs will get the implementation part nailed.

And through that process, you’ll be there, rocking it. Step up and make sure it is done right. The entire process is there; you could sleep better at night knowing that it is all in place and you are not likely to get a midnight call. Your org loves stability, you love good nights of sleep … Go out and get it.

January 11, 2023
The Right Tool for the Job: Container Edition

I have written about “the right tool for the job” repeatedly. Because (again) we are absolutists in IT. My father was an antiques restorer that dabbled in flipping houses and was rather forceful about the “right tool for the job.” When I left the service and joined IT, it was something of a shock to me that the “right tool for the job” was considered situational.

At the time, there were a large number of Microsoft zealots that loved Visual Basic, and a larger number of non-zealots that preferred C or Java. That was how we got to my first “right tool” article. VB was very strong with regard to UI and record crunching, while to-the-metal was C’s strength and Java fell in between the two—not being perfect at either. So my article stated that. Front-ending databases? Use VB. Writing drivers and low-level processes? Use C/C++. Need a bit of both and easy web page generation? Go with Java.

It is many years later and time for an article that seems just as obvious to me, but in a different IT space. We have all heard of the three architectures: Data center, cloud and mixed/multi-cloud. We have also heard of how, in this rapidly changing world of DevOps, Agile and containers, different organizations are using different deployment methodologies. That’s all good so far. Just as the variety of programming languages available today is a bonus in many ways, so are multiple deployment and operations options.

Where it gets sticky is when people include the “We’re an X shop” in conversations. Or try to do everything under the sun with one of these methodologies. For many of us, container infrastructure is perfect. Use an orchestrator like Kubernetes, build systems and security practices, then not have to care much if the Kubernetes instance it runs on is on-premises, hosted or cloud. Of course you have to care some. Networks have to connect, etc. But most of the work is done.

Then there are the people that have use cases that don’t fit this mold and have not acknowledged that fact yet. This blog is for you. We’ve all heard of companies that have containers whose life expectancy is measured in seconds. Beyond that, I was told about a company that is using containers as message queues. Honestly. Creating a container with a startup command and putting it on a queue. When it is pulled off the queue, it executes that command and then goes away. We’ll call this crowd the “Over Users”.

Finally, we have the people that cannot or will not be moving their systems anywhere. There are a variety of reasons for this, but these are companies that know their applications will continue to run in the data center for the foreseeable future. We’ll call this group the “No Needs”.

For Over Users, that is literally what serverless is for. Create a queue with a (much smaller) call to an API in a serverless environment and be done. Fewer configuration issues, less maintenance, less everything but code. Add a trigger, remove entire OS/network/configuration generation and maintenance. The most likely resistance to this advice is “But we have an investment…” Yes. And for most orgs it is an investment in unnecessary technical debt. Divest. Keep the good parts—a container architecture meant to shuffle through many containers that exist for a short time is well-architected.

For No Needs, before you port everything to containers—which I am generally in total support of—you should consider if you need to; think about if it offers any benefits and what the ROI of such a move is. Honestly. For DevOps-heavy shops or orgs that require highly portable solutions, the ROI of containers is quick and obvious. For orgs whose applications are not changing a ton and are highly unlikely to move anywhere, the ROI is a whole lot murkier. “But it gives us options!” is the most likely negative response to “We don’t need that right now.” But that’s the problem—you have those options now. What it gives you is a shifting of cost from some possible future where it might be needed to today’s already constrained budget. If there is a verifiable need—normally in portability, but any verifiable need—then make the move. Until then, spend that time and money making those apps rock.

You’re doing it. The lights are on and the applications are useful. Use the architecture that makes sense for what you are trying to achieve. Right tool for the job, as my Dad often said (he said it a bit more forcefully when I thought a chisel was a screwdriver, but that’s a different story). Keep rocking it, and stay focused on need.

September 14, 2022
Catch Emerging Security Risks Earlier by Leveraging Kubernetes Audit Logs
When a Kubernetes deployment suffers a security breach, it can be difficult for security teams to diagnose and identify the source of the intrusion and its impact. Each Kubernetes cluster may host multiple applications, and each application may be composed of dozens or even hundreds of interacting microservices deployed as pods.

The complex and dynamic communication between the applications’ components, as well as the automated Kubernetes orchestration activities, makes it difficult to understand the structure of each application and differentiate between normal behavior and malicious behavior from the observed network activity. The ephemeral nature of pods, which may exist only for moments to perform an activity before they are disposed of, makes this task even more difficult. A pod that was vulnerable may be gone by the time security teams are investigating a breach, or resume normal operation as the attacker pauses its abuse or pivots or escalates the attack to another Kubernetes resource.

K8s is by nature dynamic, flexible and scalable, and it is this same advantage for cloud native application developers that becomes a challenge for security professionals. How can one understand the source of a breach, let alone how it propagated through the system, without deep understanding of the architecture of the application?

One solution to the problem of tracing breaches is to examine the Kubernetes audit logs since they record all cluster-administration activity, even for pods that may have existed for seconds and are now gone. But K8s audit logs are complex and often convoluted, requiring deep expertise to understand. Therefore, security teams need to not only increase their proficiency with K8s audit logs, but also to apply automation and software tools to the K8s logs to detect suspicious behavior after the fact and even during their occurrence.

In fact, there are many potential threats that can be discovered when observing audit logs for suspicious behavior. They include:
- Stolen or misused credentials, enabling hackers to gain access to K8s-based clusters or pods.
- Misconfigured Rules Based Access Control (RBAC), enabling lateral attack propagation, privilege escalation and unauthorized data access or manipulation.
- Exploitation of vulnerabilities in the Kubernetes API Server, enabling bypassing of authentication, authorization, admission control or validation of cluster administration requests. This lets users gain access to privileged and sensitive resources.
- Violations of security policies diverge from compliance requirements and best practices.
Detect Breaches in Near Real Time by Watching K8s Audit Logs

There may be a vast time span between an initial breach and the detection of the compromised application or the unauthorized access of secure data and private information. Security teams must seek to close the gap between a breach attempt and its detection as much as possible, while still being able to diagnose the causes of a breach after it has been detected and assess its scope and impact.

Since K8s deployments can be vast and intricate, machine learning and artificial intelligence are preferred tools to monitor audit logs and flag suspicious behavior. An automated tool could continuously scan logs to learn what normal behavior looks like in the company’s clusters, and then flag network behavior that is unusual or suspicious for security teams to qualify, prioritize and investigate. This methodology would enable security teams to investigate possible security violations in near real time to protect and secure the applications.

In addition to detecting security risks, security tools that leverage K8s audit logs should provide reporting that is easy to understand and consume by security teams. By providing the correct user experience, companies can make their K8s logs accessible to more than the hard-core experts, effectively opening a new vista of security possibilities for more security professionals. The difficult learning curve of Kubernetes will no longer be a wall between K8s pros and newcomers with less experience, relieving pressure from understaffed security teams.

Step Up to Security and Compliance Needs

As Kubernetes moves toward the mainstream, with roughly 71% of Fortune 100 companies using Kubernetes as their main container orchestration tool, companies must also look toward compliance with standards and government regulations designed to protect financial transactions and private or personal information such as PCI, GDPR and HIPPA. The penalties for violating these standards and regulations can be severe, so companies should proactively adopt K8s audit log monitoring as a way to identify breaches earlier, to limit their impact on customers and data, and to document the behavior of the application and would-be hackers for possible legal inquiries. An effective audit log strategy will be vital in determining what happened in the case of a breach.

Leverage Audit Logs for Real-Time Security Observability in K8s

Many tools exist to secure the perimeter of the data center and to identify possible security risks during application development. The benefits of analyzing and leveraging Kubernetes audit logs at runtime have been relatively overlooked. K8s audit logs are a prime way to comprehend the behavior of any cloud-native application orchestrated by Kubernetes, as well as the best way to comprehend how the Kubernetes infrastructure itself can be abused by an attacker to compromise the applications it is hosting. As such, security teams have a responsibility to deploy tools and gain a proactive, real-time view into K8s audit logs in order to minimize the time between breach and remediation and to validate compliance conformance at runtime. Such a thoughtful approach to Kubernetes security will go a long way to protect your deployments from harm.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or Aug. 2020.

— Nitzan Niv
March 13, 2020
A K8S Tech Stack for the Real World

— ROELBOB

— ROELBOB

— ROELBOB

March 13, 2020
A New Paradigm in Container Security with Service Mesh
Containers have been around for some time, but only recently has the technology become mature enough for large enterprises to adopt it both on-prem and in the cloud. The obvious benefits for developers include the ability to dramatically reduce the time to production and the ability to lower the operational burden. This is further supported by implementing a service mesh which provides fast, reliable and secure communication among containerized infrastructure.

The Rise of Kubernetes and its Impact on Security

Kubernetes (k8s), introduced late 2014, bring container orchestration at large scale to both cloud and on-prem environments. The benefits of using k8s is clear and stipulated mainly due to the ability to become agnostic to the infrastructure. An application based on k8s can easily scale out from your datacenter into multiple cloud providers, and back. However, the challenge with Kubernetes is it creates a huge problem for security operations. Trying to enforce micro-segmentation on a large k8s cluster using traditional firewall models is not feasible.

As the move to containers rise, so will container-based cybersecurity attacks. Criminal hackers will always be on the hunt for new ways to exploit this area. As container operations are inherently dependent on the development pipeline (CI/CD) so should be their security attributes. Any other interaction point will be an afterthought resulting in slower operations or reduced security. Organizations moving to containers and k8s should act preemptively due to on-going and increasing complexity driven by scale.

Enter Service-Mesh

Implementing a service mesh layer ensures that communication among containerized infrastructure services is fast, reliable and secure. It is a configurable, low‑latency infrastructure layer that handles service-to-service communication. It is responsible for the reliable delivery of requests through the complex topology of services that make up a modern, cloud-native application. In practice, a service mesh is typically implemented as an array of lightweight network proxies called sidecars, that are deployed alongside application code without the application being aware. The mesh provides critical capabilities, including service discovery, load balancing, encryption, observability, traceability, authentication and authorization, and support for the circuit breaker pattern.

Criteria and Security Best Practices
- Pre-scanning of images isn’t enough. Enforcing runtime micro-segmentation for container services should be addressed at runtime as they introduce an easier, more prominent attack vector.
- Easy integration with development pipeline. As noted, the alternative will impact business agility or security quality.
- Micro-segmentation should be decoupled from the infra and network as otherwise, it will require some kind of application to IP/PORT/protocol mapping.
Implementing security built for k8s environments and running both on-prem and in the public cloud that uses Istio, a well-known open-source service mesh developed by Google, is recommended. Vet solutions connect directly to the code pipeline with enhanced security capabilities, such as the signing of artifacts that will be deployed to the k8s cluster in the code stage with the ability to generate signed identities. Instead of looking at machines and IPs when workloads boot up, the administrator will see that services will be able to control where they run.

This will also enable you to intuitively create identity-based micro-segmentation. This is an advantage since micro-segmentation was always a complicated task to achieve, requiring administrators to map an application to the network while keeping track of what is running in the environment.

With identities and a service mesh layer, it is only necessary to focus on your business logic. The fact that identities were generated during the CI/CD pipeline, provides the ability to control the application without the need for service discovery. Identities are represented by their business-context names, originating in the CI/CD pipeline, so the operator can easily group them together and assign policies at different layers of granularity.

K8s is a great technology that enables the harvesting of containerized environments. Adopting it can accelerate your organization’s digital transformation dramatically, but it can also require you to adopt new methods of application security. Contracting with a k8s-focused security organization will allow the business to accelerate its migration to cloud-native applications on-prem and in the cloud, in a most automated fashion.

By providing the needed security and operational capabilities, k8s clusters admins are afforded an enterprise-level micro-segmentation solution with policy orchestration, single pane of glass to your clusters in a multi/hybrid cloud deployment and the ability to connect between containerized clusters and legacy applications, as well as other resources that are external to the cluster.

— Ran Ilany
November 1, 2019