devopsy.kinsta.cloud

Tag: kubernetes

  • DevOps Chat: The Kubernetes and Multi-Cloud App Journey with Rafay

    DevOps Chat: The Kubernetes and Multi-Cloud App Journey with Rafay

    After Akamai acquired his company, Haseeb Budhani decided to take on his next challenge and start Rafay.

    Rafay focuses on the complexity and repetitive aspects of deploying and managing Kubernetes applications. Rafay addresses complexity by providing application abstraction, cluster blueprinting and enterprise-ready integration, making Rafay a great candidate for multi-region, multi-cloud, hybrid and edge/MEC adoption.

    Join us on this episode of DevOps Chats where Haseeb shares some of the experiences and challenges that led him to found Rafay, and how to accelerate your path to Kubernetes and multi-cloud applications.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Haseeb Budhani, CEO and co-founder of Rafay. Our topic today is lifecycle management for containerized apps. Haseeb, welcome to DevOps Chat.

    Haseeb Budhani: Mitch, thanks for having me. Great to talk to you.

    Ashley: Yeah, nice to talk with you. Always love talking with a fellow entrepreneur. Could you start by just introducing yourself, a little bit about your history, what you do and a little bit about Rafay?

    Budhani: Absolutely. So, Rafay the company is slightly under two years old at this point. We have a product available on the market that a number of customers are using which, essentially, helps them simplify the ongoing operations and life cycle management for their containerized applications, which they may be running in public cloud, hybrid environments, or any combination thereof.

    Fundamentally, there is a brunt of work that every company does as it relates to ongoing management of the containerized apps. You know, essentially, they build a platform internally on top of Kubernetes, and our pitch is, “Hey, why are you spending time rebuilding the—kind of reinventing the same build over and over again in every company? Let us help you with our solution which focuses on the repetitive effort that every company goes through, so that you, Mr. DevOps Engineer, can focus on the true value that your company is looking for out there. So, you focus on company value. We’ll help you solve for the repetitive work that you have to do, anyway.”

    Ashley: Very good. Now, you started this company after selling your prior company, Soha Systems, to Akamai, I think you took about a year before you jumped out and started Rafay. So, why did you pick this particular problem? What was it about it? Was it something you experienced in Akamai, in your prior company, or just as you were kinda taking that time to really think about what you wanted to do next? Why did you pick this space?

    Budhani: Yeah. You know, Akamai, it was, surprisingly, a lot of my friends who’ve kind of seen exits, you know, kind of, they talked about acquiring a company, it was tough, and I didn’t like the experience or whatever. I actually had a lot of fun at Akamai. It wasn’t as stressful as it used to be. I mean, you know how stressful startups are, so it was a pain-free experience, really. You know, an incredible sales organization. They really understood our product, took us to this whole new level.

    But, you know, sometimes an idea gets stuck in your head and it’s just impossible just to walk away from it. So, in our prior startup, so pre-acquisition by Akamai, we were practitioners of the problem that we are trying to address. So, essentially, we were the customer.

    Some of our colleagues at Soha were effectively building a platform like the one I just described, wherein we were kind of writing at this layer on top of container orchestration so that we could be running our application across many locations. So, at the time of exit, we ran—I don’t know, somewhere between 20 and 30 POPs globally across multiple cloud providers just so we could have our cloud based security platform at Soha running in a number of locations globally.

    And it was very, very painful. And I think at that point, we didn’t realize that when you’re building a platform, when you’re so focused on the core value which is the security, you know, this DevOps thing is important, but you don’t spend as much time strategically think about it. And that, after the acquisition by Akamai, Hemanth, my Co-founder at Soha and also here at Rafay, we spent a lot of time thinking about this. Like, why did we have the experience we did? Because it was not a good experience. And then we thought about, “Well, who else has this problem?”

    It turns out everybody has this problem where anybody building a SaaS platform or running applications in the cloud, particularly when it comes to containerized apps, because the community is so new, there is a whole lot of supporting cast, if you will—mythic. And it’s gonna come. It’s just, right now, it’s missing because it’s so new. And that got us thinking, “Well, somebody’s gotta solve this problem.” You know, and when you kinda look around and say, “Somebody’s gotta solve this problem” and nobody raises their hand, maybe you should do it.

    So, that got us thinking, “Well, we should—we need to go solve this problem.” So, we left. And I’m sure both of our wives were very unhappy when we said we were gonna leave Akamai. [Laughter]

    Ashley: [Laughter]

    Budhani: But—no, it was a worthy cause, and we’re very happy we left because we solved a really important problem for the community.

    Ashley: Well, congratulations for starting another startup. I think it’s one of those kind of in your blood type of things. So, talk a little bit about what’s unique about Rafay. I’m assuming you took some of the same problems that you experienced of what you’ve built at Soha and probably along the way said, “Hey, you know what? This might be a good idea for a product for containers and containerized apps.”

    And I think I would just add to the conversation by saying, as you start to build larger and larger containerized apps, you realize what some of the management challenges really are around it, and so that’s, you know, borne out of that is a new opportunity. I’m assuming you had a similar kind of experience.

    Budhani: Exactly right. You know, you read some blogs about, you know, like, Kubernetes. And it makes things look very easy. You can get something up on your laptop and you’re like, “Oh, look! I have a Kubernetes cluster running on my laptop! Isn’t this easy?”

    Ashley: [Laughter]

    Budhani: No. When you run it in production with multiple applications, it’s hard, and everybody knows it, this is not new information. But the challenge is that, because there is as yet no manual for this, we only learn by doing. So, we have to kind of jump into this and we start building stuff, and then we go, “Oh, this is hard,” and then by that time, we already have pretty large teams in most companies who are working on this. And that’s the problem and, of course, that is the opportunity, also.

    Here’s a better way of thinking about this. So, because of our experience at Soha, some things we recognized. Not everybody on our team we could train to be experts at the platform.

    Ashley: Mm-hmm.

    Budhani: Some people are experts and most people are not gonna be. And this is very typical in most companies, you have a DevOps team of five, seven, 10, 20 people who build a platform and then another hundreds or thousands of engineers, depending on the size of the company maybe consuming the platform. Because basically, they need to understand this—they need an abstraction layer. That was a key insight for us, having spent time talking to companies, you know, once we were out kind of looking at, you know, what do we do next?

    So, everybody seems to need some notion of an abstraction layer so that I, as a developer, I check in code and just magically, my application shows up somewhere. I don’t want to think about this underlying complexity. And the second thing that we saw, which VMware, surprisingly, validated two weeks ago when they did a bunch of announcements at VM World was that nobody’s running a single cluster.

    Ashley: Mm-hmm.

    Budhani: They may start with a single cluster, and because it’s easy to keep adding nodes to their cluster. But the practical approach around clusters is to run many small clusters. And VMware spent a lot of time talking about this idea of many small clusters.

    So, that implies solving for multi-cluster operations, multi-cluster federation is a critical thing. And that continues to be a gap in the community right now. There are a number of approaches that have been taken to solve for multi-cluster federation in the community, but—and, you know, a lot of companies have talked about this, there are tools that exist out there to solve this problem, but fundamentally, they make certain assumptions that are not borne out in practice. And that was a unique thing that we saw as a gap and we addressed it very elegantly in our platform.

    There’s a number of things we do, for example—I’ll give you another example. And anybody listening to this podcast, as you hear this examples, you may kinda look back to your experiences and go, “Yeah, I’ve faced this.” So, you bring up a cluster. Let’s say you are some e-commerce company. So, you need some clusters that need to be PCI ready, or PCI compliant. Some clusters, for marketing, because all they do is some experiments—I don’t need to be stringent for them. And then you have people who are just writing code, developers.

    So, you need a blueprint for a cluster, because each cluster may need different things. How do you do that? It’s not easy. It’s not easy. You know, there’s no kind of help chart for help charts. What do I do then? These are some of the problems, right?

    So, if you really spend time talking to the DevOps community, these are some of things that they struggle with and today, they all take a bunch of open source tools and try to build something for themselves. Some do it very elegantly. Some just never get to that level of perfection because they are so busy. And this is my pitch to every customer. My entire team, all of us—we do one thing. We do one thing. We help you run your containerized applications.

    Ashley: Mm-hmm.

    Budhani: You have 50 things to do. Let me help you with this, right? Let us be an extension of your team. Let us, you know, really take this one stress away from you—of course, pay money for that, [Laughter] but focus on the things that are more important to the company.

    Because getting the right script in place or whatever in place just so you can run your Kubernetes and run it better—this is not how you make money. If your competitors all ended up buying from a vendor to solve this problem versus build it themselves, they’re gonna probably sell a single more unit of widget that you’re selling, right? Focus on the value. Don’t focus on the very complex but effectively undifferentiated work.

    Ashley: Well, what you’re saying makes a lot of sense of why rebuild it all yourself? Why learn all the hard lessons that others might have learned? And I take it that you do things through your abstraction layer with your blueprinting of essentially kinda creating what we might think of in software as here’s patterns that we might use, here’s blueprints we might use for different clusters.

    Budhani: Yeah.

    Ashley: And I know a common mistake I’ve seen happen is, it’s really easy to throw everything into one cluster and then start to say, “Well, that doesn’t work. How do I start dividing it up into, besides just geographic location, what are the other techniques for doing that?” So, those are the kinda things that you’re bringing to the table with Rafay, is that correct?

    Budhani: Yeah, absolutely. Yeah. I mean, I’m sure many of us continue to have the Design Patterns book from college on our shelves, right?

    Ashley: Sure, yeah.

    Budhani:[Laughter] That’s the idea, right? Some things are best done in a specific way, and if there’s a way to do that, if it’s a best practice way to do it, you do it. Not every company may have their own quote-unquote blueprint, and that’s okay. But let’s help you manage those better.

    In effect—you know, and this is fundamentally, we talked about, practically, what Rafay is selling today. But if we fast forward three, five years, we sell something today, but we still have to have a raison d’être, right? Why are we doing this? I think, fundamentally, there is a system of record needed in our industry that maybe doesn’t exist as well as it could. Where is that one place in a company where I can go and say, “Hey, where is ____ right now? How is it doing? Three months ago, how was it? Who changed what last time? Where is that information today?”

    It’s in people’s heads, sure. It’s in a Git repository. We all talk about GitOps, right, which means somebody wrote a script and checked into a git repo. Yeah, but maybe the guy who wrote that script left six months ago—now, who understands that script? That is a very common occurrence—people write scripts and then they move onto the next job. Where is that system of record? Where is that system of governance for an application?

    Particularly with containers, there is an opportunity to very elegantly address that problem, right? Have a system that allows you to do that. One system where somebody who is non-technical can log in and say, “Hey, just from a compliance perspective, is this app running on all the right clusters right now, or not? One shot, can I know that?” That is where I think we will all go as a community.

    And as I kinda think through what are those bread crumbs to get us there, some of the things that we are working on as a product company today, my hope is that we get there. We get the opportunity at Rafay to build that system of record that people will really treat as a strategic system for their own companies. Anybody in the company should be able to come and look at, “Hey, where is my—what is the status of my app right now? How has it changed in the last month?” That, to me, is such an important and worthy goal to be focused on. Let’s see how—you know, what it takes for us as a group to get there. But that, man, that—I mean, somebody needs to go solve that problem.

    Ashley: Now, I haven’t used your product, but one of the things that I found really intriguing about your go to market and how you talk about it on your website is, you actually present use cases, more than just generic things, but something specific to, like, factory operations for IoT and customer experience app, modernizing in store retail experiences, 5G, edge deployments. Are those part of the kinda blueprints of what you talk about, or is that more, you know, on a broader scale, “Here’s how we might advise you about how to implement a containerized cluster environment?”

    Budhani: So, these are the business use cases that are—the last one we talked about was this retail use case. It’s a fascinating problem. Many retailers are out there talking to larger vendors and in some cases to their ____ providers who happen to be large customers. And they’ve asked the question, “Hey, I want to deliver a better experience inside my store. Is the point of sale system in the retail store up or not?”

    That’s a good question. How do you find out? We can’t run these tests from out of the store, because that’s a private network, basically. So, I wanna run an app inside my store. Oh, yeah? Where do I run it? Where is this infrastructure? This is not a—there’s no data center, here. They have probably one rack in the back somewhere, maybe with one or two machines. So, then maybe I should run containers here—but then, who’s gonna manage my containers, because I have 1,000 stores, it’s not one. And if I have to send IT people to every store to deploy an application and upgrade it regularly—well, I need an army now, right?

    It’s just a small question. I need to run a small, tiny, 200 ____ app. It becomes this big thing. What an interesting problem. But this is a real problem. Think about any retail location where they have a lot of refrigeration capability. They’re selling ice cream or they’re selling whatever drinks and whatnot. If any one of these refrigeration units is not working, you lose a lot of money.

    Ashley: A very big deal.

    Budhani: And, of course, it’s a bad—right? So, they run these sensors, okay? Who’s collecting this data? Where is the sensor running? Is it a tiny little sensor? But then the sensor data has to be collected maybe locally. Okay, what do I do now? Right?

    All of these small things, these are real business problems that kind of snowball into this bigger problem around application management, right? So, if I—if you consider, if we called a retail store or some head office for some retailer and said, “Hey, do you wanna buy a container management platform or life cycle management containerized app?” Like, “What is a container?” Right? [Laughter]

    Ashley: Yeah.

    Budhani: Their people know, but they may not know, right? But they understand this problem, right? It’s like, “I need to run apps here, because this is how I’m gonna make more money. I want to provide a better online and offline experience for my customers.” That opportunity, right?

    So, that’s why, as we hear use cases, as we partner with different companies, and we see very specific and repetitive use cases come up again and again, our hope is that we continue to list them on the website. So it’s more meaningful for even the business or more kind of—you know, non-technical people in the company to come to a website like this and understand why, you know, their engineers are talking to us, for example.

    Ashley: You described yourself as a SaaS based company, it’s a SaaS service. Do you also provide any operations support, kinda outsource of operations, too, or is it strictly a service that others then operate on top of your SaaS?

    Budhani: So, it’s a service, and we do provide kinda more of a solution architecture model ready to essentially assist in our customers thinking through how they’re gonna consume a platform like this, right? Because a platform like this is not—it’s not a box that you plug into your network, if you will. This is a strategic conversation where we’re thinking about how to embed it in some way into your processes or into your, you know, own organization.

    So, there’s some level of consultative conversations that are happening up front, definitely, and on an ongoing basis, also. But from an operations perspective, we don’t see that when Rafay is fundamentally a software company, you know, a lot of pre-sale support on the solution side. We are continuing to engage with a number of companies who excel at that, who look at Rafay as an enabler not just for themselves but for their customers to whom they’re delivering a broader service. So, think systems integrators, DevOps consulting organizations who, whether Rafay exists or not, they are solving this problem and they have other customers, right?

    Ashley: Mm-hmm.

    Budhani: So, those companies look at us as an accelerator for their own services and for their own offerings. So, that’s been how we’ve been approaching the market, so we can obtain a high margin business and be, effectively, a software company. That’s what we know how to do well. But there’s so many great organizations out there who have an incredible skill set as it relates to operations and integration. That’s where we step back and partner with somebody, and at that point, be our tool in the tool chain and they go deliver a great experience to their customers.

    Ashley: Very good. That helps a lot. So, talk about who your most ideal customer would be. Is it a large enterprise, medium to large? What kind of an organization would say, “This is exactly what I need—come help me”?

    Budhani: In theory, this is a horizontal problem. Anybody who’s doing, you know, running containerized applications has this issue. But fundamentally, what we, at different stages in the company, look for different kinds of companies in terms of just, you know, low hanging fruit, et cetera.

    What we find works well today is, you know, relatively mature organizations who are doing somewhere between a couple of hundred million to a billion dollars of revenue, they are modernizing their applications, moving to the cloud, becoming more and more cloud native. And they’re the ones who are very interested in going fast, because there’s some—usually, there’s some business reason why we’re doing this. They’re not just modernizing for the sake of modernizing, there’s competitive pressure, there’s cost pressure. They may need to be in different parts of the world, et cetera.

    There’s reasons why they need to be up and about quickly, and they will look for any shortcut to get them there. A shortcut in this case would be, you know, “Hey, I could build this myself. Should I?” Right? And what I just said, right? This is a question we, in some way, shape, or form always want to highlight to our customers, which is—hey, you have a really strong engineering batch. You can do anything. Should you do everything? Do you still build data centers? And that, of course—that’s a, you know, asking the question, “Are you still using, building data centers?” that’s a straw man, of course. Because, hey, I mean, maybe other than 100 companies in the world, nobody’s building data centers any more.

    Ashley: Right, right.

    Budhani: It makes a point. In our industry, we continue to go up the stack in terms of abstractions, right? I mean, all the way from assembly to op and now nobody’s building subnets any more. Private subnets, published subnets—you just make an API colony and you get one in Amazon, right? Why am I building this stuff?

    Similarly, we are providing them yet another level of abstraction which allows them to move faster. Because fundamentally, the name of the game is, “Hey, if I don’t move fast, if I don’t produce product fast, I’m gonna lose.”

    Ashley: Now, I know you worked with AWS as your et cetera VMware. Is there a particular environment that you’re best suited for? Like, if you’re a VMware customer, you’re gonna—we’re gonna connect right into your environment, or is there a technology stack that’s more easy for you to integrate than others?

    Budhani: So, this is a very agnostic platform. I mean, VMware environments can be anywhere. And I’ll highlight a specific thing that we’ve thought through and we implemented as part of the platform.

    So, let’s assume for a minute that it’s a mature organization, they do have some hybrid environments, and they maybe have some Tectonic set up and they’re running VMware there, and they also have Amazon, and that’s okay, right? So, in one environment, maybe they’re using PKS from VMware, PKS Essentials from VMware. But in Amazon, they’re using EKS, because that’s what Amazon sells for Kubernetes.

    Okay, now what do I do? Am I gonna build two sets of platforms to support two different environments? Not a lot of people do that. Or, with a platform like ours, what we say is, “Hey, just—look, if you have an opinion on Kubernetes, go with it, and essentially, we’ll add value on top of your existing Kubernetes throughout the operator.” It’s a Kubernetes concept called an operator.

    Ashley: Mm-hmm.

    Budhani: Or, if you don’t care and if you say to us, “Hey, you know what? Just bring me the, I don’t know, whatever the latest upstream Kubernetes is, 115 or whatever—just bring that.” No problem, we’ll work with that, also. But, for our SaaS platform, to help you manage your environments behind a firewall in a data center or behind a security group in Amazon, you don’t ever happen to open a single portal in the firewall for us to get in from outside. The entire system is designed to be secure such that there are sessions being launched from inside out, so your clusters run an agent, basically. They reach out and broker all the connectivity without making a security issue kind of pop up on the security team’s radar.

    So, this is another thing, by the way—anybody listening to this podcast thinking about some sort of a multi-cluster or multi-cloud solution? Please let’s make sure that you’re not signing up for a vendor who’s asking us to make maybe not the best decisions when it comes to security and asking us to open up ports, setting up IPsec links—you know, all of these are bad ideas that lead eventually to other problems that you haven’t thought about today.

    So, these are some of the things where understanding our enterprise security, understanding our enterprise problems are important up front, so we thought through these things and we designed a platform to be able to work not just in any environment, but to work in these environments in a very secure fashion.

    Ashley: Well, certainly, I think, with all your experience, you can help de-risk things and provide kind of a framework or a path for enterprises.

    Well, unfortunately, we’ve run out of time. See, I feel like we could talk about this for another hour and a half and share a lot of good stories about it, too. I’d like to thank you—thanks for being on the webcast, the podcast, here.

    Budhani: Yeah, thanks for having me. It was a really fun conversation. I look forward to talking again soon.

    Ashley: Yeah, I do, too. Keep us informed about any new news. So, I’d like to thank our guest today, Haseeb Budhani, CEO and co-founder of Rafay. I’d also like to thank, of course, you, our listeners, for joining us today. This is Mitch Ashley with DevOps, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • A New Paradigm in Container Security with Service Mesh

    A New Paradigm in Container Security with Service Mesh

    Containers have been around for some time, but only recently has the technology become mature enough for large enterprises to adopt it both on-prem and in the cloud. The obvious benefits for developers include the ability to dramatically reduce the time to production and the ability to lower the operational burden. This is further supported by implementing a service mesh which provides fast, reliable and secure communication among containerized infrastructure.

    The Rise of Kubernetes and its Impact on Security

    Kubernetes (k8s), introduced late 2014, bring container orchestration at large scale to both cloud and on-prem environments. The benefits of using k8s is clear and stipulated mainly due to the ability to become agnostic to the infrastructure. An application based on k8s can easily scale out from your datacenter into multiple cloud providers, and back. However, the challenge with Kubernetes is it creates a huge problem for security operations. Trying to enforce micro-segmentation on a large k8s cluster using traditional firewall models is not feasible.

    As the move to containers rise, so will container-based cybersecurity attacks. Criminal hackers will always be on the hunt for new ways to exploit this area. As container operations are inherently dependent on the development pipeline (CI/CD) so should be their security attributes. Any other interaction point will be an afterthought resulting in slower operations or reduced security. Organizations moving to containers and k8s should act preemptively due to on-going and increasing complexity driven by scale.

    Enter Service-Mesh

    Implementing a service mesh layer ensures that communication among containerized infrastructure services is fast, reliable and secure. It is a configurable, low‑latency infrastructure layer that handles service-to-service communication. It is responsible for the reliable delivery of requests through the complex topology of services that make up a modern, cloud-native application. In practice, a service mesh is typically implemented as an array of lightweight network proxies called sidecars, that are deployed alongside application code without the application being aware. The mesh provides critical capabilities, including service discovery, load balancing, encryption, observability, traceability, authentication and authorization, and support for the circuit breaker pattern.

    Criteria and Security Best Practices

    • Pre-scanning of images isn’t enough. Enforcing runtime micro-segmentation for container services should be addressed at runtime as they introduce an easier, more prominent attack vector.
    • Easy integration with development pipeline. As noted, the alternative will impact business agility or security quality.
    • Micro-segmentation should be decoupled from the infra and network as otherwise, it will require some kind of application to IP/PORT/protocol mapping.

    Implementing security built for k8s environments and running both on-prem and in the public cloud that uses Istio, a well-known open-source service mesh developed by Google, is recommended. Vet solutions connect directly to the code pipeline with enhanced security capabilities, such as the signing of artifacts that will be deployed to the k8s cluster in the code stage with the ability to generate signed identities. Instead of looking at machines and IPs when workloads boot up, the administrator will see that services will be able to control where they run.

    This will also enable you to intuitively create identity-based micro-segmentation. This is an advantage since micro-segmentation was always a complicated task to achieve, requiring administrators to map an application to the network while keeping track of what is running in the environment.

    With identities and a service mesh layer, it is only necessary to focus on your business logic. The fact that identities were generated during the CI/CD pipeline, provides the ability to control the application without the need for service discovery. Identities are represented by their business-context names, originating in the CI/CD pipeline, so the operator can easily group them together and assign policies at different layers of granularity.

    K8s is a great technology that enables the harvesting of containerized environments. Adopting it can accelerate your organization’s digital transformation dramatically, but it can also require you to adopt new methods of application security. Contracting with a k8s-focused security organization will allow the business to accelerate its migration to cloud-native applications on-prem and in the cloud, in a most automated fashion.

    By providing the needed security and operational capabilities, k8s clusters admins are afforded an enterprise-level micro-segmentation solution with policy orchestration, single pane of glass to your clusters in a multi/hybrid cloud deployment and the ability to connect between containerized clusters and legacy applications, as well as other resources that are external to the cluster.

    — Ran Ilany

  • Data Monoliths Go Cloud-Native with Kubernetes

    Data Monoliths Go Cloud-Native with Kubernetes

    Breaking monolithic applications into independently scalable services is the core of the cloud-native application model. By making cloud-native apps easy to deploy and manage, Kubernetes has contributed to a big change in IT practices.

    Initial cloud-native conquests started with stateless services and quickly grew to encompass applications that manage data. Database management systems have long relied on centralized servers, but cloud-native architectures are now reaching even the largest monoliths. Among the most interesting cloud-native converts are analytic databases, which hold data arranged in storage formats optimized for extremely fast analytic queries over billions ,or even trillions, of rows.

    The transformation of databases to cloud-native models has profound implications for data-driven applications. Intersecting trends in analytic database architecture and Kubernetes promise substantial benefits to the users of such applications as well as companies that operate them.

    The Rising Importance of Analytics

    Over the last decade the quantity of data that businesses generate has decoupled from the size of the business itself. Ad networks, website analytics and service observability applications often ingest 1 million events per second (or far more) and may have tables running into trillions of records. Businesses need to analyze this information rapidly–for example, to diagnose faults in cloud applications or to retrain models for displaying online ads. Flexible, real-time analytics have gone from nice-to-have to business critical.

    Meanwhile, new analytic engines have emerged that eliminate dependencies on specialized hardware and complex networking required by legacy data warehouses. Druid, ClickHouse or Elasticsearch run on commodity hardware and depend on basic compute, network and storage. Everything else, such as high availability (HA), is built into the application. Newer analytic databases offer built-in horizontal scaling using fleets of server processes. The server processes themselves typically run well in containers.

    With these changes, analytic databases are increasingly cloud-friendly if not cloud-native. It is a short hop to deployment on Kubernetes.

    Kubernetes Operators for Databases

    At the same time, as analytic databases have shifted their operational model, Kubernetes itself has become more welcoming for databases.

    One particularly important innovation for the database is custom resource definitions, or CRDs, which extend the Kubernetes API to define new types of manageable resources. Using CRDs makes it possible to have a single definition that shows the desired state of a complex application such as a data warehouse.

    Another addition is Kubernetes operators, which manage custom resource definitions. Operators track custom resource definitions and take actions when they change. They can also track other information that might be relevant to managing the custom resource, such as changes to pods, storage or even related cloud-native applications. It’s like having a human operator who is constantly watching the application, except the rules are programmed into software that runs inside Kubernetes itself.

    The operator pattern is a perfect match for analytic databases, which are complex distributed applications and may require definition of dozens or even hundreds of individual Kubernetes resources in a running system. Operators encapsulate these details as well as important procedures such as upgrade, scale-out and other operations that require careful manipulation of resources to prevent outages.

    It’s not surprising operators are popular with database developers. The OperatorHub.io registry for database operators alone has 19 entries covering databases of every kind, including analytic databases such as ClickHouse and Elastic. Many more exist outside the registry.

    A Cloud-Native Way to Consume Analytic Databases

    New data warehouse architectures and Kubernetes operators means users can now use analytic databases very differently from legacy data warehouses. Our work on ClickHouse, including development of the ClickHouse Kubernetes Operator, prompts two observations.

    First, operators make spinning up analytic databases simple and fast for all users. Amazon Redshift kicked open the door to data warehouse-as-a-service in the public cloud. Kubernetes operators enable similar capabilities not just in public clouds but any Kubernetes cluster. Individual developers can cycle through dozens of configurations in a single day while testing new analytic applications. Thanks to quick pod spin-up new analytic databases can be running in minutes or even less.

    More profoundly, users can use Kubernetes to move to an on-demand model for analytics. Rather than forcing all applications to compete for resources on a single shared cluster, each application can have an analytic database tuned to its specific requirements. Analytic databases can integrate tightly with applications in the same locations where those applications already run. Businesses can scale databases flexibly to deliver quality analytics at exactly the time they are needed and at reasonable cost. This is the heart of the cloud-native programming model.

    Modern database architectures combined with Kubernetes features, such as the operator pattern, are creating a new path to manage enormous volumes of data. For readers who are hesitant about managing data on Kubernetes, it’s time for another look.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Robert Hodges

  • Learning Production Troubleshooting Techniques in Microarchitectures

    Learning Production Troubleshooting Techniques in Microarchitectures

    One of the great things about our industry is the ability for anyone to get started doing development with a minimal amount of tools. Anyone can learn to do development with just a computer of some sort, a text editor, a tutorial and their brain.

    In fact, the initial basics of how to think through a development problem don’t require a computer at all, reducing the cost of getting started even further. Having such an easy way to get started that doesn’t require a lot of tools lowers the bar for entry, enabling more people to join in on development teams and contribute new ideas and perspectives, which enriches the entire industry.

    Getting into containerization and building modern microarchitectures is a slightly higher bar, but only slightly higher. There are a number of wonderful tutorials out there for getting started with Kubernetes, Docker, OpenShift Origin and other free and open-source tools.

    The only differences in getting started between general software development and microarchitectures specifically are: needing a computer that has the processing power and memory to run containers, and enough of a grasp of a development language to have a basic application to play with. Even that last difference isn’t as crucial because there are so many sample applications people have built for the express purpose of teaching new folks to deploy microservices onto various systems.

    However, when it comes to learning from the DevOps, SRE or ops side of managing Kubernetes deployments or other modern architectures, there’s no solid free tutorial I could find that dumped a bunch of broken sandboxes in front of the student with the express purpose of building their troubleshooting knowledge.

    To gain insight into understanding how to bring microarchitectures back from failures or fall-overs requires access to larger applications and clusters and more senior, or more experienced, mentors–unless you are lucky enough to be able to build large interconnected applications and send traffic to them regularly enough to cause failures.

    This lack of strong tutorials and a requirement for more resources makes breaking into DevOps or SRE work on microarchitectures particularly difficult if no one is willing to take a chance on a less experienced candidate. I want to see that change. We are missing out on hearing from new voices, people who may look at a problem from a different perspective or who may have unique solutions to problems we all face.

    I decided to create a repository full of deliberately broken Kubernetes builds for new practitioners to build progressively more advanced troubleshooting skills. The basic idea is to build a pyramid of environments. The base level should be easy to understand and figure out. These environments will be bare-bones simple, with only one cause for failure.

    We need to build the baseline recognition of common causes in isolation. Many of us that have debugged production systems have an almost unconscious grasp of when a common cause has appeared simply because we recognize that faint signature in the pattern of the data we’re analyzing. If you’ve never seen that signature, though, and you don’t have a mentor with you on these debugging challenges, learning to find that signal in the haystack of your data is much harder. These initial environments should make it easier to learn what those signals look like.

    The next level of environments would have some additional factors unrelated to the actual problem causing additional logs, additional spikes in traffic or additional activity. In short, these environments would have noise. We can add another layer of complexity for some of these environments by adding another service or application that runs just fine on the same cluster. This level would train the ability to rule out sections of a system that aren’t broken, to identify systems that aren’t interconnected and to work on one system without disturbing the rest of the environment.

    Further environments up the pyramid would have multiple contributing factors, red herrings or complex structures. At this point, some of the environments will need to be hosted outside of the local system. I’m still debating how to do these environments so the tutorial could still be free and easy for new folks to work with–I’m open to suggestions.

    Ideally, there would also be a written guide with answers and tips a student wouldn’t look at until they’ve solved the problem on their own, but they certainly can look up and learn regardless of whether they solved the problem. After all, troubleshooting isn’t a closed-book exam by any stretch of the imagination.

    So how do you start? Find the repo at GitHub. It’s free–open source with an MIT license–and open to contributions. There’s a contribution section of the README on the repo that explains how to get started if you want to send commits. If you want to try using these environments yourself and practice or learn troubleshooting skills, the initial environments are built on Minikube, a tool to run small Kubernetes clusters on your local machine. Follow one of the many excellent tutorials for getting Mikube up and running on your local machine, then follow GitHub’s tutorial on cloning down a repo.

    Alternatively, you can download the repo directly. Once you’ve gotten Minikube up and running and you have a copy of the repo on your local machine, follow the instructions in each environment’s directory to deploy the environment to your local cluster, and start trying to figure out exactly what went wrong.

    Want to help make this project a reality? I would love to have more contributors! In addition to being available online for help, I’m going to be at KubeCon North America 2019 in San Diego. At the conference, I’ll have a Hackers’ Corner set up in LogDNA’s booth for most of the conference, and I’ll be sitting there working on this project, among others, waiting for you. Bring your ideas, your stories, your broken environments, your laptops, your willingness to write docs, your Kubernetes knowledge, yourself—anything that can help enrich this project is welcome.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Laura Santamaria

  • DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

    DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

    Many organizations want to implement the latest DevOps/SRE practices, but many struggle with transforming the existing processes over to new streamlined processes. How does an organization transform from zero automated deploys to 100s/1000s a month? Most importantly, how do you show the value of Spinnaker to your business as a whole?

    November’s Spinnaker Summit 2019 in San Diego allows us an opportunity to hear and learn from DevOps engineers, developers and practitioners using Spinnaker open source software. Joel Vasallo, manager of Cloud DevOps at Redbox, shares a preview of his talk where he shares his experience about using Spinnaker open source to increase the number of deploys into production. Joel found Spinnaker so useful because it is very extensible, covers AWS well, strong support for Kubernetes, different deployment types and has the automation to create the software delivery platform that fits the need.

    Joel’s talk, Transforming Software Delivery using Spinnaker, is on November 17, 12:30 pm PT at Spinnaker Summit 2019 in San Diego.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Joel Vasallo who is manager of Cloud DevOps at Redbox. We know Redbox is the movies, the kiosk, the online service—great.

    Our topic today is going from zero to thousands of deploys a month. Boy, if you’re not doing a thousand a month already, that sounds like a great place to be. This is actually a preview of a talk Joel is doing at the Spinnaker Summit on “Transforming Software Delivery Using Spinnaker.” It’s 12:30 p.m. Sunday, November 17th in San Diego. Joe, welcome to DevOps Chat.

    Joel Vasallo: Hey, thanks so much for having me, Mitch. I’m glad to be here.

    Ashley: Glad to have you here. Well, tell us a little bit about yourself, what you do at Redbox and also, hopefully people know Redbox, but just a bit about Redbox.

    Vasallo: Yeah, no, my name is Joel Vasallo, I’m the Manager of Cloud DevOps at Redbox involved with software delivery to the cloud. In my spare time, I’m a GDG organizer, so a Google Developers Group here in Chicago. But about Redbox, I mean, you said a lot of people know what Redbox is. We’re the kiosk that’s usually found in your grocery store. So, we do DVDs and Blu-rays, but we also, a little known fact is, we do have an on demand platform, so video on demand, and that’s kind of our new thing that we’re doing right now. And there’s actually a lot of motivation around building fast deployments, and that’s kind of gonna be my drive for the talk.

    Ashley: Awesome. Now, just a question for you for cloud DevOps—does cloud actually include software that’s being delivered or information being delivered out to those kiosks, too, or is this really for the on demand platform in the cloud that you—

    Vasallo: Yeah, there’s some aspects of it as well. I mean, there’s a lot of APIs in use across the company, so there are some that do kind of traverse the cloud in that regard—yeah.

    Ashley: The only thing better than APIs are common APIs, right?

    Vasallo: [Laughter] Exactly.

    Ashley: Reuse.

    Vasallo: Yeah, especially if you have a solid delivery platform—of course.

    Ashley: Absolutely delivery platforms. Okay, so, let’s jump into it. We’re giving a preview of your talk, so there’ll be a lot more content, of course, in your actual talk. Tell us a little bit about, if you’re already achieved this thousands of deploys, you’ve been at this for a while. Tell us a little bit about how you got started on this journey.

    Vasallo: Yeah. You know, to preface a bit, I mean, I’ve been in the Spinnaker community since, like, late 2015, when it was still kind of an early release out of the open—in the open source community. And at a previous role at Gogo, I used to work at Gogo and we implemented Spinnaker basically from day one. And prior to that, we were using some other open source—Netflix open source tooling as well.

    But in that journey, we were going from a traditional monolithic, you know, to a microservices architecture, and the goal was to essentially build a way to deploy to the cloud as fast as possible, right? And just taking and building on those practices, you know, applying all the things, the same—the 12 factor concepts, immutable infrastructure, routing, all that stuff.

    It kind of transformed into some open source tooling that we’ve written in my history, and there’s also some practices when using Spinnaker to make such a big change in an organization, right? I mean, the whole topic of DevOps in an organization, there’s so many great talks out there, right?

    Ashley: Mm-hmm.

    Vasallo: And that’s definitely one of the—I guess, arguably, the hardest bit is, “Well, what does the DevOps mean for you?” Right? That’s the typical phrase you hear when people say, “Well, we’re DevOps.” “Well, what does that mean?” Right?

    Ashley: Well, now, you’re talking, obviously, at the Spinnaker Summit, and it sounds like you’re pretty committed to the product. This isn’t a podcast about that specifically, but you sound pretty dedicated to it. Why Spinnaker?

    Vasallo: You know, it’s—the core fact is, it kind of extends a little bit past just deployments. It has—it’s really, the biggest thing is, it’s so extensible. Right out of the box, you’re just coming out swinging. The fact that Netflix, Google, Microsoft, a lot of big names—Amazon as well, how could I forget?—are contributing and helping build this product up, you have some pretty big names out there, right? And why not lean on the shoulders of giants. They’ve seen, I would argue, probably one of the bigger challenges in terms of software delivery, especially at an in demand space such as, you know, for example, Netflix, right?

    Ashley: Mm-hmm.

    Vasallo: And building on that success, you know, you basically get those batteries included, right? You get some battle tested software, and I think that’s one of the biggest things.

    But in addition to that, I mean, it’s extensible. It covers Amazon very well, it covers Kubernetes, it has support for various deployment types and various customizations as well, some of which are customizations you need to build, and some customizations are actually built into the platform. And the best part, it’s open source, so that’s my favorite thing.

    Ashley: One of the key words I didn’t mention that I know your talk’s gonna be about is automated deploys, right? You’re not doing hundreds of thousands with a lot of manual process. Talk a little bit about the key of automation and you kind of assume a development team is gonna be all about—yeah, let’s automate this and make our lives easier, but maybe it’s not that easy to get from point A to B where you do have that automated. Talk about that part of the journey.

    Vasallo: Yeah, no, I agree with you. And I mean, it’s an unfortunate thing in the landscape right now where a lot of companies are rushing into the DevOps landscape, right, and they just say Jenkins, right—Travis CI, GitLab, whatever. And, you know, you’re essentially flooding tools on the market, right? And it’s very tough.

    But having a lot of tools doesn’t mean you’re DevOps, right? There’s some fundamental business changes that have to occur in order to achieve that delivery that we’re talking about, right?

    Ashley: Mm-hmm, yeah.

    Vasallo: So, with that, it’s kind of taking—how do I call it?—the little infinity loop of DevOps, right? Where that’s great and, you know, the plan, code, build, release—you know, that little cycle, it’s great, but you can’t use that as a guiding principle. There’s not enough detail, right? And then you can have the extreme on the other side, where it’s like the Candyland, I like to call it, where it’s like the Candyland of ITIL process where you’re just essentially passing Go and then passing down a ladder, if conditions. That’s too complex, and that’s hard to explain. So, there has to be something kind of in between.

    And that’s kind of what I’m gonna be focusing on in my talk and really asking the questions of how do you lay out an effective delivery process? What are some steps—what are the steps at your organization to get changes out the door? Because it’s a different answer for every company, right? There’s no one truth out there, unfortunate as it sounds.

    Ashley: Mm-hmm. Well, and everybody’s environment is unique. Maybe some of the problems are the same, but there’s also, you know, what is unique about your organization, whether it’s the business technology or just people, the team you’re working with has strengths in certain areas.

    Vasallo: Yeah, of course. And that’s something that I kind of build on in the talk is about building minimum viable products, right? And the argument is, many minimum viable pipelines, right? Really laying out, end to end, what needs to be done to get software delivery to occur. And the best way to start is really to look at what’s kinda currently happening, right? Lay out, you know, change has to come in, goes through phase two, phase three, phase four, and then ultimately phase five—profit, right?

    Ashley: Mm-hmm.

    Vasallo: Well, what are those phases and how much investment should be done in each spot, right? You can spend your time automating, I don’t know, some aspects of your pipeline, but what’s the return on investment, right? Would it have been better served in, for example, automating quality versus automating security scans, right? What’s gonna be the most immediate value in shaving time and getting you back from, you know, these long, drawn out pipelines to potentially fast and efficient ones, right?

    Ashley: It’s interesting. What was sort of the starting problem that took you down this path? Was the business asking you, “Hey, we need stuff faster, faster, faster”?

    Vasallo: Yeah.

    Ashley: Or was it just your own, “We don’t have resources, so let’s figure out how to automate it”? I mean, what kinda got you going on this?

    Vasallo: Yeah. So, a little bit about myself, I joined Redbox back in 2018 with the mission of building a software delivery pipeline to get software—me and my team to get software into the cloud. And with that, there’s also the transformation aspect of, “Well, what is the cloud? How do we integrate with the cloud?” And a lot of questions kinda came up.

    Fortunately, we took a big investment in doing a cloud native approach, you know, from day one. Which means no lift and shift, which is awesome. It’s arguably the hardest path, [Laughter] as we probably all know, but this means essentially from the ground up, building a software delivery platform, right, that’s integrated from the, all the way down to the development layer and all the way up to the automation layer. So, we’ve been very fortunate to work on something like that.

    That’s kind of where we started, and that was the motivation of building a truly cloud native pipeline, so, yeah, working on that stuff.

    Ashley: Now, you mentioned you’ve developed some tools, too, and with Spinnaker being open source, have you contributed a lot back to the community yourself, or are you more of a consumer of it? You know, everybody plays a different role, and not everybody’s, you know, having an open source project.

    Vasallo: I’ll selfishly say that I’m more of a consumer of it.

    Ashley: That’s okay.

    Vasallo: I would love to contribute more to it. I’ve done a few contributions in my past, but in recent times, I haven’t.

    With that being said, though, my team is empowered to make change and drive changes, so we do have one or two changes in flight, and one of them actually got merged this past week. So, we’re looking to help grow and not just be consumers. We’re hoping to also build and solve the same problems we’re seeing, maybe, for other people as well.

    Ashley: Well, good. That’s great if you get a chance to do that. Keep in mind, you know, exercising it, especially at the volume you’re talking about, that is super valuable, too. That’s what helps improve software, whether it’s open source or paid for. So, that’s another—

    Vasallo: Well, yeah, and I mean, it’s kinda like what you’re saying. If you’re not paying for the software, maybe you can pay for it within support in the sense that Spinnaker’s community is an open community. There’s a Slack channel out there and you can join—anyone’s more than welcome to join, ask questions, and get answers from potentially some high up people. And Netflix, even people in the community in general, right?

    We’re all on the same journey, and I mean, that’s arguably what kinda kept me involved in the Spinnaker community is just seeing such an open and collaborative group of individuals just willing to help each other—special interest groups every week or so meeting and just saying, “Hey, what’s the direction of the project? What else can we do? What are some challenges we’re seeing?” It’s actually been pretty fun.

    Ashley: What are some of the objections you typically see or maybe that you even ran into about automating and doing deploys at that volume?

    Vasallo: Yeah. So, it’s not like we’re doing 1,000 more changes a month, right? It’s arguably, running fast can initially be seen as a reckless approach, right? You know, if you said, “Hey, we’re averaging 10 deployments a month”—maybe that’s a standard organization—“to 1,000,” you’re gonna definitely open up some eyeballs, right? People will be like, “Wait, wait, wait. What do we need that for?”

    Ashley: “What about stability? What about up time? What about”—

    Vasallo: Exactly, exactly. But the argument is, it could be said that smaller changes typically result in less down time, right? If you can break down a purchase path change in some company, right, and maybe saying, “Hey, we’re gonna make a UI change and then we’re gonna do a back end change that’s backwards compatible and we’re gonna do, then, an API tier change to support that change such that everything hooks together,” you then broke up one large change of three major components into three separate small, individual pieces that can be rolled out safely, right? And tested and embedded as well. That’s the most important part. Because you’re only as good as your automation and testing as well.

    Ashley: Now, did you do much shift left in terms of doing more security testing or vulnerability testing earlier in the Dev cycle, too, as part of this?

    Vasallo: Well, so, that’s the beauty of it. I mean, essentially, the pipeline doesn’t, at least for us, doesn’t start at Jenkins or the build time. It starts all the way at the commit. So, we have a lot of tooling, at least at Redbox, that’s built to empower developers. The thing to also note about those 1,000 deployments—those deployments aren’t, you know, some Ops team clicking an OK button. They are developers promoting their own code, and they are empowered to write and deploy code to production when they feel it’s appropriate to release, right?

    But in doing that, we don’t just give them the keys to the city, right? We build a nice, safe, and reliable pipeline—essentially, a paved road, if you will. That’s the kind of term I’ve been hearing it described where, you know, you can ride the paved road and have fun. You can get off road and it’ll be bumpy, but the majority of the time, the paved road will get you to production as fast as possible.

    Ashley: Right. Yeah, that paved road concept comes from the Netflix paved road talk that they gave. I think that was OSCON or something.

    Vasallo: Yeah, there are definitely a lot of great things from the engineers there, a lot of inspirational things kinda came out of it. I mean, again, it’s building on the shoulders of these giants, right? They’ve seen, arguable, large percents of Internet traffic, right? And at that level of success, you know that some of those concepts definitely can be applied elsewhere, right?

    Ashley: Mm-hmm. Well, you know, one of the things that’s maybe a little bit misleading in saying automated deploys is, it isn’t just the deployment process that’s automated, is it? Talk about what all the parts that are automated to be able to deploy at that volume.

    Vasallo: Yeah, no—definitely. I mean, it’s essentially part of the—part of the talk is also gonna, the little secret of it is, it’s not like you can roll in Spinnaker and have an automated deploy and you’ll be good, right? There’s a lot of aspects of a pipeline, and as I was alluding to from a commit layer, right, whether you do your static analysis for security early on, whether you do your unit tests at build time, whether you have an automated way of storing in archiving artifacts, right? You need to know what is making it out to your environment stage prod.

    In addition to that, you need to build that infrastructure automatically, right? So, we have things—you know, you can use things like Terraform and CloudFormation as well, right?

    Ashley: Mm-hmm.

    Vasallo: But your load balancers, your security groups—again, we’re talking within the context of AWS—how do you reliably and repeatedly build your infrastructure in such a way that it can support that rapid deployment, right? You can’t just have an automated pipeline where it’s like, “Okay, hold the lights, let’s go call up somebody and say, ‘Hey, we need a load balancer. Can we get that done?’ ‘Sure, give me five minutes.’” Even five minutes is too slow, right? We need to have an automated way of creating and also getting fast feedback to our developers when these changes work and when they don’t work, right, so they can help triage and figure out what’s going on.

    Ashley: It’s almost like I’ve said before—DevOps is an overnight success 35 years in the making. [Laughter]

    Vasallo: [Laughter] That’s a good one, that’s really good.

    Ashley: Where you kinda get there one step at a time, but it is, and I think that’ll be something interesting to anticipate about your talk is really thinking about it holistically and what that entire process—not that you’re gonna go into every bit of detail in that amount of time, but it is about all of that and connecting the toolchain, the work flow, and the process, people process part of it into something that can be automated and happen at high scale.

    Vasallo: Yeah, that’s definitely something to be said there. I mean, a lot of—it’s generally a soft topic, but it’s arguably one of the more important ones is, there has to be a cultural transformation and understanding if you truly wanna be as successful as some of these giants, right?

    Building the concepts of open culture, right, working together, cross functional teams—there’s no magical fix out there in the world to fix a broken process in a broken culture, right? And that’s the toughest thing. And then that’s something, definitely, we touch on as well. And I alluded to it earlier with the concepts of, for us, that’s empowering our developers. So, building compelling tools for our developers to use every day they come to work.

    Ashley: Well, you mentioned keys to the kingdom for the developers. I can’t imagine you did all this without some form of senior leadership backing. How did you get that and where did you get it from to go down this path?

    Vasallo: Yeah. So, it definitely started very grassroots, right? Say the challenge to get to the cloud, right? That’s an easy challenge and I think a lot of—sorry, that’s an easy statement to say, but what does that mean? And for every organization, it’s different, right? For some, lift and shift is the most appropriate solution if you’re not willing to invest in the cultural changes, right?

    But for me, at least in my experience, it’s always been finding the hero in the regards of not necessarily a hero mentality, but finding that one champion who can help drive that, you know? That team who is under a tight deadline and you’re like, “Hey, you know, you’re a relatively new project. You’ve never really seen—you’ve never really gotten deployed. How about we just take an experiment and see if we can automate your app end to end, right?”

    Ashley: Mm-hmm, mm-hmm.

    Vasallo: The good news is, it’s a pretty low risk, it’s a new feature. The timeline, obviously, is always a concern, right, in terms of meeting business deliverables. But in doing that, working with those individuals, you not only can get fast feedback as to the proposal of the pipeline, you can also build a relationship that essentially is like DevOps and Devs working together to kind of create this delivery solution, if you will.

    Ashley: Hmm. Well, I feel like we just scratched the surface, which is probably a good thing, because it’s the preview of your talk. I appreciate you coming on the podcast today, Joel.

    Vasallo: No—no worries. Hey, thanks so much for having me. Again, I hope to see everyone at the Spinnaker Summit. It’s gonna be a great time, so I hope to see everyone there.

    Ashley: It is. Well, definitely, I’d like to thank Joel Vasallo, Manager of Cloud DevOps at Redbox for joining us today. Again, Joel’s talk at Spinnaker Summit, which is at 12:30 p.m. Sunday, November 17th in San Francisco, it’s titled “Transforming Software Delivery using Spinnaker,” so be sure and check that out if you’re making it. Now you have a reason to go to San Diego in the fall coming on the winter.

    Vasallo: Other than the great tacos, yeah. And everyone, join us on the Spinnaker Slack, we’re happy to help out and answer any questions as well. We’re more than a welcoming community, so hope to see more folks there.

    Ashley: That’s a great point. It’s a very vibrant community. So, we wish you all the best in your talk, and everyone, you’ve listened to another DevOps Chat podcast. I’d like to thank you, our listeners, for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud and you’ve listened to another DevOps Chat. Be careful out there.

    — Mitchell Ashley

  • Identifying and Abstracting Business Intelligence from Kubernetes Workloads

    Identifying and Abstracting Business Intelligence from Kubernetes Workloads

    In the age of big data, businesses are inundated with data points. While they know there’s an abundance of valuable resources available to them, making sense of this data to derive actionable insights is often still a challenge.

    CNCF industry survey data shows workload complexity and monitoring remain top challenges for enterprises in terms of using and deploying Kubernetes. Many understand there are valuable resources within these environments, but struggle to best identify and extract meaning from machine data. While most of this data is readily available, it just takes the right tools to gather and view intelligent insights.

    The Benefits of Open-Source Data Collection

    Perusing the CNCF website, there are a wealth of tools built around Kubernetes to enable not only monitoring but networking, storage and security. There is even a Kubernetes-specific package manager, Helm, to make the deployment and management of these resources easy and consistent.

    There are a couple of key benefits to taking advantage of these open-source collectors:

    1. They stay up to date. Each of these tools benefits from deep community support. As new versions of Kubernetes are released, the extensive use of each of these tools ensures they are quickly updated.
    2. They integrate with everything. Prometheus, for example, has an impressive list of integrations and exporters.  Regardless of your unique stack, it is likely that there is support for what you might want to export data from. The importance of these integrations cannot be overstated, as they enable the flexibility needed to grow and evolve a Kubernetes deployment over time.

    Ensuring Complete and Organized Metrics Collection

    Prometheus–endorsed by the CNCF–is the de facto tool of choice for metrics monitoring with extensive support for anything you might use to collect metrics data.

    Prometheus works by pulling data from all of the components and jobs running in Kubernetes since every component of Kubernetes exposes its metrics in a Prometheus format. The processes running behind those components then serve up the metrics on an HTTP URL. For example, the Kubernetes API Server serves its metrics on https://$API_HOST:443/metrics.

    Prometheus is particularly good at auto-discovering the jobs and services currently running in a Kubernetes cluster. As pods are added, removed or restarted, the Kubernetes Service construct keeps track of what pods exist for a given service. This auto-discovery capability is one of the primary reasons for Prometheus’ popularity, ensuring that all new and existing components are monitored.

    The Importance of Cluster Level Logging and Event Collection

    Kubernetes does not define a single standard approach to log collection, but the most common method is called cluster level logging. Cluster level logging deploys a node level logging agent to each node which then funnels data to a separate backend for storage and analysis of logs. The primary benefit of this solution is that if a pod dies, the logs detailing what happened are retained. Implementing node level logging, without funneling data to a logging backend, will not retain log data if pods die or are evicted, while cluster level logging ensures that data is captured and retailed. A common tool for implementing cluster level logging is Fluentd—or Fluentbit, a lightweight version of Fluentd—which acts as the node level logging agent funneling data to a logging backend.

    Events provide insight into decisions being made by the cluster and unexpected events that occur in Kubernetes. Events are stored by the API server on the master node and collected using the same method as log collection—via a node level logging agent like Fluentd.

    Establishing a Continuous Intelligence Dashboard

    Finally, collectors for logs, metrics, events and security can be easily deployed using Helm—an open source Kubernetes package manager. Helm can significantly simplify the setup process, reducing hundreds of lines of configuration to one. These collection plugins can be used on any Kubernetes cluster, whether one from a managed service such Amazon Elastic Kubernetes Service (EKS) or a cluster you are running entirely on your own.

    According to Sumo Logic’s recent Continuous Intelligence Report, enterprise adoption and deployments of multi-cloud grew 50% year-over-year, and 80% of users across multi-cloud environments are now utilizing Kubernetes architectures. As Kubernetes continues to become mainstream, it’s essential that businesses understand how these workloads operate and how to best extract valuable insights to inform business decisions.

    — Katie Lane

  • From Pets to X-as-Code: Enabling Control and Predictability for ITOps at Scale

    From Pets to X-as-Code: Enabling Control and Predictability for ITOps at Scale

    Amy Wheelus, vice president of network cloud at AT&T, recently spoke about declarative predictability at the Open Networking Summit in Antwerp. AT&T is the world’s largest telecommunications company providing mission critical communication services where people’s jobs–and lives–can be on the line. As they move away from labor-intensive IT operations toward cloud-native automation, there is a need to trust the declared IT infrastructure state can be predictably controlled and reproduced. This requirement to build predictability, reproducibility and trust into IT systems is not new but has grown in both scale and complexity as IT departments went from a few servers blinking in the basement to the core around which every business rotates. 

    With every customer interaction and business function becoming more digital, companies must be able to rely upon their IT departments if they hope to continue to keep pace in today’s rapidly changing world. This article traces the evolution of IT operations from Pets towards X-as-code with each improvement bringing companies closer to declarative predictability. These best practices enable enterprises to reliably deliver core business value through their IT department and keep pace in the cloud native era.

    Yesterday: Pets Versus Cattle and Infrastructure as Code

    Back when the IT department was just servers in the back room, it was all hands on deck when anything went wrong because work ground to a halt. The CEO couldn’t get their emails and the sales department couldn’t book new orders. From these early days, it was learned that in order to trust IT with critical business services, high availability and reliability needed to be built into the system. This lead to the creation of the Pets vs. Cattle analogy where each infrastructure component is interchangeable and can quickly be replaced when anything goes wrong. This allows companies to quickly recover from or even avoid outages.  

    While treating servers as cattle solves some problems, as any rancher can tell you, handling a whole heard brings its own set of challenges. As the size of managed infrastructure grows, it can be difficult to keep track of where, when and how it is actually deployed. In addition, changes to production systems to keep them healthy can lead to configuration drift creating unique snowflake servers which are pets rather than cattle. It becomes a difficult task to predict how a system will react or rely upon it in critical situations when its actual state is unknown.

    To help eliminate this problem and control how infrastructure is configured and deployed, ITOps teams moved to infrastructure-as-code. In this way of working, applications and servers are never modified once deployed. If something needs to be changed, a new version is created and the old one is decommissioned. Each deployment can be versioned to create an overview of what is in production and how it is deployed.

    Managing cattle rather than pets and using infrastructure-as-code has helped companies move from ticket-based operations with long lived machines to more scalable and dynamic infrastructure. These practices build reusability, scalability, automation and versioning into IT systems to create a simpler, more predictable deployment and management process. By being able to declare the state of its infrastructure, companies are able to predict and trust the outcome.

    Today: Cloud-Native Infrastructure

    Looking at the definition of cloud native from the Cloud Native Computing Foundation, it quickly becomes clear why many companies are turning toward these best practices. They help companies build and run resilient and manageable infrastructure that–combined with robust automation–creates dynamic, yet predictable systems. Declarative APIs and immutable infrastructure will be used as two examples to understand how cloud native builds upon and expands the techniques covered above.

    Declarative APIs make infrastructure as code and automation possible. For example, with Kubernetes, operations teams are able to declare the desired state of the system and automate the reconciliation with the actual state until they match. It originally began with just container orchestration but has since expanded to cover many other types of cloud resources including the physical infrastructure itself through Custom Resources and Cluster API respectively. Declarative APIs make infrastructure management simpler and more reproducible by automating operations and eliminating errors in production. Declarative APIs can be leveraged to create immutable infrastructure.

    Immutable infrastructure goes beyond just servers as cattle and only treating the infrastructure as code. Immutable infrastructure covers every higher layer of the stack until each one is codified and managed like cattle. Only exact copies are deployed and they can be completely replaced at any time. If anything needs to be changed, a new deployment is made and the old ones are decommissioned. Immutable infrastructure works for the whole stack to eliminate variations leading to fewer deployment failures, consistency across environments, easy horizontal scaling and a simple rollback and recovery process.

    As these two examples show, cloud native enables companies to reliably scale out dynamic IT services without having to simultaneously scale out their operations team. Building declarative predictability into systems engenders trust in automatic processes, even when critical services are running on them.

    Tomorrow: X-as-Code

    While cloud-native best practices provide many benefits, applications and infrastructure are just the beginning of the story for enterprise IT teams. In today’s world, they are presented with the Herculean tasks to manage a dynamic and sprawling environment while still keeping in compliance with internal policy along with national and international laws. This challenge moves far beyond mere servers and treating them like cattle to compliance and governance across multiple clouds and hundreds or thousands of developers. With these challenges in mind, the question still remains on how to best deliver on these demands at the enterprise scale.

    Looking across architecture, configuration, blueprinting, policy and governance, if each of these is simply viewed as another layer of the stack above the application, their operations can be radically simplified. By applying the principles of immutable infrastructure and cattle to this further layer of the stack, each of these can be defined and managed using an X-as-code model. These types of management models can already be seen taking root across the cloud-native ecosystem.

    Kubernetes creates blueprints-as-code through the use of Admission Controllers while policy-as-code can be defined in Pod Security Policies and Network Policies. The CNCF even has a policy engine, OPA, as a sandbox project that gives operators fine-grained yet flexible policy control across their entire stack. By using code to decouple policy and other layers from the actual implementation, the resulting stack is easier to understand, flexible enough to handle future requirements and less expensive to maintain.

    At Loodse, our customers were struggling to manage configuration, services and policy across their clusters. Leveraging the Kubernetes controller paradigm in Kubermatic, we are able to automate the deployment and lifecycle of each of these add-ons. This gives users the ability to declare, enforce and automate the required state of not only the infrastructure but the configuration and policy, too. Kubernetes controllers bring predictability and control to dynamic and scalable cloud-native IT even as it begins to handle more and more mission critical services.

    Cloud Native in Production: Declarative Predictability

    With IT becoming the core of every business, bringing cloud-native best practices into production enables operators to predictably declare, run and manage their infrastructure at the enterprise scale. Being able to trust in these automated operations frees up resources and opens up many new possibilities. Developers can be given guardrails and become empowered to do their job rather than waiting in a ticket queue. Operators can easily create highly available self-healing infrastructure that meets governance standards allowing them to focus on delivering a platform with differentiated business value.

    At each step of the development of modern IT practices, from cattle to X-as-code, operational processes have become more streamlined allowing both developers and operators to drive more value to the bottom line. The future of IT infrastructure and operations will be a trusted platform that automates undifferentiated heavy lifting, allowing the whole business to focus on customer value.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Bill Mulligan

  • Survey: IT Striking Balance Between Innovation, Reliability

    Survey: IT Striking Balance Between Innovation, Reliability

    When it comes to managing IT, most DevOps teams are locked in an eternal struggle to balance innovation with reliability. There’s always a keen interest in being able to take advantage of the latest and greatest IT platform. However, new IT platforms don’t achieve the level of maturity required to build and deploy mission-critical applications overnight. Enterprise IT organizations often build an application on one platform and deploy it on another platform that is perceived to be more reliable.

    But a global survey of 505 IT and other executives published by the Cloud Foundry Foundation (CFF) suggests that, at a time when adoption of microservices based on containers and serverless computing frameworks is rising, the need to strike a balance between innovation and reliability has never been more acute.

    Conducted by ClearPath Strategies on behalf of the CFF, the survey finds serverless usage and container usage are now at 14% and 37% respectively. At the same time, adoption of platform-as-a-service (PaaS) environments such as Cloud Foundry has increased to 47% from 41% in the last six months.

    The study also suggests the rate at which organizations are adopting serverless computing and containers is starting to plateau. However, organizations that have adopted serverless computing frameworks and containers are making greater use of these capabilities as they continue to mature, according to the survey results.

    CFF CTO Chip Childers said it’s not uncommon for organizations now to deploy a PaaS and Kubernetes platforms alongside each other. In some cases, applications built using containers on a Cloud Foundry PaaS will be deployed on a Kubernetes run-time at the network edge. In other cases, applications built using development frameworks running on Kubernetes will be deployed on a Cloud Foundry PaaS that is perceived to be more reliable.

    Of course, the percentage of organizations that are employing a PaaS environment is not nearly as high as those using a less opinionated framework for building applications. However, whatever the path chosen, Childers said multiple types of abstractions that simplify IT complexity will live alongside one another for many years to come. At the same time, the underlying technologies that enable those abstractions increasingly will converge. For example, the Cloud Foundry PaaS was built before Kubernetes existed. Now the CFF is moving toward replacing much of the Cloud Foundry substrate with Kubernetes.

    According to the survey results, 65% of respondents place the most value on a platform that delivers a consistent experience. However, 59% also said a platform that integrates innovations is of the greatest value to their ability to develop software applications. In addition, almost two-thirds (63%) said a technology’s ability to integrate with their environment is the “touchstone” that makes technology better.

    As is often the case, what constitutes a touchstone is in the eye of the beholder. However, DevOps teams are not likely to sacrifice reliability no matter how cool that next big thing in IT may or may not be.

    — Mike Vizard

  • Kubernetes and Service Mesh With StackRox

    Kubernetes and Service Mesh With StackRox

    DevOps and Kubernetes enable organizations to “go with security,” adding security to the code instead of on top of it. In other words, security is built-in rather than bolted on.

    This is one of the aspects of security in the DevOps world with a Kubernetes system that Ali Golshan, co-founder and CTO of StackRox, explained during this exclusive interview with Alan Shimel, Media Ops CEO and editor in chief. The other aspect is security has become continuous instead of snapshots, which provides “a lot more visibility into various stages, various deployments, as well as running environments,” he said.

    During the interview, he explored containers, orchestrations, Kubernetes and service mesh.

    “Leveraging this cloud-native infrastructure, Kube and service mesh the right way reduces your dependency on third-party tools, allows you to hand off things you typically handed off to vendors and does that inside your infrastructure, which makes it actually more secure and highly scalable,” said Golshan. “You can automate that and reduce your lock-in and your sort of dependency.”

    Check out the interview below to learn more about the best and emerging practices in Kubernetes, service mesh and the security feedback loop.

    — Veronica Haggar

  • DevOps Chat: Holistic Kubernetes and Cloud-Native App Security, With StackRox

    DevOps Chat: Holistic Kubernetes and Cloud-Native App Security, With StackRox

    More than ever, application security is a top priority. Beyond secure coding practices, a holistic app security strategy addresses the full application and infrastructure stack. This includes containers, microservices, orchestration, infrastructure software and the cloud. Bolting on the next security tool may not be the answer.

    Kamal Shah, StackRox CEO, joins us on this DevOps Chat, diving into the need for a systemic security approach across the life cycle of cloud-native, even “Kubenative,” applications. We explore how Kubernetes and cloud-native apps bring access to rich configuration information, usage visibility, runtime context, inherent security controls and compliance. It’s a fascinating conversation that will open up new paths to secure Kubernetes and cloud-native applications.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, this is Mitch Ashley from devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today I’m joined by Kamal Shah, who’s CEO of StackRox. Our topic today is security in a cloud-native world. Kamal, welcome to DevOps Chat.

    Kamal Shah: It’s great to be here, Mitch. Thank you for having me.

    Ashley: Well, I really appreciate you taking time out of your schedule. I know you’re very busy. Happy to have you on. Would you start by introducing yourself? Tell us a little bit about you and what you do. Of course as CEO we can imagine, but also a little bit about StackRox.

    Shah: Absolutely. So again, I’m Kamal Shah. I’m the CEO of StackRox, and StackRox is a leading provider of security and compliance solutions for your cloud-native infrastructure. So as you’re embarking on your journey towards microservices, containers and Kubernetes, we help you address your security and compliance requirements across the entire application life cycle from build to deploy to runtime.

    Ashley: It’s about secure software, especially in a cloud-native containerized world. That’s for sure. Well, tell us a little bit about how you approach this problem? You know, how do you think about security in a kind of contemporary app development world where you’re doing cloud-native applications? Kub-native, I know is a way that we like to talk about it. How do you approach this from a StackRox standpoint?

    Shah: Absolutely. And so the big trend that we saw when we started the company was that customers were, you know, shifting their monolithic applications built on traditional infrastructure to microservices and containers and as a result security also had to evolve to meet and secure this cloud-native infrastructure. And we saw an opportunity to build a new set of solutions that are purpose built for this cloud-native infrastructure, because when you really think about it there’s fundamental differences between traditional application and traditional security and cloud-native applications and cloud-native security, right? I mean you’re deploying containers; security then has to make sure that it understands the ephemeral and immutable nature of containers and security has to shift left and provide controls and guardrails for the build and deploy side of the application life cycle, not just at run time.

    And equally important, security now has to seamlessly integrate with DevOps processes, automation and workflow, because if you don’t then the DevOps organ is gonna reject—or sorry, the DevOps body is gonna reject the security organ.

    Ashley: [Laughs] I’m glad you talked about it in both ways because Kubernetes, containers, that obviously changed the architecture of software and how we’re doing that in the cloud. You know, it isn’t just about locking down servers anymore. It’s really app security level, but at the same time we’re creating software differently. Can you say some more about what maybe some of the unique challenges are that you’ve had to address for both cloud-native world and also shift left within DevOps?

    Shah: Yeah, absolutely. I mean the one big trend, people talk a lot about microservices and containers, but the other big thing that I would like to call out is that Kubernetes has emerged as the de facto standard for orchestrators, right? We did a survey late last year and again six months later and what we found is the adoption of Kubernetes as the orchestrator has grown from 57 percent of organizations using it to now, you know, over 86 percent of organizations using Kubernetes, right?

    Ashley: Wow.

    Shah: And for the right reasons, because it is really, really hard to scale, manage, and deploy containers in large environments, and Kubernetes was purpose built to help you do that. It was an opensource system developed by Google, and Google today uses Kubernetes or a version of Kubernetes, the internal version, to manage over 4 billion deployments at 4 billion containers at scale. So the first thing here is that you have to appreciate the importance of Kubernetes and you have to make sure that security is focused on not just the container but also at the orchestrator and specifically Kubernetes, right?

    Ashley: Sure.

    Shah: And that itself brings a whole new set of challenges because Kubernetes is a large system with significant operational complexity, right? And they just went through a security audit and the assessment team found that the configuration and deployment of Kubernetes is nontrivial, and I can get into the details as to what those challenges are, but the starting point is to make sure that you’re not just focusing on security at the container layer but also focusing at the Kubernetes layer.

    Ashley: Well, isn’t that the truth. We find so often even outside of the container world it’s oftentimes configuration issues, not just the software itself. And as you mentioned, Kubernetes, well as great as it is, it is very complex to set up and operate. It’s not a simple task. Maybe it is as a developer to get an environment running but not in a multi-cluster distributed environment in the cloud, et cetera, high bred, all that kind of thing.

    Shah: They highlighted, you know, the three reasons why it is nontrivial. The first reason was that there are certain missing operational controls, right? So you obviously have to make sure that you understand what those are and put guardrails in place for those controls. The second is that there are a lot of components that have ability to put controls in place but the default settings are confusing, and the big takeaway for me is that we need to distinguish between security controls being available by default and that doesn’t mean that they are configured securely by default, right? So the misconfiguration is therefore the number one concern that users have and it was expressed by two-thirds of the audience in the StackRox State of Container Security Report that two-thirds of the respondents were concerned about misconfigurations.

    Ashley: I believe you even have that report or some information about that on your website. I remember downloading some of that. It has some great information in it. Talk to us about then how do you approach that from a StackRox product standpoint? What parts of those issues does StackRox come in and really help you solve?

    Shah: Yeah. So, you know, what we have taken is really a holistic approach to container and Kubernetes security and we have focused on making sure that our approach is both container-native and Kubernetes-native. So what does that mean? It means two distinct things. First is making sure that we capture all the rich configuration data that is in Kubernetes and use that to address and improve the efficiency of all the use cases that we address in the product. So let’s take vulnerability management as a very simple use case, right? You can go scan images and say here are a list of all the images with a vulnerability score of seven or higher.

    Now imagine you have hundreds of images with vulnerabilities and you give them to developers. What are they gonna do? And I say, well, they’re gonna throw their hands up in the air and say, “Well, I can’t get to all of them,” and that’s where we had the Equifax problem. But by deeply integrating with Kubernetes and being Kubernetes-native, what StackRox does is takes the additional step of saying, “Hey, you know, it’s great that we have these images with vulnerabilities, that’s a good, important first step, but let’s take the second and third and fourth step to understand are those images even deployed anywhere in your production environment?”

    Ashley: Exactly.

    Shah: Right? And if they’re employed in production environment, what data is it accessing, what secret is it accessing, what’s the network configuration, is it publicly accessible, what behavior are we seeing at runtime? And by taking all this rich context we can t hen give you a prioritized list of risks in your environment so you can focus on the most important risks first as opposed to getting a laundry list of issues. So that’s an important—you know, that’s how Kubernetes-native solutions can help you prioritize risk and that’s precisely what we do at StackRox.

    Ashley: I was just going to say, I think this is an extremely valuable aspect of your product because so many security teams—and I’ve talked to even, you know, major online tech companies that say, “I don’t need more information; I need problems solved, right? Tell me what are the biggest, most important things to go after,” and I think that’s a huge part of what StackRox does.

    Shah: Exactly. And, you know, alert fatigue, as I’m sure you’ve heard, is a very common problem in the security industry.

    Ashley: That’s the nice way to say it. [Laughs]

    Shah: Yeah. And by leverage the context, the rich context available in Kubernetes, you can actually make that information more actionable for your development teams and your security teams. So that’s one example. The other example of what it means to be Kubernetes-native is really to leverage the built-in controls available in Kubernetes for policy enforcement, right? And so let’s take network segmentation as another use case where you want to segment your networks between—inside—so that not every container can talk to every container, and there are two approaches. You know, Kubernetes has something called Network Policies and you can leverage those, and it’s much more scalable and robust and it seamlessly deploys within your infrastructure, right?

    The other approach would be to go deploy a proprietary firewall, which is fraught with scale challenges and creates issues if it fails, right? And so a customer again says, “Hey look, when I’m talking about being Kubernetes-native I want to leverage Kubernetes for what it’s designed to do, leverage the native capabilities, because it’s much more scalable and robust, and I want to make sure that my controls are all in Kubernetes, have a central place for policy enforcement as opposed to multiple proprietary tools doing policy enforcement,” which is not the approach and not what Kubernetes recommends and that’s precisely why they have those built-in controls. And so that’s again another example of what—you know, the advantages of being Kubernetes-native and it allows for native enforcement of your policies.

    Ashley: Well, I think that’s another place where, you know, the DevOps intersects with other parts of the organization, DevSecOps, et cetera. You know, a security team might be used to locking down a server or at a massive application level, and as you’ve mentioned sort of outside of the Kubernetes environment locking something down at the firewall, that doesn’t deal with really what’s happening amongst all the clusters, all the containers, microservices. So you’ve really highlighted an important aspect of security, that once security teams get engaged and find out what’s possible now they can, you know, do that through StackRox. That’s gotta be a huge advantage.

    Shah: Yeah, and precisely the reason why we took this approach, because when we go talk to customers what we find is that there are security teams that care about the controls that we are talking about here and then there are platform and infrastructure teams that are responsible for the operationalization of these tools, right? They are responsible for SLAs. The SREs are the ones that have to operate these container and Kubernetes-native security solutions and, you know, they just prefer solutions that are more native to the infrastructure versus deploying proprietary third party tools. And so having a Kub-native approach also helps align security and DevOps teams, and what we are seeing now is as a result that this emergence of DevSecOps, which is all about bringing the two together and really being responsible for the security of your application and your cloud-native infrastructure.

    Ashley: I’m curious. Who is it usually that’s reaching out to you or showing interest in StackRox? Is it the developers, DevOps folks, is it security folks kind of figuring out how to talk to the dev teams? What do you see happening in the market?

    Shah: Yeah, and a great question. We typically see security teams or security architects or application architects reach out because they know they have to address it, and so that’s where the initial conversation starts. What we find is that when they involve the DevOps teams, because they have to get involved to do a proof of concept or even to deploy an operationalized solution, the DevOps teams actually much prefer a container and Kubernetes-native approach as opposed to a proprietary third party security tool because they often refer to proprietary tools as a root kit that they do not want to install in their infrastructure, right? And so it is—you have to speak to both audiences.

    You have to make sure that you cover all the use cases that security teams care about and you also have to make sure that from a day-to-day operationalization of the product it aligns with your DevOps teams, their processes, their workflows.

    Ashley: I have to believe that’s gotta be a big plus to the security team. It’s not another security tool that they bolt on to the network. It’s something that’s built into Kubernetes and managed through Kubernetes and StackRox. It’s also native to the application infrastructure versus something they have to integrate in or yet another tool to watch. It just seems to be much simpler, although it’s a complex topic, but simpler thing to implement as well as manage.

    Shah: Absolutely. I mean you said it spot on. This is security that’s built in versus security that’s bolted on, and if you think about where the security industry is headed it’s all about security as code, right? Where you built it into your applications. People talk about infrastructure as code, which is what we’ve been doing for the past five years, and now we are moving to this security as code where it presents us with a unique opportunity to address security from the very beginning and forever change the security equation in our favor.

    Ashley: I’m curious. When we talk about shift left for security, I think bundled inside that, really what’s also shifting left, is auditing and compliance. Are there things about a Kub-native environment that help us with that as well, the auditing and compliance aspects of security?

    Shah: Yeah. Absolutely. When we think about compliance and, you know, shift left you want to make sure that your Center for Internet Security benchmarks cover both your containers as well as your orchestrator in Kubernetes. If you’re doing any kind of PCI or HIPAA or NIST 800-190 compliance controls you want to make sure that you extend them not just to your containers but also to your orchestrator in Kubernetes specifically. The whole shift left movement is that if I can catch an issue early on and prevent it from being deployed in my environment that is 100 times more effective than deploying it in my environment and then catching it when something goes wrong, right? So the sooner—it’s more preventative as opposed to, hey, we’re gonna detect it—let anything run in the environment; we’ll detect everything bad that happens.

    And, you know, and as we know that’s not effective. You have to have preventative solutions, you have to have guardrails in place that harden your environment so you have fewer chances of things going wrong. And that said, you also need capabilities at runtime, so despite your best efforts, when things do go wrong, you can still catch them and prevent any damage from happening.

    Ashley: Well, that’s the whole observability tracing into microservice, et cetera, understanding what’s happening in the environment. You have to know what the environment is in the first place as well as tie it back to that data.

    Shah: Exactly. And I think what we’re going to see is logs and metrics and tracing forming observability as you point out; and similarly, you know, security is just the other side of the coin, right? So observability and security go hand in hand. What we are doing is leveraging all of that rich information that is available at the node level, at the cluster level, and leveraging that to protect your applications across the entire lifecycle.

    Ashley: Excellent. Well, we’re coming up on the end of our time here. One last thought. Is there a certain best practice or something that you’ve observed as you work with customers when they begin engaging with StackRox and see your technology and what it can do for them? What is the thing that they walk away saying, “Wow, this has really helped me and I’ve solved the following problem or knowing the following has saved me a lot from avoiding some problems”?

    Shah: Yeah. So there are a lot of, you know, great resources available on our website at www.StackRox.com, and we talk a lot on misconfigurations and we’ve highlighted some best practices. For example, making sure that you use a read-only root file system, right? Because that’s a very powerful mitigation against attempts to gain foothold in your infrastructure and the adversary’s job becomes that much more difficult. Or minimize the image contents, because a lot of file systems often contain bash or package managers which further enable an attacker to do bad things, and so remove all nonessential binaries or pay attention to your Kubernetes RBAC configuration.

    Why? Because Kubernetes API is a critical attack surface. It controls what happens in your cluster and it controls what security configurations are in your cluster, and so if you minimize who has access to that Kubernetes API it prevents—it again hardens your environment, and also upgraded to the latest version because Kubernetes, the community, is constantly adding more and more security controls, they’re constantly addressing vulnerabilities that are discovered, and so running on the latest version of Kubernetes is also an important step that you can take to secure your Kubernetes environment. So there are lots of other, you know, best practices that I would call low-hanging fruit that we’ve called out on some blogs that we’ve written about, and that’s irrespective of whether or not you’re using StackRox, right? These are just some good best practices as you embark on your journey to containers and Kubernetes.

    Ashley: You do have an excellent blog and I would certainly recommend that to our listeners. Well, it’s been great talking with you. Kamal, thanks for being on the podcast.

    Shah: Mitch, thank you so much for having me.

    Ashley: It’s definitely been my pleasure. So I’d like to thank Kamal Shah, CEO of StackRox, for joining us today. You, of course, have listened to another DevOps Chat podcast and I’d like to, of course, thank our listeners for joining us today as well. This is Mitch Ashley with devopsy.kinsta.cloud. Thank you for joining us. Be careful out there.

    — Mitchell Ashley