Tag: containers

  • SoulCycle Flexes Its Muscles with DevOps and CI/CD

    SoulCycle Flexes Its Muscles with DevOps and CI/CD

    When SoulCycle wanted to improve application development and deployment, including improving services for customers across its 80-plus indoor cycling centers in the United States and Canada, the company turned to Mark Sost, a veteran developer and engineer who worked at several media companies, including Hearst.

    Over the last two-plus years, Sost, who now holds the title of senior director of engineering at SoulCycle, has built-up the company’s engineering and DevOps teams, increasing the headcount from eight or nine developers to about 40 employees now.

    The first major project was revamping a legacy, LAMP-based stack used to support applications to focus more on a modern platform that could take advantage of DevOps methodologies, as well as continuous integration and continuous delivery practices (CI/CD).

    Sost and his team dubbed this new platform Podium. However, Podium is more than a development platform, Sost explained.

    “Podium became the ethos and the methodology, as well as the architecture and application stack for re-platforming Soul’s technology stack in total,” Sost said in a telephone interview. “We moved from that monolithic to more of a service-oriented architecture. As such, we started to break out functionality on multiple applications.”

    The re-engineering also meant investing in new infrastructure, as well as other cloud-based technologies. This includes a multi-cloud approach.

    While most of the legacy applications have now been moved to Amazon Web Services, SoulCycle also uses Google Cloud Platform for Podium and all new developments. Sost also takes advantage of Kubernetes and Docker for container orchestrations, as well as investments in other managed services.

    Other tools in the DevOps arsenal include Jenkins to run unit tests; DataDog and StackDriver for monitoring and analytics for workflows; combing Travis CI, Cloud Builder and GCR for CI; and technology from startup Harness, which provides CD-as-a-service. Sost called Harness the “last-mile” tool needed for application deployment.

    “For Harness, that’s CD for us,” Sost said. “That’s continuous delivery to our various Podium environments. We’re leveraging it for delivery to all our Podium environments. Everywhere from UAC (user account control) during the code review cycle to QA after merge, to production, load test and everywhere in-between.”

    This revamped approach to application development now means that the SoulCycle engineering team has created approximately 30 different apps over the last 24 months, this includes internal apps used within the company environment, and ultimately to customers using the facilities and equipment.

    Additionally, Podium helps create a more unified architecture to help ensure that the entire team—whether it’s DevOps, QA or the production team—can paint a 360-degree view of the problem that needs to be solved and what the implications are for the business once the solution is put into place.

    This is what Sost and his team call “Podium principles.”

    “When we say that we’re building things with Podium principles in mind, we’re talking about the fact that everything is version-controlled,” Sost said. “We have full test coverage. We have everything in terms of Kubernetes manifests to infrastructure automation written down and version control checks it as part of source code. That applications are conforming to more of that 12-factor methodology and that we’re really focused really heavily on developer empowerment.

    “I think for us, that really is a shift that Podium has introduced; a move from writing the code as an application developer and then throwing it over a wall to an operations person—a QA person—to test it and deliver it somewhere,” Sost added. “Instead, we provide a developer the tools and the patterns of principles to go about controlling more of their application development life cycle all the way through to delivery.”

    The results have helped improve the overall SoulCycle experience for customers, which includes:

    • Reducing deployment time by approximately 80 percent.
    • Instead of one hour, deployments now take one developer 10 to 15 minutes.
    • QA verification has been reduced by approximately 80 percent.
    • It now takes three QA engineers one hour to verify, compared to three to six hours previously.

    At the same time, Sost believes that the Podium approach gives developers more control over their projects and lets them focus on writing good code that creates apps people want to use.

    “It’s more, ‘Now, I know what’s involved with getting my application to people,’” Sost said. “Because at the end of the day, engineers are just concerned with writing things that people want to use. … This is how we’re going to give you more control of the code and applications that you’re writing, so that you can see meaningful results quicker, without having to wait for someone.”

    What’s next for Sost and his team at SoulCycle? The answer: Machine learning and a bit of automation to the DevOps process.

    Here, SoulCycle is using Harness tools to log more data and infrastructure monitoring into the platform, which allows the algorithm to learn more about the various development projects. This can make the code better and eliminate errors that come with constant development and deployment.

    “We’re letting it get smarter over time around how it can detect anomalies and understand whether our code was successful because there are going to be times where things slip through unit testing or slip through integration tests and there are even going to be times where they slip through load tests, where things like 99 percent are just not going to cut it,” Sost said.

    — Scott Ferguson

  • The Changing Role of APM in a Microservices World

    The Changing Role of APM in a Microservices World

    Microservices usage has skyrocketed. Fully 91 percent of organizations are using or have plans to use microservices and 92 percent expect to grow their use of microservices in the coming year. It is no longer if but when: 86 percent of organizations expect microservices to be the default within five years.

    With the growing use of microservices and containers, applications have become orders of magnitude more dynamic. It’s not unusual for containers or building blocks of the application to have a shelf-life of less than a week. However, what is great for application development can create challenges when it comes to monitoring ­and diagnosing performance issues in ever-changing applications. How does application performance monitoring (APM)  change to support microservices and dynamic cloud-native apps?

    More Moving Parts

    Small, specialized microservices may facilitate shorter development cycles, continuous releases and greater agility, but when deployed across dynamic cloud infrastructure, the sheer number of moving parts also makes it much more difficult to troubleshoot application performance. Highly distributed and transient containerized components need to be tracked and measured continuously, demanding a big data approach for APM tools. To be effective, monitoring containers and microservices requires high-definition visibility across all transactions, machine learning to pinpoint problems and a shift in focus to users from servers.

    High-Definition Data

    Because the dynamic, containerized environment is constantly changing, measurement data must be collected at high frequencies—think seconds rather than minutes. Collecting infrastructure performance metrics at one-minute intervals, or longer, can distort the real situation and sometimes miss entire events altogether. For example, at low definition, a pattern of recurring but brief spikes in CPU usage will either be averaged out or not register at all—but if detected would be a tell-tale sign of a systemic configuration issue. High-definition infrastructure metrics, collected at one-second intervals, are crucial for a comprehensive view of resource utilization and shared resource dependency issues.

    However, infrastructure metrics alone are not enough. Building a complete view of the operating environment calls for data from multiple sources, including application traces, system metrics, application logs and network behavior. The resulting data set is very large, necessitating big data collection methods and processes. In-line data compression, NoSQL back-ends, non-intrusive client agents and cloud scaling are foundational technologies for modern APM tools.

    Correlating Data End to End

    Big data analytics and visualizations are necessary to process APM data into human-scale patterns and events. Identifying root cause of a performance problem often involves following a transaction from end to end, through multiple containers, shared resources and downstream infrastructure dependencies. Next-generation APM tools dynamically map the flow of every single transaction, through all intermediate relationships. Machine learning and artificial intelligence, while still in their early stages, are proving essential to finding patterns and anomalies in the data, suggesting probable causes and flagging them for human investigation. Combining the best of machine and human skills results in faster problem identification, diagnosis and resolution. IT operations teams benefit with improved application performance, reduced repair times and higher uptime even in dynamic application environments.

    Putting the Focus on the End User

    Legacy APM tools tend to focus on infrastructure components and resources, monitoring the health and utilization of servers, storage and networks. Organizations today, driven by digital competition and customer satisfaction, need to concentrate on the user’s perspective. This means monitoring multiple types of user devices for health indicators and mapping each transaction to the backend system. Automatic instrumentation delivers real-time end user metrics that trigger alerts much earlier than server-side monitoring, and enables more effective prioritization. End user insights also help DevOps and product owners track feature and application adoption rates, compare performance before and after upgrades and directly relate technical issues and investments to business priorities.

    Monitoring Apps for Today and Tomorrow

    Clouds, containers and microservices require a different approach to performance monitoring that takes into account the speed, size and life cycle of application modules. Effectively watching these distributed and transitory processes demands a big data approach that can handle the scale of end-to-end transaction data in modern enterprise environments.

    Legacy APM tools that only take snapshots result in an incomplete view of application reality and lengthen problem resolution times. Instead, modern APM systems collect a complete data set, enabling DevOps to quickly resolve intermittent and complex issues that impact business productivity and user satisfaction. They augment their skills with machine learning to quickly identify anomalies and guide them to probable causes. The results are better application performance, enhanced competitiveness and perhaps most important, increased user satisfaction.

    — Gayle Levin

  • DevOps Chat: Container Security and Aqua 3.5 with Rani Osnat and Andy Feit

    DevOps Chat: Container Security and Aqua 3.5 with Rani Osnat and Andy Feit

    In just a little more than three years Aqua Security has set its mark in the container security space. With its major new release of Aqua 3.5, the company has again raised the bar with serverless and container encryption upgrades and feature sets.

    I sat down with the Aqua Enforcer himself, Rani Osnat, and “Boston” Andy Feit to discuss the details of this major release. Rani and Andy give us an inside peek.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s chat is a little bit about cybersecurity, container security, Kubernetes and a major new release coming out of our friends at Aqua, Aqua Security. I’m happy to be joined with the dynamic duo of marketing at Aqua, Andy Feit and Rani Osnat. Andy, Rani, welcome.

    Rani Osnat: Hi Alan, thank you.

    Andy Feit: Hi Alan, good to talk to you.

    Shimel: And just so people know, Andy, you’re joining us from Boston today. And Rani, you’re out in Israel. Is that correct?

    Feit: Yes. Yes. Today I am in _____ _____ [Crosstalk].

    Osnat: Right.

    Shimel: Modern technology. We have a worldwide panel. But guys, the big news is, Aqua just announced version 3.5 of their platform suite of tools. And you know, Aqua’s not a company that every single new time there’s a new release, and the DevOps mantra, you can’t get too excited about any one release. Right? Because there’s always a next one and a next one and a next one. But this is one to get excited about, huh?

    Feit: Absolutely.

    Osnat: We certainly think so.

    Shimel: So why should we be excited?

    Feit: Rani, you want to take that one?

    Osnat: Yeah, I’ll take this one. So, with every release we make, especially, and in the beginning of course everything’s new. But we’ve been in this space now for three years. Which is not a long time, but in this space it’s a very long time. And so now we’re at the point where we have a lot of large enterprise customers using our product. And we have a market that’s looking for innovation.

    And with every new release we try to balance these factors of, you know, offering something new that the market wants. But at the same time ensuring that our enterprise customers can make use of our platform, as they themselves grow their cloud native container implementations. So there is a maturity factor here as well as an innovation factor.

    And so on the innovation side, we’re introducing a few significant innovations. First and foremost, risk assessments for serverless functions. Which is a, you know, a kind of a sideways expansion for us into the serverless technology space, in addition to containers. Thinking that you know, what we see as, it’s basically the same teams and the same benefits that are gotten from containers people expect to get from serverless, it’s just another means to get the same end.

    And so we want to provide our customers with all the controls they need to address any challenges they have around securing those technologies. And it doesn’t matter if they use containers or serverless or both, or any sort of mixed environment. We also added something that’s innovative in the space, which is container encryption, and we can talk about that.

    And then on the side of enterprise scalability and ease of use, we’ve added quite a significant I would say rearchitecting of how we manage both administrative controls on our platforms and what users can do in terms of access. As well as the policy engine to make it a lot more scalable for multicloud, multiteam, multiapplication use.

    Shimel: Got it. Got it. Andy, did Rani leave anything out you want to add?

    Feit: No, I mean, those are the big pieces that are in the release. As he said. I mean, it’s very much being driven by our customer base and where they’re headed. And you know, in some aspects, it’s the technology elements. Like adding serverless. And in other aspects, it’s really about living with the solution. You know, as our customers, we now have customers that are we think the largest container deployments in the enterprise.

    We have some very large users and span of different industries. As they look to roll out, they’re finding they have multiple teams working on these projects. And they need to implement different levels of security. And in many cases, they’re implementing that on different technology stacks. And some are using containers, some are using serverless environments.

    And they may be using even different underlying providers of some of the infrastructure. Whether that’s tools for development or the cloud provider itself. And so we’re really, we’re becoming very heterogeneous in terms of what we need to support. And for our customers who, on the security side, are trying to look at that whole context and manage that whole context and have consistent policies, all our consistent reporting across all of them, and not be monitoring 17 different dashboards. It’s important that it be easy to do, to be able to see that whole network of activity. And that’s really what a lot of this release is about.

    Shimel: So guys, one of the things that you mentioned, both of you actually mentioned, was the serverless piece. And this is something that we are, we’re hearing a lot about from our readers and from people we speak to. You know, how quickly, and we live in crazy times, right? So the whole container revolution, if you backed up on hypervisors. And now, how quickly serverless is you know, gaining a foothold and people are building around that type of infrastructure.

    Osnat: Right.

    Shimel: Let’s, just baseline, what are some of the security challenges that you guys are seeing around this?

    Osnat: Right. So I’ll preamble that, you know, just add one more thing. There’s, you know serverless has been around almost as long as containers have, in terms of the, you know, its current incarnation of the use in the cloud. But, the use cases is quite different. And while it is gaining traction, the use cases are much more limited than containers. There’s a whole kind of religious war, you know, between the proponents of containers and the proponents of serverless. Personally I don’t believe it’s a zero sum game. I think that both are going to end up being used, and both are going to end up being used in hybrid architectures.

    Shimel: Yeah, I don’t think it’s either or.

    Osnat: Yeah, I know, but some people would like you to think it is. I don’t think it is. I think it’s both. And so we, but there are some fundamental differences between containers and serverless when it comes to security. First of all, most of the serverless workloads that happen today are cloud based and specific to a cloud provider, right? Mostly Amazon, because Amazon is the larger cloud provider in general, as well as in serverless specifically with Lambda.

    But basically you run those functions and it’s quite cloud specific. That’s one area of difference. The other is of course that you know, these are very small single-function entities that can run for a fraction of a second. So when we talk about run time security for serverless, for example, there is a lot less to do there than you have with containers.

    Containers are, there are applications that can run, they could run for a minute but they often run for a lot longer than that. With something that runs for a split of a second, there is really only so much you can do when it’s already running. So a lot of the stuff you need to do is before it ever runs. You want to make sure that what you’re running is actually something that is secure, that passed your policy, and that you know about, right?

    So when you look at the challenges, one of those is actually discovery. What functions do you have in your inventory, and what are you using them for? The second is ensuring that what’s in them is secure and not vulnerable, configured properly. And that also includes permissions. So, when you define for example a Lambda function to run on AWS, there is a set of permissions you can give it.

    And depending on how you configure it, this can be very liberal or not. And of course you want it to be less, the least liberal you can. So as not to allow it to do anything outside the scope of what it’s supposed to be doing. So right now we’re primarily concerned, and that’s also where our customers are, with preventive controls. Basically ensuring that you are, that when you run something you actually know what it is. You know that you’ve approved it. And you can run it safely. That’s the first step in what we think is the right approach to securing serverless functions.

    Shimel: Fair enough, fair enough. And back to what you said kind of right at the get go with the either or thing. It was the same arguments about, would container replace hypervisor type of thing. And I think what we see is the majority of people want containers on hypervisor.

    Osnat: That’s true. It’s –

    Feit: Yeah, and I would also add that the data centers with lots of VMware and hyper-v haven’t gone away yet either. And they won’t. They probably will never entirely go away. New development today is cloud native development. Most of your mobile customer facing development.

    It makes sense to do that in the cloud and as long as you’re doing new development, why not do it on the latest and greatest. But, and the flexibility that you get from doing something on containers or serverless or a hybrid approach of those, makes it a lot smoother, easier, scalable. You get all these wonderful benefits. But we don’t see people going in wholesale and saying, I must clean out this data center now. So I think that –

    Shimel: _____ [Crosstalk] throw out the baby with the bathwater. And people –

    Feit: Yeah.

    Osnat: Yeah. So I think that’s one thing that we did. The other, I think very interesting and innovative feature that we’re announcing is container encryption. And I want to explain the use case. So we work with, you know, we work with many organizations. For example, we work, some of our customers are ISVs, who create software for customers to use.

    And that software is proprietary IP that they’ve developed and invested in. And they often want to make sure for those reasons as well as for reasons of governance, to make sure only the latest stuff is being used, et cetera, that they have some control over where it’s running and who’s using it. And so one of the things we provide here is an ability to encrypt, you basically encrypt the container image when it’s built.

    And then in order to run it, you need to decrypt all its contents, so there’s basically a key that the customer will have, the end user will have. And that will decrypt the image as it’s running. And if you don’t have that key, the image will not run. So the container itself once it’s running is not encrypted. But up till that point, it’s encrypted and you have to decrypt it in order to run it. And then of course this prevents the sort of, someone reusing or forwarding your software and you can imagine various use cases except the ISV one where companies would want this sort of deep security control over their containers.

    Shimel: Sure. Sure. And I think it’s just the, you know it’s funny, Rani. I think you said right in the beginning that over the last three years, you know, the progress that Aqua has made, and the progress the whole industry has made in securing their containers. Like everything that’s come before it, right, with each new release, with each light bulb, _____ security we see like more polish being applied.

    So, to me this idea of container encryption is you know, maybe first we have to concentrate on just getting the containers to work in a somewhat secure manner. But now we can start thinking about next level, next, you know, how do we turn it up a notch? Things like container _____. What’s the word I’m thinking. Like the standard, the, you know, the container encryption becomes ubiquitous over time?

    Osnat: I don’t know if it will become ubiquitous. I mean, certainly today you have a lot of encryption happening with TLS and all the stuff, all the pipes that handle containers. But as for the code itself, you know, even in highly regulated environments, I don’t necessarily see everyone encrypting everything. But certainly there are some crown jewels that you will want to encrypt.

    And it’s especially those areas where you’re not talking about reusable components, right? A lot of the containers that our customers use are reusable based on open source components, that’s fine. You don’t need to encrypt your NGINX webserver container, right? That’s, it’s identical to the same, other NGINX containers people use. But if you have proprietary stuff, that’s where you want to, or something that’s especially sensitive, whether it’s for any kind of security, confidentiality reasons, or for IP reasons. That’s when you would want to use something like that.

    But what you said actually, you know, brought me to the other topic I wanted to talk about, which is how the use of these technologies has matured with some of the more leading or pioneering companies that we work with, which is why we’ve invested in these scalability, manageability features. So I want to kind of explain what it looks like in a customer environment.

    So when we started out, and we started out kind of together with the market, the companies we were talking to, even larger companies, would typically have one or two teams that actually use containers, right? And so, you were talking about isolated projects. And that’s one level of doing things. But as we move forward, you know, now we have customers that have hundreds of development teams working with containers. They make thousands of images on a daily basis, because it’s all in a CI pipeline.

    And it all gets, it all goes into this large blender where it’s being spit out at the end, through Kubernetes or mesosphere, basically through an orchestrator to a variety of delivery platforms, whether it’s OnPrem or cloud or a mix of both. And if you look at it from an enterprise perspective, what they want is a way for them to apply security policies but maintain segregation of duties. Maintain compartmentalization and multitendency models that they have. So what we’ve done is really change both our administrative access model as well as the policy engine to enable that.

    So basically, you’re talking about people who use the Aqua platform. They have one installation of the Aqua platform. But they can basically distinguish between what different teams can see and do, what different roles can see and do, and how those policies apply.

    I’ll give an example. Let’s say that you have an application that is PCI DSS related. Handling credit card transactions. And so it has to comply with certain criteria that are requirements of the PCI DSS standard. You want, and you’re, but at the basis you’re using some basic elements like, I don’t know, MongoDB or MySQL. That same MySQL image that you apply a policy to within the PCI contexts will have a different policy than in non-PCI contexts. So having contextual policy is important.

    So that’s one area. The other area is who’s got access to what. So of course the people who handle the PCI application, the developers, the DevOps team, will need to handle certain aspects of this. But they don’t necessarily need to control the policy. That’s expected to be done by the security and compliance team. And the same token, they don’t need to access, or other teams don’t need to access that PCI application.

    So we’re talking about a matrix that really separates who’s got access to what and what they can do with that access, according to the specific application or geographic location, or what cloud it’s on or what registry it uses. All of these parameters. So you know, we’ve done this and we have, now both of these models combine to basically give you this matrix that is, that has you know, dozens of parameters on each facet to basically determine who’s got access to what, and who can do, whether it’s view access or edit access or only access to the logs, for example, for auditors and so forth. And so, it’s really a step up in terms of enabling these large deployments to work in an enterprise environments.

    Shimel: Absolutely. Good stuff. Guys, I want to _____ two other topics or areas I want you to hit on. One specific to 3.5, one specific to the market. And we’re coming up on the end of our time. So you know, we hear about immutable infrastructure, and you know one of the advantages of a container environment.

    So how do you deploy, you know, now you’ve got a bunch of customers. They want to upgrade to Aqua 3.5. Talk a little bit about how do you actually upgrade or run out this upgrade deployment. Is it a question of throw out what you have and just stand up new? Because Aqua itself is containerized. Or is, what, give us a little insight, nuts and bolts if people want to upgrade. Do you know what I’m saying.

    Osnat: Yeah. So our upgrade process is you know, it is simpler because we use containers, right? So there’s nothing to install. You simply run the new container. We do have backward compatibility with our previous versions, so when there’s an upgrade path for anyone using older versions of Aqua to this version.

    We actually, we’re the last, we don’t, I mean, we release software more often than what some of our customers would like to follow, in terms of their upgrade schedule, right? And we don’t force them to do that. But in the last version, the last major release, 3.2, we actually had a lot of customers move to that. So we’re in good shape from that respect.

    And they can of course upgrade to 3.5 when it becomes available at the end of the month. So there’s really no, there’s nothing in particular they need to do except of course, we always, always recommend doing it in a test environment first. And then once you see everything works, you move that to production.

    Shimel: Excellent. Excellent. So let me add –

    Feit: There’s one more thing now I was going to jump in with, which is, we’re sort of talking about as our organizations of scale and especially if they have multiple teams that are working on different platforms and policies that need to be implemented. In some of these larger enterprises, it’s become very hard to really even see everything and know about it and track, you know, who’s running what and where, and who are these owners. And so one of the things that’s new in the product is a pretty slick interface.

    Probably that’s the prettiest part of our product right now. The dashboard’s always looked great. But you can now go in and do what we call workload explorer. And visually view all of your running workloads, whether they’re on Kubernetes or Docker or OpenShift, and you can see everything that’s running across a large distributed runtime environment. And drill down into them, and you can start doing things like highlighting those that are vulnerable or high-risk, and those that are PCI compliant can be coded to jump out, and you’ll see that much more easily.

    You can then do tabular mechanisms. Begin to filter. Show them only those running on this platform. Let’s say that there’s a vulnerability that’s exposed in a certain component. You can show me just those, and start drilling down and looking at them, seeing where are they running? Is this one on-prem? Is this one on the cloud? Which cloud provider is it running on?

    So you know, as companies have become multiteam, multicloud, multistack, it’s the matrix of what you’re looking at has become pretty complex. And as workload explorers, it’s a great way to just sort of move around in a large enterprise. See what you have and understand what you need to work on right now.

    Shimel: You know, and let’s not trivialize it. Let’s not belittle. I’m having a tough time with the word. Let’s not belittle it. You’re a hundred percent right, Andy. Today’s organizations are multi everything.

    Osnat: I agree.

    Shimel: _____ _____ distributed on so many different levels, that really, having a 360 degree view of, just of the lay of the land, where all this, how it all plays together. That in and of itself, someone could make a product out of. I mean, it’s crazy.

    Feit: It’s eye opening for customers. Just in the last, we’ve been working with this product for about a month or six weeks, with some of our early adopter customers. And some of our sales engineers have been implementing it in proof of concept environments. And when the customers see this map, and they, you start drilling down, they say wait a sec – where’s that running? Show me that one.

    They’re finding things that they didn’t realize they had because you can now visualize it and see it. You know, when it was coming in as yet another alert or a text field or you know, something coming in to your SIM system, that was one thing. You had to look at the logs and know the query to type. But when you see it visually and you jump in and start looking around, and says wait, what’s that?

    And people were, it was eye opening. Okay, I need to go talk to that person. Literally, people were discovering things they didn’t know they had in their environments. And that’s exactly how, in a big environment, this tool can be very helpful in organizing things and showing you what’s out there.

    Shimel: Absolutely, absolutely. Guys, that last question for you is not really 3.5 related but more mark related. You know, we seem to be undergoing a, well, it’s constant. But recently we’ve seen just more and more consolidation in the tech vector general within containers, around Kubernetes, you know, IBM’s Red Hat –

    Feit: Cloud, right?

    Shimel: IBM’s Red Hat acquisition, $34 billion. I mean, is the king of it. But we saw for instance, CA or Broadcom CA, whatever they call it now, spin on _____ for near a billion dollars. To Tom or Bravo. We saw, who, someone just bought, was it _____ just bought a –

    Osnat: Later insight, yep.

    Shimel: Container something.

    Osnat: Right.

    Shimel: I mean, at some point, consolidation to what Aqua’s been out here preaching for three plus years. How do you think that affects, I mean certainly now, 3.5, you’ve raised the bar again. Let everyone else try to catch up. But in this atmosphere, talk to us a little bit. What does that mean around Aqua? Or do you just keep doing your thing?

    Osnat: Yeah, that’s a good question, Alan. I mean really, I think you keep doing your thing. Because your thing is understanding where customers are headed and building solutions that address what they really need. We’ve tried to keep our eye on that ball. And we’ve been big pushers of the concept of supporting multiple platforms, from the early days.

    We were Docker first, but today we support all kinds of things including things, less broadly used technologies. But pivotal, and Microsoft containers. Supporting all the cloud. So you keep your eye on that ball with the goal of building a customer base that is satisfied and loves your product, is excited about working with you.

    And one of the things that’s important to us is our customers are going to continue to renew with us and retain. You know, we’ve only been around three years. But we look at, our largest customers are very happy and using us, and expanding their use of us. They’ve put us in a pilot two years ago, and now they’re in production and they’ve put us in production with five applications. And now they’re on 50 applications. So this is all good. We’re not very focused on hey, we must be acquired by some time or we must be public by some time.

    It’s, let’s build the right products, continue to see adoption. We’re growing very rapidly, both within our base and customers deploying more, and new names, new logos, that are using our solution. And that’s what’s important to us. And the market, you know, the market will be the market. If somebody says wait, I can’t let these guys get that much bigger. I’m going to try to snap them up. That may happen. But it’s not something that we’re really focused on. We’re focusing on building the business organically in the right way.

    I have to say, we’re all old enough to have been through the dot com boom. And you know, the craze that went on back then. And I think it’s important to keep level headed. We’ve been around for three years now. We’ve just crossed the 100-employee mark. Celebrated that. You know, we work with our customers, we’re happy about that. We keep our heads, you know, leveled and we keep doing the work. You know, someone comes along and offers $35 billion to buy us, like Red Hat, I’m sure the board will approve.

    Shimel: They might think about it as –

    Feit: They’ll think about it a couple of days, you know. But in the meantime, we’re trying to keep things sane.

    Shimel: I hear you. Guys, just one last reminder while I have you both on. I’ll see you both out at the TubeCon Native, cloud native, Linux foundation event, along with Red Hat, right? Sponsoring a Monday conference on container security. I think we spoke about it last time. And I just, I don’t want to mess up the date, but I thought it was December 9th, or 10th.

    Feit: Nope, December 10th. So it’s the Monday right before KubeCon starts, and it’s collocated with the event in Seattle. And you can register on the CNCF site when you register for KubeCon, or if you already did you can go back there and add on one of these collocated events. Ours is very focused on enterprise security, so it’s called KubeSec Enterprise Summit.

    So for larger organizations, facing you know compliance requirements and deploying containers in production, it’s a great way to learn what others are doing. We have a great lineup of speakers. We have, JPNC is speaking. We have Tinder speaking. We have Starbucks talking about their journey towards secure containers. We have Liz Rice and Michael Hassenboss, who wrote the recent book on Kubernetes security. Which is not even published in paper yet, it’s still an eBook, but it will be first delivered in paper form at KubeCon. We’ll have them in our booth, and they’ll be signing books there.

    So it’s kind of a great event that people can get exposed to all of these technologies, and hear from a lot of different people. We ran an open call for presentations, so we have everything from technology to compliance to best practices. It’s going to be a great event. We’ve got a couple hundred people already signed up. We only have room for 300. So, if any of your readers do want to attend, they should do that soon.

    Shimel: Absolutely. Well, I’ll be there as well. We’ll be reporting for it. Andy, Rani, we’re way over time as usual. I ____ _____, we’ll need to call a wrap here. Congratulations on Aqua 3.5. I’m sure it’s going to be a great benefit, and the customer base will really appreciate it, and we’ll look forward to seeing you in Seattle.

    Feit: Look forward to seeing you, Alan. It’s always great to talk to you. And share what’s going on with your listeners.

    Shimel: Thank you. Andy Feit, Rani Osnat, Aqua Security. This is Alan Shimel for devopsy.kinsta.cloud, and you’ve just listened to another DevOps Chat.

    — Alan Shimel

  • Containers: All Your Base are Belong to Us

    Containers: All Your Base are Belong to Us

    The world is still changing. No surprise there, but the direction and velocity have been modified by the growth of container usage. Back in 2015, we at devopsy.kinsta.cloud showcased a survey we performed that said container usage was staged to take off. That it did, with a vengeance. No matter the category—pilot programs, test, production—it seems to have a greater penetration each year.

    And why not? There are a whole collection of benefits to containers—smaller footprint than VMs, more portable than cloud instances, more agile than … well, any of the alternatives. And the weaknesses—the use (and abuse) of IPTables and persistent problems with persistent storage—are being addressed at a rapid clip.

    But It’s Not All Roses

    Sorry to say that there is, as always, a fly in the ointment. We have to identify weaknesses in new architectures to plan for them and overcome them. And container infrastructure is not immune to this process. Nothing is perfect.

    The first problem is with overall security. Your container management system is now the center of the world. A single point of penetration that could yield dozens to thousands of infected servers. It is like a magnet to ne’er-do-wells. Protect it, lock it down, follow advice on Container Journal and Security Boulevard about how to keep your system as protected as possible.

    That was the easy one. We have a ton of experience protecting important central systems.

    The more difficult ones all revolve around your physical infrastructure and where it meets your virtual container infrastructure. Look at all the boxes in your data center and the services they provide, then map out how those services integrate with your containers. Once you’ve mapped them, work the mappings into your DevOps projects. Load balancing and storage provisioning are two areas in which what a traditional DC does is duplicated inside the container environment, and where that processing is to occur should be a conscious choice about what is best for the application, not a decision based on what is easiest/in front of you.

    It is an (arguably) bigger issue with storage. Just because the container can grow or shrink its internal storage does not make it the right place to position a fair selection of your data. Think about what will be needed in the future—with or without the application that this container or container set represents—and make a conscious choice about what is ephemeral data that can go away should an instance die versus what is needed long-term. Work those choices into DevOps processes. It can be a pain to access persistent data that is remote to a container, so make the painful parts automated via your DevOps processes.

    Both of these problems are compounded in most (but not all) cloud environments. Too much cloud functionality is designed to lock users in. While that has been a standard practice in IT for decades, the problem with containers is that they are inherently portable. Build them to be so. Yes, lock-in functionality always brings benefits, but mobility lets a team choose where best to deploy a container system with each inflection point (purposely being vague here—a highly agile/DevOps shop could have daily inflection points, while a single team just piloting containers in their DevOps environment may only have them on occasion when something large changes with the system). Should something change at a preferred cloud vendor that makes them no longer preferred, the fewer modifications required to move to another platform the better. So building default container/container management into the DevOps toolchain is preferred to building in vendor-specific options when a choice is available. Storage is an easy one to point at and say “choice might not be available.” Allocating disk is not very portable between platforms, but if that’s one of only a handful of things that must change, portability is still an option.

    In short, containers are coming, and they’re coming fast. Build them into your DevOps toolchain as portable solutions that automate the worst of the speed bumps. Make conscious choices about where infrastructure-based functionality should reside, and implement based upon those choices.

    And keep rocking it. If moving from any other environment (with the arguable exception of cloud), containers will be faster to spin up and to spin down. Just address the issues before they hit you, and all will be well.

    — Don Macvittie

  • How DevSecOps Can Help Avoid Catastrophic Breaches

    How DevSecOps Can Help Avoid Catastrophic Breaches

    Earlier this year, Tesla’s Cloud was hijacked and used to mine cryptocurrency, exploiting a vulnerability in the company’s Kubernetes cluster. A mountain of FedEx data was recently exposed, affecting 119,000 individuals. The Equifax breach garnered international attention after an estimated 145.5 million Americans were jeopardized. In other news, we’ve reported on how the Vine Docker registry fiasco was hacked, leaving an embarrassing PR trail in its wake.

    What is common across all these scenarios? They arguably could have been avoided with basic safeguards underpinned by a healthy dose of DecSecOps. In hindsight, let’s see what specific strategies developers can adopt to avoid such horrendous leaks.

    Basic DevSecOps Can Prevent Vulnerabilities

    DevSecOps is described as placing security at the forefront of every action. Within the world of cloud tooling, this means instilling protection at every point of the build life cycle. You would think this equates to advanced container and orchestration security, powerful access management and establishing hardened oversight for internal applications, right? Actually, many modern exploits simply involve a lack of basic password protection.

    Case Study No. 1: Tesla Cloud Cryptojacking

    In mid–2018, Tesla’s Amazon servers were hijacked by malware and used to mine cryptocurrency for rogue agents. Similar cryptojacking has transpired at Gemalto, Aviva and others. According to RedLock’s report, the hackers infiltrated the Kubernetes console, which was not password-protected. In one pod, they found access credentials to Tesla’s AWS environment. The breach exposed sensitive telemetry data, though was limited to “internally used engineering test cars only,” according to Tesla.

    How to Prevent

    Password protection for Kubernetes administrative consoles is a readily apparent lesson here. All cloud accounts, even if used internally, must be better-equipped and access credentials sequestered. Surely, instilling a “security by design” mentality can help limit the number of such consoles left accessible.

    Case Study No. 2: Vine Registry Breach

    In 2016 a white hacker by the alias avicoder was able to infiltrate Vine, the video-sharing social network. The hacker used tools to discover subdomains and behind one—https://docker.vineapp.com—found an open Docker private registry that housed the Vine source code. Such a vulnerability could have been used to collect user details or inject malware for malicious purposes. Thankfully, no users were compromised, and the Twitter Bug bounty program awarded avicoder a handsome sum for the discovery.

    How to Prevent

    In this particular incident, there was nothing at fault with Docker or Docker containers. The service (which was meant to be private) was simply left public and devoid of access controls. The lesson is that URLs on the world wide web are just that—publicly exposed to the world. It does not matter if no documentation links to them; such subdomains are easy to discover with crawlers and will be exploited if found.

    Case Study No. 3: Equifax Exploits

    In the Equifax breach, millions of customer records were stolen. The Equifax report notes that hackers exploited a “website application vulnerability.” Further reports detailed that the 2017 breach utilized flaws in Apache Struts, likely through a zero-day exploit, according to the Apache Struts Project Management Committee.

    How to Prevent

    Some note that the deserialization of untrusted data inherent in Apache Struts applications leaves some major vulnerabilities and potentially malicious code execution.

    Containers, due to their short lifespan, are not persistent, therefore making consistent hacking harder. They can also isolate functionalities so as to decrease attack vector. Regarding the Equifax breach, Aquasec speculates that container usage could have lessened the impact:

    “… it’s likely that such a breach would have been more difficult with containers, and that if successful, it would have been less persistent, not as widespread, and mitigated sooner.”

    Some view containers as a formidable force if armed with behavioral analysis and firewalls for network connections. Still, others note flaws in assuming containers are safer.

    Regardless of architecture, In DevSecOps, security for customer data isn’t drafted into the background; rather, it is imbued into the entire tooling process and holds equal footing within each build.

    Instilling a DevSecOps Culture

    Instead of a build-now, secure-later attitude, DevSecOps seeks to elevate security into every decision. With each build, security is melded with greater forethought regarding the repercussions for how data is treated.

    Admin consoles for Kubernetes clusters must be armed and password-protected, and the same goes for private GitHub repositories or Docker registries. Any traversal of data through cloud SaaS services must have authorization in place. A most obvious lesson is to password protect those “hidden” registries with discoverable URLs.

    Bounty programs for white hackers continue to lead to successful bug detection. At least for these institutions, there is less evidence of compromised user data and remediation is quickened.

    Much of this can be distilled by due diligence and through breeding a culture of security. While that’s a little fluffy for some, it apparently still requires repeating for institutions that continually de-privilege these basic safeguards.

    — Bill Doerrfeld

  • CA Technologies Report Highlights App Modernization Paradox

    CA Technologies Report Highlights App Modernization Paradox

    The rush to modernize IT environments using microservices, application programming interfaces (APIs) and containers may very well turn out to represent something of a double-edged sword.

    A survey of 1,087 IT and business professionals conducted by Frost & Sullivan on behalf of CA Technologies, a unit of Brocade, finds that organizations that have modernized their application environment are eight times more likely to achieve key performance indicators (KPIs) for the business. Four out of five respondents also said agile practices are helping them build what customers really need to improve overall customer satisfaction.

    On the downside, however, the study also finds more than half of respondents (56 percent) claimed that software issues can negatively affect the achievement of KPIs. A full 69 percent of all respondents say the pressure to release new applications or updates negatively affects quality and security, and 67 percent admit that their organization sometimes trades off security for application development and deployment speed.

    Nine out of 10 respondents who claimed to widely utilize modern application architectures also concede it’s challenging to manage and monitor APIs, microservices and containers, and that APIs and microservices add new risks for their organizations. A total of 80 percent of respondents also say they view continuous application security testing and DevSecOps as critical components of their strategy to help resolve those issues.

    Ashok Reddy, general manager for DevOps at CA Technologies, said that while organizations are clearly seeing a business benefit after embracing DevOps to modernize their application environments, the process hasn’t always been easy. The challenge they now face is that the rate at which digital business transformation projects are being launched has substantially increased. Most organizations are not going to achieve those goals in a timely fashion unless they modernize their application development and deployment processes, said Reddy, noting that only then can IT become flexible enough to meet the needs of the business versus continuing to try and force the business to bend to the way their applications work.

    The paradox many organizations find themselves wresting with is that making the business more agile requires a more complex approach to IT. In theory, each microservice deployed is easier to manage than a monolithic application. But as thousands of microservices get deployed, the number of dependencies between services becomes more than the average IT organization can track. In fact, all those dependencies is one reason more organizations are starting to investigate applying machine and deep learning algorithms within the context of an artificial intelligence (AI) model to automate IT management at scale. Clearly, most IT organizations are now on a journey that starts with a simple set of microservices and ends up fundamentally transforming enterprise IT, regardless of whether anyone likes it.

    It may take a while longer before the future of IT becomes a completely settled question. But for now, it’s apparent microservices in some regards are too much of good thing. The challenge is that from a digital business perspective, those microservices have become something most organizations can’t do without.

    — Mike Vizard

  • Semaphore 2.0 Brings Continuous Delivery to Teams of All Sizes

    Semaphore 2.0 Brings Continuous Delivery to Teams of All Sizes

    Since its inception in 2012, Semaphore has been focused on making code testing and deployment a quick and easy process. The cloud-based CI/CD platform from Rendered Text enables users to get up and running within a matter of minutes—which the company claims is at least twice as fast as other offerings.

    The company is releasing Semaphore 2.0 Nov. 6, which will expand the platform to enable continuous delivery, for a product that allows teams of any size to automate any continuous integration or continuous delivery workflow with custom pipelines so users can model their CI/CD however they prefer. The latest version also better supports CI/CD for containers, giving users the ability to use Docker or Kubernetes tools in building within container environments. Other features of Semaphore 2.0 include support for YAML files and command line interfaces, so teams can manage a large number projects more easily.

    Marko Anastasov, founding partner at Rendered Text, spoke with me about the latest version of Semaphore and his thoughts on the continuous integration and continuous delivery space at large. Read on for his insights:

    Can we start with some background about Semaphore? What was the catalyst for creating it?

    Our company was an early adopter of agile software development practices, including test-driven development and continuous deployment. We relied on cloud-based products like GitHub, which were simple to use and made software development easier.

    However, when it came to implementing continuous integration, we were out of luck. I remember spending days setting up Jenkins for just one of our projects. We saw how expensive and time-consuming it would be to make it fast and scalable. It felt as if by continuously verifying our code, we were doing something wrong. Developers need to set up a fast and functional CI process in minutes, and not worry about how it works.

    What’s new in Version 2.0 that’s worth mentioning?

    Semaphore 2.0 is made to enable continuous delivery in organizations of any size. Previously, like other hosted CI/CD services, Semaphore focused on solving the integration process: building code and running tests. Delivery features were limited and tailored for classic web development.

    This time we asked ourselves, What would a continuous delivery product focused on speed and simplicity look like in the age of cloud-native computing? I’d highlight three things that stood out.

    First, for DevOps transformation to succeed, we need to automate, visualize and accelerate all software development processes in our organizations. So at the heart of Semaphore 2.0 are infinitely customizable code pipelines, configured as code, run as a service.

    Second, infrastructure should be ephemeral and scale in real time. When CI/CD capacity is limited, developers are blocked in busy times of the day. Semaphore adopts the “pay only what you use” cloud model, in which CI/CD resources scale automatically to support the team’s actual needs.

    And third, we can’t manage what we don’t measure. It doesn’t make sense to prescribe the same metrics for every project and organization. So Semaphore lets users choose their own metrics and define their own dashboards.

    What feature or features do you believe will have the most impact with users?

    In building Semaphore 2.0 we were very careful not to add any unnecessary features. All parts are designed to support each other and provide a harmonious user experience.

    For example, Kubernetes wasn’t the first solution to run software in the cloud, but it made vendor-neutral cloud computing as easy as describing what you need to run. Similarly, developers can now describe any CI/CD process they need, and Semaphore will run it immediately at any scale.

    In day-to-day use, I expect users to like how much control they have via command line and autoscaling resources. It’s the overall boost in productivity that will make the most impact.

    How important is Semaphore’s increased support for container environments?

    Containers are the biggest change in the development stack in recent history. In the transition to container-based software delivery, many teams needed to maintain their own CI/CD infrastructure, since none of the existing solutions were flexible or easy enough to use. This takes valuable time away from delivering value to customers. Semaphore 2.0 is built to support the next-generation of cloud architectures based on containers.

    What are some of your thoughts on the current state of the CI/CD market, and where do you see it heading?

    For the continuous integration part of the equation, there are many options to choose from. There’s a wide spectrum from managed to DIY solutions, and from specialized vendors to those who provide basic tools for every aspect of software development. I think the shift that’s going to happen is from basic automation to optimization. For example, Semaphore provides bare-metal performance at convenience of cloud. When a customer tells us that they’ve combined that with our test optimization features to reduce build time from hours to minutes, that makes us so happy because we know what tremendous difference that makes in their workplace.

    On the continuous delivery front, I think there’s less maturity and that the market is still developing. So far, developers have needed to build their own solutions based on complex open source tools and self-managed infrastructure. Cloud vendors have entered the space, not with complete products but low-level tools that leave a lot of complexity for the end user to manage. The next phase will be about making the entire CI/CD process easier and more accessible. At Semaphore, we’re committed to making that happen.

    — Charlene O’Hanlon

  • IBM and Red Hat: Now, That’s an Interesting Combination

    IBM and Red Hat: Now, That’s an Interesting Combination

    Like everyone else, Sunday night I got a surprise with the pre-announcement of IBM’s intent to acquire Red Hat (I might have seen it a little earlier than the rest of you, as a friend from IBM shared it as soon as it was public). I’ve been mulling this over and checking out the information being provided thus far, and have some early thoughts:

    • No matter what, some people aren’t going to like this.
    • Very much like Microsoft purchasing GitHub, you’ll see negative knee-jerk reactions.
    • Early indicators are that it is good for everyone.

    Of course, since the acquisition isn’t expected to close for about a year, all we have is early indications, but they do seem positive.

    Fear for Open Source

    IBM goes a long way to assuaging the instant fear that many will have. From the press release to the investor relations call, the company makes it very clear that Red Hat will operate on its own. IBM stops short of saying, “wholly owned subsidiary,” but is keeping all that makes Red Hat, well, Red Hat. That will likely change over time—it does for any acquisition—but for the near future, Red Hat’s strategies and values will continue. So those people who will inevitably equate this to the death of open source can be safely shushed or ignored.

    Customer Concerns

    Red Hat knows that there is an open source equivalent to everything it does, and as such it treats those willing to fork over money either for support or more polished products as if they could walk away tomorrow. There is nothing in any of the released documentation to suggest this will change. In fact, “keeping Red Hat’s strategy” specifically was mentioned to investors, and this world view informs all of their strategy. One Red Hat person I worked with years ago put it: “We need to give them a reason for paying us when they don’t have to. That’s where support and service come in.” In the short term, that does not appear to be changing.

    Cloud Boom!

    IBM is right, the future of cloud is in two narratives: one of cloud portability and another of on-premises for stuff that just isn’t that portable. IBM plus Red Hat is in a unique position to serve those two markets and offer up DevOps toolsets based on standards that will spin up whatever, wherever. Between them, they work with every major public cloud vendor, support containers and have fingers deep into the private cloud space. If their predictions that workloads will continue to move toward on-demand is accurate, it will be difficult to find a better solution set than the two can offer together.

    So Overall, it Looks Good

    I was in discussions on several social media platforms over the course of Sunday evening about this, and the consensus seems to be that it could be far worse. If Red Hat were for sale, think through all the companies that would see benefit in owning Red Hat, and could afford the price tag IBM paid. There are not a lot that could be counted on to “Let Red Hat be Red Hat.” Over the long term, IBM can’t either, but that is simply business. In the short term, it is reasonable to expect that not much will change except Red Hat customers get pitched more IBM software and gear, while IBM offers Red Hat standard with their other offerings. Other potential purchasers would likely completely fold Red Hat into their business, thereby destroying what makes the company unique. So if Red Hat was going to be sold, the face the two companies are putting on it is the best we could hope for.

    Ops Side Love

    Red Hat has done a good job of integrating containers as they came along, and Ansible provides decent configuration support for a variety of deployment targets. Merging that with IBM’s cloud work should offer a streamlined DevOps pipeline that can handle whatever target you want it to configure. Oh, certainly there will be speed bumps along the way—we do live in a time of change, which brings them along—but portability plus programmed configurability will (as IBM has hinted) equate to prequalified stacks that can then be dropped into deployment toolchains.

    Not too Long to Wait

    While the official merger will not be consummated until late next year, expect to see the two organizations use their longstanding partnership to begin exploring cross-company cooperation before that. We should see the first fruits of this arrangement relatively quickly, and unless Red Hat’s investors find some reason to vote down a HUGE raise of their per-share stock value, expect that the cooperation will continue to ramp up until the acquisition is completed.

    If Red Hat Stays the Same…

    Given that IBM has said Red Hat would stay the same for the foreseeable future, that means all you are doing with the company can keep going, and you can pick and choose which of the partnership options that are bound to come along pre-completion you wish to take advantage of. That’s a benefit to the customer. So grab what you like and keep making your DevOps environment better.

    — Don Macvittie

  • DevOps and Containers: A Network Divided

    DevOps and Containers: A Network Divided

    The growth of DevOps and containers has created some interesting split responsibility scenarios. Not that this is necessarily a bad thing, but it is something that bears watching and fixing when it becomes problematic.

    Today’s topic is networking. Plenty of bits have been spewed about the situation, but it persists and is even getting more complex as architectures diversify.

    Specifically, today’s topic is about the dual responsibility built into today’s networking. Generally speaking, network operations staff are still managing the complex hardware and software environments that deliver packets to your systems. What they do and how they do it is changing—but at a much slower pace than areas fully pulled into DevOps. There are many reasons for this, and I won’t delve into them here, but the truth is that managing a production network and dealing with hardware vendors bring unique responsibilities and constraints that aren’t there in a software world.

    That’s experience talking, from a developer who learned network administration. Software is easier and more adaptable, and that is directly reflected in the impact of DevOps.

    But we have drifted into a world in which that infrastructure is still mission-critical and still has those same constraints, while the software layer of networking—driven primarily by containers—has grown up in parallel with a different set of people often responsible for it. And that software network can be mighty complex. I shudder to think about what the iptables look like on some—maybe hundreds—of your machines.

    Cloud has much the same effect, but since cloud instances often are publicly exposed, they tend to have networking people involved to make sure routing is correct and verifiable, and to help InfoSec people determine that the instance is as locked down as possible (and we can all name companies that wish this were true).

    Still, the complexity of cloud spawned companies including CloudCoreo (now a part of VMware), which automate a bunch of network verification and security checks just because there are so many points that require touching. So cloud isn’t the nirvana of shared responsibility either. Yet.

    As DevOps and container adoption has grown, it is natural that responsibility for container/container management networking would have spilled over into the development or application administration side. After all, once a subnet is configured for a cluster and software is managing the subnet, which is integrated with the rest of container management, it feels right to have the Dev/deploy team manage it.

    And it might be. Like all things DevOps, this is a continual improvement exercise. Having someone who knows exactly what is going on between pods/clusters/networks is important, though. Breaks can cause outages that cost money. Knowledge is the key to fixing breaks and container networking is generally super-complex. Even if you are providing standard IP addresses on a corporate subnet to each container, container management networking is still complex and can still break.

    So if you haven’t already, get the networking team involved in inter-/intracontainer communications. Make certain you know what is going where and what should go where.

    You won’t regret it, when things inevitably break that cannot be repaired by dropping the instance and spawning a new one to replace it.

    — Don Macvittie

  • DevSecOps: Helping Contain the Security Problems Containers Introduce

    DevSecOps: Helping Contain the Security Problems Containers Introduce

    The idea of building security into applications as they are developed versus bolting it on at the end always made sense but proved difficult to pull off in legacy environments. Monolithic software was too complex and the teams were disparate. But the arrival of containers 1) demands the need to attack the problem earlier because the attack surface changes so rapidly, and 2), thankfully, vastly simplifies the security process when DevSecOps principles are employed.

    The DevSecOps movement, which builds on DevOps by stirring in security, has been steadily building momentum as containers shoulder increasingly critical roles in larger organizations. Heck, DevSecOps even has its own manifesto.

    Although not all organizations adhere to the approach spelled out in the manifesto by DevSecOps.org (which says it was founded by “security practitioners dedicated to the science of how to incorporate security within agile and DevOps practices”), the core elements are the same: The goal is to shift left, as some people say, meaning start the process of securing applications earlier in the development process, improve communications, and repeatedly test along the way.

    Adding the Sec into DevOps

    Obviously, DevSecOps builds on DevOps, which is all about development and operations teams working more closely together to focus on the full life cycle of applications rather than one team focusing on the beginning of the cycle and the other on the end. DevOps is about breaking down boundaries of responsibility and modifying processes so the output is more resilient and code can iterate faster.

    DevSecOps is all that plus security. “DevSecOps requires changing mindsets, processes and technology,” Gartner says. “Security and risk management leaders must adhere to the collaborative, agile nature of DevOps to be seamless and transparent in the development process, making the Sec in DevSecOps silent.”

    The need to embrace the new approach is necessary because containers upend so many conventions.

    For example, in the DevOps world security tends to be viewed as a tech thing. “Are we encrypted?” “Yes.” “Great. We have secure connections. That’s all we need to do.”

    As containers become used for more mission-critical things, that approach is too limited because there are so many more moving parts.

    Before you might have had to worry about securing a few physical servers and a few dozen VMs supporting an application, and now that application requires you to secure 200 to 400 containers, spread across a couple dozen virtual machines in both AWS and Google Cloud. The amount of work has multiplied by 10. The older security methodologies just don’t scale.

    What’s more, there are more layers to worry about—your container runtime (Docker, RKT), your orchestrator (Kubernetes, OpenShift) and your build environment (Jenkins).

    And with containers we release things faster. It isn’t once per quarter or even every month. With containers and microservices architectures and continuous integration/continuous development, we are pushing new versions of software into production on a daily or weekly basis. Existing security practices can’t keep up.

    How Do You Do It?

    There are a number of ways to realize DevSecOps, but typically organizations aren’t going to fully integrate security personnel into DevOps groups, or require developers to become security experts or security experts to become developers. That’s what Gartner was getting at with keeping the Sec in DevSecOps silent.

    The movement is all about building processes and inter-team communications and workflows, which is also how DevOps started before it become more of a delineated role/persona. Yes, developers will have to learn more about security. Yes, security team members will have to learn some particulars about containers. And, yes, workflows will need to morph.

    But the good news is that once experts from the various disciplines agree on what is important, many of the DevSecOps processes can be automated.

    When to Test

    Job one is figuring out where in the container development process code should be tested for security. The goal is to set up a series of gates developers have to go through to make the resultant code more secure. Typically—and most critically—it starts with scanning source code repositories.

    So, before an app is even built, the code the developers are writing is being checked. Are they referencing a package with known vulnerabilities, or pulling in a library that has a vulnerability or is open source and against company policy? This a workflow change that starts the security process much sooner than traditional approaches.

    The next check in the cycle typically comes when the container image is being built, say in Jenkins, the popular open source build environment. When the image is being pulled together, packaging the code up into a container and defining dependencies, a scan is done to ensure it is free of vulnerabilities and is configured to meet the best practices and policies defined by the organization.

    And finally, when the image is pushed to the registry, the image is scanned yet again before it can be put into production.

    All these checks are tied to your CI/CD pipeline and just become part of the developer’s build process. The scans themselves are automated.

    “The goal of CI is to provide rapid feedback on disparate code changes, allowing developers to correct errors as soon as possible by identifying functional defects introduced into the larger code base,” according to eSecurity Planet. “In this environment, integrated security testing is needed to provide developers a real-time threat assessment of all changes they’ve made, regardless of their operational success in the larger code base.  Without integrated security testing, there’s a risk of re-engineering solutions multiple times to address security threats detected long after functional solutions are accepted.  That wastes valuable time, money, energy and effort.”

    Gates = Slow?

    Developers traditionally have viewed security as a necessary evil. They would build something, get it in staging, test it and then right before it hits production, some security review would identify 10 things that need to be changed.

    Testing more often, earlier and using automated scans vastly reduces the chance there will be significant problems with the resulting application. So, while the idea of introducing testing gates into the development process might seem like something that would put off developers, the end result will more than make up for any inconvenience along the way. Instead of huge changes at the end, there are incremental fixes at different phases of development.

    Companies are adopting many of the DevSecOps practices today even if they aren’t calling it that. DevSecOps is, after all, more of a process and a way of working rather than a specific role. Those practices enable organizations to move faster and improve the quality of the services they deliver from a risk, security and compliance perspective.

    Of course, there is no such thing as 100 percent security, and vulnerabilities will still crop up and be exploited. But even here a DevSecOps culture can be a huge step forward for container environments.

    In legacy enterprise environments, when a vulnerability is found in something that is running live, the organization would identify it as critical and have a service level agreement that called for it to be patched in two weeks. The patch needs to be developed and the deployment planned, and probably includes some amount of infrastructure management and updating.

    With DevSecOps and container environments, you can make the change in development and push it out within a day.

    So, even as containers change the security calculus by introducing more and new things to worry about, with the adoption of DevSecOps best practices organizations should come out being more secure and fleet of foot.

    — Loris Degioanni