devopsy.kinsta.cloud

Tag: docker

  • How to Isolate Development Environments: Vagrant vs. Docker

    How to Isolate Development Environments: Vagrant vs. Docker

    “It works on my machine.” This line has become an excuse often overused by developers to escape the burden of having to investigate problems experienced by end users that they are unable to replicate on their development machines. While this may excuse developers—after all, they didn’t experience the problem during development and therefore couldn’t have anticipated it happening to some users—it doesn’t solve the problem.

    Sometimes, issues are caused by end user mistakes, such as skipping a step in an installation process. Often, however, the problem arises from some disparity in the development and server or user environments. With the vast assortment of machine configurations in use—different operating systems, hardware specifications and software versions—even the slightest differences in development and production device configurations can cause dramatic differences in the behavior of many applications.

    This can be a serious issue, as it requires members of the development and operations team to identify the root cause of the problem, which potentially introduces unexpected and often significant delays in deployment and production schedules. Especially in the DevOps culture where rapid development, frequent deployment and highly dependable releases are of highest priority, such unexpected delays are completely unwelcome.

    Thankfully, there are a number of tools that have emerged to address this very problem. In this article, we take a close look at two popular tools with two different approaches to solving the problem of environment disparity: Vagrant and Docker. Like many DevOps tools and technologies such as continuous integration, these tools work well with all types of project setups, including IT outstaffing (dedicated development teams). Let’s compare these two solutions and see which types of scenarios each one is best fit for.

    Vagrant vs. Docker

    Vagrant offers to solve the aforementioned problem by enabling developers to easily create and manage virtual machines (VMs) in which they can develop, test and deploy applications. By using a VM, a developer or DevOps engineer is assured that as long as the application is run within the virtual machine, its behavior will be the same regardless of the actual hardware and operating system of the computer that the VM is running on.

    Docker offers a similar solution, but uses a different technology. Instead of using virtual machines, it uses containers, which are similar to Linux Containers (LXC) that are included in Linux systems. Containers serve a similar purpose as virtual machines, but at a much less extensive level and, therefore, are much more lightweight. Rather than isolating an application in its own virtual computer with a full operating system, containers simply isolate an application inside an environment with the minimum components necessary to run the application. This includes system libraries and tools, settings and configuration files and a runtime environment. Compared to virtual machines, containers consume much less resources from the actual machine on which it is being hosted. Docker has a comprehensive documentation library that includes information on what Docker and Docker containers are, how to install Docker and how to execute a Docker run.

    Vagrant and Docker share some common goals. For instance, in software development, both tools enable a more streamlined and consistent software development and deployment workflow by eliminating the need to worry about the application behaving differently in different machines. Moreover, it offers some degree of security to the host machine: Since the application is isolated from the rest of the system, issues caused by one application should remain within the scope of its environment and should not break the rest of the system.

    There are, however, a number of important differences between Vagrant and Docker that make them suitable for different types of applications. Let’s examine more closely how these two differ.

    Scope of Isolation

    While both Vagrant and Docker wrap applications into isolated environments, they differ in the degree to which these applications are isolated—or, effectively, the scope of the isolated environments. For Vagrant, the scope is the entire operating system: An application can access any and all components of the full virtualized machine and operating system it is being run in. For Docker, on the other hand, the scope is much narrower, as an isolated environment is limited to a single application or service: Each container is created specifically for an application. Effectively, this implies that with Vagrant, a single isolated environment can run multiple applications at once, while with Docker, running multiple applications will typically require the use of a separate isolated environment for each one.

    Resource Consumption

    Since Vagrant virtualizes an entire machine with a full operating system, it naturally has a high overhead cost and consumes a lot of resources from the host machine. With Docker, on the other hand, an app within a container shares the same kernel—the core module or component of an operating system—with both apps in other containers and the host machine itself. This enables containers to have much less overhead and subsequently consume much less resources from the host machines than do virtual machines. In effect, a host machine running apps with containers can run up to three times as many applications as it can with virtual machines. Moreover, since launching apps within containers won’t necessitate having to wait for the virtual OS to boot up, launching apps can also be substantially faster with Docker than it is with Vagrant.

    Limitations

    Vagrant and Docker also have their own limitations. Docker, for instance, is currently natively available to two operating systems: Linux, where it uses the Linux Kernel, and Windows, where it uses the Windows kernel. Docker is also available for MacOS, although it runs Docker containers on top of a Linux virtual machine. Moreover, Docker has some security issues—its use of containers make it less secure than Vagrant. Since applications within Docker containers share the same kernel with each other and with the host machine, it’s possible for a malicious program within a Docker container to affect other containers and even harm the host machine itself.

    While Vagrant, which uses virtual environments, is much more secure in this regard, it is limited in that it is meant only as a tool for development and generally not recommended for use in production environments. In other words, it is meant to provide a convenient way to virtualize the environment of the server to which an application is eventually going to be deployed, but once in production, the application must be run directly on the host machine. On the other hand, Docker provides a consistent environment for applications throughout the entire software development workflow, from creation to production.

    Conclusion

    Being able to isolate applications into their own controlled environments has proven to be extremely useful for software development, as it eliminates many hardware and software incompatibility problems that applications may run into when executed in different machines. This is particularly useful in DevOps environments, as it streamlines the whole software development workflow by eliminating sources of unexpected problems, thereby potentially shortening development cycles.

    We have looked into two popular tools for environment isolation, Vagrant and Docker. Both isolate applications from the host machine, although to varying degrees and using different technologies. Vagrant uses virtual machines, which can support multiple applications at once, but can be heavy on resource consumption and is only recommended for use during development. Docker, on the other hand, uses containers, which support only single applications, but is much lighter in terms of resource consumption and can be used both during development and during production.

    As such, these two tools cater to different business needs. If you require the development of a complex system of multiple applications interacting with each other, using Vagrant may be beneficial, but if you’re aiming to create a single portable application and want to minimize resource consumption, then Docker is the way to go.

    — Myron Monets

  • DevOps Chat: Aqua Security Seeks To Secure Containers

    DevOps Chat: Aqua Security Seeks To Secure Containers

    The migration to container-based application infrastructures is accelerating at a faster pace even than we saw with hypervisor-based infrastructure 18 or so years ago. Hand in hand with this is the race to provide better security solutions that are native to and purpose-built for containerized environments. Aqua Security has established itself in the container security arena, recently raising $25 million in capital funding as it seeks to lock down the container security market.

    In this chat, we sat down with Rani Osnat of Aqua Security to discuss the recent funding round and the state of the container security market. Below is the streaming audio of our discussion. Below that is the transcript of our discussion.

    Audio

    Transcript

    [su_pullquote align=”right”]Rani Osnat[su_custom_gallery source=”media: 1739530″ width=”100″ height=”100″ align=”center”]
    Rani Osnat has more than 20 years of enterprise software industry experience, in project management, product management and marketing, including a decade as VP of marketing for innovative tech startups in the IT security and cloud arenas. Previously Rani was a management consultant in the London office of Booz & Co. He holds an MBA from INSEAD in Fontainebleau, France. Rani is an avid wine geek, and a slightly less avid painter.[/su_pullquote]

    Alan Shimel: Hi, everyone. This is Alan Shimel for devopsy.kinsta.cloud and Security Boulevard, and welcome to another chat.

    Today’s guest is Rani Osnat, VP of Marketing for Aqua Security, one of the leaders in the container security market, and Rani, welcome to our DevOps Chat.

    Rani Osnat: Thank you, Alan. Thanks for having me.

    Shimel: Thank you. Thank you for being here. So, Rani, the big news out of Aqua recently is, by anyone’s yardstick, a large raise of capital to fuel Aqua’s continued growth.

    Osnat: Yep, that’s correct. We raised $25,000,000.00 in a series B round led by Lightspeed Venture Partners, which is a very well-known tier one VC.

    Shimel: Yep.

    Osnat: And, it’s—well, to the best of my knowledge, it’s the biggest funding round in our space in continuous security to date, and it also brings total funding for us to $38.5 million.

    Shimel: That’s fantastic.

    Osnat: So, we’re, I think, in a pretty good position to capture what we think is a tremendous opportunity in this market.

    Shimel: Sure. And so, Rani, we were talking off mic prior to recording today—you know, beyond the sheer magnitude of the number, right, 25 is a big number, 38 is a bigger number—but beyond that, there’s also the validation. Which is, you assume a company like Lightspeed Ventures, Microsoft Ventures is another investor in the company, Shlomo Kramer, of course, from Check Point is another investor. You would assume these people are pretty bright people and they don’t throw their good money away. That, if they’re investing this money, they invest it for two things. No. 1, they see the opportunity in the market, so the market itself is ripe and worthy of that sort of investment. And then No. 2, that the team and the product at Aqua are also worthy of the investment.

    Let’s first talk about the market, if we can, though. I mean, so this is a validation, I think, of the container security market, if you will, right?

    Osnat: Yeah.

    Shimel: The containers are real, that securing them are real, and there’s gonna be a big market for that coming up. What do you think?

    Osnat: Yeah, I mean, definitely, when we started Aqua just under two years ago, we didn’t quite know how fast the market would move and what kind of market it would shape up to be. And fairly early on, even when we were in beta in early 2016, it became fairly obvious that this is an enterprise market. That is that, you know, some of the biggest adopters of container technology are not just Silicon Valley startups or web-scale companies that kind of invented this, but Fortune 500 enterprises across verticals like financials, insurance, retail, aerospace, manufacturing, and so forth, as well as government.

    And so, paradoxically, you would expect this kind of new technology to be first adopted by startups, but the real boom in adoption comes from the fact that enterprises are adopting it very quickly, and I think there are several reasons for that. One is that the benefits of using containers are overwhelmingly positive, and the benefits are pretty immediate. So, you can see benefits in agility in terms of how fast you can deliver applications to the market, you can see benefits in moving to the cloud, and reducing your infrastructure costs and operational costs. And you can see much better control over what kind of applications you’re deploying, where, and when, et cetera, largely through automation.

    So, these benefits are overwhelming, and additionally, containers are easy, right? So, it’s—you know, one of the things that made containers, which in itself is not a new technology, it’s been around for a while, but what companies like Docker did is make containers very accessible and very usable. And so, the fact that they’re easy to use makes adoption easy as well, and you know, you kind of see a pincer movement of, on the one hand, grassroots adoption through developer and DevOps teams of containers, so they develop applications using containers. And on the other hand, you’re seeing a top down approach of, you know, CIOs, CTOs seeing the benefits of containers, adopting them, or mandating their use for, for example, cloud native applications to drive cloud usage, hybrid cloud, cloud migration—all of those good things.

    Shimel: Absolutely. So, Rani, you know—granted, I’m not as involved in container and container security as you are, obviously, but to me, a couple of different thoughts. Number one, in my limited—granted, limited expertise, containers aren’t that easy, right? I mean, I think when we talk about containers and then, like, Kubernetes and stuff to manage the whole environment—for someone who hasn’t developed on them before and is used to developing, let’s say, in VMware or in Bare Metal, there is, there’s a learning curve. Maybe once you pick it up, it’s easier.

    But the other thing is, I always try to compare sort of container adoption, let’s say, to hypervisor adoption or VMware adoption, and I would say that containers, their adoption curve is higher or steeper, faster than sort of hypervisor’s.

    Osnat: Yeah, that’s correct.

    Shimel: And really what I see, the last couple—you know, we’re always doing surveys here at devopsy.kinsta.cloud, announcing the new Security Boulevard site—what we’re seeing is, you know, we always saw people messing with the containers, right? R&D testing, what have you—but the numbers of containers, the percentage of companies that have containers in product deployment, you know, in real, frontline stuff? Where it was—jeez, just two or three years ago was low teens, below 20 percent—is now climbing, getting near that critical mass number, right, of 50 percents and stuff like that. And that’s huge, right? You’re talking about early adopters moving to mainstream in a matter of months, and that’s —

    Osnat: That’s correct.

    Shimel: — that’s the numbers you see as well?

    Osnat: Yeah, we see that. And again, I don’t think this would’ve been the case if containers weren’t easy, as I said. I mean, there’s always a learning curve with new technology, but I think if you’re a new developer, a new programmer working today, working on a greenfield application, there’s really no reason for you to use something that is not containers unless it’s something obscure for technology reasons.

    But mostly, it’s a much easier way to develop applications quickly than previous approaches. And I think, you know, if we take a broader picture, a container is—this adoption of containers does not occur in a vacuum, and we actually see a convergence of three trends that are kinda coming together and containers are probably the biggest enabler of those three trends. And these trends are happening independently of each other. So, you know, one trend is one that you are very familiar with, it’s the name of your site, is DevOps, you know, the proliferation of DevOps as a method, process, organizational philosophy, et cetera, which is all about delivering applications faster and in a more continuous way, and containers make that easier than before.

    Another trend is the move to microservices architectures, you know, for web applications or cloud native applications, which is basically adopting what web scale companies like Facebook and Twitter and Netflix started, and now bringing into the mainstream. That, too, is facilitated to a great degree by containers, because implementing microservices as containers is, today, probably the easiest method of doing that. I mean, there are the so—called serverless architectures, et cetera, but they’re much less mature than containers are at the moment. Not that containers are super mature, but they’re more mature.

    And the other aspect is the whole kind of hybrid cloud, cloud native movement where, you know, it took 10 years, but enterprises are adopting cloud services in a very big way, including industries that you would never dreamed of having on the cloud, on the public cloud, like pharmaceutical, health care, insurance, banking, federal, and so forth.

    So, all of these—and again, containers, because they’re portable, because they can run anywhere in the same way, they’re a really good way to realize this notion of hybrid cloud. For example, we at Aqua have a customer—who, unfortunately, I can’t name, but it’s a well-known company. And they are, for them, this is already a reality. They are deploying applications on both Amazon and Azure using the same framework, using the same security policies, and in a way that makes these clouds interchangeable for them, so they can run application A on Azure today and they can run the same application on Amazon tomorrow. So, that’s a real change that we see, and containers are a big part of that.

    So, all of these things together, I think, are what’s driving this adoption. And yet, you know, I think we have to distinguish between adoption and penetration or, you know, maybe some other word. But the fact is that today, if you look at the total workloads that enterprises are running, containers are a very, very small percentage of that. But, if you look at any new applications delivered over the next five years as well as looking at migrating legacy applications to the cloud, you’ll see this number growing exponentially over the next few years.

    So, yes, in terms of the number of enterprises now either looking at, working with, or already deploying containers, that number is growing very fast, and I think there’s probably hardly a large enterprise on the planet today that is not at least thinking about containers if not already using them. But at the same time, you know, it’s early days for them. Most of them—not all of them, but most of them, the vast majority are looking at the first few applications being deployed, and this is just the first drop of rain before the deluge. So, it’s gonna be a lot bigger.

    Shimel: Yep. Rani, a couple of comments, questions, follow ups. So, first of all—yes, DevOps, containers, they seem to go like peanut butter and chocolate. Again, so does microservices and containers, but to me, I—you know, when I was a kid, a teacher once taught us that all spaghetti is macaroni, but not all macaroni is spaghetti.

    Osnat: Sure.

    Shimel: So, it almost seems that, you know, wherever you see containers, you see microservices, but you don’t always see containers when you see microservices, right? In some regards, microservices can be even bigger, right? It has even more of a use footprint. Actually, on devopsy.kinsta.cloud, we’re launching a microservices channel which we hope will eventually grow into a standalone site, because I do think microservices is that big and has the potential to be that big.

    Osnat: Yeah, but microservices is not a technology; containers are. Microservices is an architecture, you know what I’m saying? There’s a difference.

    Shimel: Yeah, fair enough. Yep. But, you know, bringing it back—and, of course, loud usage and all of that. But, let’s turn for a moment and talk about the mission of securing containers and all of these architectures and platforms and so forth. That’s really what Aqua is, and when we talk about why they invested money, these investors, not just because the containers are so big, but because the mission of securing them is also big.

    Osnat: That’s true. So, yeah, I mean, when you look at—so, first of all, we’re not really, I mean, the fact that we’re a container security company might allude to the fact that we secure containers. That’s not exactly true. So, you know, we don’t look to secure a container or containers, we look to secure the workloads or applications that are running on containers.

    Shimel: Mm-hmm.

    Osnat: And that’s a difference. Because—and again, here, there are several trends that come into play. One is really that these things that I mentioned before, the DevOps trends, the microservices, architecture, and moving to the cloud, they also affect how you can and want to do security. You know, if we look at the cloud first—when you go to the cloud, a lot of the basics, you know, the bread and butter security that you use to invest in as an enterprise disappears. Why? Because these are services that are already taken care of or provided by the cloud service provider.

    So, you know, when you run your applications on Amazon, you don’t really have to worry about things like data center security, infrastructure security, network security to a degree, identity management to a degree. And so, a lot of these things are—you know, they’re basically taken care of for you. So, the focus shifts to the application layer, and that is one aspect.

    Another aspect, which is a really positive one for using containers, is the fact that, since you are now breaking up the application into a bunch of microservices, you can have much more visibility and control into what the application is doing. You know, if you look at an individual container or microservice, it’s much easier to understand the intended function of that container or service, and consequently, be able to understand what is considered good behavior, and be able to identify pretty easily what is bad behavior. And you do that at a very granular level.

    Additionally, if you look at the networking aspect, in a way, the networking has been brought into the application in a more visible way than before. Before, you could have things done within the application that maybe a malicious actor did within the application, and it’s all done internally, and it’s a black box. You don’t have visibility into it until it actually gets out.

    With microservices, you are able to see the interaction between the different components of the application, sometimes at a very fine grained level and, again, be able to identify any kind of weird or abnormal communication between, you know, parts of the application that are not supposed to be communicating that way. So, that’s another benefit. So, these things really allow us to have much more control and visibility over what the application is doing.

    The last piece is around DevOps, which is often nowadays coined as DevSecOps, which is the whole notion of shifting left security—i.e., bringing some of the control points not to the run time environment but to the developer environment. And why is that? Because containers are also immutable, right? So, if you’re using containers directly, you’re not supposed to see any change in one time environments. You’re not supposed to see patching, nobody is supposed to SSH into your host that’s running containers, it’s supposed to be entirely orchestrated and automated.

    And so, anything that happens there is by definition not good. But, at the same time, you have the opportunity to make the application more secure from the start, which is in the development phase, and that includes things like, you know, removing vulnerabilities, ensuring the configuration is correct, enforcing best practices. And if you can automate that—which we do—then you’re winning, right? Because you’re delivering a much more secure application to start with, you’re gonna have a smaller attack surface, and then in run time, all you need to do is make sure that there is no drift or no unexpected changes to the run time environment, which would be considered malicious or at least suspect.

    Shimel: Got it. So, Rani, we’re running a little low on time. We’re actually out of time, but let’s keep talking. [Laughter]

    Osnat: [Laughter]

    Shimel: So, now, we’ve got—what are you gonna do, how are you using the 25 million? What’s the plan?

    Osnat: Right, so you know, in our short life of two years, we managed to launch the product very early and get quite a few enterprise customers, many of them Fortune 100 customers across many industries, and that sort of traction is what really made Lightspeed and the other investors invest in us, and we need to continue this. The market is growing, right? As we mentioned, adoption is growing at an accelerated pace, so we need to be there.

    That means, you know, investing in the quality as well as the breadth and depth of our solution. There is—containers are not a singular technology. I mean, there’s a lot of variation in the stack, not just in orchestrators and OS’s, but in the container engines themselves, and this will become even more so over the next few months as things like, you know, additional run time engines are introduced, and not just the Docker open source run time engine.

    And so, this requires investing in the right technologies and making sure that we can fulfill the requirements of our customers and, additionally, of course, we need to expand our presence, both in the U.S. as well as in other parts of the world.

    Shimel: Let me talk to you about that a sec, Rani. So, I’m gonna assume U.S. probably represents at least 50 percent of your market, is that about right?

    Osnat: That’s correct, yeah. [Cross talk]

    Shimel: And probably Ameya the lion’s share of the rest.

    Osnat: At the moment, yes, but Asia is also growing, you know, there’s—

    Shimel: I also wanted to ask—what about AP?

    Osnat: Yeah, so Asia Pacific is growing as well. There are a lot of container users in places like Australia, China, Japan, and so forth. And so, this is—these are obviously big economies, there are big companies there. They’re as advanced in terms of cloud adoption as the U.S. in some cases. So, we need to be there.

    Shimel: Absolutely. Well, how do you—so, I was actually out at RSA APJ this summer, and it was in Singapore. And, you know, I had a chance—I actually put on a DevSecOps event there with RSA. So, I had a chance to talk to many of the RSA attendees, some of the vendors and stuff.

    I mean, what’s interesting there is, your infrastructure, you know, even in Europe, AWS is an 8,000-pound gorilla, right?

    Osnat: Correct.

    Shimel: Where do you—we don’t see that in Asia Pac. AWS is still a player, obviously, but there are other solutions there.

    Osnat: Mm-hmm. Yeah, that—

    Shimel: Go ahead. How does that affect you guys, or how do you deal with it?

    Osnat: Well, it affects us partially because mostly, you know, we’re pretty much cloud agnostic, right? We run on all the major clouds. It’s not a major effort for us to run on additional clouds. You know, even today, we support Amazon and Azure, Google, IBM Bluemix. And so, it’s relatively easy for us to support additional clouds, and it’s not a—that’s not the stumbling block, in most cases, right? It’s more like things like esoteric OS’s or weird configurations that we encounter sometimes. But, in most cases, that would not be an issue. Not to mention that a lot of companies are also running private clouds, using VMware or other types of virtualization infrastructure. So, that is really not a stumbling block for us, at all.

    Shimel: Good, good. Interesting. Rani, as I said before, we’re way over time, I need to wrap this up. But, first of all, congratulations on the raise. I’m sure Aqua has 25,000,000 good uses for this money as well, and we’ll see more coming down the pipeline in terms of both functionality products as well as reach around the world. You know, always good to have you on, you have a great handle on where the container market is and of course where the container security market is.

    So, thanks for being our guest today. We’ll have you on again soon.

    Osnat: Thank you.

    Shimel: Thank you. This is Alan Shimel for DevOps and Security Boulevard Chats. Thanks for listening. We’ll hope to see you soon on another chat. Bye bye, everyone.

    — Alan Shimel

  • Characterizing and Contrasting Container Orchestrators

    Characterizing and Contrasting Container Orchestrators

    Admiral Calcote, also known as Lee Calcote (@lcalcote) or the Ginger Geek to his friends, gave a presentation titled, “Characterizing and Contrasting Container Orchestrators,” at the 2016 All Day DevOps conference.

    Okay, he isn’t really an admiral—nor does anyone call him that—but he used the title “admiral” to describe what container orchestrators do, relating it to an admiral directing a fleet of container ships. You could also say that they are like the conductor of an orchestra, directing the individuals to work together as a group toward a common goal while each musician is still able to play their own instrument.

    lee1.png

    Lee is the head of Technology Strategy at SolarWinds, and for his talk, he discussed four open-source container orchestrators: Nomad, Swarm, Kubernetes and Mesos-Marathon.

    He emphasized the obvious: There is no one perfect solution. Each organization is different, so for each solution, he looked at:

    • Genesis and purpose
    • Support and momentum
    • Host and service discovery
    • Scheduling
    • Modularity and extensibility
    • Updates and maintenance
    • Health monitoring
    • Networking and load balancing
    • Secrets management
    • High availability and scale

    Lee noted that while there are many core capabilities, any orchestrator must have cluster management and scheduling.

    lee2.png

    He then dove deeper into the four solutions. Below are summaries (the full talk is chock full of information and is online here):

    Nomad

    • Designed for both long-lived and short-lived batch processing workloads
    • Cluster manager with declarative job specifications
    • Ensures constraints are satisfied and resource utilization is optimized by efficient task packing
    • Supports all major OSs and workloads
    • Written in Go and with a Unix philosophy
    • Host discovery: Gossip protocol – Serf is used; servers advertise full set of Nomad servers to clients; creating federated clusters is simple
    • Service discovery: Integrates with Consul
    • Scheduling: two distinct phases – feasibility checking and ranking; optimistically concurrent; three scheduler types when creating jobs
    • Uses task drivers to execute a task and provide resource isolation, but it does not support pluggable task drivers
    • Built for managing multiple clusters/cluster federation

    lee3.png

    Docker Swarm 1.12

    • Simple and easy to set up
    • Architecture is not as complex as Kubernetes and Mesos
    • Written in Go – lightweight, modular and extensible
    • Strong community support
    • Host discovery: used in the formation of clusters by the Manager to discover Nodes (hosts); pull model – worker checks in with the Manager
    • Service discovery: Embedded DNS and round-robin load balancing
    • Scheduler is pluggable and is a combination of strategies and filters/constraints
    • Ability to remove “batteries”
    • Rolling updates are supported
    • Managers may be deployed in a highly available configuration, but does not support multiple failure isolation regions or federation

    lee4.png

    Kubernetes

    • An opinionated framework for building distributed systems
    • Written in Go and is lightweight, modular and extensible
    • Led by Google, Red Hat and others
    • Young – about 2 years old
    • Robust documentation and community
    • Scheduling is handled by kube-scheduler
    • Pluggable architecture and an extensible platform
    • Choice of database for service discovery or network driver and container runtime
    • Supports rolling back deployments, automating deployments and rolling updating applications
    • Inherent load balancing
    • Uses Pods, an atomic unit of scheduling. Each pod has its own IP address, no NAT required and intra-pod communication via localhost

    lee5.png

    Mesos-Marathon

    • Mesos is a distributed systems kernel
    • Mesos has been around the longest (since 2009)
    • Mesos is written in C++
    • Marathon is a framework that runs on top of Mesos
    • Mesos is used by Twitter, AirBnB, eBay, Apple, Cisco, and Yodle
    • Marathon is used by Verizon and Samsung
    • Mesos-DNS generates an SRV record for each Mesos task
    • Marathon ensures that all dynamically assigned ports are unique

    lee6.png

    Wrapping up, Lee provided the following overview comparing the different container orchestration solutions.

    lee7.png

    Lee packed a tremendous amount of information into his talk. If you use containers, his talk is worth your time and is available online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • SecOps: The Next Stride for DevOps

    SecOps: The Next Stride for DevOps

    Global IT spending is now touching a figure of $4 trillion. In today’s technology-intensive environment, data is the biggest commodity, more valuable and powerful than ever. And like Peter Parker’s uncle Ben noted, “With great power comes great responsibility.”

    Security has always been a big concern for valuable things and IT is no exception. IT as a whole is trying to push security left in the development cycle—trying to involve it in every step of the development cycle. And just when development and operations teams are getting used to the “esprit de corps” we call DevOps, I think it’s time we add the third musketeer: security.

    What is SecOps?

    SecOps is a collaboration between security and operations teams, just like development and operations teams collaborate on DevOps front. SecOps is a set of practices organizations need to follow, processes they need to execute and tools they need to use to ensure the security of their application environment. SecOps is making sure organizations do not sacrifice security to attain set performance and uptime indexes.

    In a typical development cycle—requirements gathering, design, development, testing, implementation or deployment and maintenance—security normally is introduced in the later stages, somewhere between testing and deployment or even later. But SecOps is all about making sure to introduce aspects around security much earlier or at each stage of the software development life cycle (SDLC).

    I know what you are thinking: This is going to complicate things and increase the time to delivery. This is where operations and development teams need to join forces to uncomplicate the things and make it time efficient practice. Next thing you must be thinking of is, Why so much hassle? Think of it the other way. Wouldn’t it save more time when you address the security concerns at much earlier stages than at the time of delivery or implementation? All it takes is an amalgamation of the security group, development team and operations team; a little bit of planning; and a whole lot of execution.

    SecOps + Containers

    Containerization is slowly but steadily moving from an alternative to full virtualization to a serious platform for running your applications. Containers have some obvious advantages including scalability and flexibility, and this solves most of the problems related to resources in case of application development. Reduced size; reduced time to provision application environment and testing; platforms including Docker, Solaris Zones, BSD Jails; and orchestration platforms including Kubernetes, CoreOS Fleet, Amazon ECS and OpenShift make containers a more preferred option for application development environments.

    This increased traction toward containerization points to the increased need to concentrate on security aspects. Here are some best SecOps practices for container environment you and your organizational teams can follow:

    • Authentic sources and images: Always check for authenticity of container images. There are various tools such as Docker’s security scan. With Docker Cloud and Docker Hub you can scan images to check for potential security vulnerabilities. Most images are built from some base image and not built from scratch, so there is always a threat with the used images.
    • Vulnerability management tool: There are tools available in the market to analyze container image formats and libraries for threats before you actually start using them.
    • Follow benchmarks and hardening guidelines: Always make sure you do the checks and follow hardening guidelines for containers, images, hosts and platforms before you start with production. There are few standards and benchmark checks for containers such as CIS’s Docker security benchmark, PCI compliance checklist, etc.
    • Periodic auditing: Regular auditing of your application environment can help you save yourself from the future troubles. Moreover, automation of auditing process can help in detection of unused images and containers.
    • Use of management frameworks: Use frameworks that can automate behavior profiling and control all the users, authorize the access to the containers, images and hosts.
    • Security built in to container engine systems and third-party security solutions: Third-party vendors have a number of applications for container security in addition to security systems of container management platforms.

    Dev + Sec + Ops

    With the continuously increasing business demands for new applications and software, and new practices and development trends such as DevOps, Agile, cloud, automation, CI/CD and others, traditional security needs to upgraded in the new paradigm. Thankfully, some of these practices facilitate the security. Consider CI/CD as an example: Continuous integration requires continuous integration tools, or what we call build servers. Some popular examples are Jenkins, TeamCity, GitLab CI, Travis CI, Bamboo, Go CD, CircleCI and Codeship. The best SecOps practice is to check and fix vulnerabilities at early stages as a part of CI/CD workflow. Integration between authentication, scanning, management tools and CI/CD pipeline tools could be the best possible solution to your security-related problems. Some easy-to-implement solutions can include automated security testing, static code analysis, authentication checks and login tracking.

    SecOps enable organizations in lifecycle management, analysis of security threats, incident management, optimizing and measuring the effectiveness of security controls, reduced breach response time, reduced security risks and increased business security. The basic principle on which SecOps works is Avoid, Analyze, Respond, Review, Repeat. By analyzing the security events and data, you can build incident response plans to avoid future unwanted events.

    Frameworks and Tools

    Now that you have a clear understanding of why you need your security, development and operations teams to work together, let’s see what tools and frameworks you can use.

    • Docker-native tools: If you are using Docker as a platform then you can use few security tools provided by Docker itself for the security of production environment: Docker Bench and Docker Notary. Docker Bench is a script that checks common best practices around deploying Docker containers in production. Docker Notary enables you to check whether content is from a trusted publisher.
    • Chef: Chef provides different tools including Inspec to automate security testing.
    • Puppet: Puppet provides security compliance and policy defining frameworks.
    • Ansible: Ansible provides system tracking, setting up firewall rules, user lockdown and compliance automation solutions.
    • CoreOS Clair: CoreOS Clair is an open-source project for vulnerability analysis in applications and Docker containers.
    • SaltStack: SaltStack can help in orchestration and automation of security practices solutions for containers.

    This is just a look at some of the most popular ones. If you want, there are a number of others that can look after the security of your application environment.

    The main motive behind implying SecOps practices in any organization is involving security team at all possible stages to remove any ambiguity in any stage of development rather than security team providing analysis reports to the operations team and then sitting back and enjoying the show. When these teams perform in a synergistic manner, the business focus can be shifted to other important things.

    About the Author / Chaitanya Jawale

    Chaitanya Jawale is founder and CEO of Opcito Technologies, a leader in microservice-based product engineering and automation. Chaitanya comes with strong background in product engineering, data center automation, DevOps and multi-cloud orchestration and management.

  • Sprinting in a DevOps Marathon: Tips for Success

    Sprinting in a DevOps Marathon: Tips for Success

    Remember the good old days of 2007, when Heroku was the shiny toy everyone wanted? Thanks to Moore’s law and the evolution of tech, 2007 seems like a lifetime ago. In fact, 2015 seems like a lifetime ago, now that machine learning has advanced for artificial intelligence to “wake up”—but I digress.

    DevOps and building applications in a microservice architecture appears to be here to stay. Monolithic applications have been around since the advent of the “stack,” but thanks to DevOps, we can build faster and smarter with better quality and lower cost. Consequently, it’s dumbfounding why only .5 percent of engineers operate DevOps (according to Gene Kim). DevOps would unlock trillions of dollars if the other 99.5 percent made this adoption, thereby ushering in a new era of economic prosperity.

    If you’re part of the other 99.5 percent, let’s go through the considerations for DevOps adoption—it’s the wave of the future, so you better start now!

    The Road to DevOps Success

    Assess the competency and maturity of development practices and processes of your organization across the following parameters:

    • Culture (i.e. collaboration and standardization)
    • Design and architecture
    • Data analysis and management
    • Build, continuous integration and continuous deployment
    • Assessment and testing
    • Application deployment across environments (Dev, Test, Production)
    • Continuous delivery and application upgrades/updates

    Assessing the aforementioned practices and processes result in a baseline to compare against industry standards (see Puppet’s “2017 State of DevOps Report“) and identify areas of improvement. This may seem intuitive or obvious, but true agility through DevOps requires frequent evaluations as new tools, practices and methodologies become available (i.e., you’re not still using dial up now are you?).

    Enhance and Improve the Culture with Shared Values and Objectives

    Design, Dev, Operations, QA, Product and other applicable teams historically have operated independently of one another in “silos,” which is the antithesis of DevOps. The goal need not be pushing code, configuring applications or testing; the goal should be to deliver value to the customer. Unfortunately, the culture of most organizations inhibit teams from maximizing value because they’re removed from the customer and seldom listen to the sales team (which I emphatically state as a seasoned sales professional!).

    That said, this is the easiest thing to fix by implementing the following steps:

    • Introduce a feedback loop including customers, sales, marketing and customer support to introduce new ideas to product so value is maximized.
    • Clearly articulate goals to motivate product teams and get them excited about their work.
    • Institute a culture of continuous improvement by listening to developers, operations, design, product, etc.

    Base Decisions on Facts, not Conjecture

    Seldom do organizations correlate sales and customer satisfaction metrics with product investment and product operations. All too often, organizations let sales or product dictate the best direction for the company when they need to be evaluated in conjunction due to their impact on profitability (i.e., building products to drive sales or letting sales dictate what products to build). DevOps expedites the delivery of products, thereby aligning the objectives of product with the goals of the sales team. This enhanced agility makes it’s easier to incorporate feedback from sales to enhance products while enhancing sales by delivering value faster, lowering the sales cycle and fostering happier customers!

    Choose the Right Tools, not the Popular Ones

    Months ago, the Bluemeric team started an engagement with a Fortune 500 team that was beginning its journey into containers. Long story short, this client evaluated the container platforms based on name recognition, not on the relative strengths and weaknesses or how the PaaS will achieve the client’s goals with the least amount of effort. The client reviewed dozens of platforms and entertained 10 for a proof of concept, but only one (and a relatively unknown PaaS) was able to install on its infrastructure (AWS, Azure and Google), spin up a cluster, build the application and deploy the applications. The competitors had tens if not hundreds of millions of dollars in revenue and thousands of customers, but the one that got the job done wasn’t the first, second or even third choice. It was the lightweight PaaS that Bluemeric recommended, which promised simplicity, minimal effort and ease of use that doesn’t drown customers with unnecessary features—and which actually worked. Obviously, this isn’t limited to PaaS; it’s an example on why decisions to adopt tools, methodologies or vendors must have a thoughtful evaluation, free from outside opinions and bias.

    That being said, implementing and adopting DevOps is neither easy nor fast, but it will be a requirement for all businesses in the very near future. As noted earlier, Gene Kim discovered that .5 percent of all software engineers currently operate in a DevOps environment, which is astonishing when comparing profitability against those who aren’t practicing DevOps. Once the trend becomes mainstream in the world of finance, the other 99.5 percent of software engineers better catch up or face the consequences.

    — Gabriel Deupree

  • I Can’t Go to That DevOps Conference

    I Can’t Go to That DevOps Conference

    Last year, I was able to attend 18 DevOps conferences. I saw some awesome presentations, met incredible people and expanded my knowledge of the latest practices at each conference. While my role as the DevOps Advocate for Sonatype enables me to get out to all of these conferences, not everyone has the time, the budget, or the approval to get to even one event a year.

    DevOps for the Masses

    This is what led me and my colleague, Mark Miller, to found the All Day DevOps conference last year. Our dream was to build a DevOps conference for the masses. We decided to offer 57 online sessions over 15 hours and 15 times zones, and registration would be free. Our initial goal was to see if we could find 1,000 people to attend (because that sounded like a big number at the time), but the idea of a free online event caught a viral wind and we ended up having over 13,500 people register to attend.

    Every session delivered during the conference was shared by a DevOps practitioner. We had a simple rule for the conference and that was: no vendor pitches.

    Team Time

    People also told us that they valued having the chance to talk to other attendees they met at DevOps conferences, so we set up a free Slack channel for All Day DevOps. By the time conference was over, we recorded more than 10,000 conversations between the speakers, attendees and organizers.One of the things that fascinated us most about the conference was that organizations such as Accenture, Capital One, Ericsson, Oracle, Target, Telstra and many others had more than 20 employees attending the event. A few of these had hundreds. None of them needed to approve expenses for travel, hotels or conference fees. They just watched and joined in the conversations from their homes or offices.

    Here are some of the folks who spoke at last year’s event across our CI/CD, Modern Infrastructure and Automated Security tracks. This year, we have also added a DevOps in Government track to the agenda.

    The event was so successful, we set it on the schedule again. All Day DevOps 2017 will take place Oct. 24. It’s online, it’s free and this year’s event will run for 24 hours. Therefore, no matter where you live, if you have an internet connection, you will be able to tune in to join us.

    A Few Things to Know

    1. You can register for free at www.AllDaydevopsy.kinsta.cloud. Invite your entire DevOps team to join. Everyone registering will have free access to all 57 recorded sessions from 2016.

    2. The CFP is open through May 31. If you have a story to share, we want to hear from you. Remember, no vendor pitches are permitted.

    3. All Day DevOps is a community event. Last year more than 40 DevOps meetups and software companies helped us spread the word. For those that helped, we added their logo to the site. Same applies to this year. Apply to be a supporter here.

    Please join us for this year’s event!

    Finally, a big thank you goes out to devopsy.kinsta.cloud, which helped us promote last year’s event to all of their readers. We are excited to have them helping us to promote the 2017 conference as well!

    — Derek E. Weeks

  • Get Your DevOps Right With These 7 Technologies

    Get Your DevOps Right With These 7 Technologies

    Editors’ Note: As with all of our contributed articles, the views expressed in this article are of the author, not devopsy.kinsta.cloud. Due to its vendor-agnostic stance, devopsy.kinsta.cloud does not endorse any particular technology.

    There was a time when the operations team and the developers regularly were at each other’s throats. The developers were never satisfied with production code updates and the server administrators chafed at developers piling up the serve requests and circumventing the processes.

    However, the battle lines have been somewhat erased since the advent of DevOps. IT teams adopting DevOps can move rapidly from concept to production to enhance the business impact, quality and velocity of the application delivery. Many organizations are shifting to DevOps because of its high efficiency and the number of people taking DevOps training are also increasing due to numerous job opportunities in this field.

    DevOps Technologies to Know

    A number of technologies have emerged to help in the shift to DevOps, providing services ranging from the configuration management to application portability. Following is a list of seven technologies worth noting.

    Splunk

    Splunk is used for detecting and fixing issues across the life cycle of an application in real time. It allows the developers to see data from the production environments, eliminating the need to access production machines. Splunk enables DevOps processes including continuous integration and deployment.

    The Splunk platform is used for DevOps to:

    • Provide end-to-end visibility across each and every component of the DevOps tool chain.
    • Utilize the objective metrics to make sure the code meets the quality SLAs and is operational.
    • Correlate the business metrics with changes in code to obtain new business insights.

    ScriptRock GuardRail

    ScriptRock Guardrail is designed to rapidly pinpoint the differences in configurations on desktops and servers, comparing the states between systems and over time. ScriptRock Guardrail shows what has changed, helping simplify diagnostics and decreasing downtime.

    Developers who work with agile methodologies are accustomed to a test-first way of working with the unit tests, making sure that everything works as planned. As continuous delivery becomes more prevalent, the DevOps and Ops teams need to take a similar approach. This is where the ScriptRock GuardRail comes into its own, providing you the tools to construct and build infrastructure level unit tests and view the results visually, thus making it very easy to rapidly drill into the real problem and get your services running.

    ScriptRock GuardRail focuses on addressing the key prerequisites to any DevOps, and, more precisely, initiative and automation. They are described below.

    • Visibility: The GuardRail platform delivers visual configurations showing differences between like devices, configuration drift monitoring and automated visualization and scanning of network device and server configurations.
    • Accountability: Users can curate and filter configurations to generate policies that can be utilized to ratify the configurations on a continuous or ad hoc basis.
    • Automation: All the policies can be exported to Windows PowerShell DSC, Docker, Ansible, Puppet and Chef.

    ScriptRock GuardRail works with any automation tool and can be delivered as single-tenant SaaS, multitenant SaaS or a virtual appliance on-premises.

    SaltStack

    The SaltStack Enterprise Software helps DevOps organizations by organizing the systematic motion of code into production by maintaining the complicated infrastructures refined for optimal application delivery and business service. While DevOps is not a tool, it requires a tool-centric approach to enable continuous delivery. SaltStack helps to configure as well as quickly deploy the dynamic applications and general-purpose infrastructure they run on. DevOps is gaining wide popularity around the world because of the tools like SaltStack that enhances its efficiency.

    SaltStack is a cloud infrastructure tool and delivers the systems management for application configuration, cloud building, server provisioning and data automation. It is an event-driven tool and can automate the tasks within DevOps workflow and makes the operations highly approachable for the developers. SaltStack provides a common language and common way to manage the servers and thus helps to bridge the gap between Devs and Ops.

    Puppet

    Automation is the foundation for many DevOps practices. It is the key to continuous delivery and faster deployments. Automation imposes consistency across the production, testing and development environments. It makes sure users can rapidly orchestrate the changes throughout the infrastructure: container and cloud deployments, virtual machines and bare-metal servers.

    No matter where it runs, Puppet provides a standard way of operating as well as delivering a software. Users define what they want their infrastructure and apps to look like utilizing an easy-to-read common language. From there, they can enforce, test and share the changes they want to make across their data center. Reporting and visibility is available at every step of the way to help prove compliance and make decisions.

    The Puppet Enterprise enables users to test changes before deploying them to production. With Puppet Enterprise, the infrastructure is defined as code, which enables users to adopt the best practices of agile development including automated deployment and testing, and deliver the software more reliably. This is because the configurations are consistently promoted, shared and tested across all environments, from Dev laptops through test to production.

    Atlas

    Atlas was the first commercial product released by HashiCorp, provider of open-source tools that enable the efficient deployment of software-defined applications and data centers. Atlas provides visibility into the infrastructure, including virtual machines, containers and servers, in addition to service discovery and configuration management. Building on the popular open-source projects of HashiCorp such as Terraform, Consul, Serf, Packer and Vagrant, the closed-source proprietary Atlas enables DevOps across different cloud services, which include OpenStack, Azure, Google Compute Engine and AWS, and provides a dashboard for maintaining, deploying and developing applications.

    Many engineers are shifting to Atlas today for their organizations’ customer engagement platforms. Atlas brings it in one place and enables the companies to handle end-to-end starting from configuration management to container deployments across various cloud environments. Every open-source tool is fine alone, but the capability to handle and orchestrate them as a whole is extremely powerful.

    Chef

    Chef is a cloud infrastructure and systems framework that automates the management, deploying and building of the infrastructure via repeatable, short scripts known as “recipes.” But the actual strength of Chef lies in its utilization of the pluggable configuration modules(also known as cookbooks), about 2,000 of which are available through Chef community. Facebook, the high-profile user of Chef, recently open-sourced few of its own Chef cookbooks, which include its Grocery Delivery and Taste Tester testing framework, which watches a source code repository such as Git and keeps the local Chef server in sync.

    The complex tasks are automated by Chef that are otherwise resource- and time-intensive. But, more importantly, Chef allows users to focus their efforts on improving and innovating the quality of their services. It also opens the door to more efficiency and collaboration across the organization.

    Docker

    Through its containerization technology, Docker brings portability to the applications wherein the applications run in self-contained units that can be moved across the platforms. It consists of Docker Hub, a cloud service for workflow automation and application-sharing and Docker Engine, which is a lightweight packaging and runtime tool.

    Docker has revolutionized the containerization of applications enabling DevOps to package any kind of application in a lightweight environment. The packaging is done in a way as easy as installing a mobile app. Docker is considered as the most popular DevOps tool. By using docker, the developers can containerize their apps and can run them anywhere. Built on top of Linux containers, docker is a powerful ecosystem where DevOps can use many apps (available on Docker Hub) to the maximum advantage.

    About the Author / Savaram Ravindra

    Ravindra was born and raised in Hyderabad, India, popularly known as the ‘City of Pearls’. He is presently working as a Content Writer at Appmajix Technologies. His previous professional experience includes Programmer Analyst at Cognizant Technology Solutions. He holds a Masters degree in Nanotechnology from VIT University. He enjoys spending time with his friends. He can be contacted at savaramravindra4@gmail.com. Connect with him also on Twitter and LinkedIn.

  • DevOps Chat: Purpose Built Appliances for Containerized Applications with Diamanti’s Mark Balch

    DevOps Chat: Purpose Built Appliances for Containerized Applications with Diamanti’s Mark Balch

    Back in my StillSecure days, building purpose-built appliances for security was all the rage. Over time, though, security-as-a-service captured much of the market (although security appliances are still a big seller). For containerized applications, you may think cloud is the way to go. But the folks at Diamanti say that purpose-built appliances for containerized applications are a superior option.

    I spoke with Mark Balch, their VP of product and marketing about this, and he gives us Diamanti’s views here and, frankly, makes a good case. It is early in the containerized application market and the winners are yet to be crowned. Mark tells it better than I can. Have a listen and let us know what you think.

    As usual, the streaming audio of our conversation is below and the transcript of our conversation is below that.

    Streaming Audio

    Written Transcript

    Alan Shimel: Hey, everyone. Alan Shimel, devopsy.kinsta.cloud here for another episode of DevOps Chat. Very happy to be joined today on DevOps Chat with Mark Balch, VP of products and marketing at Diamanti. Mark, welcome to DevOps Chat.

    Balch: Hey, Alan. Thanks for having me. It’s great to talk to you.

    Shimel: Thank you, Mark. So, Mark, let us first start off with Diamanti. I think that’s a fairly new company to our audience or at least to us here at devopsy.kinsta.cloud, although the company itself is not new, but, for those of our listeners who may not be familiar, why don’t you give us a little background?

    Balch: Yeah, definitely. We’ve been around since 2013, and Diamanti spent the first couple of years in stealth mode, as you may say, building our product. And we launched the company last year. We just launched our GA product this past week, in February.

    And just a very quick background or why are we here. Well, we saw the container revolution occurring. We believe that containers are the next mainstream application platform for new cloud-native applications, moving forward. And we’ve seen that users are spending a tremendous amount of time and money trying to custom-engineer their own infrastructure and figure out how to make cloud-native applications operate at scale, with reliability and performance. And we said that, “You know, we wanna make sure that users can do this out of the box, very simply, get control of their costs and be able to focus their time, instead of reinventing the wheel and infrastructure, on building the unique business applications that they’re being measured on.” So that’s why we’re here.

    Shimel: Excellent. And, you know, Mark, what was interesting is, at a time when we’re seeing software-defined everything, right, Diamanti’s out here with a purpose-built appliance for containerized applications. And I would imagine there are many people who say, “Oh, no, we don’t wanna sell hardware.” But, really, if you want control over your environment, you don’t have much of a choice.

    Balch: It’s a real tricky situation for a lot of users, right, because just go out there and spend a little bit of time searching on the web and going to meetups and looking at user testimonials about the challenge of being able to get reliable performance at a reasonable cost point in the traditional model. It’s really very difficult. People spend a lot of time custom-engineering, and that’s time that they can’t spend building their unique services. So, for us, really, it’s a very simple value proposition.

    We’re not saying that you have to customize your application or do anything proprietary; what we’re saying is that, “Continue to go with your open-source approach—standard Docker images, standard Kubernetes orchestration, standard Linux—but operate on a platform that is going to give you the automation of getting that entire application deployed properly, that’s gonna give you predictable performance and outcomes and that’s gonna let you run at a reasonable cost point.” I mean, people have massive bills, whether it’s public cloud, whether it’s looking at virtualized traditional infrastructure that was built for a different type of application. People spend a tremendous amount of time and money trying to get cloud-native applications working, and our goal is to make sure that people can get those applications working within minutes, not within weeks or months.

    Shimel: Sure. And that is the advantage of having that sort of purpose-built platform. Mark, I wanted to, first of all—you know what? Shame on me. I didn’t even ask. Can you give our audience a little bit of the background of who you are, though? And I’ve said your title, but who’s Mark Balch and how did you come to Diamanti?

    Balch: Yeah. That’s a good question. So I’ve been in the IT and application industry for 20 years. I started off my career as an engineer. I love doing development work. I love to build things. I love to deliver great products that have an impact. And then, somewhere along the way, I got into product management because I wanted to figure out whose job was it to make sure that we’re building the right products at the right time, that it could solve real needs. And so, for the last more than 10 years, I’ve been focused on product management and product marketing, doing everything from data center automation and application provisioning software to cloud automation software, and, now here at Diamanti, focused on the open-source side of application development and, again, focusing more on the outcome. Whether we’re talking about an appliance, whether we’re talking about software you can deploy, at the end of the day what I care about, as a product manager, is making sure that there’s a value at the end that’s gonna make someone’s life better, that’s going to give them a greater experience in being able to be personally successful in getting their applications into people’s hands. So that’s what motivates me.

    Shimel: Great. Appreciate that. Mark, if I may, I wanted to take a moment and talk a little bit about Kubernetes. There are obviously choices for container orchestration and container management out there, and it seems like Diamanti has clearly put their chips in the Kubernetes market. As I said, there are other choices. Can you give us a little bit of the background on why Kubernetes and—or maybe I’m wrong. Can Diamanti customers choose other container orchestration options as well?

    Balch: Yeah, you know, I think, when it comes to talking about container formats and orchestration, I think we all have to be a little humble, in that the market is still very early and very dynamic. There’ve been so many changes. Kubernetes is doing a great job. Mesos is doing a great job. Docker’s doing a great job. You’ve got now standardization happening in the Cloud Native Computing Foundation for the Open Container Initiative. So I think, honestly, it is too early to call a winner, and I think all of us have to make sure that we’re giving users choice in working with an open ecosystem. It’s not about forcing people down a particular path; it’s about giving people options. Right? So we can work with a variety of different options, but let me tell you why we have been doing a lot of work with Kubernetes. And we’ve been doing this work with Kubernetes for the last couple of years.

    Kubernetes, from a governance perspective, has been a great ecosystem to work with. Obviously, Google contributed that code to The Linux Foundation last year, and you’ve got large and small contributors from all over the ecosystem, including Diamanti, contributing to make Kubernetes bigger and better and more extensible. So what attracted us to work with Kubernetes early on was that governance, where, really, anyone could come to the table. Everyone has been welcome to pitch in and to take leadership roles, frankly. For example, Diamanti contributed something called “FlexVolume” into Kubernetes. It was upstream; it was a year ago. And we were welcomed with open arms, and now there are, literally, more than two dozen members in the storage SIG who we work with. Same thing around scheduling for Kubernetes—we upstream contributions there. Working with CNI, container network interfaces, is another example.

    So we’ve just seen a lot of great governance and a lot of extensibility, where the Kubernetes community seems very focused on this concept of extensibility, where it’s not that my solution is better than yours; it’s that we have APIs that we can use to extend different options and then users can pick and choose what’s gonna work for them, but it’s supported with a common framework. So that’s what we love about Kubernetes.

    Shimel: Got it. Interestingly, Mark, in our several studies, surveys on this with our audience, overwhelmingly—well, not overwhelmingly ’cause, actually, Mesio was pretty close, but Kubernetes clearly was the choice among our audience for container management and orchestration, so probably not a bad move there. Mark, what about people who have invested in public cloud infrastructure for containers? What’s the Diamanti value proposition to them?

    Balch: That’s a really good question, Alan, and people ask us that a lot. And, by the way, I’ll say that Diamanti is gonna be at the Google Cloud Next Conference next week, so we—

    Shimel: Really?

    Balch: Yeah, so we actually do a lot in terms of public cloud, okay? So I think the first thing is that it’s back to that concept of choice and no vendor lock-in. We need to make sure that we’re all offering users capability—we’re not trying to force them down a path with something that is proprietary—so everything we do at Diamanti is standard Docker runtime, and, as that evolves and maybe we have more standardization around runtime, that’s gonna evolve into making sure that everyone participates in the standard format. So, if you develop an application and you’re using Google Cloud or using Amazon or Azure or someone else, one of your goals, probably, should be to do it in a way where you can be portable, where you’re not locked in, where you can bring it to another provider if that provider offers you better economics, better performance, better reliability, data governance, things like that.

    And that’s where we come in. You could think of Diamanti as a provider, but we are an on-premises provider. We allow users, like MEMSQL or NBCUniversal, to be able to build an on-premises private cloud that is fully open source and you can take your existing containers, you can move them onto Diamanti, you can get all the great performance and automation benefits that we talk about. If, for some reason, you wanna go elsewhere, you wanna go to Google, to Amazon, somewhere else, we don’t stop you. You can do that. You can go back and forth, right? So we wanna make sure it’s very easy to go back and forth so that the user is in control of deciding what’s more important to them, in terms of all those things like governance, data sovereignty, performance, and economics.

    Shimel: Sure. Do you envision a time when people sort of have hybrid Diamanti-public-cloud containerized applications?

    Balch: Yeah, that’s a good question and actually brings up a really—another good wrinkle around Kubernetes. Kubernetes has an emerging project called “Ubernetes,” or its formal name, “cross-cluster federated services.” Now one thing I will say—I love the Kubernetes community, but sometimes, from a marketing perspective, we could do a little better. “Cross-cluster federated services” doesn’t really roll off the tongue very quickly, but what it means is, essentially, it can mean hybrid cloud or multiple availability zones.

    And so, just like Diamanti has made upstream contributions to the community around scheduling, we can now extend those scheduling capabilities, using Ubernetes, so we can move that across the availability zones and into hybrid cloud scenario. Right? So I definitely envision a scenario like you’re talking about, where you can have multiple Diamanti clusters in different data centers, you can have availability zones up in Google or Amazon or Azure, and you can, basically, allow the cross-cluster federated services to figure out where to deploy that application based on your performance requirements, on your sovereignty requirements, on your compliance requirements, or you can say, “Hey, I wanna make sure this thing goes into public cloud,” or, “I wanna make sure it goes into my Los Angeles data center with Diamanti.” You’re in control of that, from a policy perspective, and that’s, I think, what’s really interesting about the cross-cluster federated services road map, moving forward.

    Shimel: Absolutely. And you’re right; that does just roll right off the tongue. But, look, they’re engineers; they’re not marketing people. Mark, let me ask another question, and, not to hold your feet to the fire, but, for people who may be interested at home, can you give us sort of a ballpark pricing on Diamanti?

    Balch: Yeah, absolutely. So we’ve announced our pricing. Each Diamanti appliance has a list price of about $40,000, slightly under that, in fact. Obviously, discounts apply. So we are highly competitive with all alternatives, whether you look at commodity, whether you look at converged infrastructure, hyper-converged infrastructure. We are highly competitive from a price perspective. And then, when you look at the consolidation and the time savings, frankly, there is no competition there. So, certainly, love to chat with any of your listeners who are interested, and I pretty much guarantee, from an economic perspective, it’s gonna work out great.

    Shimel: So that brings up—you opened that box, Mark. For any of our listeners who may be interested in finding out more about Diamanti or speaking to someone, is there an e-mail address or a web address where they can check it out or contact someone?

    Balch: Yeah, definitely. So please check out our website, it’s www.diamanti.com, or just send us an e-mail, info@diamanti.com. We can get you set up really quickly.

    Shimel: Okay. Mark, we’re running low on time, but I did want to just mention that Diamanti is sort of riding the wave, if you will, of this tremendous momentum building up behind containers and micro-services and Docker, Kubernetes, etc. Where in the adoption cycle do you think all of this is? Has it really gone mainstream yet or are we still at the early-adopter stage?

    Balch: That’s the multibillion-dollar question, so we are seeing adoption all over in different kinda fits and starts, to be honest with you. So the industries we see the most adoption, Alan—we see media moving quickly. We actually do see financial services moving quickly. We see web infrastructure. We even see service methods moving ahead, wanting to offer containerized services and competing with the large public-cloud vendors. So where we see containers happening is developers are downloading Docker on their laptop; they’re able to accelerate their application lifecycle and development processes. Operations is a little slower. To be honest with you, it is lagging a little bit behind. That’s where we come in—we’re working with ops teams to help them get off the ground—and, just like always, developers ultimately have the influence where they’re defining application architecture, and then they work with their ops teams to figure out how to get those architectures deployed successfully into production.

    So there are lots of different numbers out there—451 Group had some good numbers; I’ve seen numbers from Forrester and from Gartner. The industry’s still early, maybe at only about 10 percent of adoption, and that’s a very anecdotal number, but it’s growing really quickly and we’re seeing a lot more inbound requests now because everyone’s trying to learn about it. The cat’s out of the bag; it’s not a secret anymore. And I think either people are doing it today or they see it coming quickly and they wanna figure out, when it comes down the pipe from their development team, how do they be successful with containers and actually make it work for their business.

    Shimel: Got it. Got it. Sounds great. Mark, we’re actually way over time. I apologize to our listeners and to you, but I thought we had some good information here and I didn’t wanna cut it off, but we need to wrap up. Mark Balch, thanks so much for being our guest today. Continued success with Diamanti. It sounds great. And, obviously, through Container Journal and devopsy.kinsta.cloud, this is a space we watch pretty closely, and I’m sure we’ll be hearing more from Diamanti soon.

    Balch: Thank you so much, Alan. This has been a great conversation. I appreciate it.

    Shimel: You know, one last thing. Mark, I forgot to mention. You mentioned you were going to be at the Google Cloud conference, which is the week of March 9th, something like that, right?

    Balch: Yep.

    Shimel: In March 8th and 9th?

    Balch: Yeah.

    Shimel: And then –

    Balch: March 8th through 9th.

    Shimel: – DockerCon is next month in August – in Austin.

    Balch: April.

    Shimel: In April.

    Balch: Yes. We will _____.

    Shimel: You’ll be there as well.

    Balch: Yep. Definitely gonna be there, yep, and we’ll also see you at OpenStack Summit, so we’ll be at Google Cloud Next in March, DockerCon in April, and Open Stack Summit in May.

    Shimel: And I think that one was in Boston?

    Balch: It is in Boston this year. That’s right.

    Shimel: Yes. Okay. Hey, we actually have some DevOps Connect events going on in Boston, Austin, and Chicago, but for another day. Mark Balch, Diamanti, thanks for being this episode’s guest on DevOps Chat and we’ll speak to you soon.

    Balch: Thanks a lot, Alan.

    Shimel: Thank you. This is Alan Shimel for devopsy.kinsta.cloud and we hope to see you soon on another DevOps Chats.

    — Alan Shimel

  • All Day DevOps: Exploring the New Ordinary

    All Day DevOps: Exploring the New Ordinary

    There you were. You heard about Docker, become mesmerized, floated into its new, special world … and then all chaos broke out. Things were not that easy and you scrambled for knowledge. Some folks saw the light and stepped forward, while others are still trying to figure the out the happy path home where containers will become part of their new ordinary.

    This is the true story of Daniël van Gils (@foldingbeauty), a developer at Cloud 66, and the 12 steps he laid out in his Docker journey. He shared this journey during his presentation at the recent All Day DevOps conference, and I’ll share a summary of his steps here. His journey will take you full-circle, from your “ordinary IT world” to the “very special container world” and back, to where the container world is the new ordinary.

    Containerization continues to be a hot topic in the software world, and one that is only going to grow and mature. As it matures, questions still abound:

    • How can you implement containerization into your organization?
    • Have you already started but run into roadblocks?
    • How can you be the hero of your organization’s journey?

    Daniël addressed this journey in his talk, “Docker Inside/Out,” at the recent All Day DevOps Conference.

    Daniël’s journey started in the Ordinary World (#1), the world you might be in now. You are at a conference, you read a book, you talk to some colleagues and you want to jump into containers. You receive the Call to Adventure (#2) … but you are human, you like the safe, you fear the unknown, so you want to Refuse the Call (#3).

    Thankfully, others—including Daniël—have taken the journey. They can take you under their wings, coddle you and then push you out of the nest. You Meet a Mentor (#4) and follow your heart. The heart is the good stuff—it is the core. Like most people probably would have, Daniël chose an artichoke to illustrate his point.

    The heart of the artichoke is the good stuff, but the outer leaves protect the heart and make up the whole artichoke. Each aspect of an organization has something that drives them—their heart:

    • Development – building elegantly performing code
    • Operations – stable, performing code
    • Business – delivering the right services to customers
    • Customers – consuming the service they want

    Your drive to satisfy the heart’s desire gives you the courage to Cross the Threshold (#5). You have a containerization machine, such as Docker. You put code in, creating nice little containers.

     

    Now that your infrastructure is set up, you test and know where your allies and enemies are (#6). The most important lesson here is that shortcuts are your enemy. Shortcuts are evil, promising a life of ease but delivering a life of pain.

    Daniël offers his Approach (#7) through the Docker journey to ensure you’re successful. You must KISSSSS:

    Keep It…

    • Slim – remove what you don’t need
    • Secure – ensure you have the latest updates; remove all secrets
    • Speedy – Follow best deployment practices and run performance tests
    • Stable – Use version numbers for your Docker files
    • Set – Immutable. Don’t put databases or complicated volumes into your containers. You can, but the the technology is still too new.

    The next steps in his journey cover Ordeals, Death and Rebirth (#8). Once you pass this point in the journey, you’ll understand what it takes to run Docker in production and have shown the proof of concepts. You’ll then reap the Rewards (#9) and have clear visibility to your Road Back (#10) to the new ordinary.

    For the successful heroes on this journey, now is your moment to shine.  The next steps: your Resurrection (#11) and to Return with Elixir (#12) where containers are now the new ordinary.

    If you want to learn more about Daniël’s Docker journey, be sure to watch his full All Day DevOps conference session (only 30 minutes). The other 56 presentations from the All Day DevOps Conference are available online, free-of-charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November, which hosted more than 13,500 registered attendees. Last week I discussed, “One Team, 5,000 Jobs: Life in the DevOps Jungle.” Next week, look for “Launching Products at Massive Scale: The DevOps Way” presented by Kishore Jalleda and Gopal Moore.

    — Derek E. Weeks

  • Confessions from Fannie Mae, Liberty Mutual and Capital One

    Confessions from Fannie Mae, Liberty Mutual and Capital One

    Learn from real-world stories from presenters at the All Day DevOps conference …

    True Story

    Over the past few years, Fannie Mae transformed the way in which it delivered software. Deploys increased from 1,200 per month to 15,000 per month.  At the same time, productivity increased by 28 percent while costs were reduced by 30 percent.

    But, how did Fannie Mae do it?

    During the All Day DevOps conference, more than 13,500 practitioners from around the world came together to learn from their peers in the industry. Barry Snyder, senior manager of DevOps at Fannie Mae, was one of 57 practitioners who shared his real-world journey through his enterprise transformation.

    Choke Point or Checkpoint?

    Eddie Webb, director of Software Delivery Platforms at Liberty Mutual, also shared his organization’s journey into DevOps.  Eddie described how Liberty Mutual evolved its organization, tool chain and processes to go from 20 percent of builds being deployed to 60 percent over the past 18 months. He described how the company documented its value delivery chain and identified all of its choke points in the continuous delivery process that would need to be automated to achieve its new velocity and quality targets.

    Screen Shot 2016-12-05 at 11.35.04 AM.png

    Run the Gauntlet

    The All Day DevOps conference recorded more than 57 practitioner-led sessions (30 minutes each) including those from Barry and Eddie. Every session is now posted online and is free to watch. The conference itself had three tracks: CI/CD, Automated Security and Modern Infrastructure, including:

    • Application Secret Management with KMS – Andrey Utis of Capital One
    • Getting out of the Job Jungle with Jenkins – Damien Coraboeuf of Clear2Pay
    • Serverful Computing – Taking One for the Team – Paul Czarkowski of Blue Box Group

    One of the best ways to learn about DevOps is to hear from the experiences of others who have run the gauntlet. Take 30 minutes this morning, at lunch or at home this evening to watch your first online session with All Day DevOps On-Demand.

    — Derek E. Weeks