devopsy.kinsta.cloud

Tag: docker

  • Docker On Windows, One Engineering System, Microsoft Uses DevOps To Reinvent Itself

    Docker On Windows, One Engineering System, Microsoft Uses DevOps To Reinvent Itself

    While at DOES San Francisco 2016 last week I had a chance to sit down with two presenters who work at Microsoft. One is someone I have interviewed before at both DOES and here on devopsy.kinsta.cloud, Sam Guckenheimer, Product Owner of Microsoft Visual Studio Cloud Services. Joining Sam and I is Ed Blankenship, Product Manager at Microsoft for Visual Studio Team Services. We had a wide ranging discussion on Microsoft’s own DevOps journey including the move to one engineering system, to SaaS based systems, security and finally Docker on Windows.

    Ed and Sam presented at DOES on Microsoft’s move to a single engineering system that now serves over 62,000 users within Microsoft alone! Sam also did a Lean Coffee session on security and compliance. All in all it was a great discussion with Sam and Ed. I know it is a bit long at 17+ minutes, but have a listen and enjoy.

    Also if you are interested in Docker on Windows, please check out this free and easy hands on lab Microsoft has made available to our readers here. The lab allows you to install Docker on Windows 10 and more. You don’t need anything on your end it is all self-contained in a web browser.

    You can see some of the other 50 videos we recorded from this years DOES San Francisco event on our YouTube channel, DevOpsTV.

    — Alan Shimel

  • Docker On Windows, Are You Serious? Microsoft’s Hands-on Lab Show’s You How

    Docker On Windows, Are You Serious? Microsoft’s Hands-on Lab Show’s You How

    You have probably heard or read about the availability of Docker on Windows and Azure. Both Docker and Microsoft have turned up the noise on their collaboration. But have you really looked into what these two are up to? If not, now is your chance. Microsoft has put up a great free hands on lab all about Docker, containers and Windows.

    In this hands on lab there are modules that feature Docker on Windows Server, Windows Nano Server and managing them with Windows 10. You can use Microsoft’s HyperV or not. It really is a great way to learn about containers, Docker and the Windows platform. You can play with this hands on lab by clicking here or if you prefer to cut and paste here is the link: http://msftdevops.learnondemandsystems.com/?wt.mc_id=AID557152_QSG_BLOG_22879. (Editors note: pop up blockers may stop the lab from loading. If you have an issue try disabling the pop up blocker.)

    Here is what the lab covers in a bit more detail:

    screen-shot-2016-10-27-at-1-02-39-pm

    I took the opportunity to play with the lab myself. While I am far from a developer or even an Ops person, I have to say Microsoft got this one right. Even I could complete it! The lab is supposed to take you 30 minutes but they give you 8 hours to complete. There is a great set of introduction videos that give you a full picture of what you are going to be doing. Here is an example in a still shot:

    screen-shot-2016-10-27-at-1-02-03-pm

     

    The lab is actually broken down into several modules that don’t have to be completed in one sitting. The lab requires no set up on your part at all. It runs right in your browser. The lab has you run two different Windows machines, one Win10 and one Windows Server. You can switch and log in right from the browser. The lab also has you log into PowerShell:

    screen-shot-2016-10-27-at-1-00-19-pm

    screen-shot-2016-10-27-at-2-24-52-pmIt literally gives you the CLI commands that you have to type, tells you when to switch between machines and guides you step by step through each of the several tasks. There is a great info bar on the bottom that has context sensitive instructions that walk you through the entire thing. There is also a sidebar which gives you a great oversight of the entire lab, where you are in the process and options to move around. All in all I thought they did a great job with the UX to make what could be a complex set of tasks easy.

    The whole Docker and containers on Windows theme is one has been bubbling up for some time now. While many doubted that it could ever happen, the new Microsoft seems to have made Docker right at home.

    From what I have been told there will be similar hands on labs offered for free on the Azure platform and more on containers. We will be working with Microsoft to dig in deeper here, so stay tuned for more soon!

    — Alan Shimel

  • Native Docker for Windows Arrives: Are Containers Finally Serverless?

    Native Docker for Windows Arrives: Are Containers Finally Serverless?

    Is Docker finally ready for the age of serverless computing? That’s the message from the announcement this week of native Docker container support for Windows Server 2016.

    If you’ve been following Docker for a while, you know that Docker containers originally supported only Linux operating systems. Then, Docker made its platform available for Windows users, too—but with the major caveat that Docker Engine ran inside a Linux-based virtual machine image on top. So, you could run Docker from Windows, but Linux was still the middleman.

    Real Docker Containers

    This week, that all changed. On Sept. 26, Docker announced the general availability of native Docker container support on Windows Server 2016. By “native,” Docker means that the containers run using new primitives—the equivalent of namespaces in Linux—that are built into the Windows kernel itself. There’s no more virtualization.

    But that’s not all. The complete Docker tool set also now runs natively on Windows. The Docker CLI, Swarm orchestration, data volumes and all the other building blocks of a Dockerized infrastructure are officially compatible, as well as Docker containers themselves.

    The support is the result of more than two years of collaboration between Docker and Microsoft, which is now a Top Five contributor to the Docker open-source project.

    Caveats

    Native support for Docker is a big accomplishment. But there are still some limitations, including:

    • Docker containers can only run natively on Windows Server 2016 and Windows 10. Other versions won’t work with Docker because they lack the kernel enhancements necessary to support Docker containers, Scott Johnston, Docker COO, explained in an interview.
    • Some advanced networking configuration features for Docker are not yet supported because the “plumbing” required to make those features work is not complete inside the kernel, Johnston said, adding Microsoft is working to change that.
    • Most notably, Docker containers on Windows can only run Windows apps inside the containers. In other words, you can’t run an app compiled for Linux inside a Docker container running on Windows. You would need a Windows host to do that. (And the opposite holds true: You can’t run Windows apps inside Docker containers on a Linux host.) This is not something that will change, according to Johnston.

    Docker Becomes Serverless

    Docker the company is celebrating the support as an opportunity to expand the Docker ecosystem immensely. Support for Windows “doubles the potential market opportunity” for Docker partners, the company said in a presentation about the announcement. (By the way, Docker has good reason to want to keep its partners happy by giving them a larger ecosystem to work with. The integration of Swarm into Docker earlier this year undercut the value of some of Docker’s partners’ products. But I digress.)

    The most interesting aspect of this news from a DevOps perspective, however, is that it means Docker is now much more serverless. A large part of the value of Docker containers is their ability to simplify application delivery. As long as you can package your app as a Docker container, you can deploy it on any server that has Docker running.

    You don’t have to think about the underlying server hardware or configuration. In other words, Docker allows you to become serverless, because you no longer have to think very much about the server you are deploying to.

    Previously, however, this flexibility was limited by the fact that you did have to think about the server OS. Now, with native Docker support for Windows, that’s no longer the case. It doesn’t matter whether your infrastructure is composed of Windows servers, Linux servers or both. As long as you install Docker on them, you can deploy to any server.

    Of course, there is still the issue of having to make sure your app is compiled for the right operating system. But that is easy enough to manage in most cases during the build process.

    — Chris Tozzi

  • 21 DevOps and Docker Reference Architectures

    21 DevOps and Docker Reference Architectures

    When building DevOps or continuous delivery (CD) practices relying on Docker, you can learn a great deal from others. Reference architectures reveal what choices people made, what practices they put in place and how they connected the dots.

    We at Sonatype pulled together a set of 21 reference architectures for folks building CD and DevOps practices using Docker. Why? DevOps professionals numbering 3,500 recently attended our webinar on, “Continuous Integration using Docker,” discussing just one reference architecture example. We recognized then there was a strong interest in the community to learn more.  That’s why we assembled 20 more reference architectures into this collection.

    Dive into the Details

    A picture speaks a thousand words. Beyond the images, we also share the details of each DevOps/CD Docker reference architecture. In the bottom right corner of each slide, we share the original blog or presentation source for the image. When one of the images piques your interest, you can then dive deep into the source. Cool, huh?

    Think of this set of reference architectures as open source. The slides are free to view, download and distribute. Share with others if you find them helpful.

    Help Others

    Oh, and if you have a Docker reference architecture we should add to the deck, please let us know. We would love to add it to our deck for others to learn from. Just like the others we included, you will be sure to get full attribution for your work.

    For those of you not using Docker, we did publish a set of DevOps and continuous delivery reference architectures last year on devopsy.kinsta.cloud.  With more than 86,000 views and 3,300 downloads, many people in the community have found it helpful.

    — Derek E. Weeks

  • Don’t Make Me Code: Environment Protection

    Don’t Make Me Code: Environment Protection

    Thanks for listening to “Don’t Make Me Code,” a Heavybit podcast that discusses developer experience design and the unique challenges of building developer-facing products. “Don’t Make Me Code” is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    In this episode, Steve and David are joined by Sean Li, lead product designer at Docker. Sean, Steve and David discuss Docker’s acquisition of Kitematic, the friction of attempting to replace a developer’s existing tool, and the increasingly prominent role of design in developer tools.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/274576622?secret_token=s-QHiyf” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Steve: We’re calling this episode of “Don’t Make Me Code” “Environment Protection,” and we’re here with Sean Li, lead product designer at Docker and one of the founders of Kitematic.

    Sean: Thank you.

    Steve: Thanks for being here with us. This is a really exciting conversation for me because we get to talk to a product designer for a company like Docker.

    Sean: It’s good to be here.

    Steve: Kitematic is a real interesting project for a bunch of reasons. I mean, it caught the interest of Docker. It’s also proven to be a really novel bit of development environment technology. And so we’re curious how it all got started.

    Sean: It got started in a university dorm room. I was a student from University of Waterloo in Canada, and the school actually offers a pitch competition. So we had this idea around making development environments easier, and so we pitched the idea and won. That’s how we got enough money to get started.

    Steve: Wow, and was it at all tied to Docker at that point?

    Sean: Docker was actually what we used for the back end. At the time, Docker was version 0.3. And then we’re on Hacker News, and we’re like, “Oh, this is interesting. It could be super useful technology to do what it wanted to do.” Our idea was essentially like Dropbox for your code.

    You get a Dropbox-like icon on your task menu, and then you can put code in it. For example, Django, it would just run it for you. And you can see it on the browser.

    Steve: Oh, nice. And so you have your production environment there, compiling for you and ready to go.

    Sean: Yeah. Then, when you added the code, it will sync to the cloud environment and then update right away.

    Steve: Nice. That’s awesome. It’s like a step beyond GitHub. It’s not just committing the code; all the CI and everything is built into this one workflow.

    Sean: That’s like the future plan right at the time, but at first, our MVP was super simple—just syncing code to a remote server and then seeing the changes.

    Steve: From the beginning it was implemented as a desktop application, you said like a Dropbox tray application.

    Sean: Yeah, because Dropbox was obviously the inspiration.

    Steve: And so why a desktop application?

    Sean: At the time, we actually didn’t realize the power of desktop application yet. At one point we used to pivot a lot, as startups usually do. And then we’re talking to customers, and they start realizing what we’re using underneath.

    We talk to Canadian Tire, the biggest retailer in Canada. They were looking for something like this, and they’re like, “Are you using Docker underneath? We want to use that.”

    At the time, the product we have is in the cloud, like an on-prem web browser setup. And then we started thinking of, instead of abstracting away Docker, why don’t we embrace it and then make it easier for the customers? Then we start brainstorming ideas, and it came across that having a desktop application is actually super beneficial.

    It’s like the best platform for developers, because they can import their code super easily. They can open a directory instead of having to upload files in a browser. You can also get a lot of access to their desktop, so once your tool gets more powerful, there’s more things that you can do.

    It’s also super cool because we use Chrome in an electron shell. And we wrote the UI in React, so that means we only have to worry about how the UI runs in that one browser, instead of having to worry about browser compatibilities across the board.

    Steve: So it’s not something React needed, it’s just running in a web view in the electron browser.

    Sean: And we make it feel as native as possible.

    Steve: All right, nice. So it’s a true desktop app in many ways, but specifically the UI, it’s a web UI.

    Sean: A lot of inspiration comes from the Atom Editor by GitHub. They’re using the same technology.

    Steve: And so this whole bit was open-sourced from the beginning?

    Sean: We were working on it, and then we open-sourced it.

    David: I’m curious what made you decide as you’re releasing this tool. Was open-source service a strategic decision, or you figured you could get more users by doing that? Or is it just wanting contributions from the community? Could you talk through a little bit of your inspiration behind that?

    Sean: I think the biggest part is if it’s something that touches their code, it’s really important that the developer trust it by seeing what the code is like.

    We’re really open about what we do, and also the part on gathering the feedback from the community is super, super helpful. I feel like, in general, open-source projects get more feedback from the community.

    Steve: With developer tools it seems like a great way to encourage participation. If you’ve got an open-source project, you’re inviting people to contribute. Obviously, our Opsee product for example, it’s closed, a SaaS application. So we don’t have the level of community support from developers that we would really like in some ways.

    Sean: We hired an engineer from Docker because he was a maintainer and he was super passionate about the project. It really helps on that front as well.

    Steve: Following from that, you launched this to an open-source community, did you have an idea of the kinds of users you were targeting at that point?

    Sean: We sort of did from the original product that we had, the Dropbox-for-code product. We had a signup list. A lot of these users are more leaning towards willing to use UIs and prefer something simpler. The super active ones are actually consultants that work on multiple projects.

    Steve: That’s an interesting segment. People there, maintaining a lot of projects, and they’re looking for an easy way to keep track of all that stuff. And Kitematic fit that mold.

    Sean: Yeah, and Docker being isolated also provides that value.

    Steve: That too. Now, if I’m maintaining multiple projects, I can have a Docker environment for each one. The app makes it very easy to move between those. And so specifically the React UI and the GUI, what was it like having that be part of the open-source project? Did you get contributions for that as well, or was that entirely internal?

    Sean: The React UI helped us in a way, because React is, according to a Google trend, almost on the same trajectory as Docker. So there are a lot of people willing to work on something that’s built in React. It was in a way accidental, because when we started, it was a really good technology to use.

    Steve: Did you have any designers contributing to the project?

    Sean: We do have design-minded people but not professional designers.

    Steve: For example, you’re a lead product designer at Docker and you, I assume, identify mostly as an engineer, or your background in coding. But there’s definitely this hybrid world in the developer tool space.

    Sean: Definitely. In the beginning we had hackthons. A lot of participants were trying to add features to Kitematic, then make a huge effort on figuring out the right UI right experience to build there. By default, I think what I wanted is sort of like designing an OS.

    A UI is very extensible, and you want to figure out the right points where you can extend on it. You actually focus on per component and really set an example of how you extend the current UI. When people contribute, it can follow that pattern. And you only had to make slight adjustments to it.

    Steve: That seems true of open-source projects, generally, too. You want build a broad base that people can build on top of and customize.

    Sean: Yeah, but the initial effort has to be really strong. Because otherwise you will get swayed by different opinions. And design by committee isn’t always a nice thing.

    David: That’s pretty interesting because you oftentimes see open-source projects that are sort of notorious, especially visually, for bad design or design by committee that ends up being a little too weird. I can think of some Linux desktop managers that I won’t name, but Kitematic is actually a beautiful application.

    Sean: Thank you.

    David: I’m curious how you managed to maintain that, even when you’re doing community-driven design.

    Sean: I think a big part of it is keeping focus and actually be willing to cut things. In the beginning, Kitematic actually had build, you can actually build an image and then push it. But to keep focus and make things simple before we figure out the next steps is actually making it run content only.

    Essentially the focus of Kitematic was helping users install Docker on their computer. You know, in a visual way, it feels very reliable. We solve a lot of underlying issues with VirtualBox in the beginning. And then after that choose a beautiful catalog where you can click one button and run a container. And then that’s the experience we were aiming for and focused on. So it’s super obvious what to do.

    Steve: Did you ever get push-back on those decisions from the community where you had to assert your position strongly there?

    Sean: No, I think the community is mainly supportive. What I had to manage was people trying to add too many things that are not focusing on that road map currently.

    Steve: I think I remember you saying, on the Software Engineering Daily podcast, that you did end up cutting quite a few things from the product after a bit of time. Can you talk more about that? You were trying to clean it up?

    Sean: The one thing that was cut was building images from it. Even though it’s a super important thing, we decided to delay it because we want to do it right. Instead of releasing something super early, and then get sidetracked into different things, we want to keep focus until we figure out this is it.

    And then release that as a whole. Talking about cutting more, the next version of Kitematic is sort of Docker for Mac. It really solved the core issues that Kitematic was trying to solve in the beginning, to install Docker reliably on people’s desktops in a way that it almost feels like it’s seamless.

    You don’t feel like it is running inside a VM at all, and that’s the goal in the beginning. You install Docker for Mac. When you open the terminal you can type “Docker run,” and it just works.

    Steve: Had the Docker for Mac project begun before the Kitematic acquisition?

    Sean: It was after the acquisition.

    Steve: So you were involved with that from the beginning.

    Sean: Yeah, I was the founding member of Docker for Mac.

    David: So it’s interesting to think about this this sort of internal debate. I guess with most design questions it’s customizability and configuration versus limited options, and this curated a beautiful experience. I feel like that happens even more with developer tools.

    It seems like Kitematic and Docker for Mac, some of these tools, are definitely trending towards the latter there, where you’re removing some of the optionality of Docker to make a better user experience. I think that’s really interesting.

    Sean: I think the two tools are, in a way, more complimentary. People usually use Kitematic by having the Kitematic UI side by side with the terminal. So they like the transparency that they’re getting from the Kitematic side, and the sense of it feeling reliable.

    “I see things are running. I can check the logs super easy.” On the other end, the terminal is how they’re used to working before, and that was sort of expected in the beginning. And then you go from there. You never want to displace a tool that they’re already using, one they’re so familiar with.

    So instead we embraced that but made Kitematic more of a complimentary to the Docker CLI.

    Steve: That’s really interesting. I’ve seen similar comparisons made with GitHub and some of their tooling. GitHub has their Mac app and their web UI and stuff like that. And I found myself going to them for things like diffs and comparison, merge states, something where I want to get a lot of text or get some status update.

    But for actions, if I want to do merges, everything I’m actually doing with it I’m still doing through the command line. But that’s a really interesting comparison, because it sounds like you’ve made the same distinction. To view things, maybe you go to the UI. But to actually take action?

    Sean: GitHub desktop is like a huge inspiration for all these decisions as well.

    Steve: I never really heard that distinction made this clearly, that there are certain things that developers are going to continue to want to do through the command line and through terminal. But that we can make the experience better in some ways, like just figuring out whether a container is running or not. That’s really nice to have a UI to see that all the time.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

  • Docker Automates and Democratizes Container Orchestration

    SEATTLE–(Business Wire)–DockerCon – Docker today announced Docker Engine 1.12 with built-in orchestration, a powerful combination that provides Developers and IT Operations with a simplified and automated experience to deploy and manage Dockerized distributed applications – both traditional apps and microservices – at scale in production. By adding this additional intelligence to Docker Engine, it becomes the orchestration building block, creating a model for engines to form a self-organizing, self-healing pool of machines on which to run multi-container distributed applications. When integrated into Docker Engine, these new capabilities optimize ease of use, resiliency, performance-at-scale and security – all key requirements that are missing in other orchestration systems. As a result, organizations can be assured that their dev and ops teams are aligned on unifying the software supply chain to release applications into production more rapidly and frequently.“Orchestration is at the same stage today as containerization was before Docker. You either need an army of experts to build it, or you lock yourself to a monolithic platform which will drastically reduce your choice of suppliers,” said Solomon Hykes, founder and CTO at Docker. “Three years ago we brought containerization into the mainstream by making it usable for non-experts, without lock-in. We think it’s time to do the same for orchestration. This is a necessary step for the industry to move forward, and as the leaders of the containerization market it’s our responsibility to lead this change.”

    As with all Docker tooling, this integration is always about choice and flexibility for users. “Swarm mode” is an optional feature that users can select to “turn on” built-in orchestration, or they can also elect to use either their own custom tooling or third-party orchestrators that run on Docker Engine. This approach aligns with the Docker platform’s batteries included but swappable architecture, which has spurred the growth of a vibrant and collaborative ecosystem.

    “As the adoption curve for Docker continues to grow, developers have encountered growing pains with orchestration at scale,” said Fintan Ryan, industry analyst with RedMonk. “With the inclusion of secure built-in orchestration in the 1.12 release, Docker is providing developers with a simple-to-use, yet extremely powerful, orchestration tool while further investing in a consistent, easy-to-manage experience for operations.”

    As organizations begin to make increasing investments in containerization, and with more than 60 percent of them running Docker in production, they are seeking more sophisticated orchestration tooling to expand their deployments across both applications and teams. Docker 1.12 addresses these requirements with functionality that spans the entire application stack across compute, network and storage.

    Ease of Use

    Docker 1.12 dramatically simplifies the process of creating groups of Docker Engines, also known as swarms. The self-organizing, self-healing capabilities of swarms are now backed by automated service discovery and a built-in distributed datastore. As a result, it takes just one command to add a Docker Engine and horizontally scale a swarm.

    Resilient

    The new service deployment API describes all the resources and components with a single command that allows operations teams to run and scale a service. Through the API, the swarm is aware of the application defined and will continuously check and reconcile the environment against the requirements of the application when something adverse happens. Unlike other systems, the swarm itself has no single point of failure. The state of all services is replicated in real time across a group of managers so containers can be rescheduled after any node failure.

    Performance at Scale

    Docker orchestration includes a unique in-memory caching layer that maintains state of the entire swarm, providing a non-blocking architecture which assures scheduling performance even during peak times. Additionally, the system has a built-in routing mesh technology that addresses the challenge of how to provide container-aware load balancing. The routing mesh ensures that requests are made to the right containers regardless of where they have been scheduled within the swarm.

    Secure By Default

    Each Engine is automatically assigned a cryptographic identity which ensures that only validated Engines can be accepted into a swarm. Moreover, Docker Engine comes with mutually authenticated TLS, providing authentication, authorization and end-to-end encrypted communications among every node participating in the swarm, without the operator having to take any steps to enable it.

    Availability

    There are three ways that users can get Docker 1.12, which is currently a release candidate with general availability planned for July 2016. First, it is available now as part of the newly opened public beta of Docker for Mac and Docker for Windows athttps://www.docker.com/getdocker. Second, it is available through cloud-optimized experiences that bundle custom plugins that provide deep integration between Docker and the target platform capabilities including networking, load balancing and SSH key management. Docker for AWS and Docker for Azure are the best ways to deploy Docker Engine on these platforms and are available in private beta at https://beta.docker.com. Last, Docker 1.12 is also available as a binary download or a package for all major Linux distributions at https://docs.docker.com/engine/installation/linux.

    For more information:

    About Docker, Inc.

    Docker, Inc. is the company behind the Docker open source platform, and is the chief sponsor of the Docker ecosystem. Docker is an open platform for developers and system administrators to build, ship and run distributed applications. With Docker, IT organizations shrink application delivery from months to minutes, frictionlessly move workloads between data centers and the cloud and can achieve up to 20X greater efficiency in their use of computing resources. Inspired by an active community and by transparent, open source innovation, Docker containers have been downloaded more than 4.1 billion times and Docker is used by millions of developers across thousands of the world’s most innovative organizations, including ADP, Baidu, the BBC, Goldman Sachs, Groupon, ING, Yelp, and Spotify. Docker’s rapid adoption has catalyzed an active ecosystem, resulting in more than 4,300,000 “Dockerized” applications, over 100 Docker-related startups and integration partnerships with AWS, Cloud Foundry, Google, IBM, Microsoft, OpenStack, Rackspace, Red Hat and VMware.

    Mindshare PR
    Heather Fitzsimmons, 650-279-4360
    press@docker.com

  • RightScale Releases New DevOps and Docker Trends in Follow-Up to State of the Cloud Report

    SANTA BARBARA, CA — (Marketwired) — 05/11/16 — RightScale® Inc., a demonstrated leader in enterprise universal cloud management,today announced the results of the RightScale 2016 State of the Cloud Survey: DevOps Trends. The fifth annual RightScale State of the Cloud Survey was announced in February. This follow-up report provides new data and analysis in a deep dive on DevOps, Docker, and container trends. The RightScale 2016 State of the Cloud Report series is the result of a survey to 1,060 technology professionals at large and small enterprises across a broad cross-section of industries.

    The survey results are available in the RightScale 2016 State of the Cloud Report: DevOps Trends, which can be downloaded at www.rightscale.com/2016-devops-trends-report.

    “The survey shows that the vast majority of enterprises are adopting DevOps and this usage is being driven from the bottom up by projects, teams, business units, and divisions,” said Kim Weins, vice president of marketing at RightScale. “Companies are using a portfolio of DevOps technologies that include configuration management tools as well as containers. While Chef and Puppet are still the most widely used DevOps tools, Docker has seen extraordinary growth, more than doubling in the past year. Companies report a lack of experience with containers and have made educating themselves a top priority in the coming year.”

    Highlights of the RightScale 2016 State of the Cloud Report: DevOps Trends include:

    • DevOps is growing, especially in the enterprise: DevOps adoption increased from 66 percent in 2015 to 74 percent in 2016. DevOps adoption is strongest in the enterprise (81 percent of enterprises adopting DevOps compared to 70 percent in SMBs). Enterprises are adopting DevOps from the bottom up: Adoption of DevOps by projects or teams (29 percent) and business units or divisions (31 percent) is more common than company-wide adoption (21 percent).
    • Docker usage doubles while Chef and Puppet are neck and neck: Overall, Chef, Puppet, and Docker are the top three DevOps tools used by respondents (32 percent, 32 percent, and 27 percent, respectively). Among enterprises, more use Puppet (42 percent) vs. Chef (37 percent) and Docker (29 percent). Docker is the fastest growing DevOps tool, with adoption more than doubling year-over-year from 13 percent in 2015 to 27 percent in 2016. In the enterprise, Docker also saw more than 2x growth (from 14 percent to 29 percent) while Ansible nearly tripled (from 8 percent to 23 percent). Docker could soon be the most used DevOps tool in the enterprise as 38 percent of enterprises have plans to use it. This compares to 20 percent who plan to use Chef and 19 percent who plan to use Puppet.
    • DevOps users use multiple tools: Less than half (43 percent) of companies are using a configuration tool such as Chef, Puppet, Ansible, or Salt. Use of multiple configuration tools is more common (25 percent) than a single configuration tool (18 percent) and 67 percent of companies using Chef or Puppet also use the other tool. Configuration tools are also often used with Docker; 80 percent of Docker users also leverage at least one configuration tool.
    • Container adoption is maturing, especially in enterprises: Overall, 26 percent of respondents have workloads already running in containers (8 percent in development, 18 percent in production). 36 percent of respondents are experimenting with containers, while 25 percent are learning about containers. Enterprises are using containers more than SMBs. 29 percent of enterprises have workloads running in containers versus 24 percent of SMBs, and 41 percent of enterprises are experimenting as compared to 33 percent of SMBs.
    • Docker seeing greatest adoption in Europe, with tech companies, and with enterprises: Evaluating Docker adoption across different geographies, industries, and roles, RightScale found that current use of Docker is heaviest among tech organizations (32 percent), enterprises (29 percent), and developers (28 percent). Use of Docker in Europe (34 percent) is also well above average.
    • Significant interest in containers on bare metal. Containers are currently being deployed primarily on virtual machines (29 percent) versus bare metal (12 percent). There is significant interest in deploying containers on bare metal with 24 percent of respondents having plans to do so in the future. Most containers are built using traditional Linux distributions such as Ubuntu (43 percent), CentOS (39 percent), and Red Hat (37 percent). CoreOS (12 percent) is the most widely adopted of the minimalist operating systems, which are designed specifically for containers.
    • Biggest challenge with containers is lack of experience for newbies: For respondents who are not currently using containers, lack of experience was by far the top challenge (39 percent). The top challenges cited by respondents who are already using containers were security (29 percent) and immature technology (29 percent).
    • Container focus in 2016 is education and experience: The top container initiative in 2016 will be getting more educated (62 percent), followed by conducting more experiments with containers in dev/test (44 percent) and production (28 percent), as well as expanding container use in dev/test (28 percent).

    Survey Methodology

    RightScale conducted its annual State of the Cloud Survey and RightScale State of the Cloud Report: DevOps Trends in January 2016. The survey questioned technical professionals across a broad cross-section of organizations about their adoption of cloud computing. The 1,060 respondents range from technical executives to managers and practitioners and represent organizations of varying sizes across many industries. Their answers provide a comprehensive perspective on the state of the cloud today. The margin of error is 3.07 percent.

    About RightScale

    RightScale Universal Cloud Management enables leading enterprises to accelerate delivery of cloud-based applications that engage customers and drive top-line revenue while optimizing cloud usage to reduce risk and costs. With RightScale, IT organizations can deliver instant access to a portfolio of public, private, and hybrid cloud services across business units and development teams while maintaining enterprise control. RightScale Consulting Services help companies develop cloud strategies, deliver cloud projects, and optimize cloud usage. RightScale was named a “100 Best Places to Work in 2015” by Outside Magazine and was listed in “The Best Enterprise Cloud Computing Startups to Work For in 2015” by Forbes. Since 2007, leading enterprises including Audi, Pinterest, and Yellow Pages Group have launched millions of servers through RightScale.

    Contact:
    Bret Clement
    Email Contact

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    In part one of my conversation with Stefan Thies, Sematext DevOps expert. we spoke about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Let’s continue the conversation.

    David Geer: What makes Logsene centralized? Where is it centralized a typical deployment? What are the advantages of it being centralized?

    Thies: Traditionally, systems store logs from multiple applications on every server. This makes it difficult to search logs for troubleshooting or auditing purposes. To find a relevant log entry, operators must log in to each server, find the right file on this server and search in this file for a relevant message.

    This can easily take several minutes. With systems storing logs on multiple servers, it is difficult to correlate data, so minutes can turn into frustrating multi-hour root cause-hunting sessions, often involving multiple team members. Logsene provides a central full text index for all log data coming from multiple servers and applications. There is no need to log in to multiple servers and search/grep multiple log files. All team members see all data at the same time and can collaborate much more efficiently.

    Having a central place for search makes it much easier to find logs quickly, run analytic queries or centralize alert rules in a single place.

    Geer: What is it about what Logsene does for Docker logging?

    Thies: Sematext Agent for Docker interacts with the Docker API to automate the collection, parsing and processing of logs, metrics and events. It detects containers automatically (no setup or configuration for a new container!) and supports many log formats for applications on Docker Hub such as Nginx, Apache, Elasticsearch, Solr, MongoDB, MySQL and many more.

    Geer: What does your statement mean, “Some DevOps engineers even think of Logsene as ‘ELK Stack on steroids’”?

    Thies: The ELK Stack is a combination of open source tools: Elasticsearch, Logstash and Kibana. These tools are very popular for log shipping (Logstash), indexing and search (Elasticsearch) and visualization (Kibana dashboards). However, these tools are missing many enterprise features like multiuser user interface, role-based access controls, secure transport of logs or out-of-the-box configurations and an easy query language to search logs. This is where Sematext Logsene adds the relevant enterprise features to these open source tools to make them easy to use and securely integrated into the enterprise information technology infrastructure.

    Maintaining Elasticsearch clusters is another problem for many people. It requires a lot of expertise. Using Logsene does not require this sort of expertise. The Logsene SaaS provides a managed solution where users don’t need to know anything about maintaining Elasticsearch clusters or the required infrastructure. Sematext takes care of that for them. Sematext customers can benefit from these managed services and pay a fraction of the cost compared to self-hosted ELK stacks, where storage, computing and networking infrastructure and manpower for highly skilled workers counts.

    Geer: What is Tutum? What is so important about ease of launching SPM and Logsene Docker on Tutum?

    Thies: Docker acquired Tutum in 2015. Tutum provides management of Docker containers as a cloud service. Tutum is a platform made to operate “Dockerized” application stacks. In the case of Sematext Agent for Docker, the Tutum integration deploys Sematext Docker Agent automatically to all cluster nodes in a few seconds. These nodes could run on different cloud platforms such as Amazon, Digitalocean, Rackspace or on premises. Again, a big time-saver for people running larger infrastructures or hybrid clouds. Supporting Docker products and having excellent integration with Docker products is important for Sematext.

    The Sematext Docker Agent runs on all solutions offered by Docker e.g., Docker Swarm and others such as Google Kubernetes, Google Container Engine, RancherOS, CoreOS, Amazon ECS or Hashicorp Nomad.

    Geer: Please discuss the Sematext Docker and Docker image?

    Thies: Sematext Docker image is available on Docker Hub, and we shared the Tutum Stackfile for Sematext Docker on Stackfiles.io. The easiest way to get it is to use Sematext, which generates the stackfiles for you, including tokens.

    Docker Hub is the central registry for public application images including related installation instructions. Docker users can find out quickly how to run a “Dockerized” application with the help of the instructions in Docker Hub. Typically, this requires a little configuration work before a container can be started with the right settings.

    Stackfile.io is the place where people share preconfigured application stacks (e.g., a web service together with the required database). Sematext provides a preconfigured setup for Sematext Docker Agent to monitor all containers in Tutum Cloud. This means one can launch Sematext Docker Agent and monitor all containers running in Tutum Cloud with just a single click for the chosen stack file and by entering a unique identifier (Token) for the application. That is really all it takes.

    When you log in to Sematext SPM or Logsene the general installation instructions have all required configurations displayed. Sematext users can simply copy/paste the relevant commands including unique identifiers (tokens) for their application to get the Sematext Docker Agent deployed. This, too, is a big time-saver for our users and it gets them up and running in just a few minutes.

  • Why Is There No DevOps Manifesto?

    Why Is There No DevOps Manifesto?

    Let’s start with a brief story:

    There’s a scene in a movie called “A Few Good Men” in which a lawyer is questioning a cadet. The lawyer directs the cadet to turn to the page in the thick Marine Code Manual that explains where to find the canteen. And the cadet replies, it isn’t in there.

    And the lawyer says, “Well, you eat, don’t you?”

    And the cadet says, “Yes, sir, three times a day!”

    The meals for everyone aren’t served by the book because the canteen isn’t included.

    This isn’t a history for the term DevOps or of the many people who’ve led, presented, written and interviewed about the DevOps movement. I’ve got a footnote below with some pointers for that. This is simpler than that: It’s why today there is no DevOps Manifesto and why I don’t think there will ever be.

    Now, I think I understand the desire for a DevOps Manifesto. I’ve met a lot of folks, company executives and analysts, who still dis DevOps a bit because it lacks a certified definition, an official manual or a definitive statement. Basically, a manifesto; you know, a canonical thing you can put on a shelf.

    Manifestos can be nice. They lay down the vision, the intent, the mission. They can appeal to a wide range of people, from revolutionaries to functionaries to dignitaries. It isn’t for sparking conversations so much as for orderly setting down of intent. A manifesto can bake in the meaning and content of something.

    A manifesto has that “rule of law” book feeling to them. Manifestos also have provenance and pedigree—some group of some renown sat down to discuss some topic for some specific resolutions and wrote the sum of it up. Manifesto wording can be checked to ensure real-world compliance.

    You can almost slap a copy of a Manifesto on a table to settle an argument about the manifesto’s topic. And assuming the words were crafted carefully enough, it will be the end of discussion. And that’s exactly the problem with a manifesto with respect to DevOps.

    I say: That’s not what DevOps is about.

    DevOps begins with stopping the nightmare scenario pitting Ops against Dev—or any other part of the organization. But it comes so much further. It’s for a new age of computing at webscale, using tools, teamwork and techniques to get to better, happier, more engaged outcomes for customers. Some of it is so new we are still figuring out words for it.

    The DevOps movement isn’t slowing down, either. Analysis of the Google results page shows it:

    DevOps searches over time Sep2015

    In all of this time, no one of any significance in the community has attempted to create a “manifesto for DevOps.” I’d say the Godfather of DevOps himself, Patrick Debois, was super insightful on how he went about ensuring no official manifesto could get written. (I’m not going to review the lessons learned from “The Agile Manifesto,” although good lessons were learned from the things created in the wake of having an Agile Manifesto.)

    If you have a manifesto for something, then all conversation in the domain will, rather quickly, go back to the manifesto for resolution instead of everybody working together. That’s not handy in an ever-fluid, often on-fire, scale-up fast, resilience-oriented, rapidly changing and evolving IT situation.

    As Andrew Clay Shafer (@littleidea) said recently, DevOps is about optimizing for people. With the pace of IT and tech ever accelerating, no manual or manifesto could keep up. Which reminds me of one of my favorite quotes:

    Companies die because their managers focus on the economic activity of producing goods and services and they forget that their organization’s true nature is that of a community of humans.

    — Arie De Geus

    One thing you should know for sure about DevOps: The community likes to enhance, educate, enliven, inform and energize people. Basically, everyone is wide open to embracing anything that helps out because webscale is hard. So yeah, DevOps now aligns with “microservices” because microservices can further the DevOps cause. Just like kanban, Docker containers, automated configuration and InfoSec. And anything that enables Dev and Ops to deliver more, better, faster to customers. It’s about serving the customers.

    Meanwhile, if you’re looking for more, you’ll usually find a bunch of folks … figuratively speaking … over in the kitchen at the canteen, serving up awesome and working hard on doing it better and faster.

    And everyone interested is welcome to sit down and enjoy what’s served.

    Seriously. Join in. You do not need a manifesto to.

    P.S.  It was with deep hesitation that I wrote this. There are simply so many awesome people who’ve made all this happen and I am not one of them. Literally the finest collection of technologists—and in many cases, human beings—I have the honor to know who make this community what it is.

    You should check out, in no particular order:

    • The amazing (and amazingly modest) Patrick Debois
    • The writings and teachings of Mark Burgess (one of the most under-recognized geniuses I have encountered)
    • The CALMS write-ups by John Willis (@botchagalupe) and Damon Edwards (@damonedwards)
    • Damon’s five-minute video on DevOps history
    • John’s frank exploration of the issue of burn-out in IT (and that is directly related to why we are having DevOps conversations)
    • Mary Poppendeick’s lean development work
    • John Allspaw and John Hammond’s 2009 Velocity keynote that sorta kicked this whole thing off
    • Ben Rockwood’s (@benr) comprehensive LISA11 keynote
    • The many videos of Andrew Clay Shafer (@littleidea) and his brilliant guidance. Sometimes with katana and sometimes with gattling gun.
    • Jez Humble’s books (particularly “Continuous Delivery” and “Lean Enterprise”)

    And works from folks including:

    • Jeff Sussna (@jeffsussna)
    • James Turnbull (@kartar)
    • Adam Jacob
    • John Vincent (@lusis)
    • Bridget Kromhout
    • Adrian Cockcroft
    • Gareth Rushgrove
    • Kris Buytaert
    • J. Paul Reed
    • Ernest Mueller
    • James Wickett
    • Ines Sombra
    • Karthik Gaekwad

    I mean, this is a global phenomenon, were you expecting a short list? This isn’t even complete!

    Oh, and the VisibleOps guru Gene Kim, who came and showed us all a way to really blow the doors off the DevOps news.

    These people are helping all of us do better Dev, better Ops, better IT, better business so that you can have better customers. It’s amazing. It really is. And it’s wonderful.

    Also, if you get a chance to spend time at a DevOps Days anywhere, you should.

  • DevOps Chat: Mike Pittenger, Black Duck Software on Open Source and DevOps

    DevOps Chat: Mike Pittenger, Black Duck Software on Open Source and DevOps

    I recently had a chance to speak with Mike Pittenger, Vice President of Strategy for Black Duck Software. Black Duck is a company that I have been following for many years. They pioneered the idea of auditing open source usage to ensure compliance with licenses and such. Of course over the years they have expanded into other areas as well, including security, DevOps, containers and more. Mike and I discuss Open Source and DevOps.

    Mike and I had a good conversation I hope you will enjoy. You can listen below or follow along with the transcript below that.

    Alan Shimel:               Hi, this is Alan Shimel, devopsy.kinsta.cloud, here for another DevOps Chat. Today’s guest on the chat is Mike Pittenger of Black Duck Software. Mike, welcome.

    Mike Pittenger:            Thanks very much, Alan. Nice to be here.

    Alan Shimel:                So, Mike, our audience, you know, I’m sure many people in our audience have not met you before, unless you’ve invited your relatives. But can you give us a little background on who you are, Mike, and your role with Black Duck?

    Mike Pittenger:            Sure, sure. So I’ve been in the software security space for about 16 years. I started with @stake back when it was a leading software security consultancy. I ran the product team there, and then when Symantec acquired @stake I spun off that team to form a company called Veracode. I stayed there for a couple of years and then started consulting work with Bit9, Digital Guardian, BeyondTrust, Coralogix.

    And Black Duck was a customer of mine – a client of mine – about two years ago and we were involved in a discussion of product enhancements and so on. And I came back to them about a year ago. They called me back in for another project and I found the opportunity so interesting that we – I came on full time to head up their security strategy.

    Alan Shimel:                Very good. And Mike, believe it or not there may be some folks in our audience who aren’t familiar even with Black Duck.

    Mike Pittenger:            That would be hard to believe, Alan. [Laughs] We’ve been around since –

    Alan Shimel:                A long time, I know. I’ve been writing about you all for a long time. But that being said, just in case let’s give them a little background.

    Mike Pittenger:            Sure. The company is about 13 years old and what we’re very good at is looking at source code or binaries to identify the open-source components that are used in that. Once we’ve identified those we can tell the customer anything they want to know about because we track over open-source projects and we’ve been doing it for about 13 years.

    For the first 10 or 12 years of the company’s life, the primary interest in this was the people would run our software internally in order to determine the open source, but they were interested in it because of license compliance. They wanted to make sure that they weren’t using reciprocal or restrictive licenses like GLP code and _____ distributing; something that might put their own IT at risk.

    About five years ago we started adding security information from the National Vulnerability Database and a year ago we came out with a product specifically focused on security. It’s a big deal because while license compliance is still an issue for risk managers, open-source software, because it’s become so ubiquitous, is also a concern to security people.

    There are – there have been over 4,000 new vulnerabilities – excuse me, over 6,000 new vulnerabilities reported in the National Vulnerability Database just in the last two years. So unless people are aware of the software they’re using and the components they’re using in that software, they’re unable to protect themselves from these vulnerabilities and that’s where we come in.

    Alan Shimel:                Excellent. So Mike, I wanted to talk to you a little bit today about open source. And certainly open source vulnerabilities and scanning are big topics today, especially when we talk about that in the context of containers. And I know Black Duck is actually doing some work around container security and vulnerability scanning, etc. But I wanted to just take it up another 50,000 feet and talk overall open source’s – open source’s role in this whole DevOps revolution/evolution, if you will. Can you talk about that?

    Mike Pittenger:            Sure. Yeah, so I mean the big thing, of course, is that a lot of these particular – with the popularity of containers, a lot of those are running on a lightweight version of Linux, and of course that’s open source. Where we see the risk with that is not in using Linux, it’s whether an open-source operating system or component is more or less secure than a commercial component. It becomes a religious argument and you can argue it until the end of the day.

    The fact is you just need to have awareness of what you’re using and monitor that closely. So we did an informal study of the containers on Docker Hub and found that most of them, the majority of those, had security issues with them. And it was easy to see how this happens because containers are constructed just like software is constructed. People are pulling in often open-source components and they build a LAMP stack and add their application to it.

    If they’re going to deploy a new application they may strip off the application layer, maybe strip is right down to the Linux core again, and then build it back up. But if they’re reusing these components and a vulnerability happens to be in there from the beginning, or one is disclosed after you’ve built that initial container, it’s just going to propagate itself. So people are justifiably concerned, I guess is the right word, about the security of these.

    And from our position, it’s really more about understanding the hygiene of the container, understanding what you’re using, and then monitoring that closely because with 6,000 new vulnerabilities in a two-year period, clearly your security posture can change without you doing anything.

    Alan Shimel:                Absolutely. Mike, what we’ve seen in – you know, it’s funny. I was just going over the results of a survey we’re going to be releasing soon about DevOps tools adoption. And it was split between large organizations, over 10,000, and then about an equal number of under 200. So sort of dumb-belled with not a lot in the middle. But fully 25 percent or more, of all of the tools that these people were bringing in, whether they be – let’s call it a legitimate software supply chain or in through the back door, if you will, shadow – 25 percent of them or more were open source.

    Mike Pittenger:            That’s actually lower than what we see.

    Alan Shimel:                Really?

    Mike Pittenger:            So we have a portion of our business, Black Duck On-Demand. So we’re often brought in during an M&A transaction. So if I’m going to buy your software company, I’m going to “Black Duck it.” They use it as a verb. I’m going to Black Duck it to understand the risk associated or whatever risks there might be. And so these are commercial applications. You assume they have plenty of proprietary value because somebody else is interested in acquiring that company.

    And even in those cases we’re seeing that the code base, 35 percent is open source. If you’re talking about an in-house application that, say, a bank is running, they’re building the applications, consuming them themselves, it’s not unusual to see those applications comprised of 70-80 percent open source. And it only makes sense that they’re doing that.

    I mean, open source is saving them from having to build this functionality from scratch, so it’s providing critical functionality without an acquisition cost. So it’s shortening the time to market and it’s lowering the development cost. It makes perfect sense that they’re doing this.

    Alan Shimel:                Got it. I’m surprised to hear. I thought 25 was a big number. You’re saying it’s a small number and it’s pretty significant.

    Mike Pittenger:            Yeah. It is a small – so if you look at those commercial applications – and we’re going to be publishing a report of our own pretty soon on those – but the average number of discrete open-source components in those applications is over 100.

    Alan Shimel:                Got it. Hmm…crazy. Mike, I want to just – believe it or not, as I told you before we started recording, our time here goes so quickly and we’re eight-nine minutes into our recording. I wanted to just quickly pivot into the whole container and open source and security around it and some of the things Black Duck is doing. Again, people in our audience may not be familiar. Would you give them a little – just a little taste of it?

    Mike Pittenger:            Sure. So we have partnerships with both Red Hat and Docker. In the Red Hat world we’re partnering with them for security on their open shift platform as a service and their container platform Red Hat Atomic. And the idea there is simply to be able to quickly assess the security of a container before you deploy it. So we can run our scan client as a container, pull in a Docker image from a repository and scan it before it’s deployed. Very simple.

    From our world, you can point Black Duck at a container, you can point it at typically a development process that’s integrated with a build server. But you could point it at a repository if you wanted to, and to us, a container is simply another code location _____ scanning. And when we’re looking at that, we’re not just looking at the components, we’re also looking at what makes up those components and are able to identify these down to the patch level.

    So instead of saying you’re using a bad version of _____ and Linux, we can actually look at each of the components and tell if those have been patched for reported vulnerabilities just to make the DevOps manager’s job faster, understanding what they need to worry about and what they don’t need to worry about.

    Alan Shimel:                Got it. Excellent. And you know, Mike, we only have about a minute or two left here, but obviously it’s a dynamic situation – the whole container thing, the whole what’s happening with DevOps and open source and all of this. If you can gaze into a crystal ball, where do you see us going in the next three to five years with this? Where do you see it heading?

    Mike Pittenger:            Well, I think the containers are going to obviously grow rapidly. We see some – it’s a hesitation in deploying, to switching over to containers in some worlds, but I think it’s simply trying to understand it better. An analogy might be people switching over to cloud computing a number of years ago. They want to understand both the benefits and the risks before they throw in all of their chips on it.

    So we see some of the very large companies embracing it aggressively, but a lot of the, say in the financial services industries, still testing it, playing with it, seeing – trying to decide when they’re going to jump in and whether they’re going to start in the shallow end of the pool or go right to the deep end.

    Alan Shimel:                Got it. Got it. Believe it or not, Mike, we’re at our time limit. I know it seems to go so quick. But I wanted to thank you for appearing today, and you know what? If people want to find out more about Black Duck, quick plug for their web site?

    Mike Pittenger:            Yeah, sure. It’s BlackDuckSoftware.com. We do provide a – there’s a free trial that will work on an application or on a container and you can sign up for it right on the web site. It’s www.BlackDuckSoftware.com.

    Alan Shimel:                Perfect. Mike Pittenger, Black Duck Software. Thanks for being today’s guest on DevOps Chat.

    Mike Pittenger:            Thanks very much, Alan. Nice to be here.

    Alan Shimel:                Okay. This is Alan Shimel for devopsy.kinsta.cloud.