devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • DevOps Chat: Chef Goes All-In on Open Source

    DevOps Chat: Chef Goes All-In on Open Source

    In front of next month’s ChefConf in Seattle, Chef has announced a major refinement to its business model. Affirming and clarifying its commitment to open source business models, Chef will now and in the future release all of its software as open source.

    In this DevOps Chat, we speak with Corey Scobie, SVP of Products and Engineering at Chef. He explains that Chef will be adding services, training and support to the open source software, as well as packaging all of this together.

    For the full story, be sure to listen to the podcast and check out the video version of this interview below, along with coverage on devopsy.kinsta.cloud.

    [videopress nUXwRMxU]

    Transcript

    Alan Shimel: Hey everyone, It’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to a DevOps Video Chat. Today’s DevOps Video Chat features Corey Scobie, SVP of Engineering, is it?

    Corey Scobie: Product and engineering, yep.

    Shimel: Product and engineering now at Chef. Hey, Corey, welcome.

    Scobie: Thanks, Alan, happy to be here.

    Shimel: I’m happy to have you here. Today is a red letter, big day in the history of Chef. You guys are announcing some pretty significant changes to the model is I guess what I would call them. Though, I don’t think the changes are as big as some may initially feel, but let’s go over them. I’m going to let you take the wheel here. Tell our audience.

    Scobie: It is a big day today for us at Chef. We are announcing some changes to the way we both produce and ultimately distribute our software, which I think are really exciting. There’s a lot going on in the industry right now. As you probably know, Chef has been one of the marquee players in the open source software development community for some time. Part of the announcement that we’re making today that I think is most exciting, is we’re announcing that what we’re going to do is going forward, produce 100 percent of our product-oriented software in that open source model.

    For us, that’s a change because we, over time, and through the addition of additional projects and additional software at Chef, had started following a little bit of an open source, open core path, which means that we had some proprietary software and some open source software, but when we got down to it and we thought about the history of Chef and the roots of what has made Chef such great software in the past, it really came down to collaboration. We realized the best way to make software is to make it in collaboration with the people that actually use the software, at the end of the day.

    For us, that’s the big first talking point about the announcement is that we are committed to 100 percent open source software development going forward. We’re committing all of our software to open source repos and GitHub, and then ultimately attaching it and giving it to the commons with the Apache 2.0 license.

    Shimel: Sure, so Corey, you know – so you’re talking to someone who has a bit of a history in open source, 20 years. I remember when the whole open core movement first came out. Look, Richard Stallman and that crew, they were really against open core, because they felt it was – it was almost heresy to the whole cathedral and bizarre open source kind of crowd, where if it’s open, it’s open. You can’t be a little bit pregnant. But, nevertheless, we’ve seen companies succeed with this open core or premium, where the core is free and there are premium modules on top of it, but it’s a slippery slope. In all honesty, I think Chef got caught in a little bit with that slippery slope with what’s truly open, what’s free, what is our proprietary that we make money from? I don’t know if the market really got it.

    Scobie: Yeah, one of the things that is really tough when you’re following an open core strategy is drawing a very clean line between what is free and what is not free in the world of your technology offerings. For us, that got complicated and then even conflated further as we added more projects to the list. When we opened the InSpec project for security and compliance, when we opened Habitat for application automation, what ended up happening is we had multiple projects that were governed different ways that had different lines between the free and the core part of it. I think ultimately at the end of the day, had I been at Chef during that part of the journey, I probably would have made the same decisions, I’m guessing, but at some point you get back to this reconciliation point where it’s like, “Look. The best thing that you can do for customers if number one, you can collaborate with them, and number two, you can have a really, really clean line between what is free and what is not free in your world.”

    For us, it becomes super simple with this announcement. What is free is all of the intellectual property that we’ve contributed to the commons in the world, and will contribute to the commons going forward. What isn’t free is all of the work we do to make a turnkey, enterprise-class, package that enterprise-class customers can pick up and distribute at hyperscale in their organizations, from that point.

    Shimel: That brings us to the next thing, and of course this is always the issue. So if you’re going to make all the software and the IP open, well how the heck do you make money?

    Scobie: Sure. As it turns out, we’ve got venture capital funding, investors that want to see us turn that into dollars as well. For what it’s worth, we’re feeling pretty good about the shape of the business right now. We just came off the best quarter of revenue we’ve ever had at Chef, prior to this business model, this model change around how we distribute software whatsoever. So we’re feeling really good about the shape of the business, but just the inability to have that simple conversation about what’s free and what isn’t. What we decided was as we’ve matured as a company, and as the space has matured in enterprise automation, what we found is the majority of our customers actually do want to have a very material commercial relationship with us.

    Not only do they want the software, but they want the support. They want the content. They want the expertise. They want the warranties and indemnities that go with all of that. So for us, it was very simple. We realized at the end of the day, that the most — that there was a huge constituency of our customers who what they wanted from us was a very curated commercial offering out the other end. So we’ve decided that’s where we’re going to put our effort and our energy, and we’re going to trust in the open source world to fill in the gaps where that isn’t the offering that they want as well.

    Shimel: We look at this model, Corey. Obviously, Red Hat’s the poster child for much of the open source world in that most of their software, if not all of it, was in fact open source. It was all in the packaging and the support, and that kind of thing. It sounds very much like that’s going to be the model, or am I wrong?

    Scobie: It is – no – there’s a strong analog to what Red Hat has done in the industry as well. What’s interesting is I think that that is the antithesis of what we are seeing from a lot of the more recent moves in the open source industry where there’s a desire to become more proprietary in some cases, or in other cases to try and somehow bend, I think, the definition of what is open source software. For us, the core of it, we get back to the four freedoms, and freedom zero is about the freedom to use the software, but it’s free as in freedom. It’s not free as in dollars.

    Shimel: Free as in freedom, or free as in beer, as we used to say, but Corey, to be fair to these companies that are – that we do see going another way, it’s because they didn’t cut it. They tried the Red Hat model, and for whatever reason, they didn’t cut it and they had too much pressure from their boards and the VCs and investors, and so forth. So they tried to take a path that was quicker to money. I’m not here to judge anybody’s – as someone in the Godfather once said, “Far be it from me to disparage what someone does for a living” –

    Scobie: I hold the same opinion, Alan, by the way. I sit in judgment of nobody on the decisions that they make about what’s right for their business. I just know that for us, what’s right is to try and both honor the open source roots that we came from, and to also serve the commercial customers that were really knocking at the door and saying, “Look, this is what we want from you. We want even more engagement. We want even more value out of your commercial subscription.”

    Shimel: Agreed. Before we jump into the next part of it, which is how it manifests itself within the Chef universe, I wanted to just tie a knot on this one, and mention that when we talk about this model, and you said, “Oh, we’re going to count on the open source community to fill in the gaps,” and one might say, “Well, what are those gaps? I think people have to realize what made Red Hat great was not just distributing Linux. It was that RHEL, and Fedora even before that. It was that package, that distro, that had everything you needed and wanted in there in a really easy to use – you didn’t want for anything else. It was all right there, nicely done for you, instead of running out and having to download different packages and distros from all over the internet, which is what held back Linux on the desktop for so many years. There wasn’t that one package that you could do that.

    I think that’s what you’re referring to is Chef’s strategy going forward is to have that killer distro, with all the right packages in it. If you don’t want to pay Chef for that, and I don’t think at this point, though people like software that is free as in beer, and they get free as in freedom with it, no one denies that a company should be able to make a living, that a company should be able to make a profit, pay its employers, pay its developers, and go forward. So for people who either don’t have the money, or don’t have the inclination to buy the packaged product, they can get the open source components, and the community can work on giving you the glue, in essence, that puts these all together. Correct?

    Scobie: Absolutely, for sure. Santos came out of that exact scenarios. The scientific community needed something that wasn’t going to be necessarily nearly as curated as Red Hat and Linux was, but they needed a platform for scientific computing, so CentOS was born and has a place in that world. I think our vision for that is exactly the same. If there’s a downstream project that is born of Chef InSpec or Habitat, we welcome it into the community, and we look forward to whatever innovations they bring, and we want to collaborate with anybody that wants to come and take that work out of the commons and create the derivative work out of it, that allows them to meet whatever their goals are. That is the freedom in free and open source software.

    Shimel: Absolutely. I think one of the other things though, Corey, is when we look at Chef over the last couple years, we’ve seen the whole InSpec line of security functionality, and we’ve seen Habitat. Look, Habitat was an ambitious piece of – well, it was more than a piece of software. Habitat was an ambitious play, but because there was so many balls in the air and going off in different directions and open/closed, proprietary/not, people never – it wasn’t explained well enough to people. I spent the time at ChefConf so I understood Habitat well, but there’s maybe a thousand people at a Chef conference as opposed to millions of people in the world. I think it was really hard getting InSpec and Habitat, carving out their niche in the world, amidst this whole confusion around the closed/open, supported/paid for, free/not free.

    Scobie: Yeah, that’s fair, and I think this really helps us simplify the messaging around that too. I will openly admit I think InSpec is an easier basic value proposition to parse, which is “Look, I need security and compliance in my organization and the right way to do that is express it as code”, so that should be fairly straightforward, and relatively noncontroversial. I think, with respect, to Habitat, what is interesting about Habitat is I agree with you, super ambitious project right from the get-go to redefine the abstraction of packaging and orchestrating the delivery of software in the world.

    That’s – it’s huge that there’s a huge surface area associated with that. I think what’s interesting is that we’ve learned much – we’ve learned a lot over the last couple of years about how to help people in the mental journey of how to think about Habitat in the world, and how it relates to Chef, and having that foundation of understanding and knowing Chef, which tens of thousands, if not hundreds of thousands of people already have, the ability for them to jump to where Habitat plays becomes much easier in the world where we can talk about them collaterally and not independently of one another.

    Shimel: Fair enough. I think that’s a great segue, Corey, to the next part of our conversation I wanted to have with you, and that is okay, how does this manifest itself down the line of Chef product and services? So we’re going to put all our IP out as open. I guess we should mention also, I think everything is being released under an Apache 2.0 license.

    Scobie: That’s correct. Apache 2.0.

    Shimel: Industry standard, GPL, Apache are probably the two biggest. How does this manifest itself? What changes does this mean for Chef’s products?

    Scobie: On flag day when we release new products with new packaging and new terms and conditions, it’s an evolution that we’ve been going through for some time now, which is really to understand what the interface parts are between the three core engines that we have. So we’ve created an engine for infrastructure automation, which is the Chef project, and going forward, just from a purely – from a branding simplicity perspective, it’s going to be known as Chef Infra, which is I think is fairly straightforward.

    Shimel: I’m sorry. Can you repeat that though? Chef…?

    Scobie: Infra, as in infrastructure.

    Shimel: I-N-F-R-A. Okay.

    Scobie: Yeah, and then we’re going to have Chef InSpec, for security and compliance, and the state relationship between infrastructure and security and compliance are pretty – it’s pretty intuitive, which is that if I deploy an operating system, and I want it to be locked down in a certain way and secure and meet a certain set of compliance standards, then testing that on a continuous basis, managing the drift, and being able to automatically remediate any drift that’s happening there seems like a really obvious relationship. What’s interesting though is the relationship between infrastructure and application automation. This is where things get really murky, even for longtime Chef customers who have been wildly successful in using Chef as an infrastructure automation tool. If you go and talk to those customers, what you’ll find is that they’re doing about – 30 percent of what they’re doing is about infrastructure deployment provisioning and shaping, locking down configuration management.

    70 percent of what they’re doing is deploying software on top of the operating systems in service of some application that’s going to come along at some point in time and need that – those software dependencies to be there. What we discovered, and this is really the genesis of Habitat, which I’m not even sure – we knew that what was happening in the space of Chef was not working at scale for us. That’s the reason that some folks at Chef went off and decided to create Habitat, but what we didn’t realize is why it wasn’t working as it was at the time. The reason is that there is a natural boundary for building systems from the bottom up in a layer cake approach. It doesn’t matter whether it’s Chef or Antival or Puppet or Kubernetes clusters or whatever, that there is a natural boundary layer, if what you’re doing is building systems from the bottom up.

    So you have an operating system, and then you have system libraries, and then you have packages, and then you have middleware, and eventually you have an application. At some point that dependencies graph, the modularity of everything that you lay down, and the dependencies graph gets so complicated that there’s no possibly way you can manage understanding what the implication of changing component X is on all the other components that rely on X. So the beauty of Habitat and where Infrastructure, and where Chef and Habitat come together is that Habitat takes the exact approach. It says, “Look, what you should do is you should define an application, all of the application’s dependencies and all of the orchestration steps for that application to run successfully in any environment regardless of where it is.”

    So by doing that, and by removing that layers – that bottoms up layers dependency, what we’ve really done is created an application stack, an enterprise automation stack is what we’re calling it where infrastructure and application automation marry together at a very, very clean abstraction level. The abstraction level is essentially, if you’re running software on the operating system that runs above the kernel, it should be defined by application packages and orchestration, and not by infrastructure packages and orchestration. Now, it’s architecturally it all makes sense, and it actually works really well in practice. Where it’s complicated is that organizationally, INO – traditional INO operators and other infrastructure groups have been tasked with more and more software deployment on top of their stacks. So this is a very paradigm-shifting idea. I think what is helpful for us is the enterprise automation stack allows us to create a framework and a template for how to do that in a way that helps customers adopt that methodology much, much quicker.

    Shimel: Fair, fair. So we got Chef Infra. We have Chef InSpec.

    Scobie: Yes, and Chef Habitat, Chef Habitat.

    Shimel: Chef Habitat sits – I wouldn’t say it sits on top of them.

    Scobie: Look, the stack is – we have trouble with this with our marketecture diagrams. If, what happens in the podcast now is you’ll throw up the marketecture diagram you’ll have, what you’ll see is a linear stack, but the state relationship between those three things is much more complex than can be defined in a diagram like that.

    Shimel: _____ diagram, sure.

    Scobie: Then on top of that, then there’s two other pieces to the portfolio. So we have Chef Automate, which is the observability layer on the top. Of course, if you have automation on all of these things, you want to be able to see across all of your pipelines, across all of your applications, across all of your systems. What is the state of everything that’s running? How has it been deployed? Is it in compliance or not in compliance, etcetera? So we’ve got this whole enterprise dashboard with automation up top. Then on the bottom end of the stack is we need to make it easier for developers to get started. So all of these engines are representative of either infrastructure compliance or application automation as code. What do you do as a developer? You download a bunch of tools. You get started writing code that are the artifacts that actually drive that automation. So Chef Workstation is the compilation of all of those tools and an easy way to set up your development environment to get going. That’s the entirety of the enterprise automation stack. It’s Chef Workstation, Chef Infrastructure – Chef Infra, Chef InSpec, Chef Habitat, and Chef Automate on top.

    Shimel: And repeat for me, what all of these packaged together is the –

    Scobie: Yeah, the Chef Enterprise Automation Stack.

    Shimel: That’s the whole enchilada, so to speak. This is all packaged up into one coherent type of product, if we can call it that. Fair?

    Scobie: Exactly. Yeah, one coherent distribution, absolutely.

    Shimel: Corey, you know the news broke today around this. Some are saying, “Well, Chef is pivoting. Chef is this.” How do you view this? How do you and the other folks at Chef view this? Is this really a pivot? To me, it’s – I don’t see it as a pivot.

    Scobie: Yeah, it’s not a pivot. I mean, first of all, I don’t think it’s – we’ve – you can question the effectiveness or the efficacy of our open source strategy in the past as we’ve meandered through different phases of that, but this is us staying true to the fact that we are an open source company, and so I think we’re leaning into the business model by which Chef was founded, and by which we’ve been operating for quite a while.

    Shimel: I agree with you.

    Scobie: I think from a technology perspective, we just have more clarity now in how to deliver the components and technology that have become critical to customers who are trying to solve large-scale enterprise automation problems. I don’t know that I would really describe it as a pivot. I think what we’re trying to do is use this as an opportunity to reintroduce people to Chef who maybe haven’t heard of the latest of our innovation in InSpec and Habitat, as an example, who maybe are not familiar with the work that we’ve been doing in the last three or four years. For us, it’s a little bit of a rebirth, but I wouldn’t call it a pivot.

    Shimel: I don’t know if I’d call it a rebirth. So here, look, you’re in the forest.

    Scobie: I am in the forest. There’s no question.

    Shimel: [laughter] I’m outside the forest. To me, obviously a pretty – I’m more than a casual observer. To me, Chef was always an open source company.

    Scobie: Always.

    Shimel: I think in some ways, what we’ve seen with Chef over the last five years, I would characterize as growing pains for an open source company trying to come to grips with remaining true to its open source roots, while at the same time recognizing you took other people’s money, and that’s how – I lived that life for 20 years. You take other people’s money, they want – it’s a funny thing. They want to get a return on their investment. It’s like trying to serve two masters. It’s a hard – it’s a hard –

    Scobie: Not only that, but look, the successful commercialization of any open source project or software company is how you actually create sustainable good in that community for the long-term. Obviously, you need participation from the community, but you also have to inject capital into that community if you want sustained innovation.

    Shimel: You have to extract revenue as well though. That’s where I think it gets dicey. So when I look at InSpec and Habitat, I almost feel at some level, they were somewhat hobbled by having to live in this schizophrenic, open/not open, making money/not making money. Well, if it’s totally open, we’re not making money. How much do we put into this? There was just too much of this all over the place, where to me, this is just a clarification. Look, Chef is and will remain an open source company, and our value to our investors is that No. 1, we have this tremendous community around us who are users of our software as well as contributors to the community, but No. 2, we are now offering a proprietary, a for-profit solution that takes the best of this and packages it up nicely.

    Scobie: Absolutely. Look, Alan, you make a great point. I would love to give a shout out to the community, because the Chef community is absolutely amazing. I know you’ve been to ChefConf. It’s an experience in and of itself. It’s hard to think of another tech conference that is as authentic and as welcoming and as diverse as ChefConf is, and it’s a bunch of people that come together every year, because they are passionate about solving the challenges of infrastructure management at scale, and application deployment at scale, and all of those things that Chef is interested in as well. So the community is a really key factor of it, but to your point is that there’s just this growing demand in the enterprise. We are the experts in figuring out how to do that in a really, really tight curated way. Of course, we’re going to take advantage of that market opportunity as well.

    Shimel: Sure, we got the news out. Is there any part of the news we’re missing, Corey?

    Scobie: No, I don’t think so. That really covers the announcements for today. I think the only other thing is we’ve got ChefConf 2019 coming up in Seattle May 20th – 23rd in Seattle. It’s going to be a great event. We also have a ChefConf for the first time ever, ChefConf coming to London at the end of June. It’s June 19th and 20th, I believe are the dates in London.

    Shimel: Really?

    Scobie: Yeah.

    Shimel: You know I’ll be in London those dates. I’ll have to talk to you about it.

    Scobie: Yeah, absolutely. We’d love to see you there, and we’d love to see others there as well. Yeah, so we’ve got a couple big events coming up where we’re going to showcase the latest of the innovation that we’ve had on all of the fronts, innovation, compliance, application automation, our enterprise consoles, all of that kind of stuff. so those are going to be fantastic events coming up. So other than that, we’re looking – we’re really looking forward to getting back to those community roots and really everybody being there together in the trenches together building the next generation of what this is all going to be.

    Shimel: Cool. So now let me ask you a question. Put on your wizard’s hat. If we’re talking about ChefConf 2020 now, a year from now, this strategy, this road has been – this strategy has been implemented now. We’re going down this road. How does it manifest itself over the next 12 – 24 months you think?

    Scobie: Look, we’ve been talking about that from a product strategy perspective, and I think we’ve got – there’s a lot of opportunity for us to do great work in pulling the pieces of the puzzle together, even more explicitly. We’re doing a lot of work with partners, so I anticipate that the ecosystem partners will be a big player in ChefConf 2020 as well. So we’ve been working with the likes of everybody from Microsoft, Azure DevOps, and the workflows there. We’ve got – we’ve been doing some product work, and continue to do product work with Amazon in the ops work for Chef Automate, managed and hosted to platform there. We’re quite engaged with GitHub because we’re seeing both GitHub and Artifactory and a lot of our customers in the DevOps toolchain and pipeline there. We’re doing some work with that. We’ve been doing a lot of integration with services now. So I actually think that the next phase of this is not just about the Chef parts of the stack, but how those parts of the stack connect with the rest of the ecosystem that is fast emerging in the enterprise automation space as well.

    Shimel: Sounds cool. Well, Corey, we’re about out of time. I want to thank you for taking time on what’s a crazy busy day for you guys, to brief our audience on this. Congratulations to the whole Chef team. Tell Barry and everyone else, looking forward to ChefConf. Unfortunately, I won’t be at ChefConf in Seattle. I’m in Europe, but we’re going to have a full team there doing video interviews there and everything. John Willis, our partner at Digital Anarchist will be there, and Chris Reilly. So we’ll have plenty of Chef coverage from ChefConf. Like I said, I didn’t even realize the London one. We’ll try to get over there. I think that’s Tech Week in London actually, which is a pretty big week.

    Scobie: Yeah, fantastic. Well, thank you so much, Alan. I appreciate you taking the time today as well. Look forward to seeing your colleagues at ChefConf in Seattle, and then hopefully we’ll see you in London in June.

    Shimel: Absolutely. Hey, Corey Scobie, SVP Engineering and Product. I got it right.

    Scobie: You got it right.

    Shimel: You’re talking to us about some news coming out of Chef today, a little bit of a change in how they package stuff, but they’re still an open source company. Corey, thanks very much.

    Scobie: Thank you, Alan.

    Shimel: Take care. This is Alan Shimel for devopsy.kinsta.cloud, everyone. Have a great day.

    — Alan Shimel

  • Chef Commits to 100 Percent Open Source Code

    Chef Commits to 100 Percent Open Source Code

    Chef has decided to open source its entire portfolio of IT automation software as part of an effort to make it easier for organizations to construct a DevOps pipeline using the company’s software.

    A part of that effort, Chef also launched the Chef Enterprise Automation Stack—which combines Chef Infra for managing infrastructure, Chef InSpec for maintaining compliance, Chef Habitat for managing applications, Chef Automate for managing hybrid clouds and Chef Workstation, a starter kit for launching Chef—within a single distribution of Chef software. Chef Infra is the original Chef project around which the company was launched.

    Corey Scobie, senior vice president of product and engineering for Chef, said the company previously offered a mix of open source and proprietary software that made engaging with some customers overly complex from a licensing perspective. This approach aligns the entire Chef business model around providing support for open source software, he said.

    At the same time, Chef is now open to collaborating more closely with developers who want to contribute code to its various projects. That’s critical because code contributions are now the primary means through which many IT organizations help shape the direction of products as they continue to evolve, Scobie noted.

    Of course, many IT organizations today either mandate the use of open source software or have instituted an “open source first” policy that makes it challenging for any proprietary software to be adopted.

    Chef expects that by making its software easier to consume via a unified open source transition it will be able to further accelerate adoption of what is described as a “Coded Enterprise” to managing IT. Rather than manually configuring IT assets in isolation, that approach encourages organizations to adopt best DevOps practices that treat all aspects of IT, including infrastructure, as code, said Scobie, adding that capability is more important than ever as IT organizations are increasingly require to manage heterogenous cloud computing environments.

    Chef claims that approach has already resulted in more than half of the Fortune 500 having adopted at least one module of its software and, in the fourth quarter of 2018, the highest level of bookings in the company’s history.

    While Chef helped pioneer the IT automation category, competition across this sector has intensified. Chef and its primary rivals Puppet and Red Hat are the leading proponents of frameworks for automating IT operations. Although adoption of those frameworks has increased substantially in recent years, use within IT organizations often remains fragmented, often resulting in islands of automation. Chef is clearly hoping that IT organizations will increase adoption of its software stack now that it’s  based entirely on open source software. Many organizations face challenges teaching their IT operations teams enough programming skills to automate their processes. In absence of those skills, adoption of IT automation frameworks within the context of a set of well-defined DevOps best practices often remains uneven, at best.

    — Mike Vizard

  • Helping the Little Guy: The Power of Open Source Software for Startups

    Helping the Little Guy: The Power of Open Source Software for Startups

    It’s no secret that open source software holds many benefits for businesses looking to reap the rewards of these innovative, collaborative and incidentally free efforts of developers around the world. Even major companies such as Salesforce have opened up their software for developers to test, debug and improve. Other huge players that run on open source software include Netflix and Twitter. In fact, 78 percent of companies run on open source. So, what if I told you these benefits improve exponentially for startups using OSS to launch and scale their business?

    Open source software has leveled the playing field so that even small startups can leverage advanced technologies, without needing the big budget to go with it. Here are the top ways that startups seriously benefit from running on open source software solutions.

    Power and Control

    Startups are always are running under strict budgetary restrictions as they’re just starting out. This means executing their business model in the most economical way, and open source software undeniably allows this by relieving businesses of the need to pay the expensive initial fees of proprietary software. In fact, Linux found that its users save up to 55 percent over commercial solutions, and Barclays bank reduced its software development costs by 90 percent using Linux.

    While it’s likely that down the line they may have to spend money on support, training or bespoke development needs, these choices can be made as per their business plan and actual business traction, and deployed as and when needed.

    Power in the form of expanded capabilities that comes from open source software is also in large part due to the community of talented programmers who build and craft the solution, allowing for quicker troubleshooting and accelerated development. Many hands can deliver powerful results.

    Innovation, Productivity and Agility

    Startups need to be able to experiment freely to understand what works for their business and what doesn’t. If this need to try different options is shackled down by high costs or contractual commitment, they won’t be trying out the necessary paths or making the necessary mistakes to understand which is the best route to take—not to mention the benefit of being able to take advantage of new tools as soon as they become available, especially before other big players that have developed their own proprietary software.

    Open source software allows for this kind of agility, which breeds innovation. Without the requirement to follow the dictates of a particular vendor, startups can explore all the avenues that they need to, until they know the right way for them.

    For example, startups working within AI will need to identify the algorithm that works best for their particular business needs. To identify these algorithms from the library, data scientists have to perform many trial-and-error iterations. This is usually followed by a detailed data analysis and discovery phase. Only after going through these iterative cycles will the data scientist have a full understanding of which algorithm to use, how best to slice and dice the data and how best to get the expected business results.

    If it wasn’t for open source software, going through these cycles would be prohibitively expensive. This is how top companies were expected to create AI algorithms in the past—with huge expenses on manpower and resources. Now, these open source technologies are mature enough to apply predictably, making AI startups easier and cheaper to build.

    Open source software helps startups take huge strides in their productivity levels, too. A study conducted by Frank Nagle, assistant professor of strategy at the Marshall School of Business at the University of Southern California, measured the impact of free open source software on productivity, especially since a growing number of businesses are increasingly using open source software as a key input.

    The study found productivity either always increased or stayed neutral, as a result of open source software—the impact was never negative. For every 1 percent increase in the use of free open source software, profits can go up by $100,000. This is a “huge amount for small firms,” said Nagle.

    Security and Reliability

    Due to its public nature, the software’s source code is likely more reliable than anything built by independent groups of developers. And although open source software is not automatically more secure than proprietary programs, unlike closed solutions, open source software is transparent and available for inspection. So, by opening up the software and allowing anyone to fix broken code that might put it at a risk, companies can utilize the proactivity of the community to enhance the security of their programs.

    Startups that are just starting out may be intimidated by the amount of open source software options that are out there, but this shouldn’t put them off—there’s a huge amount to gain by opting for open source technologies. Between saving on costs and resources, increased capacity for innovation and agility, and having the security of your systems granted by the open source software community and their dedication to great code, the options are countless for forward-thinking startups.

    — DJ Das

  • Best of 2018: 11 Popular Open Source DevOps Tools Worth Knowing

    Best of 2018: 11 Popular Open Source DevOps Tools Worth Knowing

    As we close out 2018, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the fourth in our weeklong series of the Best of 2018.

     

    Companies implementing the best practices of DevOps have demonstrated that they are more effective and flexible in implementing as well as designing IT tools and practices, resulting in higher revenue generation at a lower cost. For traditional organizations looking to embrace new inventions such as bitcoin wallet, the adoption of DevOps tools provides consistency, quality and efficiency.

    Open source DevOps tools are used as a way to streamline the process of development and deployment. The benefit of using open source software is that it is built with enhanced collaboration which can drive innovation and enhance flexibility in handling transforming markets and needs. Visibility into the code helps enhance overall quality and security, and also helps companies prevent vendor lock-in from proprietary vendors.

    Open Source DevOps Tools to Know

    If you are looking to accelerate an already existing program or just getting started with DevOps, below are 11 open source DevOps tools worth considering.

    Behat

    Behat is a php framework for auto testing the expectations laid by your business. It is a behavior-driven, open source development framework for php. This tool supports you in providing software that matters via test automation, deliberate discovery, and continuous communication.

    Watir

    Watir is a web application cross-platform open source testing tool. It is the most flexible and reliable tool of Ruby libraries for automating the web browsers. Just like a human being, this tool communicates with the browser so it validates text, fills out forms and clicks links.

    Supergiant

    Built on top of Kubernetes, Supergiant is an open source platform for container management. It is utilized for Kubernetes deployment on multiple clouds in a matter of minutes. The Supergiant API is used for streamlining production deployment. With the help of the packing algorithm of Supergiant, hardware costs can be lowered and the hardware you require with computed efficiency can only be utilized.

    Ansible

    Ansible automates various common tasks related to IT operations such as application deployment, configuration management and cloud provisioning. It is owned by Red Hat. It integrates with numerous other famous tools for DevOps, including Jenkins, JIRA, Git and many others. The free open source version of Ansible is available on GitHub. Red Hat offers three paid versions—premium, standard and self-support—with prices that differ depending on the required level of support and number of nodes in production.

    Nagios

    Infrastructure monitoring is an area that has numerous solutions, from Zabbix to Nagios to various other open source tools. In spite of the fact that there are much newer tools in the market today, Nagios is a well-established monitoring solution that is highly efficient due to the large contributor community creating plugins for it. Nagios has the capability to deliver results in different visual reports and representations.

    SaltStack

    SaltStack is the paid enterprise version of Salt. Salt is a highly flexible, powerful and intelligent open source software for event-driven orchestration, cloud control, configuration automation and remote execution. It helps DevOps companies by orchestrating the effective movement of code into production and keeping the complex infrastructures tuned for optimal application delivery and business service. Saltstack orchestrates the value chain of DevOps and helps to deploy as well as configure dynamic applications.

    Chef

    Chef makes it possible to manage both traditional and cloud environments with one single tool. While maintaining high availability, Chef promises to accelerate cloud adoption. The chef development kit provides you with the tools you need to develop as well as test your infrastructure automation code from your workstation locally prior to deploying changes into production. On the Chef site, many technical resources and a lot of documentation is available including various resources designed to help organizations transition to DevOps and scale their implementations of DevOps.

    Docker

    You can expect portability with Docker, which is transforming IT environments. The portability is made possible via its special containerization technology, quite often found in self-contained units. It packages everything than an application is required to run: libraries, system tools, runtime, etc. Due to this, the applications can function the same way irrespective of their deployment location. A part of Docker known as Docker Engine is a tool responsible for creation and running of Docker containers. Another part of Docker known as the Docker Hub is a service application based on cloud encompassing the concept of application-sharing and workflow automation.

    Git

    Git has become incredibly popular in the recent years for managing the source code. It has become famous, especially as a site, for hosting open source projects. Because of the ease with which it handles merging and branching, it stands out from other version control management. Many DevOps teams utilize it for managing the source code of their applications. It has great pull request and forking features. It also consists of plugins that can link with Jenkins to facilitate deployment and integration.

    Hudson

    Hudson is a tool for managing and monitoring continuous testing and integration. The key features of Hudson include support for various systems for management of source code, application servers, code analysis tools, testing frameworks, build tools, real-time notifications of test failures, change set support and easy process for installation and configuration. A huge library of plugins is present that further extend its capabilities.

    Puppet

    No matter where it runs, Puppet promises a standard way of operating and delivering software. Puppet automates deployment to boost auditability, reliability and agility. The products of Puppet provide continuous automation and delivery across the complete software delivery life cycle. The latest version of Puppet features Node Manager and Puppet Apps, which help handle large number of dynamic, variable systems.

    Conclusion

    The world of DevOps is full of unique and outstanding open source tools. The above mentioned popular DevOps tools can help effectively bridge the gap between development and production environments when compared previously. You can opt for the tool that suits your business needs and can instantly observe the difference in your business operations. And not only do these different DevOps tools function well individually, they also play well together.

    If you’re looking to build out your toolset, consider open source DevOps tools in addition to proprietary software.

    — savaramravindra

  • Best of 2018: 5 Best Node.js Frameworks to Know

    Best of 2018: 5 Best Node.js Frameworks to Know

    As we close out 2018, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the third in our weeklong series of the Best of 2018.

     

    When it comes to development platforms, the choices are wide and varied. It can be a difficult task choosing the right framework for your app, even for developers. For non-developers, the task can seem insurmountable. That’s why it’s important for app dev companies to educate their clients on the best development platform to fit the needs of their app. For most, it’s a matter of explaining the benefits and drawbacks of the many options.

    The Node.js environment, for example, has a number of different frameworks, each suited for a particular purpose. While developers need to know the technical details of each framework, clients really only need to know the factors that will impact their app.

    A Quick Node.js Explainer

    In the professional circles, much is spoken about Node.js and best Node.js frameworks as of the way to facilitate web apps development.

    What the client should know about Node.js: this is a special runtime environment used by the developers to create web apps. Its front-end (client-side) and the back-end (server-side) both can be coded using the same language: JavaScript.  Previously, those were two different areas to be dealt with different programming languages and by different developers, correspondingly.

    What to Know About Node.js

    Node.js is especially good for developing real-time apps intended for simultaneous connection of multiple users. However, it is not recommended for apps intended for complex computations.

    Some of the big names you know—Netflix, Trello, PayPal, LinkedIn, Walmart, Uber, Groupon, eBay, NASA—all use Node.js for their apps.

    Node.js offers myriad benefits to companies, including:

    • JavaScript belongs to one of the most popular programming languages—according to GitHub statistics, as of Q3 2017 JavaScript is the top programming language, with 22.5 percent of pull requests. That means will be no difficulty in finding an experienced developer;
    • Node.js’ popularity implies numerous ready-made solutions and contributions that can be used by developers for their apps;
    • Node.js gives the developers much freedom and space for creating a specific app;
    • Since the front-end and back-end are coded in the same language, only one programmer is needed;
    • Because of these factors mentioned above, development and deployment time is shortened and the app can reach the market in a short time;
    • Less search time, fewer developers and faster development mean lower costs.

    Who is behind Node.js?

    Node.js appeared in 2009 and now enjoys the support of the active and fast developing community. As of today, there are 8 million instances of Node.js, with more than 1,500 contributors and 39,672 stars on GitHub. Mark R. Hinkle, executive director of the Node.js Foundation, said his organization plans to Grow, Engage and Educate. So, the future app will have good prospects in terms of maintainability and eventual extension.

    Node.js Frameworks Worth Considering

    Express.js (35,310 Stars on GitHub)

    The classic of Node.js. appeared in 2010 and served as a basis for a long list of popular Node.js frameworks. It is considered to be minimalistic and simple. Express is not opinionated and offers the developers much freedom, as they can use various modules to build the app. Due to its flexible nature, it is best-suited for large-scale apps that are planned to be customized and extended and need long-term support.

    What should the client know about Express.js? Fast, simple, flexible

    Sails.js (18,088 Stars on GitHub)

    This framework used Express as a basis but is a complete product that can be applied as it is with the available functionality. It is suitable for fast launches. Sails is especially good for developing and setting the real-time apps requiring quick answers and is compatible with any front end.

    What should the client know about Sails.js? Complete, quick to develop, good for real-time apps

    Meteor (38,755 Stars on GitHub)

    This is a full-package popular Node.js framework that allows building real-time scalable web apps. It handles apps for any device (web, iOS, Android). With Meteor, developers can accomplish more job with less code. It suits as a perfect tool to start a project for all major platforms.

    What should the client know about Meteor.js? Full-stack, time-saving, universal

    Loopback (8,000 Stars on GitHub)

    This is a highly extensible modular framework. Loopback is specifically intended for quick and easy creating of API. It allows running the apps on-premises or in the cloud. Loopback works well for writing user’s authentication and authorization.

    What should the client know about Loopback.js? Easy and quick to develop as offers much out of box, suits for complex integrations.

    Hapi (8,701 Stars on GitHub)

    The framework appeared in 2011 and also is based on Express. It has since grown into the separate framework with a different approach: to provide as much as possible use cases out of the box. It is intended for large teams and large projects and will be too complicated for a simple app.

    What should the client know about Hapi.js? Configuration over code, reliable, time-saving due to focus on writing reusable app logic, large projects

    Bonus Framework: Koa.js (18,495 Stars on GitHub)

    This is a new-generation framework. Koa is created as lighter and more flexible version of Express. It suits both for large and small projects and provides for further customization and extension.

    What should the client know about Koa.js? Fast development, reliable app foundation, flexible

    Conclusion

    The list above should serve you in helping your clients decide which framework is best-suited for their project. Be sure to explain each framework in the simplest terms to ensure your customers understand the benefits and drawbacks of each. They will appreciate it.

    What frameworks would you add to the list? Please add them to the comments below!

    — Olga Nasonova

  • Best of 2018: 5 Open Source SIEM Tools Worth Checking Out

    Best of 2018: 5 Open Source SIEM Tools Worth Checking Out

    As we close out 2018, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the second in our weeklong series of the Best of 2018.

     

    Security information and event management (SIEM) is the cornerstone of IT security. All other network solutions are merely data flows that feed into an organization’s SIEM. Not all SIEMs are created equal, and their capabilities can vary wildly. Choosing the right one for your needs can mean the difference between detecting a security weakness and becoming just another statistic.

    A SIEM solution is a combination of a security event management (SEM) system and a security information management (SIM) system. SEMs monitor servers and networks in real time, while SIMs store the data.

    Both SEMs and SIMs provide analysis. SEMs focus on real-time event correlation, alerting and those fancy network operations center (NOC) “something has gone wrong” dashboards that one may come across in the movies. SIMs focus on bulk data computational analysis (BDCA) of large quantities of logs.

    In other words, a SEM is designed to tell you when something is going down as it happens, and SIMs are designed to spot the subtle attacks that SEMs don’t catch. Merging the two into a SIEM solution seems like a natural fit.

    Related Content:

    The Value of Commercial Open Source

    How Developers Can Take a More Proactive Approach to Security

    SIEM stacks usually consist of at least three components for data collection, data storage and data analysis. The data in question is frequently log files, but can also be netflow traffic or other real-time data feeds.

    There are a number of SIEM tools on the market, both open source and commercial. With the rise of DevOps, containers and other modern application development methods, the open source solutions are seeing a resurgence of interest. Let’s take a look at some of the top open source SIEM tools out there.

    OSSEC

    OSSEC is a popular host-based intrusion detection system (IDS) that works with Linux, Windows, MacOS and Solaris, as well as OpenBSD and FreeBSD. OSSEC is composed of two components: the host agent (responsible for collecting the logs) and the main OSSEC application (responsible for processing the logs.)

    There is also a now-deprecated GUI, but since other open source solutions do a better job of data visualization, the OSSEC project recommends using those solutions instead. Popular open-source data visualization tools include Kibana and Grafana.

    OSSEC directly monitors a number of parameters on a host. This includes log files, file integrity, rootkit detection and Windows registry monitoring. OSSEC also can perform log analysis from other network services, including most of the popular open source FTP, mail, DNS, database, web, firewall and network-based IDS solutions. OSSEC can also analyze logs from a number of commercial network services and security solutions.

    OSSEC has a number of alerting options and can be used as part of automated intrusion detection or active response solutions. OSSEC has a primitive log storage engine. By default, log messages from host agents are not retained. Once analyzed, OSSEC deletes these logs unless the <logall> option is included in the OSSEC manager’s ossec.conf file. If this option is enabled, OSSEC stores the incoming logs from agents in a text file that is rotated daily.

    Whether OSSEC counts as a “full” or “proper” SIEM is the source of numerous internet debates. OSSEC does the hard work of a SIEM: It collects data and analyzes it. OSSEC is at the core of a number of other SIEM solutions, and part of any number of application stacks that pair OSSEC with a more advanced long-term log retention system and advanced visualization capabilities.

    Snort

    Snort is a popular network-based IDS. Network-based IDS systems differ from host-based IDS systems in exactly the way one might expect—while a host-based IDS lives on an individual server or endpoint, network-based IDS systems live further out on the network, and scan all traffic that they can see.

    Snort’s purpose is to sniff, log and perform real-time analysis on network flows. It can display a real-time stream of packets to a console, dump packets to a log file or perform analysis.

    Snort relies on output plugins to determine how and where it will store the information it generates. Snort can store this information in a flat text file, output to a database, a Unix Socket, a pager or cell phone and any number of other destinations.

    Snort has a number of filtering and pattern matching options. This means that snort doesn’t have to display, log or analyze all packets traversing a network link. It can restrict itself to information pertaining to a specific host or matching specific patterns—something that is of increasing utility as network throughput of 100Gbps on individual network links becomes more common.

    Configuring Snort requires careful planning. Dumping 100Gbps worth of packets to a flat text file isn’t the best plan, and most organizations aren’t exactly going to be able to afford to ship that kind of data off to a public cloud-based log repository, either. Judicious use of Snort’s filtering and analysis capabilities combined with output plugins are required to make Snort practicably useful.

    With a narrowly configured set of alerts, Snort’s ability to send alerts directly to a cell phone becomes meaningful. A broader set of filters could be used to pass data to a long-term storage repository for more in depth analysis by another tool.

    Like OSSEC, Snort relies on third-party solutions in order to perform visualizations. As one of the earliest network-based intrusion detection solutions available, there are a number of solutions (both open source and commercial) that exist only to visualize Snort data, making the lack of a built-in visualization system somewhat irrelevant.

    Like OSSEC, Snort’s qualification as a SIEM solution is somewhat debatable. Snort collects data and analyses it, and is a core component to more complete SIEM solutions. Snort is also part of any number of application stacks which add log retention and advanced visualization capabilities.

    It is not unusual to see Snort and OSSEC working together to each fill different niches in the grand scheme of data center SIEM needs. It is also perfectly normal to see them deployed independently, especially as part of cloud-native applications, where monitoring needs can be narrower.

    Suricata is a competing open source network-based IDS that is frequently used in place of Snort.

    ELK

    The ELK stack is arguably the most popular open source SIEM tool available, though, like OSSEC and Snort, there is room for debate about whether the ELK stack even qualifies as a SIEM on its own.

    The ELK stack consists of the open source products Elasticsearch, Logstash and Kibana. Logstash is a receiver for log data from virtually any source. It can filter, process, correlate and generally enhance any log data that it encounters.

    Elasticsearch is the storage engine, and is one of the best solutions in its field currently available. Kibana is the visualization portion of the equation, and it is hands down one of the best visualization system the open source community has yet produced.

    Logstash, a part of the ELK stack, uses input plugins to collect logs; however, it also can accept input from more purpose-built solutions such as OSSEC or Snort. Combined, the ELK Stack’s log processing, storage and visualization capabilities are functionally unmatched. The solution is powerful, but that power comes at the price of complexity and a high TCO. Cloud offerings such such as Logz.io offer a hosted ELK alternative.

    The ELK stack forms the core of a growing number of commercial SIEM offerings. These solutions bring expertise in standing up, configuring and managing the applications in the stack. They also may offer canned or easier-to-design filters for Logstash than are available in the native open source solution. Both approaches offer significant added value.

    Prelude

    Prelude is a SIEM framework that unifies various other open source tools. It is the open source version of the commercial tool by the same name. Prelude aims to fill the roles that tools such as OSSEC and Snort leave out.

    Prelude accepts logs and events from multiple sources and stores them all in a single location using the Intrusion Detection Message Exchange Format (IDMEF). It provides filtering, correlation, alerting, analysis and visualization capabilities. The open source version of Prelude is significantly limited when compared to the commercial offering in all of these capabilities.

    OSSIM

    If ELK isn’t the most popular SIEM, then OSSIM likely wins the crown. The open source version of Alien Vault’s Unified Security Management offering, OSSIM is a framework like Prelude. OSSIM combines its native log storage and correlation capabilities with numerous open source projects to build a complete SIEM.

    The open source projects included in OSSIM include FProbe, Munin, Nagios, NFSen/NFDump, OpenVAS, OSSEC, PRADS, Snort, Suricata and TCPTrack. The inclusion of OpenVAS is of particular interest, as OpenVAS is used both for vulnerability assessment by correlating IDS logs with vulnerability scanner results.

    Like Prelude, the open source OSSIM is not as feature-rich as its commercial sibling. Both solutions work fine for small deployments, but experience significant performance issues at scale, ultimately driving organizations towards the commercial offerings. Log management capabilities in the open source version of OSSIM, for example, are virtually nonexistent.

    Conclusion

    A complete SIEM solution includes the ability to collect information from various data sources, store the data long term, perform analysis on that data and visualize it in useful ways. There are no clear open source winners that deliver all capabilities.

    When selecting an open source SIEM you are either going to have to use feature and performance limited solution designed to drive organizations into the arms of a commercial solution, or you are going to have to combine multiple open source projects to achieve your goals.

    Consider the ELK stack as an example. On its own, the ELK Stack can be a high-quality SIEM solution. Its input filters include public cloud infrastructure monitoring capabilities the other open source solutions can’t match.

    That said, the more dedicated collection and analysis tools, such as OSSEC and Snort, can be far more powerful when dealing with on-premises solutions. These solutions can be combined. OSSEC and Snort can feed the ELK Stack, creating a more complete solution that meets the needs of the majority of organizations.

    Open source SIEM solutions—in whole or in part—are versatile and powerful. They require experience, expertise, and above all time to deploy properly. It is for this reason that commercial offerings dominate the SIEM landscape, even when open source tools lie at the core of those commercial offerings.

    Having 80 percent of your SIEM solution handled for you is better than having to do all of it yourself. Commercial solutions handle installation, basic configuration and provide filters, correlation configurations and visualisation designs for the most common use cases. Don’t underestimate the value of these commercial features: There are a seemingly unlimited number of things to monitor in today’s data centers, and none of us have time to manually configure applications to watch them all.

    — Daniel Berman

  • Crowdsourcing, Data Science Top Talks at the 2018 Topcoder Open

    Crowdsourcing, Data Science Top Talks at the 2018 Topcoder Open

    The 2018 Topcoder Open (TCO), the world’s ultimate programming and design tournament, took place in Dallas Nov. 13-16 and featured the best technologists from around the globe competing for cash prizes and international rank. Companies of all sizes shared perspectives on digital transformation and caught a glimpse of the most coveted programmers on the planet. A handful even provided current software development challenges their in-house IT teams face for Topcoder Community members to solve.

    I caught up with Dave Messinger, chief architect of Topcoder, for his take on why 2018 brought one of Topcoder’s most challenging tournaments.

    Jessie D’Amato Ford: What’s your history with Topcoder?

    Dave Messinger: I’ve been with Topcoder since December 2001. It’s a fast-paced, innovative environment that thrives on creativity and speed, which mirrors the spirit of our unequaled Topcoder Community members. Today I run Topcoder’s technology platform, which is the core of our crowdsourced software development work with enterprise customers.

    Ford: What were a few highlights from the 2018 TCO?

    Messinger: This was Topcoder’s 17th annual event and it was especially competitive. We had 100+ finalists from 34 countries, with first-time competitors coming from Morocco, Hungary and Nicaragua. Topcoder ran 1,600+ challenges and awarded more than $2.5 million in prize money. The companies attending were impressed by not only the caliber of technologists competing and how much they accomplished in such a short period of time, but also at the efficiency, accuracy and speed of crowdsourced software development.

    Ford: Tell us about one finalist you kept an eye on while at the TCO?

    Messinger: Thomas Kranitsas (Topcoder handle: thomaskranitsas), from Greece, graduated from school and served in the Greek Air Force. Now in his third year as a Topcoder Member, thmaskranitsas has been a Code Review and Copilot on crowdsourced software development projects for companies. He sees Topcoder as an opportunity to work with the newest technologies, solve real-world business problems, earn money and meet the “best developers and designers in the world.” He finished 5th in the TCO’s First2Finish competition, with only 4.55 points separating him from the winner.

    Ford: How has crowdsourcing changing the way software is developed?

    Messinger: When Topcoder first opened its doors, the skepticism surrounding crowdsourced IT was overwhelming. Like many technology innovations, crowdsourcing had an uphill battle. It was a completely new way of developing software and managing teams—it, too, changed for the better over time. Fast forward to 2018 and now crowdsourcing is mainstream. The internet and cloud computing helped democratize development by allowing anyone in the world to access to the same resources and become familiar with multiple platforms. Other technology advancements and techniques, like microservices and DevOps, have also allowed more developers, designers and data scientists to collaborate on software projects with little ramp up time, which reduces costs and increases efficiency (both important to global organizations).

    Ford: What industries have been quick to catch on and why?

    Messinger: Topcoder has customers in almost every vertical imaginable because the methodology for developing high-quality software with a crowdsourced model is applicable to anyone. Crowdsourcing works best when developers can easily access SDKs, APIs and development environments. The 1.4 million (and growing) Topcoder Member Community has been working on projects that drive significant industry change.

    Ford: How does the TCO recognize and reward Topcoder Members?

    Messinger: There are multiple rounds to champion before Topcoder Members earn a finalist spot at the TCO in the U.S. (and we’ll fly the finalists in from wherever they are in the world). The collective intelligence in one place is astounding! It’s also a great opportunity for technologists and companies to pick up new software development techniques, as well as discuss the latest trends. Topcoder Members are passionate about helping businesses solve IT challenges.

    Ford: What skills are companies looking for in today’s developers?

    Messinger: Companies are not only looking for the best developers, but also for those who can clearly document code and collaborate well with the rest of the team. Topcoder’s viral nature and crowdsourcing methodologies hone in on those skills no matter where someone is working.

    Ford: What’s ahead for The Topcoder Open in 2019?

    Messinger: 2018 marked Topcoder’s first use of automated testing as part of our TCO development track and we’ll be expanding it in the future. TCO 2019 will bring Topcoder’s first test-a-thon competition: our QA community will participate in a live QA challenge to identify bugs in live tournament style. It will be exciting, for us and the Community, as well as companies watching these master technologists at work.

    — Jessie Ford

  • How Secure is Open Source for DevOps? 5 Considerations

    How Secure is Open Source for DevOps? 5 Considerations

    Open source libraries and frameworks have important roles to play in a DevOps culture that emphasizes shorter development life cycles, collaboration and innovation. However, it’s vital not to neglect the security of these open source components.

    This article discusses the use of open source in DevOps and gives you five things to consider in terms of the security of any open source project your teams might want to use in your software builds.

    Open Source and DevOps Overview

    Proprietary applications, many of which are developed within organizations that have adopted a DevOps approach, feature an average of 257 open source components per application. Furthermore, the average percentage of codebases that are open source within proprietary apps stands at 57 percent. It’s clear that open source fits well with DevOps in how it can help deliver quality software that can be rapidly updated and improved.

    In addition to the increasingly popular use of open source libraries and frameworks to deliver ready-made software components, quite a number of tools used in DevOps are actually open source. Examples of open source DevOps tools include Docker and Kubernetes.

    It’s crucial that development teams focus on security when using open source code, components and tools. Several of the most high-profile IT security and software breaches of recent times have resulted from vulnerabilities in open source components.

    To adequately combine security, open source and DevOps, there is a pressing need to shift left and secure the software development life cycle (SDLC), starting from the earliest stages of development.

    DevOps Open Source Security Considerations

    The Need For Security Automation

    An important driving force in achieving the aims of DevOps is to automate as much as possible. DevOps automation strategies include the use of technologies such as virtual machines and containerization to repackage applications into reusable blocks, many of which contain open source code. This high level of automation dramatically shortens the SDLC.

    However, because of this automation and the pace at which code changes and updates occur, security teams are easily left behind, and they might miss out on identifying open source vulnerabilities. InfoSec teams need to figure out ways of automating several of the most important security procedures, such as configuration checks, code analysis and vulnerability scanning. By introducing greater automation into security checks, it’s less likely that DevOps practices will lead to releasing software containing vulnerable open source components. Faster time to market combined with high-quality, secure code is the ultimate goal.

    Securing Open Source Using Open Source Tools

    Given that open source code forms the majority of the footprint of modern proprietary codebases, it makes sense to focus on the security of these libraries and frameworks first. Somewhat ironically, open source tools provide a good way to improve open source security in DevOps.

    For example, teams can use open source software management programs such as JFrog Artifactory to create binary local repositories of open source software libraries and code. These repositories give teams a single place from which they can use the latest versions of open source code that they have deemed to be secure.

    OWASP dependency-check is another useful open source tool for security teams that checks which components an application uses and whether such components contain known vulnerabilities.

    Incorporate Open Source Code-Checking Tools Into Development

    Part of the idea of shifting security to the left involves developers overcoming the inherent tendency to focus solely on application functionality without considering security. While training materials exist that teach the basics of secure apps, developers might find the material somewhat esoteric, particularly if it’s targeted toward InfoSec teams.

    One way to overcome developer resistance and shift security to the left is to integrate some open source code-checking tools into development environments. Tools such as Brakeman or Rubocop are freely available, and they can rapidly check code for common vulnerabilities.

    Hackers Targeting Open Source

    One of the difficulties with the increased use of open source is that hackers and cybercriminals are always aware of components that contain vulnerabilities. The hackers can then use this knowledge to specifically target companies developing software using those components.

    Cybercriminals know that DevOps emphasizes faster time to market, and they are always lurking to find those organizations that have become lax in their IT security checks. The Equifax data breach happened precisely because the company used a vulnerable version of the open source Apache Struts framework. The Equifax case exemplifies the potential danger of open source components in organizations where DevOps and InfoSec are out of sync.

    The Need for Policy and Governance 

    For DevOps and open source to work together to deliver secure software, there is a real need to create dedicated open source policies by DevOps organizations. When developers are given free rein to use open source libraries and frameworks without monitoring or documentation, vulnerabilities creep into applications and future trouble awaits.

    The first step is for organizations to recognize their reliance on open source before forming a governance policy that stipulates how to manage open source inventories to reduce security risks.

    Wrap Up

    Going forward, it’s clear that the open source model offers many advantages to DevOps teams in terms of achieving their goals.  Additionally, several open source tools can even enhance the security of the applications developed in DevOps. However, it’s prudent to understand the security concerns that lie at the heart of open source and shift security left so that any vulnerabilities can be found as early as possible in the SDLC.

    — Limor Wainstein

  • Infosys Launches Open Source DevOps Project

    Infosys Launches Open Source DevOps Project

    Infosys has decided to make its DevOps platform—which consists of more than 2,000 prebuilt scripts and more than 150 pipelines spanning 70-plus tools involving more than 25 different classes of technologies—available as an open source project available via GitHub.

    Ravi Kumar, president and deputy COO for Infosys, said the global systems integrator has spent the last several years turning each element of its DevOps platform into a set of microservices based on containers that can now be deployed almost anywhere. That decision made it more practical for Infosys to then offer the up the entire DevOps framework it relies on to drive thousands of projects employing more than 200,000 developers as a single open source project, he said. That framework relies on an instance of the open source Jenkins continuous integration/continuous deployment (CI/CD) framework at its core.

    Each element of the Infosys DevOps platform can be consumed via a set of templates hat Infosys has developed. The goal is to reduce the amount of time any organization needs to spend developing that framework and processes required to accelerate any digital business transformation project, Kumar noted.

    To accelerate the development process, Infosys enables organizations to automate the promotion of any module without having to create any additional scripts. In addition, Infosys makes extensive use of encryption to make sure DevOps workflows are secure.

    Most recently, during the DevOps Industry Awards 2018 event, Infosys won awards for Best Overall DevOps Project in the finance, retail and communications, in addition to best automation project and most successful cultural transformation initiative.

    Kumar noted that DevOps is now integral to any digital business transformation initiative. Each organization today is required to significantly increase the rate at which applications are developed and updated to create a modern digital experience for customers, employees and business partners, he said.

    The single biggest hindrance when it comes to making the transition to DevOps within most organizations is a lack of tools and processes. Most organizations quickly become aware there is a chicken-and-egg relationship between tools and DevOps processes. Without the right set of DevOps tools, it becomes difficult to set up a DevOps process. But the tools in of themselves have no value without a set of well-defined processes in place. To enable organizations to address that issue in a holistic manner, Infosys developed the IT Process Enabler repository alongside a DevOps maturity framework.

    Most of the adoption of DevOps within most organizations has been driven by the bottom up by individual development teams resulting in adoption of DevOps that has been uneven at best. But as those individual DevOps initiatives become increasingly successful, there is now a lot of interest at the senior most levels of organizations in finding a way to drive DevOps processes across the entire organization. That requires a level of change process management that tends to proves too elusive for many organizations to achieve on their own.

    — Mike Vizard

  • Implementing ChatOps Using Open Source Tooling

    Implementing ChatOps Using Open Source Tooling

    ChatOps is an exciting proposition for organizational Ops communication. Throwing development and operational tooling into the chat conversation can ease the development pipeline, increase transparency and increase security by streamlining bug detection and remediation efforts.

    At the heart of ChatOps are bots, scripts and chat clients; many are pre-built and open source. In this article, we’ll review a completely open source method for building out your ChatOps setup. We’ll cover integrating with Hubot, using CoffeeScript commands, and how building custom scripts can ignite ChatOps in your own environment.

    Why Open Source for ChatOps?

    Many arguments have been given as to the benefits of open source. Building upon open source inherently evades vendor lock-in, enabling more security and service longevity. Furthermore, to avoid reinventing the wheel, forking existing open source ChatOps projects naturally will be more efficient than purely greenfield initiatives.

    Open Source Chatbots

    There are many Chatbots available for use; each with unique capabilities, varying types of integrations out of the -box, and different ways to extend functionality with custom scripts. Take Lita, for example, an open source chatbot built with Ruby. It works with IRC, HipChat and Campfire. Many other open source chatbot options exist, such as Errbot, Jarvis and Lazlo.

    Example: Hubot

    By far the most adopted open source ChatOps companion is Hubot. Designed and open sourced by Github, Hubot takes market share in the ChatOps discussion. Described as “a customizable, life betterment robot,” Hubot is written in Node.js and takes commands with CoffeeScript.

    Using Hubot, developers can initiate commands with Hubot scripts to perform actions such as post images, translate languages and integrate with maps, but also initiate vital commands to your favorite DevOps cloud tools. A full catalog of scripts can be found here, and it’s quite easy to craft your own CoffeeScript commands as well.

    Example Hubot Scripts

    The community scripts that support Hubot are similarly open source, meaning that you can repackage these implementations to suit your own needs. Let’s take a look through the catalog of Github-supported Hubot scripts to see what sort of pre-constructed extensions we can seamlessly add to a Hubot environment.

    With Hubot, you can ping your major cloud development platforms, such as Heroku. Some example commands for returning status of issues on Heroku include:


    hubot heroku status - Returns the current Heroku status for app operations and tools

hubot heroku status issues &lt;limit&gt; - Returns a list of recent &lt;limit&gt; issues (default limit is 5)

hubot heroku status issue &lt;id&gt; - Returns a single issue by ID number

    Or perhaps you want to conveniently check PagerDuty from your chat client to see who is on call within your incident response team, or to gather details on current incidents. Hubout has a script for that. Using pagerduty.coffee, some example requests might be:


    hubot who's on call - return the username of who's on call

hubot pager me trigger &lt;msg&gt; - create a new incident with &lt;msg&gt;

hubot pager me incidents - return the current incidents

hubot pager me notes &lt;incident&gt; - show notes for incident #&lt;incident&gt;

hubot pager me problems - return all open incidents

    Another example could be direct interaction with the Twilio API for sending SMS. Using sms.coffee you can initiate a command with the following syntax:


    hubot sms &lt;to&gt; &lt;message&gt; - Sends &lt;message&gt; to the number &lt;to&gt;

    Configuring sms.coffee is easy, all you need to insert are:


    HUBOT_SMS_SID

HUBOT_SMS_TOKEN

HUBOT_SMS_FROM

    Of course, there is no shortage of Gifs, cat memes, and other scripts to add some humor to your workday, such as mustache insertion using image detection.

    Generating Your Own Hubot Scripts

    Aside from using third party scripts, you can always craft your own. To be an executable Hubot script, your code must be in .coffee or .js, and it must be in the correct directory (src/scripts and scripts by default). It must also export a function, as in the following:


    module.exports = (robot) -&gt;

# your code here

    Essentially, Hubot listens to all chat communication; it will hear commands and respond accordingly. To configure what it must hear in regards to your script, you must specify what to listen for, and what to respond with. For example:


    module.exports = (robot) -&gt;

robot.hear /badger/i, (res) -&gt;

# your code here

robot.respond /open the pod bay doors/i, (res) -&gt;

# your code here

    For further details, see the documentation on Scripting here.

    Getting The Most of Chatbots

    Once integrated into a chat platform such as Slack, your chatbot configuration can truly thrive. Given the wide breadth of community generated and Github-maintained scripts, DevOps teams have near unlimited potential for uniting their tooling into a single, human-readable CLI. Another benefit of choosing open source tools for such projects is that you can detach your bot and integrate with whatever chat client you prefer.

    ChatOps can add fun and efficiency to the remote workspace. We’ve covered high-level methods for adopting this into your team’s space, and hopefully, a helpful foundation into the power of open source for streamlining the power of ChatOps.

    — Bill Doerrfeld