devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • To Be Continuous Podcast: Have We Forgotten How To Code?

    To Be Continuous Podcast: Have We Forgotten How To Code?

    Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly.

    In this episode, Paul and Edith discuss the fallout after the widely covered “left-pad” incident, and while they agree that these sorts of moments move the entire open source community forward, they wonder: Have we forgotten how to code?

    [soundcloud url=”https://api.soundcloud.com/tracks/264605904?secret_token=s-Kv9Ep” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Edith: Hey, Paul. So, somebody unpublished a module called left-pad and the internet basically exploded. And now it’s been a couple weeks since then, so there’s a lot of different topics to cover in terms of what this means for open source, whether it means that people have forgotten how to program.

    Paul: So LaunchDarkly. Or, I’m sorry, the left-pad. I’m obviously looking at my own t-shirt.

    Edith: You’re looking at your LaunchDarkly t-shirt?

    Paul: The left-pad incident was pretty interesting and one of the things that we did when coming up to this was just, like, looking at all the Hacker News discussions and comments on it.

    And there’s wildly varying positions on what this means for open source, what it means for whether people can program, what it means for, like, about the Node ecosystem. So to kick this off, why don’t you give us a recap of the entire incident?

    Edith: Yeah. So, in short, a programmer had a module called Kik, K-I-K. Which is also [the name of] a very popular messaging app. I mean, even I’ve heard of it and I’m not a teenager.

    Paul: The Kik lawyers contacted the dude. Said, “Please take down your module, change it.” And the programmer said, “No. I basically don’t believe in copyright laws.” So then, the Kik people contacted NPM. The NPM people took down the module called Kik. And the guy, in I’d say, kind of a fit of pique, said, “Okay, you took down my one module, I’m gonna take down all my modules.” Basically, I’m taking my marbles and going home.

    So, he took down all of his modules. One of which turned out to be an incredibly popular module called left-pad, which did basically what the name implied. It sorted left pads. And without this in NPM, the quote I read was, “Builds all over the world started to fail very quickly.”

    Paul: So as I understand it, left-pad was used by another module, which was used by another module, which was used by something core, or like, this incredibly popular module—

    Edith: Yeah, it was basically turtles on top of turtles on top of turtles, and at the bottom turtle was this guy who—

    Paul: Took his marbles and went home.

    Edith: Yeah, or yank the bottom Jenga, basically.

    Paul: Gotcha. And so all the builds started to fail. People’s deploy started to fail. And people got pissed.

    Edith: People got pissed because they didn’t even realize that they had this dependency. So what happened was NPM basically republished it. They said that, even though you’re the original publisher—

    Paul: So there was a lot of outcry on the internet and then they republished it.

    Edith: Just basically said, “You don’t have the right to un-publish this.”

    Paul: Gotcha. So, um, but they had an Unpublish button, as I understand it. So he did something that was built into NPM to remove a module.

    Edith: Yeah, and since then they have tightened their rules a lot. They basically said that this should be something that’s used extremely rarely, if at all, and not something you can just do capriciously.

    Paul: So one thing that’s interesting about the NPM ecosystem, and the Node ecosystem in general, is they’ve had a bunch of things like this. Things that are security-related, or that are drama-related.

    And people have repeatedly come out and say, “Oh, no, JS is a tar pit, or ghetto,” or whatever it is that people use to describe poor ecosystems, that normal programmers don’t know how to code, and so on, so on.

    And what it looks to me like is, you know, each of these incidences brings the Node ecosystem quite a bit forward. To the point that, I mean Node is relatively new, it hasn’t had 15 years before prime time in the way that Ruby or Python did, or 20 years before it got big, and it’s a real, kind of bottom-up thing.

    I mean, if someone built it and there were some steroids and then someone else took over, and then NPM took over, and then. It’s this sort of weird language from the ecosystem perspective. It didn’t have the same advantages that, like, Java had, or that C or C++ had, in order to get to maturity. And so each of these incidences propels it closer to maturity.

    Edith: Yeah, I compare it somewhat to when electricity first started coming out. And then you were just working through all the issues that could happen. Like, for example, when a power line went down or a transformer blew.

    Like, you work through, Okay, what are our points of failure? Let’s not make that a point of failure again.

    Paul: Hmm. So, NPM changed their policies on this basis, right?

    Edith: Yep, so now you can’t have an individual contributor be able to unpublish, even if they originally published. Now you have to contact them, and it’s much more stringent.

    Paul: Gotcha. The response, despite people being … Well, people were annoyed at their response.

    Edith: It seems perfectly rational to me.

    Paul: Oh, I think it seems perfectly rational. This is the thing about the in trend. There’s always some set of people who are annoyed at everything you do.

    Edith: Well you know, the internet is full of billions of people. I mean, there’s gotta be some sort of standard deviation, or otherwise it would be a very boring internet.

    Paul: Right, right, and this is why there’s always haters on Hacker News. No matter what you do. If you do it one way, there’ll be one group of haters, if you do it another way, there’ll be a different group of haters.

    And so, with left-pad, there were people who were annoyed that NPM retroactively decided that they had ownership over their namespace, which previously, they had sort of gifted to people in the community, the first people to take the namespace.

    Edith: You mean the original thing about Kik?

    Paul: So there’s that, but there’s also the left-pad. They took away the sort of autonomy of the left-pad offer.

    Edith: I read a good article that basically says part of open source is, if it truly is open source, do you maintain the rights to it? Or is it a community project?

    Paul: Hmm. The traditional way of deciding that sort of ownership was the fork. That you could go, and you could fork, and when you forked, you weren’t necessarily entitled to the same name, you weren’t entitled to the same trademarks. When Debian forked Firefox, they called it Iceweasel.

    When Jenkins was forked, or Hudson was forked, it became Jenkins. GCC got forked. They all got new names. But what was the fork here, it wasn’t even a fork. It was a republish of the same thing, the same software, in a namespace that the original author didn’t control.

    So typically you’d own the URL, you know, if you have a software library, you’d own the URL. And they did own, in some sense, the URL within NPM, or the name within NPM’s space, and NPM, the sort of owners of that namespace, overrode it.

    Edith: I think they did what was necessary for good order.

    Paul: Sure. I completely agree, I think that was a good decision.

    Edith: I mean, so to take this to the logical conclusion, what if you had a module named left-pad, and you decided that, like, as a prank you were going rewrite it to be right-pad? Like, sure, that’s … Is that within your rights? Kind of, but it’s a real jerk thing to do that you probably want to prevent.

    Paul: This is kind of one of the dangers of the whole ecosystem around centralized package managers, of sorta trusted package managers, where there’s no actual trust involved. So, what I mean by that is what is published does not have any gatekeepers.

    Edith: Yeah, exactly.

    Paul: I mean, it’s great, it allows an ecosystem to grow much faster, but it has safety concerns because people generally don’t pin their packages to an exact version.

    Edith: You’re basically pinning it to a floating value.

    Paul: You’re pinning it to a floating value. And that floating value could change. There’s no guarantee of what a minor number means or what a major number means or what a patch number means or what a patch even means. There’s some sort of, like, general community guidelines in semantic versioning, but there’s no enforcement of that.

    Edith: Yeah, and I mean this basically goes back to the whole Cathedral versus Bazaar thing.

    Paul: Right, right. It’s very Bazaar. And intentionally so. It’s the thing that allows a community to grow, and is there a point at which open source communities need to, like, batten down the hatches to prevent that sort of behavior? Or prevent that sort of risk?

    Edith: Yeah, and it’s really interesting. I think this goes back to something we’ve talked about with Nadia Eghbal and Sean Byrnes a couple episodes ago about when you buy from a corporation, you have some sort of guarantee of standards that they publish somewhere. You know, this is our guarantees versus with open source, you don’t normally have those same sort of guarantees.

    Paul: Even when you do, it tends to be—

    Edith: There’s no enforcement mechanism.

    Paul: Right. So there’s no enforcement mechanism in the Bazaar.

    Edith: Well, there’s no, um … Like, so with a company, if a company has said, “Okay, we’re gonna release twice a year and it’s gonna have this level of quality,” and they break that, you can walk with your feet or your money. If an open source project stops or does something that you don’t like, there’s very little enforcement end.

    Paul: Theoretically, you could walk. You know, you could fork the thing, but when you end up in an ecosystem that has massive network effects, and in the way that NPM does, you can’t walk with it. Because the pends is all transitive. The left-pad was, you know, a turtle way at the bottom of the stack.

    Edith: It was almost like a turtle hatchling.

    Paul: So, to fork left-pad, you would have to fork the entire NPM ecosystem.

    Edith: Yeah, and it’s not even. The fork is not the issue, though. The issue was that people were depending on it being in a certain place in a certain way. It wasn’t that it wasn’t forked, it was just like this entire structure at the bottom was gone.

    Paul: I guess what I’m saying is that if you get pissed off with a certain set of developers or a certain set of practices within the NPM community, because they’re so interlinked, you don’t have the traditional open source option of forking.

    Edith: Absolutely. Absolutely.

    Paul: You could rename it. You could pick a different library, but because of the way—I’m not sure if they still do this, ’cause I read something about it, but in NPM, it used to be that the version of the software that your library took was the version that library got. Do you know what I mean by that, okay. So let’s say there’s some software, let’s call it left-pad.

    Edith: Let’s call it right-pad.

    Paul: Right-pad. So, right-pad version one is being used by library A. And right-pad version two is being used by library B. In the Java ecosystem, and many, many other ecosystems, only one version of right-pad would be used.

    And I believe in NPM, both versions of right-pad would be used. So library A will use version one, and library B will use version two. Now that’s definitely the way that it used to be, I’m not sure if they’ve actually changed that.

    Edith: Yeah, I don’t know, but that sounds very messy, very quickly.

    Paul: So both options are terrible. And there isn’t a right way to do it.

    So, really, what it depends on. So, the NPM way of doing it, every library get its own version of the dependencies, is fantastic. Because it lets that library do what it actually intended to do under the covers.

    If, however, you start having an object that gets sent around, or that two modules are, let’s say it receives a, I think, an HTTP packet or something like that would be a good example. Let’s say you take an HTTP packet from one module and just hand it to another module, and they have different understandings of the shape of that object, or what functions you can run on that object, then that’s messy.

    But, on the other hand, if you override the version of right-pad that some other library’s using, you end up with all sorts of problems. This is a major problem in the, sort of the Maven ecosystem. And at CircleCI, we used Clojure, and Clojure is built on top of the Maven ecosystem, and we would have bugs where the library that we used.

    One library that we used had a dependency, and that was in the package file for another thing. And so, that’s the version of the packets that we used. So we would have to pick. And as a result of that, we have to pick the library that works. And, often, there isn’t a library that works for both, or there isn’t a, sort of subdependency that worked for both. So those are close to forked no matter which way you do it.

    Edith: Yeah, I mean, we ran into the same issues with our STKs. I mean, ’cause we have dependencies on other libraries that sometimes conflict.

    Paul: Right, disaster.

    Edith: On the other hand, I mean, the bigger point, I think, was. So there was an interesting backlash, I think, that everybody kinda projected what they wanted on the NPM thing.

    People said, Oh, people don’t know how to program anymore. You know, why do you even use what they called left-pad? You know, why don’t you just grind your own wheat and make your own bread from scratch and eat it at lunch also?

    Paul: I have no sympathy at all for this view of the world.

    Edith: Uh, that you should make everything from scratch? Or—

    Paul: That, “Oh, we’ve forgotten how to program because we need left-pad.”

    Edith: Yeah. The fact that we have modules available that make it easier and easier and easier to program is a good thing.

    Paul: I think it’s kinda different to that. I think it’s more of a curmudgeon view of the world. It’s a get-off-my-lawn—

    Edith: Wait, that you’re the curmudgeon?

    Paul: No, the people who complain, “Have we forgotten to program anymore?” I think it’s a no true Scotsman kind of argument. Real programmers know how to write their own left-pad, real programmers write in statically typed languages, real programmers write in C, real programmers, and I’m going back in time here, real programmers write in Assembly, real programmers hand-code their. Tweak the needles on the.

    Edith: They do their own garbage collection.

    Paul: Oh, and it’s garbage collection, what is this? Garbage collection is for amateur coders, right? I mean, you could just as easily have written, have we forgotten how to manually allocate memory?

    Edith: Or, you might as well say, like, nobody knows how to make their own transistors anymore. I mean, like, the entire computing revolution has been that you no longer have to assemble a computer by hand.

    Paul: Right. And each layer of the ecosystem allows a vastly larger number of people to take part in it. So I think the thing that we’re seeing now. This has been happening for five or 10 years, but what’s getting really, really big now is people who only know how to front-type program.

    People who only know Javascript and HTML/CSS, and they go to boot camps and they come out, they maybe have some, like, vague real understanding.

    And they call themselves coders, and we, the real programmers, who know how to write a quick sort, and who know how to write left-pad, we don’t welcome them to our community. And frankly, I mean, I have felt this in the past, I have since changed my mind, but it’s a very, very common perspective.

    This podcast is brought to you by Heavybit, a program that helps startups turn products loved by developers into successful businesses. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

  • It’s Time to Slay the Universal Installer Unicorn

    It’s Time to Slay the Universal Installer Unicorn

    While many people want a universal “easy button installer,” they also want it to work on their unique snowflake of infrastructures, tools, networks and operating systems. I call them snowflakes because their small differences expand combinatorially. That divergence turns the idea of a prescriptive reference architecture into a destructive myth that limits community and ecosystem growth.

    Because there is so much needful variation and change, it is better to give up on open source projects trying to own an installer and instead focus on making their required components more resilient and portable.

    Universal Installer Fallacy

    So what does an easy button install look like? Let me tell you about one that I built.

    Crowbar Mascot

    It was 2011, the dawn of the OpenStack Era, and our team at Dell was already battle-weary from trying to make repeatable installs around Azure, Eucalyptus, Joyent and Hadoop. The simple reality was there was no success pattern we could follow deploying any one of those platforms—and certainly none cross-platform. Even though all used basically the same compute and network infrastructure, each had custom deployment code and configuration approaches. We jumped on becoming an OpenStack founding partner as a way to try to break the failure pattern.

    What was failing? Each platform had deeply embedded environmental assumptions that were not repeatable or portable.

    For example, one of those platforms required thirty (30!) full racks of gear and tethering back to the master deployer because its management and topology was hardwired into its infrastructure. Others were not much better: Installation often was “ssh apt-get install platform,” then “scp pre-built configuration files.” Even worse, those steps were always proceeded with 30 pages of prescriptive manual system prep steps that were updated daily.

    Incorrectly, we believed that the solution was to create a better installer for OpenStack.

    The Crowbar project, which is still used by SUSE, created a fully integrated install sequence from PXE boot and discovery to testing the final OpenStack installation. We could build a cloud on a complete rack of gear from boxes in minutes. The open source project even used open tools such as Chef and hardware-agnostic options to increase its community appeal. As the first installer (it predated even DevStack), the project was widely used in early OpenStack evaluations.

    Despite that success, as a v1 technology, Crowbar’s major shortcoming was that it tightly coupled components and services together. That made the installation and management process very rigid and non-conforming to disparate operational models and environments.

    Our install scars showed us the real battle lines that prevented portability. Even minor infrastructure variations resulted in broken or site-specific configurations. A major part of the challenge is that variation is not neatly contained; instead, it is spread throughout the environment. We needed to create automation that could both cope with heterogeneity across the community and be change-tolerant after installation to enable upgrades.

    We came to accept that there is no single installer. Instead, scalable installers must assume that they will operate with other parts of the system in a coordinated way, because the diversity of options makes it impossible for them to control everything. In fact, we’ve seen that smaller, composable install steps are much easier to automate consistently. Even more importantly, smaller steps make it possible to manage systemwide upgrades in a controllable way.

    Asking for an “easy button” installer is the wrong question. Instead, let’s ask, “Can we create sharable components that can be automated in multiple ways?”

    There’s no universal installation script to create cloud rainbows; however, there are strategies we can use to make individual projects installable over a wide range of options. These approaches include composability, consistent configuration and use of standard tools and patterns for operation. These items help projects focus on their own concerns and allow operators to apply best practice techniques.

    Basically, the first step is for project developers to limit their scope and delegate operational concerns to operators.

    Even in a platform-as-a-service (PaaS) project that attempts to hide operational complexity from developers, such as Kubernetes, there remains someone who must operate that platform using foundational infrastructure and tooling. We make that job more complex when we get distracted trying to apply the new concepts to the old infrastructure. The addressing of an impedance mismatch pulls effort away from the central abstraction.

    Since we do need consistent installs from the community, projects do have a responsibility to help those efforts.

    How can they help? By ensuring that their component services are well-bounded with clear input configurations, operational requirements, upgrade patterns and output expectations.

    But really, that’s a future post. Please let me know what you think and where we should go from here.

  • Backup and Recovery of Next Gen Apps in MongoDB

    Backup and Recovery of Next Gen Apps in MongoDB

    This blog is the first in a two-part series about MongoDB backup and recovery. In this first part, I will discuss the motivations for protecting data that resides on MongoDB and the existing backup and recovery solutions. In the second part, I will discuss how to solve this problem.

    In the era of big data, enterprise applications create a large volume of data that may be structured, semi-structured or unstructured in nature. In addition, application development cycles are much shorter and application availability is a critical requirement. Given these application requirements, enterprises are forced to look beyond traditional relational databases to onboard the next-generation, cloud-native Platform 3 applications—infrastructure-as-a-service (IaaS) or platform-as-a-service (PaaS) applications. NoSQL databases such as MongoDB are now being adopted and evaluated by enterprises for these next-generation applications, which include ecommerce and content management. The benefits of MongoDB are many, including dynamic schema, easy scaling through auto-sharding, tunable consistency for reads and built-in replication.

    MongoDB database also has native replication capabilities that satisfy availability requirements. However, data protection requirements for scalable point-in-time backup and recovery must be addressed. For robust data protection, enterprises need both backup and replication; Without point-in-time backups, organizations are at substantial risk of losing data due to human error, logical corruption and other operational failures. Traditional backup solutions were built to address the requirements of structured Platform 2 applications on relational databases that used shared storage and had the ACID (atomicity, consistency, isolation and durability) transaction model. Unfortunately, they fall short of addressing point-in-time backup requirements of Platform 3 applications and distributed databases (local storage, eventual consistency and the elastic nature of infrastructure). There are a few alternate script-based solutions such as Strata that enterprises are using to fill the data protection gap, but these solutions are sub-optimal at best.

    Manual Scripted Solutions

    Manual scripted solutions (MSS) leverage native MongoDB snapshot utility and scripts to transfer data to secondary storage. The scripts (via mongodump) are customized for each MongoDB cluster and require significant operational effort to scale or adapt to any topology changes (such as addition or removal of nodes to your MongoDB database). Further, these scripts are not resilient to failure scenarios such as failure of a node (primary or secondary) or intermittent network issues. Finally, recovery (the paramount value of “backup”) is a manual process, hence, time-consuming (resulting in very high application downtime) and contains data loss risk due to any bugs in the scripts. Overall, these solutions work when the MongoDB environment is small and some data loss may be permitted in the application. Some of the key issues that these solutions face are:

    • Lack of enterprise backup solution for sharded configurations
    • Database needs to be offline when the snapshots are taken
    • Both backup and recovery fail under node failure and other infrastructure failures
    • Recovery process is manual and requires verifications, which increases the recovery time
    • Recovery at collection-level requires manual recovery that is time consuming
    • Recovery to unlike topologies (sharded → unsharded) for test/dev refresh is not available

    Most enterprises use these scripted methods as a temporary quick-fix solution. It is like driving your car with flat tires: You can keep going; however, you can’t go at the speed you want to go, nor are you risk-free from disasters.

    MongoDB Paid Backup and Recovery

    MongoDB (the company) provides a couple of ways to back up MongoDB databases. Enterprises may choose from either a managed backup offering (MMS) that runs in public cloud or, if they are paid MongoDB customers, they may deploy the backup service on-premise. In addition to being exorbitantly costly, the managed backup service stores customers’ data in public cloud. Backup data transfer over WAN may not work for customers who deploy MongoDB on-premise and for the customers who need to keep their sensitive data in-house. Further, there are significant data limitations per shard to use this service.

    Using the MongoDB on-premise backup service is possible but is overly complex to deploy and operationalize (this deployment diagram speaks for itself!). Enterprises need to deploy eight servers, additional databases (with additional licensing) and about 6 to 9 times storage capacity (of the database that is backed up) for enabling on-premise backups.

    Overall, on-premise backup service is a theoretical solution that brings with it significant CAPEX and OPEX investments:

    1. Complexity of deploying multiple databases
    2. Cost of additional infrastructure (servers and storage)
    3. Cost of licensing additional MongoDB nodes
    4. Risk of failed backups when nodes fail (secondary from which backup is taken)
    5. Siloed backup infrastructure for only MongoDB database

    Look for Part two of this discussion to come shortly.

  • Data Protection for Operational Reporting Platforms

    Data Protection for Operational Reporting Platforms

    Making real-time business decisions based on data from social, mobile and cloud platforms is a requirement in today’s world. That’s why an ecosystem of next-generation operational reporting platforms such as Spark, Apache Cassandra, Kafka, Docker, Mesos, Marathon and more have come to life. Enterprises are using these technologies to re-architect their data storage and processing frameworks to achieve lower costs, scalability and faster response times. However, as enterprises onboard business-critical applications on this next-generation technology stack, they need to prepare for failures or errors that may disrupt customer-facing applications.

    One of the key benefits that operational reporting platforms provide is real-time visibility, which allows businesses to make strategic decisions based on actionable insights. However, the new applications generate a large amount of data and archaic operational reporting platforms (those based on traditional data warehouses, data marts and ETL logic) are unable to catch up and, as a result:

    • Business decisions are based on stale data
    • Fragmented data increases the risk of errors
    • Multiple copies of data increase storage costs

    These issues have driven many progressive, data-centric enterprises to re-architect their operational reporting platforms. In this new architecture, Kafka serves as a persistent message bus that ingests data from relational stores and provides failure resiliency and message buffering. Spark or Spark Streaming may be used for transformation, aggregation and other lightweight stream processing. These processing nodes may be deployed in Docker containers. Finally, Apache Cassandra serves as a distributed data storage layer that provides linear scalability and high-availability. Kafka also enables the ability to add multiple data consumers such as Hadoop/HBase through Flafka.

    Datos IO - operational reporting platforms

    However, as enterprises adopt operational reporting platforms, they want to ensure that bad data feeds and human error do not result in data loss. Apache Cassandra natively replicates data but that does not provide the capability to go back in time to recover data efficiently and at scale.

    As enterprises onboard their critical applications to achieve development agility, scalability and lower operating costs, they must also explore data protection solutions that are specifically dedicated for scale-out databases.

  • To Be Continuous Podcast: Open Source Economics

    To Be Continuous Podcast: Open Source Economics

    Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI and Edith Harbaugh, CEO of LaunchDarkly.

    In this episode of To Be Continuous, Edith and Paul are joined by Sean Byrnes, CEO of Outlier.ai, and Nadia Eghbal. The group discusses open-source economics and examines several potential evolutions of adding money to the open-source equation.

    [soundcloud url=”https://api.soundcloud.com/tracks/257873002?secret_token=s-fHPo1″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

     

    Edith: So, Nadia what do you like best about open source?

    Nadia: I like that open source is where I feel I spend time with a lot of creative people. It’s sort of like creative talent at its best. And it feeds directly back into the rest of startups and technology and everything else that I love.

    But it seems like everything starts with open source. Which makes it really magical.

    Edith: So Sean, what do you like best about open source?

    Sean: It’s the community. I enjoy learning from other people. I enjoy being able to find people I would never connect with that I have interests in common with. And a community that helps, because I consider open source to extend beyond just writing lines of code to sharing ideas, and sharing concepts and sharing solutions to problems, and while I might not always use those same solutions, I’m always inspired for new ways to solve other problems.

    Paul: So that’s super interesting, because nobody said that they like how it supports their Silicon Valley startup through sharecropping. Or … nobody said, I like how I get paid to work.

    Edith: Well, Paul, that comes later in the episode, but this would be a great time for our guests to introduce themselves.

    Nadia: I’m Nadia Eghbal. I’ve been writing recently and thinking a lot about how to support open source infrastructure.

    Originally, I have background in nonprofits and impact investing and working with foundations but most recently I was a founder of a startup and then I went into venture capital. And have sort of seen the system from all different sides and would love to marry together all those different interests and topics in which seems to happen in open source.

    Sean: Hey, I’m Sean Byrnes. I’ve been writing software most of my life. I’ve also been a founder of companies and an investor.  I would not consider myself an expert on open source and I don’t wanna claim that pedestal but I have, am, the maintainer of some open source libraries which seem to be pretty popular and I have contributed to others and I think that I’ve been lucky enough to know a lot of open source developers in my life which is, by no means, a majority of them but enough that I feel part of the community, at least.

    Paul: So, one of the topics that we were discussing just before the show started, is this idea of how open source developers get paid. Or, if open source developers should get paid.

    Nadia, sounds like you’ve been talking about this for a while or thinking about this for a while. Can you kick us off?

    Nadia: Yeah, actually, have a new thought on this topic from Michael Rogers whose, who runs Node.js Foundation. And he said recently, I think, on the last post I had written that everybody gets paid for open source. And I think that’s a really great way to start thinking about it vs. saying that nobody gets paid. Because in the end we’re contributing time to open source, right and in the end, somebody is paying you to have the time to work on open source.

    Whether it’s your employer and then you can work on open source full-time at your job. Or you’re full-time employed and that gives you the bandwidth to be able to work on open source on the weekends or whatever.

    So when I think about paying people to work on open source, I think people sort of like, bristle at the idea because they think that money is not involved in open source at all. And what I really liked about Michael’s point is that, in the end, money is actually really closely tied to open source and we’re just not talking about it.

    Sean: I get it. It’s interesting because my point of view is that it’s actually more nuanced, I agree with you 100 percent but I think that it’s very nuanced and that a lot of people do get paid for open source, it’s not even that we aspire to wanting them to get paid, right. ‘Cause there’s lots of kinds of open source. It’s funny ’cause on Medium, Nadia and I had an exchange.

    I kind of quantified it into the four categories of open source the way I see them. Which is by no means the universal standard but at least, how I see the world there, there are places where open source is part of your, a corporate product. It’s part of a whole.

    Like, LaunchDarkly. Edith’s company, has open source SDKs. It’s not because they’re doing anything grand, it reduced their sales friction if their customers can see the code before they integrate it into their app, right. So that’s one kind of open source.

    A second is what I call corporate open source, where there’s an open source project but there’s a company that sells services and products and stuff around it, right. So Confluence does that for Kafka. And Databricks does that for Spark. And MySQL was, I think, one of the first companies to really make a good model out of it. Red Hat, you could argue, had mixed success but that’s another kind where there’s a company whose corporate goal and sales is towards this project.

    There’s the third kind which I call, kinda like corporate marketing, which is you take a project out of a company and you roll it out like LinkedIn did with Kafka, where they’re actually, the company’s maintaining it they just make it available. So Google, the Angular and Facebook work react. And again there, I think, that it’s. You could argue why companies do this, some of ’em do it for recruiting, some of ’em do it for marketing. Some of ’em do it for because they’re very nice people.

    I think that the fourth category is where it’s a little bit weirder, where it’s like, what I call the community where somebody started a project. Maybe it’s an individual, maybe a few people, to solve a problem. And a lot of the most important open source products today started that way. Even Linux started that way, right. And it grew, and it got bigger, and eventually, then it became unclear because other people were using that project to build other companies and build other products, and make more money.

    And I think in that fourth category is where I think it’s less, because the first three categories is usually pretty clear who’s getting paid to do what and where the money’s coming from, but in the fourth category it gets very muddied and very quickly. Because if I start something, and the metaphor I like to use is, the open source library that I wrote was something called Likely.js which is a recommend ender system for no JS where you can, like Amazon product recommendations. And I know that there are people in the world that are using that as part of companies. And those companies are obviously making money because it to recommend products, right. And so, the question is, you know, so in that world, do they owe me anything for creating that which is underlying their products? And then, in turn, what do I owe the projects that I build Likely on top of? And then where does the flow of economics in that world go?

    Paul: So I have an interesting view on that. Years ago, I got involved in this project called PHC, which is a compiler for PHP. And there was a couple of guys who had start this and I joined a year later. And when they started it, they made it GPL. When I joined, I argued, we should change from being GPL to BSD because if someone comes along and uses that technology to build something that’s cool, I would rather that happen than not now happen and it might be that if was GPL that it wouldn’t happen. And so that, essentially, in sort of the kinda related to Likely.js you decided by the license.

    Whether or not you were going to make it publicly available or if you didn’t want people to build business on it, you could’ve made it GPL or AGPL or something along those lines and you could’ve restricted it in any way that you wanted and you chose not to. And there’s something implicit in that, that communicates to people who are building their companies on it, etc., as to what your intent was.

    Sean: No, that’s true. Although there’s a nuance in there which is that, even the GPL suffers from this, what Richard Solomon still complains about today which is, you could still use Likely.js on a server.

    Paul: Right, right, right. Well, so AGPL would work around that.

    Sean: Sure, and there’s other licenses that I could choose, but the majority of licenses out there today but even then, let me argue counter side, not because I agree with it just because, hey, let’s argue because—why not! You know, as an individual developer, I don’t have the time resources to go pursue people. I’m not gonna go sue somebody. Really? No, come on, like it’s not gonna happen. So, I think, you’re right only in the act of releasing it as open source on GitHub. I was making a decision that it’s possible for people to use this in ways that may, either A, I don’t intend, they could’ve used on Silk Road to recommend drugs for all I know, seriously, right.

    Or, to make money on it in ways that I am not gonna benefit from. And I made that decision knowing full well the impact of that. And I would actually argue most developers do that. The question is whether or not there’s remorse later on when it turns out, wait, I really wish I hadn’t made that decision, I can go back in time.

    Paul: Well, I think this is a common thing that people create a thing and then they say, well, if this is popular, I could sort of keep it or I could, there’s value here. I could keep it to myself or I could keep the value to myself. Or try to profit from it in some way. But they make, what is essentially, a marketing decision. I’m just gonna release this, I’m gonna see what happens to it. I’m gonna see if it spreads and then if it spreads, I can then make a decision as to, as to what, how possibly I can make money out of this. Or I can get value out of this. But at that point, obviously, the cat is out of the bag in terms of people actually using the software.

    Nadia: Yeah, I mean, I think it’s actually should start from the principle, like you said, of it’s really great when people, you give something away and then people can use for whatever they want. Whether it’s for a hobby project. Whether it’s for illegal drugs or whatever, you know. Businesses, whatever, that’s like, kinda one of the coolest things about open source is that you can use it for whatever you want and that’s kinda what you signed up for right.

    If you didn’t wanna do that, then don’t license it that way. I don’t know, I wouldn’t frame, at least the problems that I’m seeing and that I’m interested in, is not really about seller’s remorse, I guess of, “Oh, I made this thing and now it’s really popular and how can I make money off of it.”

    But I think it’s more really about investing back into the infrastructure that we’re all using and that requires some sort of maintenance and upkeep. And that’s why I like the term infrastructure and I’m trying to use it in that sense of things that don’t have a business model or business attached to it. Like you, literally, could not charge for Python. It’s just not gonna happen. Or the packaging associated with it or whatever, but still like millions of people are using it. It’s touched billions of people. And you literally have like one person working on a library or a project, that doesn’t seem, that seems like it’s in everyone’s best interest to make sure that it works properly.

    Edith: Well I think Nadia made a really good point that a lot of the fun of creation is the creation. And the hard work is all the maintenance and the support and the documentation after. So like, Sean, you basically got the fun.

    Sean: Oh, totally. Well I mean, if you even think back to the original “The Cathedral and the Bazaar” essay from years ago, that was one of the fundamental topics of, you know Eric was inheriting, I think it was called Pop Mail from somebody else.

    He had gotten tired of maintaining it and so he wanted somebody to take it over and he found Eric and Eric took it over. I mean, that’s, you know, I think, yes, as engineers we like creating and we like creating things in our free time and we don’t like maintaining them as much and maintenance feels more like work.

    And so, I agree with Nadia’s point that if everybody is driving economic value from it somebody has to be willing to step forward to support it. I think the problem is that I also, it’s funny that you have a background in nonprofits because I also have worked for nonprofits and I see the similar lament in nonprofit companies.

    Which is that—not nonprofit companies, nonprofit organizations, where the person who starts it usually really wants to solve this problem, they’re usually really good at that. They’re not good at the fundraising

    And in this case, the nonprofits, it’s like raising grant money or getting donations. But they have a good mission and they have a good organization to go do it. And they have this lament which is why can they not get more support to solve the problem they see in the way that they wanna solve it.

    And I see the same around maintainers of open source projects and I think the challenge is going to be that I wonder if it is even possible for the general economy to support them, to do what they wanna do, the way they wanna do it. Or if they are gonna have to start doing it the way the general—

    Paul: Right, because they have other options. They could start a consulting company. Get hired by a category two company or something like that. To build around it and support themselves that way and then their job doesn’t, then their job isn’t what they wanted their job to be which is making cool software. Their job is, you know, doing maintenance contracts for a big soul-less enterprise.

    Edith: You say that like it’s, I mean, soul-less enterprises, I mean …

    Paul: I’m a big fan of soul-less enterprises. But I’m just looking to sort of put the—

    Sean: They have a lot of money. We have to sell them.

    Paul: And if any soul-less enterprise would like to buy CircleCI they should contact me for enterprise pricing.

    Sean: Just mention the code OSS for 10 percent more.

    Edith: And by the way, thank you for wearing your LaunchDarkly t-shirt today, Paul.

    Paul: Always. So, Nadia and Sean had this conversation on Twitter a week ago which led to them being on the show. And one of the Tweets that got involved was from a Jacobian who’s one of the senior Python kind of people. And what he said is, OSS developers deserve to be paid. SV, Silicon Valley, has made massive profits by exploiting free labor. This needs to end.

    And I think that there is this, this feeling in open source by, at least by a lot of people who end up on the maintenance side of things that, there is a sort of a sharecropping going on here. And that’s something needs to happen about that.

    Edith: I think Nadia actually put this extremely well in the essay; like cool, I wish I had written that essay, it was brilliant.

    Nadia: Aww.

    Edith: But just like, what’s really fed this is the explosion of GitHub and it’s really easy now to start an open source project.

    Nadia: Yeah, I feel like right now is just a time when we’re so used to thinking of open source in a certain way and things have kind of gone along as they have for so long that we haven’t necessarily noticed or taken a step back that the past five years look completely different than the past 30 years. And stuff is just not the same anymore. And in part, I think it should be framed as a really good thing because it means that open source kind of won, in a way.

    Companies want things to be open source. They wanna learn how to open source their own projects. They wanna understand the whole thing. Everybody is defaulting to open source and that’s a really positive thing and the question is more of, not like, oh, we’re exploiting people and that really sucks but how can we help support that underlying layer that has helped make everything so great. It obviously needs some sort of reinvestment back into it.

    Edith: Yeah, I mean, one of the most fascinating thing about “The Cathedral and the Bazaar,” which was written 30 years ago, is that they referred to Linus Torvalds as this charming guy. Yeah, I know, and you’re like wow, here’s this charming, personable guy who like persuaded people through sheer force of will to work on his project.

    Sean: Yeah.

    Edith: I mean, to go back to Sean’s example about a nonprofit and everybody in the studio right now is like, whoa, that’s not …

    Nadia: Not at all.

    Edith: By anybody’s perception now, and I don’t know whether that’s just the grind down of having to keep maintaining a project or …

    Sean: First, I’d like to thank, I don’t know if his name is actually Jacobian, whatever it is, thank him for his service and work on Python ’cause I think, no matter what we agree on here, we can all agree that the people doing this are doing a service for the general community.

    Nadia: Definitely.

    For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • FLAP, Part 3: Full Data Center Provisioning

    FLAP, Part 3: Full Data Center Provisioning

    In the first two parts of this series (Part 1, Part 2), we covered server provisioning and application provisioning. Knowing a bit about each of these tools is useful (although one blog is hardly a thorough evaluation of strengths and weaknesses—I strongly suggest you do more research), knowing how they work together is the point of full layer automated provisioning (FLAP). Automating one or the other helps, but automating both—and integrating that automation—brings us to the level of “Oh, engineering needs a new test environment, and it has to be internal …” and, with a push of a button, it is delivered. That should be the long-term end goal for your organization, so staff time is free to attend to some of the many other issues that plague any given IT organization.

    The thing is, these tools combine in both better and worse ways. Some are designed to work together as if one product; some take some work to get going but the combination adds value once connected; and some combinations are painful to put together and almost never are worth the effort.

    One thing you’ll note is a repeated discussion of OS support for combined solutions. The reason we address it here, even though we have limited words and it’s already been mentioned in product write-ups, is simple: The limitations of these combinations are compounded. That is, if the server provisioning tool doesn’t support Windows, then the complete solution does not, no matter what the application provisioning tool does. This applies both ways, and is an important consideration for most shops, so we mention it here.

    Tools Built to be Together

    First off, we’ll take the easy ones. These are the combinations that were designed to work together, and do so pretty well. They vary in maturity level, but otherwise, they’re good options if they fit into your environment.

    Razor and Puppet

    This is the easy one. Razor was built to be the server provisioning part of Puppet. While the solution set is relatively new, and it shows in the installation phase (which has several manual configuration steps both before and after Razor install), it is maturing rapidly, and the two dev teams work pretty closely together. This is one of only two solution sets that uses a single UI for all provisioning—in this case the Puppet UI. Support for operating systems is broad, including all versions of Linux and Windows to boot. (get it? To boot?) If your data center is highly heterogeneous, or you are already a Puppet shop, it is worth looking closely at this combination. You just have to keep in mind that it is improving at a rapid rate, so those things you don’t like likely will change soon.

    Satellite + Puppet + Foreman

    As of version 6.0, Satellite is a collection of tools knitted together that include Puppet with Foreman. This means that Red Hat has a vested interest in making certain the two tools work well together. And they certainly seem to. While the operating systems support is limited to Red Hat OSes, the solution is pretty rock solid, and like Razor and Puppet, it is improving at a rapid rate since the switch to this new platform. The UI is independent of the tools underneath, and this is the other solution that offers a “single pane of glass” for all installation activities. The one catch with Red Hat, as mentioned before, is that the functionality of underlying systems is only supported when used with Satellite. How that fits into your organization’s plans is up to you.

    Close Enough to be Second Cousins

    There are some combinations that, while not officially interdependent, have a long history of mutual support, and thus make them strong combinations. These tools tend to come with straightforward directions to make them work together, and those instructions generally work well. Once paired, they can manage complete automation in a relatively integrated fashion.

    Note that these combinations are not even a little exclusive. They are pairings the project teams saw made good sense, and have grown out of closeness. Both sides of these solution sets interoperate with other tools (see the next section).

    Ansible + Cobbler

    Ansible and Cobbler both trace their genesis to Red Hat, and they share some employees/team members—and have for a long time. That makes it natural that the two are closely integrated. Before Ansible started discussing all of the other server provisioning tools out there, it already had Cobbler integration.

    As one would expect, there is Ansible setup, Cobbler setup, and then integration. But once combined, Cobbler can do for bare metal, Ansible can do for cloud, and together they provide a full-stack solution. Since both support most operating systems, the final solution does, too.

    Juju + MaaS

    Juju and MaaS have interoperated for a long time. Both are projects of Canonical, so it is no surprise that their interoperability continues. Years ago this author used them together and it was … difficult. But that was years ago, and just in the last year both projects have made strides in documentation and ease of use. Like most solutions in this category, there are separate installations and then integration configuration. Since both support a large number of operating systems, this solution can manage the full data center also.

    Their traditional home, and the one you’re most likely to be comfortable with them in for the short term, is Ubuntu-heavy shops. But they do support most platforms out there, so it’s worth the time to look into them even for non-Ubuntu shops.

    The Extended and Mixed Families

    Then there are the ones that have not forged solid one-on-one relationships, some because they are forging a separate path and some because they want to interoperate with everyone. Note that some of the above systems meet this category—Puppet is integrated with most of the server provisioning tools out there—but due to space considerations, we’re not going to go over those products again here. This list contains both application provisioning and server provisioning tools not mentioned above.

    Chef

    Chef has put the bulk of its focus on cloud and virtualization, with the notable exception to this trend being in AIX, where it does support server provisioning. “The bulk of its focus” is not absolute, though, and that’s on purpose. Chef has had presenters at the annual conference talk about using it with server provisioning tools, so it isn’t unaware of the integrated solution space, just that’s not where their focus has been.

    SaltStack

    The Salt crew is happy to support everyone. While they don’t talk loudly about whom they are integrated with, the product is in use with most major server provisioning tools out there. As mentioned in the SaltStack write-up in part two of this series, cloud integration is pretty broad also, making Salt perhaps the most versatile of the application provisioning tools with regard to full data center automation.

    Stacki

    Originally built to interoperate with SaltStack, Stacki (disclaimer: this author’s employer) has moved on to support most of the availableapplication provisioning tools. The catch with Stacki is that some are supported with sample integrations and video tutorials at the open source layer, while others require a license and engagement of Stacki core engineers to obtain support. So there is a video showing Ansible and open-source Stacki working together and talking about the integration, but it would require a paid license and negotiation to get SaltStack or Chef with Stacki. Because of the way Stacki is designed, it is perhaps the easiest of all these products to integrate—simply add the appropriate “Pallet” and tell Stacki to include it in installations. It is getting the correct pallet that requires licensing for some platforms.

    Conclusion

    Since any server provisioning tool that allows users to customize the install can have the agents for a given application provisioning tool be the last step of a given install, these overviews are at least a little bit arbitrary. To be precise, they are easier to use in the given configurations, but all application provisioning tools can be forced to interoperate with any of the server provisioning tools listed.

    The point of this series was really to help those looking into full data center automation quickly get up to speed on what’s available in the marketplace, and as much information as a few 1,200-word blogs will allow to help shorten the list to ones relevant to your organization. It is by no means complete, and there are issues and concerns with each of the products that only further research will bring to light, so don’t stop here.

  • FLAP, Part 2: Application Provisioning

    FLAP, Part 2: Application Provisioning

    In part 1 of this series we talked about server provisioning tools. In that post, we mentioned some of the more prominent application provisioning tools on the market with regard to integrations. This post will focus on the application layer of full layer automated provisioning (FLAP).

    Application Provisioning Who’s Who

    Application provisioning technologies most likely are familiar: Puppet, Chef, Salt, etc. We’ll take a look at each, how they compare in the areas of control mechanisms, operating systems support and ease-of-use features (integration with server provisioning tools is mostly covered from a server provisioning tool perspective in part 1 and part 3 of this series, because integration must be managed in the server provisioning tool). A blog is too short to go much deeper than that, but this will help you determine which ones should be considered in your environment. Information here is valid as of the day this blog is written, but the market is moving fast, so don’t hesitate to check at the provided links for improvements. Because this topic is a popular one, don’t hesitate to look for up-to-date information provided by other sources, either.

    It is worth mentioning that all of these tools are moving to support configuration of network equipment. For a guy who spent years worrying about programmability and interoperability at a networking gear vendor, that is very cool news. I haven’t had time to actually try any of that support yet, so I’ll leave evaluations to you.

    A word of advice: This entire marketplace has cutesy-naming disease when it comes to really important concepts. While you are grimacing at “recipes” or “grains” or whatever, learn that terminology. It may be cutesy, but the heart and soul of these products hides behind those names and the details of what concepts they encapsulate. During your evaluation period preferably—but certainly once one is chosen—you should get those terms and their meanings down pat. It will make your life much easier. The same is true with the variety of scripting formats. They’re all communicating basically the same information, but so do English and Japanese—knowing one doesn’t automatically makes you an expert in the other. Take the time to truly understand the scripting language for your tool of choice.

    Ansible

    Ansible is a full-app provisioning tool that allows users to install and customize applications through a combination of a command line and scripts. The tool is well-documented, easy to install and works very well at its designed task. Ansible offers support for Red Hat and Windows running on Docker, OpenStack and AWS. Notably missing is VMware support, but since the machines are available via IP and the OSes are standardized, VMware virtuals can be managed/orchestrated with Ansible.

    The open source version of the tool comes with command line interfaces and management is handled via command line and editing scripts. The premium add-ons such as Tower include UI, analytics and programmable APIs.

    Since the purchase by Red Hat, it is reasonable to expect an increased focus on RHEL for this tool and a commensurate reduction in non-Red Hat OSes. Not that this is assured, but history says it is a likely long-term result. If you’re a Red Hat shop, Ansible is a good place to start—though looking at Satellite—and trying to predict how Red Hat’s various open and add-on provisioning tools will evolve and emerge—is a good idea for any evaluation at this point in time.

    Chef

    Chef is one of the original application provisioning tools out there. It is managed through command line and scripts in the open source version, and through these plus a GUI in premium. Unlike Ansible, many Chef APIs are bundled with the open source product, only APIs for premium services like analytics and reporting are premium.

    Chef supports most variants of Linux and Windows, with most cloud and virtualization vendors available as platforms. In terms of completeness of support, it really is up toward the top of the list.

    If you have a largely heterogeneous environment, Chef is a good tool to look at for automating application provisioning—be it for deployment, refresh or continuous delivery.

    Juju

    Juju advertises itself as an orchestration layer above the other products in this space. While there is a small amount of truth to that claim, most of what you can do in terms of orchestration with Juju can be done with the others. In fact, Puppet has kind of found a home in entire complex multi-server systems. Juju comes with GUI in the open source version, but the product does not have a traditional API.

    Juju supports Ubuntu and Windows, and is interoperable with most of the major cloud providers. Deep server provisioning integration is really only provided for MaaS (another Canonical project), but users have made it work with other server provisioning tools. Juju scripts (charms) are primarily written in Ruby, though other languages are definitely possible. For Ubuntu shops, Juju would be the item I’d place on the top of the application provisioning list. For Windows shops, it would definitely be worth considering. For Red Hat shops, other solutions are probably more appealing.

    Puppet

    Puppet should need no introduction to readers of devopsy.kinsta.cloud. The standard in application provisioning, it uses scripts and command lines to deploy applications with a pretty broad base of user support that equates to lots of user shared scripts. Other vendors in this list also have vibrant communities contributing scripts, but Puppet, by sheer weight of numbers (and marketing), has a larger selection available.

    Supporting all major operating systems, Puppet’s documentation is solid. The only quibble is that you have to be careful when coming from a search engine about whether you’re on the open source documentation or the Professional Edition documentation. Puppet comes with APIs, though the GUI is only available in the commercial version. Puppet is the progenitor of Razor, the server provisioning tool, and while it has been used with all of the provisioning tools in the first iteration of this article, expect it to work best and provide the most value with Razor going forward.

    For highly heterogeneous shops, or shops that include multiple server provisioning tools, Puppet is a solid bet. While it will work fine in other situations, the value-add of a more focused product is worth considering if your environment maps to another tool’s strengths.

    SaltStack

    Salt is an interesting member of the application provisioning crowd. They don’t share name recognition with some of the other players, and yet have a respectable user base. Their product is certainly comparable to the others in this market, and the user community is pretty active. Not being deeply associated with any of the server provisioning tools, SaltStack has been integrated with most of them.

    The Salt Master runs on almost all flavors of Linux, and can manage all major Linux distributions plus Windows, OS X and Solaris.

    Like all of these other tools, SaltStack can manage any machine that is accessible over the network and is running a supported operating system., Additionally, the dev branch has Salt Cloud built in to the Salt Master, which spins up cloud images, automatically puts them into salt and starts managing their configuration. Expect that this type of functionality will become the norm for the market (Juju does some of this also, and others have varying levels of cloud support)—it just makes sense, and it compliments server provisioning. There is a tiny bit of overlap in the VM space, but then you can choose whether to use server provisioning or Salt Cloud to handle those instances.

    For highly heterogeneous shops that wish to use best-of-breed (actually “best fit” is a better name) server provisioning along with a solid application provisioning tool, SaltStack is a heavy hitter worth considering. If you have a more specialized environment focused on a single OS, SaltStack will work, but its advantage of broad support for both OS’s and server provisioning tools will not come into play.

    Satellite

    Satellite is Red Hat’s provisioning solution. Unlike most of the others in this list, it is a cross between server and application provisioning. In actuality, it is that “layer above provisioning” that Juju positions itself as. Under the covers, Satellite (as of release 6.0) is using Foreman for server provisioning and Puppet for application provisioning. The reason it still makes this list then is because it does do application provisioning, and does orchestration too. You can use it to spin up MariaDB or OpenStack, and it is a marketed and supported product, so it fits the list.

    As one might wholly expect, Satellite supports RHEL. It can manage instances of RHOSP and RHEV, along with VMware and Amazon, and does bare metal through Foreman also. Satellite is deeply tied to Foreman and Puppet, so if you have issues using this tool set, it’s not going to be your first choice. Because Satellite extends and improves Red Hat Network access, for Red Hat shops it should be No. 1 on the list of viable options, but if you use a lot of non-Red Hat machines, including (as of version 6) Fedora and CentOS, it is less appealing. It used to be that Spacewalk was the viable option for those CentOS and Fedora installs, but now they are separate code bases, meaning you have to maintain two different systems. At that point, it is worth looking at other options available on the market.

    The newer version 6 architecture is designed to provide a platform moving forward that is more powerful and adaptable, so expect to see good things from it if you are a Red Hat customer, possibly even cancelling out the disadvantage of no support for CentOS and Fedora.

    Summary

    All of these products are very good in their best environment, and most of them will give you pain in their very worst environment. The question is, which environment do you have? While only employees can answer that for a given environment/tool combination, hopefully there are some pointers here—and elsewhere—that can help make the decision.

    In the third and final installment, we’ll look at the combination of server and application provisioning to offer full stack automation.

  • FLAP, Part 1: Server Provisioning

    FLAP, Part 1: Server Provisioning

    We have reached the state where end-to-end data center provisioning is a reality. It’s early yet, so there is a certain amount of work involved, but today a DevOps team essentially can turn on the automation and move on to other DevOps priorities. It takes some effort, depending upon the solution chosen, but the end result is an infrastructure that provides for installation of operating system (OS), hardware (if needed) and apps, along with configuration of the whole, at the push of a button.

    There are a number of solutions ready for your data center today, and this three-part series will explore them. In the first part, we’ll talk about server provisioning—the part that takes you from uninstalled server or virtual to a functioning OS configured for your environment. The following two articles will cover application provisioning and merging the two to achieve FLAP. (Yes, I made up FLAP. Use it or not, as you see fit.)

    Disclaimer: I work for a vendor in this space. While I do not believe that impacts my objectivity, I will note where my employer is mentioned, and leave it to the reader to decide if I’m blinded to some items/issues based upon my association.

    Server Provisioning: What and Who

    A server provisioning system takes a physical or virtual server from nothing to fully installed OS. This layer of FLAP is fundamental to automating the entire process. If the system has to be installed manually, using tools and scripts to install applications just doesn’t offer the overall benefit as when the entire process is automated.

    For this discussion, we will be sticking with the major provisioning tools out there. To be sure there are more, and I have information about them, but for this short blog post, we’ll stick with the ones that hold something special—market share, history, integrations, associations—that make them a good long-term bet. Our list is alphabetical, and does not show preference by ordering.

    Cobbler

    Cobbler is one of the original server automation tool sets, and definitely has the history and stability to warrant your attention. For shops that use a lot of Fedora and CentOS, Cobbler in combination with Spacewalk is particularly appealing, as we’ll talk about later. Support includes most Linux distributions. Users have created ways to include Windows in the list of installable OSes, but there is not yet an official solution for Windows.

    Cobbler is integrated with Ansible, Spacewalk, Saltstack, Puppet(user created), and older versions of Satellite. There is some amount of concern about Cobbler’s future growth, simply because Red Hat moved Satellite off of Cobbler with release 6.0, though Red Hat still supports Cobbler. It is too soon after the split to predict how this will fall out, but currently it seems to have motivated the Cobbler core team to do more. Users concerned should check GitHub analytics to see what the trends are for committers and commits at the time they are evaluating (good advice for all of these products, but more pertinent for one where there is a question of its future).

    Cobbler’s hardware support is better than most, and in terms of application support it is a pure-play server automation tool; there is no support for applications at all—application provisioning is expected to deal with that aspect.

    Foreman

    Foreman is an open source project that is currently enjoying the sponsorship of Red Hat, but support is not limited to Red Hat OSes. In fact, Foreman will install just about any flavor of Linux, and can target most physical/virtual/cloud platforms. Using WDS, Foreman can also install Windows.

    One of the big strengths of Foreman is its interoperability—it doesn’t much care which application provisioning tool you are using. It is integrated into Satellite (with Puppet), and has support for all of the other major application provisioning tools. This means that no matter what you’re using to get your apps installed, Foreman is worth looking at for getting your servers installed.

    Like most products out there, Foreman provides CLI, API and GUI. They’re relatively stable in relation to the market, simply because Foreman implemented them earlier than most competitors, and is constantly improving them.

    In short, if you are in an environment that has a wide selection of OS, architecture and application requirements, Foreman should be at the top of your short list. For more focused solutions (in terms of OS, whether or not cloud, or application provisioning tools), it makes more sense to compare specific features/functionality with some of the more focused tools in this list.

    MaaS

    MaaS was the sleeping giant of the automation world, but it does indeed look as though it is awakening. MaaS is part of Canonical’s Ubuntu project, so there is plenty of backing, but historically its goals have been modest and its documentation more so. That changed some time over the summer/fall, and MaaS is showing increased support for OSes, better documentation and a more focused attempt to get the word out. If you are primarily an Ubuntu shop, it is well worth considering.

    Like most of the products in this offering, MaaS uses PXE to get the machines into its system, and then uses predefined images that have been imported (Ubuntu images don’t need importing) to install a machine from bare metal/virtual to working OS. MaaS proclaims support for CentOS, RedHat, SuSE, Ubuntu and Windows. For those unfamiliar with the space, that’s an impressive list. But the last time this author used MaaS, it was still only Ubuntu, so I cannot comment on the level/quality of support for each of the OSes. Due diligence should involve testing with those OSes your organization needs.

    The weakest part of some installers is non-standard hardware, such as RAID controllers, specialized network cards and the like. MaaS supports these components and includes UCS support in its packaging. Directly mentioning UCS support is only to be found with MaaS and Stacki, so if you’re a UCS shop, it is worth considering.

    In addition to CLI, MaaS touts an API and a GUI. These are important tools in the DevOps sense, because the API lets you tie MaaS into your automation architecture, and the GUI helps find issues specific to MaaS and its domain. Particularly interesting is the collection of logs that show install results and operations results clearly for each server.

    The one major drawback to MaaS is its limited application provisioning support—it is well-integrated with Juju, but none of the other big names in the space.

    In short, for Ubuntu shops using Juju, this is the tool to put at the top of your short list. For everyone else, the tool is viable, but should be considered against the features/functionality of other tools.

    Razor

    Razor is Puppet’s newer bare metal installer. It is young yet, and there are still issues to be worked out, but the large cadre of developers that are dedicated to Puppet say that Razor will continue to improve as time moves on. At current it directly supports most flavors of Linux and Windows installations. The catch to Razor is on the other end: It is designed to work with Puppet, and support for other application provisioning tools is low—indeed, is limited to user-designed solutions. Needless to say, because it is sponsored by Puppet, it is expected that official work in this area will remain slow, as it is not a priority.

    Razor does not have a dedicated GUI; instead, it uses Puppet’s GUI to report results. As with integration, this is okay for Puppet shops but pretty much eliminates Razor from consideration for other shops. (More on this in a future blog in this series.)

    Because Razor is new, there are a couple of serious caveats with the product. First, install is one of the worst in the market, with a lot of pre-install steps, a lot of install steps, and configuration of an array of underlying systems by hand. Second, each currently installed server that you do not want Razor to re-install must be entered into Razor manually on the command line. This is so limiting in a generalized data center environment (where Puppet shines) that you can expect this limitation to be resolved quickly.

    In short, for Puppet shops that do not see their application provisioning tool changing, this is the tool to put at the top of your short list. For other shops, it’s worth looking at competitors first, simply because Razor requires Puppet.

    Stacki

    Stacki is an open source project sponsored by StackIQ (as promised, this is my employer). Having roots that stretch back more than a decade, Stacki specializes in RedHat/CentOS installations with additional support for clustered software. As of this writing, CoreOS and Ubuntu have been added to Stacki’s list of supported OSes, and expect that list to grow over time.

    The biggest appeal of Stacki is the variety of ways to get systems under management. All of these apps will auto-detect a new server on the network, grab it and install it. Stacki adds the ability to list just the servers that should be clean-installed in a spreadsheet, the next time each of those machines boots they will be installed, but no other machine will. While there are ways in all of these systems to simulate this functionality—you can add exceptions to the “install all machines that boot on this subnet” rule via the command line in Razor, for example—using a spreadsheet to affirmatively manage what machines are to be installed and how gives an offline bulk way of managing installations and re-installations.

    As of this writing, Stacki does not have an API, and only with commercial add-ons can a GUI be had, but expect one or both of these things to change as the community begins guiding development efforts. Stacki is optimized to spin out hundreds of servers in minutes or hours instead of weeks or months, utilizing a peer-to-peer parallel installer to service installations faster as more servers come online. While it works as well as others in smaller environments, it really comes to its own when there are a large number of servers under management. On the initial installation front, while the other systems in this list require hand-installing and/or configuring prerequisites and sub-systems, Stacki comes as an ISO with a UI that asks a few questions and configures everything.

    Stacki has been used with most (Ansible, Puppet, Saltstack) of the major application provisioning tools this series of blogs will consider, making it versatile on the application side of the equation.

    Overall, if you’re a Red Hat shop that uses CentOS a lot—particularly a larger shop—Stacki should be the first stop on your short list. For everyone else, the tool is viable, but should be considered against the features/functionality of other tools.

    Stay tuned for part two of this series, which will cover application provisioning.

  • Top 5 Reasons to Back Up Your OpenStack Environment

    Top 5 Reasons to Back Up Your OpenStack Environment

    I am a big fan of lists. If you recall the movie “High Fidelity,” John Cusack’s character Rob and his co-workers—armed with an encyclopedic knowledge of all things musical—compile “top five” lists for every conceivable occasion, openly mock the tastes of people, and, every so often, sell a few records. In the spirit of a lists, here are the top five reasons to back up your OpenStack environment. We thought that it would be helpful to many.

    If you’re currently working with OpenStack clouds, or are in the process of embracing it, you need to include backup and recovery for OpenStack on your list of to-dos from the start. You may ask, “Why? I have orchestrated/automated everything, I just need to spin resources up and down.” Cloud environments are fundamentally different from the virtualization we have become used to. Virtualization relies on infrastructure for resiliency. In the cloud, distributed applications are responsible for their own resiliency, and are independent of the underlying infrastructure. Therefore, cloud apps need backup and recovery options, lest you face a complete machine, data or file meltdown.

    We’ve compiled the top five reasons to backup your OpenStack environment. Take a read and it will become clear why you must include backup in your OpenStack plans.

    1. Compliance — Compliance may seem like an issue for only certain industries—finance and health care come to mind—but nearly every organization has to deal with compliance issues. For some, it’s compliance to external regulations, while for others it’s simply complying with internal policies and best practices. Backup for OpenStack is an essential component for compliance; it ensures the availability of the data at a specific point in time within your OpenStack applications. I was recently involved in a dialogue/string on LinkedIn watching experienced OpenStackers go back and forth debating “pet vs. cattle.” “I am getting tired of it,” one person said. “This is cloud. It’s about scaling up and out … With automation and orchestration there is no need for cattle.” This conversation went on for days. Theory is great. Practice is reality. What people failed to discuss is that compliance or the business units will drive many requirements (such as backup) either today or tomorrow. End of story.
    2. DevOps — Backup is often the forgotten tool for DevOps. Containers are great to use, but the ability to test with real data is an issue. The reason to use backup is quite simple: test and development. With a copy of a production environment, you now can test upgrades and new features in your agile development process. Furthermore, if you lose a machine or your data, or you end up with corrupted files while in the middle of testing or development, your life will be much easier if you have an appropriate backup solution in place. Restoring machines, data and files is a better option than re-creating them entirely. Cookbooks and recipes are great as well, but you still need to couple it with the appropriate “ingredients.” It is like playing with a fake kitchen as a child … do you want to make breakfast with plastic pancakes and toast? Nothing beats the real thing. Yes, more resources are required for one over the other, but when something goes wrong—it is unfortunate—people point fingers.
    3. Disaster recovery plans — While the terms “backup” and “disaster recovery” often are used interchangeably, they are actually not the same thing. You might have what appears to be a great recovery environment (ideally one that mirrors your production environment for high availability), but without a comprehensive backup solution, your recovery might not be the painless experience you expect it to be. Perhaps the solution for intelligent disaster recovery is to marry backup with a Swift environment so you can recover from any point in time, anywhere in your organization.
    4. Save time — Often recovery can be a frantic process, driven by a business need to have the machine, data or files recovered as soon as possible. Or, more likely, sooner than possible. If you are trying to re-create something under that type of environment, there’s a likelihood that mistakes could be made. This becomes increasingly true the more complex and mature your cloud becomes (think about the growing number of workloads and applications). A much easier approach—one that provides you with peace of mind—is to have a comprehensive backup solution ready and available for when you need it. There’s also another reason backup can save you time: When you have to re-create the lost information, you are taking your time away from something else—and that means you are taking a big hit to your productivity. Save yourself and your organization a big headache with backup.
    5. Security and forensics — You never know when you are going to be asked to provide data or files related to a security issue or a forensics investigation. The right backup solution not only makes it easier to assist with the activity, but think how good you’ll look by being able to deliver the information the forensics team needs quickly.

     Are you currently looking into backup solutions for OpenStack? Now that you know why you need backup for OpenStack, learn more about how to backup OpenStack.

  • How To Maintain Open Source Code Hygiene

    How To Maintain Open Source Code Hygiene

    To maintain good open source code hygiene in the development process, you must first establish what you mean by open source code hygiene, including good and bad hygiene. “By hygiene, we just mean is it appropriate? Is it clean for the intended use?” says Mike Pittenger, VP of Strategy at Black Duck Software.

    Black Duck suggests three metrics it uses to determine whether it is appropriate to apply a given open source code project to a certain development endeavor:

    • For software that will be distributed at large, avoid open source code that the development community has made available through a GPL license.
    • For secure software, check your potential open source code for vulnerabilities in general and make sure to use the most secure version available.
    • Monitor the code periodically to check for new vulnerabilities or other concerns.

    As an example of the third point, Pittenger notes OpenSSL is no longer a viable option because of the vulnerabilities that have come to light in the form of Heartbleed.

    Checking Source Code for Open Source Concerns

    While the InfoSec Institute recommends at least 14 different open source vulnerability scanners for web applications alone, offerings such as Black Duck’s can serve as a shortcut to discovery of known bad open source components. “Instead of having to load Nessus, Metasploit or Nexpose with a Heartbleed rule and scanning every application, every server that they’ve got, they can just go into their Black Duck console, type in OpenSSL, and it would immediately show them every version they used in their software applications,” says Pittenger. Of course, you should always do your own due diligence to determine whether any scanning product is the best choice for your organization and aligns with your needs and requirements.

    Due to the many open source code reuse scenarios active in enterprise DevOps shops, you will want to continually monitor code for newly registered vulnerabilities. Enterprises can view reusing existing assets as a built-in savings by not having to purchase or create new assets. In the case of reusing legacy source code, there is also a time savings, as developers want to shrink time to market for code-based projects.

    “Let’s say you have a company that is pulling information from a database to display in pie charts. You have a rendering component that you will reuse. When you’re starting on a new product, do you build a new rendering library from scratch or do you use the existing one?” asks Pittenger.

    Even if those components are open source, there can be great savings in reusing the library.

    The fact that enterprises do this underlines the importance of continuing to check any code that leverages open source code, as this is how vulnerabilities can spread over time. It’s also important to check open source code, because it may be a form of low-hanging fruit for attackers. Open source code is fully viewable by attackers who build malware and launch attacks, whereas proprietary code is closed.

    Since people constantly discover new vulnerabilities in open source code components that previously had tested free of such holes, it’s important to monitor those components for newly uncovered weaknesses, even as you reuse open software in new projects.

    Containers are a great example. You may want to dismantle an image in varying degrees to build a new application. “If I have a base image and it’s built using a LAMP stack (i.e., a Linux OS, an Apache HTTP server, a MySQL RDBMS and PHP), I can strip down my base image, take away the existing app and take it all the way down to the Linux component, then build it up for the new application,” Pittenger said. “That is a typical, consistent reuse of open source components.”

    During the process, recheck every component for vulnerabilities. Even if everything checks out doesn’t mean you have the best assurance of security for those components going forward. By monitoring those components, you can stay on top of updates to fix any newly discovered security holes.