Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • DevOps Chat: Rachel Reinitz, IBM Bluemix Garage Method

    DevOps Chat: Rachel Reinitz, IBM Bluemix Garage Method

    newlogoIn advance of InterConnect next week in Las Vegas, I had a chance to sit down with Rachel Reinitz,Distinguished Engineer & CTO of IBM Bluemix Garage. I have met Rachel several times and it is always a pleasure to catch up on all of the innovative things her and her team are doing with Bluemix Garage. In this conversation, Rachel and I spoke about open technologies which is a focus for us this month, as well as a big focus of the Bluemix Garage method. We also spoke about “innovating like a startup, while scaling for the enterprise”.  To learn more about the IBM Bluemix Garage Method, please visit: https://www.ibm.com/devops/method/

    You can find our conversation below and a transcript of it below that. Enjoy!

    Alan Shimel:                Hi, everyone. Alan Shimel, devopsy.kinsta.cloud here for another DevOps chat. Today’s special guest is Rachel Reinitz of IBM. Rachel, welcome.

    Rachel Reinitz:            Thanks, Alan. It’s always nice to be talking with you.

    Alan Shimel:                Yes, actually I think the last time we spoke was DevOps Enterprise Summit in San Francisco. And Rachel, for our listeners who may not be familiar or didn’t get a chance to see our video which is on YouTube from DevOps Enterprise Summit, why don’t you tell them a little bit about yourself.

    Rachel Reinitz:            So I am a distinguished engineer at IBM and I co-founded and I’m the technical lead for the IBM Bluemix Garages.

    Alan Shimel:                Excellent. And you know, Rachel, there’s been a lot of announcements and news and developments, no pun intended, coming out of the IBM Bluemix Garages that some in our audience may not be familiar with. So give us some of the highlights. First of all, what is the IBM Bluemix Garage? And what’s happening with it?

    Rachel Reinitz:            So we are a consultancy within IBM. We work with customers making them successful in adopting Bluemix. And we are unique in that we are housed in startup communities. We work with both startup customers through to enterprise customers. And really what we work on is taking an idea, a problem or a concept and applying IBM design thinking, lean startup, extreme programming, DevOps to go from that idea to the definition of a minimal viable product and then onto building it and getting it out there and testing it out with users, testing the business hypotheses, all the really good DevOps practices and utilizing Bluemix which also gives us a lot of acceleration because of the kinds of services in the environment that we are in. So that’s what we do. It’s an exciting place to work and we have – and for clients to come into with designers, architects, developers. That’s pretty much what we do.

    Alan Shimel:                Got it. Rachel, I was lucky enough to visit an IBM Bluemix Garage. I guess it was at the DevOps Enterprise Summit, around that time in San Francisco. And the best way I can explain it was it’s sort of a lab, kind of a living lab where experimentation was going on as well as education and development. And it seemed like a pretty cool environment. And both for enterprise as you mentioned, anything from enterprises on up to startups.

    Rachel Reinitz:            That’s right. I think that that’s a good summary of it. We are in the startup community so we work with the startups there. And it really helps enterprises to innovate like startups, to get kind of out of their environment into our environment, a very collaborative space as you saw, when you were visiting right in the hardest heart community, we’ve actually, in some of environment news, we’ve moved to a different space still within the galvanized community in San Francisco, a bigger space, which is really exciting. We just had a little party there. And we have other locations. We are in Toronto and in London and we have a location in Nice. Each one is in a different kind of startup community. And we are working on Melbourne. So you’ll have to come investigate some of our other locations.

    Alan Shimel:                Absolutely. We are actually doing a series of roadshows, one of which will be in London in April. We’d love to get some of the IBM Bluemix Garage folks at that. But we will talk off-line. Rachel, one of the reasons I invited you on was this month on devopsy.kinsta.cloud, we are really focusing on what we call the open technologies and open enterprise. And by that we don’t mean just open source software, though certainly open source software is part of it. But there’s more to open then software, isn’t there?

    Rachel Reinitz:            Yeah, I agree. I think that’s knowledge sharing is a big part of open, sharing different kinds of knowledge in different kinds of mediums. And I think where you’re going is to the fact that we’ve now taken what we’ve been taking in the Garage and actually merged it with what we’ve been doing in open source and DevOps Transformation within IBM and we published that as the IBM Bluemix Garage method which is out there in public, open for people to use. And we’re also looking for contributions.

    Alan Shimel:                Absolutely. And it is open, right? The whole methodology, if you will, is open. And I happen to know only because I’ve spoken to you folks before, a lot of the tools you are using are open source tools, traditional open source software. But really, when we talk about the open garage method, as I’ve heard it referred to, it’s open through and through, if you will, right? It truly is open technology.

    Rachel Reinitz:            Yes. So it’s taking the lessons that we’ve been learning, it’s providing prescriptive paths and recommendations and that crosses from our software to what we are doing in the open community and open source, open-source testing tools, third-party tools. It’s what we find is working in projects that we are doing and recommendations around specific kinds of applications like a mobile application. What do we recommend? What did we learn and are sharing those learnings as you were saying out in the open? How do you go about testing them, for example? What are your different options? Things like that.

    Alan Shimel:                Fantastic. Rachel, how big is Bluemix a part of this garage method, though?

    Rachel Reinitz:            It’s definitely a big part of it in terms of we think that using Bluemix and other IBM technologies is a way to really accelerate building these kinds of applications. But it’s not exclusive to that. So when we talk about innovative practices, when we talk about some of our toolchain, no, it could be used with other technologies. Obviously I think there’s a lot of good reasons to use it with Bluemix.

    Alan Shimel:                Sure, sure. And again, you know what, Rachel? We may have some folks who perhaps aren’t even familiar with Bluemix, why don’t we quickly give them a level set. When we say Bluemix, what are we talking about?

    Rachel Reinitz:            So Bluemix is IBM’s platform as a service. It exists in public which is then hosted in our data centers, our public data centers for software. We also have a dedicated version where we provide a managed single instance of that same capability. And then we have what we call local which can deploy onto your own systems behind your firewall. And Bluemix has multiple models for writing applications with Cloud Foundry containers and VMs. And it has a whole range of services. At last count, I think we were at about 140 and those are IBM services like our SQL DB which is a form of DB2. But it’s also third-party services like New Relic. And as I said, it runs Cloud Foundry, it runs open-source runtimes. You can bring your own runtimes. So there’s a lot of openness to it.

    Alan Shimel:                Excellent, excellent. And Rachel, you mentioned IBM was looking to work with third-party partners around this open methodology, open-source. What kind of partners, what kind of contributions would you be looking for that you think makes sense?

    Rachel Reinitz:            So we have the concept of an open toolchain where different technologies can be integrated, again, around solving a particular kind of problem like mobile. And our toolchain team works with different partners. So we are very open to partners coming in and saying hey, can we be part of the toolchain and talking about how you integrate? So for example Slack is part of what we talk about there. That’s one kind of integration. Another level of integration we have is in Bluemix itself. So we have third-party services there and we have a whole program for how a company that’s providing capability that would be of interest to developers can integrate into Bluemix and be part of our ecosystem.

    Alan Shimel:                Got it, got it. And Rachel, believe it or not, we are coming up on the – I promised you it would be about 10 minutes. We are going to run a little over but I wanted to quickly pivot, if you will. Besides this month being open technologies on devopsy.kinsta.cloud, we are also featuring our coverage of the upcoming IBM Interconnect Conference in Las Vegas. And I would imagine, pardon me, I haven’t pre-checked; but I would imagine IBM’s Bluemix Garage is going to have a prominent place at Interconnect, correct?

    Rachel Reinitz:            Yes, yes. I’m really excited about Interconnect. We have, in the expo area, we actually have basically a mini garage set up, even with a motorcycle, I’m told. And we basically take you through the journey with a focus on our innovation path and we are sitting right next door to our DevOps team who also support the method so you can come and see what we are doing in terms of serving your innovation process and we show code, of course, and delivery pipeline. And that if you want to get even deeper and know the different options, we have the DevOps booth right near us. We have customers coming and talking about the work they’ve been doing with the Bluemix Garage method. We have workshops where you can get into a roundtable and really dive into it. So there’s a ton of things going on at Interconnect around the method. And of course, I’m presenting, too.

    Alan Shimel:                Really? What are you presenting on, Rachel?

    Rachel Reinitz:            I’m presenting on the Garage and on the method. I’ll be speaking, I think it’s Monday at 6:00 PM and then there’s a variety of different sessions. And then most exciting is I have some of my clients presenting with us.

    Alan Shimel:                Really? In case anybody listening is actually attending Interconnect as they very well might, can you give us – I don’t mean to put you on the spot, but can you tell us maybe customers and which sessions customers are talking at?

    Rachel Reinitz:            I’d have to look up specific sessions but we have KS2 and ManualLife, clients I’ve worked with really closely. Those are a couple of the – oh, and we have Spigot which has a really interesting cloud sourcing platform. So those are some of the customers that are going to be speaking. And there’s maybe a couple others that are ready to be talked about yet.

    Alan Shimel:                Fantastic. And of course, on the expo floor, people can actually get hands-on with the Garage for themselves and see what it’s about.

    Rachel Reinitz:            Absolutely.

    Alan Shimel:                And just again, in case anyone listening, Interconnect is February 21-25, I believe. And it’s at Mandalay Bay and MGM Grand in Las Vegas. Rachel, just one more question. What’s next?

    Rachel Reinitz:            More. More services. More services coming on Bluemix. We just had some – there’s new analytics, there’s new Watson services. It’s more coming, more projects with clients, cool Internet of things kinds of projects, more garages in the future building up the ones we have. Lots is going on. And in the method, too. More tools in the method, more prescriptive paths.

    Alan Shimel:                Do you think we will see a Bluemix Garage method 2.0?

    Rachel Reinitz:            Not 2.0 because we continuously deliver.

    Alan Shimel:                That’s exactly right. Good jump on the answer.

    Rachel Reinitz:            We came out with our MVP at DevOps Enterprise and now we will be having, you’ll see a lot more in there for Interconnect. We will just keep on going.

    Alan Shimel:                Fantastic. Rachel, we are way over time so I’m going to need to call it a day here. But Rachel Reinitz, distinguished engineer of IBM and sort of mother of IBM’s Bluemix Garage. Welcome. Or thank you for being on DevOps Chats today. Look forward to seeing you at Interconnect February 21 out in Vegas. And look forward to continuing development on the IBM Bluemix Garage and Garage method. Thanks for appearing today.

    Rachel Reinitz:            Thanks, Alan.

  • DevOps Stack on a Shoestring Budget

    DevOps Stack on a Shoestring Budget

    There are competing products, both open source and licensed, in any DevOps area that you can think of. The Operations groups in companies tend to accumulate such applications over time as determined by ongoing projects. Team members’ familiarity with a certain application could be a factor in adding it to the tools stack.

    Companies, therefore, end up with a mixed bag of freeware, open source or community versions of applications and software licensed from the get-go. In a startup environment where the Operations group might be on a shoestring budget, it might make sense to build the DevOps infrastructure on the cheap; using open source software (OSS) products would be one way to achieve that goal.

    The Shoestring Budget

    Below is a list of some of the applications and software for an effective DevOps environment. The list here includes only core applications that are needed to build the DevOps infrastructure. The basic communication and productivity tools that would be used by everyone in the company—and usually managed by IT—are not in the scope. Neither are the compilers and debugging and testing tools used by product development teams. However, in a small company setting, managing all kinds of tools might drop into the laps of the same people driving the DevOps initiatives.

    The objective here is to make a list of software that could be used license-free. That means some of the obvious applications in certain areas might not make it to the list. That doesn’t mean that the open source alternative is an inferior choice; availability of reliable product support is a major factor in most of the software purchase decision-making processes, and licensed software scores over OSS on that front.

    Git for Source Code Management (SCM)

    A code repository might be the first DevOps infrastructure component that would be set up even before a product idea is formalized into a company. Also, both developers and DevOps engineers need it for collaboration, within their own teams and across various engineering teams. Typically in a startup, developers get started with the DevOps work such as build and deployment automation before a dedicated DevOps team would even be part of the engineering staff.   

    In this category, It is hard toimagine choosing anything other than Git. Even though GitHub is not free for maintaining private code repositories, Git server can be setup on-premise for free. AWS has CodeCommit, another SaaS version of Git, as one of the services, and there is no extra cost for using it if you are already on AWS.

    Bugzilla for Bug Tracking

    The popular Atlassian suite of products, JIRA in this category and Confluence for collaborative documentation, seems to be default choices in respective categories. But, both on-premise and SaaS offerings of  JIRA and Confluence are licensed. Bugzilla is a powerful OSS alternative to JIRA that can support a large user base and it has been around for a while.

    If you are open to customizing Bugzilla for specific requirements, it could work better than some of its licensed competitions. Though its primary use is bug tracking, Bugzilla is feature-rich enough to be used for managing all phases of software development and even as an IT ticketing system.

    MediaWiki for Collaborative Documentation

    Though there are multiple, open source wiki applications available, MediaWiki might be the best supported when it comes to the availability of third-party extensions. Out-of-the-box features of MediaWiki are limited, but it can be extended using the free extensions available. Such customizations are very straightforward.

    Jenkins as Build & Continuous Integration Platform

    Though there is an enterprise offering of Jenkins from CloudBees, its open source version combined with freely downloadable plugins would make up a powerful integration platform for build, release and deployment automation—and any other areas in between. Describing Jenkins any more here would be simply preaching to the choir.

    Artifactory for Artifacts Management

    This is one of the applications a lot of companies might not have in their DevOps stack in the initial stage. However, the need to store built artifacts and to cache third-party tools is real, and the packages and binaries would normally end up in S3 buckets or shared filer volumes. There is no reason for not using the OSS version of Artifactory right from the beginning, as it covers the storage requirements already mentioned and much more:

    • Automated uploading of artifacts as part of the build process, typically done from Jenkins
    • Caching third-party tools locally for builds and deployments
    • Storing software artifacts for meeting build dependencies
    • Distributing packages and third-party tools for deployments

    Ansible for Configuration Management and Orchestration

    There are two great products in this space that could be used for free: the OSS versions of Puppet and Ansible. If you are on AWS, Chef is also available, at no extra cost, in the form of OpsWorks. But I would recommend Ansible for those getting started for these reasons:

    • It is an agentless tool and beginners would feel safer to roll out automation using it. A configuration management agent can play havoc with production systems if something gets misconfigured inadvertently.
    • Using the playbooks, deployments can be orchestrated very well using Ansible. In many companies, Puppet or Chef primarily covers system management aspects and there would be additional tools for deployments. Ansible simplifies that scenario.
    • Ansible’s support of Python turned out to be more useful, in my experience, than the support of Ruby in Puppet and Chef. The chances of a DevOps engineer knowing Python is much more than Ruby.
    • Ansible’s handling of host inventory is advanced, especially the dynamic generation of inventory lists. The latter feature is compatible with the provisioning of cloud infrastructure where its elasticity demands the use of dynamic inventory lists in configuration management systems.

    Nagios for Monitoring

    A lot of us in the DevOps community despise Nagios for its blandness and lack of features out of the box. But if you are willing to customize it by developing application-specific plugins, it could be converted into a very effective monitoring platform that can be used to consolidate outputs from the rest of the monitoring and various admin applications. I recommend it for these reasons:

    • Writing plugins for Nagios is very simple. Also, there are tons of free plugins available that would meet most of the system-level and third-party component level monitoring.
    • Developing plugins to add application-level monitoring and to integrate with third-party admin applications is easy.
    • When you want to move on to a next generation cloud-based monitoring application such as Datadog, existing Nagios plugins are normally migrated natively or as an integration.

    In a sysadmin environment where coding skills would be in short supply, Nagios could be of limited use as a systems monitoring application. But DevOps teams should treat it as an easily extendable tool that can be used to implement specific and fine-grained monitoring goals of the company.

    ELK for Log Management

    Splunk was instrumental in changing the way we dealt with log files, and, now there are several good products including Loggly, Logentries and Sumo Logic in that space. Though some of these SaaS products have a free tier and that might be enough for a small operation, it is better to start with ELK (Elasticsearch, Logstash and Kibana stack), mainly because the size of the logs can grow very fast and so would the usage of the log management application in the company. ELK requires substantial DevOps resources to set it up and maintain, as there are three applications involved.

    Closing Remarks

    It is important to have these essential applications, or similar ones with comparable features, implemented or made available to empower the engineering  staff. They facilitate the application development, lay the foundation for automation initiatives, and provide the toolkit to maintain production environments 24×7.

    Though it is normal, the organic growth of the DevOps organization and infrastructure can bring in OSS products with overlapping features, mainly because they could be used license-free. Carefully selecting the tools would help avoid the otherwise-inevitable clutter and the confusion later.

    One aspect not discussed is the total cost of use of a software. If SaaS offerings such as GitHub, Datadog and Loggly, and hosted versions of JIRA, Confluence and Artifactory are used, the cost would only be the subscription charges. OSS applications will need computing resources and some amount of customization and administrative overheads, which could add up to the overall cost.  

    If the objective is to minimize total cost, the upfront decision-making can become much more complicated than it looks. However, for a startup or a small company, getting started with the OSS stack for setting up the DevOps infrastructure can be advantageous if the applications are selected carefully.

  • To Be Continuous Podcast: The Clash in Open Source

    To Be Continuous Podcast: The Clash in Open Source

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul talk about the clash in OSS between developers looking to get paid and software companies looking for things for free.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/244279037?secret_token=s-oeBCP” params=”auto_play=false&hide_related=true&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Edith: So Paul, what do you like about open source?

    Paul: So I guess that depends on whether we’re talking about writing open source or using open sources as part of a company. Writing open source, I like just having all my code out there.

    And I’ll do that as often as possible. Anytime I start a little bit of code, a little library, I’m looking to open-source it and have a readme and have it out there for other people to use. I had a very long discussion, which I eventually won, about the …

    Edith: I like the way you think of all discussions as win or loss.

    Paul: Well, yeah, yeah. It was more on the argumentative side of discussion. But with the other co-authors of the PHP compiler I wrote, which was originally GPL. And I argued very strongly that I actually don’t care if people give back, I just want people using my software. So to me, open source is a way of having people using my software.

    Edith: That’s really interesting. So, I liked what you said at the beginning about, like, you liked open source ’cause it meant getting stuff out sooner. ‘Cause I’m a big fan of that.

    Like, I always say you should just publish stuff. Like, my co-founder John said something really good at our team meeting last week. He’s like, if it’s 90 percent done and you don’t publish it, it’s actually 0 percent done.

    Paul: It’s 0 percent done, yeah.

    Edith: Yeah, and that’s a hard lesson, ’cause you always try for perfection lately.

    Paul: Right, right, right. Well, I think there’s also some really hard downsides about that. So when you publish something before it’s ready, before you really —

    Edith: Well, ready.

    Paul: Well, there’s ready ready, but let’s say ready is, let’s say you publish it before you clarify what the software is to do, or the values or the mission of the software.

    And then you get contributors who are saying, ‘Oh, this is great, but I really want it to do this other thing,’ and you’re like, ‘Well, that’s not exactly what I was going for here.’

    Edith: Well, I’d say most open source projects are so lucky as to get contributors at all.

    Paul: True, true, true. So I’ll give you an example. I wrote the V8 bindings for Closure. So a binding that allows you to write JavaScript and run it in Closure. And I did that for a purely selfish thing, for another, for a nascent pipeline that I was building called Stefan.

    And then people came along and they wanted to do other things with it. They’re like, ‘Okay, let’s add a binding for this other thing.’ I was like—

    Edith: Well, that’s kind of like a pivot.

    Paul: It is, it is, but I really was not interested in it.

    I’d rather someone took it over, took over the project and actually ran with it where they wanted to go, and I can stick on the old version, or I can see if the new version solves it for me, but I wasn’t interested in becoming a maintainer in this thing, I was interested in solving my itch.

    And I just happened to open-source it, because I felt it might be useful for other people.

    Edith: So Paul, I hate to say this, but I agree with you.

    Paul: No, not again.

    Edith: Not again.  No, I mean, I think what you said is exactly right, that you said, I did this for this reason, and I want somebody else to take it over now.

    I think where open source runs into issues where people are like, it’s only this purpose and you can’t do anything with it besides my original thing.

    Paul: So you think that’s a good thing, or you think that’s a problem? Or it causes problems in the community?

    Edith: I think open source is so misused for so many different purposes that it’s hard to even say. Like, so, in Europe, people say open source is good because it means free software.

    Paul: Libre or gratuit?

    Edith: I’m sorry?

    Paul: Free as in beer or free as in, what’s the other side of that?

    Edith: There’s no such thing as a free lunch, is that what you’re talking about?

    Paul: No no no, like people say open source software, it’s like free-as-in-beer software, as in software that has no cost vs. software that has no price tag vs. software that is, they use the French word libre for—

    Edith: Oh yeah, that’s what I was talking about. So I remember I went to Europe in, I think it was like 2004 or 2005, and they just had this big push for open source, ’cause they thought it meant free.

    Paul: Oh, this is like the German government and that sort of thing.

    Edith:  So literally like, so I went to this customer and they’re like, ‘So it doesn’t cost you any more to give us an extra copy.’ So why shouldn’t just everything be free?

    Paul: Oh, right, yeah, well.

    Edith: And so I think open source has been misused by many businesses who think of it as just like, hey, we get all these developers working for free.

    Paul: So I think that the main advantage to business of open source is not that it’s free, but it’s that it’s low-risk.

    Edith: Low-risk?

    Paul: Lower risk. So it removes—

    Edith: Ah, I would argue—

    Paul: Well, so, what I mean is that it removes a certain element of risk from it. So there’s, Joss Belsky wrote a blog post about how you shouldn’t use, I think it was a database that you didn’t have the source code to.

    Edith: Well, so I’ll argue the reverse, and I think open source is sometimes more risky. ‘Cause I talk to people now who are like, nobody’s maintained this open source project that I depend on.

    Paul: Right, right, right. But that vs. a closed-source version.

    Edith: Well, so the opposite risk is that you end up that you are responsible for this open source project, and you’re like, well, actually, I just wanted to use this thing. I didn’t want to do the maintaining.

    Paul: So absolutely, you don’t wanna end up the only person that, well, it’s actually fine to end up the only person which is using this thing.

    Edith: Well, no, it’s not—

    Paul: It’s not ideal.

    Edith: It’s that many people are using it, but nobody is paying money, and then you’re stuck.

    Paul: Because no one is maintaining it.

    Edith: No one is maintaining it, and you depend on it.

    Paul: Right, so, but imagine that no one maintained it and all that you could get was like a binary block. That’s the real problem. The company goes out of business. You have random database version seven, the binary, that only runs on 32-bit, that you’re maintaining, like, companies are literally maintaining COBOL—

    Fortran things from 30 years ago. It sits in a machine, you can’t touch it, you can’t fix it, you can’t apply security updates to it.

    Edith: Yeah, that’s fair. So Tim Chou, who is, he’s an awesome guy, he was early Oracle, he said that people don’t buy, when they buy software, they’re not buying it what it is today.

    Paul: Right.

    Edith: ‘Cause basically, anything that you’re buying today, you could just hire 20 engineers that could build it. What they’re buying—

    Paul: Yeah, you could just copy it.

    Edith: Yeah, what they’re buying is the promise of stability, that it will continue to be maintained. That there is a vision. And that’s actually, I went over to a big customer who won’t let us use their name yet, but they said they love LaunchDarkly because we thought every day about feature flags.

    Paul: Right, right, exactly. This is why people buy SAS, because, because people think all day about CI, about QA as a service, about queues, or whatever it is from various Heavybit companies that I just shilled for, they’re, you can probably figure out yourself which ones they are by looking at the homepage.

    Edith: Well, and Paul, by the way, thank you very much for wearing your LaunchDarkly t-shirt today. It looks very handsome.

    Paul: The, I definitely agree that the benefit of the thing is that you use it every day. But you can do that with open source as well. You can have open source software that, so, for example, our front end is open-sourced.

    Edith: Oh, I’m absolutely a fan of open source. Like, we use open source every day. Our client libraries are open source. I’m not saying that it’s, I don’t think it’s a religious thing vs. open source vs. not, I just think that many times people assume open source is something that it’s not.

    Paul: Yeah, fair enough, fair enough.

    Edith: I mean that if you have open source, that just means that developers spring out of the ground willing and happy to update your software every day.

    Paul: A mixed blessing, definitely. If it even happens.

    Edith: Like, I remember, so I remember when Sun, so I used to work on a, I used to be at Epicentric, and we had a portal server, and we were killing Sun. Absolutely killing Sun. And this was in 2003, so what they said is, ‘Oh, we’re gonna make our portal server open source.’

    Paul: Right.

    Edith: And so somehow this whole thing, we’re like, everything will be open source, and like, developers will spring out of the ground, and it’s like, well, no, developers, they don’t wake up every day like, ‘I’m gonna work on your thing.’

    Paul: Right, right. They will if it’s sexy or exciting. There’s lots of developers who are showing up at Mozilla just to contribute. And in fact, they’ve fought through incredibly large hurdles to contribute to Mozilla’s software.

    Edith: Well, that’s fascinating, ’cause I see so often the reverse, where people, so what do you think about Mozilla made people want to contribute?

    Paul: I think the fact that it was an open-source-first company, and that it was a nonprofit, and it was all about the free and the open web and that sort of thing. Also, people had had very good experiences with Firefox for, like, a decade at this point by the time I was there. Well, maybe seven years, but, you know.

    Edith: It’s fascinating, ’cause I went to dinner last night with the CEO of MySQL, and I don’t know if I should say this, but he said—

    Paul: This is Marten Mickos?

    Edith: Yep. So he gave a talk—

    Paul: He has a new startup?

    Edith: Called HackerOne?

    Paul: Oh, he joined HackerOne.

    Edith: Yeah, he’s the new CEO. So the dinner was kind of like a debutante ball for him.

    Paul: Nice, nice.

    Edith: So what he said was that somebody had asked him flat out how much of MySQL was actually from contributors. He said 1 percent.

    Paul: Seems about right.

    Edith: He said the open source was more just a branding thing. Like it wasn’t actually like a productivity thing. It was more just a branding thing.

    Paul: So you have this in Mozilla as well, that any time that there’s a really avid contributor, that Mozilla would hire them.

    Edith: Yeah, that’s what he said.

    Paul: Right, and if Mozilla didn’t hire them, very often they stopped being a contributor. Like, it was a real sort of a double-edged sword.

    So someone who did it for six months, they were doing it for the love of the game. They weren’t doing it to be hired at, at least I think for the most part, they weren’t doing it to be hired at Mozilla. And then six months in, someone’s like, well, you know, you’re doing a really great job, wouldn’t you love to work on this full-time? They’re like, yeah, that’s so exciting. They’re flown out for an interview. And then it’s like, you know, actually, I don’t think so.

    Edith: Wait, they didn’t wanna work for Mozilla, or Mozilla didn’t wanna hire them?

    Paul: Mozilla didn’t wanna hire them. I mean, this isn’t a particular case. But you know, some subset of people, Mozilla didn’t wanna hire. And then what happens? Well, you kind of don’t wanna keep contributing to the company that just told you to f*ck off.

    Edith: Yeah, I mean, it’s so true. I mean, I think there’s a very human thing about volunteering.

    Paul: Right, right.

    Edith: I volunteer, I was the moderator for a long time for the Lean Startup Circle, and somebody new came on, and they’re like, yeah, I’d love to raise some money so that we can get paid, and I’m like, they cannot afford to pay me.

    Like, I don’t do this to get paid, ’cause—

    Paul: Yeah, it’s like, it’s an altruism that people are trying to do.

    Edith: Yeah, I was contributing like 10 to 15 hours a week, and at my going rate, it’s not, it’s like, I’m not doing this to get paid, I’m doing it ’cause I like giving back.

    Paul: So there’s this thing that we were talking about earlier, about the current discussion around people getting paid in open source.

    Edith: Yeah, and I think a lot of what has fueled open source is the whole 20 percent time. Like, that Google will give their engineers 20 percent for open source, and I know that other big companies, like Twitter and so on, do that too. So these people are, in essence, getting paid.

    Paul: Right, right, there’s a, in most companies, in most startups at least, there’s an unspoken rule that you’re expected to contribute to the libraries that are in your application, and that sort of thing. If you’re spending time upstreaming paths that you’ve deployed, that’s work.

    Edith: Yeah, so I think we’re agreeing, which, I thought we agreed that we were not gonna agree. It’s funny, ’cause I’m Twitter friends with a guy I’ve never met, who’s, he started Jango, and right now what he tweets a lot about is—

    Paul: This is Jacob Kaplan-Moss?

    Edith: Yeah.

    Paul: Okay.

    Edith: That people are working their nights and weekends, and contributing to open source, and then just, people are kind of, for lack of a better word, jerks. People are like—

    Paul: People are jerks.

    Edith: Yeah, like—

    Paul: Even when you’re paid, people are jerks.

    Edith: But it’s more aggravating.

    Paul: Right, right, because you’re doing it for free, or whatever.

    Edith: Yeah, and so people will, like, they’ll be like, ‘I have this pull request and it’s been open for two hours. Why haven’t you reviewed it?’

    Paul: Right, right. I mean, we get the exact same thing. We get customers who email us, or more often it’s the people who get, who are using the free version, who send us a support request and then, 30 minutes later they’re on Twitter going, I can’t believe Circle hasn’t solved my problem yet. It’s been 30 minutes.

    Edith: What do you think drives that?

    Paul: There’s a pattern across the population in general, that people are jerks.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of Circle CI, and Edith Harvaugh of LaunchDarkly.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • Open Source software license and security management with WhiteSource

    Open Source software license and security management with WhiteSource

    With the growing speed and availability of open source components, it becomes easy to add features and integration of software with other components which makes software development easier. But there are a few points to be remembered while using any open source component.

    • Security vulnerabilities
    • Licensing risks of open source component.
    • Outdated open source components.

    The above are a very important part of the the software development life cycle to eliminate any discrepancies related to security or legal issues.

    WhiteSource provides one fine platform to solve such issues without putting in much effort by developers, They can concentrate on core development instead of spending time on finding these issues which can be easily handled by WhiteSource.

    WhiteSource is an open source management solution which does

    • Open source licensing and compliance management
    • Open Source security vulnerabilities alerts and management
    • Executive dashboards, policy enforcement, and reporting

    WhiteSource checks your software and generates open source inventory report, including detail from open source. The WhiteSource gathers information on open source components and they keep the inventory updated.

    WhiteSource covers almost all commonly used languages and provide detail report, This tool can track all your open source components used knowing or unknowingly within your software.

    It sends alerts for any potential issue observed in open source component used by software. It does check for outdated/expired component, It provides detail on security issues found in any opensource component.

    WhiteSource does not store (keep track of) any software component which is not open source, this means it’s safe and your code will not be touched.

    CI tool Integration (Jenkins)

    The best part of the tool is that it has plugin available for the widely used CI tool – Jenkins.

    WhiteSource Jenkins plugin is the best and easy way to integrate with WhiteSource to run the checks during build and integration phase in an automatic way.

    Its very easy to use, not even just for Maven project also for free style projects.

    Easy to configure and use within project. Only token needs to be added and plugin automatically takes care of everything.

    Multiple options are available at job level to define the project and modules to be included.

    Logs are pretty descriptive when plugin start processing in Jenkins job.

    WhiteSource basic principle talks about

    Managing the component vs governing the component.

    Developers can leave the managing part up to WhiteSource and only concentrate on the issue of governance at the component level in software.

  • 9 Open Source DevOps Tools We Love

    9 Open Source DevOps Tools We Love

    While we have been building our SaaS company, we have been investing much time in researching and deciding which tools to include in our DevOps toolkit. We’ve based these decisions on our years of experience in the IT industry, dealing with infrastructure for the most part. From building a petabyte-scale, data analytics infrastructure, our architecture, tools, and processes have become key components of our technology and operations. We’ve taken great care in selecting, benchmarking and constantly improving our tool selection.

    As a company that is building a solution on top of an open-source stack (ELK), our team is heavily involved in the open source community, contributing to multiple projects such as Camel and Kafka while customizing tools to fit our needs. The vast majority of tools we use internally are open-source ones. By sharing the toolset that we’ve collected and honed over time, we hope to foster a discussion within the DevOps community on what further improvements can be made.

    With that, we welcome you to enjoy browsing through the following list that we’ve created. Some tools you may have known about for years, while others may be new. Of course, we invite any and all feedback, especially from those of you who are using alternatives!

    Must-Have DevOps Tools

    1. Nagios (& Icinga) 

    nagiosInfrastructure monitoring is a field that has so many solutions… from Zabbix to Nagios to dozens of other open-source tools. Despite the fact that there are now much newer kids on the block, Nagios is a veteran monitoring solution that is highly effective because of the large community of contributors who create plugins for the tool. Nagios does not include all the abilities that we had wanted around the automatic discovery of new instances and services, so we had to work around these issues with the community’s plugins. Fortunately, it wasn’t too hard, and Nagios works great.

    We also looked into Icinga, which was originally created as a fork of Nagios. Its creators aim to take Nagois to the next level with new features and a modern user experience. There is a debate within the open source community about the merits of Nagios and its stepchild, but for now we are continuing to use Nagios and are satisfied with its scale and performance. The switch to newer technology, such as Icinga, may be appropriate in the future as we progress.

    2. Monit

    monitSometimes the simplest tools are the most useful, as proven by the simple watchdog Monit. Its role is to ensure that any given process on a machine is up and running appropriately. For example, a failure occurs in Apache, Monit will help to restart the Apache process. It is very easy to setup and configure and is especially useful for multi-service architecture with hundreds of micro-services. If you are using Monit, make sure to monitor the restarts that it executes in order to surface problems and implement solutions (rather than just restarting and ignoring the failure). You can do this by monitoring Monit’s log files and ensuring that you are alerted to every restart.

    3. ELK – Elasticsearch, Logstash, Kibana – via Logz.io

    logz.ioThe ELK Stack is the most common log analytics solution in the modern IT world. It collects logs from all services, applications, networks, tools, servers, and more in an environment into a single, centralized location for processing and analysis. We use it for analytical purposes (e.g., to troubleshoot problems, monitor services, and reduce the time it takes to solve operational issues). Another use for this tool is for security and auditing (e.g., to monitor changes in security groups and changes in permissions). After receiving alerts on these issues, it is easy to act on unauthorized users and activities. We also use ELK for business intelligence, such as monitoring our users and their behavior. You can set up your own ELK or buy it as-a-service. We’ve written a guide for the community on using ELK to monitor your application performance.

    Disclaimer: Logz.io is our ELK-as-a-service that we use in our own environment. You can say that we eat our own dog food.

    4. Consul.io

    consulConsul is a great fit for service discovery and configuration in modern, elastic applications that are built from microservices. The open-source tool makes use of the latest technology in providing internal DNS names for services. It acts as a kind of broker to help you sign and register names, enabling you to access service names instead of specific machines. If, for example, you have a cluster of multiple machines, you can simply register them as a single entity under Consul and access the cluster easily. We praise this tool for its efficiency, although we still feel there is more that can be done with it. If you also use it, it would be great to hear about your own use case.

    5. Jenkins

    jenkinsEveryone knows Jenkins, right? It’s not the fastest or the fanciest, but it’s really easy to start to use and it has a great ecosystem of plugins and add-ons. It is also optimized for easy customization. We have configured Jenkins to build code, create Docker containers (see the next item), run tons of tests, and push to staging/production. It’s a great tool, but there are some issues regarding scaling and performance (which isn’t so unusual). We’ve explored other cool solutions such as Travis and CircleCI, which are both hosted solutions that don’t require any maintenance on our side. For now, however, since we’ve already invested in Jenkins, we’ll continue with it.

    6. Docker

    dockerEverything that can be said about how Docker is transforming IT environments has already been said. It’s great…life changing, even — (although we’re still experiencing some challenges with it). We use Docker in production for most services. It eases configuration management, control issues, and scaling by allowing containers to be moved from one place to another.

    We have developed our SaaS solution with a twelve-layer pipeline of data processing. Together with Jenkins and Docker, we have been able to run a full pipeline across all layers on a single Mac. It would be wrong to say that there aren’t any complications with Docker, as even small containers can take a significant amount of time to build. However, we want to ensure that our developers are as satisfied as possible and enable them to work rapidly. With all of the management involved in storage, security, networking — and everything surrounding containers — this can be a challenge.

    We see Docker progressing and look forward to welcoming the company’s new management and orchestration solutions. For those who might be having issues with Docker, we’ve also compiled a list of challenges and solutions when migrating to Docker.

    7. Ansible

    ansibleAgain, simplicity is key. Ansible is a configuration management tool that is similar to Puppet and Chef.  Personally, we found those two to have more overhead and complexity to our use case– so we decided to go with Ansible instead. We know that Puppet and Chef probably have a richer feature set, but simplicity was our desired KPI here. We see some tradeoffs between configuration management using Ansible and the option to simply kill and spin new application instances using a Docker container. With Docker, we almost never upgrade machines but opt to spin new machines instead, which reduces the need to upgrade our EC2 cloud instances. Ansible is used mostly for deployment configuration mostly. We use it to push changes and re-configure newly-deployed machines. In addition, its ecosystem is great, with an easy option to write custom applications.

    8. Collectd/Collectl

    Collectd/l are nifty little tools that gather and store statistics about the system on which they run and are much more flexible than other tools. They allow users to measure the values of multiple system metrics and unlike other log collection tools that are designed to measure specific system parameters, Collectd/l can monitor different parameters in parallel. We use these two tools to measure customer performance parameters and ship them to our ELK-as-a-Service platform. We’ve specifically wrapped a Collectl agent in a Docker container and push it with Ansible to all of our servers. It collects information every couple of seconds and then ships it to ELK to allow us to run reports and send alerts. If you’d like to see a specific example of how we do this process in our environment and how others can do the same, we’ve created a guide for everyone.

    9. Git (GitHub)

    githubGit was created 10 years ago following the Linux community’s need for SCM (Source Control Management) software that could support distributed systems. Git is probably the most common source management tool available today. After running Git internally for a short period of time, we realized that we were better suited with GitHub. In addition to its great forking and pull request features, GitHub also has plugins that can connect with Jenkins to facilitate integration and deployment. I assume that mentioning Git to modern IT teams is not breaking news, but I decided to add to it to the list due to its great value to us.

    Other picks?

    The modern DevOps world is full of outstanding and unique open source tools—it’s a jungle out there. We found the tools listed here to be the best in breed and think they should be included in every DevOps engineer’s shortlist.

    Or am I wrong? What open source DevOps tools are in your toolkit? I’d love to hear your own recommendations and experiences in the comments below.

  • Commercial vendors gift to DevOps

    Commercial vendors gift to DevOps

    Open source solutions are usually brought to life because of someone’s yearning to solve a problem and share the solution with the rest of the world. Or perhaps when inventors are merely trying to get more eyes on the code for help with hidden bugs and fixes.

    (more…)

  • DevOps: It’s All Open Source’s Fault

    DevOps: It’s All Open Source’s Fault

    Blame DevOps on open source. After all, it was the open source movement that gave developers the idea that they were free to code, free to build applications and other software for their companies without needing prior approval. It was open source that crowned developers kingmakers. It was open source that gave developers the freedom to not only write code, but also manage it.

    It was open source that gave us DevOps.

    DevOps: Freedom To Get Stuff Done

    Traditionally, IT operations has been a specialized task, an exclusive one that largely controlled the code allowed to run within the enterprise. Given the fact that all software had to be purchased before it could be run, there was little risk of developers or anyone else introducing rogue software into the hallowed halls of the data center.

    Enterprises were able to enforce policies because all software had to start with Purchasing and end with Operations. There simply was no other viable way to get software into the enterprise.

    That is, until open source.

    Open source ruined the neat and tidy relationship between Purchasing and Operations. Suddenly software didn’t have to be purchased. It could simply be downloaded. As pressures mounted on lines of business to accelerate their pace of innovation, developers stepped up: “I have an open source project for that.”

    Operations fought back, trying to crimp open source adoption. They failed.

    There simply was too much pressure on developers to get stuff done. Once Amazon gave developers a home outside the data center to deploy their code, the battle was over and a new model for building and managing code was born. Dubbed DevOps, the movement gave developers end-to-end control over their creations.

    Since 2011, DevOps adoption has increased 26%, according to a 2013 survey by Puppet Labs, resulting in 50% application failures and the ability to recover up to 12X faster. The rise in DevOps also translates into the ability to ship code 30X faster. All of which is expressed in a separate CA Technologies survey of senior IT decision-makers, which found that improvements to the customer experience is by far the biggest reason enterprises are embracing DevOps. 

    Given these improvements, there’s no way we’re going back to Operations-focused IT.

    Blessing Or Curse?

    All of which is not to suggest that DevOps has been an unalloyed blessing. In my experience, developers are often unprepared or unwilling to take on the burden of managing their applications in production. Trained for years on the idea that they could build an application and dump it on Operations to manage, developers are discovering that the “Ops” in DevOps is real, and sometimes really a pain.

    For example, one big problem is that production environments and dev/test environments are typically very different. An application that works flawlessly in a small-scale test environment may miss a whole class of errors that only occur in the presence of the more complex production environment (e.g. problems that only occur when application changes interact with a complex, zoned network topology).

    The reality is that over-optimizing for Ops (safe, secure and slow) or Development (Fast, opportunistic, iterative) proves sub-optimal. A balance is needed.

    But that balance increasingly skews toward upgrading Ops to meet developer demands. Indeed, developers are writing a host of infrastructure and tools to help other developers better manage their creations. Often such tooling is SaaS-based, and a significant percentage is open source, allowing developers to continue to skirt the controls of Operations while maintaining flexibility. Companies like Boundary, Puppet Labs, SumoLogic and more have arisen to make operations easier for DevOps professionals, among others.

    My own company, MongoDB, develops the most popular database for modern applications. Universally lauded for how easy MongoDB makes development, our tools for facilitating the operation of MongoDB at scale have lagged. Over the past two years we have been building out monitoring, back-up and other management tooling for MongoDB. It’s simply not enough anymore to forget the “Ops” in DevOps.

    No Putting The DevOps Genie Back In The Bottle

    I expect we’ll see this trend continue. The power of developers will increase in large part because the industry will better enable them to manage their applications, or to work more seamlessly with their Ops peers. But the era of Ops dictating to developers is over.

    Blame open source.