devopsy.kinsta.cloud

Category: DevOps Toolbox

9 Open Source DevOps Tools We Love

While we have been building our SaaS company, we have been investing much time in researching and deciding which tools to include in our DevOps toolkit. We’ve based these decisions on our years of experience in the IT industry, dealing with infrastructure for the most part. From building a petabyte-scale, data analytics infrastructure, our architecture, tools, and processes have become key components of our technology and operations. We’ve taken great care in selecting, benchmarking and constantly improving our tool selection.

As a company that is building a solution on top of an open-source stack (ELK), our team is heavily involved in the open source community, contributing to multiple projects such as Camel and Kafka while customizing tools to fit our needs. The vast majority of tools we use internally are open-source ones. By sharing the toolset that we’ve collected and honed over time, we hope to foster a discussion within the DevOps community on what further improvements can be made.

With that, we welcome you to enjoy browsing through the following list that we’ve created. Some tools you may have known about for years, while others may be new. Of course, we invite any and all feedback, especially from those of you who are using alternatives!

Must-Have DevOps Tools

1. Nagios (& Icinga)

Infrastructure monitoring is a field that has so many solutions… from Zabbix to Nagios to dozens of other open-source tools. Despite the fact that there are now much newer kids on the block, Nagios is a veteran monitoring solution that is highly effective because of the large community of contributors who create plugins for the tool. Nagios does not include all the abilities that we had wanted around the automatic discovery of new instances and services, so we had to work around these issues with the community’s plugins. Fortunately, it wasn’t too hard, and Nagios works great.

We also looked into Icinga, which was originally created as a fork of Nagios. Its creators aim to take Nagois to the next level with new features and a modern user experience. There is a debate within the open source community about the merits of Nagios and its stepchild, but for now we are continuing to use Nagios and are satisfied with its scale and performance. The switch to newer technology, such as Icinga, may be appropriate in the future as we progress.

2. Monit

Sometimes the simplest tools are the most useful, as proven by the simple watchdog Monit. Its role is to ensure that any given process on a machine is up and running appropriately. For example, a failure occurs in Apache, Monit will help to restart the Apache process. It is very easy to setup and configure and is especially useful for multi-service architecture with hundreds of micro-services. If you are using Monit, make sure to monitor the restarts that it executes in order to surface problems and implement solutions (rather than just restarting and ignoring the failure). You can do this by monitoring Monit’s log files and ensuring that you are alerted to every restart.

3. ELK – Elasticsearch, Logstash, Kibana – via Logz.io

The ELK Stack is the most common log analytics solution in the modern IT world. It collects logs from all services, applications, networks, tools, servers, and more in an environment into a single, centralized location for processing and analysis. We use it for analytical purposes (e.g., to troubleshoot problems, monitor services, and reduce the time it takes to solve operational issues). Another use for this tool is for security and auditing (e.g., to monitor changes in security groups and changes in permissions). After receiving alerts on these issues, it is easy to act on unauthorized users and activities. We also use ELK for business intelligence, such as monitoring our users and their behavior. You can set up your own ELK or buy it as-a-service. We’ve written a guide for the community on using ELK to monitor your application performance.

Disclaimer: Logz.io is our ELK-as-a-service that we use in our own environment. You can say that we eat our own dog food.

4. Consul.io

Consul is a great fit for service discovery and configuration in modern, elastic applications that are built from microservices. The open-source tool makes use of the latest technology in providing internal DNS names for services. It acts as a kind of broker to help you sign and register names, enabling you to access service names instead of specific machines. If, for example, you have a cluster of multiple machines, you can simply register them as a single entity under Consul and access the cluster easily. We praise this tool for its efficiency, although we still feel there is more that can be done with it. If you also use it, it would be great to hear about your own use case.

5. Jenkins

Everyone knows Jenkins, right? It’s not the fastest or the fanciest, but it’s really easy to start to use and it has a great ecosystem of plugins and add-ons. It is also optimized for easy customization. We have configured Jenkins to build code, create Docker containers (see the next item), run tons of tests, and push to staging/production. It’s a great tool, but there are some issues regarding scaling and performance (which isn’t so unusual). We’ve explored other cool solutions such as Travis and CircleCI, which are both hosted solutions that don’t require any maintenance on our side. For now, however, since we’ve already invested in Jenkins, we’ll continue with it.

6. Docker

Everything that can be said about how Docker is transforming IT environments has already been said. It’s great…life changing, even — (although we’re still experiencing some challenges with it). We use Docker in production for most services. It eases configuration management, control issues, and scaling by allowing containers to be moved from one place to another.

We have developed our SaaS solution with a twelve-layer pipeline of data processing. Together with Jenkins and Docker, we have been able to run a full pipeline across all layers on a single Mac. It would be wrong to say that there aren’t any complications with Docker, as even small containers can take a significant amount of time to build. However, we want to ensure that our developers are as satisfied as possible and enable them to work rapidly. With all of the management involved in storage, security, networking — and everything surrounding containers — this can be a challenge.

We see Docker progressing and look forward to welcoming the company’s new management and orchestration solutions. For those who might be having issues with Docker, we’ve also compiled a list of challenges and solutions when migrating to Docker.

7. Ansible

Again, simplicity is key. Ansible is a configuration management tool that is similar to Puppet and Chef. Personally, we found those two to have more overhead and complexity to our use case– so we decided to go with Ansible instead. We know that Puppet and Chef probably have a richer feature set, but simplicity was our desired KPI here. We see some tradeoffs between configuration management using Ansible and the option to simply kill and spin new application instances using a Docker container. With Docker, we almost never upgrade machines but opt to spin new machines instead, which reduces the need to upgrade our EC2 cloud instances. Ansible is used mostly for deployment configuration mostly. We use it to push changes and re-configure newly-deployed machines. In addition, its ecosystem is great, with an easy option to write custom applications.

8. Collectd/Collectl

Collectd/l are nifty little tools that gather and store statistics about the system on which they run and are much more flexible than other tools. They allow users to measure the values of multiple system metrics and unlike other log collection tools that are designed to measure specific system parameters, Collectd/l can monitor different parameters in parallel. We use these two tools to measure customer performance parameters and ship them to our ELK-as-a-Service platform. We’ve specifically wrapped a Collectl agent in a Docker container and push it with Ansible to all of our servers. It collects information every couple of seconds and then ships it to ELK to allow us to run reports and send alerts. If you’d like to see a specific example of how we do this process in our environment and how others can do the same, we’ve created a guide for everyone.

9. Git (GitHub)

Git was created 10 years ago following the Linux community’s need for SCM (Source Control Management) software that could support distributed systems. Git is probably the most common source management tool available today. After running Git internally for a short period of time, we realized that we were better suited with GitHub. In addition to its great forking and pull request features, GitHub also has plugins that can connect with Jenkins to facilitate integration and deployment. I assume that mentioning Git to modern IT teams is not breaking news, but I decided to add to it to the list due to its great value to us.

Other picks?

The modern DevOps world is full of outstanding and unique open source tools—it’s a jungle out there. We found the tools listed here to be the best in breed and think they should be included in every DevOps engineer’s shortlist.

Or am I wrong? What open source DevOps tools are in your toolkit? I’d love to hear your own recommendations and experiences in the comments below.

August 7, 2015
Scaling Lean and Agile for the enterprise: All hands on keyboard!!!
Lean and Agile methodologies are nothing new, but in the last few years we have seen the emergence of several frameworks combining elements of both. The IBM DevOps framework is one clear example. Combining principles of Lean and Agile into a single framework enables scaling of the core benefits to the entire organization, not just pockets of experts dispersed across functional silos. The goal?
- faster release cycles, increased responsiveness, reduction of waste, lower cost base
- increased software quality, customer-centric feedback loops, cross-discipline collaboration
That said, these process transformation initiatives too often remain confined within the Engineering or IT side of the house. And given that more and more large scale software projects (mobile and web) are driven and managed out of the Marketing or LOB organizations, we need to get to a place where Engineering/IT and the Business share a common vision and framework for continuous software delivery. This blog will discuss key areas where the Business needs to engage to truly scale these benefits to the organization level.

Let’s take a look at the key principles behind these frameworks and highlight those which mostly impact the behavior of analysts, project/program managers and LoB executives on the business side:
- Optimize the whole: Development teams cannot get it right by themselves. The business must be actively engaged to ensure customer/user needs are properly understood, in-turn ensuring the intended outcome of the application/system is well communicated and understood, and eventually the overall effort is continuously steering toward the right business outcomes
- Eliminate waste: Identifying/removing wait time to reduce overall turn around and cycle times (active participation during needs/requirements analysis, timely inputs into work prioritization, immediate feedback on early prototyping/release). Discover the hidden factory!!!
- Empower people: Let the real user directly represent their needs, and build interaction systems and processes that make it easy for those with the needs to engage across the lifecycle
The most common symptom that engineering and the business are not in sync is a constant flow of “management urgencies” that are forced into the current sprint or iteration. These “must do now” urgencies are a massive source of distraction for the teams executing the work on both the business and the engineering side. Directly responsible for substantial loss in productivity across the whole chain, as well as a prominent source of friction in relationships between business and IT, these distractions and interruptions can be avoided with the right level of orchestration and collaboration.

So which are the crucial touch-points where the right level of engagement from the business can generate the most impact?
- Early definition of business needs: This is not about producing lengthy system requirements documents, it’s about interaction with software architects and designers early and often to properly describe the needs/expectations of the target users/customers, it’s about making sure the overall context is defined and understood. There will be several opportunities to get into detailed requirements as the work unfolds.
- Early involvement into definition of success/acceptance criteria: When I was a BA, little humbled me as much as thinking I had done a good job describing a business requirement only to then freeze when I was trying to define objective criteria that would serve to develop user acceptance test plans. Forcing the definition of acceptance criteria at the time requirements are defined often surfaces areas of the needs that could be better defined or drives important exchanges between engineering and the business.
- Early and on-going engagement into prioritization: Gone are the days when setting up priorities was the result of a monthly or quarterly exercise; customer/user behaviors change continuously, so should your alignment to their needs. Prioritizing monthly or quarterly is like driving your car on the freeway in a pitch dark night and turning on your headlights every 2 mins or so to get an idea of where you heading… a lot can happen in 2 mins driving at 100km/hour. Prioritizing is also no longer an exercise in faith/best-guesses/loud voice in the room…. this new world is about constantly adjusting based on validated learning; running A/B testing, driving early prototyping and experimentation, monitoring application usage, social media analytics all serve to drive the right backlogs/priorities of teams across the software delivery lifecycle.
So where to start?

This transformation is a journey, not a one-time call to action, different paths can be taken depending on your organization’s goals, culture and priorities. We’ve helped hundreds of organizations over the years plan their way forward, helping them structure teams and investments while respecting their constraints and objectives. Be bold, take this on-line DevOps Practice Self-Assessment! Or visit our blog on IBM’s support for the Scaled Agile Framework (SAFe).

Also, do visit us at Agile 2015 to experience IBM DevOps solutions for collaborative development, continuous testing and continuous release and deployment.

About the Author/Roger LeBlanc

Roger is the Strategic Offering Lead, DevOps Application Platform driving the Enterprise Scaled Agile solution. He joined IBM following the acquisition of Systemcorp ALG, serving as Managing Director, Professional Services and has since been involved in ensuring a smooth transformational journey for the clients. He has a credible perspective and proven track record for meeting executive stakeholders expectations and goals. Roger has contributed to revenue growth and customer successes as well as provided strategic oversight and guidance of customer engagements.

Connect with Roger on
August 5, 2015
How ADLM Gobbles Up DevOps

The Next Evolution of Software Delivery Automation

In the 1980’s and 90’s, the business software landscape was dominated by a diverse list of cutting-edge companies such as Best Software, i2, Brock Control Systems, Mapics, Ross Systems, Infinium, FBO Systems, Manugistics and MSA (of course I could go on and on). Now long gone, these and hundreds more really great companies have been gobbled up or rendered obsolete by the rising class of ERP giants. In this article I’ll explain why history will repeat itself leading to the extinction of most DevOps tools as leading ADLM platforms continue to assert their dominance across the diverse software development and delivery automation ecosystem.

History Informs Our Future and The Evolution of ERP

You already know that for the past twenty years, most companies have leveraged some form of ERP system to manage virtually every core business processes. One benefit of this tightly integrated solution is a powerful inter-functional data flow that enables corporate agility and provides the highest level of visibility. This super-integrated architectural model has become the standard adopted by virtually every enterprise around the globe. What you may not know is today’s “ERP model” is the result of four distinct evolutionary generations that I believe help predict the next major evolution of automated software delivery.

Phase 1 – Automation
Enterprise Information Systems (EIS) – In the 1960’s, early automation systems were developed to support important individual business functions such as general ledger, inventory management, billing, payroll, etc. These systems were architected completely independently of each other and added little value to the enterprise beyond their narrow scope.

Phase 2 – Core Data Model
Manufacturing Resource Planning (MRP) – Then in the 1970’s, the idea of a master production schedule was devised so that a few of these isolated systems could gain greater visibility into future inventory and product requirements of the organization. The big idea behind the master production schedule was the creation of an open data model that could be leveraged by other systems impacted by the manufacturing production schedule.

Phase 3 – Expansion
Manufacturing Resource Planning II (MRP II) – In the 1980’s, software vendors began to build and sell off-the-shelf packages that promised “best of breed” process design. These solutions provided tightly integrated versions of the key manufacturing processes required to produce products.

Phase 4 – Business Process Domination
Enterprise Resource Planning (ERP) – Finally, in the 1990’s, business software vendors expanded well beyond the manufacturing scope by creating highly-integrated solutions that now cover just about every standard business function imaginable. These systems leverage a unified data model to dramatically improve visibility, consistency, accuracy and planning capabilities enterprise-wide.

What Does ERP Have To Do With Automated Software Delivery?

The evolution of ERP has taught us how natural pressures forced the creation of a unified and comprehensive “business data model” spanning the entire enterprise. Those software vendors with enough influence to dictate that data model were the ultimate winners in the ERP space.

In the very first generation of ERP (EIS), software was leveraged to deliver a high degree of automation to many business processes that were previously manual. Over time, it became clear that these newly automated processes were interconnected and the evolution towards a tightly integrated and unified data model was underway. The business objective that fueled each successive generation outlined above was the need to design more efficient business processes that increased organizational visibility and agility.

As Marc Andreessen famously said in 2011, “…more and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defense.” (Why Software Is Eating The World). That statement resonates even stronger four years later. Today, virtually every corporate organization is seeing the familiar pressure to deliver software more efficiently and reliably. If history is indeed our guide, any highly fragmented and/or isolated process required to deliver incremental software change will face mounting pressure to be merged into an integrated end-to-end enterprise-grade platform that can deliver improved cross-functional visibility with even greater efficiency.

Why Application Delivery Lifecycle Management Will Win

In my view, there are really only two broad solution categories in the realm of automated software design and delivery – Application Development Lifecycle Management (ADLM) and the catch-all term DevOps (which I’ve hijacked here to describe any other type of process automation that assists the software delivery process).

DevOps tools are often narrow point solutions that have sprouted from open source projects, in-house development or commercial vendors. Organizations rely heavily upon a diverse collection of these DevOps tools to help document, validate and automate a steady flow of software change along its path to end-users.

Here’s the problem DevOps tools are beginning to face: Like the EIS systems of the 60’s, fragmented DevOps tools have little or no visibility into the overall end-to-end process; however, they do generate lots of important data that is often locked away and isolated. This isolation creates a clear barrier to efficiency, visibility and agility across the software delivery process. Because of the limited function each individual DevOps tool performs, none have the gravitational pull required to define the larger data model. As comprehensive enterprise software delivery platforms emerge elsewhere, standard DevOps tools will face ever-increasing pressure to fold inside them.

Currently, Application Development Lifecycle Management solutions provide a platform to manage development projects, team resources and all manner of development activities. ADLM platforms also contain the “master production schedule” for every development initiative – past, present and future. The data contained within ADLM is now at the core of a quickly emerging software delivery data model and leading ADLM vendors are expanding their footprint pwell beyond traditional use cases. When it comes to ownership of this software delivery data model, I see no other solution category across the entire ecosystem with enough enterprise clout to pose a serious challenge to leading ADLM vendors.

Unified Software Delivery Platforms Are Already Emerging

Each of the five current ADLM leaders (according to Gartner’s most recent Magic Quadrant) are now racing to bring to market an enterprise software delivery platform that integrates many key DevOps capabilities.

Here’s my two cents on each…

The Giants in the space – IBM and Microsoft both have plenty of muscle and IP today. Clearly both are moving down the path toward a comprehensive software delivery platform. IBM acquired DevOps vender urban{code} several years ago and is hard at work building their developerWorks platform. Seemingly everyday, Microsoft is adding some kind of DevOps capability into its Visual Studio product suite. Still, I don’t see either vendor gaining much traction outside of their traditional (albeit very large) customer base. Perhaps more importantly, neither seem to have bonafide credentials within the super-influential agile development community and I believe this kind of street-cred (at least for now) is a must-have to dominate this space.

Atlassian does enjoy wide support among the agile community and no doubt has the broadest adoption footprint of any of the current ADLM leaders. Atlassian is in a strong position to mount a serious threat. However, Atlassian’s core product (JIRA), is widely believed to lack heavy-weight depth in the ADLM feature spectrum and it is often implemented as a departmental or “team tool”. They’ll have to develop deeper strategic planning and multi-team project capabilities to beat the rivals.

This May, software giant Computer Associates announced a definitive agreement to purchase ADLM heavyweight Rally and its agile development platform. In its announcement, CA said it intends to leverage Rally’s capabilities to “complement and expand CA’s strengths in the areas of DevOps and cloud management”. With the crucial addition of Rally, CA is now in a strong position to assemble its diverse capabilities into a single unified and enterprise-caliber software delivery platform. Now… can they seamlessly integrate all of the pieces-parts into a cohesive solution with a unified data model? If so, how long will it take?

Finally, I believe VersionOne may have a slight edge over the other ADLM vendors in the race toward a unified software delivery platform. I may be a bit biased because of my direct involvement in a joint project currently underway – none the less, here are four reasons why they will absolutely be a dominant force to recon with:

Vision: Robert Holler, VersionOne CEO, is clearly buying into the “enterprise software delivery platform” vision. He and his team have a well thought out strategy and they are actively executing against that strategy.
DevOps Automation: VersionOne has partnered with ClearCode Labs and both teams have been hard at work integrating ClearCode’s Continuous Delivery Automation framework into the VersionOne core product. This integration provides VersionOne the ability to orchestrate virtually any DevOps tool or platform and (just as importantly) incorporate all related data across VersionOne’s product suite to fee its quickly expanding data model.
JIRA Integration: VersionOne has just announced a tight integration into the JIRA platform. This integration will give them the ability to fold fragmented JIRA installations across the enterprise into the unified VersionOne platform providing a more strategic and enterprise-grade solution.
Availability: VersionOne’s automated delivery platform is available now and they are demonstrating their comprehensive solution to the eager agile community this week at the sold-out Agile2015 conference in Washington, DC.

Summary
The top 5 ADLM vendors are already well on their way toward developing enterprise-grade software delivery platforms that will consume many of the current “DevOps” automation solutions. Soon, development organizations will benefit from a comprehensive platform that can deliver increased efficiency, visibility and agility when compared to the heterogeneous solutions that have been cobbled together today.

About the Author/Dennis Ehle

Dennis Ehle, is a pioneer and thought leader in continuous delivery automation and agile delivery methodologies. Dennis is passionate about helping agile teams dramatically reduce the transaction cost associated with delivering incremental change. His company, ClearCode Labs, does just that by helping organizations continuously deliver high-quality software releases more frequently merging proven methodologies with empowering tools and technology. Twitter: @DennisEhle

August 4, 2015
Avoid the DevOps Tool Trap

Software development, at its most complex and dynamic is a full-court press of effort, knowledge, patience, and hard work. Anything that can reliably and cost-effectively reduce the huge number of tasks, redundancies, reports, and delays associated with the application release process is something to be embraced.

In particular, many organizations today are turning to the agile development methodology, DevOps, in order to rapidly produce software and improve operations performance. According to research by Gartner, by 2016 DevOps will become a mainstream strategy that will be employed by 25% of Global 2000 organizations. Moreover, by 2019, Gartner predicts 50% of enterprises will implement automated configuration and release management as a result of DevOps initiatives.

What many in the development community don’t realize is that solutions exist, using current technologies, to automate many of the routine—and even not so routine—elements of the application release process needed for DevOps. Instead of adding multiple application-specific automation tools, resulting in a confusing and expensive array of various silos of automation, organizations can take advantage of their existing automation suite to automate their application release processes. What we’re talking about is not simply scheduling workflows, but the actual execution of tasks, from administrative duties to build deployments, release management, and more.

Using modern IT workload automation solutions, it’s possible to offload the manual execution of a number of processes to improve overall DevOps agility. Many actions necessary to go from code to production, not to mention maintenance and cleanup tasks, can be taken off the backs of developers and operations, thus freeing them for higher level projects.

With a modern workload automation solution, organizations can automate routine and time-consuming build kickoffs and installations. For example, using pre-built job steps for Microsoft Team Foundation Server (TFS), organizations can create an automated workflow that deploys builds onto machines, installs the kits onto production environments, verifies installation files are correct, and then sends an alert to the proper users that the build is ready. After QA performs testing of the individual stories, the automation solution can then close the stories and merge them in time for a daily or weekly review meeting. Teams can also integrate TFS with a QA/test system, in order to automatically create a new work item with TFS every time a bug is logged.

On the Operations side, one of the biggest challenges in application release management is making sure machines are correctly architected. Excessive instances and regression tests can cause machine slowdowns that create havoc with development schedules. Again, workload automation comes to the rescue via workflows that purge databases, increase CPU, or anything else necessary to ensure a properly-provisioned testing environment.

Adding to this, many popular tools used in software development lack even basic scheduling capabilities. If you wish to create a work item or trigger a build, for example, you probably have to do so using a Web service or the TFS Task Scheduler Service. Even then, all you likely have at your disposal is date/time scheduling. This is where advanced workload automation really shines, since it can trigger tasks according to a pre-defined event, or even a complex combination of events. As a result, DevOps teams can avoid unnecessary project delays or lag time due that would otherwise occur with manual handoffs between teams or individuals.

While workload automation has been around for many years, today’s IT departments are increasingly using workload automation as a foundation for their wider IT systems. As new methodologies and strategies like DevOps and Big Data emerge, automation provides a foundational piece upon which to automate essential processes needed to support these new initiatives. In the area of DevOps, workload automation is helping both developers and operations build and automate release processes more effectively, with capabilities that can make the difference between headaches and headway. Don’t fall into the DevOps tool trap.

About the Author/ Jim Manias

Jim Manias is Vice President at Advanced Systems Concepts, Inc. and is responsible for the overall market strategy and planning for a range of products including ActiveBatch® Workload Automation and Job Scheduling. Jim has been with Advanced Systems Concepts since 1991 and has held multiple senior management positions in the enterprise software and hardware market. Jim can be reached at JManias@advsyscon.com

July 24, 2015
Why DevOps for the database must include three way analysis

Deployment automation is critical to practicing effective DevOps. Without deployment automation, releases still require a lot of manual steps and processes. Processes and steps that aren’t reliably repeatable, are prone to human errors, and can’t be handled consistently with high frequency.

However, the software best practices can’t simply be grafted onto database deployments. Unlike software code, a database isn’t just a collection of files. You can’t simply copy a database among your different development, testing, and production environments. Deploying a database also presents a greater business risk since it holds your most valued asset – your data.

In order to capitalize on DevOp automation advantages for databases and to promote database changes safely and quickly, all the particular components of a database; its tables, functions, and content, need to be protected.

Any successful DevOps database automation solution must be capable of intelligently dealing with conflicts and merges of code and cross updates from other teams. The solution must be able to ignore wrong code overrides and give the user ultimate control over which changes and merges are pushed out.

In order to achieve these goals a DevOps for the database solution must be able to conduct a three-way analysis on the source (that is, the object after the development work is done), the original (or “baseline”) state, and the target (or “Production”) environment, identifying the differences among these three states. Team members, gaining full knowledge of the complete scope of the project from the analysis, can review the results and options the solution presents to resolve any discrepancies. The user selects what action, if any, should be taken on the database. The solution then automatically generates the scripts to execute the chosen actions.

In this way, the solution uses automated three-way analysis and automatic script generation to eliminate both the potential for human error in manual script writing, and the potential for technological error in automated object overrides.

Reaping the Rewards
Implementing a solution that encompasses all these automation features to meet the particular challenges of database deployments, would enable a company to practice a DevOps approach to effective database automation and realize the following benefits:

· Reduced deployment costs through streamlining of development process management

· Minimizing deployment risks by enforcing change policy and best practices, and configuration management and consistency

· Improving responsiveness to customers and market needs with speedier roll-outs of task-based development updates

· Enhancing cross-team communication with clear audit trails and transparency regarding work done on objects and the state of different environments

Can you afford to be without it?

About The Author ⁄ Yaniv Yehuda, CTO, DBmaestro

Yaniv is the Co-Founder and CTO of DBmaestro, the leading provider of DevOps for Database solutions which enable control of database development and deployment. Yaniv is also the Co-Founder and the head of development for the Extreme group, a leading IT services solutions provider group.

July 23, 2015
When Good Code Goes Bad

Milk spoils. Iron rusts. And in software, good code goes bad. Yet the difference is, with the first two, you know the change has occurred. With software, those changes are not always obvious.

Your 5,100 Binaries Went Bad

There is no way to prevent software from “going bad”. As with all products, bugs and defects,are bound to happen at some point. No one and no code is immune from these issues. But who’s looking for the 5,100 software components in your organization that went bad last year (meaning new security vulnerabilities were discovered in them)? It’s all too likely, no one.

Earlier this year, I took a deep dive into the analysis of software supply chains that fuel high velocity development practices and IT operations. The analysis revealed that some of the largest development organizations were consuming an average of 240,000 open source components to expedite development, accelerate innovation, and meet time to release deadlines. While reaping huge benefits from this consumption, these same organizations were unintentionally growing their technical and security debt. How you ask?

Here is a summary of the their average debt:

Source: 2015 State of the Software Supply Chain Report

The analysis showed that over 15,000 (7.5%) of the open source components being consumed by these organizations in 2014 had known security vulnerabilities. Of those 15,000 components, an average of 66% (9,900) had known vulnerabilities dated 2013 or older. That means, they were known vulnerable components (“bad”) before they were downloaded.

The remaining 34% (approx. 5,100) might have actually been “good” components at the time they were downloaded by development teams from public open source repositories, but at some time during 2014 a new security vulnerability was discovered and a CVE identifier was assigned.

Even with the best controls in place to protect developers from downloading open source components with known vulnerabilities, the fact that new vulnerabilities are continuously discovered — about 50 new CVEs are announced daily — means that continuous integration and delivery practices also need to be complemented by continuous governance and visibility practices.

We’re Not Alone

Software development is not the only industry facing such issues of things good things going bad. I bet all of us have received notification from our car’s manufacturer that a certain part is being recalled. While the part was known to be safe when the manufacturer sourced it from suppliers, placed it on the car during assembly and shipped it, discoveries of defects are often found years after the vehicle has been on the road. The same scenario has been played over and over in the food, pharmaceutical, defense, toy, electronics, and other industries.

The difference is those other industries are known to have more mature supply chain practices where they can track and trace every part or ingredient ever used in every product they have shipped. And when a part goes bad, they can recall and repair that defective component.

In software, track and trace practices are still in the earliest stages of maturity. In a 2014 survey of 3,300 development professionals, 63% stated that their organizations have incomplete or no records of the components used in their applications. This lack of traceability means that even when good components go bad, development and IT operations teams have no way to quickly find the defective component and replace it with a better, safer version — dramatically extending mean times to repair, or in the case of no action, growing their technical and security debt. And when applications with those defective components are in the hands of customers, the ability to “recall” the impacted software or distribute newer versions of the application are hindered.

Did he recommend a “Bomb”?

A Bill of Materials (BOM; pronounced “bomb”) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product. According to Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.”

In Part 2 of this blog, “The Software BOM Squad“, I will introduce the importance of automating the creation of a “software bill of materials“ (Software BOM) to take inventory of the third party and open source components used to build an application. When software components age more like milk than wine, a software bill of materials can help you quickly identify where your 5,100 “bad” component were used.

July 22, 2015
Where Oh Where has my Hadoop gone?

Where is Hadoop? Just a few years ago, it was everywhere. Internet heavy hitters such as Facebook and Yahoo boasted about the size of their Hadoop clusters and applications. It was easy to imagine the day when Hadoop would be the one indispensable tool for anyone who dealt in large volumes of data.

(more…)

July 21, 2015
DevOps fast food: Chef on the AWS marketplace,

News from the AWS Summit in NYC today is that Chef is now available as a one click install on AWS via the AWS marketplace. While the ability to use Chef on AWS is not new (it is the leading configuration management tool on AWS), being on the marketplace makes both install and billing a one click deal. This should make it really simple to get Chef installed and running for anyone looking to leverage the many benefits Chef brings to configuration management and automation. Literally with one click install what could be simpler?

As part of this new arrangement not only will Chef on the AWS marketplace be available, but it will also be a featured offering. Chef and AWS already enjoyed a cozy relationship (they are both based in Seattle and there is a cross-pollination of personnel). This deal takes the relationship to a higher level. Both Chef and AWS are dedicating significant resources to marketing, sales and support of this initiative.

I know from speaking to AWS in the past and other marketplace vendors that AWS will often let partners list on the marketplace, but they don’t often then dedicate resources to roll out, marketing and support. Further, as far as I know Chef is the only configuration management/automation solution that is available via the marketplace like this. I don’t know if this will remain the case or for how long, but for now it is somewhat of an exclusive relationship.

I had a chance to catch up with Ken Cheney, VP of Business Development and Jay Wampold, VP of Marketing at Chef about this expanded relationship the other day. They both emphasized that this is much more than just listing Chef in the AWS marketplace. Some estimates I have seen have Chef deployed on something like 70% of AWS deployments. Ken and Jay believe that this number will continue to remain high and may even go higher as a result of the marketplace presence and the push from both Chef and AWS.

On top of their previously released expanded relationship with Microsoft via Azure, it seems Chef is putting a stranglehold on two of the largest cloud platform providers in the world. That doesn’t mean that other configuration managers don’t have presence on AWS or Azure, but Chef seems poised to be in a dominant position on these platforms.

As we see more enterprises continue to migrate more and more infrastructure to the cloud this presence will become more and more important. For now you can get Chef fast on the AWS marketplace.

July 9, 2015
HashiCorp unites open source DevOps tools with Atlas

Open source and DevOps go together like chocolate and peanut butter, or like milk and cookies (these metaphors are making me hungry). Basically, they’re inseparable and—while each is good in and of itself—they’re better when united. HashiCorp announced today that Atlas, a platform for a variety of leading open source tools, is now available to the general public to help organizations make DevOps even better.

HashiCorp has a number of popular, industry-leading open source tools at its disposal. Vagrant manages development environments. Packer automates building artifacts like Docker containers, OpenStack images, and VMware images. Consul is a tool for runtime orchestration. Serf is a protocol used by Consul for failure detection and message distribution in a server cluster. Terraform automates infrastructure provisioning on cloud platforms like Microsoft Azure and Amazon Web Services. They’re all great tools in and of themselves, but Atlas ties them all together into a powerful, cohesive whole.

Atlas has been available in beta for a while with tremendous success. Nodes under management consistently doubled month-to-month during the beta, and heavy hitters like Mozilla and Cisco jumped on board to test it out.

DevOps seeks to accelerate development and deployment. One of the primary attributes of DevOps is automation to facilitate automatic, error-free, auditable implementation of routine, repeatable functions. Many organizations strive to embrace DevOps tools and principles and leverage open source tools like Consul or Vagrant but use manual, time-consuming, error-prone processes that are difficult to scale and challenging to manage effectively.

HashiCorp believes that Atlas is the solution—a platform that brings it all together and enables organizations to manage end-to-end from configuration management to container deployments across multiple cloud environments. Each of the open source tools is fine alone, but the ability to manage and orchestrate them as a whole is very powerful.

“We initially rolled our own solution for infrastructure CI and deployment. As the team grew, our homebrew solution struggled to centrally manage configuration, Terraform state, and access control,” said Chris Lonnen, Internal Tools and Systems Architect at Mozilla. “Atlas remedies all of these challenges for us and provides a beautiful interface for audit logs, infrastructure history, system-wide monitoring, and more. We’ve never lost 9s betting on a HashiCorp product.”

According to the HashiCorp press release, you can start using the hosted version of Atlas as of today, but on-premise deployments of Atlas won’t happen until late 2015. If you want more information about on-premise Atlas you can sign up at this link: https://atlas.hashicorp.com. The development features like Vagrant box builds and hosting and Packer builds and artifact storage are free, and the first 10 nodes are free as well. Pricing is $40 per node per month beyond the first 10 nodes.

July 9, 2015
Managing many hosts is more than a Big Data problem

Back in my days at StillSecure a major problem we had with our vulnerability management platform was managing vulnerabilities over a large number of hosts. How can you effectively manage so many hosts, visualize them across the entire deployment? It gave our GUI designers and developers fits.

In today’s world of virtual instances, containers and apps, managing a large number of hosts is even a bigger challenge. It is a daunting task that even the largest companies struggle with. In fact in some ways the bigger the company, the greater the amount of hosts and the harder it is to manage. I wanted to highlight today how one company, Datadog has tackled this problem with their Host Maps feature.

The trick to managing large numbers of hosts is to balance a global view with an individual host view. It is harder than it sounds though. It is almost like joining Einstein’s Theory of Relativity with Quantum Mechanics. How can we use one view to see things on both the galactic scale and the molecular level at the same time. Datadog has developed one of the best ways to do this with their instant grouping and drill down capability.

Instant grouping allows you to virtually (no pun intended) define groups on the fly by a large number of tags and filters. So for instance you can use standard AWS tags. You can also group hosts by tags and they will be visualized this way.

This was where we ran into the wall on our vulnerability manager. We were never able to create groups on the fly. You could define groups by certain filters but drilling down from there was out of reach. With Host Map feature you can create a group of hosts that have a certain feature or attribute in common and then drill down into that group with the click of a mouse. The YouTube video below shows you much better than I have described what I am talking about:

Datadog’s designers have also made great use of color selection and shaping to help make visualization much easier for the end user. Pretty nifty use of simple technology.

Not to turn this into a love fest for Datadog Host Map, but I am not aware of others who have instituted this kind of on the fly grouping and drill down capability to address this Big Data type of problem. If you know of one, I would love to hear about it.

From the point of view of looking at how the state-of-the-art has evolved from when we were trying to tackle this issue 10 years ago, it is impressive to see how far GUI has progressed. I think it is a clear case of “mother is the necessity of invention”. It would be near impossible to manage the kind of host deployments we see now without this type of functionality.

I did have a chance to speak to a few Datadog customers when I reached out to Datadog about Host Maps. It was unanimous that they feature was as valuable to them as it looked to me. Now of course one could say that Datadog wouldn’t refer me to customers who didn’t like the product. But my impression was that this feature was indeed a big help to them.

As we continue learn to better manage data flows and manage ever growing infrastructure, I think we will continue to see the evolution of management GUIs like what Datadog has done here.

What about you? What GUI functionality have you run across that you think is worthy of praise? I am sure others would like to know.

July 7, 2015