devopsy.kinsta.cloud

Tag: containers

To DevOps Or Not To DevOps

Although enterprises today all claim to be software companies, there remain four distinct groups of organizations: the technology leaders, the technology followers, the technology hopefuls and the technology laggards.

The first group are the industry leaders who are paving the way. These are the organizations coming up with new technologies, processes and cultural shifts. They write articles and books and present at conferences, and despite perhaps working at smaller companies, they are the names you may recognize. These are the companies the rest of us aspire to be—the Googles, the Amazons and the Microsofts that keep the rest of the industry hanging on to every word.

The technology followers typically are smaller organizations that tend to adopt the latest and the greatest technologies. They are the leaders’ first wave of followers who are not afraid to try new things, nor afraid to fail. They have cultures that allow them to experiment and adapt and are given a lot of free time to invest in learning and implementation, after they watch the big guys successfully implement.

However, the first two groups are the minority of software organizations, since not everyone can lead the way, nor can everyone have the luxury of spending a significant portion of their time experimenting with new methods. After all, most of the new technologies do not have a long lifespan. They appear, generate some attention and then disappear. Front-end frameworks provide a great example of this: new ones pop up and exist for a few months before being superseded by other frameworks.

Another example of technology followers is containers—few doubt their usefulness and most of us are convinced they are the best solution. The only question is which cluster orchestration framework to choose. These frameworks are evolving so rapidly, with new ones introduced on a weekly basis, that it’s almost certain most will be defunct within a year. So, if you are into experimenting with the latest offerings, chances are you could make a wrong choice. Will it be Kubernetes? Docker Swarm? Mesosphere Marathon? Rancher OS? Cloud Foundry’s Diego? CoreOS Fleet? The question is not whether to adopt one of those tools, but rather, which one has the best shelf life.

Most of us fall under the third group, the technology hopefuls. We don’t have the ability to lead the way, nor do we have the luxury to explore the latest and greatest technologies. Yet, we cannot allow ourselves to become obsolete. The industry is moving so fast that just a few years can leave you at the back of the pack where no effort or investment can get you back in the game.

The technology laggards are the software companies that have fallen behind. This group includes businesses that are struggling to remain intact and run the risk of disappearing. These are the organizations that do not innovate, do not adapt, do not invest (enough) in improvements and do not even understand the current industry trends. They are the organizations that do not comprehend that becoming obsolete doesn’t take decades, but years—sometimes even months.

Based on the definitions of these groups, chances are, unless you are Google or Microsoft, you are not in the first or even second groups. By reading this article you are making the effort to be informed so you don’t lack the self-awareness that places you in the fourth group. Like most software organizations, you fall into the third group of technology hopefuls.

As part of this group, you have to balance between the latest and the greatest technologies and the chance of becoming obsolete. You cannot try every new solution that comes along, nor you can let years pass by until you jump on the train. The big questions you have to ask are, How can I distinguish what will stay and what will disappear? How do I make a choice between what to adopt and what to ignore?

There may not be a one-size-fits-all answer, but one process you absolutely cannot ignore is DevOps. If you’re asking yourself whether your organization should adopt DevOps, the answer is a resounding “yes.” And I encourage you to adopt it now, before you are replaced by other companies that are already living and breathing DevOps. Otherwise, you’ll be unable to adapt to the speed needed to keep up in today’s market and understand the cloud and scale accordingly when an avalanche of users come to your site after a successful marketing campaign. Lacking preparedness could cost you your business, especially in a competitive market.

DevOps is here to stay, evolving similarly to Agile by providing value and answering many of today’s challenges. The only problem is that adopting DevOps can be difficult. To implement DevOps, you might have to change your company’s culture; you might need to adopt new tools; you might need to change processes; and, you might need to change your software architecture. Changing any of those aspects is hard, but changing all of them is very hard.

Your culture and company structure need to be agile (in the literal meaning of the word). The old, monolithic tools need to be replaced with smaller, much more dynamic ones that will work well with the ever-changing nature of cloud infrastructure. Processes need to be automated, since there is no time to pause for manual intervention. To accommodate the needs of automation and frequent deployments, your applications and services will need to be relatively small for continuous delivery or deployment to become a standard process. Your teams need to be highly specialized and fully capable of working on all aspects of the software development life cycle, from requirements to post-production support.

It’s a harsh and challenging road ahead, but at the end of it lies success. You’ll experience faster time to market, greater reliability, high availability and faster adaptability to market changes. Some are already there, while others already embarked on the journey. Don’t be left out—the alternative will leave you obsolete.

To DevOps or not to DevOps is not the question. For most organizations, adopting DevOps is the answer.

About the Author / Viktor Farcic

Viktor Farcic is a Senior Consultant at CloudBees, a member of the Docker Captains group, and books author. He coded using a plethora of languages starting with Pascal (yes, he is old), Basic (before it got Visual prefix), ASP (before it got .Net suffix), C, C++, Perl, Python,ASP.Net, Visual Basic, C#, JavaScript, Java, Scala, etc. His big passions are Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD). He published “The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices,” and “Test-Driven Java Development.” He is currently working on his third book, “The DevOps 2.1 Toolkit: Docker Swarm.” Connect with him on Twitter.

November 1, 2016
Docker On Windows, Are You Serious? Microsoft’s Hands-on Lab Show’s You How

You have probably heard or read about the availability of Docker on Windows and Azure. Both Docker and Microsoft have turned up the noise on their collaboration. But have you really looked into what these two are up to? If not, now is your chance. Microsoft has put up a great free hands on lab all about Docker, containers and Windows.

In this hands on lab there are modules that feature Docker on Windows Server, Windows Nano Server and managing them with Windows 10. You can use Microsoft’s HyperV or not. It really is a great way to learn about containers, Docker and the Windows platform. You can play with this hands on lab by clicking here or if you prefer to cut and paste here is the link: http://msftdevops.learnondemandsystems.com/?wt.mc_id=AID557152_QSG_BLOG_22879. (Editors note: pop up blockers may stop the lab from loading. If you have an issue try disabling the pop up blocker.)

Here is what the lab covers in a bit more detail:

I took the opportunity to play with the lab myself. While I am far from a developer or even an Ops person, I have to say Microsoft got this one right. Even I could complete it! The lab is supposed to take you 30 minutes but they give you 8 hours to complete. There is a great set of introduction videos that give you a full picture of what you are going to be doing. Here is an example in a still shot:

The lab is actually broken down into several modules that don’t have to be completed in one sitting. The lab requires no set up on your part at all. It runs right in your browser. The lab has you run two different Windows machines, one Win10 and one Windows Server. You can switch and log in right from the browser. The lab also has you log into PowerShell:

It literally gives you the CLI commands that you have to type, tells you when to switch between machines and guides you step by step through each of the several tasks. There is a great info bar on the bottom that has context sensitive instructions that walk you through the entire thing. There is also a sidebar which gives you a great oversight of the entire lab, where you are in the process and options to move around. All in all I thought they did a great job with the UX to make what could be a complex set of tasks easy.

The whole Docker and containers on Windows theme is one has been bubbling up for some time now. While many doubted that it could ever happen, the new Microsoft seems to have made Docker right at home.

From what I have been told there will be similar hands on labs offered for free on the Azure platform and more on containers. We will be working with Microsoft to dig in deeper here, so stay tuned for more soon!

— Alan Shimel

October 31, 2016
All Day DevOps
Over the past year, I have traveled to and delivered presentations at 18 DevOps events. I’ve also heard that over the past several years, John Willis has participated in more than 200 DevOps events. But not all of us have the time or budget to get ourselves and our teams out to these events where we can learn from other practitioners.

This is why we’ve assembled the All Day DevOps Conference for Nov. 15.

Free DevOps

All Day DevOps will deliver 57 practitioner-led sessions over 15 hours and 15 time zones. No vendor pitches are allowed. To break the travel and budget barrier for you and your teams, we are delivering every session live, online and free of charge. Every session will also be recorded. We are doing this because we want to share global DevOps expertise with everyone.

DevOps Brains

More than 3,500 people have now registered for the conference. We have an awesome speaker lineup covering three tracks: Modern Infrastructure, CI/CD and Automated Security. Here is a sample of some of the great sessions being delivered:
- CI/CD
  - Meta Infrastructure as Code: How Capital One Automates our Automation Tools with an Immutable Jenkins, by George Parris (Master Software Engineer at Capital One)
  - Speaking for the Dead: Critical Positive Lessons from Waterfall and Monolithic Architectures, by Michael DeHann (Former CTO & Founder of Ansible)
  - I, For One, Welcome Our New Robot Overlords, by Mykel Alvis (DevOps Computational Demonologist at Cotiviti)
- AUTOMATED SECURITY
  - Stop Changing Your Password, by Chris Swan (CTO Global Infrastructure at CSC)
  - System Hardening using Ansible: Application Deployment + Configuration Management + Continuous Delivery, by Akash Mahajan (Lead at OWASP Bangalore)
  - Building Codes for Building Code, by Josh Corman (Director of Cyber Statecraft at Atlantic Council)
- MODERN INFRASTRUCTURE
  - Terraforming Your Infrastructure: From Beginner to Semi-Pro in 30 Minutes, Britt Treece (Production Engineer at PhishMe)
  - DevOps Friendly Doc Publishing for APIs & Microservices, by Mandy Whaley (Director of Developer Experience at Cisco)
  - Monitoring, Hold the Infrastructure – Getting the Most out of Lambda, Michael (DevOps Evangelist at DataDog)
Don’t Just Listen, Talk DevOps

One of the big pieces of input we have received from the community is that everyone wants an opportunity to interact. This is why we are building a dedicated team room in Slack for you. Every attendee will have the chance to interact with the speakers in the Slack room. Q&A will be live, online and freely accessible. We will also have Birds-of-a-Feather channels set up for those with questions to be helped by those with answers.

Register Now

If you would like to join us, can you register for free. A big thanks goes out to Alan and his team at devopsy.kinsta.cloud for helping us spread word of the event here. We are really excited to have teams from North America, Europe, Asia, and Australia joining us for the event!

www.AllDaydevopsy.kinsta.cloud

— Derek E. Weeks
October 26, 2016
So Long to Configuration Management Systems?

When I was an operations architect at Ning way back in 2009, we were struggling to manage our deployments and configurations of more than 50 different application servers, or what the cool kids today call microservices. Applications were deployed across three distinct environments: development, staging and production. None of these environments were exactly the same as another, and differences in these dependencies, together with human errors, frequently broke our deployments.

Ning was nowhere near the first company to encounter such problems. In the mid-1990s, as the public internet emerged, much of what was understood at the time about IT operations was challenged by the need to scale internet-facing systems. For many years preceding the internet boom, the maximum number of users was strictly limited to the number of employees in a company. That all changed with the World Wide Web, where the need to serve thousands and millions of simultaneous users easily exceeded the computing capacity of even the largest and most expensive computers.

As three-tier application architectures were broken down into services and distributed across dozens or hundreds of individual servers, this proliferation of servers proved to be far too much for operations teams to handle. The worst problems lived within application deployment processes, where environments and requirements were ever-changing in successive releases of software. The lifetime of a server was a long one. Once deployed, servers were subjected to repeated changes by operations and developers in what would come to be called “configuration drift.” The combined effects of changes, human errors and unknown states would create disastrous outages that damaged brands and destroyed online revenue. It would take sites days and even weeks to recover from these outages. Some would never recover.

It was in the midst of this chaos that modern configuration management systems were born. CFEngine was the first to emerge in academia, where the need to manage multiple UNIX systems was felt early on. Later, Puppet, Chef, SaltStack and Ansible emerged directly from web-scale systems. Configuration management systems fulfilled several important needs. First, they abstracted the differences between operating systems, providing a high-level language for the system administrator to describe the desired state. Second, they automatically synchronized the desired state across large numbers of servers, reducing costly human errors. Finally, they provided idempotency to ensure that destructive operations were not applied unnecessarily.

There is never a free lunch. The benefits of better abstraction were offset to a degree by the need to build and maintain evermore complicated recipes. Recipes frequently broke from time to time as package management features were updated, renamed or otherwise changed from the time the recipe was first written. Even when recipes worked, they were frequently broken by differences in environments or runtime errors. The configuration management systems themselves began to suffer from scale challenges, and evolved message bus architectures to improve their speed and responsiveness. By overcoming many of these obstacles, configuration management systems have now become a first-line tool for managing web-scale systems.

But back at Ning, modern tools such as Puppet and Chef were still in their infancy and simply could not cope with the scale of our 60,000+ node virtualized environment. Ning had a lot of brilliant software engineers floating around, and they set about creating their own deployment automation system called Galaxy, and later a meta-orchestrator called Cosmos. Galaxy and Cosmos were great tools, but they required constant care and feeding and suffered from severe scale problems. There was just a lot of complexity in dealing with these automation systems while also trying to scale a top-tier social network for a hot startup company.

As we began to analyze and post-mortem our deployment failures, it became clear that a majority of the problems stemmed from host and environment configuration errors. We needed a simpler way to deploy our software. Virtual machine images were a promising way to ensure consistency between environments with the tried and true “gold master image” approach. The problem with this approach was that building full system virtual machines was difficult in automation terms. Once deployed, virtual machines are also subject to configuration drift. Furthermore, while we were eager to adopt virtualization to get better resource isolation, we were wary of the overhead that hypervisors would create. Our benchmarks showed that if we adopted virtual machines, we would need to increase our physical server capacity to accommodate the overhead.

At the time, Linux containers were already available in the kernel, but the distributions had almost no support for managing them. At the insistence of our VP of Operations, I cooked up an alternative solution based on jails, writing a small hack of a program called Warden to build and manage them. (Note: Totally unrelated to the Warden in Cloud Foundry). Jails gave us bare metal performance, resource isolation and a way to consistently deploy the entire environment for each application server. With Warden, we could build jail images on the fly automatically and deploy them consistently with our existing tools, galaxy and cosmos. Hey, it was a Band-Aid for a difficult situation, but my experience with building Warden flipped a switch in my brain. I had seen the way lightweight virtualization could eliminate complexity, and I knew right then this was the way of the future.

Today, with the success of Docker we are finally experiencing sweeping changes in the processes and tools we use to build, deploy, and manage distributed applications. At the same time, we are about to enter a brief period of containers disillusionment as early stage solutions get washed out. When that is over, what comes afterward is something of a golden era for distributed applications. I call it the container application platform, and its somewhere between what we think of as platorm as a service (PaaS) and Serverless today. In that not-too-distant future, containers will be the default output of the software build process, and where that container gets deployed will hardly be an afterthought. This unicorn magic will come from the ability to introspect containers to determine their environment dependencies, and a fully integrated tool chain that deals with deployment, scaling and dependency resolution automatically. I will give you just one guess which company is in the best position to own that future. “One ring to rule them all.”

What happens to our beautiful configuration management systems? There has been much debate about this among smart people over the last few years. Most of these arguments boiled down to some kind of coexistence model. Indeed, configuration management systems are useful for dealing with a wide range of operational problems outside of the container platform. While it’s no surprise that Chef, Puppet, Salt and Ansible all moved quickly to include Docker support, I think we will see these tools begin to focus more on what Rob Hirschfeld calls achieving “ready state”. That is to say, everything that still needs to be done before you can lay down your container platform also still needs to be automated. Right now, that is still a great deal of work, and it’s a pretty safe place to draw the line on using the right tool for the right job.

About the Author / James Thomason

James Thomason is chief technology officer at HyperGrid from Dell, where as CTO he led the vision, technology strategy and product road map for Dell Cloud Marketplace. He joined Dell through the acquisition of Gale Technologies in 2012, where as CTO he headed the product road map, architecture and engineering of the company’s flagship cloud and converged infrastructure automation software, GaleForce. Prior to joining Gale Technologies, Thomason was the CTO and Founder of Virtiv, a San Francisco cloud automation startup focused on Linux virtualization, acquired by Gale Technologies in 2011. For over 17 years, as a specialist in distributed systems and large-scale infrastructures, Thomason has been an entrepreneur and innovator at a number of notable Silicon Valley startups, including Exodus Communications, Digital Island, Netli, NetVMG, Netscaler, 3Leaf Systems and Ning. Connect with him on LinkedIn / Twitter.

October 14, 2016
Getting Ready for All Day DevOps
DevOps Practitioners Unite

On Nov. 15, we’ll be broadcasting 54 live sessions across 15 hours and 15 time zones. The All Day DevOps conference will be live, online and free. Anyone who registers can watch.

We have invited some top-notch practitioners from around the world to speak at the event on topics ranging from DevOps transformations, to CI/CD, to automated security, to modern infrastructure (think serverless, microservices and containers). The speaker lineup will include:
- Damon Edwards, DevOps Cafe Podcast and Founder DTO Solutions
- Michael DeHaan, creator of Ansible
- Erlend Oftedal, CTO at Blank Oslo
- Hasan Yasar, Software Engineering Institute at Carnegie Mellon
- Justin Collins, Founder and CEO at Brakeman Security
- Britt Treece, DevOps Engineer at PhishMe
- Sumit Agarwal, DevOps Change Manager at Broadridge Financial
Don’t Watch Alone – Host a Viewing Party!

One of the best parts of attending a DevOps conference is getting to interact with others in the community. We have had many organizations including Capital One, Intuit and Fannie Mae approach us about hosting viewing parties at their offices—and opening the party to the public. Sound intriguing? Here’s how to do it:

Secure the location. Pick your corporate auditorium, cafeteria or open meeting space. Secure it all day for Nov. 15. Make sure you have at least one internet-accessible viewing screen.

Promote the session: One you have the approved locations and times, share them here with the All Day DevOps Conference organizers. We will help you promote your party on AllDaydevopsy.kinsta.cloud and through social media postings, directing them to your registration page.

Invite People to Register: Set up a place for people to register for your location. You’ll want to know how many people to expect because you need to know how much pizza to order. Meetup.com and Eventbrite.com make event registration easy.

Make it Fun. Order some pizza and drinks. We’ll send along some cool laptop stickers from the conference and some of the sponsors. Share pictures from the event online (@AllDayDevOps). Join the conversation in the conference Slack channels. Set up birds-of-a-feather stations. Play some music during breaks. Set up viewing screens for each of the conferences three tracks.

Just to be clear: the All Day DevOps Conference is not a collection of vendor pitches. In fact, we have tried our darnedest to ensure the agenda is purely focused on real DevOps practitioners sharing their craft. Take a moment to look at the speakers participating in each of the tracks and you’ll see what we mean.

Register today. Invite your colleagues to attend. Then plan a viewing party at your office.

— Derek E. Weeks
October 7, 2016
All Day DevOps: Bringing DevOps to the World
An Audacious Plan: Deliver DevOps to everyone.

The global audience for DevOps is expanding faster than any one person or company can keep up with. While DevOps Days and other regional events provide invaluable support to their local communities, we wanted to create a global event, offering the very best lineup of speakers across North America, Europe and Asia.

We also wanted to make sure that anyone, anywhere can attend. So, we set a high bar for our organizing team, and they delivered: $0 registration.

By creating All Day DevOps as an online event, we are able to get many people to attend and present—many of whom ordinarily would not be available because of time or travel restrictions. We expect well over 1,000 people to attend throughout the day.

15 Time Zones, 15 Hours, 3 Keynotes, 54 Sessions

Nov. 15 will see the first truly global, online DevOps Conference: All Day DevOps. Starting at 9 a.m. Greenwich Mean Time/5 a.m. Eastern Standard Time and continuing through midnight GMT/8 p.m. EST, there will be three simultaneous tracks, each track containing 18, 30-minute presentations, for a total of 27 hours of presentations.

Call for Papers

We are inviting speakers from around the world to participate in this first-of-its-kind event covering 15 time zones. Timing for the conference is set for the EMEA sessions to begin at 9 a.m. GMT, then, at 9:30 a.m. EST, move the concentration of presentations to the east coast of the United States and ending with presentations on the west coast of the United States starting at 11 a.m. PST. All sessions will be recorded, but you will need to register for the event to have access to the recordings.

The Call for Papers is now live. Please submit your paper soon. The deadline for submissions is Sept. 21.

Conference Tracks

The conference will host three tracks: Modern Infrastructure, Automated Security and CI/CD. We’ll have 54 sessions plus three keynotes on three tracks, running continuously for 15 hours.

Track One: Modern Infrastructure

Modern Infrastructure Track Hosts and Moderators: James Wickett, Signal Sciences, joined by Ernest Mueller, AlienVault and Karthik Gaekwad, Oracle

In this track, we will be discussing novel architectures, approaches and tools for building out software and services. We are not looking for the standard approaches, but the way Infrastructure will be in the next three to five years.

Track Two: DevSecOps & Automated Security — Living the Dream

Automated Security Track Hosts and Moderators: Milton Smith, Oracle, and Shannon Lietz, Intuit

You have a well-tuned development and delivery environment or you’re in the process of building one. AppSec is a top business concern. You moved DevSecOps from Big Foot and UFOs to reality, you sold the dream to staff, and now it’s beginning to pay off. This track explores DevSecOps tools and techniques to build a successful automated security program.

Track Three: CI/CD — Continuous Everything

CI/CD Track Hosts and Moderators: Derek Weeks, Sonatype, and Andi Mann, Splunk

Continuous delivery is not magic, but it does take the right effort. The most successful practitioners have taken pragmatic steps to reduce cycle times, improve collaboration and build quality in without burning people out. This track asks you to share your frontline experiences of building CI/CD pipelines, aligning key stakeholders and measuring the results. Remember, what’s now an everyday experience for you is awesome to another pursuing that path.

Conference FAQs

How to I register to attend?

Registration is open now at AllDaydevopsy.kinsta.cloud. The site will be constantly updated, sometimes multiple times per day, with the latest information. You can also follow us on Twitter under the handle @AllDayDevOps and using #AllDayDevOps.

How will the sessions be delivered?

The sessions are delivered over Google Hangouts, allowing us to have an unlimited number of attendees on each of the three channels. All you need to deliver a presentation is a reliable internet connection.

We are also arranging Slack channels for all sessions to host interactive Q&A sessions for everyone.

If my talk is selected, can I post a notice or an article about the conference on my site?

Not only is it OK, we would thank you profusely if you did. You’re welcome to use the banner and the conference icon if you need graphics for the article. Let us know when it goes up and we’ll give you a shout-out.

Who is organizing All Day DevOps?

The All Day DevOps Conference is organized by advocates in the DevOps community. We are working together to deliver a diversity of vendor-neutral sessions and speakers from around the globe to show practical usage and technical solutions of DevOps in small and large enterprises.
- James Wickett, Signal Sciences
- Derek Weeks, Sonatype
- Shannon Lietz, Intuit
- Milton Smith, Oracle
- Andi Mann, Splunk
- Mark Miller, Sonatype
- Ernest Mueller, AlienVault
- Karthik Gaekwad, Oracle
September 8, 2016
DevOps and Automation Abstraction?

Similarity breeds consistency, giving way to predictability and, thereby, dropping costs of operations and quality from a lower number of incidents and a shorter duration to resolve them. But at the heart of this chain of events is the recognition that we tend to do certain tasks the same way, over and over again, for a given type of technology. Variability then becomes our enemy. Consistency becomes our ally. So the question: How do we as DevOps service providers instill a level of consistency in what we do that is not engineer-specific (i.e. only one guy knows how it is supposed to work) and will still scale at the enterprise level?

From a people perspective, we use teams, whether co-located in person or virtually using the same backbone of tooling over wide geographical separation. From a process perspective, we use governance and education to ensure the same processes and approaches are taken at a fundamental level across different types of technologies. And finally, from a tooling perspective, we use templates or frameworks (or pick your synonym) that allows us to construct a shell that guides the automation construction to be consistent from parent to child. The combination of these three fundamental elements leads to consistency that scales. But is that all?

What Are We Missing?

Could there be another fundamental element we have overlooked—namely, abstraction? Case in point: Consider the effect of a drag-and-drop interface versus a typed one to accomplish a given task. Both look to accomplish the same outcome. One could argue the typed text approach offers more flexibility and customization to get the task done. But the drag-and-drop method offers a consistent, simplistic way of doing a similar task, the same way, over and over again. Examining both methods objectively, the drag-and-drop method offers a level of consistency the typed method never could. For the typed one, you would always have to check to ensure punctuation and lack of changes (additions or deletions) to “know” it was the same script as last time. The drag-and-drop method on the other hand, is “canned” or largely unalterable; it may limit customization (by design), but it excels at consistency.

The drag-and-drop method achieves this consistency by effectively abstracting the typed or scripted code within it from the user who employs it. The user only makes the choice to use it, when desired (or needed); the rest of the mechanics are abstracted from the user. The reason drag-and-drop methods exist is beyond just consistency, however; it is for simplicity. Apple computers would appear to be founded on this idea. For those of you old enough to remember DOS on PC computers, the parallel here immediately becomes clear. DOS was a typed-based command-line operating system (OS) fully capable of operating a PC many years ago. But Apple’s graphical interface made the same functions “intuitive” and forced the PC industry into Windows to stay competitive. Nobody chooses a command line OS like DOS when something as elegant as iOS exists, or Windows (or pick your graphical poison here).

A graphical OS abstracts core functions from the user, and makes every attempt to make an elegant intuitive packaging of the same functions. Users “get” an iPhone, which accounts for its sales. You don’t need formal education classes or an eidetic memory to use an iPhone. Users effectively have been abstracted from all the typed code it takes to create an OS like this, millions of lines of such code packaged up into swipes, beeps and elegant graphical displays. Keep in mind, at its core this OS abstracts users from variability that might exist if hundreds of companies created their own versions of iOS. The simplicity is paired with consistency, and the quality of Apple is a result (for better or worse). But whatever the quality of Apple is, it is far better than if a wide degree of variability were permitted in the vendors of iOS or the construction of it.

Extending Abstraction in DevOps

Now enter DevOps. Containers are a level of abstraction widely popular in our discipline for keeping hardware particulars away from what are typically software engineers. The value of this abstraction is well-understood. My challenge is simply to extend this thinking into the next tier of DevOps tooling. What if the ability to include database DevOps automation was simply a product of drag-and-drop methods that every build tool understood perfectly? Granted, you may have to limit the expression of this idea to say Oracle or SQL Server at first to capture the nuances and particular capabilities of products and technologies in this space. But if someone built a “container” for Oracle, I would wager the choice to use Oracle would rise substantially over its competitors, at least until those containers or abstraction layers were just as effective with other database products.

The abstraction phenomenon need not be limited to hardware, or as I have proposed to database platforms. It also could apply to .NET or Java development efforts at the top of the stack. The steps needed to take an idea, code it and put it through the build process are going to be largely similar when working in a .NET shop or a java shop (or pick your particular development tool of choice here). Even the top of the stack—that is to say, the development of applications itself—could be a viable candidate for abstraction from the mundane details as it relates to a build construction or a deployment to a given class of environment.

Creating effective containers for the application layer in effect limits variability, and increases predictability. I realize it has to be technology-type sensitive. The .NET shops have their own container. The java shops have theirs, and so on. The only question is whether we are able to include enough of the common core steps we take to produce a build with a given technology type to form an abstraction construct of value. Over time, this would include every type of technology widely in use today.

The effort to simplify through abstractions ultimately would result in our ability to include technologies such as WebSphere at our pleasure. It would also allow us to switch out WebSphere for a new competitor using the same abstraction constructs, universally, just by rebuilding all our apps one time. This kind of power is alluring to development teams. It drives application developers and operations engineers to begin to work together to look for the most cost effective technology that can perform a given spec. Being the best at something is no longer a cage from which there is no escape later.

Technology components would be selected based on performance and cost of ownership (i.e. operations) later. Switching is ultrasimplistic if all we ever used to include it in our build routine was a drag-and-drop style method with universal, well-defined parameters. A customer’s ability to switch products to the best of breed through easy or simplistic means keeps pressure on vendors to actually be the best of breed. If the performance suffers or a new release is full of bugs, developers just switch it out through abstraction (not long, complicated scripts where everything has to go right, including a full moon on the third Tuesday of the month with a Gregorian chant in the background).

Impacts to the Bottom Line

The selection of DevOps tooling often is a cantankerous one in large organizations. Debates ensue about flexibility, extensibility and whether a given tool can be customized to accommodate the unique little snowflake of my development teams. Perhaps these debates are headed in the wrong direction. Perhaps instead of prizing and valuing the extensibility or customization a given tool affords, we should be looking closer at how that extensibility is achieved.

For an enterprise to make an unwitting choice to get into the script business, is effectively stating: “I will be maintaining the scripts I use to construct a build with as much rigor as I maintain the code these scripts are designed to create builds out of.” So, we just doubled our programming staff without realizing it—one half working on the application code and the other half working on the scripting code that “automates” the delivery of a build during construction. While overall speed may be increased, it comes at the cost of doubling already expensive headcount.

As an alternative, what if enterprise customers were asking tooling vendors how simplistic their tools are to use, and how standardized are their approaches. While we may not enjoy losing a degree of flexibility as we may be unable to customize a drag-and-drop method construct, the payoffs to the business are simplicity of use and scale at an enterprise level. Not only do I refrain from doubling headcount, I can actually free resources to focus on the disciplines and areas they enjoy most instead of performing script maintenance. Abstraction as a construct: From the hardware layer up through the application layer, it may be a driving fundamental that keeps DevOps scaling and lowering the cost of innovation for many years to come—but only if you begin to ask for it.

To continue the conversation, feel free to contact me.

September 5, 2016
DevOps: The Glue of Workflow

The best tool in the DevOps continuum is the sticky wrapper that holds it all together. It is the workflow that binds it and makes sense out of the term “continuous.” CIOs often are enamored with a build or deploy tool, but it is the workflow tool in every DevOps strategy encapsulating more than one function that is the real hero. Imagine having the perfect test automation scripts but no way to execute them outside of constructing a new build. Or imagine having the perfect CMDB integration scripts but no way to call them outside of the build process. Or even worse, imagine having the need for human approval before a deployment takes place (or is promoted), and no way to integrate it outside of a custom deployment process.

Lean DevOps Construction

Attempting to make very few tools perform these functions without a workflow engine ultimately forces automation engineers to create functionality within a tool it was never designed to perform. Conversely, when a workflow wrapper is available, build and deployment processes can remain small, elegant and consistent, allowing the workflow engine to integrate extended functionality into the continuous delivery cycle. Size of the automation constructs you create does matter: the smaller the better.

Using Lean methodologies in your DevOps services, it is better to break down automation constructs into the logical functions they are intended to provide. Build automation, for example, should not have to include extensions to turn monitoring functions on or off. Build automation also should not have to worry about the deployment process; its primary singular purpose is to construct a new build. Where that new build is then deployed becomes the single purpose of the deployment process. Attempting to automate the entire cycle of an application into one automation construct from end to end (instead of smaller blocks that work together) increases both size and risk (as well as length of time to execute) of that one event.

Dealing with Humans in the Automation

Human integration is the bane of DevOps processes. Every time you have to pause automation to obtain a human approval, you introduce time delays that cannot be predicted or occur with consistency. Imagine a deployment process that requires a human to press “yes” before continuing. The introduction of a human approval can sit idle on a tooling screen for hours during lunch, or unexpected days off, or vacations (when others are afraid to take responsibility for passing a substandard build, even for further testing). Deployment tools can integrate human responses, but this is not their forte. They are generally ill-equipped to use multiple contact methods, go up the management chain when needed or ping a mobile device when preferred by the user listed as responsible. Workflow tools are ideal for all of these human-contact logistics.

The challenge during DevOps service construction in utilizing a workflow wrapper is keeping human interaction to the absolute minimum required by outside regulating agencies and still achieve compliance. The alternative is to automate processes with such quality and consistency so that human interaction would merely become redundant and unnecessary. The workflow wrapper is not an excuse to simply automate what human interaction you have today. Instead, it is a highly functioning tool to allow you to redesign more-effective automation processes, limiting human interaction where possible. The goal is to build small, independent automation constructs that work together in a highly efficient manner.

Hold the Presses

Communication or notifications also can be handled in a workflow wrapper much better than in individual DevOps toolings. Imagine for a moment that I am a release manager, wanting to know the progress (or rather obstructions to progress) for my new application. To accomplish this notification process, it would have to be embedded within my build process (to know if it passed or failed). The alternative is that my workflow wrapper checks the build logs and can send me notification in the method I prefer (email, text message, an Instagram pic of Thelma and Louise over the cliff, etc.) In addition, I may not actually want to check the status of every build submitted, as that becomes noise after a while. I may want to know only when a build candidate has passed and is ready for promotion to a further testing environment. Since my workflow wrapper would be aware of this, it could automate the notification to me quite easily.

Now add the consideration of management chain to this notification process. If I am out on vacation (which I expect to be able to do, since DevOps enables my personal life), my manager or my sponsoring executive may want to be notified if my application is not going to production for some reason during my absence. This is a level of automation that most build or deploy tools are simply not designed for. They have no idea who I report to and who that person reports to. But that departmental notification escalation chains can be incorporated easily into a workflow wrapper, and execute as expected while I am away (including pinging my cell phone just in case I need to jump in and fix something).

Release Management Solutions

Most release management (RM) or release orchestration (RO) tools have at least minimal workflow capabilities built into their offerings. Whether this capability is enough for your organization or whether you need a more capable workflow engine to wrap around it is a decision only you can make. Most release management tooling is designed to put an elegant wrapper around build automation and deployment automation. It usually contains the ability to segment by line of business or product, as well as role-based access controls (RBAC) for permissions and auditory compliance. And it usually contains at least elementary notification capabilities. What RM excels at are human interaction and tracking progress, and so it does meet the needs of many companies.

When RM is not enough is usually when the DevOps services you provide begins to extend into ancillary services and secondary automation integration capabilities. Examples include monitoring systems integration, multiple testing tooling integration/evaluation, configuration management database (CMDB) integration and capacity-on-demand environment provisioning. Highly evolved notification systems tend to require an external workflow wrapper beyond the normal capabilities an RM tool offers. A solid workflow wrapper acts as the glue that can hold disparate DevOps tooling (either from different vendors or performing different functions) together with further super-automation processes above them like an overarching umbrella.

Impacts to the Bottom Line

Choosing to offer DevOps services without any form of a workflow wrapper is like going hunting with an excellent rifle and no bullets: a lot of capability that can never reach the potential it was intended for. On the other hand, obsessing about replicating current processes within a workflow tool is like hunting a bunny rabbit with a bazooka: way too much damage caused when trying to keep things small and elegant. The key is to balance the use of a powerful tool (workflow), with Lean thinking in automation processes and constructs. Keep the building blocks small and functionally oriented; limit human interaction; and use workflow for the functions it is best-suited for—connection and communication.

The technical capabilities a workflow wrapper will add to your DevOps solutions are substantial. Your services will be far more efficient and far more powerful. However, to truly assess the financial impact of a highly functioning workflow system, one must determine the value of time and decision-making. An elegant notification system can keep you informed without sending you countless emails you learn to ignore or flooding your text inbox with data that exceeds your cellular plan. In this way, notifications are meaningful, and you can begin to make meaningful business decisions based on the information you receive, only at the times you need to see it. What is that worth to you and your organization? Only you could really assess that.

To continue the conversation, feel free to contact me.

August 17, 2016
DevOps: Commoditizing Infrastructure

Hardware vendors face a quandary. How do you distinguish yourself as superior from your competitors if you are unable to continue to grow a distinct architecture as your competitive advantage? But this dilemma does not begin and end with only hardware vendors; operating system vendors and virtual machine vendors are on the same horizon. In an effort to “uncomplicate” hardware idiosyncrasies, our industry evolves efforts from open standards to least-common-denominator functionality for a “common” language or set of features, and over time technologies such as containers emerge. If I can abstract my application from the “uniqueness” of hardware vendors, from the complexity of configuration settings, and make only assumptions about basics such as CPU, disk, memory, etc., then I am free to focus on the application and allow my “hardware mitigation technology” to perform its function.

The Relationship Between App and Storage

But to accomplish this feat, I must agree to a basic set of features or functions I intend to call, such as making “x” amount of machine memory available to my application. Whether disk or RAM, I am looking to define what my application needs and then expect the container (or hardware mitigation/abstraction manager of my choice) to manage the actual allocation. Then the quandary begins. Asking for 1TB of disk space as an example is generic enough. But does my request include the ability to specify I am looking only for disks that spin at more than 7,500 revolutions per minute, or will I be saddled with 5,000 rpm disks? There is a clear implication in speed for my app.

Ultimately, as I sit and tune my code because my app has become “slow,” it could be something as innocuous as having its actual execution occurring in a cloud provider who uses the slower disk revolutions as opposed to the faster ones. Worse, it simply could be that the same cloud provider has a batch of the slower ones in one of its data centers where I wind up executing (only on a temporary basis), and I am never fully able to figure out why my app got “slow” for a while. That kind of random bad ju-ju requires the execution of a chicken to fix (I prefer Popeye’s, myself).

Different storage vendors might have additional peculiarities that make one or the other superior in speed, reliability or performance over their respective competitors. But the container’s main functionality is to provide “disk” to my app. Specifications of specific disk vendors or providers are the problem of the cloud provider, not the app developer, in theory. But hardware refresh can be pricey, and to make a claim that “all our disks” have rpm specifications above 10,000 will be meaningful only for about 10 minutes when the 12,000 rpm models come out. And of course, there are many more architectural designs in the engineering that are intended to boost speed and reliability. If these are not to be ignored, a decision must be made as to “who” has the responsibility to manage their usage (usually the cloud provider), and “who” will complain when things get slow (usually you the customer).

And what happens when “breakthroughs” occur in a specific technology? Solid-state hard drives, for example, are not exactly new now, but they once were. When you move your app from operating on a high-speed hard drive that spins to a chip-based hard drive the operates at the speed of the RAM it is based on, the difference in speed is an order of magnitude better. The inevitable question emerges, Will my container have the ability to specify RAM-based disks instead of traditional spinning disks, as part of the disk commodity it is providing to my app?

Since the difference in speed is so great, you would think it would come up with a way. You would equally expect that providing offline storage such as tape (from the age of the dinosaur) would have its own designation in my container structure (not be equated with offline spinning drives due to the speed differential). When you go to make a backup or restore, the difference between tape and hard drive becomes as apparent as the difference between solid-state and spinning drives on app execution. So when a platform evolves, my container mitigation must evolve with it. But to offer the uniqueness of vendor superiority requires my programmers to learn who to request (or what level of hardware specs they must know), which is the whole point of wanting a commodity in the first place: I don’t have to know.

Impact of Commodity on Other Components

This same dilemma can be exhibited in CPU vendors. Intel touts its superiority, AMD would disagree, and there may always be an upstart who wants to rival the current leader in any space. Lest we think Intel-based computing is the undisputed champion, quantum computing may upset that assumption (unless, of course, Intel corners that market as well). The point here is simply to ask: how many features, how many specifications, how much do we the programming team need to know about our platform provider to understand why and where our application runs better and worse? In the days before container-style ideology, hardware vendors touted their differences and their uniqueness as the reason they were better. Now what?

Then comes our virtual machine providers, and eventually our operating systems. If I want to truly abstract from my hardware layer, I must not call an OS-unique command that only Linux or only Windows can do. I must stick to functionality that both OS instances can reasonably understand and accommodate. While this least common denominator is great for extending where my app can run (on multiple types of hardware, in multiple types of OS installations), it has the same effect on uniqueness and potentially on competitive advantage. If the intent is to come up with a one-world OS, I am generally in favor, but our journey there is sure to have certain casualties in innovation that a single OS is suited for and its competitors are not.

The inevitable question becomes, How far down, or how far up, do we want to form commodity provisioning of computing capabilities? One could argue that the simplicity of the iPhone platform is a user-centric expression of commodity provisioning at the top of the stack. Drag-and-drop programming languages are yet another expression of bucketing computing capabilities into more simplistic expressions (that more people can understand and, therefore, utilize). Commonality and simplicity do increase audience. In many segments of our industry, this is the nirvana of sales to the public.

Impacts to the Bottom Line

DevOps focuses on enabling innovation. At its core DevOps is focused on software innovation. Let’s face it—changes to hardware platforms are generally less frequent. If those hardware platforms are married to a particular app (not serving a large number of apps on the same box, same virtual instance, etc.) then at least provisioning can be tuned one to one with the app. But with the desire to turn infrastructure into a generic commodity that application programmers don’t have to think about comes the fact that they stop thinking about them. Over time they will understand less and less about why things become slow, or work better, as the knowledge of uniqueness goes away.

Depending on your market focus, ubiquity is either your friend (as it touches consumers) or your enemy (if you are the hardware vendor attempting to maintain relevance over your competitors in an increasingly commoditized market). The questions I raise in this article are designed to focus on how to keep our trend toward hardware mitigation from stifling innovation in the very hardware platforms we depend upon. In any case, I welcome your thoughts on the topic.

To continue the conversation, feel free to contact me.

August 10, 2016
NodeSource Announces Containerized N|Solid Deployment for Kubernetes
Developers can now deploy enterprise-grade Node.js on Kubernetes

San Francisco, CA – July 26, 2016 – NodeSource, the Enterprise Node Company™, today announced at Node Summit 2016 the beta launch of its containerized N|Solid™ deployment for Kubernetes. With this release, NodeSource is offering the first version of Node.js with production-ready continuous performance and security monitoring within a Kubernetes cluster.

A recent Node.js Foundation survey found that 45 percent of developers use Node.js with container technologies such as Docker. As companies look to build out their next-generation architectures, Node.js and containerization are emerging as two key components for powering rapid technical innovation.

Kubernetes makes deploying, scaling and managing containerized services fast and simple. With an N|Solid deployment for Kubernetes, developers get real-time performance monitoring and vulnerability scanning of Node.js applications running in a Kubernetes cluster.

“Node.js is ideal for containers because it allows developers to quickly build, iterate and scale cloud-native APIs, apps and microservices,” says Aparna Sinha, Product Manager, Kubernetes & Google Container Engine. “It is great to see NodeSource offering a Kubernetes deployment for Node-specific monitoring and security, allowing companies to scale with confidence.”

N|Solid combined with Kubernetes orchestration provides developer and operations teams with:

Efficient scaling of microservice/distributed architectures

Node-specific performance insights and analyses

Continuous Node.js package security vulnerability scanning

Expedited deployment of new Node applications

Automated rollouts and rollbacks

Service discovery and load balancing

Horizontal application scaling

“We are seeing a popular trend emerging among our enterprise customers with the adoption of containerization for Node.js and Kubernetes for cloud orchestration,” says Joe McCann, CEO and co-founder of NodeSource. “Adding the powerful monitoring and security features of N|Solid, we are creating the trifecta for high-performance Node.js services.”

N|Solid is available for download as a native package (Linux and Mac), Docker images and now as a pre-configured Kubernetes deployment. For more information, visit <landing page>.

About NodeSource:

NodeSource is a technology company dedicated to delivering enterprise-grade solutions in support of a sustainable ecosystem for the open source Node.js project. We aim to drive and expand the Node.js ecosystem by providing best of breed solutions that specifically target the needs of the enterprise. Customers include NASA, HomeAway, Condé Nast, Craftsy, and other progressive Node.js adopters who are leading the charge in microservices via server-side JavaScript instead of maintaining legacy monolithic applications like Java. We are a Distributed by Design organization – our teammates report to work from all corners of the world: Asia, Australia, Europe, South America, as well as NY, SF and Austin. NodeSource is backed by RRE Ventures, Crosslink Capital and Resolute.vc and our AngelList Open Source Syndicate. For more information visit www.nodesource.com and follow @nodesource.
July 26, 2016