devopsy.kinsta.cloud

Category: Features

Henry Ford Discovered The DevOps Secret Sauce (and you can too).

DevOps Celebrates it’s 100th Birthday!

Yes, you read the title right – DevOps is officially 100 years old. Way back before Dr. Deming, Agile, Jez Humble or anything labeled “continuous”, the principles of DevOps were being created and refined. But like a rebellious teen, our current iteration of DevOps may be ignoring some of the guiding principles that have proven to be so successful in previous generations.

Think about the automobile industry circa 1905. These high-tech shops were pure DevOps as cross-functional teams collectively engineered, fabricated, assembled and tested every single automobile that was “released”. These DevOps forefathers strived for the same important ideals our current generation has set out to achieve – to increase release speed, improve release quality and gain a competitive advantage.

Back then, DevOps teams were super agile and could adapt quickly to the changing needs of their end-users. Quickly, teams at Buick, Oldsmobile and Cadillac began to discover the power of interchangeable components and automation (i.e., open source tools). Productivity and efficiency skyrocketed as a result.

But starting around 1912, Henry Ford began thinking beyond the prevailing brand of DevOps. Ford came up with is own DevOps secret sauce. Unlike the other DevOps teams of the era, Ford placed relentless focus on two key concepts: repeatable processes and standardized inputs/outputs. He recognized the lack of standardization resulted in low process repeatability which in turn raised cost and lowered quality. In 1913 he began to standardize processes and the inputs they require, thereby launching the very first Continuous Delivery pipeline for the auto industry. Once implemented, Ford’s new CD pipeline was assembling, testing and releasing new automobiles at a rate far exceeding anything previously thought possible.

What was true about delivering cars in 1913 is also true about delivering software today. Without highly standardized inputs, DevOps teams can not design repeatable processes. Without repeatable processes, automation is not possible. This means DevOps craftsmen will always be required to manually build, test and deploy software change.

Today a handful of select companies like Netflix, Facebook, Ancestry.com and Flickr have largely removed this type of craftsmanship from the software delivery process, replacing it with highly-repeatable processes and relentless automation. Some call these companies “Unicorns” because they seem to have magical powers. A few months ago I talked to the DevOps team at Ancestry.com. They have hundreds of fully automated delivery pipelines and production deployments happen on average about every six minutes with the same level of fanfare as the next new Ford rolling off a production line. These organizations build, test and deploy software change effortlessly by relying on Henry Ford’s two core ideals: standardized inputs and repeatable processes.

Here’s a quick story… John DiMarco, Chief Operating Officer at CEDAR Document Technologies, is very happy with his agile Development process. New development output is very strong, so strong in fact, that Delivery is quickly becoming the agile bottleneck. To combat this problem, he’s well on his way toward a complete transformation of CEDAR’s current Delivery methodology. When we first met, he equated the process of delivering software with the effort required to deliver “finely handcrafted Italian marble”. At that time, his Delivery process was able to produce a high-quality output but it was overly reliant on individual effort and creativity. Now, one of DiMarco’s strategic objectives for 2015 is to evolve his software Delivery process toward the kind of high-tech assembly lines found at Ancestry.com.

Assembly lines require highly standardized inputs in order to function efficiently. Therefore, a clear line between Development and Delivery must exist to define the discrete point at which Delivery inputs (i.e. software change) must tightly conform to a set of very strict standards agreed upon by all stakeholders. Without this formal entry point and standardized inputs, you’ll be handcrafting marble all the way to production.

Devops needs to take a cue from our fellow microservice advocates. Imagine a Service Oriented Process (SoP) where Development and Delivery are both first-class citizens in the larger agile machine. Both services have inputs and outputs, are bound by contracts and interface through a standard “API”. With this type of mindset, Development and Delivery services can evolve as needed without affecting each other, as long as all contracts are honored.

Payoff – The Unicorn Secret Sauce

Over the past six months, I’ve talked to over 50 CIO’s and VP’s of Dev/Engineering. A few are unicorns, others are well on their way, while most are just now plotting their course. Based on these conversations, and so many others with customers and DevOps thought leaders, I have identified the Unicorn Secret Sauce.

Standardization – There is one simple reason you must break your agile Develop/Deliver stream into two (or more) discrete services – Standardized Inputs/Outputs. Whether you define two big services (develop and deliver) or twenty micro services, each service along the pipeline should have documented inputs and outputs as well as detailed contracts between any two interacting services – in writing. While this step won’t be easy, it’s absolutely critical for two reasons:
Assembly lines require standardized inputs. With no standard inputs you are handcrafting Italian Marble.
When the definition of “done” can be objectively proven via satisfied contracts, pipeline throughput is no longer dependent on cooperation and the DevOps group hug (while that’s still awesome). Now, statistics can be generated, quality can be measured and performance can be objectively quantified and compared between and across services and teams. Welcome to the DevOps Party Dr. Deming!

Test-Driven-Design – The Development service must be able to hand off a standardized package that can be easily ingested by a downstream service without the need for creative thought or subjective analysis. This has always been pretty easy when it comes to the change package itself (code), however, the question of quality has always been a much tougher nut to crack. The best way around this is by adopting some form of TDD. It doesn’t matter which teams create and define quality tests – what matters is that success and quality have been implicitly defined and that proof can be easily achieved (manually or automatically) downstream. Without tests as a primary input to delivery, there can be no contract between services and we’re back to Italian Marble.

The Journey – It’s a Marathon not a Sprint

Clearly the metamorphosis from marble craftsman to unicorn will not happen overnight or this quarter – It took Ancestry.com two years to get there. But here is a proven roadmap that will get you from here to there:

Define your version 1.0 Service Oriented Process design. Pay careful attention to ensure inputs/outputs have been standardized and clear contracts are in place between “service API’s”.
Next implement basic delivery infrastructure (the assembly line operating system). This platform will orchestrate flow across/between services, gather data, and provide real-time status, reporting and analytics. This will get all teams and service providers on the same page and rowing in the same direction leveraging a unified world view.
Next, start eating the elephant one bite at a time by continuously replacing manual services with automated replacement versions still remembering to focus on inputs/outputs, API’s and contracts.

Over time, your body of automated quality tests will grow at a speed equal to your rate of change. Soon coverage will be broad enough to accurately predict release quality allowing manual release testing to quickly decline as automated test coverage (read confidence) increases.
Twelve to twenty-four months later…congratulations, you’re a unicorn!

No doubt, many will disagree with my perspective. I value every opinion and will learn from your shared ideas. Please express yourself and add your voice to the discourse with a comment below!

About the Author/Dennis Ehle

Dennis Ehle, is a pioneer and thought leader in continuous delivery automation and agile delivery methodologies. Dennis is passionate about helping agile teams dramatically reduce the transaction cost associated with delivering incremental change. His company, ClearCode Labs, does just that by helping organizations continuously deliver high-quality software releases more frequently merging proven methodologies with empowering tools and technology. Twitter: @DennisEhle

February 5, 2015
DevOps Video Rodeo: DevOps 101

It can sometimes be difficult as a DevOps newbie to get the straight dope on what DevOps is all about without ending up in the weeds with descriptions and tutorials full of specialized details. As you ramp up, that kind of material is invaluable. But when you’re new to the game, it gets in the way of clarifying the basics.

If you or one of your colleagues is looking into the basics of DevOps, we’ve got a good place for you to start. Today we’re launching a new video round-up feature we’re calling DevOps Video Rodeo and we thought there would be no better way to kick things off than to focus on DevOps 101. We’ve collected some of our videos meant for DevOps beginners. They explain what DevOps is all about, how continuous delivery can help IT deliver better business value and how organizations can begin their DevOps journeys. Some of these are new, some are oldies but goodies, and some are considered classic talks that helped drive the way the industry first started talking about DevOps.
So let’s get started.

DevOps Demystified – An introduction to the ideas that are driving DevOps
Now director of IT and operations at Chef, Ben Joyent has long been spreading the gospel of DevOps. When he was at Joyent a couple of years ago, he gave one of the most thorough talks breaking down DevOps for newcomers that still exists today. This is ideal for practitioners on either side of the aisle who want to dive deeper into the fundamental principles behind DevOps. It’s a long, but thorough primer.

What is DevOps? – In Simple English
If you’re looking for a short summary to pass along to the upper-level, line-of-business managers who you’ll need to help support a DevOps transformation, this may be it. Don’t let the slick, vendor-produced nature of this video turn you off. Rackspace kept to the basics without trying to hide a sales pitch in there. It does an especially good job explaining the business value of DevOps.

What is DevOps
Like the previous video, this is another DevOps description that clocks in under 10 minutes, but this one is particularly good at offering explanations of how developer and sys admin roles change under the DevOps model. This is particularly good one for new DevOps acolytes to send over to colleagues they’d like to convince to join forces with them. It also offers a quick sketch of how the infrastructure-as-code model works. Plus, its quick-and-dirty, back-of-the napkin style is kind of endearing.

The (Short) History of DevOps
If you’ve ever wondered where the DevOps movement came from and who helped refine the ethos, this is about as concise of a history as you’re going to get. Nicely done by Damon Edwards of DevOps Café podcast fame.

10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
This is a classic talk, one that many DevOps early adopters point to as a pivotal demonstration that started the snowballing momentum going with a very real case study on how DevOps can improve speed and flexibility. DevOps practitioners may have moved far beyond 10 deploys a day, and John Allspaw and Paul Hammond have since moved on to other organizations, but this stands as an excellent introduction to DevOps.

February 3, 2015
A DevOps Metamorphosis Echoes Through Reverb.com, Part Two

Last time, Dear Reader, we discovered how the Reverb.com e-commerce platform, which connects musicians who want to buy and sell gear sought to automate its workflow software development process, speeding deployment and bug fixes while satisfying customers who want to trade in an online environment that is as efficient and error-free as possible. Now, the conclusion of how Reverb.com adopted DevOps to achieve its goals.

A Round Trip Through The Reverb.com CD Process

Today, Reverb.com focuses the DevOps mindset on its development process using Chef [https://www.chef.io/chef/] for configuration management and Jenkins [https://www.cloudbees.com/jenkins/about] for continuous delivery and infrastructure automation tasks. Developers use Capistrano [http://capistranorb.com/] for rolling out new code changes and running database migrations, CircleCI [https://circleci.com/] for unit testing code, and DataDog [https://www.datadoghq.com/] for metrics and monitoring. “We also use Logstash [http://logstash.net/], ElasticSearch [http://www.elasticsearch.com/about/], and Kibana in unison for log aggregation and visualization,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com.

In a typical deployment for Reverb.com, an engineer clicks the build button in Jenkins, which triggers a Chef converge on all the web heads. This rolls out any configuration changes that the developers have scheduled to go out, such as NGINX configuration updates or app config updates.

Once this is finished, the system signals the application to perform a Zero Downtime Deploy in which the application reloads without dropping any requests. “If this process finishes successfully, Jenkins kicks off a code deploy using Capistrano and simultaneously deploys new code to our web application servers. If a database migration is scheduled to be run, that will also happen on one of our worker boxes,” says Enger.

This may sound like a standard deployment so far. But that’s where DataDog and the ELK stack come into play. “That’s where we really gain the benefits of CD/CI and DevOps,” says Enger. Every time the development team converges Chef, they use a Chef Handler to submit an event to DataDog. And once Capistrano starts and finishes, another event goes to DataDog. The team has configured Logstash to watch the application logs on the web heads and to send an event to DataDog once the system renders an error. “The single point of information now becomes DataDog because we can see into our config and code changes and immediately know when a bad change goes out the door,” says Enger.

The last thing the development team looks at is the performance logs in Kibana, to see how the application is performing based on the logs it is generating. “With the ELK stack in place, the team can answer questions like: Are there any 500 errors? If so, how many and how often are they occurring? Did development increase or decrease application response times? Do people like the new feature rolled out at /my/feature?,” says Enger.

Hurdles

The biggest hurdle in moving to DevOps is the mental shift you have to make from that of a traditional systems administrator or developer to a more DevOps-based mindset. This means the operations side learns coding and development learns about load balancers.

“If we propagate continuous change in our computer systems, we should also expect to be continuously changing our mental models to adapt to new ways of doing things,” says Enger.

Benefits You Can See Through

Transparency is the number one thing the Reverb.com developers gained from this new CD approach. “Capistrano tells HipChat [https://www.hipchat.com/], “Hey, code is going out!” and I know to pull up my dashboards in Kibana and DataDog to watch for any anomalies,” says Enger. Humans can’t always be expected to communicate to each other efficiently. Computers help the Reverb.com team communicate across the organization through programmed notifications as to when changes go out.

Code deploys are now predictable and automated. No more worrying about a failed deploy due to a bad connection. “When failure does occur, we are able to open up the Jenkins Build output and find the error pretty quickly,” says Enger.

A Nice Surprise

One really positive result that surprised the development team after implementing CD/DevOps is that they learned a lot about their user behavior and traffic patterns. “Now we know that 4.30am is the slowest period on the site,” says Enger.

The team knows exactly when is the best time to roll out big configuration or code changes. They know immediately how deployments affect users and especially whether they ever break critical paths in the site, such as the checkout or signup functions. And the automated approach that DevOps provides enables them to fix bad changes within minutes of their original deployment.

The Affirming Roar of Fan Feedback

Reverb.com recently introduced more automation and monitoring to prevent site downtime and abuse by bots. The feedback: Customers don’t care about DevOps; they care about sites being operational and efficient. But it’s DevOps that makes that happen!

February 2, 2015
DevOps: Where are we with that?

There’s a great scene in the Kids in the Hall movie Brain Candy, where the CEO of a major pharmaceutical company informs his executive team that the board has recommended a massive restructuring plan that will cut their research outlay by 60%. Only seconds after first mentioning this undertaking, the CEO turns to his yes man, Marv, and asks:

“Where are we with that?”

Marv responds: “The thing you just mentioned just now?”

And after getting an affirmative response, Marv foolishly, yet not surprisingly answers:

“Oh, we’re on top of that, Don.”

It’s not hard to imagine this same scene playing out in real life where the restructuring being recommended is that found in DevOps. Marv and his fellow executives have agreed to a restructuring plan that’s going to dramatically affect the flow of work and the lives of many people at their company, but without having any real idea of why this plan is needed, how much it will cost, or even where to begin. They’ve just been asked to reduce the cost of their product by 60%, and to begin, well, five seconds ago.

Sound familiar?

There was a time when DevOps was possibly an even bigger buzzword than it is today, but that doesn’t mean it has gone anywhere, or lost popularity. In fact, its adoption has only increased since the term was coined, and it can be found everywhere from the smallest shops—all the way to massive enterprises.

So why is it still so hard to successfully pitch DevOps to executive leaders?

We’ve written before that the ambiguity around DevOps’ definition is one of the primary causes of the confusion surrounding the term, and the debate between whether it’s a “mindset” “practice” or “cultural shift” will likely never die (just don’t let anyone tell you it’s a “tool”).

Rather than argue the buzzword’s definition, many at last year’s DevOps Enterprise Summit argued that the best way to promote a DevOps initiative at your organization, especially to your executive leaders, is to not call it DevOps at all.

Since I’ve always liked the term DevOps, it’s taken me a while to come around to this concept, but I think those esteemed speakers are on to something.

Vendors, analysts, the media, and other stakeholders have all had enough time to develop their own personal assumptions around DevOps. Maybe it was first proposed to them by an overblown consultant, or it was part of the failed vision of a former executive who is now on the outs. All it takes is one negative DevOps word association within your leadership and your noble pursuit of its buy-in could be quickly derailed.

So if you don’t call it DevOps, what do you call it? “Lean IT?” “Agile Innovation Processes?” “Continuous Flow Delivery?” “ Industrialized Software Development?”

It’ll take you more words, but I’d recommend using DevOps’ results, and not any perceived buzzworthiness of the term alone, as its primary selling point. While “more words” means “more time”—something your e-staff likely has very little of—you’ll both get and hold their attention much quicker than just saying “DevOps” and expecting the applause and champagne to start flowing.

Gene Kim’s definition of DevOps supports this concept, and in a recent interview he remarked:

I would define DevOps by the outcomes. In my mind, DevOps is those set of cultural norms and technology practices that enable the fast flow of planned work from, among others, development, through tests into operations while preserving world class reliability, operation and security. DevOps is not about what you do, but what your outcomes are.

For those who are somehow just learning about DevOps, or for those who have known of its existence for years now, but who have been hesitant to pitch it to their own leadership, the good news is that the outcomes of DevOps are being proven not just at big names like Disney, GE, and Netflix—they’re even being felt by those who’ve implemented it at non-Dev or Ops fields like Security, and even Design.

Your VPs of App Dev and QA will likely be impressed by your mention of accelerated feedback loops and reduced defects. Your IT Operations leadership, no matter what distrust they harbor for app dev, would have a hard time arguing against increased visibility into the delivery pipeline, and better quality when the software lands in production. Then when the CEO hears about faster service delivery and improved time to market, you may as well start sketching out your plans for that corner office.

(Editors note: Author’s promise of a corner office is his own, and is not guaranteed by Skytap)

About the Author/Noel Wurst

Noel Wurst, Managing Editor at Skytap, Noel is responsible for setting and executing Skytap’s content marketing strategy. That includes overseeing the creation and distribution of high-quality, targeted content, and managing the team and external resources associated with those efforts. By leveraging deep online marketing, project management, and editorial experience, Noel plays a crucial role in helping generate and nurture leads with compelling content. You can follow Noel and Skytap at @NoelWurst and LinkedIn and @Skytap, Skytap’s LinkedIn

January 30, 2015
Five fruitful DevOps business drivers gaining speed

In talking with those in the enterprise trenches working to make DevOps a reality – admins, developers, CIOs, and business leaders who get it and are trying to help transform their enterprise culture and workflow to be more DevOps oriented – a number of key drivers surfaced from these discussions.

What drivers are they? Below are the Five Fruitful DevOps Business Drivers Gaining Speed right now that many say are currently acting as a catalyst for pushing DevOps forward in their organization.

These are in no particular order, nor is this list meant to be all inclusive, but these benefits are what was top of mind when speaking with enterprise IT over the past couple of months:

1. Clearing application backlog. Enterprises are busier than ever before, and more employees, partners, and customers are requiring more apps to complete their day-to-day work. These demands, plus the pressure in recent years to build more apps for more form factors is pushing many development teams further and further behind schedule. They find themselves buried in application backlogs.

Increasingly, seeing the speed and reduction in errors, more enterprises will clear through their app backlog with DevOps.

2. Improving mobile development workflow. The needs of mobile users are constantly changing, as are mobile devices and mobile apps. Mobile devices and operating systems are a never-ending river of new capabilities, updates, and integrations. The evolution is so rapid that traditional mobile development and operations teams have trouble keeping up.

It’s clear that creating dedicated mobile development teams isn’t working for all organizations that seek to maximize their productivity. But DevOps and continuous delivery and integration strategies make it more likely that development teams will be able to build and deliver these apps as fast (well, almost) as their enterprises demand.

3. DevOps is mainstream. The debates over whether DevOps is real, or not, have come to a close – except for a few who may have kept themselves buried too deeply in an underground data center for too long. The question about whether DevOps is having a positive impact on enterprise outcomes is over.

The discussion now switches, most everywhere, how to move rapidly to a DevOps organization, or how to improve the effectiveness of existing DevOps efforts.

There will be more organizations working to collaborate more effectively, increase automation, and integrate teams to increase effectiveness. DevOps and collaborative toolsets and movements such as ChatOps will also mature and help accelerate the transition.

4. Design teams, work more closely with DevOps teams.

It’s surprising that DevOps isn’t more entrenched in enterprise mobile design. One of the key benefits of DevOps is the ability it provides for enterprises to design and build applications more quickly, and be able to react more swiftly to market demands. It’s why DevOps can be so powerful for mobile teams. More effective mobile app development, testing, and operations means more effective business integration and focus on designing apps that the business needs. “all DevOps does is make sure your app works well. It has nothing to do with it being popular,” says Katz.

Some have called this focus on building better UX/UI functionality DesignOps, and more organizations are working to bring the same levels of collaboration and team integration with design and development as they’ve down with operations and development.

5. DevOps and Better Feedback from Business Leaders

With the enterprise embracing DevOps, certainly that is a culture change but the toolsets and the application service delivery platform is also changing, and with new toolsets and new workflows and tighter levels of collaboration and workflows means new assumptions of how business management can or will work with IT. Savvy businesses recognize the opportunity and are already engaging with their IT departments.

It’s the manifestation of what Gene Kim called The Third Way: Culture of Continual Experimentation and Learning:

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.

We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.

As part of the culture of “The Third Way” business leaders need to help guide IT on some of that risk taking – where it makes sense, were it doesn’t, when those risks could affect direct business outcomes.

This will help DevOps keep IT more aligned with business goals than was largely possible in large organizations previously.

January 29, 2015
A DevOps Metamorphosis Echoes Through Reverb.com, Part One

“Reverb.com is an e-commerce platform that connects musicians, enabling them to buy and sell used gear and instruments,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com. As an ecommerce site, it is critical to Reverb.com that its workflows such as checkout, offers to purchase, and messaging work in tandem with its selling workflows, which include creating new listings and managing orders.

The Reverb.com team also develops a free, comprehensive online price guide and a complex in-house content and curation system that allows them to hand pick and recommend the best gear for their customers. In the Reverb.com marketplace, all software components must run smoothly for customers to have the optimum research, buying, and selling experience when conducting trade on musical equipment.

To create those components, Reverb.com developers code primarily in Ruby on Rails on Macs. The etailer runs Linux in its production environment. “Our backend stack includes PostgreSQL for our database, ElasticSearch powering our search engine, and Redis for queueing and key value storage,” says Enger.

Not So Good, Old-Time Deployment

Prior to DevOps approaches, Reverb.com deployed code from individual user workstations. “For configuration changes to our infrastructure we had to use SSH to connect into production. All while our fingers were crossed, of course,” says Enger. This loose-knit deployment process presented Reverb.com with some obvious challenges.

“If a deploy failed due to an engineer being on a bad internet connection, we could potentially have had two updated hosts while three were still running an older version of code,” says Enger. This was especially problematic because these deployments often included database migrations.

There was also no real way to know when a deployment failed or who started the deployment. “The process was devoid of any deployment log records, so errors and anomalies were harder to spot when the deployment was swallowed in the developer’s terminal,” says Enger.

These risks made it possible for Reverb.com to face an even bigger risk: a lapse in faith on the part of its customers. Reverb.com’s reputation as a marketplace hinges on the perception of trust among its customers. Every aspect of the organization generates trust. From how Reverb.com interacts with customers as people, to how its systems present to users, the firm wants to exude confidence. “If we experience outages and slow performance, we may erode our trust with our audience,” says Enger.

Goals For DevOps

Reverb.com had many goals for DevOps in the hope that it would meet their challenges to fast, scalable, reliable delivery of code and configuration changes. Reverb.com needed to know when a change was going out. The firm needed to be able to deploy at a moment’s notice.

“All processes must be behind the click of a button or an SCM post/pre trigger,” says Enger. All Reverb.com deploys must be logged and graphed. Scaling the team and the business should become easier if the teams have quick, repeatable processes that allow them to push new features and bug fixes quickly onto the new DevOps platform. This should allow for a scalable and dynamic workflow for the Reverb.com marketplace so it can be nimble, adapting to the users’ needs as they arise.

Reverb.com’s Reasoning: An Automation Proclamation

Reverb.com made its move to DevOps on the basis that the more the automation, the less the likelihood of human error. From that, Reverb.com development staff should create fewer issues for its customer base as consumers browse the website, make purchases, and sell hardware.

“It’s extremely important to look at root causes here: if we only treat the symptoms every time the site slows or goes down, we will be playing Whac-a-Mole forever,” says Enger. By using the DevOps mindset, initiating repeatable and automated deploys, and measuring performance and application metrics in rea-time, Reverb.com developers should be able to trace when something was released, how it affected users, and whether customers are having a poor experience as a result.

In the hopes that this would equip them to quickly pinpoint issues and fix root causes, the band of coders at Reverb.com set out on the road to DevOps, fired off its first continuous delivery power chord, and awaited feedback.

Conclusion: Cliffhanger

Tune in next time when devopsy.kinsta.cloud reveals how Reverb.com adopted DevOps and used it to meet these development challenges and goals in the exciting conclusion of “A DevOps Metamorphosis Echoes Through Reverb.com”.

January 28, 2015
Docker tinkers with Docker Hub for performance improvements

Docker’s strengthening its infrastructure components to accommodate the volume of use and wide range of use cases that inevitably crop up from such hockey-stick growth. The latest improvements come by way of refinements made to Docker Hub over the last month, according to an announcement made by the company yesterday.

The Docker Hub team reported that many of the changes were precipitated by performance issues experienced at the end of 2014, following the addition of security enhancements to prevent brute force login attempts. They reported that the enhancements added “overhead to the login process.”

“Under normal load there wasn’t much difference, but as the number of login attempts increased the site performance started to degrade,” they wrote. “The decreased performance affected not just the Docker Engine logins, push, and pulls, but also the Docker Hub website as a whole.”

In the process of identifying immediate issues causing the problem, Docker’s engineers audited the entire system and is in the middle of rearchitecting the authorization, push and pull system to improve performance and isolate systems for a bulkhead effect, ensuring that there aren’t cascading problems.

In the immediate term, some of the fixes already made include bulking out infrastructure capacity, including a tripling of servers in the Docker cluster for four-times the compute power and a production database upgrade for twenty times the RAM. It also improved login infrastructure to include added database indexes and more caching. The team also tuned queries for speed and fixed an issue that was causing database locks. Docker also switch its background worker queue broker from Redis to RabbitMQ and moved background workers to their own server cluster. Additionally, improvements were made to monitoring and metrics gathering, as well as to search functions.

Meanwhile, down the line they expect to make further improvements that include API throttling, breaking up larger services to improve scalability, replacing the brute force prevention library with a better performing library and adding API metrics and monitoring to head off performance issues earlier.

“We have made some good progress, but we are not finished yet, and we still have a lot more planned,” the Docker Hub team wrote. “We will continue to make improvements with every release, so that Docker Hub only gets faster and more reliable as time goes on.”

Such hiccups as the ones in December are understandable given the meteoric growth of the Docker community. Even just six months ago, the total number of downloads of Docker were under 3 million. By December that number had risen to 100 million. In spite of some questions about the security of Docker containers, Docker’s trajectory seems promising. Just in September the company closed on $40 million in series C investment from Sequoia and over the past few months it has been announcing a number of high-powered partnerships, including those with VMware, Microsoft and AWS.

Docker’s most recent partnership was announced last week when it teamed up with Google so that the online giant could offer a private registry service for Docker on its cloud platform. Currently in beta, the Google Container Registry will provide Google cloud customers with secure hosting, sharing and management of private container repositories that will include strong access control and server-side encryption.

January 27, 2015
CFP for DevOps Connect: DevOpsSec @RSA Conference is now open
The world comes to talk security at RSA Conference. Every year in San Francisco the RSA Conference is the biggest information themed security conference in the world. This year there is a new addition to RSA Conference. DevOps Connect: DevOpsSec will be held on the Monday of RSA week at the Moscone Center. Attendance is free and open to all RSAC registered attendees. But you must pre-register at the time of registration.

The keynote and co-chairs for the DevOps Connect program are author and DevOps pioneer Gene Kim of IT Revolution Press and Joshua Corman, CTO of Sonatype and Co-Founder of RuggedSoftware.org and IamTheCavalry.org Gene and Joshua first introduced “Rugged DevOps” at RSAC 2012 and have been building collaborations and bridges between those worlds ever since.

Additionally, Mark Miller and TheNEXUS are co-producing this show along with devopsy.kinsta.cloud and of course with the sponsorship of RSA Conference.

“DevOps is one of the fastest rising trends in technology with a tremendous impact on the information security landscape,” said Linda Gray, General Manager, RSA Conference. “We are excited to hear the direction and insights speakers such as Gene Kim and Josh Corman will bring to this event and to our RSAC attendees.”

The call for papers is now open to speak at this event. The link for the CFP is: http://ow.ly/I27jQ . If you would like to speak at this event, head over now and submit your proposal. Josh and Gene have given a lot of thought in designing the CFP in terms of what the agenda for the conference will be, so please read it carefully before submitting.

What are they looking for? Here is the pre-amble to the CFP which should help give you a good idea:

DevOpsSec 2015: Call for Speakers

Whether we like it or not, DevOps is upon us. The DevOps Enterprise Summit in October 2013 showed us that it’s not just for Internet startups, but it’s for large complex, and highly regulated organizations such as Disney, Capital One, Macy’s, GE, Nordstrom, Target, U.S. Citizenship and Immigration Services at DHS, and many more.

DevOps and “continuous everything” is profoundly changing the entire software industry. Our goal is to show how the fittest in InfoSec are adapting (and how we all must adapt). The purpose of the DevOps Connect: Security Edition is to assemble the best practitioners who are forging how InfoSec integrates into DevOps, to grow the tribe, and to build relationships and collaborations with DevOps leaders and luminaries outside of the security “Echo Chamber”.

Gene and Josh first introduced “Rugged DevOps” to RSAC 2012 and have been building collaborations and bridges between those worlds ever since. For 2015, we’ve asked them to host this day-long conference with the following goals:
- bring the best and brightest of the DevOps and InfoSec communities together
- share and codify effective patterns and practices, for mutual growth and benefit
- for new-comers, demystify the DevOps foundational concepts that every InfoSec practitioner needs to know
- capture the failures and toughest problems facing our industry, and mobilize to solve them
We are looking for speakers to discuss the unique challenges of adopting DevOps in the enterprise, and the ways to overcome them. Specific topics we’re looking for include, but are not restricted to:
- How did you get senior management buy-in, what did you do, and how did you quantify the business value?
- How did you overcome objections inside the organization?
- How did you design and implement effective controls to mitigate security risks?
- Which business area (and application) did you start with, why, and what did you do?
- What were the top challenges, and how did you overcome them?
- What were your top mistakes, and what advice would you give as a result?
- In-progress stories (and even nightmares) are welcome!
Other submission guidelines:
- Talk slots will be 30 minutes, of which a few minutes should be reserved for Q&A.
- Attendees must be able to use the bulk of the presentation’s lessons without buying or using a particular tool or service.
- While we love consultants and vendors (they’re some of our favorite people!), we are looking for stories from the organizations adopting these practices. We are prioritizing practitioner presentations (or joint ones with their game changing consultant/vendors).
- You can submit up to three talks for the conference. No spamming.
- Each submission should end with a slide communicating three things:
  1) A specific challenge you could use help with (people, process, or technology).
  2) A specific offer of how you could help other attendees (people, process, or technology).
  3) If you could wave a magic wand and get any change you want from the DevOps and “continuous everything” toolchains you depend upon, what would it be and why?
The closing date for submissions is midnight Pacific time, February 15, 2015.
```
SUBMISSIONS DEADLINE: MIDNIGHT PACIFIC TIME, FEBRUARY 15, 2015
```
```
LINK:REGISTRATION FORM
```
Thanks for your interest in DevOps Connect: Security Edition and we look forward to reading your proposal!

Thank you!
Josh Corman, Gene Kim
January 26, 2015
Google Hangout: InterConnect 2015 preview from IBM & Nationwide on #DevOps
Join us for a Google Hangout* with a distinguished panel of guests as we discuss: Adopting DevOps for Two-Speed IT.

With IBM InterCnnect 2015 just a month away, three featured presenters will give us an Interconnect preview

*Google Hangouts are FREE. To participate in a Google Hangouts LIVE, a Gmail or Google Profile is required.

Date: Wednesday, January 28th, 2015

Time: 3pm eastern standard time
```
REGISTRATION LINK:(NO REGISTRATION REQUIRED*) 
```
About Our Guests:

Rosalind Radcliffe, our guest from @IBM

Distinguished Engineer – Chief Architect for CLM and DevOps, Member of the Academy of Technology

Carmen DeArdo, our guest from @Nationwide

Director – Tools and Technology Leader

Sanjeev Sharma, our guest from @IBM

IBM Worldwide Lead – DevOps Technical Sales (Executive IT Specialist),

Author of DevOps for Dummies
January 26, 2015
The role of middleware in DevOps

Middleware has a very important and strategic role in DevOps, according to Ben Henshall, middleware business unit manager at Red Hat Australia and New Zealand. ‘Rapid everything’ is part of DevOps culture, so the bloated, proprietary middleware with roots in the late 1990s and early 2000s – the pre-cloud era – is a poor fit.

He pointed to an unnamed government organization that is limited to two major code releases per year, largely because of the complexity of its middleware and associated processes, which means systems have to be brought down completely for a weekend to deploy the updates.

This is in stark contrast with current thinking, and Henshall pointed to the frequently-cited 2009 presentation by John Allspaw and Paul Hammond http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr that described how Flickr managed 10 full deployments during a typical day.

Modern middleware is designed with cloud deployments in mind, including a multitenant architecture and a small resource footprint that minimizes infrastructure requirements. The best examples are based on Open Source projects with many contributors and extensive testing, and are developed and tested in the cloud, said Henshall, presumably with Red Hat JBoss Enterprise Middleware in mind.

It may be necessary to re-architect applications in order to decouple them from old-school middleware so a more modern replacement can be used, but technologies such as containerization can simplify the task, he suggested. While the right middleware and the adoption of Platform as a Service (PaaS) help – along with technologies such as OSGi (Open Service Gateway Initiative, which allows updated Java code to be deployed without stopping and restarting a system, among other things) as implemented in Apache Karaf (which is part of Red Hat JBoss) – but it is important to design applications so they can be maintained without outages, he said.

PaaS has seen a rapid rise in popularity as it makes it easier for developers to build and test new logic, and for operations staff to manage multiple systems without customization and with the benefits of automated releases. For example, Cisco has released more than 240 applications in around 12 months by using Red Hat OpenShift PaaS, said Henshall, noting that it can run on premises or be consumed as a service run by Red Hat.

The automation provided by PaaS fits in with the operations goal of running stable, reliable and secure systems, while reducing the need for “operations ninjas” who are potential single points of failure in the event of illness or resignation.

Henshall said OpenShift customers include a tertiary education institution that adopted OpenShift to speed the development and deployment of new applications and can now provision new systems within minutes, a software vendor that has reduced the time to market for new initiatives, and a major Commonwealth Government department that has been able to meet internal requirements concerning security, supportability and risk reduction since adopting the PaaS.

Being able to support innovation in the shortest time and with the least risk “is such a huge thing” for organizations, he said, especially when it can be combined with dramatic costs reductions and easier compliance.

Vendors such as Red Had provide products and services to support the DevOps culture, he said, noting “our software is developed by people facing these issues.”

About the Author/Stephen Withers

Stephen Withers is one of Australia¹s most experienced IT journalists, covering everything from gadgets to enterprise systems. In previous lives he has been an academic, a systems programmer, an IT support manager, and an online services manager. Stephen holds an honours degree in Management Sciences, a PhD in Industrial and Business Studies, and is a senior member of the Australian Computer Society. You can see his Linkedin profile here: http://au.linkedin.com/pub/stephen-withers/0/222/687/

January 23, 2015