devopsy.kinsta.cloud

Author: contributor

Henry Ford Discovered The DevOps Secret Sauce (and you can too).

DevOps Celebrates it’s 100th Birthday!

Yes, you read the title right – DevOps is officially 100 years old. Way back before Dr. Deming, Agile, Jez Humble or anything labeled “continuous”, the principles of DevOps were being created and refined. But like a rebellious teen, our current iteration of DevOps may be ignoring some of the guiding principles that have proven to be so successful in previous generations.

Think about the automobile industry circa 1905. These high-tech shops were pure DevOps as cross-functional teams collectively engineered, fabricated, assembled and tested every single automobile that was “released”. These DevOps forefathers strived for the same important ideals our current generation has set out to achieve – to increase release speed, improve release quality and gain a competitive advantage.

Back then, DevOps teams were super agile and could adapt quickly to the changing needs of their end-users. Quickly, teams at Buick, Oldsmobile and Cadillac began to discover the power of interchangeable components and automation (i.e., open source tools). Productivity and efficiency skyrocketed as a result.

But starting around 1912, Henry Ford began thinking beyond the prevailing brand of DevOps. Ford came up with is own DevOps secret sauce. Unlike the other DevOps teams of the era, Ford placed relentless focus on two key concepts: repeatable processes and standardized inputs/outputs. He recognized the lack of standardization resulted in low process repeatability which in turn raised cost and lowered quality. In 1913 he began to standardize processes and the inputs they require, thereby launching the very first Continuous Delivery pipeline for the auto industry. Once implemented, Ford’s new CD pipeline was assembling, testing and releasing new automobiles at a rate far exceeding anything previously thought possible.

What was true about delivering cars in 1913 is also true about delivering software today. Without highly standardized inputs, DevOps teams can not design repeatable processes. Without repeatable processes, automation is not possible. This means DevOps craftsmen will always be required to manually build, test and deploy software change.

Today a handful of select companies like Netflix, Facebook, Ancestry.com and Flickr have largely removed this type of craftsmanship from the software delivery process, replacing it with highly-repeatable processes and relentless automation. Some call these companies “Unicorns” because they seem to have magical powers. A few months ago I talked to the DevOps team at Ancestry.com. They have hundreds of fully automated delivery pipelines and production deployments happen on average about every six minutes with the same level of fanfare as the next new Ford rolling off a production line. These organizations build, test and deploy software change effortlessly by relying on Henry Ford’s two core ideals: standardized inputs and repeatable processes.

Here’s a quick story… John DiMarco, Chief Operating Officer at CEDAR Document Technologies, is very happy with his agile Development process. New development output is very strong, so strong in fact, that Delivery is quickly becoming the agile bottleneck. To combat this problem, he’s well on his way toward a complete transformation of CEDAR’s current Delivery methodology. When we first met, he equated the process of delivering software with the effort required to deliver “finely handcrafted Italian marble”. At that time, his Delivery process was able to produce a high-quality output but it was overly reliant on individual effort and creativity. Now, one of DiMarco’s strategic objectives for 2015 is to evolve his software Delivery process toward the kind of high-tech assembly lines found at Ancestry.com.

Assembly lines require highly standardized inputs in order to function efficiently. Therefore, a clear line between Development and Delivery must exist to define the discrete point at which Delivery inputs (i.e. software change) must tightly conform to a set of very strict standards agreed upon by all stakeholders. Without this formal entry point and standardized inputs, you’ll be handcrafting marble all the way to production.

Devops needs to take a cue from our fellow microservice advocates. Imagine a Service Oriented Process (SoP) where Development and Delivery are both first-class citizens in the larger agile machine. Both services have inputs and outputs, are bound by contracts and interface through a standard “API”. With this type of mindset, Development and Delivery services can evolve as needed without affecting each other, as long as all contracts are honored.

Payoff – The Unicorn Secret Sauce

Over the past six months, I’ve talked to over 50 CIO’s and VP’s of Dev/Engineering. A few are unicorns, others are well on their way, while most are just now plotting their course. Based on these conversations, and so many others with customers and DevOps thought leaders, I have identified the Unicorn Secret Sauce.

Standardization – There is one simple reason you must break your agile Develop/Deliver stream into two (or more) discrete services – Standardized Inputs/Outputs. Whether you define two big services (develop and deliver) or twenty micro services, each service along the pipeline should have documented inputs and outputs as well as detailed contracts between any two interacting services – in writing. While this step won’t be easy, it’s absolutely critical for two reasons:
Assembly lines require standardized inputs. With no standard inputs you are handcrafting Italian Marble.
When the definition of “done” can be objectively proven via satisfied contracts, pipeline throughput is no longer dependent on cooperation and the DevOps group hug (while that’s still awesome). Now, statistics can be generated, quality can be measured and performance can be objectively quantified and compared between and across services and teams. Welcome to the DevOps Party Dr. Deming!

Test-Driven-Design – The Development service must be able to hand off a standardized package that can be easily ingested by a downstream service without the need for creative thought or subjective analysis. This has always been pretty easy when it comes to the change package itself (code), however, the question of quality has always been a much tougher nut to crack. The best way around this is by adopting some form of TDD. It doesn’t matter which teams create and define quality tests – what matters is that success and quality have been implicitly defined and that proof can be easily achieved (manually or automatically) downstream. Without tests as a primary input to delivery, there can be no contract between services and we’re back to Italian Marble.

The Journey – It’s a Marathon not a Sprint

Clearly the metamorphosis from marble craftsman to unicorn will not happen overnight or this quarter – It took Ancestry.com two years to get there. But here is a proven roadmap that will get you from here to there:

Define your version 1.0 Service Oriented Process design. Pay careful attention to ensure inputs/outputs have been standardized and clear contracts are in place between “service API’s”.
Next implement basic delivery infrastructure (the assembly line operating system). This platform will orchestrate flow across/between services, gather data, and provide real-time status, reporting and analytics. This will get all teams and service providers on the same page and rowing in the same direction leveraging a unified world view.
Next, start eating the elephant one bite at a time by continuously replacing manual services with automated replacement versions still remembering to focus on inputs/outputs, API’s and contracts.

Over time, your body of automated quality tests will grow at a speed equal to your rate of change. Soon coverage will be broad enough to accurately predict release quality allowing manual release testing to quickly decline as automated test coverage (read confidence) increases.
Twelve to twenty-four months later…congratulations, you’re a unicorn!

No doubt, many will disagree with my perspective. I value every opinion and will learn from your shared ideas. Please express yourself and add your voice to the discourse with a comment below!

About the Author/Dennis Ehle

Dennis Ehle, is a pioneer and thought leader in continuous delivery automation and agile delivery methodologies. Dennis is passionate about helping agile teams dramatically reduce the transaction cost associated with delivering incremental change. His company, ClearCode Labs, does just that by helping organizations continuously deliver high-quality software releases more frequently merging proven methodologies with empowering tools and technology. Twitter: @DennisEhle

February 5, 2015
Evolution Not Revolution – DevOps in the Wild
Ever heard these questions:
- What is DevOps?
- How do I hire a DevOps Engineer?
- Should we have a Director of DevOps?
- Am I doing DevOps?
- Why is DevOps failing in my organization?
More and more frequently, I run into engineers, managers and business leaders describing failures in their organization due to “DevOps.”

Oddly, digging a little deeper, I find that most of these cases are simply failure by substitution. The Operations team was failing, so they rebranded it DevOps. Alas, buzzword compliance was unable to change the trajectory of the team and the buzzword was blamed.

DevOps today is–still–an ill-defined set of practices closely associated with automation and operations. If you accept this as fact, then DevOps can neither fail or succeed. If we want to measure the success of the DevOps movement in an organization, then we need a definition that leads to accountability.

A Definition for DevOps

The DevOps movement, unlike Agile, lacks a manifesto. To fill that gap, I assert that DevOps is a philosophy and that its base definition is:

DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business.

The business in question need not be a technology company such at Docker or IBM. Indeed, all companies today are technology companies whether they like it or not. (1)

In this and future articles I will write about philosophy, models and implementation. In short, a philosophy is what I reason from to create models of behavior. Models inform the actions taken in an implementation. Implementation without philosophy leads to failure. Jingoistic adherence to philosophy without accounting for new technology leads to failure. Balance must be acheived.

As an example of this sort of failure, take a lesson from project management. Many shops adopt Jira because it can do anything. Since the tool can do anything, that means their workflow is capable of anything! Problem solved, QED. We all know tools are not process. When we look at our own track records of instituting change. too often the change stemmed from tool choice, not critical thinking about what was to be accomplished.

Tools should reflect the philosophy they are implemented to support. Without that philosophy tools will fail. The more complex the problem or tool, the more spectacular the failure. In this example, Jira did not fail, leadership failed. Specifically the thought leaders entrusted with workflow management.

Are you Doing IT DevOps Style

DevOps is about evolution. If you are doing IT DevOps style, then you are asking questions like:
- What should we change next?
- How do we create a sandbox to measure the impact of the change?
- How do we create demand for the change once vetted?
- Who do we need to be patient zero?
- What cultural paradigms are we shifting and how do we ease this for teams?
While automate-all-the-things is often associated with the term DevOps, automation without cultural shift will result in broader failure, rather than success. Like any technology problem, at its core DevOps models account for their impact on people because businesses are composed primarily of people.

Future Articles

Over the course of the next few months. I will relate my experiences at several companies. Articles will describe:
- What we changed.
- Why we changed what we did.
- What cultural issues we faced.
- What technologies we used and why.
I hope you will join me to learn how the practice of DevOps as a philosophy can lead to millions of dollars on the bottom line, happy bosses, fluffy bunnies and rainbows. If your lucky, you might even become a unicorn, though my own horn has yet to sprout.

The first case study will describe a failing company with three months capital. An amazing team. Some amazing mistakes and, ultimately, some amazing success. Technology decisions driven by the needs of a sales team. Automation driven by a business model. A successful acquisition.

References

(1) The Phoenix Project – Gene Kim, Kevin Behr, George Spafford
(2) Good to Great – Jim Collins

About the Author/Boyd E Hemphill

Boyd E Hemphill is Director of Evangelism at StackEngine (link). , an enterprise grade, Docker operations management software company. DevOps raconteur and organizer of community events in Austin, find him on Twitter or LinkedIn.
February 4, 2015
DevOps: Where are we with that?

There’s a great scene in the Kids in the Hall movie Brain Candy, where the CEO of a major pharmaceutical company informs his executive team that the board has recommended a massive restructuring plan that will cut their research outlay by 60%. Only seconds after first mentioning this undertaking, the CEO turns to his yes man, Marv, and asks:

“Where are we with that?”

Marv responds: “The thing you just mentioned just now?”

And after getting an affirmative response, Marv foolishly, yet not surprisingly answers:

“Oh, we’re on top of that, Don.”

It’s not hard to imagine this same scene playing out in real life where the restructuring being recommended is that found in DevOps. Marv and his fellow executives have agreed to a restructuring plan that’s going to dramatically affect the flow of work and the lives of many people at their company, but without having any real idea of why this plan is needed, how much it will cost, or even where to begin. They’ve just been asked to reduce the cost of their product by 60%, and to begin, well, five seconds ago.

Sound familiar?

There was a time when DevOps was possibly an even bigger buzzword than it is today, but that doesn’t mean it has gone anywhere, or lost popularity. In fact, its adoption has only increased since the term was coined, and it can be found everywhere from the smallest shops—all the way to massive enterprises.

So why is it still so hard to successfully pitch DevOps to executive leaders?

We’ve written before that the ambiguity around DevOps’ definition is one of the primary causes of the confusion surrounding the term, and the debate between whether it’s a “mindset” “practice” or “cultural shift” will likely never die (just don’t let anyone tell you it’s a “tool”).

Rather than argue the buzzword’s definition, many at last year’s DevOps Enterprise Summit argued that the best way to promote a DevOps initiative at your organization, especially to your executive leaders, is to not call it DevOps at all.

Since I’ve always liked the term DevOps, it’s taken me a while to come around to this concept, but I think those esteemed speakers are on to something.

Vendors, analysts, the media, and other stakeholders have all had enough time to develop their own personal assumptions around DevOps. Maybe it was first proposed to them by an overblown consultant, or it was part of the failed vision of a former executive who is now on the outs. All it takes is one negative DevOps word association within your leadership and your noble pursuit of its buy-in could be quickly derailed.

So if you don’t call it DevOps, what do you call it? “Lean IT?” “Agile Innovation Processes?” “Continuous Flow Delivery?” “ Industrialized Software Development?”

It’ll take you more words, but I’d recommend using DevOps’ results, and not any perceived buzzworthiness of the term alone, as its primary selling point. While “more words” means “more time”—something your e-staff likely has very little of—you’ll both get and hold their attention much quicker than just saying “DevOps” and expecting the applause and champagne to start flowing.

Gene Kim’s definition of DevOps supports this concept, and in a recent interview he remarked:

I would define DevOps by the outcomes. In my mind, DevOps is those set of cultural norms and technology practices that enable the fast flow of planned work from, among others, development, through tests into operations while preserving world class reliability, operation and security. DevOps is not about what you do, but what your outcomes are.

For those who are somehow just learning about DevOps, or for those who have known of its existence for years now, but who have been hesitant to pitch it to their own leadership, the good news is that the outcomes of DevOps are being proven not just at big names like Disney, GE, and Netflix—they’re even being felt by those who’ve implemented it at non-Dev or Ops fields like Security, and even Design.

Your VPs of App Dev and QA will likely be impressed by your mention of accelerated feedback loops and reduced defects. Your IT Operations leadership, no matter what distrust they harbor for app dev, would have a hard time arguing against increased visibility into the delivery pipeline, and better quality when the software lands in production. Then when the CEO hears about faster service delivery and improved time to market, you may as well start sketching out your plans for that corner office.

(Editors note: Author’s promise of a corner office is his own, and is not guaranteed by Skytap)

About the Author/Noel Wurst

Noel Wurst, Managing Editor at Skytap, Noel is responsible for setting and executing Skytap’s content marketing strategy. That includes overseeing the creation and distribution of high-quality, targeted content, and managing the team and external resources associated with those efforts. By leveraging deep online marketing, project management, and editorial experience, Noel plays a crucial role in helping generate and nurture leads with compelling content. You can follow Noel and Skytap at @NoelWurst and LinkedIn and @Skytap, Skytap’s LinkedIn

January 30, 2015
The role of middleware in DevOps

Middleware has a very important and strategic role in DevOps, according to Ben Henshall, middleware business unit manager at Red Hat Australia and New Zealand. ‘Rapid everything’ is part of DevOps culture, so the bloated, proprietary middleware with roots in the late 1990s and early 2000s – the pre-cloud era – is a poor fit.

He pointed to an unnamed government organization that is limited to two major code releases per year, largely because of the complexity of its middleware and associated processes, which means systems have to be brought down completely for a weekend to deploy the updates.

This is in stark contrast with current thinking, and Henshall pointed to the frequently-cited 2009 presentation by John Allspaw and Paul Hammond http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr that described how Flickr managed 10 full deployments during a typical day.

Modern middleware is designed with cloud deployments in mind, including a multitenant architecture and a small resource footprint that minimizes infrastructure requirements. The best examples are based on Open Source projects with many contributors and extensive testing, and are developed and tested in the cloud, said Henshall, presumably with Red Hat JBoss Enterprise Middleware in mind.

It may be necessary to re-architect applications in order to decouple them from old-school middleware so a more modern replacement can be used, but technologies such as containerization can simplify the task, he suggested. While the right middleware and the adoption of Platform as a Service (PaaS) help – along with technologies such as OSGi (Open Service Gateway Initiative, which allows updated Java code to be deployed without stopping and restarting a system, among other things) as implemented in Apache Karaf (which is part of Red Hat JBoss) – but it is important to design applications so they can be maintained without outages, he said.

PaaS has seen a rapid rise in popularity as it makes it easier for developers to build and test new logic, and for operations staff to manage multiple systems without customization and with the benefits of automated releases. For example, Cisco has released more than 240 applications in around 12 months by using Red Hat OpenShift PaaS, said Henshall, noting that it can run on premises or be consumed as a service run by Red Hat.

The automation provided by PaaS fits in with the operations goal of running stable, reliable and secure systems, while reducing the need for “operations ninjas” who are potential single points of failure in the event of illness or resignation.

Henshall said OpenShift customers include a tertiary education institution that adopted OpenShift to speed the development and deployment of new applications and can now provision new systems within minutes, a software vendor that has reduced the time to market for new initiatives, and a major Commonwealth Government department that has been able to meet internal requirements concerning security, supportability and risk reduction since adopting the PaaS.

Being able to support innovation in the shortest time and with the least risk “is such a huge thing” for organizations, he said, especially when it can be combined with dramatic costs reductions and easier compliance.

Vendors such as Red Had provide products and services to support the DevOps culture, he said, noting “our software is developed by people facing these issues.”

About the Author/Stephen Withers

Stephen Withers is one of Australia¹s most experienced IT journalists, covering everything from gadgets to enterprise systems. In previous lives he has been an academic, a systems programmer, an IT support manager, and an online services manager. Stephen holds an honours degree in Management Sciences, a PhD in Industrial and Business Studies, and is a senior member of the Australian Computer Society. You can see his Linkedin profile here: http://au.linkedin.com/pub/stephen-withers/0/222/687/

January 23, 2015
Change Management & Continuous Delivery: Clash or Harmony?
Introducing changes into running software systems is risky. There’s always a chance of failure, and that can have a negative impact on your business. It’s important to mitigate that risk, but there’s more than one way to do so.

Change Management is all about ensuring that changes to existing systems and software in production are done in a controlled way. Change managers review the changes that are to be implemented, assess their impact, and perform due diligence, before giving or withholding their approval.

Continuous Delivery (CD) is about improving the process of delivering software and increasing efficiency. It aims to cut down the time between ideation and realization. It’s about reducing the scope for error and increasing the speed to market to generate a competitive advantage.

Is there an inevitable clash between CM and CD? Are they really compatible? At heart, the CD pipeline is intended to achieve the same thing as CM by enabling the smooth, error-free roll-out of changes, but today they are usually coming from very different angles.

If CM is to have a future in the CD revolution, then it needs to change. Before we discuss how that might be achieved, let’s take a look at the current state of play

Is Change Management delivering?

Ask yourself what impact change management is having on your organization. If change management was working perfectly then there wouldn’t be any production problems, and yet issues still crop up. There are a few things to consider:
1. How many entries in the CM checklist could be automated? Is the change manager doing anything that really requires human input? Fact collection and verification could be a part of your CD pipeline.
2. How much value do check boxes deliver in the first place? What is change management really bringing to the table?
3. Is the process fit for purpose? Does it actually prevent problems, or is the change manager really just a fall guy, there to sign off on the release and take the blame if something goes wrong?
Clashing with Continuous Delivery

CM is intimately linked with the idea and practice of big complicated transformations, but with CD we’re not talking about enormous, disruptive changes. CD is about changing small parts of a system and doing it often. The risk to reward ratio is different when you think and work iteratively. Smaller change sets are less likely to cause problems and easier to rollback when they do.

Does the process really require human input? Heavyweight validation will slow it down and speed is one of the key benefits Continuous Delivery offers. Streamlining and automating the checklist process makes perfect sense in many business contexts. The aims of CM and CD are in sync, but CD is better placed to deliver.

A possible advantage of traditional CM is the presence of a human eye, offering intuition that a machine or a piece of software may not be capable of. By moving beyond the checklists and reimagining the CM role we can retain this advantage and push CD forward.

From Gatekeepers to Masters of Continuous Delivery

If the bulk of CM is really an administrative exercise in checking boxes then it could be an unnecessary overhead, but there is a way to retain that human oversight for organizations wary of full automation. Why not ask the former gatekeeper to own the CD process? For a CD pipeline to be truly effective someone has to guide the process, engage with all the teams involved and understand exactly why and how the process generates business value.

A change manager, paired with a build or release engineer to handle the automation, could work to help organizations move towards CD. They are uniquely well placed to identify dependencies, map processes, and help construct the CD pipeline with minimal disruption. They can help sketch out what the pipeline should verify, what data and dependencies need to be taken into account, what automated tests are required, and how it fits with other components to automate that release procedure. They also have experience mediating between business and IT.

The traditional CM approach of committee meetings and spreadsheet checklists is dated, and for many organizations it no longer makes sense. You can find much greater business value in focusing on an efficient rollback procedure that kicks in immediately and automatically when changes fail.

Take a look at your CM and ask yourself – is it really adding anything to the process, or is it just another bottleneck? It may be time for a change.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
January 21, 2015
DevOps need to pay more attention to security

Ty Miller, founder of Threat Intelligence, suggests organisations adopting DevOps need to pay more attention to security.

Over the last five years or so, the prevailing approach to IT security has changed – largely triggered by the activities of hacktivist groups such as Anonymous and LulzSec – and it is becoming an increasingly integrated part of the application lifecycle, he said.

The less attention given to security in the earlier stages, the more likely it is that the application will have to be rearchitected and redeveloped before deployment, which can be 100 times more costly than building in security from the outset.

Since DevOps is often adopted to allow more frequent code releases, it is important that security is built into the process from start to finish, as there is no longer time for extensive security testing between completion and release.

Part of the problem is that developers tend not to be security experts, and because of their backgrounds security officers generally are not application security specialists. This can leave a security gap, he said.

Miller’s recommendations include:

Code libraries The reuse of known-good code instead of reinventing the wheel each time reduces the risk of building new vulnerabilities into software.

Platform security While IT teams would not dream of omitting intrusion prevention systems and other security components from an on-premises system, “these sorts of things are quite commonly being left out” from cloud deployments possibly because they are thought to be insufficiently scalable – a symptom of the tendency to sacrifice security for flexibility. And organisations sometimes assume that SaaS products are secure, even though they can “become large scale data breaches.” Recent research suggests around 90 percent of SaaS offerings are not sufficiently secure for enterprise use, he said.

Automated builds Automatically building systems based on pre-hardened templates helps reduce the risk of accidentally overlooking something and other human errors. If you can be confident that a new server will spin up in a secure configuration, development teams can concentrate on the code they are working on.

Automated security testing Frequent releases are at odds with traditional security testing, such as engaging specialist penetration testers before a major release. “That’s thrown a spanner into the works,” Miller said. When do you do the security testing? Can you pinpoint everything that has changed since the last test, and focus testing efforts accordingly? “Automated security testing needs to be built into the DevOps process,” he advised.

Fortify, AppScan and Burp Suite are among the more reputable tools, he said.

HP’s Fortify http://www8.hp.com/us/en/software-solutions/application-security/ , a code review tool, “is very expensive [at up to $120,000 per year] but also very good,” according to Miller, even though it generates false positives as other tools do.

IBM’s AppScan http://www-03.ibm.com/software/products/en/appscan/ helps with web and mobile app development, and costs around $30,000 per year, he said. While it has a good feature set and a convenient GUI, it does need a lot of setting up and maintenance when it comes to workflows.

PortSwigger’s Burp Suite http://portswigger.net/burp/ is a semi-automated scanner for web applications. At $300 per seat it is relatively cheap, and “virtually every penetration tester around the world uses it,” said Miller. It might not be as intelligent as the big-name commercial products, but “it is a very accurate scanner.”

But automated testing is not a panacea. Automated scans can find perhaps 20 percent of vulnerabilities, but that’s 20 percent less for humans to find. And the output of such tools tends to include many false positives, so some expertise is needed to interpret the results – an inexperienced user can waste a lot of time, Miller warned.

For example, access control flaws can be hard to detect automatically. Does changing a number in a URL produce a different page in the same report, or a report for a different customer? Is any particular report public or private? A penetration tester understands the application and can easily decide whether or not there is a security issue, but the best an automated tool can manage is to warn there may be a problem here.

An organisation that has a defined set of methodologies covering the entire process from development to production, implements security as a baseline feature in code and infrastructure builds, uses automated scanning and security testing, and integrates regular penetration tests and reviews “will be significantly ahead of other organisations,” Miller suggested.

About the Author:

Stephen Withers is one of Australia¹s most experienced IT journalists, covering everything from gadgets to enterprise systems. In previous lives he has been an academic, a systems programmer, an IT support manager, and an online services manager. Stephen holds an honours degree in Management Sciences, a PhD in Industrial and Business Studies, and is a senior member of the Australian Computer Society. You can see his Linkedin profile here: http://au.linkedin.com/pub/stephen-withers/0/222/687/

January 20, 2015
Ultimate Checklist for Successful App Deployment

Deploying software releases is a mixture of planning, testing, late hours, and celebratory beers. This checklist is intended to be a guide to help improve how your team achieves this complicated and sometimes hairy task.

Part 1: Planning and Deployment Preparation

Collaboration Between Development and Operations

It is important that operations be in the loop when it comes to planning any software deployment. They need to know when the project starts, not at the last minute about your new deployment. Ensure that operations is aware of any infrastructure changes needed well in advance. If the operations team is your front-line defense for monitoring and troubleshooting your app, help them help you by collaborating on how best to monitor the app and discuss early warning signs of trouble to be alerted to. If you’re thinking of setting up DevOp team you should probably read this.

Build & Release Automation

Removing the human factor is important to duplicating software deployment success. How you deploy the code doesn’t have to be state of the art. Utilize a basic build server like Maven, Bamboo, or TFS to automate the process or even a simple script that copies the files across the network can be sometimes sufficient. The key here is to make it easily repeatable by anyone on the team and eliminate the possibility of skipping any manual steps.

Minimize the Amount of Change

When things break in IT, typically it is because something changed. When you introduce small changes in each software update it is easier to roll back those changes, or know exactly where to look for the source of the problem. As each release takes a certain amount of work in planning and testing, the key is finding the right balance of frequency and size of your releases that is optimal for your team to handle.

Create and Test SQL Change Scripts

Prior to your app deployment make sure you use tools like Red Gate SQL Compareto know what SQL schema and data changes have to be moved to production. to ensure there are no environmental differences that will cause potential headaches, I recommend never to forget to take a backup of production and run the test scripts on the backup copy before the final release. Production data differences or different database software versions can commonly cause hiccups. Also, have a plan to roll back to a previous state if the release has issues. Code is easy to roll back, but SQL requires some planning.

Know your KPIs

Every app should have some form of key performance indicators or metrics that can be used to know if everything is running smoothly. These will vary wildly from one app to another. They can be as simple as server CPU, page load times, or database performance metrics. The best metrics are more custom though. Knowing how many orders are happening per minute, or how many messages are being processed off a queue and how long it takes, or how long a certain piece of code takes to execute is a critical part of your app. Make your KPIs the heartbeat of your application and monitor them at all times.

Setup Synthetic Transactions Tests

Create a few tests that test things like your login page, key pages within your apps, APIs, etc. This will allow you to feel very confident that all systems are operational after the app deployment. I highly recommend Selenium, the leading open source framework for web automation.

Notify Your Users and Colleagues
Don’t forget to notify your users and others within your company about the upcoming changes. Make sure you have time to update documentation, train internal staff about new features, and coordinate with other departments in your company.

Part 2: After the Deployment

Monitor Server Utilization

Make sure servers are all online, in the load balancer, and still receiving traffic. Monitor server and app CPU, memory, network, and disk usage. Noutilization can be as bad as sudden high utilization.

Monitor Exception Rate

After a new release your software is likely to throw some new exceptions you have never seen. Utilize an error tracking program so you can immediately identify new errors introduced in the release and to ensure previously resolved errors haven’t been reintroduced. A comparison of your overall error rates before and after the deployment will give you good indication of the release health.

Check your KPIs

Keep an eye on your performance indicators that let you know if your application is running properly.

Watch your logs

After an app deployment make sure logging is still working and the volume of logging hasn’t changed dramatically. No logging at all can be as bad as sudden high logging rates. A centralized log management tool makes this easy to monitor. You might also want to check the logs of new features that were released to ensure they are behaving properly.

Monitor Page Load Times and HTTP Error Rates

Watch the overall page load times of your application and keep a closer eye on 2-3 requests that are mission critical to your app. Also watch out for the rate of 400 and 500 level HTTP requests to ensure they haven’t dramatically changed.

Monitor Database Performance

After your deployment make sure database CPU, IO, and overall traffic look normal. Nothing is scarier than a sudden drop in database traffic after a release… unless it’s a sudden spike in database traffic.

Monitor Key Database Queries

Setup monitors in your monitoring system to test key queries to ensure they are loading quickly and returning proper results. For example, if some important background processes work based on SQL queries, run similar queries to make sure it isn’t missing work to be done and is working properly.

Monitor Application Queues

Monitor how many messages are in your application queues to ensure everything is flowing properly and not getting backed up.

Cache Server Changes

Don’t forget about caching. Depending on how you use caching, you may need to clear your caches or make other configuration changes. Data type changes can commonly cause some weird serialization type issues in cache.

Have a celebratory beer!

At Stackify we prefer a local joint with some good craft IPAs or the one brewed by one of our developers. Make sure your team has a good tradition for after releases.

Post Mortem Meeting

It can be very valuable to stop for a few minutes and discuss how the last release went and how things potentially can be improved. These kinds of conversations can help identify team weaknesses, allow people to vent, and take note of issues that can try to be prevented in future development cycles.

Summary

Deployment success over a long period of time depends on being able to automate and create processes that minimize issues for all stakeholders. Creating some automation or scripts around your build and release processes pay big dividends in the long run. Utilizing various monitoring tools or looking for new ones that can help monitor your server and application health from every angle. Tracking trends over time can help you show off the improvements you’ve made with each release in availability, performance, KPIs, and user experience.

About the Author

Matt Watson founded Stackify in 2012 and as CEO provides the vision and leadership for the direction of the company. Prior to founding Stackify, he was the founder and CTO of VinSolutions. VinSolutions’ innovative product offering led to rapid company growth and was even featured on the Inc. 500 (#447) in 2010. Matt is an entrepreneur at heart and excels at product and software development.

Matt studied Computer Information Systems at DeVry University. He started programming when he was in middle school to automate playing video games while he was at school. He hasn’t stopped programming since and loves anything technology related. Other hobbies include working too much, travel, movies, golf, cars, and of course his family. You can reach Matt at @Stackify, @mattwatson81, www.stackify.com, https://www.facebook.com/Stackify

January 16, 2015
Continuous Delivery snapshot: a universe of fragmented tools
Technology has ushered in an era of real-time information, immediate feedback and instantaneous response. As a result, there is tremendous pressure on development teams to deliver a continuous flow of updates and fixes to software applications. In support of this objective, development philosophies and methodologies abound: agile, lean, scrum, kanban and extreme programming, just to name a few. Development teams are adopting and implementing these new methodologies at a breakneck pace in an attempt to bring efficiency and speed to the development process.

This increased velocity has placed added pressure on software delivery teams to build, validate and deploy newly developed code faster and more reliably. Over time, the DevOps movement has emerged, injecting robust automation into the delivery process while spawning a diverse collection of hundreds of single-purpose delivery automation tools. As a result, DevOps has improved delivery speed and quality. Even still, continuous development comes with its challenges. Namely, the lack of a real-time “big picture” view, of the end-to-end software delivery process.

New Challenges for CD Teams: The Unified Pipeline

While highly diversified delivery toolchains can be effective and have increased speed and agility, significant challenges continue to plague the software delivery process as a whole.

Real-time visibility and visualization – Typically, the delivery process begins as soon as newly developed code is considered “dev-complete.” Often, the sequence of steps and stages new code progresses through on the way to potential release is referred to as the delivery pipeline. In a moderately sized organization, delivery pipelines are comprised of dozens of people performing hundreds of tasks. Without a centralized “delivery portal,” team members have little or no insight into the overall upstream or downstream process flow. While delivery pipelines benefit from diverse automation, they are still a “black box,” largely absent of communication, collaboration and audit/tracking capabilities.

Data fragmentation – Important delivery data is generated at virtually every step along the delivery pipeline. Disconnected toolchains generate siloed, disaggregated and uncorrelated data that is trapped within a dozen or more automation tools. As a result, team members often lack access to a single comprehensive audit trail document that fully describes the functional and technical contents of a specific release along with any relevant data generated during each step of the delivery pipeline. At a macro level, self-organizing delivery teams do not have the real- time and historical analytic data required to quickly identify and react to exceptions or make performance optimizations across pipelines and/or teams.

Is dynamic pipeline orchestration the answer?

While the build and integration stages of a delivery pipeline are largely static and predictable, pipeline flow can become much more dynamic once manual testing begins. Defects can be discovered at any point along the delivery pipeline and judgment calls often determine a dynamic process flow. Often tools designed to automate the build/integrate phase of the pipeline are stretched beyond their original intent when it comes to pipeline orchestration. While these tools automate a critical portion of the delivery pipeline, they are far too static to support a real-world pipeline flow through the manual steps that occur in the later stages of a delivery pipeline. As a result, pipeline orchestration is largely a manual process beyond integration and into the quality assurance phases of delivery.

What’s Missing?

When it comes to alleviating the challenges associated with delivery pipelines, what will it take to ease the burden on continuous development teams?
- Detailed Inventory of Each Bundle-of-Change – As multiple revisions get merged together in downstream pipelines to form a release candidate, a clear understanding of exactly what “changes” have been incorporated within any particular change bundle (release candidate) can become crucial.
- Current Deployed Environment – At any given time, any revision or release candidate can be deployed to one or more environments or locations. The ability to quickly determine which change bundles have been deployed to which environment(s) is a fundamental need of virtually every stakeholder along the delivery process.
- Broken code lines – Release candidates that contain known material defects are usually considered terminal and/or not deployable. Identifying a release candidate as broken can dramatically reduce the amount of time invested into a terminal release candidate. Subsequent testing within environments that contain a broken release often leads to wasted effort.
- Detailed release documentation that fully describes the functional and technical contents of a specific release-in-transit along with all relevant data generated during each stage of delivery.
- Reporting that describes stalled releases awaiting manual intervention with corresponding wait times.
- Work queues that outline manual tasks or pending decisions/approvals assigned to a specific individual or group.
- Travel time details on the average time it takes a release to traverse a pipeline from start to finish.
- Reporting that describes the “base” release (starting version) and all subsequent change that has flowed thru any given development or test environment.
What is the Answer?

Without a technology that can oversee the entire development process and fill these gaps, teams often try to leverage Continuous Integration (CI) tools. While these tools greatly reduce the effort required to automate the build/integrate phase of the continuous delivery pipeline, this stretches them beyond their practical limits.

However, a new type of pipeline orchestration technology that further streamlines and advances the effectiveness of the CD process is emerging and can effectively alleviate the challenges associated with fragmented and siloed delivery pipelines. This technology, often called a “pipeline orchestration framework,” expedites movement and raises awareness of a software “release candidate” as it travels through one or more delivery pipelines.

So how do they work? Pipeline Orchestration Frameworks:
- Leverage existing tools and processes – Because Pipeline Orchestration Frameworks sit on top of existing delivery automation tools, there is no need to reconfigure current process flow or abandon existing tooling. Pipeline Optimization Frameworks simply enhance and automate what is already in place.
- Provide audit trail – Comprehensive audit trail documentation of each part of the process delivers highly detailed and valuable information regarding each new potential release. Having this audit trail ensures higher production quality results, increases cross-team awareness and reduces the time and effort required to perform root-cause analysis when defects are found (pre/post-production).
- Manage deployments – Efficiently managing the cost and friction associated with dynamic development and testing environments over time can dramatically reduce the cost of delivery. Pipeline Orchestration Frameworks are capable of automatically provisioning and/or refreshing fully formed application environments as often as needed while keeping track of environmental drift or change over time.
- Workflow automation – One of the goals of continuous delivery is to automate as much as possible along the delivery pipeline. However there are often steps that can’t (or shouldn’t) be automated, such as code reviews, exploratory testing, and manual decisions to automatically move a release into late stage test environments or even production. Pipeline Orchestration Frameworks manage and orchestrate these manual workflows as efficiently as possible, ensuring all delivery team members are aware of items in their “work queue,” and identifying stalled releases immediately.
What will 2015 Hold?

The purpose of continuous delivery is to support development teams’ need to release reliable code faster. Therefore, the delivery team must keep up with an ever-increasing development velocity. Managing the delivery process not only requires managing each of the different phases of delivery, but also orchestrating the delivery process as a whole. That, in turn, requires visibility into every part of the end-to-end process, and a 10,000-foot view of the delivery pipeline. Pipeline Orchestration Frameworks offer CD teams a powerful new tool to unlock the full potential for successful continuous development.

About the Author

Dennis Ehle is the CEO and founder of cloud sidekick, which provides an automated, self-service integration and deployment framework for enterprises seeking continuous delivery. The cloud sidekick Velocity Pipeline Optimization Platform enables agile development teams to push code through multi-stage pipelines, automatically testing in virtual environments and ultimately publishing reliable code faster. For more information, visit www.cloudsidekick.com.
January 14, 2015
Five essential Ops strategies that help make DevOps the real deal
With its roots in Agile and Lean thinking it’s not surprising that DevOps is often looked at through a development lens. After all, it’s proving absolutely essential to support accelerated delivery times; providing a much needed jolt of velocity to protracted development cycles. But of course with the increased need for speed there’s always the danger that quality can be compromised – so much so that faster time-to-value can very quickly become faster-time-to-failure.

In what seems like a byzantine age of enterprise computing, IT operations tended to call the shots when it came to delivering quality. Sure, a transactional system might suffer a glitch or two, but since constituents were largely internal and customer’s generally one step removed from the application and business process, the impact could often be controlled. Ah, so much for the good old days.

But as we know this model hasn’t just changed, it’s been completely disrupted.

Quality is now a different headed beast that bites hard in many places. It bites when customers engage directly with your brand during discrete mobile moments – like when they compare your goods in the shopping aisle with a mobile app, or assess car insurance based on your ability to provide usage-based policies from their in-vehicle software system. Yes, speed is important in the application economy, but it’s also a high quality customer experience that shapes the winners from losers.

The fabulous thing about DevOps is the fundamental premise that teams can collaborate in real-time to establish, maintain and continuously improve the quality of software systems and applications. For operations this doesn’t mean waiting for applications to reach production stage before enacting performance management processes. Rather, it involves instantiating best practices throughout the development cycle – that way teams resolve quality issues much earlier in development (where they are cheaper to fix), while also preventing a cumulative buildup of crippling technical debt.

So with application quality and customer experience being essential for DevOps success, what are some of the ops related strategies we can employ to support better business outcomes?

I see five essential, ‘rubber meets the road’ strategies ops can use to help DevOps become the real deal:
1. Focus collectively on the user experience – too often organizations become detached from business reality by looking inside-out from a silo’d technology and diagnostic perspective. DevOps practitioners put themselves in the shoes of the customers by constructing, visualizing and managing end-to-end user-centric service views that traverse functional boundaries. Additionally, user-experience management will involve gaining deeper insights into real-time performance of externally facing applications and mobile services, bringing to the forefront analytics to guide decision making and drive continuous improvement.
2. Give Developers a clear view of how code works in the real world – operational intelligence shouldn’t wait until applications are moved into production. Rather, it should be instantiated across every stage of the lifecycle – whether that’s during regression and unit testing, or pre-production. Many teams recognize this by for example activating monitoring during release automation; giving developers critical insights into both functional and non-functional code behaviors. Others go further, taking user-experience to another level by exposing key behavioral and usage-based intelligence – especially for newer mobile apps, APIs and digital services.
3. Create a single source of the truth using a common framework – to be fully effective, developers and operations should reference a single version of the truth when it comes to problem triage and resolution. Traditionally this has proved difficult with teams having discrete tools providing diagnostic insights into one aspect of technology. This however falls short due to the complex interdependencies across technologies and results in finger pointing and blames games when problems can’t easily be pigeon-holed. To address this, operations teams use a common unified framework that quickly discovers and exposes infrastructure and application configurations, and importantly the relationships and interdependencies that exist between them. Now, root-cause can be quickly established from either an application or infrastructure perspective, but always prioritized according to business and customer impact.
4. Define common goals across teams and use shared metrics – traditionally development and operations teams have been incentivized according to separate goals that are often diametrically opposed (e.g. speed of producing code –v- resilience and availability). By using shared customer and business-centric metrics, developers and operations build a ‘common language’ for evaluating progress and understanding the impact of change and continuous delivery. My advice here is quality over quantity; building out metrics that are actionable by all stakeholders. For example, this could include lead and cycle times to pinpoint operational bottlenecks and waste during deployments, or transaction response monitoring in pre-production to pinpoint code deficiencies and avoid technical debt associated with provisioning unnecessary capacity.
5. Converge around a common understanding to transcend organizational boundaries – a collection of diagnostic based monitoring and alerting provides little more than vanity metrics and feel-good green lights. Sure, they can alert operations to when something is broken, but they do little to drive the more essential element of DevOps success – feedback. Therefore common understanding involves operations automatically correlating performance information end-to-end so as to expose far more beneficial insights that developers can use to improve the code base.
At the recent Gartner Data Center, Infrastructure and Operations Management Conference in Las Vegas in December, I presented a session – Unified Infrastructure Management is Critical to Enhancing the Ops in DevOps – that further described these strategies together with supporting use-cases. It was certainly a dynamic and lively session, highlighting the viability of DevOps. As organizations look to increase speed to market of new applications while both lowering cost and increasing quality, DevOps will continue to gain traction, and I can’t wait till everyone truly understands its transformative power.

About the Author

Aruna Ravichandran is vice president of DevOps Solution Marketing and Management at CA Technologies, responsible for cross-portfolio solution marketing for DevOps across all product lines. Before joining CA Technologies, Aruna held several management positions in product marketing and engineering at Juniper Networks and HP. Aruna earned an MBA and master’s degree in Computer Engineering from Santa Clara University and holds a bachelor of science degree in computer science from Bangalore Institute of Technology. Ravichandran is on twitter @aruna13 and on linked in
January 9, 2015
7 Tips for testing the usability of mobile apps
Of the one million mobile apps, only half are used more than 5 times

If you study any random cross-section of successful mobile apps and games you’ll find that they have one common factor. All of them are accessible. They have user interfaces that people can navigate intuitively without instruction. This is partly about building on expectations and established norms with Android, iOS, or whatever platform you are developing on. But there’s no substitute for thorough usability testing.

According to Localytics, 20% of apps are used once and never opened again. Only 50% of apps are used more than five times. The window for app developers to impress and hook the user slams shut quickly and it stays shut. First impressions are vital if you want to boost retention rates and stand out in a crowd of over a million apps.

It can be difficult and expensive to extract actionable data from usability tests. Here are a few practical tips for successful usability testing:
1. Create a mock up – Starting with user-centric design is a smart move that will pay dividends down the line. You can save yourself a lot of time and money by discovering weaknesses and potential problems before the development really begins. There’s no reason you can’t mock up your user interface with simple screenshots and then frame it in an actual size smartphone surround. You can even do this with card and paper if need be. That way you can find out how potential users expect to interact with your app and identify the tasks that are most important to them. It’s a lot easier and cheaper to change your design on paper than it is to rewrite code.
1. Examine how people interact – This applies to every usability test you conduct, whether it’s a paper mockup, a prototype, or the functional app on a smartphone. If you’re doing it in person then provide minimal guidance, ideally none at all, so you can uncover genuine failure points. Make sure to take plenty of notes. It’s not enough to just record what’s happening onscreen, you need to observe what people are doing with their hands, and see their facial expressions and body language. Are they trying the wrong gestures? Are they getting frustrated? Is it working the way they expect? Close observation will deliver the best insights.
1. Test in the real world – If you can conduct tests in real world conditions in public then you might uncover some pertinent details about the usability of your app that you would miss in a controlled environment. This means conducting tests in real environments, whether it’s a busy shopping mall or a person’s home. It can be a good idea to avoid giving any instruction, that way you can see just how intuitive the design really is. There are services that offer video footage and audio narration of testers from around the world in different demographics using your app in real world conditions. Without the official framing of a test situation you have relaxed users and this can deliver different insights. Sometimes it might be more indicative of real world conditions to have someone test in their own home.
1. Keep it short and simple – The longer a test is, or the more complicated it gets, the less likely it is your users will stay focused and interested. If they have to think too much about the parameters of the test and what you’re asking them to do then they’re less representative of an end user. This is particularly important when you’re asking members of the public to test in return for a small reward or for free.
1. Vary platforms and devices – Try to let people use their own devices wherever possible. That way the device won’t be a distraction and you’ll get data about different devices. The more variations you can cover, the more useful data you can gather. If you’re releasing on more than one platform then make sure you cover each one. You’ll need to cater for differences in expectations about standard platform navigation.
1. Ask the right questions – Observation should be your primary source of data during usability tests, but you may want to ask some questions after each session. Be careful about what you ask and focus on getting actionable data. Asking if the app made them happy on a scale of one to ten isn’t going to tell you anything useful in a practical sense. You should also ask for elaboration, and where possible, have people show you what they mean using the app or prototype, rather than tell you, to avoid misconstruing any answers.
1. Assess and act – You need to aggregate the data you collect and create a plan to assess it. Make sure you draw meaningful conclusions that are backed up by a weight of data. Don’t let personal bias cloud this process. You should be acting on the things that will resonate with the most end users. Just because one vociferous tester shouts louder than the others doesn’t make their feedback more important. Take your time, get further feedback on specific areas if it’s needed, and create a plan of action that you’re confident will result in concrete improvements, then test them again to confirm.
There’s a lot of work involved in usability testing, but there’s no better way to improve the accessibility of your app and create a favorable first impression with the end user.

About the Author

Mush Honda is QA Director at KMS Technology, a provider of IT services across the software development lifecycle with offices in Atlanta and Vietnam. He was previously a tester at Ernst & Young, Nexidia, Colibrium Partners and Connecture. KMS services include application management, testing, support, professional services and staff augmentation.
January 8, 2015