devopsy.kinsta.cloud

Tag: continuous delivery

The Continuous Delivery Pipeline — What it is and Why it’s so Important in Developing Software
Continuous Delivery (CD) is a software strategy that enables organizations to deliver new features to users as fast and efficiently as possible. The core idea of CD is to create a repeatable, reliable and incrementally improving process for taking software from concept to customer. The goal of Continuous Delivery is to enable a constant flow of changes into production via an automated software production line. The Continuous Delivery pipeline is what makes it all happen.

The pipeline breaks down the software delivery process into stages. Each stage is aimed at verifying the quality of new features from a different angle to validate the new functionality and prevent errors from affecting your users. The pipeline should provide feedback to the team and visibility into the flow of changes to everyone involved in delivering the new feature/s.

There is no such thing as The Standard Pipeline, but a typical CD pipeline will include the following stages: build automation and continuous integration; test automation; and deployment automation.
- Build automation and Continuous Integration
The pipeline starts by building the binaries to create the deliverables that will be passed to the subsequent stages. New features implemented by the developers are integrated into the central code base on a continuous basis, built and unit tested. This is the most direct feedback cycle that informs the development team about the health of their application code.
- Test Automation
Throughout this stage, the new version of an application is rigorously tested to ensure that it meets all desired system qualities. It is important that all relevant aspects — whether functionality, security, performance or compliance — are verified by the pipeline. The stage may involve different types of automated or (initially, at least) manual activities.
- Deployment Automation
A deployment is required every time the application is installed in an environment for testing, but the most critical moment for deployment automation is rollout time. Since the preceding stages have verified the overall quality of the system, this is a low-risk step. The deployment can be staged, with the new version being initially released to a subset of the production environment and monitored before being completely rolled out. The deployment is automated, allowing for the reliable delivery of new functionality to users within minutes, if needed.

Your Pipeline Needs Platform Provisioning and Configuration Management

The deployment pipeline is supported by platform provisioning and system configuration management, which allow teams to create, maintain and tear down complete environments automatically or at the push of a button.

Automated platform provisioning ensures that your candidate applications are deployed to, and tests carried out against, correctly configured and reproducible environments. It also facilitates horizontal scalability and allows the business to try out new products in a sandbox environment at any time.

Orchestrating it all: Release and Pipeline Orchestration

The multiple stages in a deployment pipeline involve different groups of people collaborating and supervising the release of the new version of your application. Release and pipeline orchestration provides a top-level view of the entire pipeline, allowing you to define and control the stages and gain insight into the overall software delivery process.

By carrying out value stream mappings on your releases, you can highlight any remaining inefficiencies and hot spots, and pinpoint opportunities to improve your pipeline.

Don’t Add New Functionality Until You Get The Quality Right!

Continuous Delivery is about enabling your organization to bring new features to production, one by one, quickly and reliably. That means that every individual feature needs to be tested prior to rollout, ensuring the feature meets the quality requirements of the overall system.

In a traditional environment, development teams typically try to implement an entire new version in one go, addressing software quality properties (such as robustness, extensibility, maintainability) only when the project is close to completion. However, as deadlines loom and budget pressures grow, quality is often the first thing that is compromised.

Poor system quality, low-user satisfaction and endless “quality band-aids” can be avoided by adopting the principle of not adding new functionality before getting the quality right. You should always first meet and maintain your quality levels and only then consider gradually adding functionality to the system.

With CD, each new feature is required to meet the level of quality expected for the system as a whole, right from the start. Only once this quality level has been reached can the feature be moved to production.

Getting Started With Continuous Delivery

Obviously, organizations cannot and should not rush into adopting Continuous Delivery all at once throughout all their business units. The best approach is to focus on improving your biggest delivery bottleneck. CD will automatically show you what the next bottleneck is. This puts you on a measurable improvement path.

The main goal of using Continuous Delivery is to roll out new features and functionalities that are better than previous iterations — gradually incorporating and refining the CD principle throughout the organization. Go slowly, go smoothly — and watch the improvements!

Much of the material here is covered more extensively in the book, “The IT Manager’s Guide to Continuous Delivery” which is available at go.xebialabs.com/IT-Managers-Guide-to-CD.html

About the Author

Andrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
July 29, 2014
How to Find Your Continuous Delivery Rhythm
This article is a part of a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef community. In this article we present an interview with Jez Humble, Principal Consultant, ThoughtWorks, on digital innovation.

Focusing on speed and scale allows you to respond more rapidly to weak signals in the market, and to arbitrage those opportunities quickly. This is different from the way it used to be. In the past, we’d only respond to the strong signals in the marketplace. But now, with the lower cost of investment in product development, and the resulting optionality in that process, we can service subtle market signals effectively.

Here’s an example of this. A company noticed that when people searched for bird-cages online, very few entries came up. So, to fill the void, the company began making bird-cages. And now this business has created a new revenue stream selling bird-cages. We’re also learning that the way we service customer demand – in all its varieties – is part of the product development process. This comes with the emergence of user-experience design. And it shows that marketing, as well as technology, is an essential part of product development today. Zappos is a great case study that illustrates my point. Many companies try to minimize the time spent on customer support calls. Zappos spent a maximum amount of time on customer calls every day. And, by deeply engaging with the customer, they learn about the customer.

Code is really the key to exploring and exploiting new opportunities in an inexpensive and scalable way. And here, we have to give credit to Toyota, which has been using Lean to show us how to re-tool on the fly in response to customer demand. As a result, software is now following automobile manufacturing.

A major part of all this is continuous delivery of code, which, from my perspective, is fundamentally about reducing the transaction cost of putting out incremental changes. And I want to emphasize the importance of incremental changes and small batches; continuous delivery of code really can’t be effective if it’s one-piece flow.

Think about it this way. You don’t buy eggs at the supermarket once a year. And you don’t buy an egg as a replacement every single time you use one. No, you buy eggs on an ongoing basis in small batches. This is the kind of rhythm that powers our ability to move rapidly when there’s a weak signal in the marketplace. It’s the kind of rhythm that powers our ability to change code based on customer demand. And it’s the kind of rhythm that powers our ability to compete effectively – and with a decided advantage.

Netflix, Etsy, Amazon, Google and Facebook are five of the best at this. And I also like the way the British government’s digital service has been re-tooled and re-built with Lean, customer experience, and continuous delivery principles in mind. The government site is, without question, all about customer demand and customer service, and it has saved citizens a lot of money as well. The take-away here is that software companies shouldn’t be thinking narrowly about software; instead, they should be thinking about how their software can better serve customers. To do that, the entire business has to be run differently and more effectively. And, again, technology people – as well as product marketing people, financial people and governance people – all have to join in and help make it happen.

There are three practices of continuous delivery that can help you create the most efficient path from hypothesis to continuous feedback:
- Comprehensive configuration management
- Continuous integration
- Comprehensive automation of build, test, deploy, and infrastructure management
A new kind of “MVP”

One of the key ideas that the Lean startup movement has popularized is the minimum viable product (MVP), which is defined as the least possible amount of work you can do to test your business hypothesis.

Of course, people in manufacturing have been producing minimum viable products for decades – they’re called prototypes. I like the MVP because you’re not betting the company, or spending big wads of cash.

Another key Lean concept is validated learning; here, you gather actionable metrics based on real usage of your product, without asking people. To do this, however, you’ll need to be able to answer questions like:
- Did the changes we made to the product improve signup, retention, revenue? Or is it time to pivot?
- Which version of our new feature came out better when we did A/B testing?
- All our system metrics look okay, but a user reports that our site isn’t working. Are we down?
- Which features of our product are generating the most revenue?
These questions should be answerable by looking at a dashboard, and the information should be completely auditable. But this kind of learning is only possible if metrics are built into the “hypotheses” that are being tested. Once you have an idea for a minimum viable product, you need to start delivering software. The first step is analysis. My view is that you need to do just enough analysis up-front, and this approach needs to continue throughout the lifecycle of the project, with the emphasis on creating very small, incremental stories.

Doing less and learning more

Let’s say you’re working on a hotel booking site, and you want to add a feature to allow people to choose whether they want breakfast. Don’t create this feature for all hotels, or for all partner sites. Instead, start with a “hypothesis” that allows you to add that feature for a single hotel, with no configuration options, and get feedback for that possibility before you proceed further.

Do less, so that you can focus your efforts on learning from your users.

If you want to increase your release frequency, it’s essential to decouple your deployments from the act of taking a feature live. Feature toggles (also known as bits or switches) help you control who can see which features. This setup also allows you to release new versions of revised software that include partially complete features, containing perhaps a story or two of work, but nothing you’d want the general public using.

In addition to enabling incremental delivery of functionality, feature toggles let you turn off a problematic feature when a release goes wrong, as an alternative to rolling back your deployment.

The bottom line is that continuous delivery like this liberates us by reducing the cost and risk of releasing software, and by powering much-needed innovation at the same time.

Jez Humble is a principal consultant with ThoughtWorks and co-author of “Continuous Delivery” and “Lean Enterprise.” He began his IT career in 2000. Over the past 14 years, he has been a developer, system administrator, trainer, consultant, manager and speaker. He has worked with a variety of platforms and technologies, consulting for non-profits, telecoms, financial services and on-line retail companies. Since 2004, he has worked for ThoughtWorks and ThoughtWorks Studios in Beijing, Bangalore, London and San Francisco. He holds a BA in Physics and Philosophy from Oxford University and an MMus in Ethnomusicology from the School of Oriental and African Studies, University of London.
July 21, 2014
Revolving door of IT employment threatens continuous delivery

As the IT job market heats up following a long lull, enterprises will increasingly need to cope with the difficulty of maintaining operational continuity in the face of personnel churn. Whether the revolving door agitates the employee pool for developers or operations staff, the length of time necessary to off-board departing IT staffers and on-board new ones can particularly endanger the fast-paced routine of continuous delivery and DevOps practices.

Fortunately, the documentation and automation principles inherent to DevOps could actually also prove to be the very answer to a problem that threatens DevOps sustainability as job churn increases.

“DevOps is not just a tools play, it’s a human play–how people eventually work has a very big impact on how well DevOps is put into place,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “We have an immense shortage of talent in the market, and as engineers come and go it is almost unrealistic to expect that everyone who is there today is going to be there through the end of a project. You have to build business continuity around that and DevOps can play a key role in that.”

Suleman says he’s recently seen customers struggle with this problem first hand. In one particularly dramatic example, a smaller firm his company works with has been seeing its 10 person developer team ravaged with a churn rate of one employee turning over per month.

Recent industry numbers show that this kind of turnover may not be the result of organizational problems, but due to broader trends. Earlier this month, the Department of Labor reported that May capped off four months of dramatic job growth in the IT sector. Whereas in 2013, the DOL reported an average of 10,700 IT jobs gained per month, in May that number hit above 15,000.

“We’re now getting back on that track after some horrible months for tech professionals,” says David Foote, chief analyst at Foote Partners which has been tracking and reporting on IT labor trends since 1997.

Meanwhile, Robert Half International reports that IT salaries will rise by 5.6 percent this year. And a survey out this month from Computer Economics shows that 52 percent of hiring managers plan to increase the number of full-time IT staffers they hire this year, the highest this figure has been in three year. These statistics show how industry trends will spur on more movement between firms as skilled staffers look for greener pastures. This is particularly true for jobs seeking staffers experienced in DevOps, particularly as those DevOps vets exhibit a cross-pollenization of ops, dev and people management skill sets.

“There still remains a lot of volatility even in the IT employment marketplace which for some time been expanding mostly in the IT services industries according to these federal jobs reports. What they’re not reporting is all the new IT-business hybrid jobs that have been created and for which employers are feverishly mounting recruiting efforts,” Foote says. “They cannot find and hire them fast enough. There are probably ten or fifteen open positions for every one suitable candidate and that person is almost always already employed. They cannot find and hire them fast enough.”

So how can DevOps principles help to solve the continuity issue as employees play musical chairs? According to Suleman, most important is documenting and automating as much of the deprovisioning and provisioning process as possible. In particular, organizations should be seeking to iterate in order to speed up their on-boarding process in much the same fashion as they have for release schedules. In his example, the 10-person shop he pointed to as an example was getting killed by a two-day on-boarding process that didn’t scale well with the amount of turnover it was experiencing. One of the big reasons is that the company had no documentation standardizing the process.

“If you broke it down, a big chunk of that time was spent in just familiarizing the employee in what technologies the company used, with a lot of that shared through human interaction,” Suleman says. “That was often not a good use of time because they still needed to be relearned over the next few weeks as the engineer really deep dove into the code to figure things out on their own.”

Additionally, though, there was the issue of having no automatic way to share all of software, libraries and so on necessary to set up the new employee’s system so he or she could get started learning the environment.

“There are a lot of external dependencies as a part of that process. You may have an instruction saying go ahead and install the latest database version, but the version I’ll download today will be different from one you downloaded last month and something that will work on your computer won’t on mine,” Suleman says. “What ends up happening is those two days are spent figuring out all those discrepancies.”

In this case, a document may be helpful but even more helpful would be to find a way to automate the creation of up-to-date images of different roles’ environments so that all it takes is one simple install to get a new employee running from day one. This is exactly what Suleman says his company helped the customer do, creating a single, pre-packaged USB stick to go along with newly created documentation so that a two-day process was slimmed down to 30 minutes.

July 8, 2014
DevOps: Getting Past Audit

New business processes (and technologies) too often throw auditors into a tailspin. It’s no one’s fault, really: the decision making on what IT and process controls that need to be in place moves slow – and it is the tail to the rapidly moving technology dog. In fact, while there is no way a standard or regulatory body can predict technological change from year to year, there are ways to help ease the relationship between auditors and organizations embracing DevOps and continuous deployment practices.

“DevOps audit is an interesting problem,” says David Mortman, distinguished engineer at Dell (formerly Entratius). “Audit is something you just can’t avoid, especially in any industry that’s regulatory controlled. And auditors are a lot like security people, in my experience, which is to say that they have a lot of responsibility and they are justifiably concerned about things they don’t understand,” Mortman says.

And concern their certainly is when it comes to any organization that must deal with acronyms like HIPAA, PCI DSS, SSAE 16, SOX, FedRAMP, among others means few things will raise auditors’ eyebrows more than hearing that developers are performing deploys on their own code.

That’s why it’s important to know how to manage these audits, and continuously improve the audit process.

Rapid technological change, glacial control placement
One of the first things to note is that these standards and management programs move slow. “If you are familiar with ISO 27001, which is a security management program, they just released a new version at the end of September [2013]. From 2005 to now, the standard hadn’t changed,” said James J. DeLuccia, a senior manager in the advisory services practice of Ernst & Young LLP in his Flowcon presentation, Successfully Establishing and Representing DevOps in an Audit. But technology certainly has changed in that time: increased use of virtualization, the rise of cloud computing, ubiquitous mobile, as well as changes in development and operational processes such as continuous deployment, continuous integration, and DevOps.

One of the biggest issues that come up with disciplines such as DevOps and continuous deployment is separation, or segregation, of duties – that development, testing and operations should be separated duties. Not only do many government and industry mandates require it, so does many enterprise security policies. That’s why DevOps, when not explained or approached properly, can create bug-eyed expressions in unfamiliar auditors.

The goal of separation of duties comes down, essentially, to proper change management. If the developer who wrote the code can deploy the code into production, and they are given too much access (such as root) or shared access, there’s a lost control check, and potentially no log of the event. But it’s a concern that is overstated, experts say.

“In a proper DevOps environment, the person deploying the code doesn’t need to have root access. There’s no reason why, when properly setup, developers can’t logon, push the deploy button, and create an auditable log that says at 2:32 on Friday the 18th, David deployed a version of the code. And you have it down to the hundredth of second who did what,” says Mortman.

As DeLuccia explained in his talk, when auditors are asking to see change control logs they are really trying to see that these controls are in place. “They’re really asking for help to understand this linkage,” he said. Depending on the policy and requirements, the auditor needs to see reports that demonstrate, in a statistically relevant way, that everything in IT is working the way it needs to and is said to be working. “Because remember, when we’re putting together these reports, we’re trying to say, everything is working the way you say it’s working,” he said.

Auditing DevOps: An education process
When done right, Mortman says DevOps and continuous deployments create more accurate logging and change management. “You have a log of every change that’s ever happened and it’s all generated automatically. You don’t have to worry about someone changing something. When setup properly, they don’t have the access to change anything other than the very limited roles of what that application they can deploy to,” Mortman says. “So in a lot of ways, in fact you’re actually maintaining better control over your environment.

The challenge there, as DeLuccia explained, is that audit programs are built a year in advance. “When they come to see you, they’re already have it planned out based off last year’s risk. That’s already too late [to keep up with rapidly changing technology and processes]. You really want to plan this a year in advance,” he said. The goal, he explained is to become part of the process and show how the procedures they are asking the IT team to execute against are in line with reality of the processes and the technology on the ground.

“Much like any other industry, you have an education problem,” says Mortman. “Security people run into this all the time. The business will say, “Hey, we want to do this new thing, and the security people reply, “Whoa, hold on a second. You can’t do that. That’s not secure. Or, we can’t audit that.” So a lot of the challenge with DevOps is an educational challenge. And you need to educate auditors that the control processes they are looking for are in place and that there are adequate logs that can be audited and evaluated,” says Mortman.

May 14, 2014
Continuous integration for better security

One of the big advantages to smaller deploys and continuous integration is that it can make it easier to provide more proactive security. In short, continuous integration (and the associated automated testing it enables) makes it easier to focus your security team on analyzing areas where the security risk is higher.

If the notion of rapid development, rapid deployment, and continuous integration make your security team nervous you may be able to get them engaged by sharing proven ideas from companies who’ve made this approach work well in the real world. For example, I was recently flipping through a presentation from Etsy on SlideShare, and discovered a clear explanation of how continuous integration enables better security.

At right, you see slide 34 from this (very long but clear) deck. this is the first in a series of slides that talks about how you can identify high-risk areas of code, then use those designations to drive automated checks for changes to those areas of code. The power comes in when you automatically alert the security team that someone has just check in code which changes one of the high-risk areas.

This seems like a commonsense way to focus high attention on high risk.

Another positive benefit of this approach is that it forces the conversation upfront about what constitutes a high-risk area code. This is an opportunity to get security, ops, and the business on the same page about where risk is likely to occur. When you incorporate this approach with periodic reviews of where things stand, it allows you to refine your approach to automated analysis based on your experience. In other words, during post-incident review if you discover one of those”Gee I wish I’d known that I had of time” sorts of situations, you can add a check into the automation library that looks for those conditions in the future. This sort of continuous improvement model benefits everyone.

Oh, by the way – one of the scenarios this approach can check for is the introduction of new third-party libraries. In today’s climate of elevated concern over third-party risk (as well as fears of security flaws in open-source components) having that capability can be a very good thing.

What about you? How are you integrating continuous security checks into your continuous integration?

April 10, 2014
DevOps Needs a Tsunami To Jump The Chasm

All start up business endeavors must go through the phase of crossing the chasm. Most of those business endeavors usually fall prey to the chasm. The chasm is this black hole that everyone claims to understand but no one truly does. We have all seen businesses cross the chasm or fail to cross the chasm and fall into the black hole. But no one seems to understand why and how that crossing happens. The bestselling book by Geoffrey A. Moore titled “Crossing the Chasm,” is all about the heart and soul required to get early stage technology across the chasm, from early adopters to mainstream customers.

There is a big difference between people who are enthusiastic to try leading edge technologies and the rest of the inhabitants, who tend to be much more conservative. The reasons that people spend time trying to figure out how to cross or jump, if you will, the chasm is because that black hole is what stands between them and making their fortune. The early adopters are just a small sample of the market and there is no money in building a business or technology around them. To make real money, you need to cross over into the mainstream.

In Geoff’s model, the innovators are all about technology. You know those people, when a new iphone is released, they spend days standing on line to get latest and greatest new technology. The early adopters are the visionaries that are always on the bleeding edge. The mainstream are the pragmatists, the people who practice risk avoidance. These pragmatists are the key to jumping the black hole, they move the needle.

DevOps first order of business was getting on the curve and to that end DevOps has evangelists. But, understand that what worked in converting these early adopters probably won’t work for the pragmatists. These DevOps evangelists and visionaries want DevOps power tools, as opposed to the pragmatists who simply want to push the button and say “that was easy”.

By all appearances DevOps has been accepted by the early adoption set. Just how does DevOps cross the chasm? Jumping the black hole after all can be quite a daunting task. To do so DevOps has to create what I refer to as the tsunami effect.

The Tsunami effect is where DevOps gains acceptance in multiple industries or markets in what seems like a gold rush, it’s here that fortunes are made. More accurately, it’s where cult communities transform overnight into mainstream communities. Many technologies simply create a single tornado which remains in a silo forever never gaining enough momentum to really emerge. Another common path what I call an earthquake, there is a lot of hype but it tends to fizzle out quickly, if not tended to properly. Right now DevOps has all the appearances of an earthquake, Devops evangelists have created a lot of buzz, but acceptance beyond a set of power tools is still a huge question.

Considering DevOps is in the earthquake phase, as opposed to the tornado phase, DevOps cannot be successful converting one group at a time. The strategy in the tornado phase is convert industries and markets one at a time, via tornados forming within industries or markets. The tactical execution is to get smaller industries or markets committed and therefore the larger industries or markets will then embrace the movement. That is simply crossing the chasm. DevOps has too many cultural implications to convert believers over a long period of time; DevOps requires reorganization, not a subtle shift.

Since I consider DevOps to be in the earthquake phase, crossing the chasm is not an option. DevOps must jump the chasm in a surge of overwhelming momentum, hence creating a tsunami. DevOps has created a very closely knit culture, DevOps needs to push hard to create a tsunami and flood the pragmatic population. But, be advised, DevOps has to be all but universally accepted, there is no almost in bringing together Dev and Ops. But if DevOps wins, it wins big, DevOps will go from being a niche culture to a truly mainstream practice.

So what’s the strategy to create a tsunami? And more importantly, how do you move beyond the earthquake phase? Here’s how I see it according to Geoff, with my DevOps spin: First, DevOps must clearly establish that the mission-critical software development process is broken and is causing management enough pain that it would gamble on unproven methodology that is populated with a bunch of prima donnas. I am not convinced the DevOps cult has clearly established this as fact. Is the development and operations process broken or have just new power tools been created to improve speed and accuracy? Second, DevOps needs to present the total solution, not just tools. Overwhelm the community; each new DevOps tool should leverage the success and technology of the last hence creating a methodology. Even more importantly, because the DevOps tools are closely related, credibility can cross industries and markets and create the momentum required for a tsunami. Once Devops has reached critical mass by providing the total solution in a few industries or markets simultaneously the tsunami is created and DevOps has jumped the chasm.

The tsunami will emerge suddenly with everyone seeming to adopt DevOps all at once. This will result in explosive growth and will pull, willingly or unwillingly, everyone into its path. It’s driven and sustained by the pragmatists. Here, everybody has to get the latest thing all at once; the masses have been converted to DevOps. DevOps needs mass-market success, DevOps needs to become a tsunami and everyone will want to play DevOps.

Now that you know the formula, go forward and create the perfect storm.

If you would like to say something; email parker@devops.com and don’t forget to comment here.

April 10, 2014
My DevOps Bookshelf

Aside from the many technical manuals on our bookshelves, I bet everyone has a few that are more process or philosophically oriented which hold a special place in our hearts. To me, these are the ones we continue to go back to and that have helped develop us as engineers, software, operations, quality assurance or otherwise. In no particular order, the following are the top four, not strictly technical, books in my library.

Web Operations by John Allspaw and Jesse Robbins

Allspaw and Robbins cover many topics in this text ranging from monitoring and metrics to databases and operations development relations. I appreciate this one so much it is required reading material for anyone who joins my operations teams. In my opinion, it illustrates many best practices for an operations organization.

Continuous Delivery by Dave Farley and Jez Humble

This book bridges the gap between good software development practices and getting to continuous delivery. It made me really think about testing and quality assurance, which in my opinion is one of the most difficult parts of a continuous delivery world to wrap your head around. While there isn’t one right answer, you should really think about the role testing plays in the overall process.

The Phoenix Project by Gene Kim, Kevin Behr and George Spafford

“The Pheonix Project” is a relatively new addition to my library. While it may sound strange, this book was an absolute page-turner and I really related to it. It presents many Agile and DevOps concepts via a fictional story that doesn’t seem fictional because we have all been there. Despite the fact that it is fictional, the concepts presented and techniques are applicable to each of our real-world situations.

Etsy – Moving Fast at Scale

While this isn’t actually a book, but a video, to me this is a pivotal piece of information has continued to guide my pursuit of DevOps methodologies and practices, so it holds a special place on my figurative bookshelf. In the video, the guys at Esty discuss a number of topics including deployment processes, application toggles, metrics and more. Most importantly, they not only discuss the why but also the how and I find myself frequently referencing this video.

For me each one of these books or videos has played an important role in pursuit and real-world implementations of DevOps practices. What are the texts digital or otherwise that you hold dear and/or have influenced you?

April 2, 2014
Don’t Fear, Testing Team! DevOps is here!

I love Gene Kim’s analogy about how all unicorns used to be horses; its a great visual that I use all the time when encouraging IT organizations to change from thinking “why not” to “why not us” when it comes to starting their DevOps transformation. If culture is really one of the greatest enemies to faster adoption, then you have to start with where the horses live.

One of the biggest herd of horses in every organization is the team of people dedicated to testing and validating applications. Even though these teams are not being included in the conversation, they are in fact the most pivotal part in the transformation.

A Dangerous Love Triangle

DevOps is touted as the marriage of development and operations; a match made in heaven. One small problem though: in most traditional organizations there is a love triangle between operations, development, and testing, as the flow of bits roll from one team to the other until the application is delivered to production. With the conversation shifting to how can development and operations work better together the testing teams are being alienated. The question is where does a testing team fit into this conversation, when the focus is all about bringing development and operations closer together? How can the testing team assert themselves to help facilitate this transformation?

Horn Growing Engagement

The Phoenix Project is a great narrative that introduces robust DevOps concepts. One cannot go far on devopsy.kinsta.cloud without seeing “The Three Ways” and what they mean. The first “way” is about getting a really good understanding of the delivery chain and how it works as a whole to deliver applications to the end user. The team that knows the most about this is the testing team as they are the glue that binds operations and development together. If you are going to start growing a horn, you need to start here.

Engage with the testing team early and often. This team knows more about the flow of information from development to production than any other team. They understand the process of moving applications from development environments, testing them, and delivering them to production. They understand the infrastructure and how it should work and what operations team needs to deploy applications. This knowledge is the perfect horn-growing fertilizer.

Performance-Driven Feedback

In my last post I talked about how performance and change management were two pillars to start building on. As you start to move from simply understanding the system as a whole and increasing the rate of feedback, performance data is critical. Remember, in science there is no right or wrong answer – only data that says the outcome is desired or not. Performance data can help take the emotion out of determining whether or not processes are going right or wrong after change is made. Most of the performance engineering folks are embedded in the testing groups since this type of expertise is needed to drive a successful DevOps transition. However, be careful as you start to utilize these resources as they have the potential to become your biggest constraints. Protect them and get the mechanics of performance testing and engineering out of their head and into the heads of your development and operations teams.

Continuous Experimentation

Once you have the testing team on board and helping in this process an interesting event should occur. The testing teams will start to embed themselves into Development and Operations. This is when an environment of experimentation becomes possible, just like in a factory, quality is being tested continuously. This leads to a more combined focus on the task at hand while ensuring the quality of the applications and the speed of delivery.

So really the phrase should be: DevOps, don’t fear! The testing team is here!

March 31, 2014
DevOps Lessons Learned – The “DevOps Engineer”

Back in 2008 I was part of a startup that had success and sold to large company three years later. During the startup years, there was three of us in IT plus an offshore team that did our web development. The three core resources owned everything from the API down while the offshore team owned the front end. Since we were small, we each owned our own part of the architecture and managed the deployments and operations of the system with few issues. Once we were purchased, our world changed drastically.

Our team quickly ramped up to over 20 people while my original two architects soon left for greener pastures. Now we had a lot more people involved in the process and none were domain experts. Things quickly started spiraling out of control as server sprawl started creeping in and deployments were becoming increasing challenging and error prone. We hired our first “DevOps” person to help us reign all of this back in. Then things got really ugly. For those who have followed my DevOps writings and my rant called “No you are not a DevOps Engineer“, what I am about to describe is the basis for my position on DevOps.

DevOps as a Role: Fail Pattern

We never intended to create a DevOps silo or position. We just wanted someone with expertise with cloud and automation since everyone else on the team was a developer and many of them were new to both the cloud and the platform. Unintentionally, this person became labelled as DevOps and he was soon joined with three other people which became an official DevOps team. We now were the proud owner of a new silo which became the new bottleneck.

The DevOps team had their hands full to regain control over the environment. Permissions had to be pulled back to prevent server sprawl from continuing. Development teams now had to request environments from the DevOps team. When we were a startup, this was easy and manageable. As the team got larger and more concurrent projects were being run, this became challenging. The DevOps guys were so busy keeping up with the demand that they were not always in tune with the goals and the deliverables of the different projects. The app dev teams were so focused on “agile” that they were not including the DevOps guys early enough in the projects.

The DevOps team became drastically overworked. They were monitoring and operating the systems, they were being pulled from their deliverables to provision environments, and they were trying to put controls around a platform that experienced sudden growth from a 3 person team of founders to a 20+ person team of new people. To make matters worse, we knew what we wanted to do to fix these problems but there were not enough hours in the day to get there. We had mapped out an entire continuous integration and delivery strategy, but we just could not get enough time to focus on it.

DevOps started to get a bad rap from the app dev team because there was often delays in getting environments provisioned in time. We experienced environmental issues between dev, QA, and stage because of the lack of timely communication of the environment requirements from the app dev team and because the different app dev teams were not being consistent (e.g. different version of Python and different libraries).

All of this led to frustration from the business, especially from the business people who came from the startup and was used to frequent deployments with high quality.

Lesson Learned

Eventually the team turned this around. Although they still have an official DevOps team, the team works much closer with the app dev teams. A high level of standardization is now in place for the environments. The teams share operational requirements early in the project life-cycle and developers assist in the environment creation. The lesson learned here is that creating a DevOps role or organization is not the correct approach. Creating a new silo just creates new bottlenecks. DevOps is not a role, it is a way of getting quality, reliable software to market faster. The sooner an organization realizes this the sooner they can get to a successful state.

March 27, 2014
Chuck Norris doesn’t need DevOps, but have we beat “What is DevOps” to death yet?
Some interesting and lots of the usual around the DeOps blog beat last week and over the weekend. An interesting piece by Daniel Bryant on his Tai-Dev Blog on why “Chuck Norris doesn’t do DevOps”. It is actually a video of a presentation he gave at the recent London Java Community event and is the subject of a talk that Bryant has been working on for some time. You can also see the slides from the show on slideshare here.

Chuck Norris doesn’t do DevOps because he doesn’t have to. Chuck can do everything and he can do it by himself. He has no need to work with Dev or Ops, he does it all. But for the rest of us, to get something done we have to work as part of a team. As such we need DevOps to help us work better, faster deeper. So until we are all as good as Chuck Norris, we need to keep at it.

To understand why Chuck Norris doesn’t do DevOps, do we have to also define DevOps? Hasn’t been there enough good descriptions of what DevOps is? Usually when you keep seeing people trying to solve the same issue over and over again, it generally means it hasn’t been done satisfactorily to date. Is this the case with a definition of DevOps? You would think so by how “What is DevOps” blog posts we see.

Vishal Biyani on his blog last week writes exactly that. It is actually part of one of a multi-part series of blog posts that will according to Biyani cover:
- Getting basics right: What is DevOps anyway? (This post)
- It’s all about delivering: CI & CD
- Configuration Chaos : Managing your infrastructure, as code!
- Monitor, measure and move: From logs to performance and beyond
- Putting it all together: Collaborate, communicate and continuously improve
- Further Reading
Giving Biyani credit he refers back to Magnus Hedemark at Redhat. The first part explains the basic premise behind DevOps, a brief history and the motive behind the DevOps movement.

I think this is what we should do more often. Refer back to a well written definition that we can point to with consistency rather than having as many definitions of DevOps as there are stars in the sky. An example is OlinData who back in January wrote a well done “what is DevOps” post. It is no worse than countless others I have seen. The issue, how many more people will write their own definition instead of just pointing at OlinData’s or Dev2Ops.com definition or any others.

As long as we spend so many cycles defining what DevOps is, we are wasting cycles that we could be using on doing DevOps and expanding DevOps knowledge. I think we have beaten the what it is fairly well by now, let’s move on.

An example of moving on is the “BlueMIx” 100 Days announced by IBM and written about on this IBM blog. I will be writing more about IBM and BlueMix, but suffice to say here, IBM is scheduling 100 workshops around the world to help Developers do DevOps with some pretty cool tools from Big Blue. It promises to be a great event if you can get to one in your area.

Another good example of blogs moving DevOps forward is the Ranger4 blog. They had a good post up with the 7 Habits of Highly Effective Continuous Delivery Experts. Good article that take it to the next level.

Anyway, that is my report today. For those that notice these things, I will be handling the Cup o’ Joe blog here for a while, as reading blogs is something I do. You can write me at ashimmy@devops.com if you have something on your blog you want us to mention.
March 24, 2014